Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on kv-engine-jepsen-host-2-ubuntu1804 (jepsen non-vm-host) in workspace /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly The recommended git tool is: NONE No credentials specified > git rev-parse --resolve-git-dir /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/.git # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url ssh://review.couchbase.org:29418/jepsen.couchbase # timeout=10 Fetching upstream changes from ssh://review.couchbase.org:29418/jepsen.couchbase > git --version # timeout=10 > git --version # 'git version 2.17.1' > git fetch --tags --progress -- ssh://review.couchbase.org:29418/jepsen.couchbase +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 Checking out Revision bcf52879039c4f293b7fc0f305221c3d2f918f42 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f bcf52879039c4f293b7fc0f305221c3d2f918f42 # timeout=10 Commit message: "Reorganize .gitignore and exclusions for *.log *.deb *.rpm *.txt *.svg" > git rev-list --no-walk bcf52879039c4f293b7fc0f305221c3d2f918f42 # timeout=10 The recommended git tool is: NONE No credentials specified > git rev-parse bcf52879039c4f293b7fc0f305221c3d2f918f42^{commit} # timeout=10 The recommended git tool is: NONE No credentials specified [GitCheckoutListener] Recording commits of 'git ssh://review.couchbase.org:29418/jepsen.couchbase' [GitCheckoutListener] Found previous build 'kv-engine-cheshire-cat-jepsen-nightly #795' that contains recorded Git commits [GitCheckoutListener] -> Starting recording of new commits since 'bcf52879039c4f293b7fc0f305221c3d2f918f42' [GitCheckoutListener] -> Git commit decorator could not be created for SCM 'hudson.plugins.git.GitSCM@631413ce' [GitCheckoutListener] -> No new commits found [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content CC=/usr/lib/ccache/gcc-8 CXX=/usr/lib/ccache/g++-8 [EnvInject] - Variables injected successfully. [kv-engine-cheshire-cat-jepsen-nightly] $ /bin/bash /tmp/jenkins1970683201362484287.sh Setting NETWORK_INTERFACE to eth0 ------ Testing Jepsen changes against latest build ------ grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-767/20220213T204108.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-767/20220213T235128.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-766/20220212T164745.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-764/20220210T173714.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-775/20220221T220946.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-775/20220221T222230.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-775/20220221T225807.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-775/20220221T201820.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-775/20220221T232712.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-771/20220217T160429.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-786/20220303T235037.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-786/20220303T233927.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-786/20220303T205458.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-786/20220303T232623.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-778/20220224T165937.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-770/20220216T234156.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-770/20220216T225156.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-772/20220218T141333.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-772/20220218T161437.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-772/20220219T004104.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-769/20220215T151327.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-784/20220301T232148.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-784/20220301T231027.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-784/20220301T225648.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-776/20220222T221756.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-776/20220222T221917.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-765/20220211T230358.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-765/20220211T230224.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-765/20220211T160739.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-765/20220211T231503.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-777/20220223T213229.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-777/20220223T214620.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T144355.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T192558.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T145658.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T143202.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T152659.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T173142.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T171105.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T193428.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T093439.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T182722.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T151014.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T172423.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T025153.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T172454.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T152733.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T172136.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T185300.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T152247.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T210140.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T182108.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T174945.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T172520.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T165836.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-783/20220228T174033.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-768/20220215T000131.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-768/20220214T202014.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-768/20220214T185826.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-768/20220214T230625.000-0800/results.edn: No such file or directory grep: ./store/Couchbase-jenkins-kv-engine-cheshire-cat-jepsen-nightly-768/20220215T000006.000-0800/results.edn: No such file or directory Couchbase Server: 7.0.4-7218-1 Using Node IPs 172.23.105.187 172.23.107.228 172.23.106.61 172.23.107.227 ################################################################ Running test suit: suites/kv-engine-cv/cheshire-cat/cv-nightly.conf # Syncwrite Sanity tests workload=register,replicas=2,no-autofailover,rate=0,durability=25:25:25:25 # Syncwrite failover # We're only Sequentially consistent in hard failover situations, so linizrable checker is likely to fail. So just test graceful. workload=failover,failover-type=graceful,recovery-type=delta,replicas=1,no-autofailover,disrupt-count=1,rate=0,durability=25:25:25:25 workload=failover,scopes=3,collections=1,failover-type=graceful,recovery-type=delta,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:0:100:0,cas workload=failover,failover-type=graceful,recovery-type=full,replicas=1,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0,use-json-docs,use-subdoc # Syncwrite rebalance workload=rebalance,scenario=sequential-rebalance-out-in,replicas=1,no-autofailover,disrupt-count=1,rate=0,durability=25:25:25:25,cas workload=rebalance,scenario=sequential-rebalance-out-in,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0 workload=rebalance,scenario=bulk-rebalance-out-in,no-autofailover,replicas=2,disrupt-count=2,kv-timeout=1.5,durability=0:0:100:0,cas workload=rebalance,scenario=swap-rebalance,no-autofailover,replicas=1,disrupt-count=1,kv-timeout=30,durability=0:0:0:100,use-json-docs,doc-padding-size=4096 # Syncwrite partition workload=partition,no-autofailover,replicas=1,disrupt-count=1,disrupt-time=10,kv-timeout=15,durability=0:0:100:0,cas # Syncwrite partition-failover (These tests are commented out till we have a sequential consistency checker as they require hard failover to work) #workload=partition-failover,failover-type=hard,recovery-type=delta,replicas=2,no-autofailover,disrupt-count=1,durability=0:100:0:0 #workload=partition-failover,failover-type=hard,recovery-type=full,replicas=1,no-autofailover,disrupt-count=1,durability=0:0:100:0 # Syncwrite kill workload=kill,scenario=kill-ns-server,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value workload=kill,scopes=1,collections=3,scenario=kill-babysitter,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value,cas workload=kill,scenario=kill-memcached,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value,use-json-docs,doc-padding-size=4096 workload=kill,scenario=hard-reboot,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=full,cas workload=kill,scenario=suspend-process,process-to-suspend=memcached,process-suspend-time=35,replicas=1,no-autofailover,durability=0:0:100:0,eviction-policy=full,cas # Syncwrite set-kill workload=set-kill,scenario=kill-memcached,replicas=1,no-autofailover,durability=0:0:100:0,eviction-policy=full,use-json-docs,doc-padding-size=64 workload=set-kill,scenario=kill-memcached,replicas=1,no-autofailover,durability=0:0:0:100,eviction-policy=value workload=set-kill,scenario=kill-ns-server,replicas=2,no-autofailover,durability=0:100:0:0,eviction-policy=value workload=set-kill,scenario=kill-babysitter,replicas=2,no-autofailover,durability=0:100:0:0,eviction-policy=value workload=set-kill,scenario=hard-reboot,replicas=1,no-autofailover,durability=0:0:100:0,eviction-policy=value workload=set-kill,scenario=suspend-process,process-to-suspend=memcached,process-suspend-time=10,replicas=2,autofailover,durability=0:0:100:0,eviction-policy=value workload=set-kill,scenario=kill-memcached-on-slow-disk,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value,manipulate-disks workload=set-kill,scenario=kill-memcached-on-slow-disk,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=full,manipulate-disks # Ephemeral Syncwrite tests workload=register,replicas=2,no-autofailover,rate=0,durability=50:50:0:0,bucket-type=ephemeral,cas workload=failover,failover-type=hard,recovery-type=delta,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0,bucket-type=ephemeral,cas workload=rebalance,scenario=sequential-rebalance-out-in,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0,bucket-type=ephemeral,cas workload=set-kill,scenario=kill-memcached,replicas=2,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral workload=set-kill,scenario=kill-memcached,replicas=1,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral,use-json-docs,use-subdoc workload=set-kill,scenario=kill-ns-server,replicas=2,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral,use-json-docs,doc-padding-size=64 workload=set-kill,scenario=kill-babysitter,replicas=2,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral # Disk Failure Tests workload=disk-failure,manipulate-disks,replicas=1,durability=0:100:0:0,recovery-type=full,eviction-policy=value,use-json-docs,use-subdoc workload=disk-failure,manipulate-disks,replicas=1,durability=0:0:100:0,recovery-type=delta,eviction-policy=full,cas workload=disk-failure,manipulate-disks,replicas=1,durability=0:0:0:100,recovery-type=delta,eviction-policy=full,cas workload=disk-failure,manipulate-disks,disrupt-count=2,replicas=2,durability=0:0:0:100,recovery-type=delta,eviction-policy=full # Disk Failure and kill Tests workload=kill-n-disk-failure,manipulate-disks,replicas=1,durability=0:0:0:100,recovery-type=delta,eviction-policy=full################################################################ Creating key-pair ./resources/my.key and ./resources/my.key.pub. Key pair already exists. Removing node entries from ~/.ssh/known_hosts. # Host 172.23.105.187 found: line 14 /home/couchbase/.ssh/known_hosts updated. Original contents retained as /home/couchbase/.ssh/known_hosts.old # Host 172.23.107.228 found: line 14 /home/couchbase/.ssh/known_hosts updated. Original contents retained as /home/couchbase/.ssh/known_hosts.old # Host 172.23.106.61 found: line 14 /home/couchbase/.ssh/known_hosts updated. Original contents retained as /home/couchbase/.ssh/known_hosts.old # Host 172.23.107.227 found: line 14 /home/couchbase/.ssh/known_hosts updated. Original contents retained as /home/couchbase/.ssh/known_hosts.old Publishing public key ./resources/my.key.pub to all nodes. ################################################################ Running test: #1 Test params: workload=register,replicas=2,no-autofailover,rate=0,durability=25:25:25:25 ./run.sh: line 163: cd: ./store/latest: No such file or directory Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --package couchbase-server-enterprise_cheshire-cat-ubuntu18.04_amd64.deb --workload=register --replicas=2 --no-autofailover --rate=0 --durability=25:25:25:25 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-1.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 1 unknown: 0 fail: 0 crash: 0 memcached errors (not reported as total): 0 1/1 = 100.00% ################################################################ ################################################################ Running test: #2 Test params: workload=failover,failover-type=graceful,recovery-type=delta,replicas=1,no-autofailover,disrupt-count=1,rate=0,durability=25:25:25:25 Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=failover --failover-type=graceful --recovery-type=delta --replicas=1 --no-autofailover --disrupt-count=1 --rate=0 --durability=25:25:25:25 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-2.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 2 unknown: 0 fail: 0 crash: 0 memcached errors (not reported as total): 0 2/2 = 100.00% ################################################################ ################################################################ Running test: #3 Test params: workload=failover,scopes=3,collections=1,failover-type=graceful,recovery-type=delta,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:0:100:0,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=failover --scopes=3 --collections=1 --failover-type=graceful --recovery-type=delta --replicas=2 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:0:100:0 --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-3.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 3 unknown: 0 fail: 0 crash: 0 memcached errors (not reported as total): 0 3/3 = 100.00% ################################################################ ################################################################ Running test: #4 Test params: workload=failover,failover-type=graceful,recovery-type=full,replicas=1,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0,use-json-docs,use-subdoc Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=failover --failover-type=graceful --recovery-type=full --replicas=1 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:100:0:0 --use-json-docs --use-subdoc --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-4.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 4 unknown: 0 fail: 0 crash: 0 memcached errors (not reported as total): 0 4/4 = 100.00% ################################################################ ################################################################ Running test: #5 Test params: workload=rebalance,scenario=sequential-rebalance-out-in,replicas=1,no-autofailover,disrupt-count=1,rate=0,durability=25:25:25:25,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=rebalance --scenario=sequential-rebalance-out-in --replicas=1 --no-autofailover --disrupt-count=1 --rate=0 --durability=25:25:25:25 --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-5.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 5 unknown: 0 fail: 0 crash: 0 memcached errors (not reported as total): 0 5/5 = 100.00% ################################################################ ################################################################ Running test: #6 Test params: workload=rebalance,scenario=sequential-rebalance-out-in,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0 Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=rebalance --scenario=sequential-rebalance-out-in --replicas=2 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:100:0:0 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-6.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 6 unknown: 0 fail: 0 crash: 0 memcached errors (not reported as total): 0 6/6 = 100.00% ################################################################ ################################################################ Running test: #7 Test params: workload=rebalance,scenario=bulk-rebalance-out-in,no-autofailover,replicas=2,disrupt-count=2,kv-timeout=1.5,durability=0:0:100:0,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=rebalance --scenario=bulk-rebalance-out-in --no-autofailover --replicas=2 --disrupt-count=2 --kv-timeout=1.5 --durability=0:0:100:0 --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-7.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 7 unknown: 0 fail: 0 crash: 0 memcached errors (not reported as total): 0 7/7 = 100.00% ################################################################ ################################################################ Running test: #8 Test params: workload=rebalance,scenario=swap-rebalance,no-autofailover,replicas=1,disrupt-count=1,kv-timeout=30,durability=0:0:0:100,use-json-docs,doc-padding-size=4096 Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-8.log ./run.sh: line 184: 114362 Aborted (core dumped) lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-8.log file to use error_msg_from_memcached.log Critical or Error log messages where found in memcached.log's ******************* memcached errors ************************ ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:05.314399-07:00 CRITICAL *** Fatal error encountered during exception handling *** ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:05.385595-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:983 ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511398-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.4-7218). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/02ab548f-a217-4202-ae2772af-a3a69ae6.dmp before terminating. ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511441-07:00 CRITICAL Stack backtrace of crashed thread: ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511693-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x67f213] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511760-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x6a03ca] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511808-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x6a0708] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511833-07:00 CRITICAL #3 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f7e2ac2e000+0x12980] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511877-07:00 CRITICAL #4 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f7e285cc000+0x3ee87] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511914-07:00 CRITICAL #5 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f7e285cc000+0x407f1] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512014-07:00 CRITICAL #6 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xa1efc] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512075-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x68cfef] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512152-07:00 CRITICAL #8 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xacf96] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512245-07:00 CRITICAL #9 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xad001] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512336-07:00 CRITICAL #10 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xad294] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512381-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x8f847] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512499-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x4b0c35] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512562-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x4b216c] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512610-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x4a4772] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512645-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x4a48f7] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512684-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x4ab945] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512728-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x2aaf37] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512819-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x2a6080] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512889-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x71ad70] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512936-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x702b2a] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512980-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x71dd29] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513025-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x2a475d] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513120-07:00 CRITICAL #23 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xd62f0] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513133-07:00 CRITICAL #24 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f7e2ac2e000+0x76db] ./172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513189-07:00 CRITICAL #25 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f7e285cc000+0x12161f] ************************************************************* 17:05:14.516 [main] INFO jepsen.cli - Test options: {:scopes nil, :hashdump true, :cycles 1, :ssh-private-key "./resources/my.key", :enable-memcached-debug-log-level true, :bucket-type :couchbase, :persist-to 0, :collect-core-files false, :manipulate-disks false, :concurrency 4, :pool-size 6, :autoreprovision-maxnodes 1, :leave-db-running? false, :use-checker nil, :skip-teardown false, :logging-json? false, :dcp-set-read false, :replicate-to 0, :cluster-run false, :cas false, :ssh {:dummy? false, :username "root", :password "root", :strict-host-key-checking false, :private-key-path "./resources/my.key"}, :net-interface "eth0", :collections nil, :doc-padding-size 4096, :use-subdoc false, :argv ("test" "--nodes-file" "./nodes" "--username" "root" "--ssh-private-key=./resources/my.key" "--workload=rebalance" "--scenario=swap-rebalance" "--no-autofailover" "--replicas=1" "--disrupt-count=1" "--kv-timeout=30" "--durability=0:0:0:100" "--use-json-docs" "--doc-padding-size=4096" "--enable-tcp-capture" "--hashdump" "--enable-memcached-debug-log-level" "--net-interface=eth0"), :durability (0 0 0 100), :disable-out-of-order-execution false, :nodes ["172.23.105.187" "172.23.107.228" "172.23.106.61" "172.23.107.227"], :test-count 1, :install-path "/opt/couchbase", :doc-count 30, :process-suspend-time 10, :perf-graphs false, :replicas 1, :autofailover false, :storage-backend :couchstore, :cbcollect true, :connect-timeout 10, :autoreprovision true, :disrupt-count 1, :collect-data-files false, :doc-threads 3, :kv-timeout #object[java.time.Duration 0x478cebdb "PT30S"], :disable-auto-compaction false, :time-limit 60, :use-json-docs true, :eviction-policy "fullEviction", :scenario :swap-rebalance, :workload "rebalance", :recovery-type :delta, :enable-tcp-capture true} INFO [2022-03-13 17:05:14,987] jepsen test runner - jepsen.core Test version bcf52879039c4f293b7fc0f305221c3d2f918f42 INFO [2022-03-13 17:05:14,989] jepsen test runner - jepsen.core Command line: lein run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 INFO [2022-03-13 17:05:15,169] jepsen test runner - jepsen.core Running test: {:scopes nil :remote #jepsen.control.retry.Remote{:remote #jepsen.control.scp.Remote{:cmd-remote #jepsen.control.clj_ssh.Remote{:concurrency-limit 8, :conn-spec nil, :session nil, :semaphore nil}, :conn-spec nil}, :conn nil} :hashdump true :cycles 1 :ssh-private-key "./resources/my.key" :enable-memcached-debug-log-level true :bucket-type :couchbase :persist-to 0 :collect-core-files false :manipulate-disks false :concurrency 90 :pool-size 6 :autoreprovision-maxnodes 1 :db #object[couchbase.core$couchbase_remote$reify__3667 "0x7c751692" "couchbase.core$couchbase_remote$reify__3667@7c751692"] :leave-db-running? false :use-checker nil :skip-teardown false :name "Couchbase" :logging-json? false :dcp-set-read false :start-time #object[org.joda.time.DateTime "0x820b78a" "2022-03-13T17:05:14.000-07:00"] :net #object[jepsen.net$reify__9402 "0x15a41088" "jepsen.net$reify__9402@15a41088"] :client #couchbase.clients.NewRegisterClient{:cluster nil, :bucket nil, :collection nil, :env nil} :replicate-to 0 :barrier #object[java.util.concurrent.CyclicBarrier "0x53335328" "java.util.concurrent.CyclicBarrier@53335328"] :cluster-run false :cas false :pure-generators true :ssh {:dummy? false, :username "root", :password "root", :strict-host-key-checking false, :private-key-path "./resources/my.key"} :net-interface "eth0" :collections nil :doc-padding-size 4096 :checker #object[jepsen.checker$compose$reify__8903 "0xf1e1d66" "jepsen.checker$compose$reify__8903@f1e1d66"] :use-subdoc false :argv ("test" "--nodes-file" "./nodes" "--username" "root" "--ssh-private-key=./resources/my.key" "--workload=rebalance" "--scenario=swap-rebalance" "--no-autofailover" "--replicas=1" "--disrupt-count=1" "--kv-timeout=30" "--durability=0:0:0:100" "--use-json-docs" "--doc-padding-size=4096" "--enable-tcp-capture" "--hashdump" "--enable-memcached-debug-log-level" "--net-interface=eth0") :db-intialized #object[clojure.lang.Atom "0x43edd2ea" {:status :ready, :val false}] :nemesis #unprintable "couchbase.nemesis$couchbase$reify__3984@4cd9ee2c" :durability (0 0 0 100) :disable-out-of-order-execution false :workload-type :register :nodes ["172.23.105.187" "172.23.107.228" "172.23.106.61" "172.23.107.227"] :test-count 1 :install-path "/opt/couchbase" :doc-count 30 :process-suspend-time 10 :perf-graphs false :replicas 1 :autofailover false :storage-backend :couchstore :cbcollect true :connect-timeout 10 :autoreprovision true :disrupt-count 1 :generator (couchbase.workload.Stopable {:gen (jepsen.generator.Any {:gens [(jepsen.generator.OnThreads {:f #{:nemesis}, :gen [[{:type :sleep, :value 5} {:type :info, :f :rebalance-out, :targeter #object[couchbase.nemesis$start_stop_targeter$start_stop_targeter_invoke__3737 "0x47f416d0" "couchbase.nemesis$start_stop_targeter$start_stop_targeter_invoke__3737@47f416d0"], :target-count 1, :target-action :start}] ([{:type :sleep, :value 5} {:type :info, :f :swap-rebalance, :targeter #object[couchbase.nemesis$start_stop_targeter$start_stop_targeter_invoke__3737 "0x47f416d0" "couchbase.nemesis$start_stop_targeter$start_stop_targeter_invoke__3737@47f416d0"], :target-count 1, :target-action :swap}]) {:type :sleep, :value 10} {:type :stop-test}]}) (jepsen.generator.OnThreads {:f #object[clojure.core$complement$fn__5686 "0x4acdf6e7" "clojure.core$complement$fn__5686@4acdf6e7"], :gen (jepsen.generator.OnThreads {:f #object[clojure.core$complement$fn__5686 "0x2de82bf8" "clojure.core$complement$fn__5686@2de82bf8"], :gen (jepsen.independent.ConcurrentGenerator {:n 3, :fgen #object[couchbase.workload$register_client_gen$fn__3426 "0x54ec8ab3" "couchbase.workload$register_client_gen$fn__3426@54ec8ab3"], :group->threads nil, :thread->group nil, :keys (0 1 2 3 4 5 6 7 ...), :gens nil})})})]})}) :collect-data-files false :doc-threads 3 :kv-timeout #object[java.time.Duration "0x478cebdb" "PT30S"] :disable-auto-compaction false :os #object[jepsen.os$reify__2507 "0x1d686622" "jepsen.os$reify__2507@1d686622"] :time-limit 60 :use-json-docs true :eviction-policy "fullEviction" :scenario :swap-rebalance :workload "rebalance" :recovery-type :delta :enable-tcp-capture true} INFO [2022-03-13 17:05:16,423] jepsen test runner - jepsen.db Tearing down DB INFO [2022-03-13 17:05:16,429] jepsen node 172.23.107.227 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:05:16,428] jepsen node 172.23.106.61 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:05:16,429] jepsen node 172.23.105.187 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:05:16,429] jepsen node 172.23.107.228 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:05:18,443] jepsen node 172.23.106.61 - couchbase.util Teardown Complete INFO [2022-03-13 17:05:18,783] jepsen node 172.23.107.228 - couchbase.util Teardown Complete INFO [2022-03-13 17:05:18,992] jepsen node 172.23.105.187 - couchbase.util Teardown Complete INFO [2022-03-13 17:05:19,204] jepsen node 172.23.107.227 - couchbase.util Teardown Complete INFO [2022-03-13 17:05:19,208] jepsen test runner - jepsen.db Setting up DB INFO [2022-03-13 17:05:19,220] jepsen node 172.23.107.227 - couchbase.util Setting up Couchbase Server INFO [2022-03-13 17:05:19,220] jepsen node 172.23.105.187 - couchbase.util Setting up Couchbase Server INFO [2022-03-13 17:05:19,220] jepsen node 172.23.106.61 - couchbase.util Setting up Couchbase Server INFO [2022-03-13 17:05:19,220] jepsen node 172.23.107.228 - couchbase.util Setting up Couchbase Server INFO [2022-03-13 17:05:19,220] jepsen node 172.23.105.187 - couchbase.util Setting up data paths INFO [2022-03-13 17:05:19,220] jepsen node 172.23.107.227 - couchbase.util Setting up data paths INFO [2022-03-13 17:05:19,220] jepsen node 172.23.106.61 - couchbase.util Setting up data paths INFO [2022-03-13 17:05:19,222] jepsen node 172.23.107.228 - couchbase.util Setting up data paths INFO [2022-03-13 17:05:19,323] jepsen node 172.23.107.227 - couchbase.util Changing permissions for /opt/couchbase INFO [2022-03-13 17:05:19,323] jepsen node 172.23.107.228 - couchbase.util Changing permissions for /opt/couchbase INFO [2022-03-13 17:05:19,324] jepsen node 172.23.105.187 - couchbase.util Changing permissions for /opt/couchbase INFO [2022-03-13 17:05:19,323] jepsen node 172.23.106.61 - couchbase.util Changing permissions for /opt/couchbase INFO [2022-03-13 17:05:19,626] jepsen node 172.23.107.228 - couchbase.util Starting daemon INFO [2022-03-13 17:05:19,626] jepsen node 172.23.105.187 - couchbase.util Starting daemon INFO [2022-03-13 17:05:19,626] jepsen node 172.23.107.227 - couchbase.util Starting daemon INFO [2022-03-13 17:05:19,629] jepsen node 172.23.106.61 - couchbase.util Starting daemon INFO [2022-03-13 17:05:42,575] jepsen node 172.23.107.227 - couchbase.util Daemon started INFO [2022-03-13 17:05:42,679] jepsen node 172.23.107.227 - couchbase.util packet dump file name Couchbase-172.23.107.227.pcap INFO [2022-03-13 17:05:44,537] jepsen node 172.23.105.187 - couchbase.util Daemon started INFO [2022-03-13 17:05:44,538] jepsen node 172.23.106.61 - couchbase.util Daemon started INFO [2022-03-13 17:05:44,640] jepsen node 172.23.105.187 - couchbase.util packet dump file name Couchbase-172.23.105.187.pcap INFO [2022-03-13 17:05:44,640] jepsen node 172.23.106.61 - couchbase.util packet dump file name Couchbase-172.23.106.61.pcap INFO [2022-03-13 17:05:45,029] jepsen node 172.23.107.228 - couchbase.util Daemon started INFO [2022-03-13 17:05:45,131] jepsen node 172.23.107.228 - couchbase.util packet dump file name Couchbase-172.23.107.228.pcap INFO [2022-03-13 17:05:45,336] jepsen test runner - jepsen.db Setting up primary 172.23.105.187 INFO [2022-03-13 17:05:45,339] jepsen node 172.23.105.187 - couchbase.util Creating couchbase cluster from 172.23.105.187 INFO [2022-03-13 17:05:45,689] jepsen node 172.23.105.187 - couchbase.util Adding node 172.23.107.228 to cluster INFO [2022-03-13 17:05:56,683] jepsen node 172.23.105.187 - couchbase.util Adding node 172.23.106.61 to cluster INFO [2022-03-13 17:06:04,816] jepsen node 172.23.105.187 - couchbase.util Adding node 172.23.107.227 to cluster INFO [2022-03-13 17:06:12,708] jepsen node 172.23.105.187 - couchbase.util Rebalance status: {} INFO [2022-03-13 17:06:12,709] jepsen node 172.23.105.187 - couchbase.util Rebalance check count: 0 INFO [2022-03-13 17:06:12,709] jepsen node 172.23.105.187 - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:13,730] jepsen node 172.23.105.187 - couchbase.util Rebalance status: {:status none} INFO [2022-03-13 17:06:13,730] jepsen node 172.23.105.187 - couchbase.util Rebalance check count: 1 INFO [2022-03-13 17:06:13,730] jepsen node 172.23.105.187 - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:13,731] jepsen node 172.23.105.187 - couchbase.util Rebalance complete INFO [2022-03-13 17:06:13,829] jepsen node 172.23.105.187 - couchbase.util Waiting for bucket warmup to complete... INFO [2022-03-13 17:06:13,856] jepsen node 172.23.105.187 - couchbase.util Setting memcached log level to debug INFO [2022-03-13 17:06:13,886] jepsen node 172.23.105.187 - couchbase.util Setup complete INFO [2022-03-13 17:06:13,889] jepsen test runner - jepsen.core Relative time begins now INFO [2022-03-13 17:06:13,898] jepsen node 172.23.105.187 - couchbase.cbclients Opening new client INFO [2022-03-13 17:06:16,455] cb-events - com.couchbase.core [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.2.3","clientGitHash":"${buildNumber}","coreVersion":"2.2.3","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.2.3 (Linux 4.15.0-143-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"hasTrustCertificates":false,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":30000,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000,"eventingMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0} {"coreId":"0xa312307a00000001","seedNodes":[{"address":"172.23.107.227"}]} INFO [2022-03-13 17:06:16,463] jepsen node 172.23.105.187 - couchbase.cbclients Checking value of 'com.couchbase.unorderedExecutionEnabled' : nil INFO [2022-03-13 17:06:16,464] jepsen node 172.23.105.187 - couchbase.cbclients Opening new client INFO [2022-03-13 17:06:16,481] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:06:16,493] jepsen node 172.23.105.187 - couchbase.cbclients Checking value of 'com.couchbase.unorderedExecutionEnabled' : nil INFO [2022-03-13 17:06:16,493] jepsen node 172.23.105.187 - couchbase.cbclients Opening new client INFO [2022-03-13 17:06:16,519] jepsen node 172.23.105.187 - couchbase.cbclients Checking value of 'com.couchbase.unorderedExecutionEnabled' : nil INFO [2022-03-13 17:06:16,519] jepsen node 172.23.105.187 - couchbase.cbclients Opening new client INFO [2022-03-13 17:06:16,550] jepsen node 172.23.105.187 - couchbase.cbclients Checking value of 'com.couchbase.unorderedExecutionEnabled' : nil INFO [2022-03-13 17:06:16,551] jepsen node 172.23.105.187 - couchbase.cbclients Opening new client INFO [2022-03-13 17:06:16,572] cb-events - com.couchbase.core [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.2.3","clientGitHash":"${buildNumber}","coreVersion":"2.2.3","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.2.3 (Linux 4.15.0-143-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"hasTrustCertificates":false,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":30000,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000,"eventingMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0} {"coreId":"0xa312307a00000002","seedNodes":[{"address":"172.23.107.228"}]} INFO [2022-03-13 17:06:16,572] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:06:16,592] jepsen node 172.23.105.187 - couchbase.cbclients Checking value of 'com.couchbase.unorderedExecutionEnabled' : nil INFO [2022-03-13 17:06:16,592] jepsen node 172.23.105.187 - couchbase.cbclients Opening new client INFO [2022-03-13 17:06:16,605] cb-events - com.couchbase.core [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.2.3","clientGitHash":"${buildNumber}","coreVersion":"2.2.3","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.2.3 (Linux 4.15.0-143-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"hasTrustCertificates":false,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":30000,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000,"eventingMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0} {"coreId":"0xa312307a00000003","seedNodes":[{"address":"172.23.107.227"}]} INFO [2022-03-13 17:06:16,608] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:06:16,635] jepsen node 172.23.105.187 - couchbase.cbclients Checking value of 'com.couchbase.unorderedExecutionEnabled' : nil INFO [2022-03-13 17:06:16,636] cb-events - com.couchbase.core [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.2.3","clientGitHash":"${buildNumber}","coreVersion":"2.2.3","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.2.3 (Linux 4.15.0-143-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"hasTrustCertificates":false,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":30000,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000,"eventingMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0} {"coreId":"0xa312307a00000004","seedNodes":[{"address":"172.23.107.227"}]} INFO [2022-03-13 17:06:16,637] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:06:16,675] cb-events - com.couchbase.core [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.2.3","clientGitHash":"${buildNumber}","coreVersion":"2.2.3","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.2.3 (Linux 4.15.0-143-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"hasTrustCertificates":false,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":30000,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000,"eventingMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0} {"coreId":"0xa312307a00000005","seedNodes":[{"address":"172.23.106.61"}]} INFO [2022-03-13 17:06:16,675] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:06:16,706] cb-events - com.couchbase.core [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.2.3","clientGitHash":"${buildNumber}","coreVersion":"2.2.3","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.2.3 (Linux 4.15.0-143-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"hasTrustCertificates":false,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":30000,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000,"eventingMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0} {"coreId":"0xa312307a00000006","seedNodes":[{"address":"172.23.107.228"}]} INFO [2022-03-13 17:06:16,707] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:06:17,277] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:06:17,329] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:06:17,330] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:06:17,330] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:06:17,378] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:06:17,379] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:06:17,450] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:06:17,451] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:06:17,476] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:06:17,485] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:06:17,485] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:06:17,513] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:06:17,514] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:06:17,514] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:06:17,514] cb-events - com.couchbase.core [com.couchbase.core][BucketOpenedEvent][982ms] Opened bucket "default" {"coreId":"0xa312307a00000003"} INFO [2022-03-13 17:06:17,538] cb-events - com.couchbase.core [com.couchbase.core][BucketOpenedEvent][896ms] Opened bucket "default" {"coreId":"0xa312307a00000006"} INFO [2022-03-13 17:06:17,551] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:06:17,552] cb-events - com.couchbase.core [com.couchbase.core][BucketOpenedEvent][934ms] Opened bucket "default" {"coreId":"0xa312307a00000004"} INFO [2022-03-13 17:06:17,577] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:06:17,578] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:06:17,578] cb-events - com.couchbase.core [com.couchbase.core][BucketOpenedEvent][1012ms] Opened bucket "default" {"coreId":"0xa312307a00000002"} INFO [2022-03-13 17:06:17,580] cb-events - com.couchbase.core [com.couchbase.core][BucketOpenedEvent][926ms] Opened bucket "default" {"coreId":"0xa312307a00000005"} INFO [2022-03-13 17:06:17,586] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:06:17,687] cb-events - com.couchbase.core [com.couchbase.core][BucketOpenedEvent][1223ms] Opened bucket "default" {"coreId":"0xa312307a00000001"} INFO [2022-03-13 17:06:21,811] jepsen worker 41 - jepsen.util 41 :invoke :read [13 nil] INFO [2022-03-13 17:06:21,814] jepsen worker 3 - jepsen.util 3 :invoke :read [1 nil] INFO [2022-03-13 17:06:21,828] jepsen worker 54 - jepsen.util 54 :invoke :write [18 30] INFO [2022-03-13 17:06:21,839] jepsen worker 39 - jepsen.util 39 :invoke :read [13 nil] INFO [2022-03-13 17:06:21,853] jepsen worker 62 - jepsen.util 62 :invoke :write [20 8] INFO [2022-03-13 17:06:21,862] jepsen worker 13 - jepsen.util 13 :invoke :read [4 nil] INFO [2022-03-13 17:06:21,879] jepsen worker 78 - jepsen.util 78 :invoke :read [26 nil] INFO [2022-03-13 17:06:21,901] jepsen worker 41 - jepsen.util 41 :ok :read [13 :nil] INFO [2022-03-13 17:06:21,906] jepsen worker 39 - jepsen.util 39 :ok :read [13 :nil] INFO [2022-03-13 17:06:21,901] jepsen worker 78 - jepsen.util 78 :ok :read [26 :nil] INFO [2022-03-13 17:06:21,908] jepsen worker 3 - jepsen.util 3 :ok :read [1 :nil] INFO [2022-03-13 17:06:21,910] jepsen worker 13 - jepsen.util 13 :ok :read [4 :nil] INFO [2022-03-13 17:06:21,911] jepsen worker 9 - jepsen.util 9 :invoke :read [3 nil] INFO [2022-03-13 17:06:21,917] jepsen worker 9 - jepsen.util 9 :ok :read [3 :nil] INFO [2022-03-13 17:06:21,922] jepsen worker 41 - jepsen.util 41 :invoke :read [13 nil] INFO [2022-03-13 17:06:21,928] jepsen worker 41 - jepsen.util 41 :ok :read [13 :nil] INFO [2022-03-13 17:06:21,934] jepsen worker 8 - jepsen.util 8 :invoke :write [2 26] INFO [2022-03-13 17:06:21,942] jepsen worker 82 - jepsen.util 82 :invoke :write [27 25] INFO [2022-03-13 17:06:21,949] jepsen worker 36 - jepsen.util 36 :invoke :read [12 nil] INFO [2022-03-13 17:06:21,960] jepsen worker 59 - jepsen.util 59 :invoke :write [19 23] INFO [2022-03-13 17:06:21,967] jepsen worker 36 - jepsen.util 36 :ok :read [12 :nil] INFO [2022-03-13 17:06:21,975] jepsen worker 38 - jepsen.util 38 :invoke :write [12 18] INFO [2022-03-13 17:06:22,005] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:06:22,011] jepsen worker 51 - jepsen.util 51 :ok :read [17 :nil] INFO [2022-03-13 17:06:22,033] jepsen worker 18 - jepsen.util 18 :invoke :read [6 nil] INFO [2022-03-13 17:06:22,045] jepsen worker 18 - jepsen.util 18 :ok :read [6 :nil] INFO [2022-03-13 17:06:22,054] jepsen worker 2 - jepsen.util 2 :invoke :read [0 nil] INFO [2022-03-13 17:06:22,070] jepsen worker 70 - jepsen.util 70 :invoke :read [23 nil] INFO [2022-03-13 17:06:22,074] jepsen worker 2 - jepsen.util 2 :ok :read [0 :nil] INFO [2022-03-13 17:06:22,083] jepsen worker 70 - jepsen.util 70 :ok :read [23 :nil] INFO [2022-03-13 17:06:22,093] jepsen worker 79 - jepsen.util 79 :invoke :write [26 23] INFO [2022-03-13 17:06:22,107] jepsen worker 49 - jepsen.util 49 :invoke :write [16 4] INFO [2022-03-13 17:06:22,126] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:06:22,138] jepsen worker 51 - jepsen.util 51 :ok :read [17 :nil] INFO [2022-03-13 17:06:22,149] jepsen worker 60 - jepsen.util 60 :invoke :write [20 29] INFO [2022-03-13 17:06:22,163] jepsen worker 58 - jepsen.util 58 :invoke :write [19 1] INFO [2022-03-13 17:06:22,196] jepsen worker 85 - jepsen.util 85 :invoke :read [28 nil] INFO [2022-03-13 17:06:22,202] jepsen worker 85 - jepsen.util 85 :ok :read [28 :nil] INFO [2022-03-13 17:06:22,237] jepsen worker 12 - jepsen.util 12 :invoke :read [4 nil] INFO [2022-03-13 17:06:22,244] jepsen worker 12 - jepsen.util 12 :ok :read [4 :nil] INFO [2022-03-13 17:06:22,262] jepsen worker 5 - jepsen.util 5 :invoke :write [1 28] INFO [2022-03-13 17:06:22,282] jepsen worker 80 - jepsen.util 80 :invoke :read [26 nil] INFO [2022-03-13 17:06:22,292] jepsen worker 80 - jepsen.util 80 :ok :read [26 :nil] INFO [2022-03-13 17:06:22,321] jepsen worker 50 - jepsen.util 50 :invoke :read [16 nil] INFO [2022-03-13 17:06:22,331] jepsen worker 50 - jepsen.util 50 :ok :read [16 :nil] INFO [2022-03-13 17:06:22,354] jepsen worker 31 - jepsen.util 31 :invoke :write [10 49] INFO [2022-03-13 17:06:22,361] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:06:22,374] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:06:22,391] jepsen worker 25 - jepsen.util 25 :invoke :read [8 nil] INFO [2022-03-13 17:06:22,418] jepsen worker 25 - jepsen.util 25 :ok :read [8 :nil] INFO [2022-03-13 17:06:22,434] jepsen worker 9 - jepsen.util 9 :invoke :read [3 nil] INFO [2022-03-13 17:06:22,458] jepsen worker 56 - jepsen.util 56 :invoke :write [18 24] INFO [2022-03-13 17:06:22,484] jepsen worker 9 - jepsen.util 9 :ok :read [3 :nil] INFO [2022-03-13 17:06:22,495] jepsen worker 2 - jepsen.util 2 :invoke :read [0 nil] INFO [2022-03-13 17:06:22,505] jepsen worker 2 - jepsen.util 2 :ok :read [0 :nil] INFO [2022-03-13 17:06:22,517] jepsen worker 55 - jepsen.util 55 :invoke :read [18 nil] INFO [2022-03-13 17:06:22,525] jepsen worker 55 - jepsen.util 55 :ok :read [18 :nil] INFO [2022-03-13 17:06:22,682] jepsen worker 73 - jepsen.util 73 :invoke :read [24 nil] INFO [2022-03-13 17:06:22,696] jepsen worker 73 - jepsen.util 73 :ok :read [24 :nil] INFO [2022-03-13 17:06:22,704] jepsen worker 55 - jepsen.util 55 :invoke :write [18 32] INFO [2022-03-13 17:06:22,711] jepsen worker 68 - jepsen.util 68 :invoke :write [22 46] INFO [2022-03-13 17:06:22,778] jepsen worker 78 - jepsen.util 78 :invoke :write [26 15] INFO [2022-03-13 17:06:22,817] jepsen worker 74 - jepsen.util 74 :invoke :read [24 nil] INFO [2022-03-13 17:06:22,841] jepsen worker 9 - jepsen.util 9 :invoke :write [3 23] INFO [2022-03-13 17:06:22,884] jepsen worker 74 - jepsen.util 74 :ok :read [24 :nil] INFO [2022-03-13 17:06:22,887] jepsen worker 57 - jepsen.util 57 :invoke :read [19 nil] INFO [2022-03-13 17:06:22,898] jepsen worker 57 - jepsen.util 57 :ok :read [19 :nil] INFO [2022-03-13 17:06:22,901] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:06:22,907] jepsen worker 51 - jepsen.util 51 :ok :read [17 :nil] INFO [2022-03-13 17:06:22,918] jepsen worker 32 - jepsen.util 32 :invoke :write [10 15] INFO [2022-03-13 17:06:22,928] jepsen worker 6 - jepsen.util 6 :invoke :write [2 44] INFO [2022-03-13 17:06:22,933] jepsen worker 11 - jepsen.util 11 :invoke :write [3 16] INFO [2022-03-13 17:06:22,939] jepsen worker 47 - jepsen.util 47 :invoke :read [15 nil] INFO [2022-03-13 17:06:22,946] jepsen worker 70 - jepsen.util 70 :invoke :write [23 23] INFO [2022-03-13 17:06:22,970] jepsen worker 47 - jepsen.util 47 :ok :read [15 :nil] INFO [2022-03-13 17:06:22,991] jepsen worker 2 - jepsen.util 2 :invoke :write [0 3] INFO [2022-03-13 17:06:23,003] jepsen worker 35 - jepsen.util 35 :invoke :write [11 13] INFO [2022-03-13 17:06:23,026] jepsen worker 50 - jepsen.util 50 :invoke :read [16 nil] INFO [2022-03-13 17:06:23,059] jepsen worker 80 - jepsen.util 80 :invoke :write [26 36] INFO [2022-03-13 17:06:23,074] jepsen worker 72 - jepsen.util 72 :invoke :read [24 nil] INFO [2022-03-13 17:06:23,103] jepsen worker 57 - jepsen.util 57 :invoke :read [19 nil] INFO [2022-03-13 17:06:23,103] jepsen worker 30 - jepsen.util 30 :invoke :write [10 5] INFO [2022-03-13 17:06:23,107] jepsen worker 72 - jepsen.util 72 :ok :read [24 :nil] INFO [2022-03-13 17:06:23,110] jepsen worker 7 - jepsen.util 7 :invoke :read [2 nil] INFO [2022-03-13 17:06:23,164] jepsen worker 50 - jepsen.util 50 :ok :read [16 :nil] INFO [2022-03-13 17:06:23,122] jepsen worker 85 - jepsen.util 85 :invoke :write [28 45] INFO [2022-03-13 17:06:23,116] jepsen worker 25 - jepsen.util 25 :invoke :write [8 42] INFO [2022-03-13 17:06:23,221] jepsen worker 36 - jepsen.util 36 :invoke :write [12 41] INFO [2022-03-13 17:06:23,229] jepsen worker 7 - jepsen.util 7 :ok :read [2 :nil] INFO [2022-03-13 17:06:23,256] jepsen worker 67 - jepsen.util 67 :invoke :read [22 nil] INFO [2022-03-13 17:06:23,303] jepsen worker 57 - jepsen.util 57 :ok :read [19 :nil] INFO [2022-03-13 17:06:23,306] jepsen worker 4 - jepsen.util 4 :invoke :read [1 nil] INFO [2022-03-13 17:06:23,312] jepsen worker 4 - jepsen.util 4 :ok :read [1 :nil] INFO [2022-03-13 17:06:23,366] jepsen worker 67 - jepsen.util 67 :ok :read [22 :nil] INFO [2022-03-13 17:06:23,370] jepsen worker 61 - jepsen.util 61 :invoke :write [20 3] INFO [2022-03-13 17:06:23,377] jepsen worker 1 - jepsen.util 1 :invoke :read [0 nil] INFO [2022-03-13 17:06:23,524] jepsen worker 1 - jepsen.util 1 :ok :read [0 :nil] INFO [2022-03-13 17:06:23,585] jepsen worker 4 - jepsen.util 4 :invoke :write [1 30] INFO [2022-03-13 17:06:23,599] jepsen worker 64 - jepsen.util 64 :invoke :read [21 nil] INFO [2022-03-13 17:06:23,622] jepsen worker 20 - jepsen.util 20 :invoke :read [6 nil] INFO [2022-03-13 17:06:23,627] jepsen worker 48 - jepsen.util 48 :invoke :write [16 14] INFO [2022-03-13 17:06:23,633] jepsen worker 81 - jepsen.util 81 :invoke :read [27 nil] INFO [2022-03-13 17:06:23,682] jepsen worker 88 - jepsen.util 88 :invoke :read [29 nil] INFO [2022-03-13 17:06:23,694] jepsen worker 81 - jepsen.util 81 :ok :read [27 :nil] INFO [2022-03-13 17:06:23,696] jepsen worker 76 - jepsen.util 76 :invoke :write [25 35] INFO [2022-03-13 17:06:23,694] jepsen worker 64 - jepsen.util 64 :ok :read [21 :nil] INFO [2022-03-13 17:06:23,700] jepsen worker 13 - jepsen.util 13 :invoke :write [4 4] INFO [2022-03-13 17:06:23,716] jepsen worker 53 - jepsen.util 53 :invoke :write [17 20] INFO [2022-03-13 17:06:23,742] jepsen worker 18 - jepsen.util 18 :invoke :read [6 nil] INFO [2022-03-13 17:06:23,754] jepsen worker 18 - jepsen.util 18 :ok :read [6 :nil] INFO [2022-03-13 17:06:23,777] jepsen worker 88 - jepsen.util 88 :ok :read [29 :nil] INFO [2022-03-13 17:06:23,812] jepsen worker 20 - jepsen.util 20 :ok :read [6 :nil] INFO [2022-03-13 17:06:23,838] jepsen worker 37 - jepsen.util 37 :invoke :read [12 nil] INFO [2022-03-13 17:06:23,844] jepsen worker 17 - jepsen.util 17 :invoke :write [5 40] INFO [2022-03-13 17:06:23,859] jepsen worker 37 - jepsen.util 37 :ok :read [12 :nil] INFO [2022-03-13 17:06:23,877] jepsen worker 0 - jepsen.util 0 :invoke :read [0 nil] INFO [2022-03-13 17:06:23,891] jepsen worker 0 - jepsen.util 0 :ok :read [0 :nil] INFO [2022-03-13 17:06:23,962] jepsen worker 26 - jepsen.util 26 :invoke :write [8 11] INFO [2022-03-13 17:06:23,965] jepsen worker 23 - jepsen.util 23 :invoke :write [7 43] INFO [2022-03-13 17:06:23,977] jepsen worker 84 - jepsen.util 84 :invoke :write [28 20] INFO [2022-03-13 17:06:24,010] jepsen worker 72 - jepsen.util 72 :invoke :write [24 35] INFO [2022-03-13 17:06:24,030] jepsen worker 3 - jepsen.util 3 :invoke :read [1 nil] INFO [2022-03-13 17:06:24,034] jepsen worker 3 - jepsen.util 3 :ok :read [1 :nil] INFO [2022-03-13 17:06:24,045] jepsen worker 12 - jepsen.util 12 :invoke :read [4 nil] INFO [2022-03-13 17:06:24,049] jepsen worker 12 - jepsen.util 12 :ok :read [4 :nil] INFO [2022-03-13 17:06:24,073] jepsen worker 7 - jepsen.util 7 :invoke :write [2 21] INFO [2022-03-13 17:06:24,144] jepsen worker 69 - jepsen.util 69 :invoke :write [23 17] INFO [2022-03-13 17:06:24,147] jepsen worker 73 - jepsen.util 73 :invoke :read [24 nil] INFO [2022-03-13 17:06:24,151] jepsen worker 73 - jepsen.util 73 :ok :read [24 :nil] INFO [2022-03-13 17:06:24,184] jepsen worker 81 - jepsen.util 81 :invoke :write [27 33] INFO [2022-03-13 17:06:24,280] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:06:24,286] jepsen worker 22 - jepsen.util 22 :invoke :write [7 30] INFO [2022-03-13 17:06:24,288] jepsen worker 29 - jepsen.util 29 :invoke :read [9 nil] INFO [2022-03-13 17:06:24,289] jepsen worker 33 - jepsen.util 33 :invoke :read [11 nil] INFO [2022-03-13 17:06:24,292] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:06:24,293] jepsen worker 16 - jepsen.util 16 :invoke :write [5 10] INFO [2022-03-13 17:06:24,294] jepsen worker 29 - jepsen.util 29 :ok :read [9 :nil] INFO [2022-03-13 17:06:24,311] jepsen worker 33 - jepsen.util 33 :ok :read [11 :nil] INFO [2022-03-13 17:06:24,368] jepsen worker 37 - jepsen.util 37 :invoke :write [12 44] INFO [2022-03-13 17:06:24,372] jepsen worker 73 - jepsen.util 73 :invoke :read [24 nil] INFO [2022-03-13 17:06:24,396] jepsen worker 73 - jepsen.util 73 :ok :read [24 :nil] INFO [2022-03-13 17:06:24,411] jepsen worker 83 - jepsen.util 83 :invoke :read [27 nil] INFO [2022-03-13 17:06:24,415] jepsen worker 83 - jepsen.util 83 :ok :read [27 :nil] INFO [2022-03-13 17:06:24,425] jepsen worker 67 - jepsen.util 67 :invoke :read [22 nil] INFO [2022-03-13 17:06:24,428] jepsen worker 67 - jepsen.util 67 :ok :read [22 :nil] INFO [2022-03-13 17:06:24,451] jepsen worker 41 - jepsen.util 41 :invoke :read [13 nil] INFO [2022-03-13 17:06:24,454] jepsen worker 41 - jepsen.util 41 :ok :read [13 :nil] INFO [2022-03-13 17:06:24,464] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:06:24,468] jepsen worker 15 - jepsen.util 15 :invoke :read [5 nil] INFO [2022-03-13 17:06:24,469] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:06:24,471] jepsen worker 15 - jepsen.util 15 :ok :read [5 :nil] INFO [2022-03-13 17:06:24,492] jepsen worker 57 - jepsen.util 57 :invoke :read [19 nil] INFO [2022-03-13 17:06:24,496] jepsen worker 57 - jepsen.util 57 :ok :read [19 :nil] INFO [2022-03-13 17:06:24,579] jepsen worker 0 - jepsen.util 0 :invoke :write [0 41] INFO [2022-03-13 17:06:24,740] jepsen worker 73 - jepsen.util 73 :invoke :read [24 nil] INFO [2022-03-13 17:06:24,742] jepsen worker 88 - jepsen.util 88 :invoke :read [29 nil] INFO [2022-03-13 17:06:24,744] jepsen worker 86 - jepsen.util 86 :invoke :read [28 nil] INFO [2022-03-13 17:06:24,751] jepsen worker 52 - jepsen.util 52 :invoke :read [17 nil] INFO [2022-03-13 17:06:24,754] jepsen worker 52 - jepsen.util 52 :ok :read [17 :nil] INFO [2022-03-13 17:06:24,757] jepsen worker 50 - jepsen.util 50 :invoke :read [16 nil] INFO [2022-03-13 17:06:24,760] jepsen worker 74 - jepsen.util 74 :invoke :read [24 nil] INFO [2022-03-13 17:06:24,763] jepsen worker 74 - jepsen.util 74 :ok :read [24 :nil] INFO [2022-03-13 17:06:24,816] jepsen worker 73 - jepsen.util 73 :ok :read [24 :nil] INFO [2022-03-13 17:06:24,823] jepsen worker 86 - jepsen.util 86 :ok :read [28 :nil] INFO [2022-03-13 17:06:24,823] jepsen worker 50 - jepsen.util 50 :ok :read [16 :nil] INFO [2022-03-13 17:06:24,823] jepsen worker 88 - jepsen.util 88 :ok :read [29 :nil] INFO [2022-03-13 17:06:24,825] jepsen worker 64 - jepsen.util 64 :invoke :read [21 nil] INFO [2022-03-13 17:06:24,828] jepsen worker 64 - jepsen.util 64 :ok :read [21 :nil] INFO [2022-03-13 17:06:24,942] jepsen worker 29 - jepsen.util 29 :invoke :write [9 31] INFO [2022-03-13 17:06:24,947] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:06:25,031] jepsen worker 87 - jepsen.util 87 :invoke :write [29 49] INFO [2022-03-13 17:06:25,037] jepsen worker 63 - jepsen.util 63 :invoke :write [21 36] INFO [2022-03-13 17:06:25,043] jepsen worker 74 - jepsen.util 74 :invoke :read [24 nil] INFO [2022-03-13 17:06:25,047] jepsen worker 74 - jepsen.util 74 :ok :read [24 :nil] INFO [2022-03-13 17:06:25,048] jepsen worker 57 - jepsen.util 57 :invoke :read [19 nil] INFO [2022-03-13 17:06:25,050] jepsen worker 50 - jepsen.util 50 :invoke :read [16 nil] INFO [2022-03-13 17:06:25,051] jepsen worker 57 - jepsen.util 57 :ok :read [19 :nil] INFO [2022-03-13 17:06:25,051] jepsen worker 10 - jepsen.util 10 :invoke :write [3 49] INFO [2022-03-13 17:06:25,050] jepsen worker 19 - jepsen.util 19 :invoke :read [6 nil] INFO [2022-03-13 17:06:25,059] jepsen worker 50 - jepsen.util 50 :ok :read [16 :nil] INFO [2022-03-13 17:06:25,065] jepsen worker 19 - jepsen.util 19 :ok :read [6 :nil] INFO [2022-03-13 17:06:25,071] jepsen worker 45 - jepsen.util 45 :invoke :read [15 nil] INFO [2022-03-13 17:06:25,075] jepsen worker 45 - jepsen.util 45 :ok :read [15 :nil] INFO [2022-03-13 17:06:25,098] jepsen worker 57 - jepsen.util 57 :invoke :read [19 nil] INFO [2022-03-13 17:06:25,099] jepsen worker 45 - jepsen.util 45 :invoke :write [15 48] INFO [2022-03-13 17:06:25,119] jepsen worker 57 - jepsen.util 57 :ok :read [19 :nil] INFO [2022-03-13 17:06:25,124] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:06:25,128] jepsen worker 73 - jepsen.util 73 :invoke :write [24 6] INFO [2022-03-13 17:06:25,138] jepsen worker 1 - jepsen.util 1 :invoke :read [0 nil] INFO [2022-03-13 17:06:25,141] jepsen worker 1 - jepsen.util 1 :ok :read [0 :nil] INFO [2022-03-13 17:06:25,179] jepsen worker 74 - jepsen.util 74 :invoke :write [24 49] INFO [2022-03-13 17:06:25,183] jepsen worker 21 - jepsen.util 21 :invoke :write [7 39] INFO [2022-03-13 17:06:25,212] jepsen worker 20 - jepsen.util 20 :invoke :read [6 nil] INFO [2022-03-13 17:06:25,215] jepsen worker 20 - jepsen.util 20 :ok :read [6 :nil] INFO [2022-03-13 17:06:25,216] jepsen worker 52 - jepsen.util 52 :invoke :write [17 43] INFO [2022-03-13 17:06:25,249] jepsen worker 67 - jepsen.util 67 :invoke :read [22 nil] INFO [2022-03-13 17:06:25,252] jepsen worker 67 - jepsen.util 67 :ok :read [22 :nil] INFO [2022-03-13 17:06:25,253] jepsen worker 50 - jepsen.util 50 :invoke :write [16 49] INFO [2022-03-13 17:06:25,292] jepsen worker 47 - jepsen.util 47 :invoke :read [15 nil] INFO [2022-03-13 17:06:25,296] jepsen worker 47 - jepsen.util 47 :ok :read [15 :nil] INFO [2022-03-13 17:06:25,297] jepsen worker 27 - jepsen.util 27 :invoke :write [9 36] INFO [2022-03-13 17:06:25,318] jepsen worker 71 - jepsen.util 71 :invoke :write [23 49] INFO [2022-03-13 17:06:25,329] jepsen worker 86 - jepsen.util 86 :invoke :write [28 6] INFO [2022-03-13 17:06:25,332] jepsen worker 28 - jepsen.util 28 :invoke :write [9 14] INFO [2022-03-13 17:06:25,335] jepsen worker 12 - jepsen.util 12 :invoke :read [4 nil] INFO [2022-03-13 17:06:25,343] jepsen worker 12 - jepsen.util 12 :ok :read [4 :nil] INFO [2022-03-13 17:06:25,394] jepsen worker 75 - jepsen.util 75 :invoke :write [25 33] INFO [2022-03-13 17:06:25,414] jepsen worker 42 - jepsen.util 42 :invoke :write [14 45] INFO [2022-03-13 17:06:25,418] jepsen worker 88 - jepsen.util 88 :invoke :write [29 8] INFO [2022-03-13 17:06:25,420] jepsen worker 19 - jepsen.util 19 :invoke :write [6 5] INFO [2022-03-13 17:06:25,463] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:06:25,465] jepsen worker 67 - jepsen.util 67 :invoke :write [22 36] INFO [2022-03-13 17:06:25,467] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:06:25,467] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:06:25,470] jepsen worker 51 - jepsen.util 51 :ok :read [17 :nil] INFO [2022-03-13 17:06:25,495] jepsen worker 15 - jepsen.util 15 :invoke :write [5 44] INFO [2022-03-13 17:06:25,512] jepsen worker 89 - jepsen.util 89 :invoke :read [29 nil] INFO [2022-03-13 17:06:25,519] jepsen worker 89 - jepsen.util 89 :ok :read [29 :nil] INFO [2022-03-13 17:06:25,534] jepsen worker 89 - jepsen.util 89 :invoke :write [29 35] INFO [2022-03-13 17:06:25,556] jepsen worker 1 - jepsen.util 1 :invoke :write [0 46] INFO [2022-03-13 17:06:25,580] jepsen worker 64 - jepsen.util 64 :invoke :write [21 27] INFO [2022-03-13 17:06:25,584] jepsen worker 12 - jepsen.util 12 :invoke :read [4 nil] INFO [2022-03-13 17:06:25,587] jepsen worker 12 - jepsen.util 12 :ok :read [4 :nil] INFO [2022-03-13 17:06:25,589] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:06:25,592] jepsen worker 51 - jepsen.util 51 :ok :read [17 :nil] INFO [2022-03-13 17:06:25,742] jepsen worker 65 - jepsen.util 65 :invoke :read [21 nil] INFO [2022-03-13 17:06:25,742] jepsen worker 34 - jepsen.util 34 :invoke :write [11 47] INFO [2022-03-13 17:06:25,759] jepsen worker 65 - jepsen.util 65 :ok :read [21 :nil] INFO [2022-03-13 17:06:25,800] jepsen worker 24 - jepsen.util 24 :invoke :write [8 33] INFO [2022-03-13 17:06:25,807] jepsen worker 20 - jepsen.util 20 :invoke :read [6 nil] INFO [2022-03-13 17:06:25,809] jepsen worker 77 - jepsen.util 77 :invoke :write [25 39] INFO [2022-03-13 17:06:25,810] jepsen worker 40 - jepsen.util 40 :invoke :write [13 1] INFO [2022-03-13 17:06:25,810] jepsen worker 20 - jepsen.util 20 :ok :read [6 :nil] INFO [2022-03-13 17:06:25,869] jepsen worker 44 - jepsen.util 44 :invoke :write [14 41] INFO [2022-03-13 17:06:25,870] jepsen worker 3 - jepsen.util 3 :invoke :write [1 29] INFO [2022-03-13 17:06:25,872] jepsen worker 51 - jepsen.util 51 :invoke :write [17 8] INFO [2022-03-13 17:06:25,874] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:06:25,884] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:06:25,948] jepsen worker 46 - jepsen.util 46 :invoke :write [15 5] INFO [2022-03-13 17:06:25,949] jepsen worker 43 - jepsen.util 43 :invoke :write [14 9] INFO [2022-03-13 17:06:25,985] jepsen worker 66 - jepsen.util 66 :invoke :write [22 41] INFO [2022-03-13 17:06:25,986] jepsen worker 47 - jepsen.util 47 :invoke :read [15 nil] INFO [2022-03-13 17:06:25,989] jepsen worker 47 - jepsen.util 47 :ok :read [15 :nil] INFO [2022-03-13 17:06:26,032] jepsen worker 18 - jepsen.util 18 :invoke :write [6 13] INFO [2022-03-13 17:06:26,033] jepsen worker 33 - jepsen.util 33 :invoke :read [11 nil] INFO [2022-03-13 17:06:26,034] jepsen worker 57 - jepsen.util 57 :invoke :write [19 49] INFO [2022-03-13 17:06:26,038] jepsen worker 33 - jepsen.util 33 :ok :read [11 :nil] INFO [2022-03-13 17:06:26,059] jepsen worker 41 - jepsen.util 41 :invoke :write [13 22] INFO [2022-03-13 17:06:26,060] jepsen worker 33 - jepsen.util 33 :invoke :write [11 6] INFO [2022-03-13 17:06:26,060] jepsen worker 83 - jepsen.util 83 :invoke :read [27 nil] INFO [2022-03-13 17:06:26,061] jepsen worker 39 - jepsen.util 39 :invoke :read [13 nil] INFO [2022-03-13 17:06:26,063] jepsen worker 83 - jepsen.util 83 :ok :read [27 :nil] INFO [2022-03-13 17:06:26,063] jepsen worker 39 - jepsen.util 39 :ok :read [13 :nil] INFO [2022-03-13 17:06:26,098] jepsen worker 14 - jepsen.util 14 :invoke :write [4 5] INFO [2022-03-13 17:06:26,099] jepsen worker 47 - jepsen.util 47 :invoke :read [15 nil] INFO [2022-03-13 17:06:26,100] jepsen worker 83 - jepsen.util 83 :invoke :read [27 nil] INFO [2022-03-13 17:06:26,100] jepsen worker 65 - jepsen.util 65 :invoke :write [21 49] INFO [2022-03-13 17:06:26,102] jepsen worker 47 - jepsen.util 47 :ok :read [15 :nil] INFO [2022-03-13 17:06:26,102] jepsen worker 83 - jepsen.util 83 :ok :read [27 :nil] INFO [2022-03-13 17:06:26,152] jepsen worker 20 - jepsen.util 20 :invoke :write [6 22] INFO [2022-03-13 17:06:26,153] jepsen worker 83 - jepsen.util 83 :invoke :write [27 11] INFO [2022-03-13 17:06:26,153] jepsen worker 39 - jepsen.util 39 :invoke :write [13 13] INFO [2022-03-13 17:06:26,154] jepsen worker 47 - jepsen.util 47 :invoke :read [15 nil] INFO [2022-03-13 17:06:26,154] jepsen worker 12 - jepsen.util 12 :invoke :write [4 26] INFO [2022-03-13 17:06:26,158] jepsen worker 47 - jepsen.util 47 :ok :read [15 :nil] INFO [2022-03-13 17:06:26,189] jepsen worker 47 - jepsen.util 47 :invoke :write [15 46] INFO [2022-03-13 17:06:26,792] jepsen worker nemesis - jepsen.util :nemesis :info :rebalance-out nil INFO [2022-03-13 17:06:29,155] jepsen worker nemesis - couchbase.util Rebalancing nodes (172.23.107.227) out of cluster INFO [2022-03-13 17:06:29,273] jepsen worker nemesis - couchbase.util Rebalance status: {} INFO [2022-03-13 17:06:29,273] jepsen worker nemesis - couchbase.util Rebalance check count: 0 INFO [2022-03-13 17:06:29,273] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:30,281] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:06:30,282] jepsen worker nemesis - couchbase.util Rebalance check count: 1 INFO [2022-03-13 17:06:30,282] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:31,310] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.0117647058823529}, :ns_1@172.23.107.227 {:progress 0.00390625}, :ns_1@172.23.105.187 {:progress 0.01953125}, :ns_1@172.23.107.228 {:progress 0.02723735408560313}} INFO [2022-03-13 17:06:31,310] jepsen worker nemesis - couchbase.util Rebalance check count: 2 INFO [2022-03-13 17:06:31,310] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:32,319] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07450980392156858}, :ns_1@172.23.107.227 {:progress 0.05859375}, :ns_1@172.23.105.187 {:progress 0.09765625}, :ns_1@172.23.107.228 {:progress 0.08949416342412453}} INFO [2022-03-13 17:06:32,320] jepsen worker nemesis - couchbase.util Rebalance check count: 3 INFO [2022-03-13 17:06:32,320] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:33,329] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1294117647058823}, :ns_1@172.23.107.227 {:progress 0.109375}, :ns_1@172.23.105.187 {:progress 0.15625}, :ns_1@172.23.107.228 {:progress 0.1595330739299611}} INFO [2022-03-13 17:06:33,330] jepsen worker nemesis - couchbase.util Rebalance check count: 4 INFO [2022-03-13 17:06:33,330] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:34,342] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2}, :ns_1@172.23.107.227 {:progress 0.15625}, :ns_1@172.23.105.187 {:progress 0.2265625}, :ns_1@172.23.107.228 {:progress 0.2217898832684825}} INFO [2022-03-13 17:06:34,343] jepsen worker nemesis - couchbase.util Rebalance check count: 5 INFO [2022-03-13 17:06:34,343] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:35,364] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2666666666666667}, :ns_1@172.23.107.227 {:progress 0.19921875}, :ns_1@172.23.105.187 {:progress 0.2890625}, :ns_1@172.23.107.228 {:progress 0.2840466926070039}} INFO [2022-03-13 17:06:35,364] jepsen worker nemesis - couchbase.util Rebalance check count: 6 INFO [2022-03-13 17:06:35,364] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:36,386] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3294117647058824}, :ns_1@172.23.107.227 {:progress 0.25}, :ns_1@172.23.105.187 {:progress 0.42578125}, :ns_1@172.23.107.228 {:progress 0.3540856031128404}} INFO [2022-03-13 17:06:36,386] jepsen worker nemesis - couchbase.util Rebalance check count: 7 INFO [2022-03-13 17:06:36,386] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:37,398] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.388235294117647}, :ns_1@172.23.107.227 {:progress 0.2890625}, :ns_1@172.23.105.187 {:progress 0.51171875}, :ns_1@172.23.107.228 {:progress 0.4124513618677043}} INFO [2022-03-13 17:06:37,398] jepsen worker nemesis - couchbase.util Rebalance check count: 8 INFO [2022-03-13 17:06:37,398] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:38,409] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.4509803921568627}, :ns_1@172.23.107.227 {:progress 0.33984375}, :ns_1@172.23.105.187 {:progress 0.5625}, :ns_1@172.23.107.228 {:progress 0.4824902723735408}} INFO [2022-03-13 17:06:38,409] jepsen worker nemesis - couchbase.util Rebalance check count: 9 INFO [2022-03-13 17:06:38,409] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:39,444] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.5254901960784314}, :ns_1@172.23.107.227 {:progress 0.37890625}, :ns_1@172.23.105.187 {:progress 0.64453125}, :ns_1@172.23.107.228 {:progress 0.5603112840466926}} INFO [2022-03-13 17:06:39,445] jepsen worker nemesis - couchbase.util Rebalance check count: 10 INFO [2022-03-13 17:06:39,445] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:40,467] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.607843137254902}, :ns_1@172.23.107.227 {:progress 0.41796875}, :ns_1@172.23.105.187 {:progress 0.72265625}, :ns_1@172.23.107.228 {:progress 0.6420233463035019}} INFO [2022-03-13 17:06:40,467] jepsen worker nemesis - couchbase.util Rebalance check count: 11 INFO [2022-03-13 17:06:40,467] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:41,482] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.6901960784313725}, :ns_1@172.23.107.227 {:progress 0.45703125}, :ns_1@172.23.105.187 {:progress 0.79296875}, :ns_1@172.23.107.228 {:progress 0.7237354085603113}} INFO [2022-03-13 17:06:41,483] jepsen worker nemesis - couchbase.util Rebalance check count: 12 INFO [2022-03-13 17:06:41,483] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:42,494] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.8}, :ns_1@172.23.107.227 {:progress 0.5}, :ns_1@172.23.105.187 {:progress 0.85546875}, :ns_1@172.23.107.228 {:progress 0.8365758754863813}} INFO [2022-03-13 17:06:42,494] jepsen worker nemesis - couchbase.util Rebalance check count: 13 INFO [2022-03-13 17:06:42,495] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:43,503] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.8196078431372549}, :ns_1@172.23.107.227 {:progress 0.55859375}, :ns_1@172.23.105.187 {:progress 0.8828125}, :ns_1@172.23.107.228 {:progress 0.8560311284046693}} INFO [2022-03-13 17:06:43,504] jepsen worker nemesis - couchbase.util Rebalance check count: 14 INFO [2022-03-13 17:06:43,504] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:44,525] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.8470588235294118}, :ns_1@172.23.107.227 {:progress 0.640625}, :ns_1@172.23.105.187 {:progress 0.9140625}, :ns_1@172.23.107.228 {:progress 0.8793774319066148}} INFO [2022-03-13 17:06:44,526] jepsen worker nemesis - couchbase.util Rebalance check count: 15 INFO [2022-03-13 17:06:44,526] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:45,538] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.8666666666666667}, :ns_1@172.23.107.227 {:progress 0.703125}, :ns_1@172.23.105.187 {:progress 0.93359375}, :ns_1@172.23.107.228 {:progress 0.9027237354085603}} INFO [2022-03-13 17:06:45,538] jepsen worker nemesis - couchbase.util Rebalance check count: 16 INFO [2022-03-13 17:06:45,539] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:46,584] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.8901960784313725}, :ns_1@172.23.107.227 {:progress 0.7734375}, :ns_1@172.23.105.187 {:progress 0.96484375}, :ns_1@172.23.107.228 {:progress 0.9182879377431906}} INFO [2022-03-13 17:06:46,585] jepsen worker nemesis - couchbase.util Rebalance check count: 17 INFO [2022-03-13 17:06:46,585] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:47,600] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.9098039215686274}, :ns_1@172.23.107.227 {:progress 0.8515625}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 0.9416342412451362}} INFO [2022-03-13 17:06:47,600] jepsen worker nemesis - couchbase.util Rebalance check count: 18 INFO [2022-03-13 17:06:47,600] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:48,614] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.9294117647058824}, :ns_1@172.23.107.227 {:progress 0.91796875}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 0.9883268482490273}} INFO [2022-03-13 17:06:48,614] jepsen worker nemesis - couchbase.util Rebalance check count: 19 INFO [2022-03-13 17:06:48,615] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:49,950] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:49,966] jepsen worker nemesis - couchbase.util Rebalance check count: 20 INFO [2022-03-13 17:06:49,966] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:06:50,726] jepsen worker 8 - jepsen.util 8 :ok :write [2 26] INFO [2022-03-13 17:06:50,885] jepsen worker 8 - jepsen.util 8 :invoke :write [2 30] INFO [2022-03-13 17:06:51,024] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:51,025] jepsen worker nemesis - couchbase.util Rebalance check count: 21 INFO [2022-03-13 17:06:51,025] jepsen worker nemesis - couchbase.util Rebalance stuck count: 1 INFO [2022-03-13 17:06:51,362] jepsen worker 8 - jepsen.util 8 :ok :write [2 30] INFO [2022-03-13 17:06:51,368] jepsen worker 8 - jepsen.util 8 :invoke :write [2 36] INFO [2022-03-13 17:06:51,622] jepsen worker 82 - jepsen.util 82 :ok :write [27 25] INFO [2022-03-13 17:06:51,624] jepsen worker 82 - jepsen.util 82 :invoke :write [27 7] INFO [2022-03-13 17:06:51,722] jepsen worker 8 - jepsen.util 8 :ok :write [2 36] INFO [2022-03-13 17:06:51,723] jepsen worker 8 - jepsen.util 8 :invoke :write [2 4] INFO [2022-03-13 17:06:51,943] jepsen worker 8 - jepsen.util 8 :ok :write [2 4] INFO [2022-03-13 17:06:51,955] jepsen worker 8 - jepsen.util 8 :invoke :write [2 1] INFO [2022-03-13 17:06:52,032] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:52,032] jepsen worker nemesis - couchbase.util Rebalance check count: 22 INFO [2022-03-13 17:06:52,032] jepsen worker nemesis - couchbase.util Rebalance stuck count: 2 INFO [2022-03-13 17:06:52,037] jepsen worker 82 - jepsen.util 82 :ok :write [27 7] INFO [2022-03-13 17:06:52,047] jepsen worker 82 - jepsen.util 82 :invoke :read [27 nil] INFO [2022-03-13 17:06:52,194] jepsen worker 8 - jepsen.util 8 :ok :write [2 1] INFO [2022-03-13 17:06:52,197] jepsen worker 8 - jepsen.util 8 :invoke :write [2 4] INFO [2022-03-13 17:06:52,443] jepsen worker 8 - jepsen.util 8 :ok :write [2 4] INFO [2022-03-13 17:06:52,446] jepsen worker 8 - jepsen.util 8 :invoke :read [2 nil] INFO [2022-03-13 17:06:52,491] jepsen worker 54 - jepsen.util 54 :ok :write [18 30] INFO [2022-03-13 17:06:52,492] jepsen worker 54 - jepsen.util 54 :invoke :write [18 34] INFO [2022-03-13 17:06:52,672] jepsen worker 82 - jepsen.util 82 :ok :read [27 7] INFO [2022-03-13 17:06:52,684] jepsen worker 54 - jepsen.util 54 :ok :write [18 34] INFO [2022-03-13 17:06:52,686] jepsen worker 54 - jepsen.util 54 :invoke :read [18 nil] INFO [2022-03-13 17:06:52,690] jepsen worker 82 - jepsen.util 82 :invoke :write [27 48] INFO [2022-03-13 17:06:52,990] jepsen worker 54 - jepsen.util 54 :ok :read [18 34] INFO [2022-03-13 17:06:53,040] jepsen worker 54 - jepsen.util 54 :invoke :write [18 30] INFO [2022-03-13 17:06:53,040] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:53,040] jepsen worker nemesis - couchbase.util Rebalance check count: 23 INFO [2022-03-13 17:06:53,040] jepsen worker nemesis - couchbase.util Rebalance stuck count: 3 INFO [2022-03-13 17:06:53,086] jepsen worker 8 - jepsen.util 8 :ok :read [2 4] INFO [2022-03-13 17:06:53,092] jepsen worker 8 - jepsen.util 8 :invoke :read [2 nil] INFO [2022-03-13 17:06:53,128] jepsen worker 82 - jepsen.util 82 :ok :write [27 48] INFO [2022-03-13 17:06:53,132] jepsen worker 82 - jepsen.util 82 :invoke :write [27 26] INFO [2022-03-13 17:06:53,300] jepsen worker 62 - jepsen.util 62 :ok :write [20 8] INFO [2022-03-13 17:06:53,438] jepsen worker 54 - jepsen.util 54 :ok :write [18 30] INFO [2022-03-13 17:06:53,558] jepsen worker 82 - jepsen.util 82 :ok :write [27 26] INFO [2022-03-13 17:06:53,560] jepsen worker 62 - jepsen.util 62 :invoke :read [20 nil] INFO [2022-03-13 17:06:53,561] jepsen worker 82 - jepsen.util 82 :invoke :write [27 36] INFO [2022-03-13 17:06:53,562] jepsen worker 54 - jepsen.util 54 :invoke :write [18 30] INFO [2022-03-13 17:06:53,636] jepsen worker 62 - jepsen.util 62 :ok :read [20 8] INFO [2022-03-13 17:06:53,637] jepsen worker 62 - jepsen.util 62 :invoke :write [20 3] INFO [2022-03-13 17:06:53,805] jepsen worker 8 - jepsen.util 8 :ok :read [2 4] INFO [2022-03-13 17:06:53,816] jepsen worker 8 - jepsen.util 8 :invoke :read [2 nil] INFO [2022-03-13 17:06:53,995] jepsen worker 8 - jepsen.util 8 :ok :read [2 4] INFO [2022-03-13 17:06:54,027] jepsen worker 8 - jepsen.util 8 :invoke :read [2 nil] INFO [2022-03-13 17:06:54,050] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:54,050] jepsen worker nemesis - couchbase.util Rebalance check count: 24 INFO [2022-03-13 17:06:54,050] jepsen worker nemesis - couchbase.util Rebalance stuck count: 4 INFO [2022-03-13 17:06:54,058] jepsen worker 54 - jepsen.util 54 :ok :write [18 30] INFO [2022-03-13 17:06:54,061] jepsen worker 54 - jepsen.util 54 :invoke :write [18 31] INFO [2022-03-13 17:06:54,217] jepsen worker 8 - jepsen.util 8 :ok :read [2 4] INFO [2022-03-13 17:06:54,241] jepsen worker 8 - jepsen.util 8 :invoke :write [2 5] INFO [2022-03-13 17:06:54,361] jepsen worker 62 - jepsen.util 62 :ok :write [20 3] INFO [2022-03-13 17:06:54,362] jepsen worker 62 - jepsen.util 62 :invoke :write [20 10] INFO [2022-03-13 17:06:54,441] jepsen worker 79 - jepsen.util 79 :ok :write [26 23] INFO [2022-03-13 17:06:54,562] jepsen worker 79 - jepsen.util 79 :invoke :write [26 25] INFO [2022-03-13 17:06:54,583] jepsen worker 82 - jepsen.util 82 :ok :write [27 36] INFO [2022-03-13 17:06:54,587] jepsen worker 82 - jepsen.util 82 :invoke :read [27 nil] INFO [2022-03-13 17:06:54,669] jepsen worker 54 - jepsen.util 54 :ok :write [18 31] INFO [2022-03-13 17:06:54,671] jepsen worker 54 - jepsen.util 54 :invoke :write [18 17] INFO [2022-03-13 17:06:54,802] jepsen worker 82 - jepsen.util 82 :ok :read [27 36] INFO [2022-03-13 17:06:54,807] jepsen worker 82 - jepsen.util 82 :invoke :write [27 28] INFO [2022-03-13 17:06:54,888] jepsen worker 60 - jepsen.util 60 :ok :write [20 29] INFO [2022-03-13 17:06:55,058] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:55,059] jepsen worker nemesis - couchbase.util Rebalance check count: 25 INFO [2022-03-13 17:06:55,059] jepsen worker nemesis - couchbase.util Rebalance stuck count: 5 INFO [2022-03-13 17:06:55,135] jepsen worker 60 - jepsen.util 60 :invoke :write [20 15] INFO [2022-03-13 17:06:55,160] jepsen worker 54 - jepsen.util 54 :ok :write [18 17] INFO [2022-03-13 17:06:55,161] jepsen worker 54 - jepsen.util 54 :invoke :write [18 6] INFO [2022-03-13 17:06:55,163] jepsen worker 79 - jepsen.util 79 :ok :write [26 25] INFO [2022-03-13 17:06:55,163] jepsen worker 8 - jepsen.util 8 :ok :write [2 5] INFO [2022-03-13 17:06:55,164] jepsen worker 79 - jepsen.util 79 :invoke :write [26 11] INFO [2022-03-13 17:06:55,165] jepsen worker 8 - jepsen.util 8 :invoke :write [2 12] WARN [2022-03-13 17:06:55,173] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":202694,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":202694,"total_server_duration_us":107430,"encode_duration_us":1324649,"operation_id":"0x551","timeout_ms":10000,"last_server_duration_us":107430,"total_duration_us":902483}],"total_count":1}} INFO [2022-03-13 17:06:55,526] jepsen worker 38 - jepsen.util 38 :ok :write [12 18] INFO [2022-03-13 17:06:55,528] jepsen worker 38 - jepsen.util 38 :invoke :write [12 46] INFO [2022-03-13 17:06:55,538] jepsen worker 82 - jepsen.util 82 :ok :write [27 28] INFO [2022-03-13 17:06:55,544] jepsen worker 82 - jepsen.util 82 :invoke :read [27 nil] INFO [2022-03-13 17:06:55,641] jepsen worker 82 - jepsen.util 82 :ok :read [27 28] INFO [2022-03-13 17:06:55,666] jepsen worker 82 - jepsen.util 82 :invoke :read [27 nil] INFO [2022-03-13 17:06:55,816] jepsen worker 54 - jepsen.util 54 :ok :write [18 6] INFO [2022-03-13 17:06:55,820] jepsen worker 54 - jepsen.util 54 :invoke :write [18 2] INFO [2022-03-13 17:06:55,846] jepsen worker 82 - jepsen.util 82 :ok :read [27 28] INFO [2022-03-13 17:06:55,846] jepsen worker 82 - jepsen.util 82 :invoke :write [27 8] INFO [2022-03-13 17:06:56,069] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:56,069] jepsen worker nemesis - couchbase.util Rebalance check count: 26 INFO [2022-03-13 17:06:56,069] jepsen worker nemesis - couchbase.util Rebalance stuck count: 6 INFO [2022-03-13 17:06:56,209] jepsen worker 59 - jepsen.util 59 :ok :write [19 23] INFO [2022-03-13 17:06:56,210] jepsen worker 59 - jepsen.util 59 :invoke :read [19 nil] INFO [2022-03-13 17:06:56,212] jepsen worker 68 - jepsen.util 68 :ok :write [22 46] INFO [2022-03-13 17:06:56,221] jepsen worker 68 - jepsen.util 68 :invoke :write [22 40] INFO [2022-03-13 17:06:56,260] jepsen worker 79 - jepsen.util 79 :ok :write [26 11] INFO [2022-03-13 17:06:56,260] jepsen worker 79 - jepsen.util 79 :invoke :read [26 nil] INFO [2022-03-13 17:06:56,342] jepsen worker 59 - jepsen.util 59 :ok :read [19 23] INFO [2022-03-13 17:06:56,391] jepsen worker 59 - jepsen.util 59 :invoke :write [19 24] INFO [2022-03-13 17:06:56,363] jepsen worker 79 - jepsen.util 79 :ok :read [26 11] INFO [2022-03-13 17:06:56,415] jepsen worker 79 - jepsen.util 79 :invoke :write [26 8] WARN [2022-03-13 17:06:56,678] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":581402,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":581402,"total_server_duration_us":560280,"encode_duration_us":397121,"operation_id":"0x62b","timeout_ms":10000,"last_server_duration_us":560280,"total_duration_us":586979}],"total_count":1}} WARN [2022-03-13 17:06:56,689] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":799026,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":799026,"total_server_duration_us":763355,"encode_duration_us":217587,"operation_id":"0x5f5","timeout_ms":10000,"last_server_duration_us":763355,"total_duration_us":803318},{"operation_name":"upsert","last_dispatch_duration_us":497742,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":497742,"total_server_duration_us":483318,"encode_duration_us":227086,"operation_id":"0x62c","timeout_ms":10000,"last_server_duration_us":483318,"total_duration_us":502637}],"total_count":2}} INFO [2022-03-13 17:06:56,707] jepsen worker 54 - jepsen.util 54 :ok :write [18 2] INFO [2022-03-13 17:06:56,709] jepsen worker 38 - jepsen.util 38 :ok :write [12 46] INFO [2022-03-13 17:06:56,711] jepsen worker 38 - jepsen.util 38 :invoke :read [12 nil] INFO [2022-03-13 17:06:56,712] jepsen worker 54 - jepsen.util 54 :invoke :read [18 nil] WARN [2022-03-13 17:06:56,779] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":633384,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":633384,"total_server_duration_us":618768,"encode_duration_us":86230,"operation_id":"0x5f3","timeout_ms":10000,"last_server_duration_us":618768,"total_duration_us":637448}],"total_count":1}} WARN [2022-03-13 17:06:56,781] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":1002211,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1002211,"total_server_duration_us":980103,"encode_duration_us":88800,"operation_id":"0x62e","timeout_ms":10000,"last_server_duration_us":980103,"total_duration_us":1006019},{"operation_name":"upsert","last_dispatch_duration_us":725553,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":725553,"total_server_duration_us":709647,"encode_duration_us":174443,"operation_id":"0x632","timeout_ms":10000,"last_server_duration_us":709647,"total_duration_us":731622}],"total_count":2}} INFO [2022-03-13 17:06:56,800] jepsen worker 54 - jepsen.util 54 :ok :read [18 2] INFO [2022-03-13 17:06:56,834] jepsen worker 54 - jepsen.util 54 :invoke :write [18 47] WARN [2022-03-13 17:06:56,883] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":555649,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":693786,"total_server_duration_us":565788,"encode_duration_us":231606,"operation_id":"0x5ff","timeout_ms":10000,"last_server_duration_us":510517,"total_duration_us":734321}],"total_count":1}} INFO [2022-03-13 17:06:57,078] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:57,078] jepsen worker nemesis - couchbase.util Rebalance check count: 27 INFO [2022-03-13 17:06:57,079] jepsen worker nemesis - couchbase.util Rebalance stuck count: 7 INFO [2022-03-13 17:06:57,852] jepsen worker 62 - jepsen.util 62 :ok :write [20 10] INFO [2022-03-13 17:06:57,856] jepsen worker 62 - jepsen.util 62 :invoke :write [20 15] INFO [2022-03-13 17:06:58,085] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:58,085] jepsen worker nemesis - couchbase.util Rebalance check count: 28 INFO [2022-03-13 17:06:58,085] jepsen worker nemesis - couchbase.util Rebalance stuck count: 8 INFO [2022-03-13 17:06:58,166] jepsen worker 58 - jepsen.util 58 :ok :write [19 1] INFO [2022-03-13 17:06:58,167] jepsen worker 58 - jepsen.util 58 :invoke :write [19 5] INFO [2022-03-13 17:06:59,092] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:06:59,092] jepsen worker nemesis - couchbase.util Rebalance check count: 29 INFO [2022-03-13 17:06:59,092] jepsen worker nemesis - couchbase.util Rebalance stuck count: 9 INFO [2022-03-13 17:06:59,978] jepsen worker 56 - jepsen.util 56 :ok :write [18 24] INFO [2022-03-13 17:06:59,979] jepsen worker 56 - jepsen.util 56 :invoke :write [18 33] INFO [2022-03-13 17:07:00,102] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:00,102] jepsen worker nemesis - couchbase.util Rebalance check count: 30 INFO [2022-03-13 17:07:00,103] jepsen worker nemesis - couchbase.util Rebalance stuck count: 10 INFO [2022-03-13 17:07:00,107] jepsen worker 53 - jepsen.util 53 :ok :write [17 20] INFO [2022-03-13 17:07:00,108] jepsen worker 53 - jepsen.util 53 :invoke :write [17 5] INFO [2022-03-13 17:07:00,128] jepsen worker 8 - jepsen.util 8 :ok :write [2 12] INFO [2022-03-13 17:07:00,131] jepsen worker 8 - jepsen.util 8 :invoke :write [2 26] INFO [2022-03-13 17:07:01,110] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:01,110] jepsen worker nemesis - couchbase.util Rebalance check count: 31 INFO [2022-03-13 17:07:01,110] jepsen worker nemesis - couchbase.util Rebalance stuck count: 11 INFO [2022-03-13 17:07:02,116] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:02,116] jepsen worker nemesis - couchbase.util Rebalance check count: 32 INFO [2022-03-13 17:07:02,117] jepsen worker nemesis - couchbase.util Rebalance stuck count: 12 INFO [2022-03-13 17:07:02,314] jepsen worker 82 - jepsen.util 82 :ok :write [27 8] INFO [2022-03-13 17:07:02,314] jepsen worker 80 - jepsen.util 80 :ok :write [26 36] INFO [2022-03-13 17:07:02,316] jepsen worker 80 - jepsen.util 80 :invoke :read [26 nil] INFO [2022-03-13 17:07:02,317] jepsen worker 82 - jepsen.util 82 :invoke :write [27 40] INFO [2022-03-13 17:07:02,429] jepsen worker 80 - jepsen.util 80 :ok :read [26 36] INFO [2022-03-13 17:07:02,431] jepsen worker 80 - jepsen.util 80 :invoke :read [26 nil] INFO [2022-03-13 17:07:02,494] jepsen worker 80 - jepsen.util 80 :ok :read [26 36] INFO [2022-03-13 17:07:02,494] jepsen worker 80 - jepsen.util 80 :invoke :read [26 nil] INFO [2022-03-13 17:07:02,519] jepsen worker 80 - jepsen.util 80 :ok :read [26 36] INFO [2022-03-13 17:07:02,519] jepsen worker 80 - jepsen.util 80 :invoke :write [26 35] INFO [2022-03-13 17:07:03,127] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:03,127] jepsen worker nemesis - couchbase.util Rebalance check count: 33 INFO [2022-03-13 17:07:03,127] jepsen worker nemesis - couchbase.util Rebalance stuck count: 13 INFO [2022-03-13 17:07:04,133] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:04,134] jepsen worker nemesis - couchbase.util Rebalance check count: 34 INFO [2022-03-13 17:07:04,134] jepsen worker nemesis - couchbase.util Rebalance stuck count: 14 INFO [2022-03-13 17:07:05,140] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:05,140] jepsen worker nemesis - couchbase.util Rebalance check count: 35 INFO [2022-03-13 17:07:05,141] jepsen worker nemesis - couchbase.util Rebalance stuck count: 15 WARN [2022-03-13 17:07:05,183] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":4678572,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":4678572,"total_server_duration_us":4658092,"encode_duration_us":259302,"operation_id":"0x631","timeout_ms":10000,"last_server_duration_us":4658092,"total_duration_us":4684068},{"operation_name":"upsert","last_dispatch_duration_us":2815915,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2815915,"total_server_duration_us":1921248,"encode_duration_us":377888,"operation_id":"0x630","timeout_ms":10000,"last_server_duration_us":1921248,"total_duration_us":2820059},{"operation_name":"upsert","last_dispatch_duration_us":1526910,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1526910,"total_server_duration_us":1503378,"encode_duration_us":21924,"operation_id":"0x622","timeout_ms":10000,"last_server_duration_us":1503378,"total_duration_us":1532461},{"operation_name":"upsert","last_dispatch_duration_us":869392,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":869392,"total_server_duration_us":850317,"encode_duration_us":11704,"operation_id":"0x65c","timeout_ms":10000,"last_server_duration_us":850317,"total_duration_us":874385},{"operation_name":"upsert","last_dispatch_duration_us":635473,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":635473,"total_server_duration_us":622165,"encode_duration_us":281013,"operation_id":"0x60e","timeout_ms":10000,"last_server_duration_us":622165,"total_duration_us":640699},{"operation_name":"upsert","last_dispatch_duration_us":547811,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":547811,"total_server_duration_us":530059,"encode_duration_us":101882,"operation_id":"0x62f","timeout_ms":10000,"last_server_duration_us":530059,"total_duration_us":552830}],"total_count":6}} INFO [2022-03-13 17:07:05,237] jepsen worker 60 - jepsen.util 60 :info :write [20 15] :RequestTimeoutException INFO [2022-03-13 17:07:05,238] jepsen worker 60 - jepsen.util 150 :invoke :read [20 nil] INFO [2022-03-13 17:07:05,332] jepsen worker 76 - jepsen.util 76 :info :write [25 35] :SyncWriteAmbiguous INFO [2022-03-13 17:07:05,334] jepsen worker 76 - jepsen.util 166 :invoke :write [25 28] INFO [2022-03-13 17:07:05,466] jepsen worker 68 - jepsen.util 68 :info :write [22 40] :SyncWriteAmbiguous INFO [2022-03-13 17:07:05,467] jepsen worker 68 - jepsen.util 158 :invoke :read [22 nil] INFO [2022-03-13 17:07:05,559] jepsen worker 68 - jepsen.util 158 :ok :read [22 46] INFO [2022-03-13 17:07:05,559] jepsen worker 68 - jepsen.util 158 :invoke :read [22 nil] INFO [2022-03-13 17:07:05,660] jepsen worker 68 - jepsen.util 158 :ok :read [22 46] INFO [2022-03-13 17:07:05,661] jepsen worker 68 - jepsen.util 158 :invoke :write [22 48] INFO [2022-03-13 17:07:05,698] jepsen worker 49 - jepsen.util 49 :info :write [16 4] :RequestTimeoutException INFO [2022-03-13 17:07:05,699] jepsen worker 49 - jepsen.util 139 :invoke :write [16 25] INFO [2022-03-13 17:07:05,708] jepsen worker 6 - jepsen.util 6 :info :write [2 44] :RequestTimeoutException INFO [2022-03-13 17:07:05,709] jepsen worker 6 - jepsen.util 96 :invoke :write [2 0] INFO [2022-03-13 17:07:05,786] jepsen worker 2 - jepsen.util 2 :info :write [0 3] :SyncWriteAmbiguous INFO [2022-03-13 17:07:05,787] jepsen worker 38 - jepsen.util 38 :ok :read [12 46] INFO [2022-03-13 17:07:05,787] jepsen worker 2 - jepsen.util 92 :invoke :read [0 nil] INFO [2022-03-13 17:07:05,788] jepsen worker 38 - jepsen.util 38 :invoke :write [12 40] WARN [2022-03-13 17:07:05,897] cb-events - com.couchbase.endpoint [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"bucket":"default","channelId":"A312307A00000001/000000009880CB75","circuitBreaker":"DISABLED","coreId":"0xa312307a00000001","local":"172.23.104.79:33114","remote":"172.23.107.227:11210","type":"KV"} WARN [2022-03-13 17:07:05,898] cb-events - com.couchbase.endpoint [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"bucket":"default","channelId":"A312307A00000004/00000000C91F8C95","circuitBreaker":"DISABLED","coreId":"0xa312307a00000004","local":"172.23.104.79:33126","remote":"172.23.107.227:11210","type":"KV"} WARN [2022-03-13 17:07:05,898] cb-events - com.couchbase.endpoint [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"bucket":"default","channelId":"A312307A00000006/0000000089EF1B1F","circuitBreaker":"DISABLED","coreId":"0xa312307a00000006","local":"172.23.104.79:33182","remote":"172.23.107.227:11210","type":"KV"} WARN [2022-03-13 17:07:05,898] cb-events - com.couchbase.endpoint [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"bucket":"default","channelId":"A312307A00000005/0000000077FF400F","circuitBreaker":"DISABLED","coreId":"0xa312307a00000005","local":"172.23.104.79:33170","remote":"172.23.107.227:11210","type":"KV"} WARN [2022-03-13 17:07:05,904] cb-events - com.couchbase.endpoint [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"bucket":"default","channelId":"A312307A00000002/0000000010EEC5D3","circuitBreaker":"DISABLED","coreId":"0xa312307a00000002","local":"172.23.104.79:33176","remote":"172.23.107.227:11210","type":"KV"} WARN [2022-03-13 17:07:05,905] cb-events - com.couchbase.endpoint [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {"bucket":"default","channelId":"A312307A00000003/0000000095A95C5F","circuitBreaker":"DISABLED","coreId":"0xa312307a00000003","local":"172.23.104.79:33120","remote":"172.23.107.227:11210","type":"KV"} INFO [2022-03-13 17:07:06,155] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:06,155] jepsen worker nemesis - couchbase.util Rebalance check count: 36 INFO [2022-03-13 17:07:06,155] jepsen worker nemesis - couchbase.util Rebalance stuck count: 16 INFO [2022-03-13 17:07:06,402] jepsen worker 5 - jepsen.util 5 :ok :write [1 28] INFO [2022-03-13 17:07:06,402] jepsen worker 5 - jepsen.util 5 :invoke :read [1 nil] INFO [2022-03-13 17:07:06,474] jepsen worker 20 - jepsen.util 20 :info :write [6 22] :SyncWriteAmbiguous INFO [2022-03-13 17:07:06,475] jepsen worker 20 - jepsen.util 110 :invoke :read [6 nil] INFO [2022-03-13 17:07:06,536] jepsen worker 9 - jepsen.util 9 :info :write [3 23] :SyncWriteAmbiguous INFO [2022-03-13 17:07:06,537] jepsen worker 9 - jepsen.util 99 :invoke :write [3 5] INFO [2022-03-13 17:07:06,542] jepsen worker 20 - jepsen.util 110 :ok :read [6 :nil] INFO [2022-03-13 17:07:06,543] jepsen worker 20 - jepsen.util 110 :invoke :read [6 nil] INFO [2022-03-13 17:07:06,688] jepsen worker 79 - jepsen.util 79 :info :write [26 8] :RequestTimeoutException INFO [2022-03-13 17:07:06,692] jepsen worker 79 - jepsen.util 169 :invoke :write [26 4] WARN [2022-03-13 17:07:06,700] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10057662,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10057662,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9027816,"encode_duration_us":143224,"operation_id":"0x654","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10005638}],"total_count":1}} WARN [2022-03-13 17:07:06,701] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":34786,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":5115960,"total_server_duration_us":341248,"encode_duration_us":94598,"operation_id":"0x655","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10006538},{"operation_name":"upsert","last_dispatch_duration_us":10057662,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10057662,"total_server_duration_us":9027816,"encode_duration_us":143224,"operation_id":"0x654","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10005638},{"operation_name":"get","last_dispatch_duration_us":9056476,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9056476,"total_server_duration_us":100,"operation_id":"0x683","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":9058332},{"operation_name":"upsert","last_dispatch_duration_us":9031680,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9031680,"total_server_duration_us":9019331,"encode_duration_us":99524,"operation_id":"0x691","timeout_ms":10000,"last_server_duration_us":9019331,"total_duration_us":9036746},{"operation_name":"upsert","last_dispatch_duration_us":4953382,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":4953382,"total_server_duration_us":4941897,"encode_duration_us":112018,"operation_id":"0x690","timeout_ms":10000,"last_server_duration_us":4941897,"total_duration_us":4956505},{"operation_name":"upsert","last_dispatch_duration_us":1163679,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1163679,"total_server_duration_us":1143074,"encode_duration_us":11779,"operation_id":"0x643","timeout_ms":10000,"last_server_duration_us":1143074,"total_duration_us":1169000}],"total_count":6}} INFO [2022-03-13 17:07:06,707] jepsen worker 20 - jepsen.util 110 :ok :read [6 :nil] WARN [2022-03-13 17:07:06,708] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9050561,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9050561,"total_server_duration_us":9034183,"encode_duration_us":58519,"operation_id":"0x685","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":9055023},{"operation_name":"upsert","last_dispatch_duration_us":6195356,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":6195356,"total_server_duration_us":6178259,"encode_duration_us":267875,"operation_id":"0x65e","timeout_ms":10000,"last_server_duration_us":6178259,"total_duration_us":6199644}],"total_count":2}} WARN [2022-03-13 17:07:06,708] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9045331,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9045331,"total_server_duration_us":9028877,"encode_duration_us":296970,"operation_id":"0x661","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":9051598},{"operation_name":"upsert","last_dispatch_duration_us":2712336,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2925331,"total_server_duration_us":2822670,"encode_duration_us":70490,"operation_id":"0x601","timeout_ms":10000,"last_server_duration_us":2701067,"total_duration_us":3418471}],"total_count":2}} INFO [2022-03-13 17:07:06,712] jepsen worker 20 - jepsen.util 110 :invoke :write [6 24] INFO [2022-03-13 17:07:06,796] jepsen worker 59 - jepsen.util 59 :info :write [19 24] :RequestTimeoutException INFO [2022-03-13 17:07:06,798] jepsen worker 59 - jepsen.util 149 :invoke :write [19 23] WARN [2022-03-13 17:07:06,800] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":3452717,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3452717,"total_server_duration_us":20921,"encode_duration_us":256290,"operation_id":"0x682","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10001461},{"operation_name":"upsert","last_dispatch_duration_us":8892784,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8892784,"total_server_duration_us":8881921,"encode_duration_us":148253,"operation_id":"0x699","timeout_ms":10000,"last_server_duration_us":8881921,"total_duration_us":8896798},{"operation_name":"upsert","last_dispatch_duration_us":4051921,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4051921,"total_server_duration_us":4036425,"encode_duration_us":126750,"operation_id":"0x65d","timeout_ms":10000,"last_server_duration_us":4036425,"total_duration_us":4057592},{"operation_name":"upsert","last_dispatch_duration_us":3268207,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":3555614,"total_server_duration_us":3308369,"encode_duration_us":124357,"operation_id":"0x662","timeout_ms":10000,"last_server_duration_us":3249807,"total_duration_us":3590638}],"total_count":4}} INFO [2022-03-13 17:07:06,816] jepsen worker 26 - jepsen.util 26 :info :write [8 11] :SyncWriteAmbiguous INFO [2022-03-13 17:07:06,817] jepsen worker 26 - jepsen.util 116 :invoke :read [8 nil] INFO [2022-03-13 17:07:06,872] jepsen worker 70 - jepsen.util 70 :info :write [23 23] :RequestTimeoutException INFO [2022-03-13 17:07:06,873] jepsen worker 70 - jepsen.util 160 :invoke :write [23 17] WARN [2022-03-13 17:07:06,900] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":66092,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":637614,"total_server_duration_us":290828,"encode_duration_us":82213,"operation_id":"0x62d","timeout_ms":10000,"last_server_duration_us":38274,"total_duration_us":10017110},{"operation_name":"upsert","last_dispatch_duration_us":9058308,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9058308,"total_server_duration_us":9030999,"encode_duration_us":119172,"operation_id":"0x692","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":9061677}],"total_count":2}} INFO [2022-03-13 17:07:07,022] jepsen worker 54 - jepsen.util 54 :info :write [18 47] :RequestTimeoutException INFO [2022-03-13 17:07:07,023] jepsen worker 54 - jepsen.util 144 :invoke :write [18 28] INFO [2022-03-13 17:07:07,147] jepsen worker 13 - jepsen.util 13 :info :write [4 4] :RequestTimeoutException INFO [2022-03-13 17:07:07,147] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:07,172] jepsen worker 35 - jepsen.util 35 :info :write [11 13] :SyncWriteAmbiguous INFO [2022-03-13 17:07:07,173] jepsen worker 35 - jepsen.util 125 :invoke :write [11 36] INFO [2022-03-13 17:07:07,200] jepsen worker 31 - jepsen.util 31 :info :write [10 49] :SyncWriteAmbiguous INFO [2022-03-13 17:07:07,203] jepsen worker 31 - jepsen.util 121 :invoke :read [10 nil] INFO [2022-03-13 17:07:07,468] jepsen worker 2 - jepsen.util 92 :ok :read [0 :nil] INFO [2022-03-13 17:07:07,468] jepsen worker 2 - jepsen.util 92 :invoke :read [0 nil] INFO [2022-03-13 17:07:07,497] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:07,498] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:07,501] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:07,501] jepsen worker nemesis - couchbase.util Rebalance check count: 37 INFO [2022-03-13 17:07:07,501] jepsen worker nemesis - couchbase.util Rebalance stuck count: 17 INFO [2022-03-13 17:07:07,601] jepsen worker 2 - jepsen.util 92 :ok :read [0 :nil] INFO [2022-03-13 17:07:07,602] jepsen worker 2 - jepsen.util 92 :invoke :read [0 nil] INFO [2022-03-13 17:07:07,636] jepsen worker 5 - jepsen.util 5 :ok :read [1 28] INFO [2022-03-13 17:07:07,637] jepsen worker 5 - jepsen.util 5 :invoke :read [1 nil] INFO [2022-03-13 17:07:07,663] jepsen worker 23 - jepsen.util 23 :info :write [7 43] :RequestTimeoutException INFO [2022-03-13 17:07:07,663] jepsen worker 23 - jepsen.util 113 :invoke :write [7 41] INFO [2022-03-13 17:07:07,672] jepsen worker 25 - jepsen.util 25 :info :write [8 42] :RequestTimeoutException INFO [2022-03-13 17:07:07,672] jepsen worker 25 - jepsen.util 115 :invoke :write [8 24] INFO [2022-03-13 17:07:07,688] jepsen worker 2 - jepsen.util 92 :ok :read [0 :nil] INFO [2022-03-13 17:07:07,689] jepsen worker 2 - jepsen.util 92 :invoke :read [0 nil] INFO [2022-03-13 17:07:07,739] jepsen worker 17 - jepsen.util 17 :info :write [5 40] :SyncWriteAmbiguous INFO [2022-03-13 17:07:07,740] jepsen worker 17 - jepsen.util 107 :invoke :write [5 0] INFO [2022-03-13 17:07:07,748] jepsen worker 55 - jepsen.util 55 :info :write [18 32] :RequestTimeoutException INFO [2022-03-13 17:07:07,749] jepsen worker 55 - jepsen.util 145 :invoke :read [18 nil] INFO [2022-03-13 17:07:07,764] jepsen worker 5 - jepsen.util 5 :ok :read [1 28] INFO [2022-03-13 17:07:07,765] jepsen worker 5 - jepsen.util 5 :invoke :read [1 nil] INFO [2022-03-13 17:07:07,779] jepsen worker 2 - jepsen.util 92 :ok :read [0 :nil] INFO [2022-03-13 17:07:07,780] jepsen worker 2 - jepsen.util 92 :invoke :read [0 nil] INFO [2022-03-13 17:07:07,850] jepsen worker 2 - jepsen.util 92 :ok :read [0 :nil] INFO [2022-03-13 17:07:07,851] jepsen worker 2 - jepsen.util 92 :invoke :write [0 1] INFO [2022-03-13 17:07:07,868] jepsen worker 4 - jepsen.util 4 :info :write [1 30] :RequestTimeoutException INFO [2022-03-13 17:07:07,869] jepsen worker 4 - jepsen.util 94 :invoke :write [1 33] INFO [2022-03-13 17:07:07,873] jepsen worker 62 - jepsen.util 62 :info :write [20 15] :RequestTimeoutException INFO [2022-03-13 17:07:07,873] jepsen worker 5 - jepsen.util 5 :ok :read [1 28] INFO [2022-03-13 17:07:07,874] jepsen worker 5 - jepsen.util 5 :invoke :write [1 32] INFO [2022-03-13 17:07:07,874] jepsen worker 62 - jepsen.util 152 :invoke :write [20 35] INFO [2022-03-13 17:07:07,924] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:07,924] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:07,968] jepsen worker 56 - jepsen.util 56 :ok :write [18 33] INFO [2022-03-13 17:07:07,969] jepsen worker 56 - jepsen.util 56 :invoke :write [18 12] INFO [2022-03-13 17:07:07,970] jepsen worker 86 - jepsen.util 86 :info :write [28 6] :RequestTimeoutException INFO [2022-03-13 17:07:07,971] jepsen worker 86 - jepsen.util 176 :invoke :read [28 nil] INFO [2022-03-13 17:07:07,988] jepsen worker 30 - jepsen.util 30 :info :write [10 5] :RequestTimeoutException INFO [2022-03-13 17:07:07,988] jepsen worker 30 - jepsen.util 120 :invoke :read [10 nil] INFO [2022-03-13 17:07:08,020] jepsen worker 55 - jepsen.util 145 :ok :read [18 33] INFO [2022-03-13 17:07:08,020] jepsen worker 55 - jepsen.util 145 :invoke :read [18 nil] INFO [2022-03-13 17:07:08,068] jepsen worker 78 - jepsen.util 78 :info :write [26 15] :RequestTimeoutException INFO [2022-03-13 17:07:08,069] jepsen worker 78 - jepsen.util 168 :invoke :write [26 16] INFO [2022-03-13 17:07:08,129] jepsen worker 55 - jepsen.util 145 :ok :read [18 33] INFO [2022-03-13 17:07:08,130] jepsen worker 55 - jepsen.util 145 :invoke :read [18 nil] INFO [2022-03-13 17:07:08,138] jepsen worker 73 - jepsen.util 73 :info :write [24 6] :RequestTimeoutException INFO [2022-03-13 17:07:08,138] jepsen worker 73 - jepsen.util 163 :invoke :write [24 24] INFO [2022-03-13 17:07:08,160] jepsen worker 55 - jepsen.util 145 :ok :read [18 33] INFO [2022-03-13 17:07:08,161] jepsen worker 55 - jepsen.util 145 :invoke :write [18 2] INFO [2022-03-13 17:07:08,199] jepsen worker 48 - jepsen.util 48 :info :write [16 14] :RequestTimeoutException INFO [2022-03-13 17:07:08,200] jepsen worker 48 - jepsen.util 138 :invoke :write [16 27] INFO [2022-03-13 17:07:08,223] jepsen worker 36 - jepsen.util 36 :info :write [12 41] :SyncWriteAmbiguous INFO [2022-03-13 17:07:08,224] jepsen worker 36 - jepsen.util 126 :invoke :write [12 37] INFO [2022-03-13 17:07:08,263] jepsen worker 58 - jepsen.util 58 :info :write [19 5] :RequestTimeoutException INFO [2022-03-13 17:07:08,264] jepsen worker 58 - jepsen.util 148 :invoke :read [19 nil] INFO [2022-03-13 17:07:08,315] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:08,318] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:08,379] jepsen worker 72 - jepsen.util 72 :info :write [24 35] :SyncWriteAmbiguous INFO [2022-03-13 17:07:08,381] jepsen worker 72 - jepsen.util 162 :invoke :write [24 34] INFO [2022-03-13 17:07:08,513] jepsen worker 61 - jepsen.util 61 :info :write [20 3] :RequestTimeoutException INFO [2022-03-13 17:07:08,516] jepsen worker 61 - jepsen.util 151 :invoke :write [20 36] INFO [2022-03-13 17:07:08,527] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:08,530] jepsen worker 13 - jepsen.util 103 :invoke :write [4 22] INFO [2022-03-13 17:07:08,669] jepsen worker 85 - jepsen.util 85 :info :write [28 45] :RequestTimeoutException INFO [2022-03-13 17:07:08,671] jepsen worker 85 - jepsen.util 175 :invoke :write [28 29] INFO [2022-03-13 17:07:08,823] jepsen worker 81 - jepsen.util 81 :info :write [27 33] :RequestTimeoutException INFO [2022-03-13 17:07:08,823] jepsen worker 81 - jepsen.util 171 :invoke :read [27 nil] INFO [2022-03-13 17:07:08,926] jepsen worker 32 - jepsen.util 32 :info :write [10 15] :RequestTimeoutException INFO [2022-03-13 17:07:08,927] jepsen worker 32 - jepsen.util 122 :invoke :read [10 nil] INFO [2022-03-13 17:07:08,941] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:08,942] jepsen worker nemesis - couchbase.util Rebalance check count: 38 INFO [2022-03-13 17:07:08,942] jepsen worker nemesis - couchbase.util Rebalance stuck count: 18 INFO [2022-03-13 17:07:09,038] jepsen worker 11 - jepsen.util 11 :info :write [3 16] :RequestTimeoutException INFO [2022-03-13 17:07:09,040] jepsen worker 11 - jepsen.util 101 :invoke :read [3 nil] INFO [2022-03-13 17:07:09,043] jepsen worker 11 - jepsen.util 101 :ok :read [3 :nil] INFO [2022-03-13 17:07:09,044] jepsen worker 11 - jepsen.util 101 :invoke :write [3 31] INFO [2022-03-13 17:07:09,047] jepsen worker 46 - jepsen.util 46 :info :write [15 5] :SyncWriteAmbiguous INFO [2022-03-13 17:07:09,050] jepsen worker 46 - jepsen.util 136 :invoke :write [15 5] INFO [2022-03-13 17:07:09,063] jepsen worker 18 - jepsen.util 18 :info :write [6 13] :RequestTimeoutException INFO [2022-03-13 17:07:09,064] jepsen worker 18 - jepsen.util 108 :invoke :read [6 nil] INFO [2022-03-13 17:07:09,078] jepsen worker 69 - jepsen.util 69 :info :write [23 17] :RequestTimeoutException INFO [2022-03-13 17:07:09,084] jepsen worker 69 - jepsen.util 159 :invoke :write [23 32] INFO [2022-03-13 17:07:09,127] jepsen worker 16 - jepsen.util 16 :info :write [5 10] :RequestTimeoutException INFO [2022-03-13 17:07:09,128] jepsen worker 16 - jepsen.util 106 :invoke :write [5 19] INFO [2022-03-13 17:07:09,140] jepsen worker 27 - jepsen.util 27 :info :write [9 36] :SyncWriteAmbiguous INFO [2022-03-13 17:07:09,141] jepsen worker 27 - jepsen.util 117 :invoke :write [9 1] INFO [2022-03-13 17:07:09,154] jepsen worker 42 - jepsen.util 42 :info :write [14 45] :SyncWriteAmbiguous INFO [2022-03-13 17:07:09,155] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:09,173] jepsen worker 64 - jepsen.util 64 :info :write [21 27] :SyncWriteAmbiguous INFO [2022-03-13 17:07:09,174] jepsen worker 64 - jepsen.util 154 :invoke :read [21 nil] INFO [2022-03-13 17:07:09,259] jepsen worker 22 - jepsen.util 22 :info :write [7 30] :SyncWriteAmbiguous INFO [2022-03-13 17:07:09,261] jepsen worker 22 - jepsen.util 112 :invoke :write [7 26] INFO [2022-03-13 17:07:09,275] jepsen worker 84 - jepsen.util 84 :info :write [28 20] :RequestTimeoutException INFO [2022-03-13 17:07:09,276] jepsen worker 84 - jepsen.util 174 :invoke :read [28 nil] INFO [2022-03-13 17:07:09,282] jepsen worker 84 - jepsen.util 174 :ok :read [28 :nil] INFO [2022-03-13 17:07:09,283] jepsen worker 84 - jepsen.util 174 :invoke :write [28 41] INFO [2022-03-13 17:07:09,311] jepsen worker 8 - jepsen.util 8 :info :write [2 26] :SyncWriteAmbiguous INFO [2022-03-13 17:07:09,312] jepsen worker 8 - jepsen.util 98 :invoke :write [2 28] INFO [2022-03-13 17:07:09,343] jepsen worker 21 - jepsen.util 21 :info :write [7 39] :RequestTimeoutException INFO [2022-03-13 17:07:09,344] jepsen worker 21 - jepsen.util 111 :invoke :write [7 28] INFO [2022-03-13 17:07:09,367] jepsen worker 71 - jepsen.util 71 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,370] jepsen worker 7 - jepsen.util 7 :info :write [2 21] :RequestTimeoutException INFO [2022-03-13 17:07:09,370] jepsen worker 71 - jepsen.util 71 :invoke :write [23 43] INFO [2022-03-13 17:07:09,372] jepsen worker 7 - jepsen.util 97 :invoke :write [2 17] INFO [2022-03-13 17:07:09,391] jepsen worker 74 - jepsen.util 74 :info :write [24 49] :RequestTimeoutException INFO [2022-03-13 17:07:09,391] jepsen worker 74 - jepsen.util 164 :invoke :write [24 28] INFO [2022-03-13 17:07:09,397] jepsen worker 44 - jepsen.util 44 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,398] jepsen worker 44 - jepsen.util 44 :invoke :read [14 nil] INFO [2022-03-13 17:07:09,419] jepsen worker 37 - jepsen.util 37 :info :write [12 44] :RequestTimeoutException INFO [2022-03-13 17:07:09,420] jepsen worker 37 - jepsen.util 127 :invoke :write [12 41] INFO [2022-03-13 17:07:09,470] jepsen worker 77 - jepsen.util 77 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,471] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:07:09,472] jepsen worker 79 - jepsen.util 169 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,473] jepsen worker 79 - jepsen.util 169 :invoke :read [26 nil] INFO [2022-03-13 17:07:09,498] jepsen worker 69 - jepsen.util 159 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,499] jepsen worker 69 - jepsen.util 159 :invoke :write [23 5] INFO [2022-03-13 17:07:09,568] jepsen worker 35 - jepsen.util 125 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,568] jepsen worker 35 - jepsen.util 125 :invoke :write [11 49] INFO [2022-03-13 17:07:09,579] jepsen worker 53 - jepsen.util 53 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,580] jepsen worker 53 - jepsen.util 53 :invoke :read [17 nil] INFO [2022-03-13 17:07:09,602] jepsen worker 14 - jepsen.util 14 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,603] jepsen worker 14 - jepsen.util 14 :invoke :write [4 48] INFO [2022-03-13 17:07:09,644] jepsen worker 8 - jepsen.util 98 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,644] jepsen worker 44 - jepsen.util 44 :ok :read [14 :nil] INFO [2022-03-13 17:07:09,645] jepsen worker 8 - jepsen.util 98 :invoke :write [2 25] INFO [2022-03-13 17:07:09,650] jepsen worker 44 - jepsen.util 44 :invoke :write [14 17] INFO [2022-03-13 17:07:09,666] jepsen worker 62 - jepsen.util 152 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,667] jepsen worker 62 - jepsen.util 152 :invoke :read [20 nil] INFO [2022-03-13 17:07:09,688] jepsen worker 83 - jepsen.util 83 :info :write [27 11] :RequestTimeoutException INFO [2022-03-13 17:07:09,689] jepsen worker 83 - jepsen.util 173 :invoke :read [27 nil] INFO [2022-03-13 17:07:09,691] jepsen worker 75 - jepsen.util 75 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,692] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:09,708] jepsen worker 79 - jepsen.util 169 :ok :read [26 36] INFO [2022-03-13 17:07:09,708] jepsen worker 79 - jepsen.util 169 :invoke :write [26 7] INFO [2022-03-13 17:07:09,728] jepsen worker 63 - jepsen.util 63 :info :write [21 36] :RequestTimeoutException INFO [2022-03-13 17:07:09,729] jepsen worker 63 - jepsen.util 153 :invoke :read [21 nil] INFO [2022-03-13 17:07:09,732] jepsen worker 63 - jepsen.util 153 :ok :read [21 :nil] INFO [2022-03-13 17:07:09,733] jepsen worker 63 - jepsen.util 153 :invoke :read [21 nil] INFO [2022-03-13 17:07:09,735] jepsen worker 63 - jepsen.util 153 :ok :read [21 :nil] INFO [2022-03-13 17:07:09,735] jepsen worker 63 - jepsen.util 153 :invoke :read [21 nil] INFO [2022-03-13 17:07:09,737] jepsen worker 63 - jepsen.util 153 :ok :read [21 :nil] INFO [2022-03-13 17:07:09,738] jepsen worker 63 - jepsen.util 153 :invoke :write [21 20] INFO [2022-03-13 17:07:09,747] jepsen worker 10 - jepsen.util 10 :info :write [3 49] :RequestTimeoutException INFO [2022-03-13 17:07:09,748] jepsen worker 10 - jepsen.util 100 :invoke :read [3 nil] INFO [2022-03-13 17:07:09,758] jepsen worker 35 - jepsen.util 125 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,759] jepsen worker 35 - jepsen.util 125 :invoke :read [11 nil] INFO [2022-03-13 17:07:09,762] jepsen worker 52 - jepsen.util 52 :info :write [17 43] :RequestTimeoutException INFO [2022-03-13 17:07:09,763] jepsen worker 52 - jepsen.util 142 :invoke :write [17 38] INFO [2022-03-13 17:07:09,793] jepsen worker 70 - jepsen.util 160 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,794] jepsen worker 15 - jepsen.util 15 :info :write [5 44] :RequestTimeoutException INFO [2022-03-13 17:07:09,796] jepsen worker 51 - jepsen.util 51 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,796] jepsen worker 70 - jepsen.util 160 :invoke :read [23 nil] INFO [2022-03-13 17:07:09,797] jepsen worker 15 - jepsen.util 105 :invoke :write [5 0] INFO [2022-03-13 17:07:09,803] jepsen worker 51 - jepsen.util 51 :invoke :write [17 33] INFO [2022-03-13 17:07:09,820] jepsen worker 8 - jepsen.util 98 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,821] jepsen worker 8 - jepsen.util 98 :invoke :read [2 nil] INFO [2022-03-13 17:07:09,842] jepsen worker 0 - jepsen.util 0 :info :write [0 41] :RequestTimeoutException INFO [2022-03-13 17:07:09,843] jepsen worker 0 - jepsen.util 90 :invoke :read [0 nil] INFO [2022-03-13 17:07:09,852] jepsen worker 43 - jepsen.util 43 :info :write [14 9] :RequestTimeoutException INFO [2022-03-13 17:07:09,853] jepsen worker 43 - jepsen.util 133 :invoke :write [14 44] INFO [2022-03-13 17:07:09,884] jepsen worker 44 - jepsen.util 44 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,885] jepsen worker 44 - jepsen.util 44 :invoke :read [14 nil] INFO [2022-03-13 17:07:09,896] jepsen worker 19 - jepsen.util 19 :info :write [6 5] :RequestTimeoutException INFO [2022-03-13 17:07:09,898] jepsen worker 19 - jepsen.util 109 :invoke :write [6 16] INFO [2022-03-13 17:07:09,912] jepsen worker 24 - jepsen.util 24 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,912] jepsen worker 12 - jepsen.util 12 :info :write [4 26] :RequestTimeoutException INFO [2022-03-13 17:07:09,913] jepsen worker 24 - jepsen.util 24 :invoke :write [8 42] INFO [2022-03-13 17:07:09,913] jepsen worker 12 - jepsen.util 102 :invoke :read [4 nil] INFO [2022-03-13 17:07:09,930] jepsen worker 13 - jepsen.util 103 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,931] jepsen worker 13 - jepsen.util 103 :invoke :write [4 11] INFO [2022-03-13 17:07:09,949] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:09,949] jepsen worker nemesis - couchbase.util Rebalance check count: 39 INFO [2022-03-13 17:07:09,949] jepsen worker nemesis - couchbase.util Rebalance stuck count: 19 INFO [2022-03-13 17:07:09,984] jepsen worker 35 - jepsen.util 125 :ok :read [11 :nil] INFO [2022-03-13 17:07:09,984] jepsen worker 79 - jepsen.util 169 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:09,986] jepsen worker 79 - jepsen.util 169 :invoke :write [26 46] INFO [2022-03-13 17:07:09,989] jepsen worker 35 - jepsen.util 125 :invoke :read [11 nil] INFO [2022-03-13 17:07:09,995] jepsen worker 35 - jepsen.util 125 :ok :read [11 :nil] INFO [2022-03-13 17:07:09,995] jepsen worker 44 - jepsen.util 44 :ok :read [14 :nil] INFO [2022-03-13 17:07:10,001] jepsen worker 69 - jepsen.util 159 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:07:10,001] jepsen worker 35 - jepsen.util 125 :invoke :read [11 nil] INFO [2022-03-13 17:07:10,001] jepsen worker 44 - jepsen.util 44 :invoke :read [14 nil] INFO [2022-03-13 17:07:10,002] jepsen worker 69 - jepsen.util 159 :invoke :write [23 46] INFO [2022-03-13 17:07:10,003] jepsen worker 35 - jepsen.util 125 :ok :read [11 :nil] INFO [2022-03-13 17:07:10,006] jepsen worker 25 - jepsen.util 115 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:07:10,006] jepsen worker 70 - jepsen.util 160 :ok :read [23 :nil] INFO [2022-03-13 17:07:10,007] jepsen worker 35 - jepsen.util 125 :invoke :write [11 12] INFO [2022-03-13 17:07:10,007] jepsen worker 40 - jepsen.util 40 :info :write [13 1] :RequestTimeoutException INFO [2022-03-13 17:07:10,008] jepsen worker 44 - jepsen.util 44 :ok :read [14 :nil] INFO [2022-03-13 17:07:10,009] jepsen worker 53 - jepsen.util 53 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:10,013] jepsen worker 25 - jepsen.util 115 :invoke :write [8 8] INFO [2022-03-13 17:07:10,014] jepsen worker 70 - jepsen.util 160 :invoke :write [23 25] INFO [2022-03-13 17:07:10,018] jepsen worker 8 - jepsen.util 98 :ok :read [2 12] INFO [2022-03-13 17:07:10,020] jepsen worker 44 - jepsen.util 44 :invoke :read [14 nil] INFO [2022-03-13 17:07:10,022] jepsen worker 40 - jepsen.util 130 :invoke :write [13 12] INFO [2022-03-13 17:07:10,023] jepsen worker 8 - jepsen.util 98 :invoke :write [2 30] INFO [2022-03-13 17:07:10,030] jepsen worker 53 - jepsen.util 53 :invoke :write [17 26] INFO [2022-03-13 17:07:10,033] jepsen worker 44 - jepsen.util 44 :ok :read [14 :nil] INFO [2022-03-13 17:07:10,034] jepsen worker 44 - jepsen.util 44 :invoke :read [14 nil] INFO [2022-03-13 17:07:10,052] jepsen worker 1 - jepsen.util 1 :info :write [0 46] :RequestTimeoutException INFO [2022-03-13 17:07:10,053] jepsen worker 1 - jepsen.util 91 :invoke :read [0 nil] INFO [2022-03-13 17:07:10,087] jepsen worker 76 - jepsen.util 166 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:07:10,088] jepsen worker 41 - jepsen.util 41 :info :write [13 22] :RequestTimeoutException INFO [2022-03-13 17:07:10,088] jepsen worker 76 - jepsen.util 166 :invoke :write [25 44] INFO [2022-03-13 17:07:10,089] jepsen worker 41 - jepsen.util 131 :invoke :write [13 38] INFO [2022-03-13 17:07:10,112] jepsen worker 21 - jepsen.util 111 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:07:10,113] jepsen worker 21 - jepsen.util 111 :invoke :write [7 37] INFO [2022-03-13 17:07:10,113] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:07:10,116] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:07:10,123] jepsen worker 79 - jepsen.util 169 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:10,125] jepsen worker 44 - jepsen.util 44 :ok :read [14 :nil] INFO [2022-03-13 17:07:10,125] jepsen worker 79 - jepsen.util 169 :invoke :read [26 nil] INFO [2022-03-13 17:07:10,127] jepsen worker 44 - jepsen.util 44 :invoke :read [14 nil] INFO [2022-03-13 17:07:10,139] jepsen worker 45 - jepsen.util 45 :info :write [15 48] :RequestTimeoutException INFO [2022-03-13 17:07:10,140] jepsen worker 45 - jepsen.util 135 :invoke :write [15 7] INFO [2022-03-13 17:07:10,141] jepsen worker 67 - jepsen.util 67 :info :write [22 36] :RequestTimeoutException INFO [2022-03-13 17:07:10,143] jepsen worker 67 - jepsen.util 157 :invoke :read [22 nil] INFO [2022-03-13 17:07:10,148] jepsen worker 39 - jepsen.util 39 :info :write [13 13] :RequestTimeoutException INFO [2022-03-13 17:07:10,149] jepsen worker 29 - jepsen.util 29 :info :write [9 31] :RequestTimeoutException INFO [2022-03-13 17:07:10,150] jepsen worker 39 - jepsen.util 129 :invoke :write [13 47] INFO [2022-03-13 17:07:10,151] jepsen worker 29 - jepsen.util 119 :invoke :read [9 nil] INFO [2022-03-13 17:07:10,167] jepsen worker 12 - jepsen.util 102 :ok :read [4 :nil] INFO [2022-03-13 17:07:10,168] jepsen worker 12 - jepsen.util 102 :invoke :read [4 nil] INFO [2022-03-13 17:07:10,171] jepsen worker 33 - jepsen.util 33 :info :write [11 6] :RequestTimeoutException INFO [2022-03-13 17:07:10,172] jepsen worker 33 - jepsen.util 123 :invoke :write [11 18] INFO [2022-03-13 17:07:10,178] jepsen worker 71 - jepsen.util 71 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:07:10,178] jepsen worker 75 - jepsen.util 75 :ok :read [25 :nil] INFO [2022-03-13 17:07:10,179] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:07:10,186] jepsen worker 75 - jepsen.util 75 :invoke :write [25 7] INFO [2022-03-13 17:07:10,187] jepsen worker 62 - jepsen.util 152 :ok :read [20 10] INFO [2022-03-13 17:07:10,188] jepsen worker 62 - jepsen.util 152 :invoke :write [20 12] INFO [2022-03-13 17:07:10,201] jepsen worker 57 - jepsen.util 57 :info :write [19 49] :RequestTimeoutException INFO [2022-03-13 17:07:10,201] jepsen worker 47 - jepsen.util 47 :info :write [15 46] :RequestTimeoutException INFO [2022-03-13 17:07:10,202] jepsen worker 47 - jepsen.util 137 :invoke :write [15 47] INFO [2022-03-13 17:07:10,203] jepsen worker 57 - jepsen.util 147 :invoke :write [19 25] INFO [2022-03-13 17:07:10,206] jepsen worker 29 - jepsen.util 119 :ok :read [9 :nil] INFO [2022-03-13 17:07:10,207] jepsen worker 29 - jepsen.util 119 :invoke :write [9 47] INFO [2022-03-13 17:07:10,227] jepsen worker 28 - jepsen.util 28 :info :write [9 14] :RequestTimeoutException INFO [2022-03-13 17:07:10,229] jepsen worker 28 - jepsen.util 118 :invoke :write [9 16] INFO [2022-03-13 17:07:10,244] jepsen worker 87 - jepsen.util 87 :info :write [29 49] :RequestTimeoutException INFO [2022-03-13 17:07:10,245] jepsen worker 87 - jepsen.util 177 :invoke :write [29 36] INFO [2022-03-13 17:07:10,301] jepsen worker 89 - jepsen.util 89 :info :write [29 35] :RequestTimeoutException INFO [2022-03-13 17:07:10,303] jepsen worker 89 - jepsen.util 179 :invoke :write [29 9] INFO [2022-03-13 17:07:10,305] jepsen worker 65 - jepsen.util 65 :info :write [21 49] :RequestTimeoutException INFO [2022-03-13 17:07:10,307] jepsen worker 65 - jepsen.util 155 :invoke :write [21 27] INFO [2022-03-13 17:07:10,317] jepsen worker 3 - jepsen.util 3 :info :write [1 29] :RequestTimeoutException INFO [2022-03-13 17:07:10,317] jepsen worker 3 - jepsen.util 93 :invoke :write [1 12] INFO [2022-03-13 17:07:10,325] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:07:10,326] jepsen worker 77 - jepsen.util 77 :invoke :write [25 6] INFO [2022-03-13 17:07:10,326] jepsen worker 34 - jepsen.util 34 :info :write [11 47] :RequestTimeoutException INFO [2022-03-13 17:07:10,328] jepsen worker 44 - jepsen.util 44 :ok :read [14 :nil] INFO [2022-03-13 17:07:10,328] jepsen worker 34 - jepsen.util 124 :invoke :read [11 nil] INFO [2022-03-13 17:07:10,330] jepsen worker 44 - jepsen.util 44 :invoke :write [14 43] INFO [2022-03-13 17:07:10,342] jepsen worker 79 - jepsen.util 169 :ok :read [26 36] INFO [2022-03-13 17:07:10,344] jepsen worker 79 - jepsen.util 169 :invoke :write [26 26] INFO [2022-03-13 17:07:10,347] jepsen worker 50 - jepsen.util 50 :info :write [16 49] :RequestTimeoutException INFO [2022-03-13 17:07:10,351] jepsen worker 50 - jepsen.util 140 :invoke :read [16 nil] INFO [2022-03-13 17:07:10,394] jepsen worker 66 - jepsen.util 66 :info :write [22 41] :RequestTimeoutException INFO [2022-03-13 17:07:10,395] jepsen worker 66 - jepsen.util 156 :invoke :read [22 nil] INFO [2022-03-13 17:07:10,398] jepsen worker 88 - jepsen.util 88 :info :write [29 8] :RequestTimeoutException INFO [2022-03-13 17:07:10,399] jepsen worker 88 - jepsen.util 178 :invoke :read [29 nil] INFO [2022-03-13 17:07:10,499] jepsen worker 12 - jepsen.util 102 :ok :read [4 :nil] INFO [2022-03-13 17:07:10,500] jepsen worker 12 - jepsen.util 102 :invoke :write [4 8] INFO [2022-03-13 17:07:10,708] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:07:10,709] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:07:10,818] jepsen worker 34 - jepsen.util 124 :ok :read [11 :nil] INFO [2022-03-13 17:07:10,819] jepsen worker 34 - jepsen.util 124 :invoke :write [11 42] INFO [2022-03-13 17:07:10,957] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:10,957] jepsen worker nemesis - couchbase.util Rebalance check count: 40 INFO [2022-03-13 17:07:10,957] jepsen worker nemesis - couchbase.util Rebalance stuck count: 20 INFO [2022-03-13 17:07:10,991] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:07:10,991] jepsen worker 71 - jepsen.util 71 :invoke :write [23 32] INFO [2022-03-13 17:07:11,486] jepsen worker 88 - jepsen.util 178 :ok :read [29 :nil] INFO [2022-03-13 17:07:11,486] jepsen worker 88 - jepsen.util 178 :invoke :write [29 27] INFO [2022-03-13 17:07:11,633] jepsen worker 58 - jepsen.util 148 :ok :read [19 1] INFO [2022-03-13 17:07:11,634] jepsen worker 58 - jepsen.util 148 :invoke :read [19 nil] INFO [2022-03-13 17:07:11,963] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:11,963] jepsen worker nemesis - couchbase.util Rebalance check count: 41 INFO [2022-03-13 17:07:11,964] jepsen worker nemesis - couchbase.util Rebalance stuck count: 21 INFO [2022-03-13 17:07:12,240] jepsen worker 60 - jepsen.util 150 :ok :read [20 10] INFO [2022-03-13 17:07:12,241] jepsen worker 60 - jepsen.util 150 :invoke :write [20 46] INFO [2022-03-13 17:07:12,348] jepsen worker 82 - jepsen.util 82 :info :write [27 40] :RequestTimeoutException INFO [2022-03-13 17:07:12,349] jepsen worker 82 - jepsen.util 172 :invoke :write [27 47] INFO [2022-03-13 17:07:12,546] jepsen worker 80 - jepsen.util 80 :info :write [26 35] :RequestTimeoutException INFO [2022-03-13 17:07:12,547] jepsen worker 80 - jepsen.util 170 :invoke :read [26 nil] INFO [2022-03-13 17:07:12,760] jepsen worker 58 - jepsen.util 148 :ok :read [19 1] INFO [2022-03-13 17:07:12,762] jepsen worker 58 - jepsen.util 148 :invoke :read [19 nil] INFO [2022-03-13 17:07:12,878] jepsen worker 80 - jepsen.util 170 :ok :read [26 36] INFO [2022-03-13 17:07:12,879] jepsen worker 80 - jepsen.util 170 :invoke :read [26 nil] INFO [2022-03-13 17:07:12,970] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:12,971] jepsen worker nemesis - couchbase.util Rebalance check count: 42 INFO [2022-03-13 17:07:12,971] jepsen worker nemesis - couchbase.util Rebalance stuck count: 22 INFO [2022-03-13 17:07:13,088] jepsen worker 80 - jepsen.util 170 :ok :read [26 36] INFO [2022-03-13 17:07:13,088] jepsen worker 80 - jepsen.util 170 :invoke :write [26 33] INFO [2022-03-13 17:07:13,978] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:13,979] jepsen worker nemesis - couchbase.util Rebalance check count: 43 INFO [2022-03-13 17:07:13,979] jepsen worker nemesis - couchbase.util Rebalance stuck count: 23 INFO [2022-03-13 17:07:14,388] jepsen worker 86 - jepsen.util 176 :ok :read [28 :nil] INFO [2022-03-13 17:07:14,388] jepsen worker 86 - jepsen.util 176 :invoke :read [28 nil] INFO [2022-03-13 17:07:14,560] jepsen worker 73 - jepsen.util 163 :ok :write [24 24] INFO [2022-03-13 17:07:14,560] jepsen worker 73 - jepsen.util 163 :invoke :read [24 nil] INFO [2022-03-13 17:07:14,733] jepsen worker 68 - jepsen.util 158 :info :write [22 48] :SyncWriteAmbiguous INFO [2022-03-13 17:07:14,733] jepsen worker 68 - jepsen.util 248 :invoke :read [22 nil] INFO [2022-03-13 17:07:14,863] jepsen worker 67 - jepsen.util 157 :ok :read [22 46] INFO [2022-03-13 17:07:14,863] jepsen worker 67 - jepsen.util 157 :invoke :read [22 nil] INFO [2022-03-13 17:07:14,904] jepsen worker 67 - jepsen.util 157 :ok :read [22 46] INFO [2022-03-13 17:07:14,905] jepsen worker 67 - jepsen.util 157 :invoke :write [22 10] INFO [2022-03-13 17:07:14,988] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:14,989] jepsen worker nemesis - couchbase.util Rebalance check count: 44 INFO [2022-03-13 17:07:14,989] jepsen worker nemesis - couchbase.util Rebalance stuck count: 24 INFO [2022-03-13 17:07:15,040] jepsen worker 50 - jepsen.util 140 :ok :read [16 :nil] INFO [2022-03-13 17:07:15,041] jepsen worker 50 - jepsen.util 140 :invoke :write [16 13] INFO [2022-03-13 17:07:15,065] jepsen worker 81 - jepsen.util 171 :ok :read [27 8] INFO [2022-03-13 17:07:15,066] jepsen worker 81 - jepsen.util 171 :invoke :read [27 nil] WARN [2022-03-13 17:07:15,117] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7862025,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":7911337,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":7869826,"encode_duration_us":120338,"operation_id":"0x724","timeout_ms":10000,"last_server_duration_us":7847106,"total_duration_us":10006334},{"operation_name":"upsert","last_dispatch_duration_us":9079760,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9079760,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9023573,"encode_duration_us":285520,"operation_id":"0x6ed","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10009691}],"total_count":2}} WARN [2022-03-13 17:07:15,118] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":48508,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8809094,"total_server_duration_us":647011,"encode_duration_us":84887,"operation_id":"0x727","timeout_ms":10000,"last_server_duration_us":28768,"total_duration_us":10010858},{"operation_name":"upsert","last_dispatch_duration_us":9079760,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9079760,"total_server_duration_us":9023573,"encode_duration_us":285520,"operation_id":"0x6ed","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10009691},{"operation_name":"upsert","last_dispatch_duration_us":64610,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7820888,"total_server_duration_us":313034,"encode_duration_us":84132,"operation_id":"0x6c5","timeout_ms":10000,"last_server_duration_us":43727,"total_duration_us":10009189},{"operation_name":"upsert","last_dispatch_duration_us":34289,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9469446,"total_server_duration_us":281483,"encode_duration_us":112736,"operation_id":"0x6f0","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10007876},{"operation_name":"upsert","last_dispatch_duration_us":7862025,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7911337,"total_server_duration_us":7869826,"encode_duration_us":120338,"operation_id":"0x724","timeout_ms":10000,"last_server_duration_us":7847106,"total_duration_us":10006334},{"operation_name":"upsert","last_dispatch_duration_us":8200374,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8602008,"total_server_duration_us":259877,"encode_duration_us":180911,"operation_id":"0x68e","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10004678},{"operation_name":"upsert","last_dispatch_duration_us":7685556,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7804430,"total_server_duration_us":88016,"encode_duration_us":310935,"operation_id":"0x720","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10001759},{"operation_name":"upsert","last_dispatch_duration_us":57198,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8564222,"total_server_duration_us":339147,"encode_duration_us":26600,"operation_id":"0x73a","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10001206},{"operation_name":"upsert","last_dispatch_duration_us":102413,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":164141,"total_server_duration_us":113238,"encode_duration_us":111670,"operation_id":"0x735","timeout_ms":10000,"last_server_duration_us":72862,"total_duration_us":10000878},{"operation_name":"upsert","last_dispatch_duration_us":9958729,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9958729,"total_server_duration_us":9026756,"encode_duration_us":109746,"operation_id":"0x6f4","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":9963313}],"total_count":19}} INFO [2022-03-13 17:07:15,152] jepsen worker 81 - jepsen.util 171 :ok :read [27 8] INFO [2022-03-13 17:07:15,152] jepsen worker 81 - jepsen.util 171 :invoke :read [27 nil] INFO [2022-03-13 17:07:15,222] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][1284us] Node disconnected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:07:15,280] jepsen worker 81 - jepsen.util 171 :ok :read [27 8] INFO [2022-03-13 17:07:15,281] jepsen worker 81 - jepsen.util 171 :invoke :read [27 nil] INFO [2022-03-13 17:07:15,338] jepsen worker 81 - jepsen.util 171 :ok :read [27 8] INFO [2022-03-13 17:07:15,338] jepsen worker 81 - jepsen.util 171 :invoke :write [27 13] INFO [2022-03-13 17:07:15,670] jepsen worker 9 - jepsen.util 99 :info :write [3 5] :SyncWriteAmbiguous INFO [2022-03-13 17:07:15,671] jepsen worker 9 - jepsen.util 189 :invoke :read [3 nil] INFO [2022-03-13 17:07:15,674] jepsen worker 9 - jepsen.util 189 :ok :read [3 :nil] INFO [2022-03-13 17:07:15,674] jepsen worker 9 - jepsen.util 189 :invoke :read [3 nil] INFO [2022-03-13 17:07:15,677] jepsen worker 9 - jepsen.util 189 :ok :read [3 :nil] INFO [2022-03-13 17:07:15,677] jepsen worker 9 - jepsen.util 189 :invoke :read [3 nil] INFO [2022-03-13 17:07:15,680] jepsen worker 9 - jepsen.util 189 :ok :read [3 :nil] INFO [2022-03-13 17:07:15,680] jepsen worker 9 - jepsen.util 189 :invoke :read [3 nil] INFO [2022-03-13 17:07:15,683] jepsen worker 9 - jepsen.util 189 :ok :read [3 :nil] INFO [2022-03-13 17:07:15,683] jepsen worker 9 - jepsen.util 189 :invoke :read [3 nil] INFO [2022-03-13 17:07:15,685] jepsen worker 9 - jepsen.util 189 :ok :read [3 :nil] INFO [2022-03-13 17:07:15,686] jepsen worker 9 - jepsen.util 189 :invoke :write [3 37] INFO [2022-03-13 17:07:15,719] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][198us] Node disconnected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:07:15,720] jepsen worker 86 - jepsen.util 176 :ok :read [28 :nil] INFO [2022-03-13 17:07:15,721] jepsen worker 86 - jepsen.util 176 :invoke :write [28 35] INFO [2022-03-13 17:07:15,728] jepsen worker 49 - jepsen.util 139 :info :write [16 25] :RequestTimeoutException INFO [2022-03-13 17:07:15,729] jepsen worker 49 - jepsen.util 229 :invoke :read [16 nil] INFO [2022-03-13 17:07:15,731] jepsen worker 6 - jepsen.util 96 :info :write [2 0] :RequestTimeoutException INFO [2022-03-13 17:07:15,732] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:15,735] jepsen worker 58 - jepsen.util 148 :ok :read [19 1] INFO [2022-03-13 17:07:15,736] jepsen worker 58 - jepsen.util 148 :invoke :read [19 nil] INFO [2022-03-13 17:07:15,790] jepsen worker 73 - jepsen.util 163 :ok :read [24 24] INFO [2022-03-13 17:07:15,792] jepsen worker 73 - jepsen.util 163 :invoke :write [24 13] INFO [2022-03-13 17:07:15,814] jepsen worker 58 - jepsen.util 148 :ok :read [19 1] INFO [2022-03-13 17:07:15,815] jepsen worker 58 - jepsen.util 148 :invoke :write [19 23] INFO [2022-03-13 17:07:15,817] jepsen worker 38 - jepsen.util 38 :info :write [12 40] :RequestTimeoutException INFO [2022-03-13 17:07:15,818] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][198us] Node disconnected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:07:15,819] jepsen worker 38 - jepsen.util 128 :invoke :write [12 14] INFO [2022-03-13 17:07:16,002] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:16,002] jepsen worker nemesis - couchbase.util Rebalance check count: 45 INFO [2022-03-13 17:07:16,002] jepsen worker nemesis - couchbase.util Rebalance stuck count: 25 INFO [2022-03-13 17:07:16,138] jepsen worker 22 - jepsen.util 112 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,138] jepsen worker 22 - jepsen.util 112 :invoke :read [7 nil] INFO [2022-03-13 17:07:16,143] jepsen worker 76 - jepsen.util 166 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,143] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:16,149] jepsen worker 0 - jepsen.util 90 :ok :read [0 41] INFO [2022-03-13 17:07:16,150] jepsen worker 0 - jepsen.util 90 :invoke :write [0 0] INFO [2022-03-13 17:07:16,164] jepsen worker 20 - jepsen.util 110 :info :write [6 24] :SyncWriteAmbiguous INFO [2022-03-13 17:07:16,164] jepsen worker 20 - jepsen.util 200 :invoke :read [6 nil] INFO [2022-03-13 17:07:16,175] jepsen worker 77 - jepsen.util 77 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,175] jepsen worker 25 - jepsen.util 115 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,176] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:07:16,176] jepsen worker 25 - jepsen.util 115 :invoke :write [8 19] INFO [2022-03-13 17:07:16,264] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:16,264] jepsen worker 42 - jepsen.util 132 :invoke :write [14 19] INFO [2022-03-13 17:07:16,280] jepsen worker 6 - jepsen.util 186 :ok :read [2 12] INFO [2022-03-13 17:07:16,281] jepsen worker 6 - jepsen.util 186 :invoke :write [2 35] INFO [2022-03-13 17:07:16,372] jepsen worker 22 - jepsen.util 112 :ok :read [7 :nil] INFO [2022-03-13 17:07:16,372] jepsen worker 22 - jepsen.util 112 :invoke :write [7 7] INFO [2022-03-13 17:07:16,451] jepsen worker 26 - jepsen.util 116 :ok :read [8 :nil] INFO [2022-03-13 17:07:16,452] jepsen worker 26 - jepsen.util 116 :invoke :write [8 25] WARN [2022-03-13 17:07:16,624] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16926967,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16926967,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9084117,"encode_duration_us":146807,"operation_id":"0x6be","timeout_ms":10000,"last_server_duration_us":9084117,"total_duration_us":10007809}],"total_count":1}} INFO [2022-03-13 17:07:16,645] jepsen worker 61 - jepsen.util 151 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,646] jepsen worker 61 - jepsen.util 151 :invoke :write [20 7] INFO [2022-03-13 17:07:16,672] jepsen worker 7 - jepsen.util 97 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,674] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:16,708] jepsen worker 62 - jepsen.util 152 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,709] jepsen worker 62 - jepsen.util 152 :invoke :read [20 nil] WARN [2022-03-13 17:07:16,721] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9063310,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9063310,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9047983,"encode_duration_us":84463,"operation_id":"0x72e","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10004565},{"operation_name":"upsert","last_dispatch_duration_us":9038939,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9038939,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9028877,"encode_duration_us":137607,"operation_id":"0x6f1","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10009567},{"operation_name":"upsert","last_dispatch_duration_us":13328055,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":13328055,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9049045,"encode_duration_us":153998,"operation_id":"0x6c1","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10009259}],"total_count":3}} WARN [2022-03-13 17:07:16,721] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9038939,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9038939,"total_server_duration_us":9028877,"encode_duration_us":137607,"operation_id":"0x6f1","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10009567},{"operation_name":"upsert","last_dispatch_duration_us":13328055,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":13328055,"total_server_duration_us":9049045,"encode_duration_us":153998,"operation_id":"0x6c1","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10009259},{"operation_name":"upsert","last_dispatch_duration_us":42801,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":735499,"total_server_duration_us":223959,"encode_duration_us":19271,"operation_id":"0x7e9","timeout_ms":10000,"last_server_duration_us":25275,"total_duration_us":10008517},{"operation_name":"upsert","last_dispatch_duration_us":47319,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":6771244,"total_server_duration_us":225261,"encode_duration_us":191277,"operation_id":"0x70d","timeout_ms":10000,"last_server_duration_us":26507,"total_duration_us":10006628},{"operation_name":"upsert","last_dispatch_duration_us":35155,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":559772,"total_server_duration_us":79087,"encode_duration_us":69448,"operation_id":"0x73c","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10006166},{"operation_name":"upsert","last_dispatch_duration_us":283633,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":8128802,"total_server_duration_us":387937,"encode_duration_us":19782,"operation_id":"0x789","timeout_ms":10000,"last_server_duration_us":60128,"total_duration_us":10005750},{"operation_name":"upsert","last_dispatch_duration_us":71722,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8217578,"total_server_duration_us":895991,"encode_duration_us":232337,"operation_id":"0x71a","timeout_ms":10000,"last_server_duration_us":48449,"total_duration_us":10005488},{"operation_name":"upsert","last_dispatch_duration_us":9063310,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9063310,"total_server_duration_us":9047983,"encode_duration_us":84463,"operation_id":"0x72e","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10004565},{"operation_name":"upsert","last_dispatch_duration_us":7294737,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":7429758,"total_server_duration_us":163669,"encode_duration_us":17945,"operation_id":"0x742","timeout_ms":10000,"last_server_duration_us":84574,"total_duration_us":10003439},{"operation_name":"upsert","last_dispatch_duration_us":32418,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":133285,"total_server_duration_us":81938,"encode_duration_us":120260,"operation_id":"0x6f8","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10003034}],"total_count":16}} WARN [2022-03-13 17:07:16,724] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":1550308,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11181125,"total_server_duration_us":45547,"encode_duration_us":46248,"operation_id":"0x73f","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10010254},{"operation_name":"upsert","last_dispatch_duration_us":159579,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":298497,"total_server_duration_us":45751,"encode_duration_us":121787,"operation_id":"0x6f7","timeout_ms":10000,"last_server_duration_us":24669,"total_duration_us":10009747},{"operation_name":"upsert","last_dispatch_duration_us":206604,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":291094,"total_server_duration_us":57305,"encode_duration_us":199131,"operation_id":"0x728","timeout_ms":10000,"last_server_duration_us":37653,"total_duration_us":10007716},{"operation_name":"upsert","last_dispatch_duration_us":9200459,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":17761394,"total_server_duration_us":244078,"encode_duration_us":220898,"operation_id":"0x6b2","timeout_ms":10000,"last_server_duration_us":70690,"total_duration_us":10006468},{"operation_name":"upsert","last_dispatch_duration_us":168991,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9226846,"total_server_duration_us":175080,"encode_duration_us":103941,"operation_id":"0x68d","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10006072},{"operation_name":"upsert","last_dispatch_duration_us":9156078,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9384714,"total_server_duration_us":99222,"encode_duration_us":23907,"operation_id":"0x725","timeout_ms":10000,"last_server_duration_us":56401,"total_duration_us":10005635},{"operation_name":"upsert","encode_duration_us":98151,"operation_id":"0x738","timeout_ms":10000,"total_duration_us":10005163},{"operation_name":"upsert","last_dispatch_duration_us":9187931,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9187931,"total_server_duration_us":9071357,"encode_duration_us":50359,"operation_id":"0x71c","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10004150},{"operation_name":"upsert","last_dispatch_duration_us":840997,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1571700,"total_server_duration_us":158758,"encode_duration_us":17957,"operation_id":"0x7e4","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10002757},{"operation_name":"upsert","last_dispatch_duration_us":215158,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":325176,"total_server_duration_us":42646,"encode_duration_us":10589,"operation_id":"0x6bd","timeout_ms":10000,"last_server_duration_us":23306,"total_duration_us":10002495}],"total_count":24}} WARN [2022-03-13 17:07:16,725] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","encode_duration_us":19586,"operation_id":"0x780","timeout_ms":10000,"total_duration_us":10010185},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":96770,"operation_id":"0x734","timeout_ms":10000,"total_duration_us":10009644},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":20430,"operation_id":"0x7e3","timeout_ms":10000,"total_duration_us":10007967},{"operation_name":"upsert","last_dispatch_duration_us":16926967,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":16926967,"total_server_duration_us":9084117,"encode_duration_us":146807,"operation_id":"0x6be","timeout_ms":10000,"last_server_duration_us":9084117,"total_duration_us":10007809},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":227075,"operation_id":"0x6df","timeout_ms":10000,"total_duration_us":10007484},{"operation_name":"upsert","encode_duration_us":19511,"operation_id":"0x747","timeout_ms":10000,"total_duration_us":10006817},{"operation_name":"upsert","last_dispatch_duration_us":8350306,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8996662,"total_server_duration_us":230454,"encode_duration_us":130220,"operation_id":"0x6f9","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10006141},{"operation_name":"upsert","last_dispatch_duration_us":62899,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":334383,"total_server_duration_us":129644,"encode_duration_us":73763,"operation_id":"0x733","timeout_ms":10000,"last_server_duration_us":36525,"total_duration_us":10004571},{"operation_name":"upsert","last_dispatch_duration_us":8568612,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8808482,"total_server_duration_us":94726,"encode_duration_us":39369,"operation_id":"0x736","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10002743},{"operation_name":"upsert","last_dispatch_duration_us":516014,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":8468161,"total_server_duration_us":646937,"encode_duration_us":145942,"operation_id":"0x6cc","timeout_ms":10000,"last_server_duration_us":388143,"total_duration_us":10001269}],"total_count":19}} WARN [2022-03-13 17:07:16,726] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9189700,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12642418,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":94468,"encode_duration_us":256290,"operation_id":"0x682","timeout_ms":10000,"last_server_duration_us":73547,"total_duration_us":10001461},{"operation_name":"upsert","last_dispatch_duration_us":16236841,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16236841,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9037366,"encode_duration_us":162987,"operation_id":"0x73d","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10007553},{"operation_name":"upsert","last_dispatch_duration_us":9064961,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9409907,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9248317,"encode_duration_us":396256,"operation_id":"0x689","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10005460},{"operation_name":"upsert","last_dispatch_duration_us":12038514,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12038514,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":9033121,"encode_duration_us":256718,"operation_id":"0x68f","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10009554}],"total_count":4}} INFO [2022-03-13 17:07:16,753] jepsen worker 70 - jepsen.util 160 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,754] jepsen worker 70 - jepsen.util 160 :invoke :write [23 48] INFO [2022-03-13 17:07:16,761] jepsen worker 22 - jepsen.util 112 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,762] jepsen worker 22 - jepsen.util 112 :invoke :write [7 36] INFO [2022-03-13 17:07:16,821] jepsen worker 69 - jepsen.util 159 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,821] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:16,822] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:16,823] jepsen worker 76 - jepsen.util 166 :invoke :write [25 14] WARN [2022-03-13 17:07:16,825] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":840997,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":1571700,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":158758,"encode_duration_us":17957,"operation_id":"0x7e4","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10002757},{"operation_name":"upsert","last_dispatch_duration_us":9187931,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9187931,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":9071357,"encode_duration_us":50359,"operation_id":"0x71c","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10004150}],"total_count":2}} WARN [2022-03-13 17:07:16,827] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9139867,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17637515,"total_server_duration_us":9128765,"encode_duration_us":112251,"operation_id":"0x6f2","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10017095},{"operation_name":"upsert","last_dispatch_duration_us":12038514,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":12038514,"total_server_duration_us":9033121,"encode_duration_us":256718,"operation_id":"0x68f","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10009554},{"operation_name":"upsert","last_dispatch_duration_us":33784,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":760518,"total_server_duration_us":290344,"encode_duration_us":58491,"operation_id":"0x740","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10008967},{"operation_name":"upsert","last_dispatch_duration_us":16236841,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16236841,"total_server_duration_us":9037366,"encode_duration_us":162987,"operation_id":"0x73d","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10007553},{"operation_name":"upsert","last_dispatch_duration_us":9064961,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9409907,"total_server_duration_us":9248317,"encode_duration_us":396256,"operation_id":"0x689","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10005460},{"operation_name":"upsert","last_dispatch_duration_us":35517,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":1962489,"total_server_duration_us":300818,"encode_duration_us":150983,"operation_id":"0x6ee","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10005379},{"operation_name":"upsert","last_dispatch_duration_us":68753,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9022894,"total_server_duration_us":274648,"encode_duration_us":93989,"operation_id":"0x6f5","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10004096},{"operation_name":"upsert","last_dispatch_duration_us":5407044,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8153466,"total_server_duration_us":307679,"encode_duration_us":187216,"operation_id":"0x6bf","timeout_ms":10000,"last_server_duration_us":44944,"total_duration_us":10001594},{"operation_name":"upsert","last_dispatch_duration_us":116642,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":204329,"total_server_duration_us":57734,"encode_duration_us":18516,"operation_id":"0x748","timeout_ms":10000,"last_server_duration_us":34512,"total_duration_us":10001556},{"operation_name":"upsert","last_dispatch_duration_us":9257906,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9257906,"total_server_duration_us":74648,"encode_duration_us":96405,"operation_id":"0x73b","timeout_ms":10000,"last_server_duration_us":74648,"total_duration_us":9371376}],"total_count":18}} INFO [2022-03-13 17:07:16,829] jepsen worker 59 - jepsen.util 149 :info :write [19 23] :RequestTimeoutException INFO [2022-03-13 17:07:16,830] jepsen worker 59 - jepsen.util 239 :invoke :write [19 7] INFO [2022-03-13 17:07:16,852] jepsen worker 61 - jepsen.util 151 :fail :write [20 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,853] jepsen worker 61 - jepsen.util 151 :invoke :write [20 23] INFO [2022-03-13 17:07:16,892] jepsen worker 42 - jepsen.util 132 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:16,893] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:16,910] jepsen worker 54 - jepsen.util 144 :info :write [18 28] :SyncWriteAmbiguous INFO [2022-03-13 17:07:16,911] jepsen worker 54 - jepsen.util 234 :invoke :write [18 3] WARN [2022-03-13 17:07:16,921] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17129741,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17659601,"total_server_duration_us":9298476,"encode_duration_us":159464,"operation_id":"0x6c4","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":10009400},{"operation_name":"upsert","last_dispatch_duration_us":63246,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":526260,"total_server_duration_us":418758,"encode_duration_us":266975,"operation_id":"0x71b","timeout_ms":10000,"last_server_duration_us":49835,"total_duration_us":10009122},{"operation_name":"upsert","last_dispatch_duration_us":67768,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":685472,"total_server_duration_us":133173,"encode_duration_us":126415,"operation_id":"0x6ef","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10007590},{"operation_name":"upsert","last_dispatch_duration_us":69928,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":201340,"total_server_duration_us":74259,"encode_duration_us":38559,"operation_id":"0x743","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10005958},{"operation_name":"upsert","last_dispatch_duration_us":73310,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9592134,"total_server_duration_us":351676,"encode_duration_us":185792,"operation_id":"0x6c3","timeout_ms":10000,"last_server_duration_us":60380,"total_duration_us":10004606},{"operation_name":"upsert","last_dispatch_duration_us":51653,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8192200,"total_server_duration_us":295527,"encode_duration_us":36763,"operation_id":"0x741","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10002405},{"operation_name":"upsert","last_dispatch_duration_us":34679,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8486771,"total_server_duration_us":270834,"encode_duration_us":78007,"operation_id":"0x72c","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10002239},{"operation_name":"upsert","last_dispatch_duration_us":17098997,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17686308,"total_server_duration_us":9292266,"encode_duration_us":82733,"operation_id":"0x6bc","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10001299},{"operation_name":"upsert","last_dispatch_duration_us":418592,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":9612079,"total_server_duration_us":281138,"encode_duration_us":63598,"operation_id":"0x726","timeout_ms":10000,"last_server_duration_us":49024,"total_duration_us":9837649},{"operation_name":"upsert","last_dispatch_duration_us":492982,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":9217287,"total_server_duration_us":516982,"encode_duration_us":24638,"operation_id":"0x744","timeout_ms":10000,"last_server_duration_us":73684,"total_duration_us":9449198}],"total_count":21}} WARN [2022-03-13 17:07:16,922] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9073615,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9711230,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9330317,"encode_duration_us":82213,"operation_id":"0x62d","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10017110}],"total_count":1}} INFO [2022-03-13 17:07:17,093] jepsen worker 6 - jepsen.util 186 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,098] jepsen worker 25 - jepsen.util 115 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,098] jepsen worker 80 - jepsen.util 170 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,099] jepsen worker 22 - jepsen.util 112 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,101] jepsen worker 18 - jepsen.util 108 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,101] jepsen worker 20 - jepsen.util 200 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,102] jepsen worker 24 - jepsen.util 24 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,102] jepsen worker 80 - jepsen.util 170 :invoke :write [26 12] INFO [2022-03-13 17:07:17,103] jepsen worker 25 - jepsen.util 115 :invoke :read [8 nil] INFO [2022-03-13 17:07:17,103] jepsen worker 18 - jepsen.util 108 :invoke :read [6 nil] INFO [2022-03-13 17:07:17,104] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:07:17,104] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:17,105] jepsen worker 20 - jepsen.util 200 :invoke :read [6 nil] INFO [2022-03-13 17:07:17,105] jepsen worker 22 - jepsen.util 112 :invoke :read [7 nil] INFO [2022-03-13 17:07:17,106] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:07:17,107] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:07:17,108] jepsen worker 51 - jepsen.util 51 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,110] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:17,123] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:17,123] jepsen worker nemesis - couchbase.util Rebalance check count: 46 INFO [2022-03-13 17:07:17,123] jepsen worker nemesis - couchbase.util Rebalance stuck count: 26 INFO [2022-03-13 17:07:17,124] jepsen worker 21 - jepsen.util 111 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,125] jepsen worker 21 - jepsen.util 111 :invoke :read [7 nil] INFO [2022-03-13 17:07:17,132] jepsen worker 60 - jepsen.util 150 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,134] jepsen worker 60 - jepsen.util 150 :invoke :write [20 43] INFO [2022-03-13 17:07:17,151] jepsen worker 18 - jepsen.util 108 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,151] jepsen worker 20 - jepsen.util 200 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,151] jepsen worker 18 - jepsen.util 108 :invoke :read [6 nil] INFO [2022-03-13 17:07:17,152] jepsen worker 20 - jepsen.util 200 :invoke :write [6 42] INFO [2022-03-13 17:07:17,154] jepsen worker 18 - jepsen.util 108 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,155] jepsen worker 18 - jepsen.util 108 :invoke :read [6 nil] INFO [2022-03-13 17:07:17,157] jepsen worker 18 - jepsen.util 108 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,157] jepsen worker 18 - jepsen.util 108 :invoke :read [6 nil] INFO [2022-03-13 17:07:17,182] jepsen worker 83 - jepsen.util 173 :ok :read [27 8] INFO [2022-03-13 17:07:17,184] jepsen worker 83 - jepsen.util 173 :invoke :write [27 47] INFO [2022-03-13 17:07:17,205] jepsen worker 53 - jepsen.util 53 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,205] jepsen worker 18 - jepsen.util 108 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,206] jepsen worker 53 - jepsen.util 53 :invoke :write [17 19] INFO [2022-03-13 17:07:17,206] jepsen worker 18 - jepsen.util 108 :invoke :read [6 nil] INFO [2022-03-13 17:07:17,217] jepsen worker 43 - jepsen.util 133 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,218] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:17,226] jepsen worker 61 - jepsen.util 151 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,226] jepsen worker 22 - jepsen.util 112 :ok :read [7 :nil] INFO [2022-03-13 17:07:17,227] jepsen worker 61 - jepsen.util 151 :invoke :read [20 nil] INFO [2022-03-13 17:07:17,227] jepsen worker 22 - jepsen.util 112 :invoke :write [7 47] INFO [2022-03-13 17:07:17,239] jepsen worker 18 - jepsen.util 108 :ok :read [6 :nil] INFO [2022-03-13 17:07:17,240] jepsen worker 18 - jepsen.util 108 :invoke :write [6 7] INFO [2022-03-13 17:07:17,279] jepsen worker 26 - jepsen.util 116 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,280] jepsen worker 26 - jepsen.util 116 :invoke :write [8 1] INFO [2022-03-13 17:07:17,310] jepsen worker 12 - jepsen.util 102 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,310] jepsen worker 80 - jepsen.util 170 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,310] jepsen worker 12 - jepsen.util 102 :invoke :read [4 nil] INFO [2022-03-13 17:07:17,311] jepsen worker 80 - jepsen.util 170 :invoke :read [26 nil] INFO [2022-03-13 17:07:17,351] jepsen worker 7 - jepsen.util 97 :ok :read [2 12] INFO [2022-03-13 17:07:17,352] jepsen worker 7 - jepsen.util 97 :invoke :write [2 26] INFO [2022-03-13 17:07:17,364] jepsen worker 33 - jepsen.util 123 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,366] jepsen worker 33 - jepsen.util 123 :invoke :write [11 23] INFO [2022-03-13 17:07:17,399] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:07:17,399] jepsen worker 12 - jepsen.util 102 :ok :read [4 :nil] INFO [2022-03-13 17:07:17,400] jepsen worker 21 - jepsen.util 111 :ok :read [7 :nil] INFO [2022-03-13 17:07:17,400] jepsen worker 24 - jepsen.util 24 :invoke :write [8 42] INFO [2022-03-13 17:07:17,401] jepsen worker 12 - jepsen.util 102 :invoke :read [4 nil] INFO [2022-03-13 17:07:17,402] jepsen worker 21 - jepsen.util 111 :invoke :write [7 8] INFO [2022-03-13 17:07:17,405] jepsen worker 12 - jepsen.util 102 :ok :read [4 :nil] INFO [2022-03-13 17:07:17,406] jepsen worker 12 - jepsen.util 102 :invoke :write [4 46] INFO [2022-03-13 17:07:17,407] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:07:17,419] jepsen worker 60 - jepsen.util 150 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,420] jepsen worker 60 - jepsen.util 150 :invoke :write [20 19] INFO [2022-03-13 17:07:17,420] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:07:17,448] jepsen worker 1 - jepsen.util 91 :ok :read [0 41] INFO [2022-03-13 17:07:17,453] jepsen worker 1 - jepsen.util 91 :invoke :write [0 30] INFO [2022-03-13 17:07:17,463] jepsen worker 62 - jepsen.util 152 :ok :read [20 10] INFO [2022-03-13 17:07:17,465] jepsen worker 62 - jepsen.util 152 :invoke :read [20 nil] INFO [2022-03-13 17:07:17,474] jepsen worker 71 - jepsen.util 71 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,474] jepsen worker 61 - jepsen.util 151 :ok :read [20 10] INFO [2022-03-13 17:07:17,475] jepsen worker 61 - jepsen.util 151 :invoke :read [20 nil] INFO [2022-03-13 17:07:17,476] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:07:17,490] jepsen worker 70 - jepsen.util 160 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,491] jepsen worker 25 - jepsen.util 115 :ok :read [8 :nil] INFO [2022-03-13 17:07:17,491] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:07:17,492] jepsen worker 70 - jepsen.util 160 :invoke :read [23 nil] INFO [2022-03-13 17:07:17,493] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:07:17,493] jepsen worker 25 - jepsen.util 115 :invoke :write [8 22] INFO [2022-03-13 17:07:17,523] jepsen worker 68 - jepsen.util 248 :ok :read [22 46] INFO [2022-03-13 17:07:17,525] jepsen worker 68 - jepsen.util 248 :invoke :write [22 14] INFO [2022-03-13 17:07:17,561] jepsen worker 22 - jepsen.util 112 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,563] jepsen worker 22 - jepsen.util 112 :invoke :read [7 nil] INFO [2022-03-13 17:07:17,600] jepsen worker 23 - jepsen.util 113 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,600] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:17,601] jepsen worker 42 - jepsen.util 132 :invoke :write [14 12] INFO [2022-03-13 17:07:17,601] jepsen worker 23 - jepsen.util 113 :invoke :write [7 32] INFO [2022-03-13 17:07:17,608] jepsen worker 80 - jepsen.util 170 :ok :read [26 36] INFO [2022-03-13 17:07:17,609] jepsen worker 80 - jepsen.util 170 :invoke :read [26 nil] INFO [2022-03-13 17:07:17,625] jepsen worker 53 - jepsen.util 53 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,626] jepsen worker 53 - jepsen.util 53 :invoke :read [17 nil] INFO [2022-03-13 17:07:17,676] jepsen worker 60 - jepsen.util 150 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,677] jepsen worker 60 - jepsen.util 150 :invoke :write [20 31] INFO [2022-03-13 17:07:17,685] jepsen worker 24 - jepsen.util 24 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,685] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:07:17,933] jepsen worker 21 - jepsen.util 111 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,936] jepsen worker 52 - jepsen.util 142 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,937] jepsen worker 70 - jepsen.util 160 :ok :read [23 :nil] INFO [2022-03-13 17:07:17,937] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:17,937] jepsen worker 32 - jepsen.util 122 :ok :read [10 :nil] INFO [2022-03-13 17:07:17,937] jepsen worker 33 - jepsen.util 123 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,938] jepsen worker 31 - jepsen.util 121 :ok :read [10 :nil] INFO [2022-03-13 17:07:17,938] jepsen worker 10 - jepsen.util 100 :ok :read [3 :nil] INFO [2022-03-13 17:07:17,939] jepsen worker 34 - jepsen.util 124 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,941] jepsen worker 17 - jepsen.util 107 :info :write [5 0] :RequestTimeoutException INFO [2022-03-13 17:07:17,942] jepsen worker 22 - jepsen.util 112 :ok :read [7 :nil] INFO [2022-03-13 17:07:17,943] jepsen worker 2 - jepsen.util 92 :info :write [0 1] :RequestTimeoutException INFO [2022-03-13 17:07:17,943] jepsen worker 4 - jepsen.util 94 :info :write [1 33] :RequestTimeoutException INFO [2022-03-13 17:07:17,944] jepsen worker 78 - jepsen.util 168 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,947] jepsen worker 5 - jepsen.util 5 :info :write [1 32] :RequestTimeoutException INFO [2022-03-13 17:07:17,947] jepsen worker 32 - jepsen.util 122 :invoke :write [10 49] INFO [2022-03-13 17:07:17,949] jepsen worker 17 - jepsen.util 197 :invoke :write [5 9] INFO [2022-03-13 17:07:17,949] jepsen worker 2 - jepsen.util 182 :invoke :write [0 29] INFO [2022-03-13 17:07:17,955] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,956] jepsen worker 69 - jepsen.util 159 :invoke :write [23 46] INFO [2022-03-13 17:07:17,958] jepsen worker 21 - jepsen.util 111 :invoke :read [7 nil] INFO [2022-03-13 17:07:17,955] jepsen worker 76 - jepsen.util 166 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,963] jepsen worker 70 - jepsen.util 160 :invoke :write [23 40] INFO [2022-03-13 17:07:17,966] jepsen worker 33 - jepsen.util 123 :invoke :read [11 nil] INFO [2022-03-13 17:07:17,968] jepsen worker 12 - jepsen.util 102 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,968] jepsen worker 53 - jepsen.util 53 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,968] jepsen worker 25 - jepsen.util 115 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,969] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:07:17,969] jepsen worker 62 - jepsen.util 152 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,970] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:07:17,971] jepsen worker 22 - jepsen.util 112 :invoke :write [7 22] INFO [2022-03-13 17:07:17,971] jepsen worker 21 - jepsen.util 111 :ok :read [7 :nil] INFO [2022-03-13 17:07:17,971] jepsen worker 33 - jepsen.util 123 :ok :read [11 :nil] INFO [2022-03-13 17:07:17,977] jepsen worker 77 - jepsen.util 77 :invoke :read [25 nil] INFO [2022-03-13 17:07:17,982] jepsen worker 6 - jepsen.util 186 :ok :read [2 12] INFO [2022-03-13 17:07:17,982] jepsen worker 62 - jepsen.util 152 :invoke :read [20 nil] INFO [2022-03-13 17:07:17,985] jepsen worker 77 - jepsen.util 77 :ok :read [25 :nil] INFO [2022-03-13 17:07:17,987] jepsen worker 62 - jepsen.util 152 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:17,992] jepsen worker 80 - jepsen.util 170 :ok :read [26 36] INFO [2022-03-13 17:07:17,992] jepsen worker 33 - jepsen.util 123 :invoke :write [11 1] INFO [2022-03-13 17:07:17,997] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:07:18,000] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:07:18,001] jepsen worker 10 - jepsen.util 100 :invoke :read [3 nil] INFO [2022-03-13 17:07:18,003] jepsen worker 56 - jepsen.util 56 :info :write [18 12] :RequestTimeoutException INFO [2022-03-13 17:07:18,004] jepsen worker 5 - jepsen.util 95 :invoke :read [1 nil] INFO [2022-03-13 17:07:18,006] jepsen worker 12 - jepsen.util 102 :invoke :write [4 3] INFO [2022-03-13 17:07:18,010] jepsen worker 61 - jepsen.util 151 :ok :read [20 10] INFO [2022-03-13 17:07:18,018] jepsen worker 31 - jepsen.util 121 :invoke :read [10 nil] INFO [2022-03-13 17:07:18,019] jepsen worker 4 - jepsen.util 184 :invoke :read [1 nil] INFO [2022-03-13 17:07:18,020] jepsen worker 6 - jepsen.util 186 :invoke :write [2 49] INFO [2022-03-13 17:07:18,021] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:18,022] jepsen worker 10 - jepsen.util 100 :ok :read [3 :nil] INFO [2022-03-13 17:07:18,023] jepsen worker 52 - jepsen.util 142 :invoke :write [17 31] INFO [2022-03-13 17:07:18,025] jepsen worker 56 - jepsen.util 146 :invoke :read [18 nil] INFO [2022-03-13 17:07:18,028] jepsen worker 77 - jepsen.util 77 :invoke :write [25 9] INFO [2022-03-13 17:07:18,030] jepsen worker 34 - jepsen.util 124 :invoke :write [11 44] INFO [2022-03-13 17:07:18,038] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][89us] Node disconnected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:07:18,039] jepsen worker 80 - jepsen.util 170 :invoke :write [26 10] INFO [2022-03-13 17:07:18,046] jepsen worker 53 - jepsen.util 53 :invoke :write [17 31] INFO [2022-03-13 17:07:18,048] jepsen worker 25 - jepsen.util 115 :invoke :write [8 19] INFO [2022-03-13 17:07:18,048] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:18,049] jepsen worker 21 - jepsen.util 111 :invoke :write [7 25] INFO [2022-03-13 17:07:18,059] jepsen worker 26 - jepsen.util 116 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,064] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:07:18,064] jepsen worker 26 - jepsen.util 116 :invoke :write [8 40] INFO [2022-03-13 17:07:18,065] jepsen worker 62 - jepsen.util 152 :invoke :write [20 10] INFO [2022-03-13 17:07:18,065] jepsen worker 61 - jepsen.util 151 :invoke :read [20 nil] INFO [2022-03-13 17:07:18,065] jepsen worker 10 - jepsen.util 100 :invoke :write [3 1] INFO [2022-03-13 17:07:18,066] jepsen worker 31 - jepsen.util 121 :ok :read [10 :nil] INFO [2022-03-13 17:07:18,070] jepsen worker 78 - jepsen.util 168 :invoke :read [26 nil] INFO [2022-03-13 17:07:18,074] jepsen worker 31 - jepsen.util 121 :invoke :read [10 nil] INFO [2022-03-13 17:07:18,092] jepsen worker 56 - jepsen.util 146 :ok :read [18 33] INFO [2022-03-13 17:07:18,093] jepsen worker 56 - jepsen.util 146 :invoke :write [18 12] INFO [2022-03-13 17:07:18,097] jepsen worker 60 - jepsen.util 150 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,098] jepsen worker 60 - jepsen.util 150 :invoke :write [20 6] INFO [2022-03-13 17:07:18,128] jepsen worker 70 - jepsen.util 160 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,130] jepsen worker 70 - jepsen.util 160 :invoke :read [23 nil] INFO [2022-03-13 17:07:18,141] jepsen worker 31 - jepsen.util 121 :ok :read [10 :nil] INFO [2022-03-13 17:07:18,141] jepsen worker 31 - jepsen.util 121 :invoke :read [10 nil] INFO [2022-03-13 17:07:18,146] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:18,147] jepsen worker nemesis - couchbase.util Rebalance check count: 47 INFO [2022-03-13 17:07:18,147] jepsen worker nemesis - couchbase.util Rebalance stuck count: 27 INFO [2022-03-13 17:07:18,186] jepsen worker 55 - jepsen.util 145 :info :write [18 2] :RequestTimeoutException INFO [2022-03-13 17:07:18,187] jepsen worker 55 - jepsen.util 235 :invoke :write [18 2] INFO [2022-03-13 17:07:18,210] jepsen worker 4 - jepsen.util 184 :ok :read [1 28] INFO [2022-03-13 17:07:18,211] jepsen worker 4 - jepsen.util 184 :invoke :read [1 nil] INFO [2022-03-13 17:07:18,217] jepsen worker 75 - jepsen.util 75 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,218] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:18,219] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:18,219] jepsen worker 5 - jepsen.util 95 :ok :read [1 28] INFO [2022-03-13 17:07:18,220] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:18,221] jepsen worker 5 - jepsen.util 95 :invoke :write [1 18] INFO [2022-03-13 17:07:18,233] jepsen worker 12 - jepsen.util 102 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,233] jepsen worker 22 - jepsen.util 112 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,234] jepsen worker 12 - jepsen.util 102 :invoke :write [4 6] INFO [2022-03-13 17:07:18,235] jepsen worker 22 - jepsen.util 112 :invoke :read [7 nil] INFO [2022-03-13 17:07:18,245] jepsen worker 30 - jepsen.util 120 :ok :read [10 :nil] INFO [2022-03-13 17:07:18,246] jepsen worker 30 - jepsen.util 120 :invoke :read [10 nil] INFO [2022-03-13 17:07:18,247] jepsen worker 48 - jepsen.util 138 :info :write [16 27] :RequestTimeoutException INFO [2022-03-13 17:07:18,248] jepsen worker 48 - jepsen.util 228 :invoke :read [16 nil] INFO [2022-03-13 17:07:18,248] jepsen worker 36 - jepsen.util 126 :info :write [12 37] :RequestTimeoutException INFO [2022-03-13 17:07:18,249] jepsen worker 36 - jepsen.util 216 :invoke :read [12 nil] INFO [2022-03-13 17:07:18,268] jepsen worker 7 - jepsen.util 97 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,272] jepsen worker 4 - jepsen.util 184 :ok :read [1 28] INFO [2022-03-13 17:07:18,275] jepsen worker 4 - jepsen.util 184 :invoke :write [1 49] INFO [2022-03-13 17:07:18,275] jepsen worker 7 - jepsen.util 97 :invoke :write [2 19] INFO [2022-03-13 17:07:18,295] jepsen worker 33 - jepsen.util 123 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,295] jepsen worker 33 - jepsen.util 123 :invoke :read [11 nil] INFO [2022-03-13 17:07:18,325] jepsen worker 31 - jepsen.util 121 :ok :read [10 :nil] INFO [2022-03-13 17:07:18,326] jepsen worker 31 - jepsen.util 121 :invoke :write [10 45] INFO [2022-03-13 17:07:18,331] jepsen worker 80 - jepsen.util 170 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,331] jepsen worker 61 - jepsen.util 151 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,332] jepsen worker 61 - jepsen.util 151 :invoke :read [20 nil] INFO [2022-03-13 17:07:18,333] jepsen worker 80 - jepsen.util 170 :invoke :write [26 27] INFO [2022-03-13 17:07:18,339] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][331us] Node disconnected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:07:18,382] jepsen worker 78 - jepsen.util 168 :ok :read [26 36] INFO [2022-03-13 17:07:18,384] jepsen worker 78 - jepsen.util 168 :invoke :write [26 15] INFO [2022-03-13 17:07:18,395] jepsen worker 14 - jepsen.util 14 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,396] jepsen worker 14 - jepsen.util 14 :invoke :write [4 0] INFO [2022-03-13 17:07:18,405] jepsen worker 72 - jepsen.util 162 :info :write [24 34] :RequestTimeoutException INFO [2022-03-13 17:07:18,405] jepsen worker 64 - jepsen.util 154 :ok :read [21 :nil] INFO [2022-03-13 17:07:18,406] jepsen worker 72 - jepsen.util 252 :invoke :write [24 36] INFO [2022-03-13 17:07:18,418] jepsen worker 64 - jepsen.util 154 :invoke :read [21 nil] INFO [2022-03-13 17:07:18,445] jepsen worker 23 - jepsen.util 113 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,446] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:18,462] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:07:18,462] jepsen worker 77 - jepsen.util 77 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,463] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:07:18,464] jepsen worker 77 - jepsen.util 77 :invoke :write [25 6] INFO [2022-03-13 17:07:18,493] jepsen worker 22 - jepsen.util 112 :ok :read [7 :nil] INFO [2022-03-13 17:07:18,493] jepsen worker 60 - jepsen.util 150 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,494] jepsen worker 61 - jepsen.util 151 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,494] jepsen worker 22 - jepsen.util 112 :invoke :write [7 35] INFO [2022-03-13 17:07:18,494] jepsen worker 61 - jepsen.util 151 :invoke :write [20 41] INFO [2022-03-13 17:07:18,495] jepsen worker 60 - jepsen.util 150 :invoke :write [20 36] INFO [2022-03-13 17:07:18,521] jepsen worker 13 - jepsen.util 103 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,522] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:07:18,524] jepsen worker 13 - jepsen.util 103 :invoke :write [4 16] INFO [2022-03-13 17:07:18,525] jepsen worker 71 - jepsen.util 71 :invoke :write [23 42] INFO [2022-03-13 17:07:18,560] jepsen worker 53 - jepsen.util 53 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,561] jepsen worker 53 - jepsen.util 53 :invoke :read [17 nil] INFO [2022-03-13 17:07:18,561] jepsen worker 80 - jepsen.util 170 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,562] jepsen worker 80 - jepsen.util 170 :invoke :read [26 nil] INFO [2022-03-13 17:07:18,606] jepsen worker 78 - jepsen.util 168 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,607] jepsen worker 78 - jepsen.util 168 :invoke :write [26 47] INFO [2022-03-13 17:07:18,613] jepsen worker 42 - jepsen.util 132 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,614] jepsen worker 42 - jepsen.util 132 :invoke :write [14 11] INFO [2022-03-13 17:07:18,649] jepsen worker 21 - jepsen.util 111 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,650] jepsen worker 21 - jepsen.util 111 :invoke :read [7 nil] INFO [2022-03-13 17:07:18,693] jepsen worker 25 - jepsen.util 115 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,693] jepsen worker 33 - jepsen.util 123 :ok :read [11 :nil] INFO [2022-03-13 17:07:18,694] jepsen worker 70 - jepsen.util 160 :ok :read [23 :nil] INFO [2022-03-13 17:07:18,695] jepsen worker 25 - jepsen.util 115 :invoke :read [8 nil] INFO [2022-03-13 17:07:18,695] jepsen worker 33 - jepsen.util 123 :invoke :read [11 nil] INFO [2022-03-13 17:07:18,696] jepsen worker 53 - jepsen.util 53 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,696] jepsen worker 85 - jepsen.util 175 :info :write [28 29] :RequestTimeoutException INFO [2022-03-13 17:07:18,697] jepsen worker 53 - jepsen.util 53 :invoke :read [17 nil] INFO [2022-03-13 17:07:18,697] jepsen worker 70 - jepsen.util 160 :invoke :write [23 15] INFO [2022-03-13 17:07:18,698] jepsen worker 85 - jepsen.util 265 :invoke :write [28 25] INFO [2022-03-13 17:07:18,706] jepsen worker 53 - jepsen.util 53 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,706] jepsen worker 33 - jepsen.util 123 :ok :read [11 :nil] INFO [2022-03-13 17:07:18,706] jepsen worker 25 - jepsen.util 115 :ok :read [8 :nil] INFO [2022-03-13 17:07:18,707] jepsen worker 33 - jepsen.util 123 :invoke :write [11 38] INFO [2022-03-13 17:07:18,708] jepsen worker 53 - jepsen.util 53 :invoke :write [17 39] INFO [2022-03-13 17:07:18,708] jepsen worker 25 - jepsen.util 115 :invoke :write [8 44] INFO [2022-03-13 17:07:18,721] jepsen worker 60 - jepsen.util 150 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,721] jepsen worker 30 - jepsen.util 120 :ok :read [10 :nil] INFO [2022-03-13 17:07:18,721] jepsen worker 64 - jepsen.util 154 :ok :read [21 :nil] INFO [2022-03-13 17:07:18,723] jepsen worker 30 - jepsen.util 120 :invoke :write [10 47] INFO [2022-03-13 17:07:18,729] jepsen worker 64 - jepsen.util 154 :invoke :read [21 nil] INFO [2022-03-13 17:07:18,730] jepsen worker 60 - jepsen.util 150 :invoke :read [20 nil] INFO [2022-03-13 17:07:18,772] jepsen worker 62 - jepsen.util 152 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,773] jepsen worker 62 - jepsen.util 152 :invoke :write [20 35] INFO [2022-03-13 17:07:18,804] jepsen worker 69 - jepsen.util 159 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,805] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:18,805] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:18,805] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,805] jepsen worker 76 - jepsen.util 166 :invoke :write [25 33] INFO [2022-03-13 17:07:18,806] jepsen worker 51 - jepsen.util 51 :invoke :write [17 41] INFO [2022-03-13 17:07:18,846] jepsen worker 52 - jepsen.util 142 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,847] jepsen worker 52 - jepsen.util 142 :invoke :write [17 27] INFO [2022-03-13 17:07:18,885] jepsen worker 22 - jepsen.util 112 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,885] jepsen worker 22 - jepsen.util 112 :invoke :write [7 31] INFO [2022-03-13 17:07:18,927] jepsen worker 12 - jepsen.util 102 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,927] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:07:18,927] jepsen worker 70 - jepsen.util 160 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:07:18,928] jepsen worker 12 - jepsen.util 102 :invoke :write [4 16] INFO [2022-03-13 17:07:18,929] jepsen worker 70 - jepsen.util 160 :invoke :read [23 nil] INFO [2022-03-13 17:07:18,930] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:07:19,000] jepsen worker 75 - jepsen.util 75 :ok :read [25 :nil] INFO [2022-03-13 17:07:19,000] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:19,001] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:19,001] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:19,059] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:19,061] jepsen worker 23 - jepsen.util 113 :invoke :write [7 28] INFO [2022-03-13 17:07:19,074] jepsen worker 46 - jepsen.util 136 :info :write [15 5] :RequestTimeoutException INFO [2022-03-13 17:07:19,075] jepsen worker 46 - jepsen.util 226 :invoke :write [15 41] INFO [2022-03-13 17:07:19,081] jepsen worker 11 - jepsen.util 101 :info :write [3 31] :RequestTimeoutException INFO [2022-03-13 17:07:19,082] jepsen worker 11 - jepsen.util 191 :invoke :read [3 nil] INFO [2022-03-13 17:07:19,091] jepsen worker 80 - jepsen.util 170 :ok :read [26 36] INFO [2022-03-13 17:07:19,092] jepsen worker 80 - jepsen.util 170 :invoke :write [26 9] INFO [2022-03-13 17:07:19,102] jepsen worker 21 - jepsen.util 111 :ok :read [7 :nil] INFO [2022-03-13 17:07:19,103] jepsen worker 21 - jepsen.util 111 :invoke :write [7 31] INFO [2022-03-13 17:07:19,151] jepsen worker 16 - jepsen.util 106 :info :write [5 19] :RequestTimeoutException INFO [2022-03-13 17:07:19,152] jepsen worker 16 - jepsen.util 196 :invoke :read [5 nil] INFO [2022-03-13 17:07:19,154] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:19,154] jepsen worker nemesis - couchbase.util Rebalance check count: 48 INFO [2022-03-13 17:07:19,154] jepsen worker nemesis - couchbase.util Rebalance stuck count: 28 INFO [2022-03-13 17:07:19,168] jepsen worker 27 - jepsen.util 117 :info :write [9 1] :RequestTimeoutException INFO [2022-03-13 17:07:19,169] jepsen worker 27 - jepsen.util 207 :invoke :read [9 nil] INFO [2022-03-13 17:07:19,205] jepsen worker 60 - jepsen.util 150 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:19,206] jepsen worker 60 - jepsen.util 150 :invoke :write [20 11] INFO [2022-03-13 17:07:19,302] jepsen worker 29 - jepsen.util 119 :info :write [9 47] :SyncWriteAmbiguous INFO [2022-03-13 17:07:19,303] jepsen worker 29 - jepsen.util 209 :invoke :read [9 nil] INFO [2022-03-13 17:07:19,308] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:07:19,308] jepsen worker 24 - jepsen.util 24 :invoke :read [8 nil] INFO [2022-03-13 17:07:19,319] jepsen worker 84 - jepsen.util 174 :info :write [28 41] :RequestTimeoutException INFO [2022-03-13 17:07:19,319] jepsen worker 84 - jepsen.util 264 :invoke :write [28 42] INFO [2022-03-13 17:07:19,332] jepsen worker 27 - jepsen.util 207 :ok :read [9 :nil] INFO [2022-03-13 17:07:19,333] jepsen worker 27 - jepsen.util 207 :invoke :read [9 nil] INFO [2022-03-13 17:07:19,335] jepsen worker 27 - jepsen.util 207 :ok :read [9 :nil] INFO [2022-03-13 17:07:19,336] jepsen worker 27 - jepsen.util 207 :invoke :read [9 nil] INFO [2022-03-13 17:07:19,338] jepsen worker 27 - jepsen.util 207 :ok :read [9 :nil] INFO [2022-03-13 17:07:19,338] jepsen worker 27 - jepsen.util 207 :invoke :write [9 14] INFO [2022-03-13 17:07:19,415] jepsen worker 74 - jepsen.util 164 :info :write [24 28] :RequestTimeoutException INFO [2022-03-13 17:07:19,416] jepsen worker 74 - jepsen.util 254 :invoke :read [24 nil] INFO [2022-03-13 17:07:19,446] jepsen worker 37 - jepsen.util 127 :info :write [12 41] :RequestTimeoutException INFO [2022-03-13 17:07:19,447] jepsen worker 37 - jepsen.util 217 :invoke :write [12 11] INFO [2022-03-13 17:07:19,507] jepsen worker 70 - jepsen.util 160 :ok :read [23 :nil] INFO [2022-03-13 17:07:19,508] jepsen worker 70 - jepsen.util 160 :invoke :read [23 nil] INFO [2022-03-13 17:07:19,761] jepsen worker 63 - jepsen.util 153 :info :write [21 20] :RequestTimeoutException INFO [2022-03-13 17:07:19,761] jepsen worker 63 - jepsen.util 243 :invoke :write [21 32] INFO [2022-03-13 17:07:19,824] jepsen worker 15 - jepsen.util 105 :info :write [5 0] :RequestTimeoutException INFO [2022-03-13 17:07:19,825] jepsen worker 15 - jepsen.util 195 :invoke :write [5 7] INFO [2022-03-13 17:07:19,931] jepsen worker 19 - jepsen.util 109 :info :write [6 16] :RequestTimeoutException INFO [2022-03-13 17:07:19,931] jepsen worker 70 - jepsen.util 160 :ok :read [23 :nil] INFO [2022-03-13 17:07:19,932] jepsen worker 19 - jepsen.util 199 :invoke :read [6 nil] INFO [2022-03-13 17:07:19,932] jepsen worker 24 - jepsen.util 24 :ok :read [8 :nil] INFO [2022-03-13 17:07:19,932] jepsen worker 70 - jepsen.util 160 :invoke :read [23 nil] INFO [2022-03-13 17:07:19,933] jepsen worker 24 - jepsen.util 24 :invoke :write [8 23] INFO [2022-03-13 17:07:20,041] jepsen worker 35 - jepsen.util 125 :info :write [11 12] :RequestTimeoutException INFO [2022-03-13 17:07:20,041] jepsen worker 35 - jepsen.util 215 :invoke :read [11 nil] INFO [2022-03-13 17:07:20,050] jepsen worker 40 - jepsen.util 130 :info :write [13 12] :RequestTimeoutException INFO [2022-03-13 17:07:20,051] jepsen worker 8 - jepsen.util 98 :info :write [2 30] :RequestTimeoutException INFO [2022-03-13 17:07:20,051] jepsen worker 40 - jepsen.util 220 :invoke :write [13 10] INFO [2022-03-13 17:07:20,052] jepsen worker 8 - jepsen.util 188 :invoke :write [2 5] INFO [2022-03-13 17:07:20,117] jepsen worker 41 - jepsen.util 131 :info :write [13 38] :RequestTimeoutException INFO [2022-03-13 17:07:20,118] jepsen worker 41 - jepsen.util 221 :invoke :write [13 45] INFO [2022-03-13 17:07:20,160] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:20,161] jepsen worker nemesis - couchbase.util Rebalance check count: 49 INFO [2022-03-13 17:07:20,161] jepsen worker nemesis - couchbase.util Rebalance stuck count: 29 INFO [2022-03-13 17:07:20,176] jepsen worker 45 - jepsen.util 135 :info :write [15 7] :RequestTimeoutException INFO [2022-03-13 17:07:20,178] jepsen worker 39 - jepsen.util 129 :info :write [13 47] :RequestTimeoutException INFO [2022-03-13 17:07:20,178] jepsen worker 45 - jepsen.util 225 :invoke :write [15 45] INFO [2022-03-13 17:07:20,179] jepsen worker 39 - jepsen.util 219 :invoke :write [13 32] INFO [2022-03-13 17:07:20,236] jepsen worker 47 - jepsen.util 137 :info :write [15 47] :RequestTimeoutException INFO [2022-03-13 17:07:20,237] jepsen worker 47 - jepsen.util 227 :invoke :write [15 48] INFO [2022-03-13 17:07:20,246] jepsen worker 57 - jepsen.util 147 :info :write [19 25] :RequestTimeoutException INFO [2022-03-13 17:07:20,247] jepsen worker 57 - jepsen.util 237 :invoke :read [19 nil] INFO [2022-03-13 17:07:20,261] jepsen worker 28 - jepsen.util 118 :info :write [9 16] :RequestTimeoutException INFO [2022-03-13 17:07:20,262] jepsen worker 28 - jepsen.util 208 :invoke :write [9 20] INFO [2022-03-13 17:07:20,268] jepsen worker 87 - jepsen.util 177 :info :write [29 36] :RequestTimeoutException INFO [2022-03-13 17:07:20,270] jepsen worker 87 - jepsen.util 267 :invoke :write [29 44] INFO [2022-03-13 17:07:20,297] jepsen worker 70 - jepsen.util 160 :ok :read [23 :nil] INFO [2022-03-13 17:07:20,297] jepsen worker 70 - jepsen.util 160 :invoke :write [23 17] INFO [2022-03-13 17:07:20,334] jepsen worker 65 - jepsen.util 155 :info :write [21 27] :RequestTimeoutException INFO [2022-03-13 17:07:20,335] jepsen worker 65 - jepsen.util 245 :invoke :write [21 33] INFO [2022-03-13 17:07:20,341] jepsen worker 89 - jepsen.util 179 :info :write [29 9] :RequestTimeoutException INFO [2022-03-13 17:07:20,341] jepsen worker 89 - jepsen.util 269 :invoke :read [29 nil] INFO [2022-03-13 17:07:20,351] jepsen worker 44 - jepsen.util 44 :info :write [14 43] :RequestTimeoutException INFO [2022-03-13 17:07:20,352] jepsen worker 44 - jepsen.util 134 :invoke :write [14 38] INFO [2022-03-13 17:07:20,357] jepsen worker 3 - jepsen.util 93 :info :write [1 12] :RequestTimeoutException INFO [2022-03-13 17:07:20,358] jepsen worker 3 - jepsen.util 183 :invoke :read [1 nil] INFO [2022-03-13 17:07:20,371] jepsen worker 79 - jepsen.util 169 :info :write [26 26] :RequestTimeoutException INFO [2022-03-13 17:07:20,372] jepsen worker 79 - jepsen.util 259 :invoke :read [26 nil] INFO [2022-03-13 17:07:20,569] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:20,569] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:20,571] jepsen worker 35 - jepsen.util 215 :ok :read [11 :nil] INFO [2022-03-13 17:07:20,572] jepsen worker 35 - jepsen.util 215 :invoke :read [11 nil] INFO [2022-03-13 17:07:21,129] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:21,130] jepsen worker 69 - jepsen.util 159 :invoke :write [23 7] INFO [2022-03-13 17:07:21,257] jepsen worker 75 - jepsen.util 75 :ok :read [25 :nil] INFO [2022-03-13 17:07:21,258] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:21,259] jepsen worker 43 - jepsen.util 133 :invoke :write [14 40] INFO [2022-03-13 17:07:21,259] jepsen worker 75 - jepsen.util 75 :invoke :write [25 22] INFO [2022-03-13 17:07:21,511] jepsen worker 88 - jepsen.util 178 :info :write [29 27] :RequestTimeoutException INFO [2022-03-13 17:07:21,512] jepsen worker 88 - jepsen.util 268 :invoke :write [29 32] INFO [2022-03-13 17:07:21,688] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:21,688] jepsen worker nemesis - couchbase.util Rebalance check count: 50 INFO [2022-03-13 17:07:21,688] jepsen worker nemesis - couchbase.util Rebalance stuck count: 30 INFO [2022-03-13 17:07:22,034] jepsen worker 35 - jepsen.util 215 :ok :read [11 :nil] INFO [2022-03-13 17:07:22,035] jepsen worker 35 - jepsen.util 215 :invoke :read [11 nil] INFO [2022-03-13 17:07:22,141] jepsen worker 35 - jepsen.util 215 :ok :read [11 :nil] INFO [2022-03-13 17:07:22,141] jepsen worker 35 - jepsen.util 215 :invoke :read [11 nil] INFO [2022-03-13 17:07:22,143] jepsen worker 35 - jepsen.util 215 :ok :read [11 :nil] INFO [2022-03-13 17:07:22,144] jepsen worker 35 - jepsen.util 215 :invoke :write [11 15] INFO [2022-03-13 17:07:22,374] jepsen worker 82 - jepsen.util 172 :info :write [27 47] :RequestTimeoutException INFO [2022-03-13 17:07:22,375] jepsen worker 82 - jepsen.util 262 :invoke :write [27 0] INFO [2022-03-13 17:07:22,694] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:22,694] jepsen worker nemesis - couchbase.util Rebalance check count: 51 INFO [2022-03-13 17:07:22,694] jepsen worker nemesis - couchbase.util Rebalance stuck count: 31 INFO [2022-03-13 17:07:23,120] jepsen worker 79 - jepsen.util 259 :ok :read [26 36] INFO [2022-03-13 17:07:23,120] jepsen worker 79 - jepsen.util 259 :invoke :write [26 20] INFO [2022-03-13 17:07:23,591] jepsen worker 27 - jepsen.util 207 :ok :write [9 14] INFO [2022-03-13 17:07:23,591] jepsen worker 27 - jepsen.util 207 :invoke :write [9 30] INFO [2022-03-13 17:07:23,702] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 1}, :ns_1@172.23.107.227 {:progress 1}, :ns_1@172.23.105.187 {:progress 1}, :ns_1@172.23.107.228 {:progress 1}} INFO [2022-03-13 17:07:23,702] jepsen worker nemesis - couchbase.util Rebalance check count: 52 INFO [2022-03-13 17:07:23,702] jepsen worker nemesis - couchbase.util Rebalance stuck count: 32 INFO [2022-03-13 17:07:24,094] jepsen worker 49 - jepsen.util 229 :ok :read [16 :nil] INFO [2022-03-13 17:07:24,094] jepsen worker 49 - jepsen.util 229 :invoke :read [16 nil] INFO [2022-03-13 17:07:24,095] jepsen worker 49 - jepsen.util 229 :ok :read [16 :nil] INFO [2022-03-13 17:07:24,097] jepsen worker 49 - jepsen.util 229 :invoke :write [16 48] INFO [2022-03-13 17:07:24,324] jepsen worker 66 - jepsen.util 156 :ok :read [22 46] INFO [2022-03-13 17:07:24,324] jepsen worker 66 - jepsen.util 156 :invoke :read [22 nil] INFO [2022-03-13 17:07:24,365] jepsen worker 16 - jepsen.util 196 :ok :read [5 :nil] INFO [2022-03-13 17:07:24,366] jepsen worker 16 - jepsen.util 196 :invoke :write [5 37] INFO [2022-03-13 17:07:24,389] jepsen worker 66 - jepsen.util 156 :ok :read [22 46] INFO [2022-03-13 17:07:24,389] jepsen worker 66 - jepsen.util 156 :invoke :write [22 3] INFO [2022-03-13 17:07:24,465] jepsen worker 81 - jepsen.util 171 :info :write [27 13] :SyncWriteAmbiguous INFO [2022-03-13 17:07:24,467] jepsen worker 81 - jepsen.util 261 :invoke :write [27 38] INFO [2022-03-13 17:07:24,572] jepsen worker 48 - jepsen.util 228 :ok :read [16 :nil] INFO [2022-03-13 17:07:24,572] jepsen worker 48 - jepsen.util 228 :invoke :read [16 nil] INFO [2022-03-13 17:07:24,708] jepsen worker nemesis - couchbase.util Rebalance status: {:status none} INFO [2022-03-13 17:07:24,708] jepsen worker nemesis - couchbase.util Rebalance check count: 53 INFO [2022-03-13 17:07:24,708] jepsen worker nemesis - couchbase.util Rebalance stuck count: 32 INFO [2022-03-13 17:07:24,709] jepsen worker nemesis - couchbase.util Rebalance complete INFO [2022-03-13 17:07:24,719] jepsen worker nemesis - jepsen.util :nemesis :info :rebalance-out ("172.23.107.227") INFO [2022-03-13 17:07:24,862] jepsen worker 9 - jepsen.util 189 :info :write [3 37] :SyncWriteAmbiguous INFO [2022-03-13 17:07:24,863] jepsen worker 9 - jepsen.util 279 :invoke :write [3 14] INFO [2022-03-13 17:07:24,870] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][100us] Node disconnected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:07:24,889] jepsen worker 73 - jepsen.util 163 :info :write [24 13] :SyncWriteAmbiguous INFO [2022-03-13 17:07:24,891] jepsen worker 73 - jepsen.util 253 :invoke :write [24 4] INFO [2022-03-13 17:07:24,931] jepsen worker 67 - jepsen.util 157 :info :write [22 10] :RequestTimeoutException INFO [2022-03-13 17:07:24,932] jepsen worker 67 - jepsen.util 247 :invoke :write [22 12] INFO [2022-03-13 17:07:25,071] jepsen worker 50 - jepsen.util 140 :info :write [16 13] :RequestTimeoutException INFO [2022-03-13 17:07:25,072] jepsen worker 50 - jepsen.util 230 :invoke :write [16 6] WARN [2022-03-13 17:07:25,171] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7716404,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7887231,"total_server_duration_us":133264,"encode_duration_us":19511,"operation_id":"0x914","timeout_ms":10000,"last_server_duration_us":43289,"total_duration_us":10053417},{"operation_name":"upsert","last_dispatch_duration_us":7441802,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7710458,"total_server_duration_us":161635,"encode_duration_us":30337,"operation_id":"0x920","timeout_ms":10000,"last_server_duration_us":29230,"total_duration_us":10009685},{"operation_name":"upsert","last_dispatch_duration_us":41637,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1025024,"total_server_duration_us":421987,"encode_duration_us":22246,"operation_id":"0x9af","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10009106},{"operation_name":"upsert","last_dispatch_duration_us":35758,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":906004,"total_server_duration_us":433530,"encode_duration_us":23901,"operation_id":"0x9d2","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10006467},{"operation_name":"upsert","last_dispatch_duration_us":152539,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":863398,"total_server_duration_us":358541,"encode_duration_us":20007,"operation_id":"0xa8a","timeout_ms":10000,"last_server_duration_us":26507,"total_duration_us":10004470},{"operation_name":"upsert","last_dispatch_duration_us":36980,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1043234,"total_server_duration_us":504599,"encode_duration_us":18047,"operation_id":"0x9a1","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10003832},{"operation_name":"upsert","last_dispatch_duration_us":8758591,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":12022386,"total_server_duration_us":805438,"encode_duration_us":17755,"operation_id":"0x9d8","timeout_ms":10000,"last_server_duration_us":58004,"total_duration_us":10002471},{"operation_name":"upsert","last_dispatch_duration_us":8759516,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":12093034,"total_server_duration_us":601802,"encode_duration_us":23502,"operation_id":"0x9db","timeout_ms":10000,"last_server_duration_us":59751,"total_duration_us":10002350},{"operation_name":"upsert","last_dispatch_duration_us":7525793,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7774702,"total_server_duration_us":134090,"encode_duration_us":19618,"operation_id":"0x96b","timeout_ms":10000,"last_server_duration_us":34215,"total_duration_us":10002173},{"operation_name":"upsert","last_dispatch_duration_us":8680592,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":11740228,"total_server_duration_us":705423,"encode_duration_us":24480,"operation_id":"0x9bb","timeout_ms":10000,"last_server_duration_us":85303,"total_duration_us":10002153}],"total_count":16}} WARN [2022-03-13 17:07:25,172] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16783012,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16947154,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9149543,"encode_duration_us":111670,"operation_id":"0x735","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10000878},{"operation_name":"upsert","last_dispatch_duration_us":9540772,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12210005,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":9697793,"encode_duration_us":30764,"operation_id":"0x9ba","timeout_ms":10000,"last_server_duration_us":9101143,"total_duration_us":10001987},{"operation_name":"upsert","last_dispatch_duration_us":16598368,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24419257,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9337668,"encode_duration_us":84132,"operation_id":"0x6c5","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009189}],"total_count":3}} INFO [2022-03-13 17:07:25,320] jepsen worker 0 - jepsen.util 90 :info :write [0 0] :SyncWriteAmbiguous INFO [2022-03-13 17:07:25,322] jepsen worker 0 - jepsen.util 180 :invoke :write [0 38] INFO [2022-03-13 17:07:25,750] jepsen worker 86 - jepsen.util 176 :info :write [28 35] :RequestTimeoutException INFO [2022-03-13 17:07:25,751] jepsen worker 86 - jepsen.util 266 :invoke :write [28 27] INFO [2022-03-13 17:07:25,844] jepsen worker 38 - jepsen.util 128 :info :write [12 14] :RequestTimeoutException INFO [2022-03-13 17:07:25,845] jepsen worker 38 - jepsen.util 218 :invoke :write [12 44] INFO [2022-03-13 17:07:25,850] jepsen worker 58 - jepsen.util 148 :info :write [19 23] :RequestTimeoutException INFO [2022-03-13 17:07:25,852] jepsen worker 58 - jepsen.util 238 :invoke :write [19 15] INFO [2022-03-13 17:07:26,110] jepsen worker 89 - jepsen.util 269 :ok :read [29 :nil] INFO [2022-03-13 17:07:26,111] jepsen worker 89 - jepsen.util 269 :invoke :write [29 17] INFO [2022-03-13 17:07:26,262] jepsen worker 44 - jepsen.util 134 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,263] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:26,264] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:26,265] jepsen worker 44 - jepsen.util 134 :invoke :write [14 36] INFO [2022-03-13 17:07:26,481] jepsen worker 57 - jepsen.util 237 :ok :read [19 1] INFO [2022-03-13 17:07:26,481] jepsen worker 57 - jepsen.util 237 :invoke :read [19 nil] INFO [2022-03-13 17:07:26,507] jepsen worker 57 - jepsen.util 237 :ok :read [19 1] INFO [2022-03-13 17:07:26,508] jepsen worker 57 - jepsen.util 237 :invoke :write [19 26] INFO [2022-03-13 17:07:26,603] jepsen worker 3 - jepsen.util 183 :ok :read [1 28] INFO [2022-03-13 17:07:26,603] jepsen worker 3 - jepsen.util 183 :invoke :read [1 nil] INFO [2022-03-13 17:07:26,614] jepsen worker 44 - jepsen.util 134 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,614] jepsen worker 44 - jepsen.util 134 :invoke :write [14 47] INFO [2022-03-13 17:07:26,674] jepsen worker 69 - jepsen.util 159 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,674] jepsen worker 7 - jepsen.util 97 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,676] jepsen worker 7 - jepsen.util 97 :invoke :write [2 8] INFO [2022-03-13 17:07:26,691] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:26,701] jepsen worker 3 - jepsen.util 183 :ok :read [1 28] INFO [2022-03-13 17:07:26,701] jepsen worker 3 - jepsen.util 183 :invoke :write [1 6] INFO [2022-03-13 17:07:26,741] jepsen worker 34 - jepsen.util 124 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,741] jepsen worker 34 - jepsen.util 124 :invoke :write [11 49] WARN [2022-03-13 17:07:26,768] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17404566,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17738949,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":8761286,"encode_duration_us":73763,"operation_id":"0x733","timeout_ms":10000,"last_server_duration_us":8631642,"total_duration_us":10004571},{"operation_name":"upsert","last_dispatch_duration_us":25327674,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25327674,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9083054,"encode_duration_us":227075,"operation_id":"0x6df","timeout_ms":10000,"last_server_duration_us":9083054,"total_duration_us":10007484}],"total_count":2}} WARN [2022-03-13 17:07:26,768] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":69122,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":69122,"total_server_duration_us":49,"operation_id":"0x9dc","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":13911921},{"operation_name":"get","last_dispatch_duration_us":10732684,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10732684,"total_server_duration_us":38,"operation_id":"0x8c8","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":10733102},{"operation_name":"upsert","encode_duration_us":18409,"operation_id":"0x9ca","timeout_ms":10000,"total_duration_us":10007930},{"operation_name":"upsert","encode_duration_us":22185,"operation_id":"0x8c3","timeout_ms":10000,"total_duration_us":10007210},{"operation_name":"upsert","last_dispatch_duration_us":2472493,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11951839,"total_server_duration_us":268720,"encode_duration_us":22221,"operation_id":"0x96d","timeout_ms":10000,"last_server_duration_us":36017,"total_duration_us":10003378},{"operation_name":"upsert","last_dispatch_duration_us":9170501,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9170501,"total_server_duration_us":19108,"encode_duration_us":18860,"operation_id":"0x9d3","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10003227},{"operation_name":"upsert","encode_duration_us":20788,"operation_id":"0x921","timeout_ms":10000,"total_duration_us":10001735},{"operation_name":"get","last_dispatch_duration_us":9008836,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9008836,"total_server_duration_us":15,"operation_id":"0x965","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":9009230},{"operation_name":"get","last_dispatch_duration_us":35844,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":35844,"total_server_duration_us":27,"operation_id":"0xaab","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":8363115},{"operation_name":"get","last_dispatch_duration_us":8187791,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8187791,"total_server_duration_us":56,"operation_id":"0x9a0","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8188199}],"total_count":24}} WARN [2022-03-13 17:07:26,772] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9118443,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17863254,"total_server_duration_us":9275160,"encode_duration_us":23431,"operation_id":"0x90d","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10175100},{"operation_name":"upsert","last_dispatch_duration_us":33967,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7832295,"total_server_duration_us":315495,"encode_duration_us":28066,"operation_id":"0x9d0","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10004408},{"operation_name":"upsert","last_dispatch_duration_us":102322,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":634447,"total_server_duration_us":183015,"encode_duration_us":21231,"operation_id":"0x978","timeout_ms":10000,"last_server_duration_us":34116,"total_duration_us":10003983},{"operation_name":"upsert","last_dispatch_duration_us":59320,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":580569,"total_server_duration_us":194625,"encode_duration_us":34201,"operation_id":"0x9da","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10003539},{"operation_name":"upsert","last_dispatch_duration_us":9082536,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9082536,"total_server_duration_us":9040551,"encode_duration_us":22291,"operation_id":"0x9c1","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10003431},{"operation_name":"upsert","last_dispatch_duration_us":9396353,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9544890,"total_server_duration_us":9061734,"encode_duration_us":19626,"operation_id":"0x8c5","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":9864654},{"operation_name":"upsert","last_dispatch_duration_us":339414,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2009553,"total_server_duration_us":505198,"encode_duration_us":29804,"operation_id":"0x91c","timeout_ms":10000,"last_server_duration_us":92892,"total_duration_us":9842283},{"operation_name":"upsert","last_dispatch_duration_us":9133454,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9133454,"total_server_duration_us":9050106,"encode_duration_us":28052,"operation_id":"0xab9","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9139859},{"operation_name":"upsert","last_dispatch_duration_us":194324,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1394244,"total_server_duration_us":549549,"encode_duration_us":24682,"operation_id":"0x955","timeout_ms":10000,"last_server_duration_us":44611,"total_duration_us":8102261},{"operation_name":"get","last_dispatch_duration_us":8029557,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8029557,"total_server_duration_us":43,"operation_id":"0x967","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8030054}],"total_count":17}} WARN [2022-03-13 17:07:26,773] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":257627,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":3149870,"total_server_duration_us":174287,"encode_duration_us":25789,"operation_id":"0x9d6","timeout_ms":10000,"last_server_duration_us":56524,"total_duration_us":10010710},{"operation_name":"upsert","last_dispatch_duration_us":169984,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":715656,"total_server_duration_us":220405,"encode_duration_us":25734,"operation_id":"0xab4","timeout_ms":10000,"last_server_duration_us":50301,"total_duration_us":10009008},{"operation_name":"upsert","last_dispatch_duration_us":9086259,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13824912,"total_server_duration_us":9293719,"encode_duration_us":17735,"operation_id":"0xa15","timeout_ms":10000,"last_server_duration_us":9069231,"total_duration_us":10005902},{"operation_name":"upsert","last_dispatch_duration_us":64523,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1126298,"total_server_duration_us":329608,"encode_duration_us":25251,"operation_id":"0xaae","timeout_ms":10000,"last_server_duration_us":26685,"total_duration_us":10002656},{"operation_name":"upsert","last_dispatch_duration_us":7528239,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8108573,"total_server_duration_us":382970,"encode_duration_us":32789,"operation_id":"0x969","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10002056},{"operation_name":"upsert","last_dispatch_duration_us":229810,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":2067186,"total_server_duration_us":92665,"encode_duration_us":25263,"operation_id":"0x9bc","timeout_ms":10000,"last_server_duration_us":49487,"total_duration_us":10001410},{"operation_name":"upsert","last_dispatch_duration_us":885136,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1683636,"total_server_duration_us":176101,"encode_duration_us":20590,"operation_id":"0x909","timeout_ms":10000,"last_server_duration_us":114454,"total_duration_us":9914196},{"operation_name":"upsert","last_dispatch_duration_us":9066794,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9066794,"total_server_duration_us":9046921,"encode_duration_us":19813,"operation_id":"0x9cf","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9073072},{"operation_name":"upsert","last_dispatch_duration_us":826966,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1303304,"total_server_duration_us":105636,"encode_duration_us":20078,"operation_id":"0x9a3","timeout_ms":10000,"last_server_duration_us":41552,"total_duration_us":8149003},{"operation_name":"upsert","last_dispatch_duration_us":575688,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":575688,"total_server_duration_us":45278,"encode_duration_us":19210,"operation_id":"0x975","timeout_ms":10000,"last_server_duration_us":45278,"total_duration_us":7277739}],"total_count":23}} INFO [2022-03-13 17:07:26,781] jepsen worker 6 - jepsen.util 186 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,781] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:26,829] jepsen worker 71 - jepsen.util 71 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,830] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:07:26,857] jepsen worker 59 - jepsen.util 239 :info :write [19 7] :RequestTimeoutException INFO [2022-03-13 17:07:26,857] jepsen worker 59 - jepsen.util 329 :invoke :read [19 nil] INFO [2022-03-13 17:07:26,858] jepsen worker 42 - jepsen.util 132 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,863] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] WARN [2022-03-13 17:07:26,866] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":14569005,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16531495,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9343492,"encode_duration_us":150983,"operation_id":"0x6ee","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10005379},{"operation_name":"upsert","last_dispatch_duration_us":9139867,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17637515,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9128765,"encode_duration_us":112251,"operation_id":"0x6f2","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10017095},{"operation_name":"upsert","last_dispatch_duration_us":14506311,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15266829,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9318160,"encode_duration_us":58491,"operation_id":"0x740","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10008967},{"operation_name":"upsert","last_dispatch_duration_us":17310982,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17883118,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9205802,"encode_duration_us":22682,"operation_id":"0x916","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10045079}],"total_count":4}} WARN [2022-03-13 17:07:26,866] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10254324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10254324,"total_server_duration_us":56,"operation_id":"0x919","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":10255077},{"operation_name":"upsert","last_dispatch_duration_us":35514,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9467281,"total_server_duration_us":262659,"encode_duration_us":26362,"operation_id":"0x913","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10064238},{"operation_name":"upsert","last_dispatch_duration_us":17310982,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17883118,"total_server_duration_us":9205802,"encode_duration_us":22682,"operation_id":"0x916","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10045079},{"operation_name":"upsert","last_dispatch_duration_us":131325,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":347901,"total_server_duration_us":60425,"encode_duration_us":17409,"operation_id":"0x9a8","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10008700},{"operation_name":"upsert","last_dispatch_duration_us":109127,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":277190,"total_server_duration_us":56192,"encode_duration_us":17197,"operation_id":"0xa2a","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10006743},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":19518,"operation_id":"0x9d4","timeout_ms":10000,"total_duration_us":10006087},{"operation_name":"upsert","last_dispatch_duration_us":6924482,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7190219,"total_server_duration_us":141703,"encode_duration_us":20098,"operation_id":"0xab3","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10003795},{"operation_name":"upsert","last_dispatch_duration_us":9233154,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9233154,"total_server_duration_us":55182,"encode_duration_us":19124,"operation_id":"0x968","timeout_ms":10000,"last_server_duration_us":55182,"total_duration_us":10003585},{"operation_name":"upsert","last_dispatch_duration_us":9994092,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9994092,"total_server_duration_us":24841,"encode_duration_us":19824,"operation_id":"0x948","timeout_ms":10000,"last_server_duration_us":24841,"total_duration_us":10002427},{"operation_name":"upsert","last_dispatch_duration_us":10304459,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10304459,"total_server_duration_us":30445,"encode_duration_us":21853,"operation_id":"0x91a","timeout_ms":10000,"last_server_duration_us":30445,"total_duration_us":10001871}],"total_count":23}} WARN [2022-03-13 17:07:26,873] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9082536,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9082536,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9040551,"encode_duration_us":22291,"operation_id":"0x9c1","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10003431},{"operation_name":"upsert","last_dispatch_duration_us":17193246,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17928745,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9261325,"encode_duration_us":19271,"operation_id":"0x7e9","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10008517},{"operation_name":"upsert","last_dispatch_duration_us":9118443,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17863254,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9275160,"encode_duration_us":23431,"operation_id":"0x90d","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10175100}],"total_count":3}} WARN [2022-03-13 17:07:26,873] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":106558,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8517093,"total_server_duration_us":293086,"encode_duration_us":22694,"operation_id":"0x971","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10009522},{"operation_name":"upsert","last_dispatch_duration_us":7238244,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7453725,"total_server_duration_us":67235,"encode_duration_us":32517,"operation_id":"0x9d1","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10008968},{"operation_name":"upsert","last_dispatch_duration_us":72139,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":6113818,"total_server_duration_us":313035,"encode_duration_us":23861,"operation_id":"0x9c8","timeout_ms":10000,"last_server_duration_us":41123,"total_duration_us":10007805},{"operation_name":"upsert","last_dispatch_duration_us":3495500,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9966995,"total_server_duration_us":3825674,"encode_duration_us":18637,"operation_id":"0x91f","timeout_ms":10000,"last_server_duration_us":3469265,"total_duration_us":10005148},{"operation_name":"upsert","last_dispatch_duration_us":263914,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1080300,"total_server_duration_us":350917,"encode_duration_us":19603,"operation_id":"0x960","timeout_ms":10000,"last_server_duration_us":31585,"total_duration_us":10004131},{"operation_name":"upsert","last_dispatch_duration_us":9063782,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9063782,"total_server_duration_us":9043735,"encode_duration_us":20707,"operation_id":"0xab2","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":9073498},{"operation_name":"upsert","last_dispatch_duration_us":991123,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4114350,"total_server_duration_us":476712,"encode_duration_us":19255,"operation_id":"0x996","timeout_ms":10000,"last_server_duration_us":72181,"total_duration_us":8772225},{"operation_name":"upsert","last_dispatch_duration_us":440414,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3422140,"total_server_duration_us":231776,"encode_duration_us":102267,"operation_id":"0xb4e","timeout_ms":10000,"last_server_duration_us":35412,"total_duration_us":8607230},{"operation_name":"upsert","last_dispatch_duration_us":1111376,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4393533,"total_server_duration_us":420350,"encode_duration_us":23850,"operation_id":"0x9b1","timeout_ms":10000,"last_server_duration_us":39214,"total_duration_us":8565492},{"operation_name":"upsert","last_dispatch_duration_us":1095075,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3862588,"total_server_duration_us":558307,"encode_duration_us":18120,"operation_id":"0x9cd","timeout_ms":10000,"last_server_duration_us":76730,"total_duration_us":8012063}],"total_count":35}} WARN [2022-03-13 17:07:26,874] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9086259,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13824912,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9293719,"encode_duration_us":17735,"operation_id":"0xa15","timeout_ms":10000,"last_server_duration_us":9069231,"total_duration_us":10005902}],"total_count":1}} INFO [2022-03-13 17:07:26,884] jepsen worker 59 - jepsen.util 329 :ok :read [19 1] INFO [2022-03-13 17:07:26,885] jepsen worker 59 - jepsen.util 329 :invoke :write [19 37] INFO [2022-03-13 17:07:26,901] jepsen worker 43 - jepsen.util 133 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,903] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:26,928] jepsen worker 7 - jepsen.util 97 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,928] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] WARN [2022-03-13 17:07:26,974] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17098997,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17686308,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9292266,"encode_duration_us":82733,"operation_id":"0x6bc","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10001299},{"operation_name":"upsert","last_dispatch_duration_us":16323325,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16524665,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9153059,"encode_duration_us":38559,"operation_id":"0x743","timeout_ms":10000,"last_server_duration_us":9078800,"total_duration_us":10005958},{"operation_name":"upsert","last_dispatch_duration_us":3495500,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9966995,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":3825674,"encode_duration_us":18637,"operation_id":"0x91f","timeout_ms":10000,"last_server_duration_us":3469265,"total_duration_us":10005148},{"operation_name":"upsert","last_dispatch_duration_us":17129741,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17659601,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9298476,"encode_duration_us":159464,"operation_id":"0x6c4","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":10009400}],"total_count":4}} INFO [2022-03-13 17:07:26,991] jepsen worker 23 - jepsen.util 113 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,991] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,992] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:26,993] jepsen worker 76 - jepsen.util 166 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:07:26,993] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:26,994] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:27,027] jepsen worker 14 - jepsen.util 14 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,027] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:27,028] jepsen worker 14 - jepsen.util 14 :invoke :write [4 45] INFO [2022-03-13 17:07:27,029] jepsen worker 69 - jepsen.util 159 :invoke :write [23 0] INFO [2022-03-13 17:07:27,048] jepsen worker 26 - jepsen.util 116 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,048] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:27,048] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,048] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:27,049] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,049] jepsen worker 7 - jepsen.util 97 :invoke :write [2 19] INFO [2022-03-13 17:07:27,050] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:27,051] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:27,051] jepsen worker 26 - jepsen.util 116 :invoke :write [8 4] INFO [2022-03-13 17:07:27,052] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:27,092] jepsen worker 34 - jepsen.util 124 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,093] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:27,093] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:07:27,094] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:07:27,095] jepsen worker 34 - jepsen.util 124 :invoke :write [11 37] INFO [2022-03-13 17:07:27,095] jepsen worker 43 - jepsen.util 133 :invoke :write [14 19] INFO [2022-03-13 17:07:27,117] jepsen worker 52 - jepsen.util 142 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,118] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:27,118] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:27,119] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,119] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:27,120] jepsen worker 42 - jepsen.util 132 :invoke :write [14 11] INFO [2022-03-13 17:07:27,121] jepsen worker 54 - jepsen.util 234 :info :write [18 3] :RequestTimeoutException INFO [2022-03-13 17:07:27,121] jepsen worker 23 - jepsen.util 113 :invoke :write [7 24] INFO [2022-03-13 17:07:27,122] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:27,122] jepsen worker 54 - jepsen.util 324 :invoke :read [18 nil] INFO [2022-03-13 17:07:27,153] jepsen worker 74 - jepsen.util 254 :ok :read [24 24] INFO [2022-03-13 17:07:27,154] jepsen worker 74 - jepsen.util 254 :invoke :write [24 23] INFO [2022-03-13 17:07:27,169] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:27,169] jepsen worker 13 - jepsen.util 103 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,170] jepsen worker 13 - jepsen.util 103 :invoke :write [4 11] INFO [2022-03-13 17:07:27,170] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:27,187] jepsen worker 20 - jepsen.util 200 :info :write [6 42] :RequestTimeoutException INFO [2022-03-13 17:07:27,187] jepsen worker 20 - jepsen.util 290 :invoke :write [6 47] INFO [2022-03-13 17:07:27,191] jepsen worker 26 - jepsen.util 116 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,192] jepsen worker 26 - jepsen.util 116 :invoke :write [8 2] INFO [2022-03-13 17:07:27,206] jepsen worker 83 - jepsen.util 173 :info :write [27 47] :RequestTimeoutException INFO [2022-03-13 17:07:27,207] jepsen worker 83 - jepsen.util 263 :invoke :read [27 nil] INFO [2022-03-13 17:07:27,247] jepsen worker 14 - jepsen.util 14 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,247] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:27,267] jepsen worker 18 - jepsen.util 108 :info :write [6 7] :RequestTimeoutException INFO [2022-03-13 17:07:27,267] jepsen worker 18 - jepsen.util 198 :invoke :read [6 nil] INFO [2022-03-13 17:07:27,275] jepsen worker 7 - jepsen.util 97 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,276] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,276] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,276] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:27,277] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:27,278] jepsen worker 52 - jepsen.util 142 :invoke :write [17 16] INFO [2022-03-13 17:07:27,324] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:07:27,324] jepsen worker 69 - jepsen.util 159 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,324] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:07:27,325] jepsen worker 69 - jepsen.util 159 :invoke :write [23 36] INFO [2022-03-13 17:07:27,351] jepsen worker 23 - jepsen.util 113 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,352] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:27,398] jepsen worker 43 - jepsen.util 133 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,399] jepsen worker 43 - jepsen.util 133 :invoke :write [14 2] INFO [2022-03-13 17:07:27,427] jepsen worker 42 - jepsen.util 132 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,428] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:27,457] jepsen worker 32 - jepsen.util 122 :info :write [10 49] :SyncWriteAmbiguous INFO [2022-03-13 17:07:27,458] jepsen worker 32 - jepsen.util 212 :invoke :read [10 nil] INFO [2022-03-13 17:07:27,479] jepsen worker 34 - jepsen.util 124 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,480] jepsen worker 34 - jepsen.util 124 :invoke :write [11 8] INFO [2022-03-13 17:07:27,480] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:27,481] jepsen worker 76 - jepsen.util 166 :invoke :write [25 27] INFO [2022-03-13 17:07:27,491] jepsen worker 1 - jepsen.util 91 :info :write [0 30] :RequestTimeoutException INFO [2022-03-13 17:07:27,496] jepsen worker 1 - jepsen.util 181 :invoke :write [0 32] INFO [2022-03-13 17:07:27,513] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,513] jepsen worker 26 - jepsen.util 116 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,514] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,515] jepsen worker 26 - jepsen.util 116 :invoke :read [8 nil] INFO [2022-03-13 17:07:27,516] jepsen worker 7 - jepsen.util 97 :invoke :write [2 34] INFO [2022-03-13 17:07:27,516] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:27,557] jepsen worker 68 - jepsen.util 248 :info :write [22 14] :RequestTimeoutException INFO [2022-03-13 17:07:27,558] jepsen worker 68 - jepsen.util 338 :invoke :read [22 nil] INFO [2022-03-13 17:07:27,559] jepsen worker 13 - jepsen.util 103 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,559] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:27,590] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:27,590] jepsen worker 52 - jepsen.util 142 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,590] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:27,590] jepsen worker 26 - jepsen.util 116 :ok :read [8 :nil] INFO [2022-03-13 17:07:27,590] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,592] jepsen worker 26 - jepsen.util 116 :invoke :write [8 42] INFO [2022-03-13 17:07:27,593] jepsen worker 42 - jepsen.util 132 :invoke :write [14 15] INFO [2022-03-13 17:07:27,607] jepsen worker 6 - jepsen.util 186 :invoke :write [2 41] INFO [2022-03-13 17:07:27,622] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:27,623] jepsen worker 23 - jepsen.util 113 :invoke :write [7 18] INFO [2022-03-13 17:07:27,636] jepsen worker 75 - jepsen.util 75 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,638] jepsen worker 75 - jepsen.util 75 :invoke :write [25 5] INFO [2022-03-13 17:07:27,638] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:07:27,639] jepsen worker 71 - jepsen.util 71 :invoke :read [23 nil] INFO [2022-03-13 17:07:27,669] jepsen worker 7 - jepsen.util 97 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,669] jepsen worker 7 - jepsen.util 97 :invoke :write [2 26] INFO [2022-03-13 17:07:27,722] jepsen worker 69 - jepsen.util 159 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,723] jepsen worker 69 - jepsen.util 159 :invoke :write [23 42] INFO [2022-03-13 17:07:27,757] jepsen worker 26 - jepsen.util 116 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,758] jepsen worker 26 - jepsen.util 116 :invoke :write [8 13] INFO [2022-03-13 17:07:27,847] jepsen worker 64 - jepsen.util 154 :ok :read [21 :nil] INFO [2022-03-13 17:07:27,848] jepsen worker 64 - jepsen.util 154 :invoke :read [21 nil] INFO [2022-03-13 17:07:27,854] jepsen worker 43 - jepsen.util 133 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,855] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:27,875] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,875] jepsen worker 42 - jepsen.util 132 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,877] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:27,877] jepsen worker 42 - jepsen.util 132 :invoke :write [14 40] INFO [2022-03-13 17:07:27,947] jepsen worker 34 - jepsen.util 124 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,948] jepsen worker 34 - jepsen.util 124 :invoke :write [11 37] INFO [2022-03-13 17:07:27,968] jepsen worker 6 - jepsen.util 186 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:07:27,969] jepsen worker 6 - jepsen.util 186 :invoke :write [2 46] INFO [2022-03-13 17:07:27,999] jepsen worker 17 - jepsen.util 197 :info :write [5 9] :RequestTimeoutException INFO [2022-03-13 17:07:28,000] jepsen worker 17 - jepsen.util 287 :invoke :read [5 nil] INFO [2022-03-13 17:07:28,010] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:28,010] jepsen worker 2 - jepsen.util 182 :info :write [0 29] :RequestTimeoutException INFO [2022-03-13 17:07:28,011] jepsen worker 76 - jepsen.util 166 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,011] jepsen worker 2 - jepsen.util 272 :invoke :write [0 4] INFO [2022-03-13 17:07:28,012] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:28,013] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,020] jepsen worker 19 - jepsen.util 199 :ok :read [6 :nil] INFO [2022-03-13 17:07:28,021] jepsen worker 19 - jepsen.util 199 :invoke :read [6 nil] INFO [2022-03-13 17:07:28,033] jepsen worker 29 - jepsen.util 209 :ok :read [9 14] INFO [2022-03-13 17:07:28,034] jepsen worker 29 - jepsen.util 209 :invoke :write [9 18] INFO [2022-03-13 17:07:28,039] jepsen worker 23 - jepsen.util 113 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,040] jepsen worker 23 - jepsen.util 113 :invoke :write [7 7] INFO [2022-03-13 17:07:28,108] jepsen worker 10 - jepsen.util 100 :info :write [3 1] :RequestTimeoutException INFO [2022-03-13 17:07:28,109] jepsen worker 10 - jepsen.util 190 :invoke :read [3 nil] INFO [2022-03-13 17:07:28,129] jepsen worker 75 - jepsen.util 75 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,129] jepsen worker 71 - jepsen.util 71 :ok :read [23 :nil] INFO [2022-03-13 17:07:28,129] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,130] jepsen worker 71 - jepsen.util 71 :invoke :write [23 31] INFO [2022-03-13 17:07:28,156] jepsen worker 56 - jepsen.util 146 :info :write [18 12] :RequestTimeoutException INFO [2022-03-13 17:07:28,157] jepsen worker 56 - jepsen.util 236 :invoke :read [18 nil] INFO [2022-03-13 17:07:28,164] jepsen worker 7 - jepsen.util 97 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,165] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:28,177] jepsen worker 54 - jepsen.util 324 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,178] jepsen worker 54 - jepsen.util 324 :invoke :write [18 8] INFO [2022-03-13 17:07:28,211] jepsen worker 55 - jepsen.util 235 :info :write [18 2] :RequestTimeoutException INFO [2022-03-13 17:07:28,212] jepsen worker 55 - jepsen.util 325 :invoke :read [18 nil] INFO [2022-03-13 17:07:28,245] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:28,245] jepsen worker 69 - jepsen.util 159 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,247] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:28,247] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:28,264] jepsen worker 5 - jepsen.util 95 :info :write [1 18] :RequestTimeoutException INFO [2022-03-13 17:07:28,265] jepsen worker 5 - jepsen.util 185 :invoke :read [1 nil] INFO [2022-03-13 17:07:28,291] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,291] jepsen worker 26 - jepsen.util 116 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,292] jepsen worker 52 - jepsen.util 142 :invoke :write [17 48] INFO [2022-03-13 17:07:28,293] jepsen worker 26 - jepsen.util 116 :invoke :write [8 40] INFO [2022-03-13 17:07:28,316] jepsen worker 4 - jepsen.util 184 :info :write [1 49] :RequestTimeoutException INFO [2022-03-13 17:07:28,317] jepsen worker 4 - jepsen.util 274 :invoke :read [1 nil] INFO [2022-03-13 17:07:28,342] jepsen worker 51 - jepsen.util 51 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,343] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:28,358] jepsen worker 31 - jepsen.util 121 :info :write [10 45] :RequestTimeoutException INFO [2022-03-13 17:07:28,359] jepsen worker 31 - jepsen.util 211 :invoke :write [10 18] INFO [2022-03-13 17:07:28,363] jepsen worker 42 - jepsen.util 132 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,364] jepsen worker 42 - jepsen.util 132 :invoke :write [14 2] INFO [2022-03-13 17:07:28,412] jepsen worker 34 - jepsen.util 124 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,412] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,413] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:28,413] jepsen worker 75 - jepsen.util 75 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,413] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,414] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,414] jepsen worker 13 - jepsen.util 103 :invoke :write [4 37] INFO [2022-03-13 17:07:28,415] jepsen worker 34 - jepsen.util 124 :invoke :read [11 nil] INFO [2022-03-13 17:07:28,443] jepsen worker 72 - jepsen.util 252 :info :write [24 36] :RequestTimeoutException INFO [2022-03-13 17:07:28,444] jepsen worker 72 - jepsen.util 342 :invoke :read [24 nil] INFO [2022-03-13 17:07:28,473] jepsen worker 6 - jepsen.util 186 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,474] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:28,488] jepsen worker 77 - jepsen.util 77 :info :write [25 6] :RequestTimeoutException INFO [2022-03-13 17:07:28,489] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,537] jepsen worker 61 - jepsen.util 151 :info :write [20 41] :RequestTimeoutException INFO [2022-03-13 17:07:28,538] jepsen worker 61 - jepsen.util 241 :invoke :read [20 nil] INFO [2022-03-13 17:07:28,540] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:28,540] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:28,540] jepsen worker 71 - jepsen.util 71 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,540] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:28,541] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,541] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,541] jepsen worker 75 - jepsen.util 75 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,541] jepsen worker 43 - jepsen.util 133 :invoke :write [14 20] INFO [2022-03-13 17:07:28,542] jepsen worker 34 - jepsen.util 124 :ok :read [11 :nil] INFO [2022-03-13 17:07:28,542] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:28,542] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:28,543] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:28,544] jepsen worker 71 - jepsen.util 71 :invoke :write [23 1] INFO [2022-03-13 17:07:28,544] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,545] jepsen worker 34 - jepsen.util 124 :invoke :read [11 nil] INFO [2022-03-13 17:07:28,545] jepsen worker 75 - jepsen.util 75 :invoke :write [25 7] INFO [2022-03-13 17:07:28,576] jepsen worker 23 - jepsen.util 113 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,576] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,577] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:28,578] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:28,629] jepsen worker 77 - jepsen.util 167 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,629] jepsen worker 13 - jepsen.util 103 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,630] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:28,631] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:28,631] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,631] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,631] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:28,632] jepsen worker 34 - jepsen.util 124 :ok :read [11 :nil] INFO [2022-03-13 17:07:28,632] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:28,633] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:28,634] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,634] jepsen worker 34 - jepsen.util 124 :invoke :write [11 41] INFO [2022-03-13 17:07:28,635] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,635] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:28,637] jepsen worker 78 - jepsen.util 168 :info :write [26 47] :RequestTimeoutException INFO [2022-03-13 17:07:28,637] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:28,684] jepsen worker 52 - jepsen.util 142 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,685] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:28,721] jepsen worker 43 - jepsen.util 133 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,722] jepsen worker 43 - jepsen.util 133 :invoke :write [14 49] INFO [2022-03-13 17:07:28,743] jepsen worker 26 - jepsen.util 116 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,744] jepsen worker 26 - jepsen.util 116 :invoke :write [8 32] INFO [2022-03-13 17:07:28,744] jepsen worker 25 - jepsen.util 115 :info :write [8 44] :RequestTimeoutException INFO [2022-03-13 17:07:28,745] jepsen worker 53 - jepsen.util 53 :info :write [17 39] :RequestTimeoutException INFO [2022-03-13 17:07:28,745] jepsen worker 33 - jepsen.util 123 :info :write [11 38] :RequestTimeoutException INFO [2022-03-13 17:07:28,746] jepsen worker 25 - jepsen.util 205 :invoke :read [8 nil] INFO [2022-03-13 17:07:28,747] jepsen worker 85 - jepsen.util 265 :info :write [28 25] :RequestTimeoutException INFO [2022-03-13 17:07:28,748] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:28,749] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:28,750] jepsen worker 85 - jepsen.util 355 :invoke :read [28 nil] INFO [2022-03-13 17:07:28,764] jepsen worker 30 - jepsen.util 120 :info :write [10 47] :RequestTimeoutException INFO [2022-03-13 17:07:28,768] jepsen worker 30 - jepsen.util 210 :invoke :write [10 33] INFO [2022-03-13 17:07:28,792] jepsen worker 71 - jepsen.util 71 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,793] jepsen worker 71 - jepsen.util 71 :invoke :write [23 33] INFO [2022-03-13 17:07:28,798] jepsen worker 62 - jepsen.util 152 :info :write [20 35] :RequestTimeoutException INFO [2022-03-13 17:07:28,799] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:28,876] jepsen worker 42 - jepsen.util 132 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,876] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,877] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:28,878] jepsen worker 42 - jepsen.util 132 :invoke :write [14 14] INFO [2022-03-13 17:07:28,878] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,878] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,878] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:28,879] jepsen worker 25 - jepsen.util 205 :ok :read [8 :nil] INFO [2022-03-13 17:07:28,879] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:28,880] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:28,880] jepsen worker 52 - jepsen.util 142 :invoke :write [17 12] INFO [2022-03-13 17:07:28,881] jepsen worker 25 - jepsen.util 205 :invoke :read [8 nil] INFO [2022-03-13 17:07:28,906] jepsen worker 22 - jepsen.util 112 :info :write [7 31] :RequestTimeoutException INFO [2022-03-13 17:07:28,907] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:28,934] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:28,934] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:28,934] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,933] jepsen worker 75 - jepsen.util 75 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,934] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:28,934] jepsen worker 77 - jepsen.util 167 :ok :read [25 :nil] INFO [2022-03-13 17:07:28,935] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:28,935] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,935] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,936] jepsen worker 76 - jepsen.util 166 :invoke :write [25 27] INFO [2022-03-13 17:07:28,937] jepsen worker 69 - jepsen.util 159 :invoke :write [23 27] INFO [2022-03-13 17:07:28,937] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:28,937] jepsen worker 13 - jepsen.util 103 :invoke :write [4 26] INFO [2022-03-13 17:07:28,942] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:28,958] jepsen worker 12 - jepsen.util 102 :info :write [4 16] :RequestTimeoutException INFO [2022-03-13 17:07:28,960] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:28,964] jepsen worker 26 - jepsen.util 116 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,964] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,965] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:28,965] jepsen worker 25 - jepsen.util 205 :ok :read [8 :nil] INFO [2022-03-13 17:07:28,965] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:28,966] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:28,967] jepsen worker 25 - jepsen.util 205 :invoke :write [8 21] INFO [2022-03-13 17:07:28,968] jepsen worker 6 - jepsen.util 186 :invoke :write [2 37] INFO [2022-03-13 17:07:28,968] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:28,968] jepsen worker 26 - jepsen.util 116 :invoke :write [8 10] INFO [2022-03-13 17:07:29,009] jepsen worker 34 - jepsen.util 124 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,011] jepsen worker 34 - jepsen.util 124 :invoke :write [11 4] INFO [2022-03-13 17:07:29,033] jepsen worker 42 - jepsen.util 132 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,034] jepsen worker 42 - jepsen.util 132 :invoke :write [14 22] INFO [2022-03-13 17:07:29,101] jepsen worker 36 - jepsen.util 216 :ok :read [12 46] INFO [2022-03-13 17:07:29,101] jepsen worker 36 - jepsen.util 216 :invoke :read [12 nil] INFO [2022-03-13 17:07:29,103] jepsen worker 43 - jepsen.util 133 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,104] jepsen worker 43 - jepsen.util 133 :invoke :write [14 28] INFO [2022-03-13 17:07:29,107] jepsen worker 46 - jepsen.util 226 :info :write [15 41] :RequestTimeoutException INFO [2022-03-13 17:07:29,107] jepsen worker 46 - jepsen.util 316 :invoke :read [15 nil] INFO [2022-03-13 17:07:29,117] jepsen worker 80 - jepsen.util 170 :info :write [26 9] :RequestTimeoutException INFO [2022-03-13 17:07:29,117] jepsen worker 80 - jepsen.util 260 :invoke :read [26 nil] INFO [2022-03-13 17:07:29,128] jepsen worker 21 - jepsen.util 111 :info :write [7 31] :RequestTimeoutException INFO [2022-03-13 17:07:29,129] jepsen worker 21 - jepsen.util 201 :invoke :write [7 34] INFO [2022-03-13 17:07:29,144] jepsen worker 52 - jepsen.util 142 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,144] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:29,144] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,145] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:29,145] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:29,146] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:29,147] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:29,147] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:29,195] jepsen worker 36 - jepsen.util 216 :ok :read [12 46] INFO [2022-03-13 17:07:29,196] jepsen worker 36 - jepsen.util 216 :invoke :read [12 nil] INFO [2022-03-13 17:07:29,215] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:29,215] jepsen worker 71 - jepsen.util 71 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,216] jepsen worker 75 - jepsen.util 75 :ok :read [25 :nil] INFO [2022-03-13 17:07:29,216] jepsen worker 14 - jepsen.util 14 :invoke :write [4 17] INFO [2022-03-13 17:07:29,217] jepsen worker 71 - jepsen.util 71 :invoke :write [23 48] INFO [2022-03-13 17:07:29,219] jepsen worker 36 - jepsen.util 216 :ok :read [12 46] INFO [2022-03-13 17:07:29,219] jepsen worker 36 - jepsen.util 216 :invoke :read [12 nil] INFO [2022-03-13 17:07:29,220] jepsen worker 75 - jepsen.util 75 :invoke :write [25 18] INFO [2022-03-13 17:07:29,224] jepsen worker 77 - jepsen.util 167 :ok :read [25 :nil] INFO [2022-03-13 17:07:29,225] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,226] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:29,226] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:29,226] jepsen worker 60 - jepsen.util 150 :info :write [20 11] :RequestTimeoutException INFO [2022-03-13 17:07:29,227] jepsen worker 60 - jepsen.util 240 :invoke :write [20 4] INFO [2022-03-13 17:07:29,249] jepsen worker 36 - jepsen.util 216 :ok :read [12 46] INFO [2022-03-13 17:07:29,250] jepsen worker 36 - jepsen.util 216 :invoke :read [12 nil] INFO [2022-03-13 17:07:29,255] jepsen worker 25 - jepsen.util 205 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,255] jepsen worker 25 - jepsen.util 205 :invoke :write [8 4] INFO [2022-03-13 17:07:29,280] jepsen worker 36 - jepsen.util 216 :ok :read [12 46] INFO [2022-03-13 17:07:29,282] jepsen worker 36 - jepsen.util 216 :invoke :read [12 nil] INFO [2022-03-13 17:07:29,309] jepsen worker 36 - jepsen.util 216 :ok :read [12 46] INFO [2022-03-13 17:07:29,309] jepsen worker 36 - jepsen.util 216 :invoke :write [12 34] INFO [2022-03-13 17:07:29,336] jepsen worker 13 - jepsen.util 103 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,337] jepsen worker 13 - jepsen.util 103 :invoke :write [4 47] INFO [2022-03-13 17:07:29,347] jepsen worker 84 - jepsen.util 264 :info :write [28 42] :RequestTimeoutException INFO [2022-03-13 17:07:29,348] jepsen worker 84 - jepsen.util 354 :invoke :write [28 26] INFO [2022-03-13 17:07:29,356] jepsen worker 26 - jepsen.util 116 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,356] jepsen worker 26 - jepsen.util 116 :invoke :read [8 nil] INFO [2022-03-13 17:07:29,390] jepsen worker 76 - jepsen.util 166 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,391] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:29,434] jepsen worker 6 - jepsen.util 186 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,435] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:29,471] jepsen worker 37 - jepsen.util 217 :info :write [12 11] :RequestTimeoutException INFO [2022-03-13 17:07:29,472] jepsen worker 37 - jepsen.util 307 :invoke :read [12 nil] INFO [2022-03-13 17:07:29,473] jepsen worker 4 - jepsen.util 274 :ok :read [1 28] INFO [2022-03-13 17:07:29,474] jepsen worker 4 - jepsen.util 274 :invoke :write [1 9] INFO [2022-03-13 17:07:29,512] jepsen worker 69 - jepsen.util 159 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,513] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:29,557] jepsen worker 42 - jepsen.util 132 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,557] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,557] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:29,557] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,557] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:29,559] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:29,560] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:29,561] jepsen worker 22 - jepsen.util 202 :invoke :write [7 37] INFO [2022-03-13 17:07:29,562] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:29,563] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:29,610] jepsen worker 34 - jepsen.util 124 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,610] jepsen worker 77 - jepsen.util 167 :ok :read [25 :nil] INFO [2022-03-13 17:07:29,610] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,612] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:29,613] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:29,614] jepsen worker 34 - jepsen.util 124 :invoke :write [11 20] INFO [2022-03-13 17:07:29,643] jepsen worker 25 - jepsen.util 205 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,644] jepsen worker 26 - jepsen.util 116 :ok :read [8 :nil] INFO [2022-03-13 17:07:29,644] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,644] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:29,644] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,645] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,645] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:29,645] jepsen worker 25 - jepsen.util 205 :invoke :write [8 9] INFO [2022-03-13 17:07:29,646] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:29,647] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:29,648] jepsen worker 26 - jepsen.util 116 :invoke :write [8 37] INFO [2022-03-13 17:07:29,649] jepsen worker 52 - jepsen.util 142 :invoke :write [17 20] INFO [2022-03-13 17:07:29,651] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:29,651] jepsen worker 23 - jepsen.util 113 :invoke :write [7 2] INFO [2022-03-13 17:07:29,702] jepsen worker 71 - jepsen.util 71 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,705] jepsen worker 71 - jepsen.util 71 :invoke :write [23 21] INFO [2022-03-13 17:07:29,721] jepsen worker nemesis - jepsen.util :nemesis :info :swap-rebalance nil INFO [2022-03-13 17:07:29,734] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:29,734] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,734] jepsen worker 22 - jepsen.util 202 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,735] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:29,735] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:29,734] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,736] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:29,736] jepsen worker 6 - jepsen.util 186 :invoke :write [2 0] INFO [2022-03-13 17:07:29,788] jepsen worker 63 - jepsen.util 243 :info :write [21 32] :RequestTimeoutException INFO [2022-03-13 17:07:29,788] jepsen worker 63 - jepsen.util 333 :invoke :read [21 nil] INFO [2022-03-13 17:07:29,800] jepsen worker 75 - jepsen.util 75 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,800] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:29,827] jepsen worker 25 - jepsen.util 205 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,828] jepsen worker 25 - jepsen.util 205 :invoke :write [8 0] INFO [2022-03-13 17:07:29,851] jepsen worker 15 - jepsen.util 195 :info :write [5 7] :RequestTimeoutException INFO [2022-03-13 17:07:29,851] jepsen worker 15 - jepsen.util 285 :invoke :write [5 19] INFO [2022-03-13 17:07:29,870] jepsen worker 14 - jepsen.util 14 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,870] jepsen worker 76 - jepsen.util 166 :ok :read [25 :nil] INFO [2022-03-13 17:07:29,870] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:29,871] jepsen worker 69 - jepsen.util 159 :invoke :write [23 34] INFO [2022-03-13 17:07:29,871] jepsen worker 14 - jepsen.util 14 :invoke :write [4 45] INFO [2022-03-13 17:07:29,872] jepsen worker 76 - jepsen.util 166 :invoke :write [25 16] INFO [2022-03-13 17:07:29,891] jepsen worker 26 - jepsen.util 116 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,892] jepsen worker 26 - jepsen.util 116 :invoke :write [8 12] INFO [2022-03-13 17:07:29,933] jepsen worker 13 - jepsen.util 103 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,934] jepsen worker 77 - jepsen.util 167 :ok :read [25 :nil] INFO [2022-03-13 17:07:29,934] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,934] jepsen worker 13 - jepsen.util 103 :invoke :write [4 49] INFO [2022-03-13 17:07:29,935] jepsen worker 51 - jepsen.util 51 :invoke :write [17 40] INFO [2022-03-13 17:07:29,935] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:29,958] jepsen worker 24 - jepsen.util 24 :info :write [8 23] :RequestTimeoutException INFO [2022-03-13 17:07:29,958] jepsen worker 23 - jepsen.util 113 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,960] jepsen worker 24 - jepsen.util 114 :invoke :read [8 nil] INFO [2022-03-13 17:07:29,961] jepsen worker 23 - jepsen.util 113 :invoke :write [7 19] INFO [2022-03-13 17:07:29,999] jepsen worker 34 - jepsen.util 124 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:07:29,999] jepsen worker 80 - jepsen.util 260 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,000] jepsen worker 34 - jepsen.util 124 :invoke :write [11 43] INFO [2022-03-13 17:07:30,000] jepsen worker 80 - jepsen.util 260 :invoke :write [26 4] INFO [2022-03-13 17:07:30,021] jepsen worker 52 - jepsen.util 142 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,022] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:30,022] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:30,023] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,024] jepsen worker 22 - jepsen.util 202 :invoke :write [7 26] INFO [2022-03-13 17:07:30,024] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:30,025] jepsen worker 52 - jepsen.util 142 :invoke :write [17 40] INFO [2022-03-13 17:07:30,025] jepsen worker 42 - jepsen.util 132 :invoke :write [14 43] INFO [2022-03-13 17:07:30,076] jepsen worker 40 - jepsen.util 220 :info :write [13 10] :RequestTimeoutException INFO [2022-03-13 17:07:30,077] jepsen worker 40 - jepsen.util 310 :invoke :read [13 nil] INFO [2022-03-13 17:07:30,091] jepsen worker 8 - jepsen.util 188 :info :write [2 5] :RequestTimeoutException INFO [2022-03-13 17:07:30,092] jepsen worker 8 - jepsen.util 278 :invoke :write [2 38] INFO [2022-03-13 17:07:30,100] jepsen worker 43 - jepsen.util 133 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,100] jepsen worker 75 - jepsen.util 75 :ok :read [25 :nil] INFO [2022-03-13 17:07:30,101] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:30,102] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:30,121] jepsen worker 17 - jepsen.util 287 :ok :read [5 :nil] INFO [2022-03-13 17:07:30,121] jepsen worker 6 - jepsen.util 186 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,122] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:30,123] jepsen worker 17 - jepsen.util 287 :invoke :read [5 nil] INFO [2022-03-13 17:07:30,124] jepsen worker 17 - jepsen.util 287 :ok :read [5 :nil] INFO [2022-03-13 17:07:30,125] jepsen worker 17 - jepsen.util 287 :invoke :write [5 32] INFO [2022-03-13 17:07:30,148] jepsen worker 41 - jepsen.util 221 :info :write [13 45] :RequestTimeoutException INFO [2022-03-13 17:07:30,150] jepsen worker 41 - jepsen.util 311 :invoke :write [13 36] INFO [2022-03-13 17:07:30,168] jepsen worker 69 - jepsen.util 159 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,169] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:30,200] jepsen worker 45 - jepsen.util 225 :info :write [15 45] :RequestTimeoutException INFO [2022-03-13 17:07:30,201] jepsen worker 45 - jepsen.util 315 :invoke :write [15 6] INFO [2022-03-13 17:07:30,204] jepsen worker 39 - jepsen.util 219 :info :write [13 32] :RequestTimeoutException INFO [2022-03-13 17:07:30,205] jepsen worker 39 - jepsen.util 309 :invoke :write [13 19] INFO [2022-03-13 17:07:30,240] jepsen worker 25 - jepsen.util 205 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,241] jepsen worker 25 - jepsen.util 205 :invoke :read [8 nil] INFO [2022-03-13 17:07:30,267] jepsen worker 47 - jepsen.util 227 :info :write [15 48] :RequestTimeoutException INFO [2022-03-13 17:07:30,268] jepsen worker 47 - jepsen.util 317 :invoke :write [15 39] INFO [2022-03-13 17:07:30,292] jepsen worker 28 - jepsen.util 208 :info :write [9 20] :RequestTimeoutException INFO [2022-03-13 17:07:30,292] jepsen worker 28 - jepsen.util 298 :invoke :read [9 nil] INFO [2022-03-13 17:07:30,298] jepsen worker 87 - jepsen.util 267 :info :write [29 44] :RequestTimeoutException INFO [2022-03-13 17:07:30,299] jepsen worker 87 - jepsen.util 357 :invoke :write [29 34] INFO [2022-03-13 17:07:30,304] jepsen worker 14 - jepsen.util 14 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,305] jepsen worker 14 - jepsen.util 14 :invoke :write [4 29] INFO [2022-03-13 17:07:30,326] jepsen worker 26 - jepsen.util 116 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,326] jepsen worker 26 - jepsen.util 116 :invoke :read [8 nil] INFO [2022-03-13 17:07:30,335] jepsen worker 70 - jepsen.util 160 :info :write [23 17] :RequestTimeoutException INFO [2022-03-13 17:07:30,336] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:30,354] jepsen worker 28 - jepsen.util 298 :ok :read [9 14] INFO [2022-03-13 17:07:30,354] jepsen worker 28 - jepsen.util 298 :invoke :read [9 nil] INFO [2022-03-13 17:07:30,361] jepsen worker 65 - jepsen.util 245 :info :write [21 33] :RequestTimeoutException INFO [2022-03-13 17:07:30,362] jepsen worker 65 - jepsen.util 335 :invoke :read [21 nil] INFO [2022-03-13 17:07:30,367] jepsen worker 51 - jepsen.util 51 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,367] jepsen worker 51 - jepsen.util 51 :invoke :write [17 30] INFO [2022-03-13 17:07:30,387] jepsen worker 28 - jepsen.util 298 :ok :read [9 14] INFO [2022-03-13 17:07:30,388] jepsen worker 28 - jepsen.util 298 :invoke :read [9 nil] INFO [2022-03-13 17:07:30,394] jepsen worker 23 - jepsen.util 113 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,395] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:30,423] jepsen worker 77 - jepsen.util 167 :ok :read [25 6] INFO [2022-03-13 17:07:30,424] jepsen worker 77 - jepsen.util 167 :invoke :write [25 5] INFO [2022-03-13 17:07:30,426] jepsen worker 28 - jepsen.util 298 :ok :read [9 14] INFO [2022-03-13 17:07:30,427] jepsen worker 28 - jepsen.util 298 :invoke :read [9 nil] INFO [2022-03-13 17:07:30,456] jepsen worker 28 - jepsen.util 298 :ok :read [9 14] INFO [2022-03-13 17:07:30,457] jepsen worker 28 - jepsen.util 298 :invoke :write [9 36] INFO [2022-03-13 17:07:30,461] jepsen worker 13 - jepsen.util 103 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,462] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:30,515] jepsen worker 42 - jepsen.util 132 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,515] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,516] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:30,516] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] INFO [2022-03-13 17:07:30,555] jepsen worker 80 - jepsen.util 260 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,555] jepsen worker 80 - jepsen.util 260 :invoke :read [26 nil] INFO [2022-03-13 17:07:30,576] jepsen worker 22 - jepsen.util 202 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,576] jepsen worker 26 - jepsen.util 116 :ok :read [8 :nil] INFO [2022-03-13 17:07:30,577] jepsen worker 25 - jepsen.util 205 :ok :read [8 :nil] INFO [2022-03-13 17:07:30,578] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:30,578] jepsen worker 26 - jepsen.util 116 :invoke :read [8 nil] INFO [2022-03-13 17:07:30,578] jepsen worker 25 - jepsen.util 205 :invoke :write [8 48] INFO [2022-03-13 17:07:30,611] jepsen worker 34 - jepsen.util 124 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,612] jepsen worker 34 - jepsen.util 124 :invoke :write [11 34] INFO [2022-03-13 17:07:30,633] jepsen worker 52 - jepsen.util 142 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,633] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:30,635] jepsen worker 23 - jepsen.util 113 :invoke :write [7 10] INFO [2022-03-13 17:07:30,636] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:30,676] jepsen worker 14 - jepsen.util 14 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,676] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:30,676] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:30,677] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:30,698] jepsen worker 8 - jepsen.util 278 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,698] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,699] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,699] jepsen worker 8 - jepsen.util 278 :invoke :write [2 35] INFO [2022-03-13 17:07:30,699] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:30,699] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:30,700] jepsen worker 26 - jepsen.util 116 :ok :read [8 :nil] INFO [2022-03-13 17:07:30,700] jepsen worker 6 - jepsen.util 186 :invoke :write [2 17] INFO [2022-03-13 17:07:30,702] jepsen worker 26 - jepsen.util 116 :invoke :write [8 13] INFO [2022-03-13 17:07:30,702] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:30,703] jepsen worker 42 - jepsen.util 132 :invoke :write [14 11] INFO [2022-03-13 17:07:30,706] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:30,769] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:30,769] jepsen worker 51 - jepsen.util 51 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,769] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:30,770] jepsen worker 51 - jepsen.util 51 :invoke :write [17 25] INFO [2022-03-13 17:07:30,814] jepsen worker 25 - jepsen.util 205 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,814] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,815] jepsen worker 52 - jepsen.util 142 :invoke :write [17 46] INFO [2022-03-13 17:07:30,815] jepsen worker 25 - jepsen.util 205 :invoke :write [8 23] INFO [2022-03-13 17:07:30,881] jepsen worker 77 - jepsen.util 167 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,881] jepsen worker 80 - jepsen.util 260 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,882] jepsen worker 77 - jepsen.util 167 :invoke :write [25 46] INFO [2022-03-13 17:07:30,882] jepsen worker 80 - jepsen.util 260 :invoke :read [26 nil] INFO [2022-03-13 17:07:30,902] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:30,902] jepsen worker 23 - jepsen.util 113 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,902] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,902] jepsen worker 23 - jepsen.util 113 :invoke :write [7 18] INFO [2022-03-13 17:07:30,903] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:30,904] jepsen worker 22 - jepsen.util 202 :invoke :write [7 6] INFO [2022-03-13 17:07:30,938] jepsen worker 34 - jepsen.util 124 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,938] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:30,939] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:30,939] jepsen worker 34 - jepsen.util 124 :invoke :write [11 14] INFO [2022-03-13 17:07:30,959] jepsen worker 6 - jepsen.util 186 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:07:30,960] jepsen worker 6 - jepsen.util 186 :invoke :write [2 0] INFO [2022-03-13 17:07:30,971] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:30,971] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:30,972] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:30,973] jepsen worker 13 - jepsen.util 103 :invoke :write [4 8] INFO [2022-03-13 17:07:30,973] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:30,974] jepsen worker 69 - jepsen.util 159 :invoke :write [23 8] INFO [2022-03-13 17:07:30,987] jepsen worker 75 - jepsen.util 75 :ok :read [25 6] INFO [2022-03-13 17:07:30,988] jepsen worker 75 - jepsen.util 75 :invoke :write [25 32] INFO [2022-03-13 17:07:31,009] jepsen worker 51 - jepsen.util 51 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,010] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:31,052] jepsen worker 8 - jepsen.util 278 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,053] jepsen worker 8 - jepsen.util 278 :invoke :write [2 37] INFO [2022-03-13 17:07:31,131] jepsen worker 80 - jepsen.util 260 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,131] jepsen worker 71 - jepsen.util 71 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,132] jepsen worker 80 - jepsen.util 260 :invoke :write [26 31] INFO [2022-03-13 17:07:31,132] jepsen worker 71 - jepsen.util 71 :invoke :write [23 16] INFO [2022-03-13 17:07:31,176] jepsen worker 26 - jepsen.util 116 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,177] jepsen worker 26 - jepsen.util 116 :invoke :write [8 5] INFO [2022-03-13 17:07:31,234] jepsen worker 77 - jepsen.util 167 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,234] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:31,235] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:31,236] jepsen worker 77 - jepsen.util 167 :invoke :write [25 2] INFO [2022-03-13 17:07:31,260] jepsen worker 52 - jepsen.util 142 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,261] jepsen worker 52 - jepsen.util 142 :invoke :write [17 22] INFO [2022-03-13 17:07:31,311] jepsen worker 34 - jepsen.util 124 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,312] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:31,312] jepsen worker 34 - jepsen.util 124 :invoke :write [11 14] INFO [2022-03-13 17:07:31,313] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:31,538] jepsen worker 88 - jepsen.util 268 :info :write [29 32] :RequestTimeoutException INFO [2022-03-13 17:07:31,539] jepsen worker 88 - jepsen.util 358 :invoke :read [29 nil] INFO [2022-03-13 17:07:31,548] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:31,549] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:31,587] jepsen worker 26 - jepsen.util 116 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,588] jepsen worker 26 - jepsen.util 116 :invoke :read [8 nil] INFO [2022-03-13 17:07:31,640] jepsen worker 77 - jepsen.util 167 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,641] jepsen worker 77 - jepsen.util 167 :invoke :write [25 9] INFO [2022-03-13 17:07:31,641] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:31,642] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:31,670] jepsen worker 25 - jepsen.util 205 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,671] jepsen worker 25 - jepsen.util 205 :invoke :write [8 42] INFO [2022-03-13 17:07:31,707] jepsen worker nemesis - couchbase.util Adding node 172.23.107.227 to cluster INFO [2022-03-13 17:07:31,715] jepsen worker 34 - jepsen.util 124 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,728] jepsen worker 34 - jepsen.util 124 :invoke :read [11 nil] INFO [2022-03-13 17:07:31,743] jepsen worker 52 - jepsen.util 142 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,743] jepsen worker 52 - jepsen.util 142 :invoke :write [17 26] INFO [2022-03-13 17:07:31,786] jepsen worker 69 - jepsen.util 159 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,786] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:31,786] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,787] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:31,788] jepsen worker 69 - jepsen.util 159 :invoke :write [23 3] INFO [2022-03-13 17:07:31,789] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:31,808] jepsen worker 6 - jepsen.util 186 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,809] jepsen worker 6 - jepsen.util 186 :invoke :write [2 36] INFO [2022-03-13 17:07:31,851] jepsen worker 75 - jepsen.util 75 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,852] jepsen worker 75 - jepsen.util 75 :invoke :write [25 13] INFO [2022-03-13 17:07:31,874] jepsen worker 8 - jepsen.util 278 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,874] jepsen worker 26 - jepsen.util 116 :ok :read [8 :nil] INFO [2022-03-13 17:07:31,879] jepsen worker 26 - jepsen.util 116 :invoke :write [8 29] INFO [2022-03-13 17:07:31,880] jepsen worker 8 - jepsen.util 278 :invoke :write [2 47] INFO [2022-03-13 17:07:31,919] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:31,919] jepsen worker 13 - jepsen.util 103 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:31,920] jepsen worker 13 - jepsen.util 103 :invoke :write [4 13] INFO [2022-03-13 17:07:31,920] jepsen worker 14 - jepsen.util 14 :invoke :write [4 12] INFO [2022-03-13 17:07:32,177] jepsen worker 35 - jepsen.util 215 :info :write [11 15] :RequestTimeoutException INFO [2022-03-13 17:07:32,177] jepsen worker 35 - jepsen.util 305 :invoke :write [11 39] INFO [2022-03-13 17:07:32,302] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:32,302] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:32,303] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:32,304] jepsen worker 51 - jepsen.util 51 :invoke :write [17 11] INFO [2022-03-13 17:07:32,306] jepsen worker 27 - jepsen.util 207 :ok :write [9 30] INFO [2022-03-13 17:07:32,307] jepsen worker 27 - jepsen.util 207 :invoke :read [9 nil] INFO [2022-03-13 17:07:32,400] jepsen worker 82 - jepsen.util 262 :info :write [27 0] :RequestTimeoutException INFO [2022-03-13 17:07:32,401] jepsen worker 82 - jepsen.util 352 :invoke :write [27 4] INFO [2022-03-13 17:07:32,453] jepsen worker 27 - jepsen.util 207 :ok :read [9 30] INFO [2022-03-13 17:07:32,453] jepsen worker 27 - jepsen.util 207 :invoke :read [9 nil] INFO [2022-03-13 17:07:32,493] jepsen worker 27 - jepsen.util 207 :ok :read [9 30] INFO [2022-03-13 17:07:32,494] jepsen worker 27 - jepsen.util 207 :invoke :read [9 nil] INFO [2022-03-13 17:07:32,541] jepsen worker 27 - jepsen.util 207 :ok :read [9 30] INFO [2022-03-13 17:07:32,542] jepsen worker 27 - jepsen.util 207 :invoke :read [9 nil] INFO [2022-03-13 17:07:32,587] jepsen worker 27 - jepsen.util 207 :ok :read [9 30] INFO [2022-03-13 17:07:32,588] jepsen worker 27 - jepsen.util 207 :invoke :write [9 17] INFO [2022-03-13 17:07:32,913] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:32,913] jepsen worker 53 - jepsen.util 143 :invoke :write [17 12] INFO [2022-03-13 17:07:32,948] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:32,949] jepsen worker 43 - jepsen.util 133 :invoke :write [14 4] INFO [2022-03-13 17:07:33,151] jepsen worker 79 - jepsen.util 259 :info :write [26 20] :RequestTimeoutException INFO [2022-03-13 17:07:33,151] jepsen worker 79 - jepsen.util 349 :invoke :write [26 19] INFO [2022-03-13 17:07:33,412] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:33,412] jepsen worker 7 - jepsen.util 97 :invoke :write [2 6] INFO [2022-03-13 17:07:33,802] jepsen worker 48 - jepsen.util 228 :ok :read [16 :nil] INFO [2022-03-13 17:07:33,802] jepsen worker 66 - jepsen.util 156 :info :write [22 3] :SyncWriteAmbiguous INFO [2022-03-13 17:07:33,802] jepsen worker 48 - jepsen.util 228 :invoke :read [16 nil] INFO [2022-03-13 17:07:33,803] jepsen worker 66 - jepsen.util 246 :invoke :read [22 nil] INFO [2022-03-13 17:07:33,973] jepsen worker 9 - jepsen.util 279 :info :write [3 14] :SyncWriteAmbiguous INFO [2022-03-13 17:07:33,975] jepsen worker 9 - jepsen.util 369 :invoke :read [3 nil] INFO [2022-03-13 17:07:34,073] jepsen worker 48 - jepsen.util 228 :ok :read [16 :nil] INFO [2022-03-13 17:07:34,074] jepsen worker 48 - jepsen.util 228 :invoke :write [16 31] INFO [2022-03-13 17:07:34,096] jepsen worker 5 - jepsen.util 185 :ok :read [1 28] INFO [2022-03-13 17:07:34,097] jepsen worker 5 - jepsen.util 185 :invoke :write [1 6] INFO [2022-03-13 17:07:34,118] jepsen worker 49 - jepsen.util 229 :info :write [16 48] :RequestTimeoutException INFO [2022-03-13 17:07:34,119] jepsen worker 49 - jepsen.util 319 :invoke :write [16 8] INFO [2022-03-13 17:07:34,281] jepsen worker 40 - jepsen.util 310 :ok :read [13 :nil] INFO [2022-03-13 17:07:34,282] jepsen worker 40 - jepsen.util 310 :invoke :write [13 2] INFO [2022-03-13 17:07:34,322] jepsen worker 34 - jepsen.util 124 :ok :read [11 :nil] INFO [2022-03-13 17:07:34,323] jepsen worker 34 - jepsen.util 124 :invoke :read [11 nil] INFO [2022-03-13 17:07:34,389] jepsen worker 16 - jepsen.util 196 :info :write [5 37] :RequestTimeoutException INFO [2022-03-13 17:07:34,389] jepsen worker 16 - jepsen.util 286 :invoke :read [5 nil] INFO [2022-03-13 17:07:34,426] jepsen worker 68 - jepsen.util 338 :ok :read [22 46] INFO [2022-03-13 17:07:34,427] jepsen worker 68 - jepsen.util 338 :invoke :read [22 nil] INFO [2022-03-13 17:07:34,494] jepsen worker 81 - jepsen.util 261 :info :write [27 38] :RequestTimeoutException INFO [2022-03-13 17:07:34,495] jepsen worker 81 - jepsen.util 351 :invoke :write [27 17] INFO [2022-03-13 17:07:34,617] jepsen worker 68 - jepsen.util 338 :ok :read [22 46] INFO [2022-03-13 17:07:34,618] jepsen worker 68 - jepsen.util 338 :invoke :read [22 nil] INFO [2022-03-13 17:07:34,621] jepsen worker 18 - jepsen.util 198 :ok :read [6 :nil] INFO [2022-03-13 17:07:34,622] jepsen worker 18 - jepsen.util 198 :invoke :write [6 15] INFO [2022-03-13 17:07:34,763] jepsen worker 68 - jepsen.util 338 :ok :read [22 46] INFO [2022-03-13 17:07:34,764] jepsen worker 68 - jepsen.util 338 :invoke :read [22 nil] INFO [2022-03-13 17:07:34,792] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:34,792] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:34,792] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:34,795] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:34,796] jepsen worker 12 - jepsen.util 192 :invoke :write [4 3] INFO [2022-03-13 17:07:34,797] jepsen worker 70 - jepsen.util 250 :invoke :write [23 11] INFO [2022-03-13 17:07:34,848] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:34,848] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:34,911] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:34,911] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:34,912] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:34,913] jepsen worker 78 - jepsen.util 258 :invoke :write [26 47] INFO [2022-03-13 17:07:34,917] jepsen worker 73 - jepsen.util 253 :info :write [24 4] :RequestTimeoutException INFO [2022-03-13 17:07:34,918] jepsen worker 73 - jepsen.util 343 :invoke :read [24 nil] INFO [2022-03-13 17:07:34,945] jepsen worker 73 - jepsen.util 343 :ok :read [24 24] INFO [2022-03-13 17:07:34,946] jepsen worker 73 - jepsen.util 343 :invoke :write [24 35] INFO [2022-03-13 17:07:34,951] jepsen worker 68 - jepsen.util 338 :ok :read [22 46] INFO [2022-03-13 17:07:34,951] jepsen worker 68 - jepsen.util 338 :invoke :read [22 nil] INFO [2022-03-13 17:07:34,960] jepsen worker 67 - jepsen.util 247 :info :write [22 12] :RequestTimeoutException INFO [2022-03-13 17:07:34,961] jepsen worker 67 - jepsen.util 337 :invoke :read [22 nil] INFO [2022-03-13 17:07:34,980] jepsen worker 34 - jepsen.util 124 :ok :read [11 :nil] INFO [2022-03-13 17:07:34,981] jepsen worker 34 - jepsen.util 124 :invoke :write [11 39] INFO [2022-03-13 17:07:35,079] jepsen worker 89 - jepsen.util 269 :ok :write [29 17] INFO [2022-03-13 17:07:35,079] jepsen worker 89 - jepsen.util 269 :invoke :read [29 nil] INFO [2022-03-13 17:07:35,098] jepsen worker 50 - jepsen.util 230 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:07:35,100] jepsen worker 50 - jepsen.util 320 :invoke :read [16 nil] INFO [2022-03-13 17:07:35,103] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,105] jepsen worker 62 - jepsen.util 242 :invoke :write [20 37] INFO [2022-03-13 17:07:35,126] jepsen worker 68 - jepsen.util 338 :ok :read [22 46] INFO [2022-03-13 17:07:35,127] jepsen worker 68 - jepsen.util 338 :invoke :write [22 11] INFO [2022-03-13 17:07:35,184] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:35,185] jepsen worker 33 - jepsen.util 213 :invoke :write [11 10] WARN [2022-03-13 17:07:35,186] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9062708,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9183427,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9050417,"encode_duration_us":18787,"operation_id":"0xbc4","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10003368},{"operation_name":"upsert","last_dispatch_duration_us":9283490,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10308514,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9691987,"encode_duration_us":22246,"operation_id":"0x9af","timeout_ms":10000,"last_server_duration_us":9270000,"total_duration_us":10009106},{"operation_name":"upsert","last_dispatch_duration_us":9233428,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10409244,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":9570412,"encode_duration_us":20936,"operation_id":"0xc42","timeout_ms":10000,"last_server_duration_us":9133103,"total_duration_us":10008069}],"total_count":3}} WARN [2022-03-13 17:07:35,186] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":16691,"operation_id":"0xbe8","timeout_ms":10000,"total_duration_us":10011132},{"operation_name":"upsert","last_dispatch_duration_us":173467,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":4413550,"total_server_duration_us":652625,"encode_duration_us":27202,"operation_id":"0xbe2","timeout_ms":10000,"last_server_duration_us":97638,"total_duration_us":10010654},{"operation_name":"upsert","last_dispatch_duration_us":58337,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":510991,"total_server_duration_us":314566,"encode_duration_us":19103,"operation_id":"0xc7a","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10008448},{"operation_name":"upsert","last_dispatch_duration_us":9233428,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10409244,"total_server_duration_us":9570412,"encode_duration_us":20936,"operation_id":"0xc42","timeout_ms":10000,"last_server_duration_us":9133103,"total_duration_us":10008069},{"operation_name":"upsert","last_dispatch_duration_us":11017278,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11017278,"total_server_duration_us":22223,"encode_duration_us":201081,"operation_id":"0xae6","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10007806},{"operation_name":"upsert","last_dispatch_duration_us":7673121,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8391777,"total_server_duration_us":329415,"encode_duration_us":20730,"operation_id":"0xa8b","timeout_ms":10000,"last_server_duration_us":34413,"total_duration_us":10007718},{"operation_name":"upsert","last_dispatch_duration_us":9062708,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9183427,"total_server_duration_us":9050417,"encode_duration_us":18787,"operation_id":"0xbc4","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10003368},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":20922,"operation_id":"0xb51","timeout_ms":10000,"total_duration_us":10001549}],"total_count":8}} INFO [2022-03-13 17:07:35,307] jepsen worker 51 - jepsen.util 51 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,307] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:35,335] jepsen worker 83 - jepsen.util 263 :ok :read [27 8] INFO [2022-03-13 17:07:35,335] jepsen worker 83 - jepsen.util 263 :invoke :write [27 0] INFO [2022-03-13 17:07:35,349] jepsen worker 7 - jepsen.util 97 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,350] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:35,350] jepsen worker 0 - jepsen.util 180 :info :write [0 38] :RequestTimeoutException INFO [2022-03-13 17:07:35,351] jepsen worker 0 - jepsen.util 270 :invoke :read [0 nil] INFO [2022-03-13 17:07:35,364] jepsen worker 78 - jepsen.util 258 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,364] jepsen worker 12 - jepsen.util 192 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,366] jepsen worker 12 - jepsen.util 192 :invoke :write [4 32] INFO [2022-03-13 17:07:35,367] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:35,424] jepsen worker 37 - jepsen.util 307 :ok :read [12 46] INFO [2022-03-13 17:07:35,425] jepsen worker 37 - jepsen.util 307 :invoke :read [12 nil] INFO [2022-03-13 17:07:35,430] jepsen worker 77 - jepsen.util 167 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,431] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:35,488] jepsen worker 60 - jepsen.util 240 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,488] jepsen worker 70 - jepsen.util 250 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,489] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:35,490] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:35,516] jepsen worker 6 - jepsen.util 186 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,517] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:35,518] jepsen worker 89 - jepsen.util 269 :ok :read [29 17] INFO [2022-03-13 17:07:35,519] jepsen worker 89 - jepsen.util 269 :invoke :read [29 nil] INFO [2022-03-13 17:07:35,529] jepsen worker 67 - jepsen.util 337 :ok :read [22 46] INFO [2022-03-13 17:07:35,531] jepsen worker 67 - jepsen.util 337 :invoke :read [22 nil] INFO [2022-03-13 17:07:35,582] jepsen worker 44 - jepsen.util 134 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,584] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:35,587] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:35,589] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:35,596] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:35,597] jepsen worker 0 - jepsen.util 270 :ok :read [0 41] INFO [2022-03-13 17:07:35,597] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:35,598] jepsen worker 0 - jepsen.util 270 :invoke :read [0 nil] INFO [2022-03-13 17:07:35,600] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:35,600] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:35,602] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:35,603] jepsen worker 44 - jepsen.util 134 :invoke :write [14 7] INFO [2022-03-13 17:07:35,607] jepsen worker 85 - jepsen.util 355 :ok :read [28 25] INFO [2022-03-13 17:07:35,608] jepsen worker 85 - jepsen.util 355 :invoke :read [28 nil] INFO [2022-03-13 17:07:35,620] jepsen worker 62 - jepsen.util 242 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,620] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,620] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,620] jepsen worker 62 - jepsen.util 242 :invoke :write [20 2] INFO [2022-03-13 17:07:35,621] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:35,621] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:35,670] jepsen worker 14 - jepsen.util 14 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,671] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:35,709] jepsen worker 66 - jepsen.util 246 :ok :read [22 46] INFO [2022-03-13 17:07:35,709] jepsen worker 66 - jepsen.util 246 :invoke :read [22 nil] INFO [2022-03-13 17:07:35,715] jepsen worker 37 - jepsen.util 307 :ok :read [12 46] INFO [2022-03-13 17:07:35,716] jepsen worker 37 - jepsen.util 307 :invoke :read [12 nil] INFO [2022-03-13 17:07:35,725] jepsen worker 89 - jepsen.util 269 :ok :read [29 17] INFO [2022-03-13 17:07:35,726] jepsen worker 89 - jepsen.util 269 :invoke :write [29 49] INFO [2022-03-13 17:07:35,729] jepsen worker 33 - jepsen.util 213 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,730] jepsen worker 33 - jepsen.util 213 :invoke :write [11 9] INFO [2022-03-13 17:07:35,747] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,747] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,749] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:35,749] jepsen worker 0 - jepsen.util 270 :ok :read [0 41] INFO [2022-03-13 17:07:35,750] jepsen worker 0 - jepsen.util 270 :invoke :read [0 nil] INFO [2022-03-13 17:07:35,751] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:35,757] jepsen worker 85 - jepsen.util 355 :ok :read [28 25] INFO [2022-03-13 17:07:35,757] jepsen worker 44 - jepsen.util 134 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,758] jepsen worker 85 - jepsen.util 355 :invoke :write [28 36] INFO [2022-03-13 17:07:35,759] jepsen worker 44 - jepsen.util 134 :invoke :write [14 10] INFO [2022-03-13 17:07:35,774] jepsen worker 86 - jepsen.util 266 :info :write [28 27] :RequestTimeoutException INFO [2022-03-13 17:07:35,775] jepsen worker 86 - jepsen.util 356 :invoke :read [28 nil] INFO [2022-03-13 17:07:35,787] jepsen worker 67 - jepsen.util 337 :ok :read [22 46] INFO [2022-03-13 17:07:35,788] jepsen worker 67 - jepsen.util 337 :invoke :read [22 nil] INFO [2022-03-13 17:07:35,793] jepsen worker 53 - jepsen.util 143 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,794] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:35,816] jepsen worker 79 - jepsen.util 349 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,817] jepsen worker 79 - jepsen.util 349 :invoke :read [26 nil] INFO [2022-03-13 17:07:35,831] jepsen worker 88 - jepsen.util 358 :ok :read [29 17] INFO [2022-03-13 17:07:35,832] jepsen worker 88 - jepsen.util 358 :invoke :write [29 15] INFO [2022-03-13 17:07:35,845] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,845] jepsen worker 44 - jepsen.util 134 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,847] jepsen worker 53 - jepsen.util 143 :invoke :write [17 37] INFO [2022-03-13 17:07:35,847] jepsen worker 44 - jepsen.util 134 :invoke :write [14 20] INFO [2022-03-13 17:07:35,867] jepsen worker 38 - jepsen.util 218 :info :write [12 44] :RequestTimeoutException INFO [2022-03-13 17:07:35,868] jepsen worker 38 - jepsen.util 308 :invoke :read [12 nil] INFO [2022-03-13 17:07:35,876] jepsen worker 0 - jepsen.util 270 :ok :read [0 41] INFO [2022-03-13 17:07:35,877] jepsen worker 0 - jepsen.util 270 :invoke :read [0 nil] INFO [2022-03-13 17:07:35,878] jepsen worker 58 - jepsen.util 238 :info :write [19 15] :RequestTimeoutException INFO [2022-03-13 17:07:35,879] jepsen worker 58 - jepsen.util 328 :invoke :write [19 20] INFO [2022-03-13 17:07:35,883] jepsen worker 66 - jepsen.util 246 :ok :read [22 46] INFO [2022-03-13 17:07:35,883] jepsen worker 66 - jepsen.util 246 :invoke :write [22 44] INFO [2022-03-13 17:07:35,901] jepsen worker 12 - jepsen.util 192 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,901] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:35,903] jepsen worker 12 - jepsen.util 192 :invoke :write [4 24] INFO [2022-03-13 17:07:35,903] jepsen worker 70 - jepsen.util 250 :invoke :write [23 12] INFO [2022-03-13 17:07:35,917] jepsen worker 43 - jepsen.util 133 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,922] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:35,989] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,989] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,989] jepsen worker 42 - jepsen.util 132 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,990] jepsen worker 42 - jepsen.util 132 :invoke :write [14 20] INFO [2022-03-13 17:07:35,992] jepsen worker 62 - jepsen.util 242 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,992] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,992] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:35,992] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:35,995] jepsen worker 7 - jepsen.util 97 :invoke :write [2 7] INFO [2022-03-13 17:07:35,996] jepsen worker 78 - jepsen.util 258 :invoke :write [26 23] INFO [2022-03-13 17:07:35,997] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:35,998] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:36,005] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,006] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,007] jepsen worker 62 - jepsen.util 242 :invoke :write [20 30] INFO [2022-03-13 17:07:36,007] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:36,009] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,011] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:36,012] jepsen worker 53 - jepsen.util 143 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,013] jepsen worker 53 - jepsen.util 143 :invoke :write [17 13] INFO [2022-03-13 17:07:36,052] jepsen worker 0 - jepsen.util 270 :ok :read [0 41] INFO [2022-03-13 17:07:36,053] jepsen worker 0 - jepsen.util 270 :invoke :write [0 42] INFO [2022-03-13 17:07:36,080] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,080] jepsen worker 33 - jepsen.util 213 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,081] jepsen worker 60 - jepsen.util 240 :invoke :write [20 45] INFO [2022-03-13 17:07:36,082] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:36,118] jepsen worker 44 - jepsen.util 134 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,118] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:36,145] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,145] jepsen worker 34 - jepsen.util 124 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,146] jepsen worker 34 - jepsen.util 124 :invoke :write [11 26] INFO [2022-03-13 17:07:36,147] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,187] jepsen worker 12 - jepsen.util 192 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,188] jepsen worker 12 - jepsen.util 192 :invoke :write [4 16] INFO [2022-03-13 17:07:36,190] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:36,190] jepsen worker 53 - jepsen.util 143 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,191] jepsen worker 44 - jepsen.util 134 :invoke :write [14 9] INFO [2022-03-13 17:07:36,192] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,193] jepsen worker 52 - jepsen.util 142 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,195] jepsen worker 52 - jepsen.util 142 :invoke :write [17 27] INFO [2022-03-13 17:07:36,196] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,197] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,199] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,200] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,203] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,204] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,206] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,207] jepsen worker 53 - jepsen.util 143 :invoke :write [17 28] INFO [2022-03-13 17:07:36,211] jepsen worker 32 - jepsen.util 212 :ok :read [10 :nil] INFO [2022-03-13 17:07:36,215] jepsen worker 32 - jepsen.util 212 :invoke :read [10 nil] INFO [2022-03-13 17:07:36,225] jepsen worker 78 - jepsen.util 258 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,226] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:36,237] jepsen worker 32 - jepsen.util 212 :ok :read [10 :nil] INFO [2022-03-13 17:07:36,237] jepsen worker 32 - jepsen.util 212 :invoke :write [10 6] INFO [2022-03-13 17:07:36,253] jepsen worker 56 - jepsen.util 236 :ok :read [18 12] INFO [2022-03-13 17:07:36,254] jepsen worker 56 - jepsen.util 236 :invoke :read [18 nil] INFO [2022-03-13 17:07:36,311] jepsen worker 56 - jepsen.util 236 :ok :read [18 12] INFO [2022-03-13 17:07:36,312] jepsen worker 56 - jepsen.util 236 :invoke :read [18 nil] INFO [2022-03-13 17:07:36,349] jepsen worker 44 - jepsen.util 134 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,350] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:36,380] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:36,380] jepsen worker 70 - jepsen.util 250 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,380] jepsen worker 62 - jepsen.util 242 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,381] jepsen worker 70 - jepsen.util 250 :invoke :write [23 28] INFO [2022-03-13 17:07:36,382] jepsen worker 62 - jepsen.util 242 :invoke :write [20 43] INFO [2022-03-13 17:07:36,382] jepsen worker 33 - jepsen.util 213 :invoke :write [11 27] INFO [2022-03-13 17:07:36,396] jepsen worker 76 - jepsen.util 166 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,398] jepsen worker 76 - jepsen.util 166 :invoke :read [25 nil] INFO [2022-03-13 17:07:36,412] jepsen worker 56 - jepsen.util 236 :ok :read [18 12] INFO [2022-03-13 17:07:36,412] jepsen worker 56 - jepsen.util 236 :invoke :write [18 23] INFO [2022-03-13 17:07:36,485] jepsen worker 53 - jepsen.util 143 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,485] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:36,486] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:36,487] jepsen worker 12 - jepsen.util 192 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,488] jepsen worker 60 - jepsen.util 240 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,488] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:36,488] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,489] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,489] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:36,491] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:36,491] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:36,492] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:36,492] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,493] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,493] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:36,494] jepsen worker 44 - jepsen.util 134 :invoke :write [14 23] INFO [2022-03-13 17:07:36,495] jepsen worker 23 - jepsen.util 113 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,496] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,497] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:36,498] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,500] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,501] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:36,502] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,504] jepsen worker 53 - jepsen.util 143 :invoke :write [17 44] INFO [2022-03-13 17:07:36,509] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:36,515] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:36,527] jepsen worker 77 - jepsen.util 167 :ok :read [25 6] INFO [2022-03-13 17:07:36,530] jepsen worker 77 - jepsen.util 167 :invoke :read [25 nil] INFO [2022-03-13 17:07:36,536] jepsen worker 57 - jepsen.util 237 :info :write [19 26] :RequestTimeoutException INFO [2022-03-13 17:07:36,537] jepsen worker 57 - jepsen.util 327 :invoke :write [19 37] INFO [2022-03-13 17:07:36,553] jepsen worker 37 - jepsen.util 307 :ok :read [12 46] INFO [2022-03-13 17:07:36,554] jepsen worker 37 - jepsen.util 307 :invoke :read [12 nil] INFO [2022-03-13 17:07:36,592] jepsen worker 33 - jepsen.util 213 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,592] jepsen worker 62 - jepsen.util 242 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,593] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,593] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,593] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:36,594] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:36,595] jepsen worker 62 - jepsen.util 242 :invoke :write [20 32] INFO [2022-03-13 17:07:36,595] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,596] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,596] jepsen worker 60 - jepsen.util 240 :invoke :write [20 38] INFO [2022-03-13 17:07:36,597] jepsen worker 78 - jepsen.util 258 :invoke :write [26 39] INFO [2022-03-13 17:07:36,597] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:36,627] jepsen worker 70 - jepsen.util 250 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,627] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:36,628] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:36,629] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:36,629] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:36,630] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:36,631] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:36,631] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:36,631] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:36,632] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:36,632] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:36,633] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:36,633] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:36,633] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:36,634] jepsen worker 33 - jepsen.util 213 :invoke :write [11 21] INFO [2022-03-13 17:07:36,634] jepsen worker 70 - jepsen.util 250 :invoke :write [23 49] INFO [2022-03-13 17:07:36,634] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:36,636] jepsen worker 12 - jepsen.util 192 :invoke :write [4 41] INFO [2022-03-13 17:07:36,658] jepsen worker 62 - jepsen.util 242 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,659] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:36,686] jepsen worker 78 - jepsen.util 258 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,687] jepsen worker 78 - jepsen.util 258 :invoke :write [26 21] INFO [2022-03-13 17:07:36,726] jepsen worker 3 - jepsen.util 183 :info :write [1 6] :RequestTimeoutException INFO [2022-03-13 17:07:36,727] jepsen worker 3 - jepsen.util 273 :invoke :write [1 0] INFO [2022-03-13 17:07:36,734] jepsen worker 69 - jepsen.util 159 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,735] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:36,735] jepsen worker 44 - jepsen.util 134 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,736] jepsen worker 44 - jepsen.util 134 :invoke :write [14 32] INFO [2022-03-13 17:07:36,737] jepsen worker 60 - jepsen.util 240 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,737] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,738] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:36,738] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] WARN [2022-03-13 17:07:36,787] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8857237,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11850214,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":8312674,"encode_duration_us":18705,"operation_id":"0xb6a","timeout_ms":10000,"last_server_duration_us":7993608,"total_duration_us":10003752},{"operation_name":"upsert","last_dispatch_duration_us":10189277,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":13897599,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":9316473,"encode_duration_us":19127,"operation_id":"0xbdf","timeout_ms":10000,"last_server_duration_us":9104337,"total_duration_us":10005039}],"total_count":2}} WARN [2022-03-13 17:07:36,787] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":52763,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9817096,"total_server_duration_us":304905,"encode_duration_us":20486,"operation_id":"0xb5f","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10010443},{"operation_name":"upsert","last_dispatch_duration_us":9310671,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9378726,"total_server_duration_us":47653,"encode_duration_us":39022,"operation_id":"0xb2e","timeout_ms":10000,"last_server_duration_us":27132,"total_duration_us":10009606},{"operation_name":"upsert","last_dispatch_duration_us":129640,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":129640,"total_server_duration_us":24239,"encode_duration_us":17375,"operation_id":"0xbe9","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10009310},{"operation_name":"upsert","last_dispatch_duration_us":10194452,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":15701406,"total_server_duration_us":396893,"encode_duration_us":20881,"operation_id":"0xbb2","timeout_ms":10000,"last_server_duration_us":33721,"total_duration_us":10008656},{"operation_name":"upsert","last_dispatch_duration_us":9331051,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9367994,"total_server_duration_us":45091,"encode_duration_us":17970,"operation_id":"0xc19","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10007448},{"operation_name":"upsert","last_dispatch_duration_us":43458,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":43458,"total_server_duration_us":19418,"encode_duration_us":21662,"operation_id":"0xbe3","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10006902},{"operation_name":"upsert","last_dispatch_duration_us":120428,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":365024,"total_server_duration_us":45858,"encode_duration_us":18575,"operation_id":"0xc7b","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10006458},{"operation_name":"upsert","last_dispatch_duration_us":9728363,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":15981233,"total_server_duration_us":531044,"encode_duration_us":17892,"operation_id":"0xba7","timeout_ms":10000,"last_server_duration_us":26953,"total_duration_us":10006427},{"operation_name":"upsert","last_dispatch_duration_us":10054567,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":15622637,"total_server_duration_us":529323,"encode_duration_us":18875,"operation_id":"0xbb9","timeout_ms":10000,"last_server_duration_us":54215,"total_duration_us":10005720},{"operation_name":"upsert","last_dispatch_duration_us":10189277,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":13897599,"total_server_duration_us":9316473,"encode_duration_us":19127,"operation_id":"0xbdf","timeout_ms":10000,"last_server_duration_us":9104337,"total_duration_us":10005039}],"total_count":25}} WARN [2022-03-13 17:07:36,793] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":5652883,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9405087,"total_server_duration_us":773248,"encode_duration_us":30855,"operation_id":"0xb70","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10010792},{"operation_name":"upsert","last_dispatch_duration_us":9290217,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10184660,"total_server_duration_us":9331538,"encode_duration_us":21193,"operation_id":"0xc2e","timeout_ms":10000,"last_server_duration_us":9101143,"total_duration_us":10010355},{"operation_name":"upsert","last_dispatch_duration_us":10552385,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11437272,"total_server_duration_us":5599600,"encode_duration_us":37197,"operation_id":"0xba0","timeout_ms":10000,"last_server_duration_us":5313837,"total_duration_us":10010321},{"operation_name":"upsert","last_dispatch_duration_us":4816730,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10915332,"total_server_duration_us":349199,"encode_duration_us":30289,"operation_id":"0xb5d","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10010274},{"operation_name":"upsert","last_dispatch_duration_us":9064450,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10564379,"total_server_duration_us":9633955,"encode_duration_us":24909,"operation_id":"0xaed","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008751},{"operation_name":"upsert","last_dispatch_duration_us":7830587,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8646218,"total_server_duration_us":525589,"encode_duration_us":20221,"operation_id":"0xbe7","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10008693},{"operation_name":"upsert","last_dispatch_duration_us":55057,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1542089,"total_server_duration_us":604738,"encode_duration_us":18034,"operation_id":"0xaf4","timeout_ms":10000,"last_server_duration_us":39529,"total_duration_us":10007737},{"operation_name":"upsert","last_dispatch_duration_us":5848271,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9629855,"total_server_duration_us":748329,"encode_duration_us":23324,"operation_id":"0xb84","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10005496},{"operation_name":"upsert","last_dispatch_duration_us":5901899,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":7468262,"total_server_duration_us":649499,"encode_duration_us":17182,"operation_id":"0xbaf","timeout_ms":10000,"last_server_duration_us":38482,"total_duration_us":10003245},{"operation_name":"upsert","last_dispatch_duration_us":36444,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":6057647,"total_server_duration_us":326577,"encode_duration_us":17506,"operation_id":"0xc96","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10002974}],"total_count":18}} WARN [2022-03-13 17:07:36,793] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9064450,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10564379,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9633955,"encode_duration_us":24909,"operation_id":"0xaed","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008751},{"operation_name":"upsert","last_dispatch_duration_us":9290217,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10184660,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9331538,"encode_duration_us":21193,"operation_id":"0xc2e","timeout_ms":10000,"last_server_duration_us":9101143,"total_duration_us":10010355},{"operation_name":"upsert","last_dispatch_duration_us":10552385,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11437272,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":5599600,"encode_duration_us":37197,"operation_id":"0xba0","timeout_ms":10000,"last_server_duration_us":5313837,"total_duration_us":10010321}],"total_count":3}} INFO [2022-03-13 17:07:36,794] jepsen worker 22 - jepsen.util 202 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,795] jepsen worker 79 - jepsen.util 349 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,795] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:07:36,795] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16383269,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16623216,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9168832,"encode_duration_us":39344,"operation_id":"0xb59","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002625},{"operation_name":"upsert","last_dispatch_duration_us":17401547,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17401547,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9068168,"encode_duration_us":19518,"operation_id":"0x9d4","timeout_ms":10000,"last_server_duration_us":9068168,"total_duration_us":10006087},{"operation_name":"upsert","last_dispatch_duration_us":16419484,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17146670,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9259922,"encode_duration_us":25007,"operation_id":"0xb03","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10005174},{"operation_name":"upsert","last_dispatch_duration_us":9760209,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12149380,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":9344019,"encode_duration_us":25882,"operation_id":"0xbea","timeout_ms":10000,"last_server_duration_us":9092628,"total_duration_us":10010556},{"operation_name":"upsert","last_dispatch_duration_us":16423754,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16771655,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9124342,"encode_duration_us":17409,"operation_id":"0x9a8","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":10008700},{"operation_name":"upsert","last_dispatch_duration_us":16367827,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16645018,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9109485,"encode_duration_us":17197,"operation_id":"0xa2a","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10006743}],"total_count":6}} INFO [2022-03-13 17:07:36,795] jepsen worker 79 - jepsen.util 349 :invoke :write [26 8] WARN [2022-03-13 17:07:36,796] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":54406,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":54406,"total_server_duration_us":38,"operation_id":"0xb58","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":10835058},{"operation_name":"upsert","last_dispatch_duration_us":36252,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":366753,"total_server_duration_us":147137,"encode_duration_us":17438,"operation_id":"0xc7c","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10009716},{"operation_name":"upsert","encode_duration_us":52416,"operation_id":"0xb33","timeout_ms":10000,"total_duration_us":10007086},{"operation_name":"upsert","last_dispatch_duration_us":7652850,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10336520,"total_server_duration_us":391004,"encode_duration_us":18015,"operation_id":"0xbec","timeout_ms":10000,"last_server_duration_us":25275,"total_duration_us":10006869},{"operation_name":"upsert","last_dispatch_duration_us":86042,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":132459,"total_server_duration_us":48231,"encode_duration_us":17886,"operation_id":"0xc2f","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10005949},{"operation_name":"upsert","last_dispatch_duration_us":121415,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":197457,"total_server_duration_us":45146,"encode_duration_us":20708,"operation_id":"0xbdc","timeout_ms":10000,"last_server_duration_us":23983,"total_duration_us":10003817},{"operation_name":"upsert","last_dispatch_duration_us":7656067,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10011638,"total_server_duration_us":332793,"encode_duration_us":18606,"operation_id":"0xbe5","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10001693},{"operation_name":"upsert","last_dispatch_duration_us":183625,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3243210,"total_server_duration_us":308640,"encode_duration_us":53679,"operation_id":"0xb4a","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":8928633},{"operation_name":"upsert","last_dispatch_duration_us":8690464,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8690464,"total_server_duration_us":8630601,"encode_duration_us":17676,"operation_id":"0xc4f","timeout_ms":10000,"last_server_duration_us":8630601,"total_duration_us":8696331},{"operation_name":"upsert","last_dispatch_duration_us":336411,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":6549851,"total_server_duration_us":478570,"encode_duration_us":76254,"operation_id":"0xb46","timeout_ms":10000,"last_server_duration_us":38482,"total_duration_us":8683225}],"total_count":34}} INFO [2022-03-13 17:07:36,796] jepsen worker 22 - jepsen.util 202 :invoke :write [7 24] INFO [2022-03-13 17:07:36,797] jepsen worker 6 - jepsen.util 186 :invoke :read [2 nil] INFO [2022-03-13 17:07:36,833] jepsen worker 53 - jepsen.util 143 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,835] jepsen worker 53 - jepsen.util 143 :invoke :write [17 36] INFO [2022-03-13 17:07:36,844] jepsen worker 33 - jepsen.util 213 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,845] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:36,883] jepsen worker 75 - jepsen.util 75 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,884] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:36,884] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,884] jepsen worker 75 - jepsen.util 75 :invoke :write [25 38] INFO [2022-03-13 17:07:36,885] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:36,886] jepsen worker 51 - jepsen.util 51 :invoke :write [17 10] WARN [2022-03-13 17:07:36,894] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":153550,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9732662,"total_server_duration_us":217289,"encode_duration_us":17404,"operation_id":"0xca5","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10009942},{"operation_name":"upsert","last_dispatch_duration_us":16800343,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":17253515,"total_server_duration_us":8618654,"encode_duration_us":53585,"operation_id":"0xb50","timeout_ms":10000,"last_server_duration_us":8430728,"total_duration_us":10008783},{"operation_name":"upsert","last_dispatch_duration_us":17044726,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":17044726,"total_server_duration_us":9055417,"encode_duration_us":22343,"operation_id":"0xbb7","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10008058},{"operation_name":"upsert","last_dispatch_duration_us":52723,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8201224,"total_server_duration_us":173214,"encode_duration_us":23294,"operation_id":"0xb16","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10007037},{"operation_name":"upsert","last_dispatch_duration_us":11263532,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11263532,"total_server_duration_us":22471,"encode_duration_us":19794,"operation_id":"0xc79","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10005062},{"operation_name":"upsert","last_dispatch_duration_us":9134968,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9134968,"total_server_duration_us":9032060,"encode_duration_us":20522,"operation_id":"0xbe1","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10003635},{"operation_name":"upsert","last_dispatch_duration_us":8652698,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17645455,"total_server_duration_us":311772,"encode_duration_us":18104,"operation_id":"0xaee","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10003029},{"operation_name":"upsert","last_dispatch_duration_us":135249,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9894406,"total_server_duration_us":202870,"encode_duration_us":20742,"operation_id":"0xcbc","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10002937},{"operation_name":"upsert","last_dispatch_duration_us":171943,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":171943,"total_server_duration_us":20362,"encode_duration_us":20946,"operation_id":"0xc97","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10002885},{"operation_name":"upsert","last_dispatch_duration_us":16539843,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":18057525,"total_server_duration_us":9276867,"encode_duration_us":23740,"operation_id":"0xadb","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10001923}],"total_count":59}} WARN [2022-03-13 17:07:36,895] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9760209,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":12149380,"total_server_duration_us":9344019,"encode_duration_us":25882,"operation_id":"0xbea","timeout_ms":10000,"last_server_duration_us":9092628,"total_duration_us":10010556},{"operation_name":"upsert","last_dispatch_duration_us":8226887,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":14523145,"total_server_duration_us":767641,"encode_duration_us":24706,"operation_id":"0xb9d","timeout_ms":10000,"last_server_duration_us":45055,"total_duration_us":10010155},{"operation_name":"upsert","last_dispatch_duration_us":9202299,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9539040,"total_server_duration_us":68631,"encode_duration_us":27253,"operation_id":"0xb64","timeout_ms":10000,"last_server_duration_us":25888,"total_duration_us":10008800},{"operation_name":"upsert","last_dispatch_duration_us":9012324,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":15343351,"total_server_duration_us":639061,"encode_duration_us":27188,"operation_id":"0xb9e","timeout_ms":10000,"last_server_duration_us":53854,"total_duration_us":10008632},{"operation_name":"upsert","last_dispatch_duration_us":9053700,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":15406399,"total_server_duration_us":579374,"encode_duration_us":28186,"operation_id":"0xb9f","timeout_ms":10000,"last_server_duration_us":25712,"total_duration_us":10008458},{"operation_name":"upsert","last_dispatch_duration_us":8568248,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8639227,"total_server_duration_us":52162,"encode_duration_us":18566,"operation_id":"0xc61","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10007091},{"operation_name":"upsert","last_dispatch_duration_us":8370303,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8403395,"total_server_duration_us":41050,"encode_duration_us":17451,"operation_id":"0xbe4","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10006370},{"operation_name":"upsert","last_dispatch_duration_us":16419484,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17146670,"total_server_duration_us":9259922,"encode_duration_us":25007,"operation_id":"0xb03","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10005174},{"operation_name":"upsert","last_dispatch_duration_us":147878,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8960871,"total_server_duration_us":103532,"encode_duration_us":17266,"operation_id":"0xc92","timeout_ms":10000,"last_server_duration_us":25976,"total_duration_us":10004722},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":35870,"operation_id":"0xba3","timeout_ms":10000,"total_duration_us":10004359}],"total_count":27}} INFO [2022-03-13 17:07:36,911] jepsen worker 59 - jepsen.util 329 :info :write [19 37] :RequestTimeoutException INFO [2022-03-13 17:07:36,911] jepsen worker 59 - jepsen.util 419 :invoke :read [19 nil] INFO [2022-03-13 17:07:36,953] jepsen worker 44 - jepsen.util 134 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,953] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:36,970] jepsen worker 64 - jepsen.util 154 :ok :read [21 :nil] INFO [2022-03-13 17:07:36,972] jepsen worker 42 - jepsen.util 132 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:07:36,974] jepsen worker 64 - jepsen.util 154 :invoke :read [21 nil] INFO [2022-03-13 17:07:36,979] jepsen worker 42 - jepsen.util 132 :invoke :read [14 nil] WARN [2022-03-13 17:07:36,995] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16539843,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18057525,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9276867,"encode_duration_us":23740,"operation_id":"0xadb","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10001923},{"operation_name":"upsert","last_dispatch_duration_us":9069133,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15182951,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9368452,"encode_duration_us":23861,"operation_id":"0x9c8","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10007805},{"operation_name":"upsert","last_dispatch_duration_us":9134968,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9134968,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9032060,"encode_duration_us":20522,"operation_id":"0xbe1","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10003635},{"operation_name":"upsert","last_dispatch_duration_us":16800343,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17253515,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":8618654,"encode_duration_us":53585,"operation_id":"0xb50","timeout_ms":10000,"last_server_duration_us":8430728,"total_duration_us":10008783},{"operation_name":"upsert","last_dispatch_duration_us":17044726,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17044726,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9055417,"encode_duration_us":22343,"operation_id":"0xbb7","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10008058}],"total_count":5}} INFO [2022-03-13 17:07:37,006] jepsen worker 59 - jepsen.util 419 :ok :read [19 1] INFO [2022-03-13 17:07:37,007] jepsen worker 59 - jepsen.util 419 :invoke :write [19 35] INFO [2022-03-13 17:07:37,019] jepsen worker 34 - jepsen.util 124 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,020] jepsen worker 34 - jepsen.util 124 :invoke :read [11 nil] INFO [2022-03-13 17:07:37,043] jepsen worker 7 - jepsen.util 97 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,044] jepsen worker 7 - jepsen.util 97 :invoke :read [2 nil] INFO [2022-03-13 17:07:37,045] jepsen worker 19 - jepsen.util 199 :ok :read [6 :nil] INFO [2022-03-13 17:07:37,045] jepsen worker 19 - jepsen.util 199 :invoke :read [6 nil] INFO [2022-03-13 17:07:37,073] jepsen worker 70 - jepsen.util 250 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,073] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,073] jepsen worker 78 - jepsen.util 258 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,073] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,073] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:37,074] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:37,075] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:37,075] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,076] jepsen worker 62 - jepsen.util 242 :invoke :write [20 18] INFO [2022-03-13 17:07:37,076] jepsen worker 60 - jepsen.util 240 :invoke :write [20 49] INFO [2022-03-13 17:07:37,077] jepsen worker 78 - jepsen.util 258 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,078] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:37,078] jepsen worker 53 - jepsen.util 143 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,078] jepsen worker 78 - jepsen.util 258 :invoke :write [26 44] INFO [2022-03-13 17:07:37,079] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,079] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:37,080] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,080] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:37,081] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,081] jepsen worker 44 - jepsen.util 134 :invoke :write [14 36] INFO [2022-03-13 17:07:37,084] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,084] jepsen worker 53 - jepsen.util 143 :invoke :write [17 0] INFO [2022-03-13 17:07:37,142] jepsen worker 71 - jepsen.util 71 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,143] jepsen worker 71 - jepsen.util 71 :invoke :write [23 13] INFO [2022-03-13 17:07:37,168] jepsen worker 11 - jepsen.util 191 :ok :read [3 :nil] INFO [2022-03-13 17:07:37,169] jepsen worker 11 - jepsen.util 191 :invoke :read [3 nil] INFO [2022-03-13 17:07:37,171] jepsen worker 12 - jepsen.util 192 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,171] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:37,171] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:37,172] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:37,173] jepsen worker 33 - jepsen.util 213 :invoke :write [11 21] INFO [2022-03-13 17:07:37,174] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:37,177] jepsen worker 74 - jepsen.util 254 :info :write [24 23] :RequestTimeoutException INFO [2022-03-13 17:07:37,178] jepsen worker 74 - jepsen.util 344 :invoke :write [24 43] INFO [2022-03-13 17:07:37,182] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:37,183] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:37,183] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:37,184] jepsen worker 70 - jepsen.util 250 :invoke :write [23 32] INFO [2022-03-13 17:07:37,185] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:37,186] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:37,193] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:37,193] jepsen worker 52 - jepsen.util 142 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,195] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,195] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:37,212] jepsen worker 20 - jepsen.util 290 :info :write [6 47] :RequestTimeoutException INFO [2022-03-13 17:07:37,213] jepsen worker 20 - jepsen.util 380 :invoke :read [6 nil] INFO [2022-03-13 17:07:37,268] jepsen worker 55 - jepsen.util 325 :ok :read [18 12] INFO [2022-03-13 17:07:37,269] jepsen worker 55 - jepsen.util 325 :invoke :read [18 nil] INFO [2022-03-13 17:07:37,283] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:37,283] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:37,304] jepsen worker 44 - jepsen.util 134 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,305] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:37,309] jepsen worker 80 - jepsen.util 260 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,311] jepsen worker 80 - jepsen.util 260 :invoke :read [26 nil] INFO [2022-03-13 17:07:37,326] jepsen worker 11 - jepsen.util 191 :ok :read [3 :nil] INFO [2022-03-13 17:07:37,327] jepsen worker 11 - jepsen.util 191 :invoke :write [3 36] INFO [2022-03-13 17:07:37,380] jepsen worker 62 - jepsen.util 242 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,381] jepsen worker 53 - jepsen.util 143 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,381] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:37,381] jepsen worker 26 - jepsen.util 116 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,382] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:37,382] jepsen worker 6 - jepsen.util 186 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,384] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,385] jepsen worker 26 - jepsen.util 116 :invoke :read [8 nil] INFO [2022-03-13 17:07:37,386] jepsen worker 6 - jepsen.util 186 :invoke :write [2 24] INFO [2022-03-13 17:07:37,387] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,388] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,388] jepsen worker 55 - jepsen.util 325 :ok :read [18 12] INFO [2022-03-13 17:07:37,389] jepsen worker 55 - jepsen.util 325 :invoke :read [18 nil] INFO [2022-03-13 17:07:37,390] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,391] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,391] jepsen worker 44 - jepsen.util 134 :invoke :write [14 11] INFO [2022-03-13 17:07:37,393] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,393] jepsen worker 64 - jepsen.util 154 :ok :read [21 :nil] INFO [2022-03-13 17:07:37,394] jepsen worker 19 - jepsen.util 199 :ok :read [6 :nil] INFO [2022-03-13 17:07:37,394] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,395] jepsen worker 19 - jepsen.util 199 :invoke :read [6 nil] INFO [2022-03-13 17:07:37,396] jepsen worker 64 - jepsen.util 154 :invoke :write [21 10] INFO [2022-03-13 17:07:37,396] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,397] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,410] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,411] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,412] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,413] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,415] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,416] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,503] jepsen worker 44 - jepsen.util 134 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,503] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,504] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:37,505] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,506] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:37,507] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,507] jepsen worker 44 - jepsen.util 134 :invoke :write [14 1] INFO [2022-03-13 17:07:37,508] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,509] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,510] jepsen worker 33 - jepsen.util 213 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,511] jepsen worker 33 - jepsen.util 213 :invoke :write [11 3] INFO [2022-03-13 17:07:37,513] jepsen worker 13 - jepsen.util 103 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,513] jepsen worker 13 - jepsen.util 103 :invoke :write [4 45] INFO [2022-03-13 17:07:37,518] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,520] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,521] jepsen worker 53 - jepsen.util 143 :invoke :write [17 19] INFO [2022-03-13 17:07:37,521] jepsen worker 1 - jepsen.util 181 :info :write [0 32] :RequestTimeoutException INFO [2022-03-13 17:07:37,521] jepsen worker 1 - jepsen.util 271 :invoke :write [0 41] INFO [2022-03-13 17:07:37,568] jepsen worker 60 - jepsen.util 240 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,568] jepsen worker 22 - jepsen.util 202 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,568] jepsen worker 42 - jepsen.util 132 :ok :read [14 :nil] INFO [2022-03-13 17:07:37,569] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:37,569] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:37,570] jepsen worker 42 - jepsen.util 132 :invoke :write [14 5] INFO [2022-03-13 17:07:37,637] jepsen worker 44 - jepsen.util 134 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,638] jepsen worker 44 - jepsen.util 134 :invoke :write [14 42] INFO [2022-03-13 17:07:37,669] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:37,669] jepsen worker 51 - jepsen.util 51 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,670] jepsen worker 43 - jepsen.util 133 :invoke :read [14 nil] INFO [2022-03-13 17:07:37,670] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,689] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:37,689] jepsen worker 70 - jepsen.util 250 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,690] jepsen worker 12 - jepsen.util 192 :invoke :write [4 13] INFO [2022-03-13 17:07:37,690] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:37,698] jepsen worker 79 - jepsen.util 349 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,700] jepsen worker 7 - jepsen.util 97 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,700] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:37,700] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,700] jepsen worker 79 - jepsen.util 349 :invoke :write [26 1] INFO [2022-03-13 17:07:37,702] jepsen worker 7 - jepsen.util 97 :invoke :write [2 42] INFO [2022-03-13 17:07:37,704] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:37,707] jepsen worker 52 - jepsen.util 142 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,714] jepsen worker 53 - jepsen.util 143 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,715] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:37,739] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:37,739] jepsen worker 34 - jepsen.util 124 :ok :read [11 :nil] INFO [2022-03-13 17:07:37,739] jepsen worker 75 - jepsen.util 75 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,740] jepsen worker 75 - jepsen.util 75 :invoke :write [25 31] INFO [2022-03-13 17:07:37,742] jepsen worker 34 - jepsen.util 124 :invoke :read [11 nil] INFO [2022-03-13 17:07:37,744] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:37,746] jepsen worker 19 - jepsen.util 199 :ok :read [6 :nil] INFO [2022-03-13 17:07:37,747] jepsen worker 19 - jepsen.util 199 :invoke :write [6 23] INFO [2022-03-13 17:07:37,769] jepsen worker 25 - jepsen.util 205 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,770] jepsen worker 25 - jepsen.util 205 :invoke :read [8 nil] INFO [2022-03-13 17:07:37,797] jepsen worker 55 - jepsen.util 325 :ok :read [18 12] INFO [2022-03-13 17:07:37,798] jepsen worker 55 - jepsen.util 325 :invoke :read [18 nil] INFO [2022-03-13 17:07:37,803] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,803] jepsen worker 78 - jepsen.util 258 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,803] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,804] jepsen worker 78 - jepsen.util 258 :invoke :write [26 29] INFO [2022-03-13 17:07:37,805] jepsen worker 62 - jepsen.util 242 :invoke :write [20 11] INFO [2022-03-13 17:07:37,805] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:37,807] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,808] jepsen worker 60 - jepsen.util 240 :invoke :read [20 nil] INFO [2022-03-13 17:07:37,811] jepsen worker 60 - jepsen.util 240 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,812] jepsen worker 60 - jepsen.util 240 :invoke :write [20 9] INFO [2022-03-13 17:07:37,817] jepsen worker 77 - jepsen.util 167 :ok :read [25 6] INFO [2022-03-13 17:07:37,818] jepsen worker 77 - jepsen.util 167 :invoke :write [25 42] INFO [2022-03-13 17:07:37,832] jepsen worker 44 - jepsen.util 134 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,832] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,833] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:37,834] jepsen worker 53 - jepsen.util 143 :invoke :write [17 13] INFO [2022-03-13 17:07:37,835] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:37,835] jepsen worker 44 - jepsen.util 134 :invoke :write [14 26] INFO [2022-03-13 17:07:37,914] jepsen worker 33 - jepsen.util 213 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,915] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:37,915] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:37,916] jepsen worker 70 - jepsen.util 250 :invoke :write [23 25] INFO [2022-03-13 17:07:37,944] jepsen worker 44 - jepsen.util 134 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,945] jepsen worker 44 - jepsen.util 134 :invoke :write [14 4] INFO [2022-03-13 17:07:37,947] jepsen worker 71 - jepsen.util 71 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,948] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:37,948] jepsen worker 71 - jepsen.util 71 :invoke :write [23 20] INFO [2022-03-13 17:07:37,948] jepsen worker 80 - jepsen.util 260 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:37,949] jepsen worker 80 - jepsen.util 260 :invoke :write [26 18] INFO [2022-03-13 17:07:37,954] jepsen worker 69 - jepsen.util 159 :invoke :read [23 nil] INFO [2022-03-13 17:07:38,016] jepsen worker 12 - jepsen.util 192 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,017] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:38,017] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:38,018] jepsen worker 33 - jepsen.util 213 :invoke :read [11 nil] INFO [2022-03-13 17:07:38,033] jepsen worker 2 - jepsen.util 272 :info :write [0 4] :RequestTimeoutException INFO [2022-03-13 17:07:38,034] jepsen worker 2 - jepsen.util 362 :invoke :write [0 34] INFO [2022-03-13 17:07:38,056] jepsen worker 53 - jepsen.util 143 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,056] jepsen worker 8 - jepsen.util 278 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,057] jepsen worker 26 - jepsen.util 116 :ok :read [8 :nil] INFO [2022-03-13 17:07:38,058] jepsen worker 26 - jepsen.util 116 :invoke :write [8 3] INFO [2022-03-13 17:07:38,058] jepsen worker 53 - jepsen.util 143 :invoke :write [17 4] INFO [2022-03-13 17:07:38,059] jepsen worker 8 - jepsen.util 278 :invoke :read [2 nil] INFO [2022-03-13 17:07:38,064] jepsen worker 29 - jepsen.util 209 :info :write [9 18] :RequestTimeoutException INFO [2022-03-13 17:07:38,064] jepsen worker 29 - jepsen.util 299 :invoke :write [9 42] INFO [2022-03-13 17:07:38,068] jepsen worker 62 - jepsen.util 242 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,068] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:38,089] jepsen worker 76 - jepsen.util 166 :ok :read [25 6] INFO [2022-03-13 17:07:38,090] jepsen worker 76 - jepsen.util 166 :invoke :write [25 12] INFO [2022-03-13 17:07:38,106] jepsen worker 43 - jepsen.util 133 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,106] jepsen worker 13 - jepsen.util 103 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,106] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,107] jepsen worker 34 - jepsen.util 124 :ok :read [11 :nil] INFO [2022-03-13 17:07:38,107] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:38,107] jepsen worker 43 - jepsen.util 133 :invoke :write [14 1] INFO [2022-03-13 17:07:38,108] jepsen worker 14 - jepsen.util 14 :invoke :write [4 13] INFO [2022-03-13 17:07:38,109] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:38,110] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:38,111] jepsen worker 34 - jepsen.util 124 :invoke :write [11 3] INFO [2022-03-13 17:07:38,128] jepsen worker 6 - jepsen.util 186 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,129] jepsen worker 6 - jepsen.util 186 :invoke :write [2 1] INFO [2022-03-13 17:07:38,152] jepsen worker 44 - jepsen.util 134 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,153] jepsen worker 44 - jepsen.util 134 :invoke :write [14 44] INFO [2022-03-13 17:07:38,192] jepsen worker 75 - jepsen.util 75 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,192] jepsen worker 75 - jepsen.util 75 :invoke :read [25 nil] INFO [2022-03-13 17:07:38,198] jepsen worker 54 - jepsen.util 324 :info :write [18 8] :RequestTimeoutException INFO [2022-03-13 17:07:38,198] jepsen worker 54 - jepsen.util 414 :invoke :read [18 nil] INFO [2022-03-13 17:07:38,228] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:38,228] jepsen worker 42 - jepsen.util 132 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,228] jepsen worker 52 - jepsen.util 142 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,229] jepsen worker 52 - jepsen.util 142 :invoke :write [17 11] INFO [2022-03-13 17:07:38,230] jepsen worker 23 - jepsen.util 113 :invoke :write [7 27] INFO [2022-03-13 17:07:38,230] jepsen worker 42 - jepsen.util 132 :invoke :write [14 46] INFO [2022-03-13 17:07:38,270] jepsen worker 53 - jepsen.util 143 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,271] jepsen worker 53 - jepsen.util 143 :invoke :write [17 36] INFO [2022-03-13 17:07:38,287] jepsen worker 69 - jepsen.util 159 :ok :read [23 :nil] INFO [2022-03-13 17:07:38,287] jepsen worker 77 - jepsen.util 167 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,289] jepsen worker 69 - jepsen.util 159 :invoke :write [23 40] INFO [2022-03-13 17:07:38,296] jepsen worker 77 - jepsen.util 167 :invoke :write [25 18] INFO [2022-03-13 17:07:38,307] jepsen worker 7 - jepsen.util 97 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,309] jepsen worker 7 - jepsen.util 97 :invoke :write [2 29] INFO [2022-03-13 17:07:38,354] jepsen worker 71 - jepsen.util 71 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,355] jepsen worker 71 - jepsen.util 71 :invoke :write [23 19] INFO [2022-03-13 17:07:38,376] jepsen worker 79 - jepsen.util 349 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,376] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:38,376] jepsen worker 8 - jepsen.util 278 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,377] jepsen worker 79 - jepsen.util 349 :invoke :write [26 2] INFO [2022-03-13 17:07:38,379] jepsen worker 8 - jepsen.util 278 :invoke :write [2 47] INFO [2022-03-13 17:07:38,380] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:38,384] jepsen worker 31 - jepsen.util 211 :info :write [10 18] :RequestTimeoutException INFO [2022-03-13 17:07:38,386] jepsen worker 31 - jepsen.util 301 :invoke :write [10 3] INFO [2022-03-13 17:07:38,405] jepsen worker 44 - jepsen.util 134 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,405] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,424] jepsen worker 55 - jepsen.util 325 :ok :read [18 12] INFO [2022-03-13 17:07:38,424] jepsen worker 55 - jepsen.util 325 :invoke :read [18 nil] INFO [2022-03-13 17:07:38,452] jepsen worker 80 - jepsen.util 260 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,452] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:38,453] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,453] jepsen worker 80 - jepsen.util 260 :invoke :write [26 12] INFO [2022-03-13 17:07:38,454] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:38,454] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:38,519] jepsen worker 53 - jepsen.util 143 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,519] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,519] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:38,520] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,521] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,522] jepsen worker 53 - jepsen.util 143 :invoke :write [17 3] INFO [2022-03-13 17:07:38,523] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,527] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,528] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,529] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,530] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,530] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,532] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,533] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,534] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,534] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,536] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,536] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,537] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,538] jepsen worker 44 - jepsen.util 134 :invoke :write [14 35] INFO [2022-03-13 17:07:38,539] jepsen worker 26 - jepsen.util 116 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,540] jepsen worker 26 - jepsen.util 116 :invoke :write [8 42] INFO [2022-03-13 17:07:38,606] jepsen worker 53 - jepsen.util 143 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,607] jepsen worker 78 - jepsen.util 258 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,607] jepsen worker 70 - jepsen.util 250 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,608] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:38,608] jepsen worker 33 - jepsen.util 213 :ok :read [11 :nil] INFO [2022-03-13 17:07:38,608] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:38,611] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:38,611] jepsen worker 78 - jepsen.util 258 :invoke :read [26 nil] INFO [2022-03-13 17:07:38,612] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:38,613] jepsen worker 33 - jepsen.util 213 :invoke :write [11 19] INFO [2022-03-13 17:07:38,628] jepsen worker 76 - jepsen.util 166 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,629] jepsen worker 76 - jepsen.util 166 :invoke :write [25 8] INFO [2022-03-13 17:07:38,679] jepsen worker 23 - jepsen.util 113 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,680] jepsen worker 23 - jepsen.util 113 :invoke :read [7 nil] INFO [2022-03-13 17:07:38,684] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,684] jepsen worker 53 - jepsen.util 143 :invoke :read [17 nil] INFO [2022-03-13 17:07:38,686] jepsen worker 53 - jepsen.util 143 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,687] jepsen worker 53 - jepsen.util 143 :invoke :write [17 35] INFO [2022-03-13 17:07:38,693] jepsen worker 44 - jepsen.util 134 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,693] jepsen worker 44 - jepsen.util 134 :invoke :write [14 45] INFO [2022-03-13 17:07:38,730] jepsen worker 14 - jepsen.util 14 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,731] jepsen worker 14 - jepsen.util 14 :invoke :read [4 nil] INFO [2022-03-13 17:07:38,760] jepsen worker 6 - jepsen.util 186 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,764] jepsen worker 6 - jepsen.util 186 :invoke :write [2 32] INFO [2022-03-13 17:07:38,791] jepsen worker 30 - jepsen.util 210 :info :write [10 33] :RequestTimeoutException INFO [2022-03-13 17:07:38,792] jepsen worker 30 - jepsen.util 300 :invoke :read [10 nil] INFO [2022-03-13 17:07:38,795] jepsen worker 75 - jepsen.util 75 :ok :read [25 6] INFO [2022-03-13 17:07:38,796] jepsen worker 75 - jepsen.util 75 :invoke :write [25 39] INFO [2022-03-13 17:07:38,797] jepsen worker 53 - jepsen.util 143 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,800] jepsen worker 53 - jepsen.util 143 :invoke :write [17 27] INFO [2022-03-13 17:07:38,822] jepsen worker 34 - jepsen.util 124 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,823] jepsen worker 34 - jepsen.util 124 :invoke :write [11 4] INFO [2022-03-13 17:07:38,841] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:38,841] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:38,841] jepsen worker 70 - jepsen.util 250 :invoke :read [23 nil] INFO [2022-03-13 17:07:38,842] jepsen worker 12 - jepsen.util 192 :invoke :read [4 nil] INFO [2022-03-13 17:07:38,901] jepsen worker 42 - jepsen.util 132 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,901] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:38,902] jepsen worker 22 - jepsen.util 202 :invoke :read [7 nil] INFO [2022-03-13 17:07:38,903] jepsen worker 42 - jepsen.util 132 :invoke :write [14 49] INFO [2022-03-13 17:07:38,927] jepsen worker 44 - jepsen.util 134 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:07:38,928] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:38,935] jepsen worker 38 - jepsen.util 308 :ok :read [12 46] INFO [2022-03-13 17:07:38,936] jepsen worker 67 - jepsen.util 337 :ok :read [22 46] INFO [2022-03-13 17:07:38,936] jepsen worker 38 - jepsen.util 308 :invoke :write [12 22] INFO [2022-03-13 17:07:38,937] jepsen worker 67 - jepsen.util 337 :invoke :write [22 39] INFO [2022-03-13 17:07:38,972] jepsen worker 54 - jepsen.util 414 :ok :read [18 12] INFO [2022-03-13 17:07:38,972] jepsen worker 54 - jepsen.util 414 :invoke :write [18 42] INFO [2022-03-13 17:07:38,975] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:38,975] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:39,074] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:39,075] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:39,082] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:39,083] jepsen worker 44 - jepsen.util 134 :invoke :read [14 nil] INFO [2022-03-13 17:07:39,150] jepsen worker 44 - jepsen.util 134 :ok :read [14 :nil] INFO [2022-03-13 17:07:39,150] jepsen worker 21 - jepsen.util 201 :info :write [7 34] :RequestTimeoutException INFO [2022-03-13 17:07:39,151] jepsen worker 44 - jepsen.util 134 :invoke :write [14 9] INFO [2022-03-13 17:07:39,151] jepsen worker 21 - jepsen.util 291 :invoke :write [7 9] INFO [2022-03-13 17:07:39,190] jepsen worker 70 - jepsen.util 250 :ok :read [23 :nil] INFO [2022-03-13 17:07:39,190] jepsen worker 12 - jepsen.util 192 :ok :read [4 :nil] INFO [2022-03-13 17:07:39,191] jepsen worker 70 - jepsen.util 250 :invoke :write [23 32] INFO [2022-03-13 17:07:39,192] jepsen worker 12 - jepsen.util 192 :invoke :write [4 8] INFO [2022-03-13 17:07:39,197] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:39,198] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:39,200] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:39,202] jepsen worker 13 - jepsen.util 103 :invoke :read [4 nil] INFO [2022-03-13 17:07:39,330] jepsen worker 36 - jepsen.util 216 :info :write [12 34] :RequestTimeoutException INFO [2022-03-13 17:07:39,331] jepsen worker 36 - jepsen.util 306 :invoke :read [12 nil] INFO [2022-03-13 17:07:39,390] jepsen worker 84 - jepsen.util 354 :info :write [28 26] :RequestTimeoutException INFO [2022-03-13 17:07:39,391] jepsen worker 84 - jepsen.util 444 :invoke :read [28 nil] INFO [2022-03-13 17:07:39,399] jepsen worker 23 - jepsen.util 113 :ok :read [7 :nil] INFO [2022-03-13 17:07:39,400] jepsen worker 23 - jepsen.util 113 :invoke :write [7 18] INFO [2022-03-13 17:07:39,463] jepsen worker 55 - jepsen.util 325 :ok :read [18 12] INFO [2022-03-13 17:07:39,463] jepsen worker 55 - jepsen.util 325 :invoke :read [18 nil] INFO [2022-03-13 17:07:39,494] jepsen worker 22 - jepsen.util 202 :ok :read [7 :nil] INFO [2022-03-13 17:07:39,495] jepsen worker 22 - jepsen.util 202 :invoke :write [7 24] INFO [2022-03-13 17:07:39,503] jepsen worker 4 - jepsen.util 274 :info :write [1 9] :RequestTimeoutException INFO [2022-03-13 17:07:39,504] jepsen worker 4 - jepsen.util 364 :invoke :write [1 31] INFO [2022-03-13 17:07:39,520] jepsen worker 62 - jepsen.util 242 :ok :read [20 10] INFO [2022-03-13 17:07:39,521] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:39,695] jepsen worker 55 - jepsen.util 325 :ok :read [18 12] INFO [2022-03-13 17:07:39,696] jepsen worker 55 - jepsen.util 325 :invoke :write [18 26] INFO [2022-03-13 17:07:39,886] jepsen worker 15 - jepsen.util 285 :info :write [5 19] :RequestTimeoutException INFO [2022-03-13 17:07:39,887] jepsen worker 15 - jepsen.util 375 :invoke :read [5 nil] INFO [2022-03-13 17:07:40,007] jepsen worker 37 - jepsen.util 307 :ok :read [12 46] INFO [2022-03-13 17:07:40,008] jepsen worker 37 - jepsen.util 307 :invoke :write [12 48] INFO [2022-03-13 17:07:40,140] jepsen worker 61 - jepsen.util 241 :ok :read [20 10] INFO [2022-03-13 17:07:40,141] jepsen worker 61 - jepsen.util 241 :invoke :read [20 nil] INFO [2022-03-13 17:07:40,151] jepsen worker 17 - jepsen.util 287 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:07:40,153] jepsen worker 17 - jepsen.util 377 :invoke :write [5 13] INFO [2022-03-13 17:07:40,174] jepsen worker 41 - jepsen.util 311 :info :write [13 36] :RequestTimeoutException INFO [2022-03-13 17:07:40,175] jepsen worker 41 - jepsen.util 401 :invoke :write [13 16] INFO [2022-03-13 17:07:40,210] jepsen worker 24 - jepsen.util 114 :ok :read [8 :nil] INFO [2022-03-13 17:07:40,210] jepsen worker 24 - jepsen.util 114 :invoke :write [8 37] INFO [2022-03-13 17:07:40,227] jepsen worker 45 - jepsen.util 315 :info :write [15 6] :RequestTimeoutException INFO [2022-03-13 17:07:40,227] jepsen worker 39 - jepsen.util 309 :info :write [13 19] :RequestTimeoutException INFO [2022-03-13 17:07:40,228] jepsen worker 45 - jepsen.util 405 :invoke :read [15 nil] INFO [2022-03-13 17:07:40,229] jepsen worker 39 - jepsen.util 399 :invoke :read [13 nil] INFO [2022-03-13 17:07:40,290] jepsen worker 47 - jepsen.util 317 :info :write [15 39] :RequestTimeoutException INFO [2022-03-13 17:07:40,291] jepsen worker 47 - jepsen.util 407 :invoke :read [15 nil] INFO [2022-03-13 17:07:40,331] jepsen worker 87 - jepsen.util 357 :info :write [29 34] :RequestTimeoutException INFO [2022-03-13 17:07:40,332] jepsen worker 87 - jepsen.util 447 :invoke :read [29 nil] INFO [2022-03-13 17:07:40,335] jepsen worker 61 - jepsen.util 241 :ok :read [20 10] INFO [2022-03-13 17:07:40,336] jepsen worker 61 - jepsen.util 241 :invoke :read [20 nil] INFO [2022-03-13 17:07:40,410] jepsen worker 51 - jepsen.util 51 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:40,410] jepsen worker 14 - jepsen.util 14 :ok :read [4 :nil] INFO [2022-03-13 17:07:40,410] jepsen worker 13 - jepsen.util 103 :ok :read [4 :nil] INFO [2022-03-13 17:07:40,410] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:40,411] jepsen worker 13 - jepsen.util 103 :invoke :write [4 14] INFO [2022-03-13 17:07:40,411] jepsen worker 14 - jepsen.util 14 :invoke :write [4 40] INFO [2022-03-13 17:07:40,419] jepsen worker 45 - jepsen.util 405 :ok :read [15 48] INFO [2022-03-13 17:07:40,419] jepsen worker 45 - jepsen.util 405 :invoke :read [15 nil] INFO [2022-03-13 17:07:40,455] jepsen worker 50 - jepsen.util 320 :ok :read [16 :nil] INFO [2022-03-13 17:07:40,456] jepsen worker 50 - jepsen.util 320 :invoke :write [16 10] INFO [2022-03-13 17:07:40,488] jepsen worker 25 - jepsen.util 205 :ok :read [8 :nil] INFO [2022-03-13 17:07:40,488] jepsen worker 28 - jepsen.util 298 :info :write [9 36] :RequestTimeoutException INFO [2022-03-13 17:07:40,489] jepsen worker 28 - jepsen.util 388 :invoke :read [9 nil] INFO [2022-03-13 17:07:40,490] jepsen worker 25 - jepsen.util 205 :invoke :write [8 29] INFO [2022-03-13 17:07:40,527] jepsen worker 39 - jepsen.util 399 :ok :read [13 :nil] INFO [2022-03-13 17:07:40,528] jepsen worker 39 - jepsen.util 399 :invoke :write [13 18] INFO [2022-03-13 17:07:40,539] jepsen worker 61 - jepsen.util 241 :ok :read [20 10] INFO [2022-03-13 17:07:40,539] jepsen worker 61 - jepsen.util 241 :invoke :read [20 nil] INFO [2022-03-13 17:07:40,545] jepsen worker 84 - jepsen.util 444 :ok :read [28 25] INFO [2022-03-13 17:07:40,546] jepsen worker 84 - jepsen.util 444 :invoke :write [28 0] INFO [2022-03-13 17:07:40,614] jepsen worker 45 - jepsen.util 405 :ok :read [15 48] INFO [2022-03-13 17:07:40,614] jepsen worker 45 - jepsen.util 405 :invoke :read [15 nil] INFO [2022-03-13 17:07:40,618] jepsen worker 47 - jepsen.util 407 :ok :read [15 48] INFO [2022-03-13 17:07:40,619] jepsen worker 47 - jepsen.util 407 :invoke :write [15 41] INFO [2022-03-13 17:07:40,733] jepsen worker 61 - jepsen.util 241 :ok :read [20 10] INFO [2022-03-13 17:07:40,734] jepsen worker 61 - jepsen.util 241 :invoke :write [20 5] INFO [2022-03-13 17:07:40,783] jepsen worker 45 - jepsen.util 405 :ok :read [15 48] INFO [2022-03-13 17:07:40,784] jepsen worker 45 - jepsen.util 405 :invoke :read [15 nil] INFO [2022-03-13 17:07:40,939] jepsen worker 45 - jepsen.util 405 :ok :read [15 48] INFO [2022-03-13 17:07:40,939] jepsen worker 45 - jepsen.util 405 :invoke :read [15 nil] INFO [2022-03-13 17:07:41,026] jepsen worker 45 - jepsen.util 405 :ok :read [15 48] INFO [2022-03-13 17:07:41,027] jepsen worker 45 - jepsen.util 405 :invoke :write [15 37] INFO [2022-03-13 17:07:41,152] jepsen worker 51 - jepsen.util 51 :ok :read [17 20] INFO [2022-03-13 17:07:41,153] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:42,033] jepsen worker 51 - jepsen.util 51 :ok :read [17 20] INFO [2022-03-13 17:07:42,033] jepsen worker 51 - jepsen.util 51 :invoke :read [17 nil] INFO [2022-03-13 17:07:42,201] jepsen worker 35 - jepsen.util 305 :info :write [11 39] :RequestTimeoutException INFO [2022-03-13 17:07:42,202] jepsen worker 35 - jepsen.util 395 :invoke :write [11 38] INFO [2022-03-13 17:07:42,429] jepsen worker 82 - jepsen.util 352 :info :write [27 4] :RequestTimeoutException INFO [2022-03-13 17:07:42,429] jepsen worker 82 - jepsen.util 442 :invoke :read [27 nil] INFO [2022-03-13 17:07:42,610] jepsen worker 27 - jepsen.util 207 :info :write [9 17] :RequestTimeoutException INFO [2022-03-13 17:07:42,611] jepsen worker 27 - jepsen.util 297 :invoke :read [9 nil] INFO [2022-03-13 17:07:42,648] jepsen worker 51 - jepsen.util 51 :ok :read [17 20] INFO [2022-03-13 17:07:42,649] jepsen worker 51 - jepsen.util 51 :invoke :write [17 8] INFO [2022-03-13 17:07:42,739] jepsen worker 20 - jepsen.util 380 :ok :read [6 :nil] INFO [2022-03-13 17:07:42,739] jepsen worker 30 - jepsen.util 300 :ok :read [10 :nil] INFO [2022-03-13 17:07:42,740] jepsen worker 20 - jepsen.util 380 :invoke :write [6 1] INFO [2022-03-13 17:07:42,747] jepsen worker 30 - jepsen.util 300 :invoke :read [10 nil] INFO [2022-03-13 17:07:42,844] jepsen worker 30 - jepsen.util 300 :ok :read [10 :nil] INFO [2022-03-13 17:07:42,845] jepsen worker 30 - jepsen.util 300 :invoke :read [10 nil] INFO [2022-03-13 17:07:42,858] jepsen worker 28 - jepsen.util 388 :ok :read [9 30] INFO [2022-03-13 17:07:42,858] jepsen worker 28 - jepsen.util 388 :invoke :write [9 19] INFO [2022-03-13 17:07:42,964] jepsen worker 30 - jepsen.util 300 :ok :read [10 :nil] INFO [2022-03-13 17:07:42,964] jepsen worker 30 - jepsen.util 300 :invoke :write [10 2] INFO [2022-03-13 17:07:43,499] jepsen worker 16 - jepsen.util 286 :ok :read [5 :nil] INFO [2022-03-13 17:07:43,500] jepsen worker 16 - jepsen.util 286 :invoke :read [5 nil] INFO [2022-03-13 17:07:43,575] jepsen worker 15 - jepsen.util 375 :ok :read [5 :nil] INFO [2022-03-13 17:07:43,576] jepsen worker 15 - jepsen.util 375 :invoke :write [5 15] INFO [2022-03-13 17:07:43,586] jepsen worker 16 - jepsen.util 286 :ok :read [5 :nil] INFO [2022-03-13 17:07:43,587] jepsen worker 16 - jepsen.util 286 :invoke :write [5 40] INFO [2022-03-13 17:07:43,596] jepsen worker 10 - jepsen.util 190 :ok :read [3 :nil] INFO [2022-03-13 17:07:43,597] jepsen worker 10 - jepsen.util 190 :invoke :read [3 nil] INFO [2022-03-13 17:07:43,600] jepsen worker 86 - jepsen.util 356 :ok :read [28 25] INFO [2022-03-13 17:07:43,600] jepsen worker 86 - jepsen.util 356 :invoke :read [28 nil] INFO [2022-03-13 17:07:43,603] jepsen worker 87 - jepsen.util 447 :ok :read [29 17] INFO [2022-03-13 17:07:43,604] jepsen worker 87 - jepsen.util 447 :invoke :read [29 nil] INFO [2022-03-13 17:07:43,651] jepsen worker 63 - jepsen.util 333 :ok :read [21 :nil] INFO [2022-03-13 17:07:43,651] jepsen worker 86 - jepsen.util 356 :ok :read [28 25] INFO [2022-03-13 17:07:43,652] jepsen worker 63 - jepsen.util 333 :invoke :write [21 45] INFO [2022-03-13 17:07:43,652] jepsen worker 86 - jepsen.util 356 :invoke :read [28 nil] INFO [2022-03-13 17:07:43,653] jepsen worker 72 - jepsen.util 342 :ok :read [24 24] INFO [2022-03-13 17:07:43,654] jepsen worker 72 - jepsen.util 342 :invoke :write [24 25] INFO [2022-03-13 17:07:43,666] jepsen worker 46 - jepsen.util 316 :ok :read [15 48] INFO [2022-03-13 17:07:43,667] jepsen worker 46 - jepsen.util 316 :invoke :read [15 nil] INFO [2022-03-13 17:07:43,677] jepsen worker 87 - jepsen.util 447 :ok :read [29 17] INFO [2022-03-13 17:07:43,678] jepsen worker 87 - jepsen.util 447 :invoke :write [29 11] INFO [2022-03-13 17:07:43,690] jepsen worker 65 - jepsen.util 335 :ok :read [21 :nil] INFO [2022-03-13 17:07:43,690] jepsen worker 9 - jepsen.util 369 :ok :read [3 :nil] INFO [2022-03-13 17:07:43,690] jepsen worker 65 - jepsen.util 335 :invoke :read [21 nil] INFO [2022-03-13 17:07:43,691] jepsen worker 9 - jepsen.util 369 :invoke :write [3 47] INFO [2022-03-13 17:07:43,733] jepsen worker 86 - jepsen.util 356 :ok :read [28 25] INFO [2022-03-13 17:07:43,733] jepsen worker 86 - jepsen.util 356 :invoke :read [28 nil] INFO [2022-03-13 17:07:43,847] jepsen worker 86 - jepsen.util 356 :ok :read [28 25] INFO [2022-03-13 17:07:43,847] jepsen worker 86 - jepsen.util 356 :invoke :write [28 16] INFO [2022-03-13 17:07:44,014] jepsen worker 5 - jepsen.util 185 :info :write [1 6] :SyncWriteAmbiguous INFO [2022-03-13 17:07:44,015] jepsen worker 5 - jepsen.util 275 :invoke :read [1 nil] INFO [2022-03-13 17:07:44,044] jepsen worker 81 - jepsen.util 351 :info :write [27 17] :SyncWriteAmbiguous INFO [2022-03-13 17:07:44,044] jepsen worker 81 - jepsen.util 441 :invoke :read [27 nil] INFO [2022-03-13 17:07:44,098] jepsen worker 48 - jepsen.util 228 :info :write [16 31] :RequestTimeoutException INFO [2022-03-13 17:07:44,099] jepsen worker 48 - jepsen.util 318 :invoke :read [16 nil] INFO [2022-03-13 17:07:44,146] jepsen worker 49 - jepsen.util 319 :info :write [16 8] :RequestTimeoutException INFO [2022-03-13 17:07:44,147] jepsen worker 49 - jepsen.util 409 :invoke :read [16 nil] INFO [2022-03-13 17:07:44,220] jepsen worker 48 - jepsen.util 318 :ok :read [16 :nil] INFO [2022-03-13 17:07:44,220] jepsen worker 48 - jepsen.util 318 :invoke :write [16 46] INFO [2022-03-13 17:07:44,283] jepsen worker 81 - jepsen.util 441 :ok :read [27 8] INFO [2022-03-13 17:07:44,284] jepsen worker 81 - jepsen.util 441 :invoke :write [27 36] INFO [2022-03-13 17:07:44,309] jepsen worker 40 - jepsen.util 310 :info :write [13 2] :RequestTimeoutException INFO [2022-03-13 17:07:44,309] jepsen worker 40 - jepsen.util 400 :invoke :read [13 nil] INFO [2022-03-13 17:07:44,609] jepsen worker 10 - jepsen.util 190 :ok :read [3 :nil] INFO [2022-03-13 17:07:44,610] jepsen worker 10 - jepsen.util 190 :invoke :read [3 nil] INFO [2022-03-13 17:07:44,646] jepsen worker 18 - jepsen.util 198 :info :write [6 15] :RequestTimeoutException INFO [2022-03-13 17:07:44,647] jepsen worker 18 - jepsen.util 288 :invoke :write [6 29] INFO [2022-03-13 17:07:44,648] jepsen worker 46 - jepsen.util 316 :ok :read [15 48] INFO [2022-03-13 17:07:44,648] jepsen worker 46 - jepsen.util 316 :invoke :write [15 49] INFO [2022-03-13 17:07:44,664] jepsen worker 27 - jepsen.util 297 :ok :read [9 30] INFO [2022-03-13 17:07:44,666] jepsen worker 27 - jepsen.util 297 :invoke :write [9 9] INFO [2022-03-13 17:07:44,773] jepsen worker 5 - jepsen.util 275 :ok :read [1 28] INFO [2022-03-13 17:07:44,774] jepsen worker 5 - jepsen.util 275 :invoke :write [1 7] INFO [2022-03-13 17:07:44,873] jepsen worker 65 - jepsen.util 335 :ok :read [21 :nil] INFO [2022-03-13 17:07:44,874] jepsen worker 65 - jepsen.util 335 :invoke :write [21 41] INFO [2022-03-13 17:07:44,941] jepsen worker 9 - jepsen.util 369 :fail :write [3 47] :ServerOutOfMemoryException INFO [2022-03-13 17:07:44,941] jepsen worker 9 - jepsen.util 369 :invoke :read [3 nil] INFO [2022-03-13 17:07:44,981] jepsen worker 73 - jepsen.util 343 :info :write [24 35] :RequestTimeoutException INFO [2022-03-13 17:07:44,982] jepsen worker 73 - jepsen.util 433 :invoke :write [24 33] INFO [2022-03-13 17:07:45,021] jepsen worker 10 - jepsen.util 190 :ok :read [3 :nil] INFO [2022-03-13 17:07:45,022] jepsen worker 10 - jepsen.util 190 :invoke :write [3 33] INFO [2022-03-13 17:07:45,158] jepsen worker 68 - jepsen.util 338 :info :write [22 11] :RequestTimeoutException INFO [2022-03-13 17:07:45,159] jepsen worker 68 - jepsen.util 428 :invoke :read [22 nil] WARN [2022-03-13 17:07:45,209] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19964900,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21008135,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9540904,"encode_duration_us":18047,"operation_id":"0x9a1","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10003832},{"operation_name":"upsert","last_dispatch_duration_us":15220753,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19634303,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":7076693,"encode_duration_us":27202,"operation_id":"0xbe2","timeout_ms":10000,"last_server_duration_us":6424068,"total_duration_us":10010654},{"operation_name":"upsert","last_dispatch_duration_us":15354221,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15865213,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9372108,"encode_duration_us":19103,"operation_id":"0xc7a","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10008448}],"total_count":3}} WARN [2022-03-13 17:07:45,209] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18083678,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":18083678,"total_server_duration_us":23,"operation_id":"0xbb5","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":18084206},{"operation_name":"get","last_dispatch_duration_us":11580077,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":11580077,"total_server_duration_us":32,"operation_id":"0xd45","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11583526},{"operation_name":"get","last_dispatch_duration_us":10248130,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10248130,"total_server_duration_us":23,"operation_id":"0xddd","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":10248571},{"operation_name":"upsert","last_dispatch_duration_us":8610730,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":15508081,"total_server_duration_us":271625,"encode_duration_us":16957,"operation_id":"0xcc5","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10007925},{"operation_name":"upsert","last_dispatch_duration_us":116358,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9804101,"total_server_duration_us":530972,"encode_duration_us":17821,"operation_id":"0xe67","timeout_ms":10000,"last_server_duration_us":47306,"total_duration_us":10004692},{"operation_name":"upsert","last_dispatch_duration_us":9128633,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":15916655,"total_server_duration_us":161506,"encode_duration_us":19230,"operation_id":"0xcf4","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10004278},{"operation_name":"upsert","last_dispatch_duration_us":828438,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9969232,"total_server_duration_us":595262,"encode_duration_us":18266,"operation_id":"0xd6e","timeout_ms":10000,"last_server_duration_us":382594,"total_duration_us":10003327},{"operation_name":"upsert","last_dispatch_duration_us":11056846,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":11056846,"total_server_duration_us":67353,"encode_duration_us":18248,"operation_id":"0xd92","timeout_ms":10000,"last_server_duration_us":67353,"total_duration_us":10001794},{"operation_name":"upsert","last_dispatch_duration_us":687644,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8535584,"total_server_duration_us":289562,"encode_duration_us":19810,"operation_id":"0xdf2","timeout_ms":10000,"last_server_duration_us":42091,"total_duration_us":10000840},{"operation_name":"get","last_dispatch_duration_us":9108656,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9108656,"total_server_duration_us":27,"operation_id":"0xeac","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9109130}],"total_count":16}} INFO [2022-03-13 17:07:45,216] jepsen worker 68 - jepsen.util 428 :ok :read [22 46] INFO [2022-03-13 17:07:45,217] jepsen worker 68 - jepsen.util 428 :invoke :write [22 24] INFO [2022-03-13 17:07:45,364] jepsen worker 83 - jepsen.util 263 :info :write [27 0] :RequestTimeoutException INFO [2022-03-13 17:07:45,367] jepsen worker 83 - jepsen.util 353 :invoke :read [27 nil] INFO [2022-03-13 17:07:45,596] jepsen worker 9 - jepsen.util 369 :ok :read [3 :nil] INFO [2022-03-13 17:07:45,597] jepsen worker 9 - jepsen.util 369 :invoke :read [3 nil] INFO [2022-03-13 17:07:45,754] jepsen worker 89 - jepsen.util 269 :info :write [29 49] :RequestTimeoutException INFO [2022-03-13 17:07:45,755] jepsen worker 89 - jepsen.util 359 :invoke :read [29 nil] INFO [2022-03-13 17:07:45,787] jepsen worker 85 - jepsen.util 355 :info :write [28 36] :RequestTimeoutException INFO [2022-03-13 17:07:45,788] jepsen worker 85 - jepsen.util 445 :invoke :write [28 46] INFO [2022-03-13 17:07:45,867] jepsen worker 88 - jepsen.util 358 :info :write [29 15] :RequestTimeoutException INFO [2022-03-13 17:07:45,868] jepsen worker 88 - jepsen.util 448 :invoke :read [29 nil] INFO [2022-03-13 17:07:45,921] jepsen worker 58 - jepsen.util 328 :info :write [19 20] :RequestTimeoutException INFO [2022-03-13 17:07:45,921] jepsen worker 58 - jepsen.util 418 :invoke :read [19 nil] INFO [2022-03-13 17:07:45,923] jepsen worker 66 - jepsen.util 246 :info :write [22 44] :RequestTimeoutException INFO [2022-03-13 17:07:45,924] jepsen worker 66 - jepsen.util 336 :invoke :write [22 35] INFO [2022-03-13 17:07:46,076] jepsen worker 0 - jepsen.util 270 :info :write [0 42] :RequestTimeoutException INFO [2022-03-13 17:07:46,077] jepsen worker 0 - jepsen.util 360 :invoke :write [0 47] INFO [2022-03-13 17:07:46,266] jepsen worker 32 - jepsen.util 212 :info :write [10 6] :RequestTimeoutException INFO [2022-03-13 17:07:46,267] jepsen worker 32 - jepsen.util 302 :invoke :write [10 28] INFO [2022-03-13 17:07:46,273] jepsen worker 9 - jepsen.util 369 :ok :read [3 :nil] INFO [2022-03-13 17:07:46,274] jepsen worker 9 - jepsen.util 369 :invoke :read [3 nil] INFO [2022-03-13 17:07:46,436] jepsen worker 56 - jepsen.util 236 :info :write [18 23] :RequestTimeoutException INFO [2022-03-13 17:07:46,437] jepsen worker 56 - jepsen.util 326 :invoke :read [18 nil] INFO [2022-03-13 17:07:46,564] jepsen worker 57 - jepsen.util 327 :info :write [19 37] :RequestTimeoutException INFO [2022-03-13 17:07:46,566] jepsen worker 57 - jepsen.util 417 :invoke :write [19 17] INFO [2022-03-13 17:07:46,614] jepsen worker 56 - jepsen.util 326 :ok :read [18 12] INFO [2022-03-13 17:07:46,615] jepsen worker 56 - jepsen.util 326 :invoke :write [18 3] INFO [2022-03-13 17:07:46,756] jepsen worker 3 - jepsen.util 273 :info :write [1 0] :RequestTimeoutException INFO [2022-03-13 17:07:46,757] jepsen worker 3 - jepsen.util 363 :invoke :write [1 25] WARN [2022-03-13 17:07:46,807] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15485800,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":15485800,"total_server_duration_us":38,"operation_id":"0xd25","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":15486276},{"operation_name":"get","last_dispatch_duration_us":15191284,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":15191284,"total_server_duration_us":8069,"operation_id":"0xd39","timeout_ms":30000,"last_server_duration_us":8069,"total_duration_us":15193645},{"operation_name":"get","last_dispatch_duration_us":14540684,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":14540684,"total_server_duration_us":27,"operation_id":"0xd8b","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14542265},{"operation_name":"get","last_dispatch_duration_us":13861283,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13861283,"total_server_duration_us":19,"operation_id":"0xdd0","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":13861727},{"operation_name":"get","last_dispatch_duration_us":13326515,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13326515,"total_server_duration_us":6191,"operation_id":"0xdf9","timeout_ms":30000,"last_server_duration_us":6191,"total_duration_us":13327016},{"operation_name":"upsert","last_dispatch_duration_us":13465631,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13465631,"total_server_duration_us":19966,"encode_duration_us":17543,"operation_id":"0xdef","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10006740},{"operation_name":"upsert","last_dispatch_duration_us":1260206,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10370294,"total_server_duration_us":105385,"encode_duration_us":16996,"operation_id":"0xecb","timeout_ms":10000,"last_server_duration_us":65645,"total_duration_us":10005775},{"operation_name":"get","last_dispatch_duration_us":9713762,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9713762,"total_server_duration_us":11,"operation_id":"0xea6","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":9714065},{"operation_name":"get","last_dispatch_duration_us":5353359,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":5353359,"total_server_duration_us":43,"operation_id":"0xedc","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5353894},{"operation_name":"get","last_dispatch_duration_us":2030073,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":2030073,"total_server_duration_us":17290,"operation_id":"0x10ee","timeout_ms":30000,"last_server_duration_us":17290,"total_duration_us":2033575}],"total_count":17}} WARN [2022-03-13 17:07:46,813] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":814126,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":3285748,"total_server_duration_us":2737116,"encode_duration_us":22611,"operation_id":"0xed9","timeout_ms":10000,"last_server_duration_us":788761,"total_duration_us":10011171},{"operation_name":"upsert","last_dispatch_duration_us":44347,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":3507501,"total_server_duration_us":316885,"encode_duration_us":17532,"operation_id":"0xda8","timeout_ms":10000,"last_server_duration_us":28676,"total_duration_us":10008053},{"operation_name":"upsert","last_dispatch_duration_us":7728814,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8145110,"total_server_duration_us":249023,"encode_duration_us":33526,"operation_id":"0xf1f","timeout_ms":10000,"last_server_duration_us":27132,"total_duration_us":10006722},{"operation_name":"upsert","last_dispatch_duration_us":7179995,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10848807,"total_server_duration_us":385931,"encode_duration_us":34483,"operation_id":"0xda1","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10006113},{"operation_name":"upsert","last_dispatch_duration_us":8903942,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10701660,"total_server_duration_us":9026897,"encode_duration_us":18887,"operation_id":"0xdec","timeout_ms":10000,"last_server_duration_us":8890351,"total_duration_us":10005450},{"operation_name":"upsert","last_dispatch_duration_us":9044469,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10884745,"total_server_duration_us":9148383,"encode_duration_us":26908,"operation_id":"0xdf5","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10003826},{"operation_name":"upsert","last_dispatch_duration_us":45953,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":6219873,"total_server_duration_us":411233,"encode_duration_us":19442,"operation_id":"0xcdd","timeout_ms":10000,"last_server_duration_us":25712,"total_duration_us":10002707},{"operation_name":"upsert","last_dispatch_duration_us":9044049,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9831204,"total_server_duration_us":9312868,"encode_duration_us":18461,"operation_id":"0xd9e","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10001830},{"operation_name":"upsert","last_dispatch_duration_us":64835,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":2220625,"total_server_duration_us":1263880,"encode_duration_us":19929,"operation_id":"0xe7e","timeout_ms":10000,"last_server_duration_us":43948,"total_duration_us":10000999},{"operation_name":"upsert","last_dispatch_duration_us":812407,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3003471,"total_server_duration_us":218270,"encode_duration_us":29605,"operation_id":"0xe64","timeout_ms":10000,"last_server_duration_us":107430,"total_duration_us":6145244}],"total_count":40}} WARN [2022-03-13 17:07:46,813] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18397777,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18810449,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9250380,"encode_duration_us":17946,"operation_id":"0xc57","timeout_ms":10000,"last_server_duration_us":9118182,"total_duration_us":10002831},{"operation_name":"upsert","last_dispatch_duration_us":10072454,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10448199,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9287068,"encode_duration_us":19982,"operation_id":"0xe00","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10010586}],"total_count":2}} WARN [2022-03-13 17:07:46,814] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10072454,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10448199,"total_server_duration_us":9287068,"encode_duration_us":19982,"operation_id":"0xe00","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10010586},{"operation_name":"upsert","last_dispatch_duration_us":3528835,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":4953858,"total_server_duration_us":112579,"encode_duration_us":20221,"operation_id":"0xede","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10009981},{"operation_name":"upsert","last_dispatch_duration_us":1901040,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5539826,"total_server_duration_us":190239,"encode_duration_us":17933,"operation_id":"0xeaa","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10007826},{"operation_name":"upsert","last_dispatch_duration_us":3470798,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":3995645,"total_server_duration_us":114636,"encode_duration_us":19440,"operation_id":"0xe6b","timeout_ms":10000,"last_server_duration_us":27583,"total_duration_us":10006846},{"operation_name":"upsert","last_dispatch_duration_us":3991160,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5356766,"total_server_duration_us":224967,"encode_duration_us":18138,"operation_id":"0xea7","timeout_ms":10000,"last_server_duration_us":24325,"total_duration_us":10004860},{"operation_name":"upsert","last_dispatch_duration_us":7284623,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10827580,"total_server_duration_us":466779,"encode_duration_us":16820,"operation_id":"0xd2a","timeout_ms":10000,"last_server_duration_us":58874,"total_duration_us":10002206},{"operation_name":"upsert","last_dispatch_duration_us":9128871,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9695507,"total_server_duration_us":9157108,"encode_duration_us":20998,"operation_id":"0xea8","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9893609},{"operation_name":"upsert","last_dispatch_duration_us":9153619,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9490702,"total_server_duration_us":9077816,"encode_duration_us":18989,"operation_id":"0xeb7","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":9527412},{"operation_name":"get","last_dispatch_duration_us":5523871,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5523871,"total_server_duration_us":23,"operation_id":"0xfab","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":5524398},{"operation_name":"get","last_dispatch_duration_us":3945863,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":3945863,"total_server_duration_us":15,"operation_id":"0x104f","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":3946130}],"total_count":15}} INFO [2022-03-13 17:07:46,825] jepsen worker 36 - jepsen.util 306 :ok :read [12 46] INFO [2022-03-13 17:07:46,826] jepsen worker 36 - jepsen.util 306 :invoke :write [12 7] WARN [2022-03-13 17:07:46,907] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13804497,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19404891,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9320160,"encode_duration_us":23624,"operation_id":"0xbc1","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10001812},{"operation_name":"upsert","last_dispatch_duration_us":16956102,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18498191,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9634676,"encode_duration_us":18034,"operation_id":"0xaf4","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10007737}],"total_count":2}} WARN [2022-03-13 17:07:46,913] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":142189,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1611496,"total_server_duration_us":383833,"encode_duration_us":20026,"operation_id":"0xf0f","timeout_ms":10000,"last_server_duration_us":42417,"total_duration_us":10008424},{"operation_name":"upsert","last_dispatch_duration_us":4805656,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5260967,"total_server_duration_us":168005,"encode_duration_us":17461,"operation_id":"0xea9","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10008041},{"operation_name":"upsert","last_dispatch_duration_us":4864167,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5372948,"total_server_duration_us":175763,"encode_duration_us":26000,"operation_id":"0xdd6","timeout_ms":10000,"last_server_duration_us":33230,"total_duration_us":10007156},{"operation_name":"upsert","last_dispatch_duration_us":280879,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":4313483,"total_server_duration_us":826721,"encode_duration_us":21668,"operation_id":"0xf3f","timeout_ms":10000,"last_server_duration_us":32258,"total_duration_us":10006621},{"operation_name":"upsert","last_dispatch_duration_us":95268,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":4148436,"total_server_duration_us":925142,"encode_duration_us":18350,"operation_id":"0xf46","timeout_ms":10000,"last_server_duration_us":33426,"total_duration_us":10004330},{"operation_name":"upsert","last_dispatch_duration_us":446002,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10139327,"total_server_duration_us":332681,"encode_duration_us":19204,"operation_id":"0xcdb","timeout_ms":10000,"last_server_duration_us":106144,"total_duration_us":10002666},{"operation_name":"upsert","last_dispatch_duration_us":70136,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":621337,"total_server_duration_us":156622,"encode_duration_us":17918,"operation_id":"0xdf0","timeout_ms":10000,"last_server_duration_us":35815,"total_duration_us":10002665},{"operation_name":"upsert","last_dispatch_duration_us":44992,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1026581,"total_server_duration_us":246234,"encode_duration_us":31645,"operation_id":"0xf1a","timeout_ms":10000,"last_server_duration_us":31872,"total_duration_us":10002376},{"operation_name":"upsert","last_dispatch_duration_us":4739552,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5860124,"total_server_duration_us":318143,"encode_duration_us":21046,"operation_id":"0xf2e","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10001590},{"operation_name":"upsert","last_dispatch_duration_us":946880,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2510969,"total_server_duration_us":197917,"encode_duration_us":16851,"operation_id":"0xe36","timeout_ms":10000,"last_server_duration_us":84283,"total_duration_us":6159421}],"total_count":41}} WARN [2022-03-13 17:07:46,913] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9044049,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9831204,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9312868,"encode_duration_us":18461,"operation_id":"0xd9e","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10001830},{"operation_name":"upsert","last_dispatch_duration_us":8903942,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10701660,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9026897,"encode_duration_us":18887,"operation_id":"0xdec","timeout_ms":10000,"last_server_duration_us":8890351,"total_duration_us":10005450},{"operation_name":"upsert","last_dispatch_duration_us":9044469,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10884745,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9148383,"encode_duration_us":26908,"operation_id":"0xdf5","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10003826}],"total_count":3}} WARN [2022-03-13 17:07:46,913] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18216964,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27382784,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9099419,"encode_duration_us":20428,"operation_id":"0x977","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10001600},{"operation_name":"upsert","last_dispatch_duration_us":14321915,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14321915,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9049045,"encode_duration_us":19399,"operation_id":"0xdee","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10003390},{"operation_name":"upsert","last_dispatch_duration_us":16367853,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16367853,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9040551,"encode_duration_us":18218,"operation_id":"0xd22","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10002811},{"operation_name":"upsert","last_dispatch_duration_us":18201824,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18201824,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9074546,"encode_duration_us":35870,"operation_id":"0xba3","timeout_ms":10000,"last_server_duration_us":9074546,"total_duration_us":10004359}],"total_count":4}} WARN [2022-03-13 17:07:46,914] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7122918,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9892858,"total_server_duration_us":134712,"encode_duration_us":18678,"operation_id":"0xee3","timeout_ms":10000,"last_server_duration_us":30256,"total_duration_us":10009425},{"operation_name":"upsert","last_dispatch_duration_us":1480593,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11153157,"total_server_duration_us":443318,"encode_duration_us":21267,"operation_id":"0xd23","timeout_ms":10000,"last_server_duration_us":192926,"total_duration_us":10007484},{"operation_name":"upsert","last_dispatch_duration_us":3695481,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":3695481,"total_server_duration_us":23729,"encode_duration_us":32480,"operation_id":"0xf20","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10007132},{"operation_name":"upsert","last_dispatch_duration_us":3542180,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":4392427,"total_server_duration_us":44141,"encode_duration_us":20491,"operation_id":"0xf09","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10006004},{"operation_name":"upsert","last_dispatch_duration_us":3529392,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":3529392,"total_server_duration_us":22471,"encode_duration_us":20893,"operation_id":"0xf62","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10004863},{"operation_name":"upsert","last_dispatch_duration_us":14321915,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":14321915,"total_server_duration_us":9049045,"encode_duration_us":19399,"operation_id":"0xdee","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10003390},{"operation_name":"upsert","last_dispatch_duration_us":16367853,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16367853,"total_server_duration_us":9040551,"encode_duration_us":18218,"operation_id":"0xd22","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10002811},{"operation_name":"upsert","last_dispatch_duration_us":967035,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9974527,"total_server_duration_us":260640,"encode_duration_us":21801,"operation_id":"0xd33","timeout_ms":10000,"last_server_duration_us":200638,"total_duration_us":10002279},{"operation_name":"get","last_dispatch_duration_us":9120048,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9120048,"total_server_duration_us":32,"operation_id":"0xd18","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9120479},{"operation_name":"get","last_dispatch_duration_us":9022669,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9022669,"total_server_duration_us":23,"operation_id":"0xd21","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":9023057}],"total_count":16}} WARN [2022-03-13 17:07:47,013] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17841754,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18013697,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9067283,"encode_duration_us":20946,"operation_id":"0xc97","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10002885}],"total_count":1}} INFO [2022-03-13 17:07:47,036] jepsen worker 59 - jepsen.util 419 :info :write [19 35] :RequestTimeoutException INFO [2022-03-13 17:07:47,037] jepsen worker 59 - jepsen.util 509 :invoke :read [19 nil] INFO [2022-03-13 17:07:47,075] jepsen worker 9 - jepsen.util 369 :ok :read [3 :nil] INFO [2022-03-13 17:07:47,076] jepsen worker 9 - jepsen.util 369 :invoke :write [3 21] INFO [2022-03-13 17:07:47,211] jepsen worker 74 - jepsen.util 344 :info :write [24 43] :RequestTimeoutException INFO [2022-03-13 17:07:47,212] jepsen worker 74 - jepsen.util 434 :invoke :read [24 nil] INFO [2022-03-13 17:07:47,214] jepsen worker 74 - jepsen.util 434 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:47,214] jepsen worker 74 - jepsen.util 434 :invoke :write [24 24] INFO [2022-03-13 17:07:47,351] jepsen worker 11 - jepsen.util 191 :info :write [3 36] :RequestTimeoutException INFO [2022-03-13 17:07:47,352] jepsen worker 11 - jepsen.util 281 :invoke :read [3 nil] INFO [2022-03-13 17:07:47,424] jepsen worker 64 - jepsen.util 154 :info :write [21 10] :RequestTimeoutException INFO [2022-03-13 17:07:47,425] jepsen worker 64 - jepsen.util 244 :invoke :read [21 nil] INFO [2022-03-13 17:07:47,497] jepsen worker 11 - jepsen.util 281 :ok :read [3 :nil] INFO [2022-03-13 17:07:47,498] jepsen worker 11 - jepsen.util 281 :invoke :write [3 43] INFO [2022-03-13 17:07:47,560] jepsen worker 1 - jepsen.util 271 :info :write [0 41] :RequestTimeoutException INFO [2022-03-13 17:07:47,561] jepsen worker 1 - jepsen.util 361 :invoke :read [0 nil] INFO [2022-03-13 17:07:47,640] jepsen worker 64 - jepsen.util 244 :ok :read [21 :nil] INFO [2022-03-13 17:07:47,641] jepsen worker 64 - jepsen.util 244 :invoke :write [21 10] INFO [2022-03-13 17:07:47,774] jepsen worker 19 - jepsen.util 199 :info :write [6 23] :RequestTimeoutException INFO [2022-03-13 17:07:47,774] jepsen worker 19 - jepsen.util 289 :invoke :read [6 nil] INFO [2022-03-13 17:07:47,838] jepsen worker 60 - jepsen.util 240 :info :write [20 9] :RequestTimeoutException INFO [2022-03-13 17:07:47,839] jepsen worker 60 - jepsen.util 330 :invoke :write [20 20] INFO [2022-03-13 17:07:48,054] jepsen worker 2 - jepsen.util 362 :info :write [0 34] :RequestTimeoutException INFO [2022-03-13 17:07:48,055] jepsen worker 2 - jepsen.util 452 :invoke :write [0 36] INFO [2022-03-13 17:07:48,097] jepsen worker 29 - jepsen.util 299 :info :write [9 42] :RequestTimeoutException INFO [2022-03-13 17:07:48,097] jepsen worker 29 - jepsen.util 389 :invoke :read [9 nil] INFO [2022-03-13 17:07:48,136] jepsen worker 43 - jepsen.util 133 :info :write [14 1] :RequestTimeoutException INFO [2022-03-13 17:07:48,138] jepsen worker 43 - jepsen.util 223 :invoke :write [14 17] INFO [2022-03-13 17:07:48,266] jepsen worker 29 - jepsen.util 389 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:48,266] jepsen worker 29 - jepsen.util 389 :invoke :write [9 0] INFO [2022-03-13 17:07:48,270] jepsen worker 52 - jepsen.util 142 :info :write [17 11] :RequestTimeoutException INFO [2022-03-13 17:07:48,271] jepsen worker 52 - jepsen.util 232 :invoke :read [17 nil] INFO [2022-03-13 17:07:48,317] jepsen worker 69 - jepsen.util 159 :info :write [23 40] :RequestTimeoutException INFO [2022-03-13 17:07:48,318] jepsen worker 69 - jepsen.util 249 :invoke :write [23 30] INFO [2022-03-13 17:07:48,337] jepsen worker 77 - jepsen.util 167 :info :write [25 18] :RequestTimeoutException INFO [2022-03-13 17:07:48,338] jepsen worker 77 - jepsen.util 257 :invoke :read [25 nil] INFO [2022-03-13 17:07:48,340] jepsen worker 7 - jepsen.util 97 :info :write [2 29] :RequestTimeoutException INFO [2022-03-13 17:07:48,341] jepsen worker 7 - jepsen.util 187 :invoke :read [2 nil] INFO [2022-03-13 17:07:48,372] jepsen worker 77 - jepsen.util 257 :ok :read [25 6] INFO [2022-03-13 17:07:48,373] jepsen worker 77 - jepsen.util 257 :invoke :write [25 41] INFO [2022-03-13 17:07:48,380] jepsen worker 80 - jepsen.util 260 :info :write [26 12] :SyncWriteAmbiguous INFO [2022-03-13 17:07:48,381] jepsen worker 80 - jepsen.util 350 :invoke :write [26 41] INFO [2022-03-13 17:07:48,387] jepsen worker 71 - jepsen.util 71 :info :write [23 19] :RequestTimeoutException INFO [2022-03-13 17:07:48,388] jepsen worker 71 - jepsen.util 161 :invoke :read [23 nil] INFO [2022-03-13 17:07:48,410] jepsen worker 79 - jepsen.util 349 :info :write [26 2] :RequestTimeoutException INFO [2022-03-13 17:07:48,411] jepsen worker 8 - jepsen.util 278 :info :write [2 47] :RequestTimeoutException INFO [2022-03-13 17:07:48,411] jepsen worker 31 - jepsen.util 301 :info :write [10 3] :RequestTimeoutException INFO [2022-03-13 17:07:48,412] jepsen worker 79 - jepsen.util 439 :invoke :write [26 41] INFO [2022-03-13 17:07:48,413] jepsen worker 8 - jepsen.util 368 :invoke :write [2 17] INFO [2022-03-13 17:07:48,413] jepsen worker 31 - jepsen.util 391 :invoke :read [10 nil] INFO [2022-03-13 17:07:48,458] jepsen worker 53 - jepsen.util 143 :info :write [17 27] :SyncWriteAmbiguous INFO [2022-03-13 17:07:48,459] jepsen worker 53 - jepsen.util 233 :invoke :read [17 nil] INFO [2022-03-13 17:07:48,570] jepsen worker 26 - jepsen.util 116 :info :write [8 42] :RequestTimeoutException INFO [2022-03-13 17:07:48,571] jepsen worker 26 - jepsen.util 206 :invoke :write [8 32] INFO [2022-03-13 17:07:48,654] jepsen worker 33 - jepsen.util 213 :info :write [11 19] :RequestTimeoutException INFO [2022-03-13 17:07:48,655] jepsen worker 33 - jepsen.util 303 :invoke :write [11 49] INFO [2022-03-13 17:07:48,657] jepsen worker 76 - jepsen.util 166 :info :write [25 8] :RequestTimeoutException INFO [2022-03-13 17:07:48,658] jepsen worker 76 - jepsen.util 256 :invoke :write [25 37] INFO [2022-03-13 17:07:48,790] jepsen worker 6 - jepsen.util 186 :info :write [2 32] :RequestTimeoutException INFO [2022-03-13 17:07:48,791] jepsen worker 6 - jepsen.util 276 :invoke :write [2 44] INFO [2022-03-13 17:07:48,827] jepsen worker 75 - jepsen.util 75 :info :write [25 39] :RequestTimeoutException INFO [2022-03-13 17:07:48,827] jepsen worker 75 - jepsen.util 165 :invoke :write [25 23] INFO [2022-03-13 17:07:48,857] jepsen worker 34 - jepsen.util 124 :info :write [11 4] :RequestTimeoutException INFO [2022-03-13 17:07:48,857] jepsen worker 34 - jepsen.util 214 :invoke :write [11 49] INFO [2022-03-13 17:07:48,873] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:48,874] jepsen worker 62 - jepsen.util 242 :invoke :read [20 nil] INFO [2022-03-13 17:07:48,887] jepsen worker 78 - jepsen.util 258 :ok :read [26 36] INFO [2022-03-13 17:07:48,888] jepsen worker 78 - jepsen.util 258 :invoke :write [26 33] INFO [2022-03-13 17:07:48,931] jepsen worker 42 - jepsen.util 132 :info :write [14 49] :RequestTimeoutException INFO [2022-03-13 17:07:48,931] jepsen worker 42 - jepsen.util 222 :invoke :write [14 43] INFO [2022-03-13 17:07:48,968] jepsen worker 38 - jepsen.util 308 :info :write [12 22] :RequestTimeoutException INFO [2022-03-13 17:07:48,969] jepsen worker 38 - jepsen.util 398 :invoke :write [12 5] INFO [2022-03-13 17:07:48,974] jepsen worker 67 - jepsen.util 337 :info :write [22 39] :RequestTimeoutException INFO [2022-03-13 17:07:48,975] jepsen worker 67 - jepsen.util 427 :invoke :write [22 45] INFO [2022-03-13 17:07:49,007] jepsen worker 54 - jepsen.util 414 :info :write [18 42] :RequestTimeoutException INFO [2022-03-13 17:07:49,009] jepsen worker 54 - jepsen.util 504 :invoke :write [18 40] INFO [2022-03-13 17:07:49,056] jepsen worker 83 - jepsen.util 353 :ok :read [27 8] INFO [2022-03-13 17:07:49,057] jepsen worker 83 - jepsen.util 353 :invoke :read [27 nil] INFO [2022-03-13 17:07:49,107] jepsen worker 40 - jepsen.util 400 :ok :read [13 :nil] INFO [2022-03-13 17:07:49,108] jepsen worker 40 - jepsen.util 400 :invoke :read [13 nil] INFO [2022-03-13 17:07:49,174] jepsen worker 82 - jepsen.util 442 :ok :read [27 8] INFO [2022-03-13 17:07:49,174] jepsen worker 82 - jepsen.util 442 :invoke :read [27 nil] INFO [2022-03-13 17:07:49,178] jepsen worker 44 - jepsen.util 134 :info :write [14 9] :SyncWriteAmbiguous INFO [2022-03-13 17:07:49,179] jepsen worker 44 - jepsen.util 224 :invoke :write [14 35] INFO [2022-03-13 17:07:49,180] jepsen worker 21 - jepsen.util 291 :info :write [7 9] :RequestTimeoutException INFO [2022-03-13 17:07:49,181] jepsen worker 21 - jepsen.util 381 :invoke :read [7 nil] INFO [2022-03-13 17:07:49,215] jepsen worker 70 - jepsen.util 250 :info :write [23 32] :RequestTimeoutException INFO [2022-03-13 17:07:49,216] jepsen worker 70 - jepsen.util 340 :invoke :write [23 31] INFO [2022-03-13 17:07:49,219] jepsen worker 7 - jepsen.util 187 :ok :read [2 5] INFO [2022-03-13 17:07:49,219] jepsen worker 52 - jepsen.util 232 :ok :read [17 20] INFO [2022-03-13 17:07:49,220] jepsen worker 7 - jepsen.util 187 :invoke :read [2 nil] INFO [2022-03-13 17:07:49,220] jepsen worker 52 - jepsen.util 232 :invoke :read [17 nil] INFO [2022-03-13 17:07:49,224] jepsen worker 12 - jepsen.util 192 :info :write [4 8] :RequestTimeoutException INFO [2022-03-13 17:07:49,225] jepsen worker 12 - jepsen.util 282 :invoke :read [4 nil] INFO [2022-03-13 17:07:49,334] jepsen worker 40 - jepsen.util 400 :ok :read [13 :nil] INFO [2022-03-13 17:07:49,335] jepsen worker 40 - jepsen.util 400 :invoke :write [13 39] INFO [2022-03-13 17:07:49,370] jepsen worker 82 - jepsen.util 442 :ok :read [27 8] INFO [2022-03-13 17:07:49,371] jepsen worker 82 - jepsen.util 442 :invoke :read [27 nil] INFO [2022-03-13 17:07:49,391] jepsen worker 52 - jepsen.util 232 :ok :read [17 20] INFO [2022-03-13 17:07:49,391] jepsen worker 52 - jepsen.util 232 :invoke :write [17 4] INFO [2022-03-13 17:07:49,430] jepsen worker 23 - jepsen.util 113 :info :write [7 18] :RequestTimeoutException INFO [2022-03-13 17:07:49,431] jepsen worker 23 - jepsen.util 203 :invoke :read [7 nil] INFO [2022-03-13 17:07:49,487] jepsen worker 82 - jepsen.util 442 :ok :read [27 8] INFO [2022-03-13 17:07:49,488] jepsen worker 82 - jepsen.util 442 :invoke :write [27 4] INFO [2022-03-13 17:07:49,531] jepsen worker 22 - jepsen.util 202 :info :write [7 24] :RequestTimeoutException INFO [2022-03-13 17:07:49,532] jepsen worker 22 - jepsen.util 292 :invoke :read [7 nil] INFO [2022-03-13 17:07:49,537] jepsen worker 4 - jepsen.util 364 :info :write [1 31] :RequestTimeoutException INFO [2022-03-13 17:07:49,537] jepsen worker 4 - jepsen.util 454 :invoke :read [1 nil] INFO [2022-03-13 17:07:49,566] jepsen worker 23 - jepsen.util 203 :ok :read [7 :nil] INFO [2022-03-13 17:07:49,567] jepsen worker 23 - jepsen.util 203 :invoke :read [7 nil] INFO [2022-03-13 17:07:49,709] jepsen worker 23 - jepsen.util 203 :ok :read [7 :nil] INFO [2022-03-13 17:07:49,710] jepsen worker 23 - jepsen.util 203 :invoke :write [7 19] INFO [2022-03-13 17:07:49,710] jepsen worker 84 - jepsen.util 444 :info :write [28 0] :SyncWriteAmbiguous INFO [2022-03-13 17:07:49,711] jepsen worker 84 - jepsen.util 534 :invoke :write [28 9] INFO [2022-03-13 17:07:49,721] jepsen worker 55 - jepsen.util 325 :info :write [18 26] :RequestTimeoutException INFO [2022-03-13 17:07:49,721] jepsen worker 55 - jepsen.util 415 :invoke :read [18 nil] INFO [2022-03-13 17:07:49,799] jepsen worker 62 - jepsen.util 242 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:49,799] jepsen worker 62 - jepsen.util 242 :invoke :write [20 3] INFO [2022-03-13 17:07:49,812] jepsen worker 53 - jepsen.util 233 :ok :read [17 20] INFO [2022-03-13 17:07:49,813] jepsen worker 53 - jepsen.util 233 :invoke :read [17 nil] INFO [2022-03-13 17:07:49,834] jepsen worker 88 - jepsen.util 448 :ok :read [29 17] INFO [2022-03-13 17:07:49,835] jepsen worker 88 - jepsen.util 448 :invoke :read [29 nil] INFO [2022-03-13 17:07:49,880] jepsen worker 1 - jepsen.util 361 :ok :read [0 41] INFO [2022-03-13 17:07:49,881] jepsen worker 1 - jepsen.util 361 :invoke :read [0 nil] INFO [2022-03-13 17:07:49,922] jepsen worker 88 - jepsen.util 448 :ok :read [29 17] INFO [2022-03-13 17:07:49,923] jepsen worker 88 - jepsen.util 448 :invoke :write [29 12] INFO [2022-03-13 17:07:50,010] jepsen worker 1 - jepsen.util 361 :ok :read [0 41] INFO [2022-03-13 17:07:50,010] jepsen worker 1 - jepsen.util 361 :invoke :read [0 nil] INFO [2022-03-13 17:07:50,044] jepsen worker 37 - jepsen.util 307 :info :write [12 48] :RequestTimeoutException INFO [2022-03-13 17:07:50,045] jepsen worker 37 - jepsen.util 397 :invoke :read [12 nil] INFO [2022-03-13 17:07:50,104] jepsen worker 55 - jepsen.util 415 :ok :read [18 12] INFO [2022-03-13 17:07:50,105] jepsen worker 55 - jepsen.util 415 :invoke :read [18 nil] INFO [2022-03-13 17:07:50,178] jepsen worker 17 - jepsen.util 377 :info :write [5 13] :RequestTimeoutException INFO [2022-03-13 17:07:50,179] jepsen worker 17 - jepsen.util 467 :invoke :read [5 nil] INFO [2022-03-13 17:07:50,194] jepsen worker 1 - jepsen.util 361 :ok :read [0 41] INFO [2022-03-13 17:07:50,195] jepsen worker 1 - jepsen.util 361 :invoke :write [0 8] INFO [2022-03-13 17:07:50,200] jepsen worker 41 - jepsen.util 401 :info :write [13 16] :RequestTimeoutException INFO [2022-03-13 17:07:50,201] jepsen worker 41 - jepsen.util 491 :invoke :write [13 46] INFO [2022-03-13 17:07:50,240] jepsen worker 24 - jepsen.util 114 :info :write [8 37] :RequestTimeoutException INFO [2022-03-13 17:07:50,247] jepsen worker 24 - jepsen.util 204 :invoke :read [8 nil] INFO [2022-03-13 17:07:50,353] jepsen worker 17 - jepsen.util 467 :ok :read [5 32] INFO [2022-03-13 17:07:50,354] jepsen worker 17 - jepsen.util 467 :invoke :write [5 9] INFO [2022-03-13 17:07:50,446] jepsen worker 55 - jepsen.util 415 :ok :read [18 12] INFO [2022-03-13 17:07:50,447] jepsen worker 14 - jepsen.util 14 :info :write [4 40] :RequestTimeoutException INFO [2022-03-13 17:07:50,447] jepsen worker 13 - jepsen.util 103 :info :write [4 14] :RequestTimeoutException INFO [2022-03-13 17:07:50,447] jepsen worker 55 - jepsen.util 415 :invoke :write [18 10] INFO [2022-03-13 17:07:50,449] jepsen worker 14 - jepsen.util 104 :invoke :read [4 nil] INFO [2022-03-13 17:07:50,449] jepsen worker 13 - jepsen.util 193 :invoke :read [4 nil] INFO [2022-03-13 17:07:50,486] jepsen worker 50 - jepsen.util 320 :info :write [16 10] :RequestTimeoutException INFO [2022-03-13 17:07:50,487] jepsen worker 50 - jepsen.util 410 :invoke :write [16 18] INFO [2022-03-13 17:07:50,520] jepsen worker 25 - jepsen.util 205 :info :write [8 29] :RequestTimeoutException INFO [2022-03-13 17:07:50,521] jepsen worker 25 - jepsen.util 295 :invoke :read [8 nil] INFO [2022-03-13 17:07:50,557] jepsen worker 39 - jepsen.util 399 :info :write [13 18] :RequestTimeoutException INFO [2022-03-13 17:07:50,558] jepsen worker 39 - jepsen.util 489 :invoke :write [13 20] INFO [2022-03-13 17:07:50,641] jepsen worker 25 - jepsen.util 295 :ok :read [8 :nil] INFO [2022-03-13 17:07:50,641] jepsen worker 25 - jepsen.util 295 :invoke :write [8 16] INFO [2022-03-13 17:07:50,646] jepsen worker 47 - jepsen.util 407 :info :write [15 41] :RequestTimeoutException INFO [2022-03-13 17:07:50,647] jepsen worker 47 - jepsen.util 497 :invoke :read [15 nil] INFO [2022-03-13 17:07:50,761] jepsen worker 61 - jepsen.util 241 :info :write [20 5] :RequestTimeoutException INFO [2022-03-13 17:07:50,762] jepsen worker 61 - jepsen.util 331 :invoke :write [20 20] INFO [2022-03-13 17:07:50,908] jepsen worker 21 - jepsen.util 381 :ok :read [7 :nil] INFO [2022-03-13 17:07:50,908] jepsen worker 21 - jepsen.util 381 :invoke :write [7 46] INFO [2022-03-13 17:07:50,934] jepsen worker 7 - jepsen.util 187 :ok :read [2 5] INFO [2022-03-13 17:07:50,935] jepsen worker 7 - jepsen.util 187 :invoke :write [2 17] INFO [2022-03-13 17:07:50,966] jepsen worker 53 - jepsen.util 233 :ok :read [17 20] INFO [2022-03-13 17:07:50,967] jepsen worker 53 - jepsen.util 233 :invoke :write [17 19] INFO [2022-03-13 17:07:51,020] jepsen worker 71 - jepsen.util 161 :ok :read [23 :nil] INFO [2022-03-13 17:07:51,020] jepsen worker 13 - jepsen.util 193 :ok :read [4 :nil] INFO [2022-03-13 17:07:51,020] jepsen worker 71 - jepsen.util 161 :invoke :read [23 nil] INFO [2022-03-13 17:07:51,021] jepsen worker 13 - jepsen.util 193 :invoke :read [4 nil] INFO [2022-03-13 17:07:51,054] jepsen worker 45 - jepsen.util 405 :info :write [15 37] :RequestTimeoutException INFO [2022-03-13 17:07:51,055] jepsen worker 45 - jepsen.util 495 :invoke :read [15 nil] INFO [2022-03-13 17:07:51,122] jepsen worker 71 - jepsen.util 161 :ok :read [23 :nil] INFO [2022-03-13 17:07:51,122] jepsen worker 12 - jepsen.util 282 :ok :read [4 :nil] INFO [2022-03-13 17:07:51,123] jepsen worker 71 - jepsen.util 161 :invoke :read [23 nil] INFO [2022-03-13 17:07:51,124] jepsen worker 12 - jepsen.util 282 :invoke :write [4 8] INFO [2022-03-13 17:07:51,376] jepsen worker 71 - jepsen.util 161 :ok :read [23 :nil] INFO [2022-03-13 17:07:51,377] jepsen worker 71 - jepsen.util 161 :invoke :read [23 nil] INFO [2022-03-13 17:07:51,547] jepsen worker 71 - jepsen.util 161 :ok :read [23 :nil] INFO [2022-03-13 17:07:51,548] jepsen worker 71 - jepsen.util 161 :invoke :read [23 nil] INFO [2022-03-13 17:07:51,605] jepsen worker 45 - jepsen.util 495 :ok :read [15 48] INFO [2022-03-13 17:07:51,605] jepsen worker 45 - jepsen.util 495 :invoke :read [15 nil] INFO [2022-03-13 17:07:51,674] jepsen worker 13 - jepsen.util 193 :ok :read [4 :nil] INFO [2022-03-13 17:07:51,675] jepsen worker 13 - jepsen.util 193 :invoke :write [4 16] INFO [2022-03-13 17:07:51,713] jepsen worker 45 - jepsen.util 495 :ok :read [15 48] INFO [2022-03-13 17:07:51,714] jepsen worker 45 - jepsen.util 495 :invoke :write [15 43] INFO [2022-03-13 17:07:51,800] jepsen worker 71 - jepsen.util 161 :ok :read [23 :nil] INFO [2022-03-13 17:07:51,801] jepsen worker 71 - jepsen.util 161 :invoke :read [23 nil] INFO [2022-03-13 17:07:51,944] jepsen worker 71 - jepsen.util 161 :ok :read [23 :nil] INFO [2022-03-13 17:07:51,945] jepsen worker 71 - jepsen.util 161 :invoke :read [23 nil] INFO [2022-03-13 17:07:52,065] jepsen worker 71 - jepsen.util 161 :ok :read [23 :nil] INFO [2022-03-13 17:07:52,066] jepsen worker 71 - jepsen.util 161 :invoke :write [23 30] INFO [2022-03-13 17:07:52,231] jepsen worker 35 - jepsen.util 395 :info :write [11 38] :RequestTimeoutException INFO [2022-03-13 17:07:52,231] jepsen worker 35 - jepsen.util 485 :invoke :read [11 nil] INFO [2022-03-13 17:07:52,419] jepsen worker 35 - jepsen.util 485 :ok :read [11 :nil] INFO [2022-03-13 17:07:52,420] jepsen worker 35 - jepsen.util 485 :invoke :read [11 nil] INFO [2022-03-13 17:07:52,676] jepsen worker 51 - jepsen.util 51 :info :write [17 8] :RequestTimeoutException INFO [2022-03-13 17:07:52,677] jepsen worker 51 - jepsen.util 141 :invoke :read [17 nil] INFO [2022-03-13 17:07:52,708] jepsen worker 51 - jepsen.util 141 :ok :read [17 20] INFO [2022-03-13 17:07:52,708] jepsen worker 51 - jepsen.util 141 :invoke :read [17 nil] INFO [2022-03-13 17:07:52,751] jepsen worker 51 - jepsen.util 141 :ok :read [17 20] INFO [2022-03-13 17:07:52,751] jepsen worker 51 - jepsen.util 141 :invoke :write [17 21] INFO [2022-03-13 17:07:52,768] jepsen worker 20 - jepsen.util 380 :info :write [6 1] :RequestTimeoutException INFO [2022-03-13 17:07:52,769] jepsen worker 20 - jepsen.util 470 :invoke :write [6 43] INFO [2022-03-13 17:07:52,888] jepsen worker 28 - jepsen.util 388 :info :write [9 19] :RequestTimeoutException INFO [2022-03-13 17:07:52,889] jepsen worker 28 - jepsen.util 478 :invoke :read [9 nil] INFO [2022-03-13 17:07:52,988] jepsen worker 30 - jepsen.util 300 :info :write [10 2] :RequestTimeoutException INFO [2022-03-13 17:07:52,989] jepsen worker 30 - jepsen.util 390 :invoke :read [10 nil] INFO [2022-03-13 17:07:53,157] jepsen worker 15 - jepsen.util 375 :info :write [5 15] :SyncWriteAmbiguous INFO [2022-03-13 17:07:53,158] jepsen worker 15 - jepsen.util 465 :invoke :write [5 29] INFO [2022-03-13 17:07:53,182] jepsen worker 58 - jepsen.util 418 :ok :read [19 35] INFO [2022-03-13 17:07:53,183] jepsen worker 58 - jepsen.util 418 :invoke :write [19 44] INFO [2022-03-13 17:07:53,194] jepsen worker 19 - jepsen.util 289 :ok :read [6 :nil] INFO [2022-03-13 17:07:53,195] jepsen worker 19 - jepsen.util 289 :invoke :write [6 42] INFO [2022-03-13 17:07:53,442] jepsen worker 48 - jepsen.util 318 :info :write [16 46] :SyncWriteAmbiguous INFO [2022-03-13 17:07:53,443] jepsen worker 48 - jepsen.util 408 :invoke :read [16 nil] INFO [2022-03-13 17:07:53,518] jepsen worker 87 - jepsen.util 447 :info :write [29 11] :SyncWriteAmbiguous INFO [2022-03-13 17:07:53,519] jepsen worker 87 - jepsen.util 537 :invoke :read [29 nil] INFO [2022-03-13 17:07:53,547] jepsen worker 87 - jepsen.util 537 :ok :read [29 17] INFO [2022-03-13 17:07:53,548] jepsen worker 87 - jepsen.util 537 :invoke :write [29 42] INFO [2022-03-13 17:07:53,563] jepsen worker 49 - jepsen.util 409 :ok :read [16 :nil] INFO [2022-03-13 17:07:53,563] jepsen worker 49 - jepsen.util 409 :invoke :read [16 nil] INFO [2022-03-13 17:07:53,611] jepsen worker 16 - jepsen.util 286 :info :write [5 40] :RequestTimeoutException INFO [2022-03-13 17:07:53,612] jepsen worker 16 - jepsen.util 376 :invoke :write [5 41] INFO [2022-03-13 17:07:53,628] jepsen worker 59 - jepsen.util 509 :ok :read [19 35] INFO [2022-03-13 17:07:53,628] jepsen worker 59 - jepsen.util 509 :invoke :write [19 41] INFO [2022-03-13 17:07:53,652] jepsen worker 89 - jepsen.util 359 :ok :read [29 17] INFO [2022-03-13 17:07:53,653] jepsen worker 89 - jepsen.util 359 :invoke :read [29 nil] INFO [2022-03-13 17:07:53,677] jepsen worker 63 - jepsen.util 333 :info :write [21 45] :RequestTimeoutException INFO [2022-03-13 17:07:53,677] jepsen worker 72 - jepsen.util 342 :info :write [24 25] :RequestTimeoutException INFO [2022-03-13 17:07:53,679] jepsen worker 63 - jepsen.util 423 :invoke :write [21 0] INFO [2022-03-13 17:07:53,679] jepsen worker 72 - jepsen.util 432 :invoke :write [24 0] INFO [2022-03-13 17:07:53,705] jepsen worker 49 - jepsen.util 409 :ok :read [16 :nil] INFO [2022-03-13 17:07:53,706] jepsen worker 49 - jepsen.util 409 :invoke :write [16 4] INFO [2022-03-13 17:07:53,744] jepsen worker 89 - jepsen.util 359 :ok :read [29 17] INFO [2022-03-13 17:07:53,745] jepsen worker 89 - jepsen.util 359 :invoke :write [29 22] INFO [2022-03-13 17:07:53,749] jepsen worker 4 - jepsen.util 454 :ok :read [1 28] INFO [2022-03-13 17:07:53,750] jepsen worker 4 - jepsen.util 454 :invoke :read [1 nil] INFO [2022-03-13 17:07:53,786] jepsen worker 35 - jepsen.util 485 :ok :read [11 :nil] INFO [2022-03-13 17:07:53,787] jepsen worker 35 - jepsen.util 485 :invoke :read [11 nil] INFO [2022-03-13 17:07:53,798] jepsen worker 14 - jepsen.util 104 :ok :read [4 :nil] INFO [2022-03-13 17:07:53,798] jepsen worker 14 - jepsen.util 104 :invoke :read [4 nil] INFO [2022-03-13 17:07:53,835] jepsen worker 4 - jepsen.util 454 :ok :read [1 28] INFO [2022-03-13 17:07:53,836] jepsen worker 4 - jepsen.util 454 :invoke :read [1 nil] INFO [2022-03-13 17:07:53,864] jepsen worker 14 - jepsen.util 104 :ok :read [4 :nil] INFO [2022-03-13 17:07:53,865] jepsen worker 14 - jepsen.util 104 :invoke :write [4 27] INFO [2022-03-13 17:07:53,883] jepsen worker 86 - jepsen.util 356 :info :write [28 16] :RequestTimeoutException INFO [2022-03-13 17:07:53,884] jepsen worker 86 - jepsen.util 446 :invoke :read [28 nil] INFO [2022-03-13 17:07:53,904] jepsen worker 4 - jepsen.util 454 :ok :read [1 28] INFO [2022-03-13 17:07:53,905] jepsen worker 4 - jepsen.util 454 :invoke :read [1 nil] INFO [2022-03-13 17:07:53,914] jepsen worker 22 - jepsen.util 292 :ok :read [7 :nil] INFO [2022-03-13 17:07:53,915] jepsen worker 22 - jepsen.util 292 :invoke :write [7 22] INFO [2022-03-13 17:07:54,022] jepsen worker 4 - jepsen.util 454 :ok :read [1 28] INFO [2022-03-13 17:07:54,023] jepsen worker 4 - jepsen.util 454 :invoke :read [1 nil] INFO [2022-03-13 17:07:54,036] jepsen worker 86 - jepsen.util 446 :ok :read [28 25] INFO [2022-03-13 17:07:54,036] jepsen worker 86 - jepsen.util 446 :invoke :read [28 nil] INFO [2022-03-13 17:07:54,091] jepsen worker 4 - jepsen.util 454 :ok :read [1 28] INFO [2022-03-13 17:07:54,092] jepsen worker 4 - jepsen.util 454 :invoke :write [1 16] INFO [2022-03-13 17:07:54,137] jepsen worker 86 - jepsen.util 446 :ok :read [28 25] INFO [2022-03-13 17:07:54,138] jepsen worker 86 - jepsen.util 446 :invoke :write [28 13] INFO [2022-03-13 17:07:54,178] jepsen worker 40 - jepsen.util 400 :ok :write [13 39] INFO [2022-03-13 17:07:54,179] jepsen worker 40 - jepsen.util 400 :invoke :write [13 1] INFO [2022-03-13 17:07:54,310] jepsen worker 81 - jepsen.util 441 :info :write [27 36] :RequestTimeoutException INFO [2022-03-13 17:07:54,312] jepsen worker 81 - jepsen.util 531 :invoke :write [27 31] INFO [2022-03-13 17:07:54,677] jepsen worker 18 - jepsen.util 288 :info :write [6 29] :RequestTimeoutException INFO [2022-03-13 17:07:54,677] jepsen worker 46 - jepsen.util 316 :info :write [15 49] :RequestTimeoutException INFO [2022-03-13 17:07:54,679] jepsen worker 46 - jepsen.util 406 :invoke :read [15 nil] INFO [2022-03-13 17:07:54,679] jepsen worker 18 - jepsen.util 378 :invoke :read [6 nil] INFO [2022-03-13 17:07:54,706] jepsen worker 27 - jepsen.util 297 :info :write [9 9] :RequestTimeoutException INFO [2022-03-13 17:07:54,707] jepsen worker 27 - jepsen.util 387 :invoke :write [9 26] INFO [2022-03-13 17:07:54,807] jepsen worker 5 - jepsen.util 275 :info :write [1 7] :RequestTimeoutException INFO [2022-03-13 17:07:54,808] jepsen worker 5 - jepsen.util 365 :invoke :write [1 45] INFO [2022-03-13 17:07:54,827] jepsen worker 46 - jepsen.util 406 :ok :read [15 48] INFO [2022-03-13 17:07:54,827] jepsen worker 46 - jepsen.util 406 :invoke :read [15 nil] INFO [2022-03-13 17:07:54,855] jepsen worker 46 - jepsen.util 406 :ok :read [15 48] INFO [2022-03-13 17:07:54,855] jepsen worker 46 - jepsen.util 406 :invoke :read [15 nil] INFO [2022-03-13 17:07:54,886] jepsen worker 46 - jepsen.util 406 :ok :read [15 48] INFO [2022-03-13 17:07:54,887] jepsen worker 46 - jepsen.util 406 :invoke :read [15 nil] INFO [2022-03-13 17:07:54,907] jepsen worker 18 - jepsen.util 378 :ok :read [6 :nil] INFO [2022-03-13 17:07:54,908] jepsen worker 18 - jepsen.util 378 :invoke :write [6 11] INFO [2022-03-13 17:07:54,917] jepsen worker 65 - jepsen.util 335 :info :write [21 41] :RequestTimeoutException INFO [2022-03-13 17:07:54,917] jepsen worker 65 - jepsen.util 425 :invoke :read [21 nil] INFO [2022-03-13 17:07:54,936] jepsen worker 46 - jepsen.util 406 :ok :read [15 48] INFO [2022-03-13 17:07:54,937] jepsen worker 46 - jepsen.util 406 :invoke :write [15 42] INFO [2022-03-13 17:07:55,016] jepsen worker 73 - jepsen.util 433 :info :write [24 33] :RequestTimeoutException INFO [2022-03-13 17:07:55,017] jepsen worker 73 - jepsen.util 523 :invoke :write [24 36] INFO [2022-03-13 17:07:55,046] jepsen worker 10 - jepsen.util 190 :info :write [3 33] :RequestTimeoutException INFO [2022-03-13 17:07:55,047] jepsen worker 10 - jepsen.util 280 :invoke :write [3 12] WARN [2022-03-13 17:07:55,226] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9962948,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18498532,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9668192,"encode_duration_us":19810,"operation_id":"0xdf2","timeout_ms":10000,"last_server_duration_us":9378630,"total_duration_us":10000840}],"total_count":1}} WARN [2022-03-13 17:07:55,227] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":242517,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3063449,"total_server_duration_us":963667,"encode_duration_us":19672,"operation_id":"0x1091","timeout_ms":10000,"last_server_duration_us":52301,"total_duration_us":10009499},{"operation_name":"upsert","last_dispatch_duration_us":46946,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1304227,"total_server_duration_us":313063,"encode_duration_us":24494,"operation_id":"0x1124","timeout_ms":10000,"last_server_duration_us":25101,"total_duration_us":10008677},{"operation_name":"upsert","last_dispatch_duration_us":54144,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1147676,"total_server_duration_us":312983,"encode_duration_us":17712,"operation_id":"0x112a","timeout_ms":10000,"last_server_duration_us":30539,"total_duration_us":10007990},{"operation_name":"upsert","last_dispatch_duration_us":10341041,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13419087,"total_server_duration_us":197318,"encode_duration_us":17488,"operation_id":"0x1086","timeout_ms":10000,"last_server_duration_us":102959,"total_duration_us":10006438},{"operation_name":"upsert","last_dispatch_duration_us":11926133,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":17414503,"total_server_duration_us":436749,"encode_duration_us":17761,"operation_id":"0xfb0","timeout_ms":10000,"last_server_duration_us":38274,"total_duration_us":10005901},{"operation_name":"upsert","last_dispatch_duration_us":177329,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2758032,"total_server_duration_us":1156830,"encode_duration_us":22163,"operation_id":"0x10b9","timeout_ms":10000,"last_server_duration_us":43727,"total_duration_us":10004107},{"operation_name":"upsert","last_dispatch_duration_us":7726510,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11168278,"total_server_duration_us":231909,"encode_duration_us":20279,"operation_id":"0x1021","timeout_ms":10000,"last_server_duration_us":50068,"total_duration_us":10003788},{"operation_name":"upsert","last_dispatch_duration_us":2941113,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":5742540,"total_server_duration_us":784875,"encode_duration_us":29039,"operation_id":"0xfaa","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10002507},{"operation_name":"upsert","last_dispatch_duration_us":93532,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1382074,"total_server_duration_us":337701,"encode_duration_us":21190,"operation_id":"0x110f","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10002147},{"operation_name":"upsert","last_dispatch_duration_us":9240086,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9752112,"total_server_duration_us":9195093,"encode_duration_us":25823,"operation_id":"0x111b","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":9811600}],"total_count":13}} INFO [2022-03-13 17:07:55,241] jepsen worker 68 - jepsen.util 428 :info :write [22 24] :RequestTimeoutException INFO [2022-03-13 17:07:55,247] jepsen worker 68 - jepsen.util 518 :invoke :read [22 nil] INFO [2022-03-13 17:07:55,336] jepsen worker 68 - jepsen.util 518 :ok :read [22 46] INFO [2022-03-13 17:07:55,337] jepsen worker 68 - jepsen.util 518 :invoke :write [22 4] INFO [2022-03-13 17:07:55,814] jepsen worker 85 - jepsen.util 445 :info :write [28 46] :RequestTimeoutException INFO [2022-03-13 17:07:55,815] jepsen worker 85 - jepsen.util 535 :invoke :write [28 37] INFO [2022-03-13 17:07:55,951] jepsen worker 66 - jepsen.util 336 :info :write [22 35] :RequestTimeoutException INFO [2022-03-13 17:07:55,951] jepsen worker 66 - jepsen.util 426 :invoke :write [22 30] INFO [2022-03-13 17:07:56,072] jepsen worker 35 - jepsen.util 485 :ok :read [11 :nil] INFO [2022-03-13 17:07:56,073] jepsen worker 35 - jepsen.util 485 :invoke :write [11 4] INFO [2022-03-13 17:07:56,098] jepsen worker 0 - jepsen.util 360 :info :write [0 47] :RequestTimeoutException INFO [2022-03-13 17:07:56,099] jepsen worker 0 - jepsen.util 450 :invoke :read [0 nil] INFO [2022-03-13 17:07:56,292] jepsen worker 19 - jepsen.util 289 :fail :write [6 42] :ServerOutOfMemoryException INFO [2022-03-13 17:07:56,293] jepsen worker 19 - jepsen.util 289 :invoke :write [6 30] INFO [2022-03-13 17:07:56,300] jepsen worker 32 - jepsen.util 302 :info :write [10 28] :RequestTimeoutException INFO [2022-03-13 17:07:56,301] jepsen worker 32 - jepsen.util 392 :invoke :read [10 nil] INFO [2022-03-13 17:07:56,606] jepsen worker 57 - jepsen.util 417 :info :write [19 17] :RequestTimeoutException INFO [2022-03-13 17:07:56,607] jepsen worker 57 - jepsen.util 507 :invoke :read [19 nil] INFO [2022-03-13 17:07:56,644] jepsen worker 56 - jepsen.util 326 :info :write [18 3] :RequestTimeoutException INFO [2022-03-13 17:07:56,645] jepsen worker 56 - jepsen.util 416 :invoke :write [18 22] WARN [2022-03-13 17:07:56,728] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10262722,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10262722,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":9127773,"encode_duration_us":19216,"operation_id":"0xff8","timeout_ms":10000,"last_server_duration_us":9127773,"total_duration_us":10005460},{"operation_name":"upsert","last_dispatch_duration_us":9905205,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13900850,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":5982994,"encode_duration_us":19440,"operation_id":"0xe6b","timeout_ms":10000,"last_server_duration_us":5868358,"total_duration_us":10006846},{"operation_name":"upsert","last_dispatch_duration_us":14154662,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14154662,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9064980,"encode_duration_us":24399,"operation_id":"0x105b","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10006511},{"operation_name":"upsert","last_dispatch_duration_us":13284647,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18238506,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9409417,"encode_duration_us":20221,"operation_id":"0xede","timeout_ms":10000,"last_server_duration_us":9296838,"total_duration_us":10009981}],"total_count":4}} INFO [2022-03-13 17:07:56,796] jepsen worker 3 - jepsen.util 363 :info :write [1 25] :RequestTimeoutException INFO [2022-03-13 17:07:56,797] jepsen worker 3 - jepsen.util 453 :invoke :read [1 nil] INFO [2022-03-13 17:07:56,820] jepsen worker 3 - jepsen.util 453 :ok :read [1 28] INFO [2022-03-13 17:07:56,821] jepsen worker 3 - jepsen.util 453 :invoke :read [1 nil] WARN [2022-03-13 17:07:56,822] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":152584,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8975755,"total_server_duration_us":241180,"encode_duration_us":17986,"operation_id":"0x10ac","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10010194},{"operation_name":"upsert","last_dispatch_duration_us":149478,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8853505,"total_server_duration_us":258703,"encode_duration_us":20223,"operation_id":"0x1099","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10009902},{"operation_name":"upsert","last_dispatch_duration_us":30119,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":4417827,"total_server_duration_us":297786,"encode_duration_us":30028,"operation_id":"0x1181","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10009281},{"operation_name":"upsert","last_dispatch_duration_us":8032317,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":12391643,"total_server_duration_us":247495,"encode_duration_us":19420,"operation_id":"0xf7d","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10007897},{"operation_name":"upsert","last_dispatch_duration_us":519609,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7032497,"total_server_duration_us":925572,"encode_duration_us":23263,"operation_id":"0x100a","timeout_ms":10000,"last_server_duration_us":93043,"total_duration_us":10007585},{"operation_name":"upsert","last_dispatch_duration_us":35258,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5226393,"total_server_duration_us":964137,"encode_duration_us":20651,"operation_id":"0x113f","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10007451},{"operation_name":"upsert","last_dispatch_duration_us":1022802,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":4516527,"total_server_duration_us":1120226,"encode_duration_us":17791,"operation_id":"0x1118","timeout_ms":10000,"last_server_duration_us":743065,"total_duration_us":10006198},{"operation_name":"upsert","last_dispatch_duration_us":117174,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5677312,"total_server_duration_us":796914,"encode_duration_us":35648,"operation_id":"0x1146","timeout_ms":10000,"last_server_duration_us":88394,"total_duration_us":10006041},{"operation_name":"upsert","last_dispatch_duration_us":144107,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6422964,"total_server_duration_us":957700,"encode_duration_us":21949,"operation_id":"0x1140","timeout_ms":10000,"last_server_duration_us":57262,"total_duration_us":10005945},{"operation_name":"upsert","last_dispatch_duration_us":82772,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5028820,"total_server_duration_us":679063,"encode_duration_us":34894,"operation_id":"0x1141","timeout_ms":10000,"last_server_duration_us":40482,"total_duration_us":10004307}],"total_count":19}} WARN [2022-03-13 17:07:56,828] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":282818,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2445085,"total_server_duration_us":156278,"encode_duration_us":19820,"operation_id":"0x101b","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10010925},{"operation_name":"upsert","last_dispatch_duration_us":9636242,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":10901755,"total_server_duration_us":9264768,"encode_duration_us":17490,"operation_id":"0x10da","timeout_ms":10000,"last_server_duration_us":9207861,"total_duration_us":10010181},{"operation_name":"upsert","last_dispatch_duration_us":7653730,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9013054,"total_server_duration_us":99931,"encode_duration_us":17806,"operation_id":"0x1061","timeout_ms":10000,"last_server_duration_us":25976,"total_duration_us":10009751},{"operation_name":"upsert","last_dispatch_duration_us":348153,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2616473,"total_server_duration_us":193958,"encode_duration_us":30012,"operation_id":"0x1017","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10009403},{"operation_name":"upsert","last_dispatch_duration_us":300489,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2586328,"total_server_duration_us":196680,"encode_duration_us":22967,"operation_id":"0x101e","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10009064},{"operation_name":"upsert","last_dispatch_duration_us":955865,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":4514293,"total_server_duration_us":1456266,"encode_duration_us":23228,"operation_id":"0x1170","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10008347},{"operation_name":"upsert","last_dispatch_duration_us":320818,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1774434,"total_server_duration_us":125456,"encode_duration_us":21847,"operation_id":"0x1069","timeout_ms":10000,"last_server_duration_us":29230,"total_duration_us":10007385},{"operation_name":"upsert","last_dispatch_duration_us":9531817,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":11470965,"total_server_duration_us":9222590,"encode_duration_us":21380,"operation_id":"0x103c","timeout_ms":10000,"last_server_duration_us":9124576,"total_duration_us":10007192},{"operation_name":"upsert","last_dispatch_duration_us":13361917,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13361917,"total_server_duration_us":21977,"encode_duration_us":17480,"operation_id":"0x108e","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10006325},{"operation_name":"upsert","last_dispatch_duration_us":330080,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3477912,"total_server_duration_us":209574,"encode_duration_us":24282,"operation_id":"0x10a4","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10005326}],"total_count":19}} WARN [2022-03-13 17:07:56,828] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10258059,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":10258059,"total_server_duration_us":132218,"operation_id":"0x1042","timeout_ms":30000,"last_server_duration_us":132218,"total_duration_us":10260527},{"operation_name":"upsert","last_dispatch_duration_us":1378492,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5111088,"total_server_duration_us":910888,"encode_duration_us":17680,"operation_id":"0x10f4","timeout_ms":10000,"last_server_duration_us":70421,"total_duration_us":10009857},{"operation_name":"upsert","last_dispatch_duration_us":6181798,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10243952,"total_server_duration_us":1993753,"encode_duration_us":19331,"operation_id":"0x10f9","timeout_ms":10000,"last_server_duration_us":67089,"total_duration_us":10009519},{"operation_name":"upsert","last_dispatch_duration_us":14154662,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":14154662,"total_server_duration_us":9064980,"encode_duration_us":24399,"operation_id":"0x105b","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10006511},{"operation_name":"upsert","last_dispatch_duration_us":7406426,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11536670,"total_server_duration_us":921173,"encode_duration_us":17042,"operation_id":"0x10fa","timeout_ms":10000,"last_server_duration_us":31872,"total_duration_us":10006069},{"operation_name":"upsert","last_dispatch_duration_us":10262722,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":10262722,"total_server_duration_us":9127773,"encode_duration_us":19216,"operation_id":"0xff8","timeout_ms":10000,"last_server_duration_us":9127773,"total_duration_us":10005460},{"operation_name":"upsert","last_dispatch_duration_us":3307910,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8318361,"total_server_duration_us":221090,"encode_duration_us":20705,"operation_id":"0x108d","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10003105},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":18195,"operation_id":"0x116e","timeout_ms":10000,"total_duration_us":10001788},{"operation_name":"get","last_dispatch_duration_us":9351162,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9351162,"total_server_duration_us":43,"operation_id":"0x1077","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9351553},{"operation_name":"upsert","last_dispatch_duration_us":9197301,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9197301,"total_server_duration_us":9160840,"encode_duration_us":17567,"operation_id":"0x1129","timeout_ms":10000,"last_server_duration_us":9160840,"total_duration_us":9201847}],"total_count":19}} INFO [2022-03-13 17:07:56,845] jepsen worker 3 - jepsen.util 453 :ok :read [1 28] INFO [2022-03-13 17:07:56,846] jepsen worker 3 - jepsen.util 453 :invoke :write [1 35] INFO [2022-03-13 17:07:56,854] jepsen worker 36 - jepsen.util 306 :info :write [12 7] :RequestTimeoutException INFO [2022-03-13 17:07:56,855] jepsen worker 36 - jepsen.util 396 :invoke :read [12 nil] INFO [2022-03-13 17:07:56,928] jepsen worker 32 - jepsen.util 392 :ok :read [10 :nil] WARN [2022-03-13 17:07:56,928] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":147108,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":4857568,"total_server_duration_us":2649014,"encode_duration_us":17736,"operation_id":"0x1185","timeout_ms":10000,"last_server_duration_us":25450,"total_duration_us":10010984},{"operation_name":"upsert","last_dispatch_duration_us":3927804,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6989229,"total_server_duration_us":737271,"encode_duration_us":21194,"operation_id":"0x1066","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10010165},{"operation_name":"upsert","last_dispatch_duration_us":101466,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9440746,"total_server_duration_us":98936,"encode_duration_us":25029,"operation_id":"0x108b","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10010150},{"operation_name":"upsert","last_dispatch_duration_us":201382,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6160054,"total_server_duration_us":911943,"encode_duration_us":18856,"operation_id":"0xfbd","timeout_ms":10000,"last_server_duration_us":99814,"total_duration_us":10009070},{"operation_name":"upsert","last_dispatch_duration_us":818911,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":818911,"total_server_duration_us":27673,"encode_duration_us":17156,"operation_id":"0x116a","timeout_ms":10000,"last_server_duration_us":27673,"total_duration_us":10007867},{"operation_name":"upsert","last_dispatch_duration_us":4067759,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":8947659,"total_server_duration_us":1213604,"encode_duration_us":32332,"operation_id":"0x1047","timeout_ms":10000,"last_server_duration_us":32355,"total_duration_us":10007846},{"operation_name":"upsert","last_dispatch_duration_us":8779272,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8779272,"total_server_duration_us":21324,"encode_duration_us":29633,"operation_id":"0x105c","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10006324},{"operation_name":"upsert","last_dispatch_duration_us":2979055,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":8522892,"total_server_duration_us":939186,"encode_duration_us":18370,"operation_id":"0x1065","timeout_ms":10000,"last_server_duration_us":37036,"total_duration_us":10004012},{"operation_name":"upsert","last_dispatch_duration_us":730749,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":5585377,"total_server_duration_us":1689898,"encode_duration_us":23955,"operation_id":"0x10c2","timeout_ms":10000,"last_server_duration_us":676885,"total_duration_us":10002052},{"operation_name":"upsert","last_dispatch_duration_us":4427336,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11078545,"total_server_duration_us":89271,"encode_duration_us":17451,"operation_id":"0x1005","timeout_ms":10000,"last_server_duration_us":24928,"total_duration_us":10000959}],"total_count":19}} WARN [2022-03-13 17:07:56,928] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15952986,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15952986,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9052231,"encode_duration_us":34807,"operation_id":"0xfd4","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002692},{"operation_name":"upsert","last_dispatch_duration_us":9531817,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11470965,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":9222590,"encode_duration_us":21380,"operation_id":"0x103c","timeout_ms":10000,"last_server_duration_us":9124576,"total_duration_us":10007192},{"operation_name":"upsert","last_dispatch_duration_us":16740626,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20248128,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9382927,"encode_duration_us":17532,"operation_id":"0xda8","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10008053},{"operation_name":"upsert","last_dispatch_duration_us":9636242,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10901755,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":9264768,"encode_duration_us":17490,"operation_id":"0x10da","timeout_ms":10000,"last_server_duration_us":9207861,"total_duration_us":10010181}],"total_count":4}} INFO [2022-03-13 17:07:56,928] jepsen worker 73 - jepsen.util 523 :fail :write [24 36] :ServerOutOfMemoryException INFO [2022-03-13 17:07:56,929] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:56,930] jepsen worker 32 - jepsen.util 392 :invoke :write [10 33] WARN [2022-03-13 17:07:56,938] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":160465,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8533580,"total_server_duration_us":118798,"encode_duration_us":20738,"operation_id":"0x1145","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10010869},{"operation_name":"upsert","last_dispatch_duration_us":209570,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3791228,"total_server_duration_us":248153,"encode_duration_us":22045,"operation_id":"0x1057","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10010609},{"operation_name":"upsert","last_dispatch_duration_us":7992416,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12373542,"total_server_duration_us":1279813,"encode_duration_us":22857,"operation_id":"0x105e","timeout_ms":10000,"last_server_duration_us":79687,"total_duration_us":10010494},{"operation_name":"upsert","last_dispatch_duration_us":7713073,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10996489,"total_server_duration_us":226720,"encode_duration_us":24376,"operation_id":"0x1097","timeout_ms":10000,"last_server_duration_us":26864,"total_duration_us":10009911},{"operation_name":"upsert","last_dispatch_duration_us":7612299,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10604239,"total_server_duration_us":229945,"encode_duration_us":16794,"operation_id":"0x10f2","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10009710},{"operation_name":"upsert","last_dispatch_duration_us":7708139,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10580343,"total_server_duration_us":577868,"encode_duration_us":21650,"operation_id":"0x101c","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10009506},{"operation_name":"upsert","last_dispatch_duration_us":8166124,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":13327189,"total_server_duration_us":106054,"encode_duration_us":22499,"operation_id":"0x1079","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10008626},{"operation_name":"upsert","last_dispatch_duration_us":8208023,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9010803,"total_server_duration_us":8412174,"encode_duration_us":19909,"operation_id":"0xf92","timeout_ms":10000,"last_server_duration_us":8074371,"total_duration_us":10008512},{"operation_name":"upsert","last_dispatch_duration_us":331583,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2581012,"total_server_duration_us":187429,"encode_duration_us":31559,"operation_id":"0x101a","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10008017},{"operation_name":"upsert","last_dispatch_duration_us":7631122,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10818103,"total_server_duration_us":218589,"encode_duration_us":27388,"operation_id":"0x1098","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10007500}],"total_count":21}} INFO [2022-03-13 17:07:57,034] jepsen worker 31 - jepsen.util 391 :ok :read [10 :nil] INFO [2022-03-13 17:07:57,035] jepsen worker 31 - jepsen.util 391 :invoke :read [10 nil] WARN [2022-03-13 17:07:57,038] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9467841,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11303251,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9202536,"encode_duration_us":22513,"operation_id":"0x1019","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10003861},{"operation_name":"upsert","last_dispatch_duration_us":9462772,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13611209,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9995436,"encode_duration_us":18350,"operation_id":"0xf46","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":10004330},{"operation_name":"upsert","last_dispatch_duration_us":8208023,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9010803,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":8412174,"encode_duration_us":19909,"operation_id":"0xf92","timeout_ms":10000,"last_server_duration_us":8074371,"total_duration_us":10008512},{"operation_name":"upsert","last_dispatch_duration_us":9563641,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11707838,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9307027,"encode_duration_us":22893,"operation_id":"0x1053","timeout_ms":10000,"last_server_duration_us":9162975,"total_duration_us":10006887},{"operation_name":"upsert","last_dispatch_duration_us":9545656,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13859139,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":10205351,"encode_duration_us":21668,"operation_id":"0xf3f","timeout_ms":10000,"last_server_duration_us":9378630,"total_duration_us":10006621}],"total_count":5}} INFO [2022-03-13 17:07:57,067] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,068] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:57,096] jepsen worker 9 - jepsen.util 369 :info :write [3 21] :RequestTimeoutException INFO [2022-03-13 17:07:57,097] jepsen worker 9 - jepsen.util 459 :invoke :write [3 21] INFO [2022-03-13 17:07:57,164] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,164] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:57,188] jepsen worker 60 - jepsen.util 330 :info :write [20 20] :SyncWriteAmbiguous INFO [2022-03-13 17:07:57,189] jepsen worker 60 - jepsen.util 420 :invoke :read [20 nil] INFO [2022-03-13 17:07:57,193] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,194] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:57,239] jepsen worker 74 - jepsen.util 434 :info :write [24 24] :RequestTimeoutException INFO [2022-03-13 17:07:57,239] jepsen worker 74 - jepsen.util 524 :invoke :write [24 40] INFO [2022-03-13 17:07:57,241] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,242] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:57,243] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,244] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:57,301] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,302] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:57,304] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,305] jepsen worker 73 - jepsen.util 523 :invoke :read [24 nil] INFO [2022-03-13 17:07:57,307] jepsen worker 73 - jepsen.util 523 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,308] jepsen worker 73 - jepsen.util 523 :invoke :write [24 38] INFO [2022-03-13 17:07:57,317] jepsen worker 83 - jepsen.util 353 :ok :read [27 4] INFO [2022-03-13 17:07:57,318] jepsen worker 83 - jepsen.util 353 :invoke :write [27 34] INFO [2022-03-13 17:07:57,404] jepsen worker 36 - jepsen.util 396 :ok :read [12 46] INFO [2022-03-13 17:07:57,405] jepsen worker 36 - jepsen.util 396 :invoke :write [12 10] INFO [2022-03-13 17:07:57,408] jepsen worker 37 - jepsen.util 397 :ok :read [12 46] INFO [2022-03-13 17:07:57,409] jepsen worker 37 - jepsen.util 397 :invoke :write [12 25] INFO [2022-03-13 17:07:57,426] jepsen worker 78 - jepsen.util 258 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,428] jepsen worker 78 - jepsen.util 258 :invoke :write [26 21] INFO [2022-03-13 17:07:57,473] jepsen worker 48 - jepsen.util 408 :ok :read [16 :nil] INFO [2022-03-13 17:07:57,474] jepsen worker 48 - jepsen.util 408 :invoke :read [16 nil] INFO [2022-03-13 17:07:57,507] jepsen worker 62 - jepsen.util 242 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,507] jepsen worker 35 - jepsen.util 485 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,508] jepsen worker 35 - jepsen.util 485 :invoke :write [11 38] INFO [2022-03-13 17:07:57,513] jepsen worker 62 - jepsen.util 242 :invoke :write [20 13] INFO [2022-03-13 17:07:57,520] jepsen worker 11 - jepsen.util 281 :info :write [3 43] :RequestTimeoutException INFO [2022-03-13 17:07:57,522] jepsen worker 11 - jepsen.util 371 :invoke :write [3 11] INFO [2022-03-13 17:07:57,599] jepsen worker 60 - jepsen.util 420 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,600] jepsen worker 60 - jepsen.util 420 :invoke :read [20 nil] INFO [2022-03-13 17:07:57,674] jepsen worker 64 - jepsen.util 244 :info :write [21 10] :RequestTimeoutException INFO [2022-03-13 17:07:57,675] jepsen worker 64 - jepsen.util 334 :invoke :read [21 nil] INFO [2022-03-13 17:07:57,771] jepsen worker 28 - jepsen.util 478 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,772] jepsen worker 28 - jepsen.util 478 :invoke :write [9 34] INFO [2022-03-13 17:07:57,793] jepsen worker 47 - jepsen.util 497 :ok :read [15 48] INFO [2022-03-13 17:07:57,794] jepsen worker 47 - jepsen.util 497 :invoke :read [15 nil] INFO [2022-03-13 17:07:57,809] jepsen worker 60 - jepsen.util 420 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,810] jepsen worker 60 - jepsen.util 420 :invoke :read [20 nil] INFO [2022-03-13 17:07:57,856] jepsen worker 48 - jepsen.util 408 :ok :read [16 :nil] INFO [2022-03-13 17:07:57,856] jepsen worker 48 - jepsen.util 408 :invoke :read [16 nil] INFO [2022-03-13 17:07:57,926] jepsen worker 48 - jepsen.util 408 :ok :read [16 :nil] INFO [2022-03-13 17:07:57,927] jepsen worker 48 - jepsen.util 408 :invoke :read [16 nil] INFO [2022-03-13 17:07:57,990] jepsen worker 60 - jepsen.util 420 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:57,990] jepsen worker 60 - jepsen.util 420 :invoke :write [20 10] INFO [2022-03-13 17:07:57,997] jepsen worker 48 - jepsen.util 408 :ok :read [16 :nil] INFO [2022-03-13 17:07:57,998] jepsen worker 48 - jepsen.util 408 :invoke :write [16 33] INFO [2022-03-13 17:07:58,088] jepsen worker 2 - jepsen.util 452 :info :write [0 36] :RequestTimeoutException INFO [2022-03-13 17:07:58,089] jepsen worker 2 - jepsen.util 542 :invoke :write [0 49] INFO [2022-03-13 17:07:58,163] jepsen worker 43 - jepsen.util 223 :info :write [14 17] :RequestTimeoutException INFO [2022-03-13 17:07:58,164] jepsen worker 43 - jepsen.util 313 :invoke :read [14 nil] INFO [2022-03-13 17:07:58,291] jepsen worker 29 - jepsen.util 389 :info :write [9 0] :RequestTimeoutException INFO [2022-03-13 17:07:58,291] jepsen worker 29 - jepsen.util 479 :invoke :write [9 22] INFO [2022-03-13 17:07:58,347] jepsen worker 69 - jepsen.util 249 :info :write [23 30] :RequestTimeoutException INFO [2022-03-13 17:07:58,347] jepsen worker 69 - jepsen.util 339 :invoke :write [23 11] INFO [2022-03-13 17:07:58,564] jepsen worker 77 - jepsen.util 257 :info :write [25 41] :RequestTimeoutException INFO [2022-03-13 17:07:58,564] jepsen worker 79 - jepsen.util 439 :info :write [26 41] :RequestTimeoutException INFO [2022-03-13 17:07:58,564] jepsen worker 80 - jepsen.util 350 :info :write [26 41] :RequestTimeoutException INFO [2022-03-13 17:07:58,564] jepsen worker 77 - jepsen.util 347 :invoke :read [25 nil] INFO [2022-03-13 17:07:58,565] jepsen worker 80 - jepsen.util 440 :invoke :write [26 9] INFO [2022-03-13 17:07:58,566] jepsen worker 79 - jepsen.util 529 :invoke :read [26 nil] INFO [2022-03-13 17:07:58,571] jepsen worker 8 - jepsen.util 368 :info :write [2 17] :RequestTimeoutException INFO [2022-03-13 17:07:58,576] jepsen worker 44 - jepsen.util 224 :info :write [14 35] :SyncWriteAmbiguous INFO [2022-03-13 17:07:58,578] jepsen worker 8 - jepsen.util 458 :invoke :read [2 nil] INFO [2022-03-13 17:07:58,579] jepsen worker 44 - jepsen.util 314 :invoke :read [14 nil] INFO [2022-03-13 17:07:58,601] jepsen worker 26 - jepsen.util 206 :info :write [8 32] :RequestTimeoutException INFO [2022-03-13 17:07:58,602] jepsen worker 26 - jepsen.util 296 :invoke :read [8 nil] INFO [2022-03-13 17:07:58,679] jepsen worker 24 - jepsen.util 204 :ok :read [8 :nil] INFO [2022-03-13 17:07:58,680] jepsen worker 24 - jepsen.util 204 :invoke :write [8 45] INFO [2022-03-13 17:07:58,684] jepsen worker 33 - jepsen.util 303 :info :write [11 49] :RequestTimeoutException INFO [2022-03-13 17:07:58,685] jepsen worker 33 - jepsen.util 393 :invoke :read [11 nil] INFO [2022-03-13 17:07:58,686] jepsen worker 76 - jepsen.util 256 :info :write [25 37] :RequestTimeoutException INFO [2022-03-13 17:07:58,687] jepsen worker 76 - jepsen.util 346 :invoke :read [25 nil] INFO [2022-03-13 17:07:58,817] jepsen worker 6 - jepsen.util 276 :info :write [2 44] :RequestTimeoutException INFO [2022-03-13 17:07:58,818] jepsen worker 6 - jepsen.util 366 :invoke :read [2 nil] INFO [2022-03-13 17:07:58,851] jepsen worker 75 - jepsen.util 165 :info :write [25 23] :RequestTimeoutException INFO [2022-03-13 17:07:58,852] jepsen worker 75 - jepsen.util 255 :invoke :read [25 nil] INFO [2022-03-13 17:07:58,889] jepsen worker 34 - jepsen.util 214 :info :write [11 49] :RequestTimeoutException INFO [2022-03-13 17:07:58,889] jepsen worker 34 - jepsen.util 304 :invoke :read [11 nil] INFO [2022-03-13 17:07:58,951] jepsen worker 76 - jepsen.util 346 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:58,951] jepsen worker 76 - jepsen.util 346 :invoke :write [25 39] INFO [2022-03-13 17:07:58,960] jepsen worker 42 - jepsen.util 222 :info :write [14 43] :RequestTimeoutException INFO [2022-03-13 17:07:58,961] jepsen worker 42 - jepsen.util 312 :invoke :read [14 nil] INFO [2022-03-13 17:07:58,990] jepsen worker 79 - jepsen.util 529 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:58,991] jepsen worker 79 - jepsen.util 529 :invoke :read [26 nil] INFO [2022-03-13 17:07:59,004] jepsen worker 38 - jepsen.util 398 :info :write [12 5] :RequestTimeoutException INFO [2022-03-13 17:07:59,005] jepsen worker 38 - jepsen.util 488 :invoke :read [12 nil] INFO [2022-03-13 17:07:59,006] jepsen worker 67 - jepsen.util 427 :info :write [22 45] :RequestTimeoutException INFO [2022-03-13 17:07:59,007] jepsen worker 67 - jepsen.util 517 :invoke :read [22 nil] INFO [2022-03-13 17:07:59,046] jepsen worker 54 - jepsen.util 504 :info :write [18 40] :RequestTimeoutException INFO [2022-03-13 17:07:59,047] jepsen worker 54 - jepsen.util 594 :invoke :write [18 12] INFO [2022-03-13 17:07:59,141] jepsen worker 6 - jepsen.util 366 :ok :read [2 5] INFO [2022-03-13 17:07:59,142] jepsen worker 6 - jepsen.util 366 :invoke :write [2 16] INFO [2022-03-13 17:07:59,236] jepsen worker 79 - jepsen.util 529 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:59,237] jepsen worker 79 - jepsen.util 529 :invoke :read [26 nil] INFO [2022-03-13 17:07:59,240] jepsen worker 70 - jepsen.util 340 :info :write [23 31] :RequestTimeoutException INFO [2022-03-13 17:07:59,240] jepsen worker 70 - jepsen.util 430 :invoke :write [23 21] INFO [2022-03-13 17:07:59,417] jepsen worker 52 - jepsen.util 232 :info :write [17 4] :RequestTimeoutException INFO [2022-03-13 17:07:59,417] jepsen worker 52 - jepsen.util 322 :invoke :write [17 47] INFO [2022-03-13 17:07:59,452] jepsen worker 79 - jepsen.util 529 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:59,453] jepsen worker 79 - jepsen.util 529 :invoke :read [26 nil] INFO [2022-03-13 17:07:59,514] jepsen worker 82 - jepsen.util 442 :info :write [27 4] :RequestTimeoutException INFO [2022-03-13 17:07:59,515] jepsen worker 82 - jepsen.util 532 :invoke :write [27 45] INFO [2022-03-13 17:07:59,620] jepsen worker 79 - jepsen.util 529 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:07:59,621] jepsen worker 79 - jepsen.util 529 :invoke :write [26 43] INFO [2022-03-13 17:07:59,736] jepsen worker 23 - jepsen.util 203 :info :write [7 19] :RequestTimeoutException INFO [2022-03-13 17:07:59,737] jepsen worker 23 - jepsen.util 293 :invoke :read [7 nil] INFO [2022-03-13 17:07:59,748] jepsen worker 84 - jepsen.util 534 :info :write [28 9] :RequestTimeoutException INFO [2022-03-13 17:07:59,748] jepsen worker 84 - jepsen.util 624 :invoke :read [28 nil] INFO [2022-03-13 17:07:59,946] jepsen worker 88 - jepsen.util 448 :info :write [29 12] :RequestTimeoutException INFO [2022-03-13 17:07:59,947] jepsen worker 88 - jepsen.util 538 :invoke :write [29 2] INFO [2022-03-13 17:08:00,226] jepsen worker 41 - jepsen.util 491 :info :write [13 46] :RequestTimeoutException INFO [2022-03-13 17:08:00,226] jepsen worker 1 - jepsen.util 361 :info :write [0 8] :RequestTimeoutException INFO [2022-03-13 17:08:00,227] jepsen worker 41 - jepsen.util 581 :invoke :read [13 nil] INFO [2022-03-13 17:08:00,228] jepsen worker 1 - jepsen.util 451 :invoke :write [0 49] INFO [2022-03-13 17:08:00,377] jepsen worker 17 - jepsen.util 467 :info :write [5 9] :RequestTimeoutException INFO [2022-03-13 17:08:00,377] jepsen worker 17 - jepsen.util 557 :invoke :read [5 nil] INFO [2022-03-13 17:08:00,470] jepsen worker 55 - jepsen.util 415 :info :write [18 10] :RequestTimeoutException INFO [2022-03-13 17:08:00,471] jepsen worker 55 - jepsen.util 505 :invoke :read [18 nil] INFO [2022-03-13 17:08:00,514] jepsen worker 50 - jepsen.util 410 :info :write [16 18] :RequestTimeoutException INFO [2022-03-13 17:08:00,515] jepsen worker 50 - jepsen.util 500 :invoke :write [16 1] INFO [2022-03-13 17:08:00,586] jepsen worker 39 - jepsen.util 489 :info :write [13 20] :RequestTimeoutException INFO [2022-03-13 17:08:00,587] jepsen worker 39 - jepsen.util 579 :invoke :write [13 40] INFO [2022-03-13 17:08:00,667] jepsen worker 25 - jepsen.util 295 :info :write [8 16] :RequestTimeoutException INFO [2022-03-13 17:08:00,669] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:00,788] jepsen worker 61 - jepsen.util 331 :info :write [20 20] :RequestTimeoutException INFO [2022-03-13 17:08:00,789] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:00,793] jepsen worker 30 - jepsen.util 390 :ok :read [10 :nil] INFO [2022-03-13 17:08:00,794] jepsen worker 30 - jepsen.util 390 :invoke :write [10 6] INFO [2022-03-13 17:08:00,939] jepsen worker 21 - jepsen.util 381 :info :write [7 46] :RequestTimeoutException INFO [2022-03-13 17:08:00,939] jepsen worker 21 - jepsen.util 471 :invoke :read [7 nil] INFO [2022-03-13 17:08:00,958] jepsen worker 7 - jepsen.util 187 :info :write [2 17] :RequestTimeoutException INFO [2022-03-13 17:08:00,959] jepsen worker 7 - jepsen.util 277 :invoke :read [2 nil] INFO [2022-03-13 17:08:00,983] jepsen worker 31 - jepsen.util 391 :ok :read [10 :nil] INFO [2022-03-13 17:08:00,984] jepsen worker 64 - jepsen.util 334 :ok :read [21 :nil] INFO [2022-03-13 17:08:00,985] jepsen worker 31 - jepsen.util 391 :invoke :read [10 nil] INFO [2022-03-13 17:08:00,985] jepsen worker 64 - jepsen.util 334 :invoke :read [21 nil] INFO [2022-03-13 17:08:00,988] jepsen worker 53 - jepsen.util 233 :info :write [17 19] :RequestTimeoutException INFO [2022-03-13 17:08:00,989] jepsen worker 53 - jepsen.util 323 :invoke :write [17 7] INFO [2022-03-13 17:08:01,155] jepsen worker 12 - jepsen.util 282 :info :write [4 8] :RequestTimeoutException INFO [2022-03-13 17:08:01,155] jepsen worker 12 - jepsen.util 372 :invoke :write [4 36] INFO [2022-03-13 17:08:01,689] jepsen worker 47 - jepsen.util 497 :ok :read [15 48] INFO [2022-03-13 17:08:01,690] jepsen worker 47 - jepsen.util 497 :invoke :read [15 nil] INFO [2022-03-13 17:08:01,708] jepsen worker 13 - jepsen.util 193 :info :write [4 16] :RequestTimeoutException INFO [2022-03-13 17:08:01,709] jepsen worker 13 - jepsen.util 283 :invoke :write [4 3] INFO [2022-03-13 17:08:01,740] jepsen worker 45 - jepsen.util 495 :info :write [15 43] :RequestTimeoutException INFO [2022-03-13 17:08:01,741] jepsen worker 45 - jepsen.util 585 :invoke :write [15 3] INFO [2022-03-13 17:08:01,808] jepsen worker 55 - jepsen.util 505 :ok :read [18 40] INFO [2022-03-13 17:08:01,809] jepsen worker 55 - jepsen.util 505 :invoke :write [18 32] INFO [2022-03-13 17:08:01,837] jepsen worker 31 - jepsen.util 391 :ok :read [10 :nil] INFO [2022-03-13 17:08:01,837] jepsen worker 64 - jepsen.util 334 :ok :read [21 :nil] INFO [2022-03-13 17:08:01,838] jepsen worker 31 - jepsen.util 391 :invoke :write [10 3] INFO [2022-03-13 17:08:01,838] jepsen worker 64 - jepsen.util 334 :invoke :write [21 0] INFO [2022-03-13 17:08:02,094] jepsen worker 71 - jepsen.util 161 :info :write [23 30] :RequestTimeoutException INFO [2022-03-13 17:08:02,095] jepsen worker 47 - jepsen.util 497 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:02,097] jepsen worker 71 - jepsen.util 251 :invoke :write [23 40] INFO [2022-03-13 17:08:02,097] jepsen worker 47 - jepsen.util 497 :invoke :write [15 41] INFO [2022-03-13 17:08:02,240] jepsen worker 51 - jepsen.util 141 :info :write [17 21] :SyncWriteAmbiguous INFO [2022-03-13 17:08:02,242] jepsen worker 51 - jepsen.util 231 :invoke :read [17 nil] INFO [2022-03-13 17:08:02,763] jepsen worker 87 - jepsen.util 537 :info :write [29 42] :SyncWriteAmbiguous INFO [2022-03-13 17:08:02,764] jepsen worker 87 - jepsen.util 627 :invoke :read [29 nil] INFO [2022-03-13 17:08:02,797] jepsen worker 20 - jepsen.util 470 :info :write [6 43] :RequestTimeoutException INFO [2022-03-13 17:08:02,797] jepsen worker 20 - jepsen.util 560 :invoke :read [6 nil] INFO [2022-03-13 17:08:02,858] jepsen worker 87 - jepsen.util 627 :ok :read [29 17] INFO [2022-03-13 17:08:02,859] jepsen worker 87 - jepsen.util 627 :invoke :read [29 nil] INFO [2022-03-13 17:08:02,884] jepsen worker 87 - jepsen.util 627 :ok :read [29 17] INFO [2022-03-13 17:08:02,885] jepsen worker 87 - jepsen.util 627 :invoke :write [29 14] INFO [2022-03-13 17:08:02,888] jepsen worker 49 - jepsen.util 409 :info :write [16 4] :SyncWriteAmbiguous INFO [2022-03-13 17:08:02,889] jepsen worker 49 - jepsen.util 499 :invoke :read [16 nil] INFO [2022-03-13 17:08:02,916] jepsen worker 84 - jepsen.util 624 :ok :read [28 25] INFO [2022-03-13 17:08:02,917] jepsen worker 84 - jepsen.util 624 :invoke :write [28 19] INFO [2022-03-13 17:08:03,112] jepsen worker 14 - jepsen.util 104 :info :write [4 27] :SyncWriteAmbiguous INFO [2022-03-13 17:08:03,114] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:03,191] jepsen worker 15 - jepsen.util 465 :info :write [5 29] :RequestTimeoutException INFO [2022-03-13 17:08:03,191] jepsen worker 15 - jepsen.util 555 :invoke :read [5 nil] INFO [2022-03-13 17:08:03,207] jepsen worker 58 - jepsen.util 418 :info :write [19 44] :RequestTimeoutException INFO [2022-03-13 17:08:03,207] jepsen worker 58 - jepsen.util 508 :invoke :read [19 nil] INFO [2022-03-13 17:08:03,256] jepsen worker 67 - jepsen.util 517 :ok :read [22 46] INFO [2022-03-13 17:08:03,257] jepsen worker 67 - jepsen.util 517 :invoke :write [22 34] INFO [2022-03-13 17:08:03,269] jepsen worker 86 - jepsen.util 446 :info :write [28 13] :SyncWriteAmbiguous INFO [2022-03-13 17:08:03,270] jepsen worker 86 - jepsen.util 536 :invoke :write [28 12] INFO [2022-03-13 17:08:03,508] jepsen worker 65 - jepsen.util 425 :ok :read [21 :nil] INFO [2022-03-13 17:08:03,509] jepsen worker 65 - jepsen.util 425 :invoke :write [21 25] INFO [2022-03-13 17:08:03,573] jepsen worker 20 - jepsen.util 560 :ok :read [6 :nil] INFO [2022-03-13 17:08:03,573] jepsen worker 20 - jepsen.util 560 :invoke :read [6 nil] INFO [2022-03-13 17:08:03,636] jepsen worker 16 - jepsen.util 376 :info :write [5 41] :RequestTimeoutException INFO [2022-03-13 17:08:03,637] jepsen worker 16 - jepsen.util 466 :invoke :read [5 nil] INFO [2022-03-13 17:08:03,650] jepsen worker 59 - jepsen.util 509 :info :write [19 41] :RequestTimeoutException INFO [2022-03-13 17:08:03,651] jepsen worker 59 - jepsen.util 599 :invoke :read [19 nil] INFO [2022-03-13 17:08:03,688] jepsen worker 58 - jepsen.util 508 :ok :read [19 35] INFO [2022-03-13 17:08:03,689] jepsen worker 58 - jepsen.util 508 :invoke :read [19 nil] INFO [2022-03-13 17:08:03,701] jepsen worker 63 - jepsen.util 423 :info :write [21 0] :RequestTimeoutException INFO [2022-03-13 17:08:03,702] jepsen worker 63 - jepsen.util 513 :invoke :read [21 nil] INFO [2022-03-13 17:08:03,707] jepsen worker 72 - jepsen.util 432 :info :write [24 0] :RequestTimeoutException INFO [2022-03-13 17:08:03,709] jepsen worker 72 - jepsen.util 522 :invoke :write [24 31] INFO [2022-03-13 17:08:03,719] jepsen worker 58 - jepsen.util 508 :ok :read [19 35] INFO [2022-03-13 17:08:03,720] jepsen worker 58 - jepsen.util 508 :invoke :read [19 nil] INFO [2022-03-13 17:08:03,753] jepsen worker 16 - jepsen.util 466 :ok :read [5 32] INFO [2022-03-13 17:08:03,754] jepsen worker 16 - jepsen.util 466 :invoke :read [5 nil] INFO [2022-03-13 17:08:03,758] jepsen worker 58 - jepsen.util 508 :ok :read [19 35] INFO [2022-03-13 17:08:03,758] jepsen worker 58 - jepsen.util 508 :invoke :read [19 nil] INFO [2022-03-13 17:08:03,770] jepsen worker 89 - jepsen.util 359 :info :write [29 22] :RequestTimeoutException INFO [2022-03-13 17:08:03,771] jepsen worker 89 - jepsen.util 449 :invoke :read [29 nil] INFO [2022-03-13 17:08:03,793] jepsen worker 16 - jepsen.util 466 :ok :read [5 32] INFO [2022-03-13 17:08:03,794] jepsen worker 16 - jepsen.util 466 :invoke :write [5 40] INFO [2022-03-13 17:08:03,794] jepsen worker 58 - jepsen.util 508 :ok :read [19 35] INFO [2022-03-13 17:08:03,795] jepsen worker 58 - jepsen.util 508 :invoke :write [19 35] INFO [2022-03-13 17:08:03,825] jepsen worker 89 - jepsen.util 449 :ok :read [29 17] INFO [2022-03-13 17:08:03,826] jepsen worker 89 - jepsen.util 449 :invoke :write [29 23] INFO [2022-03-13 17:08:03,936] jepsen worker 22 - jepsen.util 292 :info :write [7 22] :RequestTimeoutException INFO [2022-03-13 17:08:03,937] jepsen worker 22 - jepsen.util 382 :invoke :read [7 nil] INFO [2022-03-13 17:08:04,045] jepsen worker 55 - jepsen.util 505 :fail :write [18 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,046] jepsen worker 55 - jepsen.util 505 :invoke :read [18 nil] INFO [2022-03-13 17:08:04,086] jepsen worker 20 - jepsen.util 560 :ok :read [6 :nil] INFO [2022-03-13 17:08:04,087] jepsen worker 20 - jepsen.util 560 :invoke :write [6 18] INFO [2022-03-13 17:08:04,113] jepsen worker 10 - jepsen.util 280 :fail :write [3 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,114] jepsen worker 10 - jepsen.util 280 :invoke :read [3 nil] INFO [2022-03-13 17:08:04,114] jepsen worker 9 - jepsen.util 459 :fail :write [3 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,116] jepsen worker 9 - jepsen.util 459 :invoke :write [3 24] INFO [2022-03-13 17:08:04,120] jepsen worker 4 - jepsen.util 454 :info :write [1 16] :RequestTimeoutException INFO [2022-03-13 17:08:04,121] jepsen worker 4 - jepsen.util 544 :invoke :write [1 1] INFO [2022-03-13 17:08:04,201] jepsen worker 64 - jepsen.util 334 :fail :write [21 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,202] jepsen worker 64 - jepsen.util 334 :invoke :read [21 nil] INFO [2022-03-13 17:08:04,205] jepsen worker 32 - jepsen.util 392 :fail :write [10 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,205] jepsen worker 63 - jepsen.util 513 :ok :read [21 :nil] INFO [2022-03-13 17:08:04,207] jepsen worker 63 - jepsen.util 513 :invoke :read [21 nil] INFO [2022-03-13 17:08:04,207] jepsen worker 32 - jepsen.util 392 :invoke :write [10 23] INFO [2022-03-13 17:08:04,215] jepsen worker 40 - jepsen.util 400 :info :write [13 1] :RequestTimeoutException INFO [2022-03-13 17:08:04,216] jepsen worker 40 - jepsen.util 490 :invoke :read [13 nil] INFO [2022-03-13 17:08:04,334] jepsen worker 81 - jepsen.util 531 :info :write [27 31] :RequestTimeoutException INFO [2022-03-13 17:08:04,334] jepsen worker 69 - jepsen.util 339 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,334] jepsen worker 42 - jepsen.util 312 :ok :read [14 :nil] INFO [2022-03-13 17:08:04,334] jepsen worker 26 - jepsen.util 296 :ok :read [8 :nil] INFO [2022-03-13 17:08:04,335] jepsen worker 21 - jepsen.util 471 :ok :read [7 :nil] INFO [2022-03-13 17:08:04,336] jepsen worker 42 - jepsen.util 312 :invoke :read [14 nil] INFO [2022-03-13 17:08:04,337] jepsen worker 26 - jepsen.util 296 :invoke :write [8 4] INFO [2022-03-13 17:08:04,339] jepsen worker 21 - jepsen.util 471 :invoke :read [7 nil] INFO [2022-03-13 17:08:04,339] jepsen worker 69 - jepsen.util 339 :invoke :write [23 41] INFO [2022-03-13 17:08:04,340] jepsen worker 81 - jepsen.util 621 :invoke :write [27 48] INFO [2022-03-13 17:08:04,376] jepsen worker 71 - jepsen.util 251 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,377] jepsen worker 21 - jepsen.util 471 :ok :read [7 :nil] INFO [2022-03-13 17:08:04,377] jepsen worker 42 - jepsen.util 312 :ok :read [14 :nil] INFO [2022-03-13 17:08:04,377] jepsen worker 71 - jepsen.util 251 :invoke :read [23 nil] INFO [2022-03-13 17:08:04,378] jepsen worker 21 - jepsen.util 471 :invoke :read [7 nil] INFO [2022-03-13 17:08:04,379] jepsen worker 42 - jepsen.util 312 :invoke :write [14 30] INFO [2022-03-13 17:08:04,453] jepsen worker 10 - jepsen.util 280 :ok :read [3 :nil] INFO [2022-03-13 17:08:04,454] jepsen worker 10 - jepsen.util 280 :invoke :write [3 27] INFO [2022-03-13 17:08:04,465] jepsen worker 69 - jepsen.util 339 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:04,465] jepsen worker 69 - jepsen.util 339 :invoke :read [23 nil] INFO [2022-03-13 17:08:04,566] jepsen worker 21 - jepsen.util 471 :ok :read [7 :nil] INFO [2022-03-13 17:08:04,566] jepsen worker 71 - jepsen.util 251 :ok :read [23 :nil] INFO [2022-03-13 17:08:04,567] jepsen worker 21 - jepsen.util 471 :invoke :read [7 nil] INFO [2022-03-13 17:08:04,568] jepsen worker 71 - jepsen.util 251 :invoke :write [23 26] INFO [2022-03-13 17:08:04,624] jepsen worker 21 - jepsen.util 471 :ok :read [7 :nil] INFO [2022-03-13 17:08:04,624] jepsen worker 69 - jepsen.util 339 :ok :read [23 :nil] INFO [2022-03-13 17:08:04,624] jepsen worker 21 - jepsen.util 471 :invoke :write [7 13] INFO [2022-03-13 17:08:04,627] jepsen worker 69 - jepsen.util 339 :invoke :write [23 37] INFO [2022-03-13 17:08:04,631] jepsen worker 63 - jepsen.util 513 :ok :read [21 :nil] INFO [2022-03-13 17:08:04,631] jepsen worker 63 - jepsen.util 513 :invoke :read [21 nil] INFO [2022-03-13 17:08:04,731] jepsen worker 27 - jepsen.util 387 :info :write [9 26] :RequestTimeoutException INFO [2022-03-13 17:08:04,732] jepsen worker 27 - jepsen.util 477 :invoke :read [9 nil] INFO [2022-03-13 17:08:04,828] jepsen worker 5 - jepsen.util 365 :info :write [1 45] :RequestTimeoutException INFO [2022-03-13 17:08:04,829] jepsen worker 5 - jepsen.util 455 :invoke :read [1 nil] INFO [2022-03-13 17:08:04,855] jepsen worker nemesis - couchbase.util Rebalancing nodes (172.23.106.61) out of cluster INFO [2022-03-13 17:08:04,923] jepsen worker 38 - jepsen.util 488 :ok :read [12 46] INFO [2022-03-13 17:08:04,924] jepsen worker 38 - jepsen.util 488 :invoke :write [12 45] INFO [2022-03-13 17:08:04,936] jepsen worker 18 - jepsen.util 378 :info :write [6 11] :RequestTimeoutException INFO [2022-03-13 17:08:04,937] jepsen worker 18 - jepsen.util 468 :invoke :read [6 nil] INFO [2022-03-13 17:08:04,977] jepsen worker 46 - jepsen.util 406 :info :write [15 42] :RequestTimeoutException INFO [2022-03-13 17:08:04,977] jepsen worker 46 - jepsen.util 496 :invoke :read [15 nil] INFO [2022-03-13 17:08:05,025] jepsen worker 55 - jepsen.util 505 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:05,026] jepsen worker 55 - jepsen.util 505 :invoke :read [18 nil] INFO [2022-03-13 17:08:05,107] jepsen worker 5 - jepsen.util 455 :ok :read [1 28] INFO [2022-03-13 17:08:05,109] jepsen worker 5 - jepsen.util 455 :invoke :write [1 33] INFO [2022-03-13 17:08:05,255] jepsen worker 27 - jepsen.util 477 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:05,257] jepsen worker 27 - jepsen.util 477 :invoke :write [9 28] INFO [2022-03-13 17:08:05,307] jepsen worker 63 - jepsen.util 513 :ok :read [21 :nil] INFO [2022-03-13 17:08:05,308] jepsen worker 63 - jepsen.util 513 :invoke :write [21 10] INFO [2022-03-13 17:08:05,366] jepsen worker 68 - jepsen.util 518 :info :write [22 4] :RequestTimeoutException INFO [2022-03-13 17:08:05,367] jepsen worker 68 - jepsen.util 608 :invoke :read [22 nil] WARN [2022-03-13 17:08:05,372] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8463737,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9155605,"total_server_duration_us":388037,"encode_duration_us":18549,"operation_id":"0x11b7","timeout_ms":10000,"last_server_duration_us":64603,"total_duration_us":10171116},{"operation_name":"upsert","last_dispatch_duration_us":37923,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1038851,"total_server_duration_us":407875,"encode_duration_us":17932,"operation_id":"0x12b9","timeout_ms":10000,"last_server_duration_us":26418,"total_duration_us":10009037},{"operation_name":"upsert","last_dispatch_duration_us":335099,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9944741,"total_server_duration_us":433808,"encode_duration_us":17099,"operation_id":"0x11dc","timeout_ms":10000,"last_server_duration_us":44279,"total_duration_us":10005635},{"operation_name":"upsert","last_dispatch_duration_us":1441114,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9565952,"total_server_duration_us":231056,"encode_duration_us":18029,"operation_id":"0x12d4","timeout_ms":10000,"last_server_duration_us":41983,"total_duration_us":10004778},{"operation_name":"upsert","last_dispatch_duration_us":1445913,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10007083,"total_server_duration_us":353860,"encode_duration_us":18499,"operation_id":"0x12a8","timeout_ms":10000,"last_server_duration_us":141868,"total_duration_us":10002965},{"operation_name":"upsert","last_dispatch_duration_us":42113,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7399220,"total_server_duration_us":344588,"encode_duration_us":23146,"operation_id":"0x116d","timeout_ms":10000,"last_server_duration_us":31298,"total_duration_us":10001935},{"operation_name":"upsert","last_dispatch_duration_us":72582,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1137462,"total_server_duration_us":545925,"encode_duration_us":19953,"operation_id":"0x12a6","timeout_ms":10000,"last_server_duration_us":38378,"total_duration_us":10001398},{"operation_name":"upsert","last_dispatch_duration_us":108716,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7452842,"total_server_duration_us":357825,"encode_duration_us":21887,"operation_id":"0x114b","timeout_ms":10000,"last_server_duration_us":37447,"total_duration_us":10000849},{"operation_name":"upsert","last_dispatch_duration_us":9353056,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9353056,"total_server_duration_us":9093693,"encode_duration_us":18435,"operation_id":"0x11e8","timeout_ms":10000,"last_server_duration_us":9093693,"total_duration_us":9373204},{"operation_name":"upsert","last_dispatch_duration_us":9062260,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9256523,"total_server_duration_us":9167691,"encode_duration_us":24039,"operation_id":"0x11ad","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9322914}],"total_count":22}} WARN [2022-03-13 17:08:05,372] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15553235,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16700912,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9362028,"encode_duration_us":17712,"operation_id":"0x112a","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10007990}],"total_count":1}} INFO [2022-03-13 17:08:05,415] jepsen worker 64 - jepsen.util 334 :ok :read [21 :nil] INFO [2022-03-13 17:08:05,416] jepsen worker 64 - jepsen.util 334 :invoke :write [21 47] INFO [2022-03-13 17:08:05,666] jepsen worker 46 - jepsen.util 496 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:05,667] jepsen worker 46 - jepsen.util 496 :invoke :write [15 37] INFO [2022-03-13 17:08:05,840] jepsen worker 85 - jepsen.util 535 :info :write [28 37] :RequestTimeoutException INFO [2022-03-13 17:08:05,840] jepsen worker 85 - jepsen.util 625 :invoke :write [28 21] INFO [2022-03-13 17:08:05,957] jepsen worker 44 - jepsen.util 314 :ok :read [14 :nil] INFO [2022-03-13 17:08:05,959] jepsen worker 44 - jepsen.util 314 :invoke :write [14 33] INFO [2022-03-13 17:08:05,978] jepsen worker 66 - jepsen.util 426 :info :write [22 30] :RequestTimeoutException INFO [2022-03-13 17:08:05,979] jepsen worker 66 - jepsen.util 516 :invoke :read [22 nil] INFO [2022-03-13 17:08:05,994] jepsen worker 34 - jepsen.util 304 :ok :read [11 :nil] INFO [2022-03-13 17:08:05,995] jepsen worker 34 - jepsen.util 304 :invoke :write [11 22] INFO [2022-03-13 17:08:06,317] jepsen worker 19 - jepsen.util 289 :info :write [6 30] :RequestTimeoutException INFO [2022-03-13 17:08:06,317] jepsen worker 19 - jepsen.util 379 :invoke :write [6 40] INFO [2022-03-13 17:08:06,327] jepsen worker 61 - jepsen.util 421 :ok :read [20 10] INFO [2022-03-13 17:08:06,327] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:06,375] jepsen worker 18 - jepsen.util 468 :ok :read [6 :nil] INFO [2022-03-13 17:08:06,377] jepsen worker 18 - jepsen.util 468 :invoke :read [6 nil] INFO [2022-03-13 17:08:06,445] jepsen worker 55 - jepsen.util 505 :ok :read [18 40] INFO [2022-03-13 17:08:06,446] jepsen worker 55 - jepsen.util 505 :invoke :read [18 nil] INFO [2022-03-13 17:08:06,666] jepsen worker 56 - jepsen.util 416 :info :write [18 22] :RequestTimeoutException INFO [2022-03-13 17:08:06,667] jepsen worker 56 - jepsen.util 506 :invoke :write [18 3] WARN [2022-03-13 17:08:06,775] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":790187,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10327440,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":339230,"encode_duration_us":19153,"operation_id":"0x12a9","timeout_ms":10000,"last_server_duration_us":70556,"total_duration_us":10008091}],"total_count":1}} INFO [2022-03-13 17:08:06,871] jepsen worker 3 - jepsen.util 453 :info :write [1 35] :RequestTimeoutException INFO [2022-03-13 17:08:06,872] jepsen worker 3 - jepsen.util 543 :invoke :read [1 nil] WARN [2022-03-13 17:08:06,874] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9297477,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13154894,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9786142,"encode_duration_us":19315,"operation_id":"0x1119","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10002512},{"operation_name":"upsert","last_dispatch_duration_us":9098327,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10180607,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9289443,"encode_duration_us":19217,"operation_id":"0x121f","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10002624},{"operation_name":"upsert","last_dispatch_duration_us":9182653,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10911589,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9459633,"encode_duration_us":19745,"operation_id":"0x120b","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10005184},{"operation_name":"upsert","last_dispatch_duration_us":9036300,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10663151,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9568981,"encode_duration_us":17826,"operation_id":"0x1236","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10006364},{"operation_name":"upsert","last_dispatch_duration_us":9090090,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13507918,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9337275,"encode_duration_us":30028,"operation_id":"0x1181","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10009281},{"operation_name":"upsert","last_dispatch_duration_us":9990127,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14506655,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":11099825,"encode_duration_us":17791,"operation_id":"0x1118","timeout_ms":10000,"last_server_duration_us":9979599,"total_duration_us":10006198},{"operation_name":"upsert","last_dispatch_duration_us":9106797,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11208351,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9339284,"encode_duration_us":23113,"operation_id":"0x11e2","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007713},{"operation_name":"upsert","last_dispatch_duration_us":9100390,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10967330,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9384518,"encode_duration_us":20277,"operation_id":"0x11ba","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10130724},{"operation_name":"upsert","last_dispatch_duration_us":9227992,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10474816,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9328053,"encode_duration_us":21357,"operation_id":"0x121c","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10009144},{"operation_name":"upsert","last_dispatch_duration_us":9107945,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11906448,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9592849,"encode_duration_us":16911,"operation_id":"0x11d7","timeout_ms":10000,"last_server_duration_us":9087309,"total_duration_us":10009767}],"total_count":11}} WARN [2022-03-13 17:08:06,875] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9100390,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10967330,"total_server_duration_us":9384518,"encode_duration_us":20277,"operation_id":"0x11ba","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10130724},{"operation_name":"upsert","last_dispatch_duration_us":9107945,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":11906448,"total_server_duration_us":9592849,"encode_duration_us":16911,"operation_id":"0x11d7","timeout_ms":10000,"last_server_duration_us":9087309,"total_duration_us":10009767},{"operation_name":"upsert","last_dispatch_duration_us":9227992,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10474816,"total_server_duration_us":9328053,"encode_duration_us":21357,"operation_id":"0x121c","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10009144},{"operation_name":"upsert","last_dispatch_duration_us":9106797,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11208351,"total_server_duration_us":9339284,"encode_duration_us":23113,"operation_id":"0x11e2","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007713},{"operation_name":"upsert","last_dispatch_duration_us":9036300,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10663151,"total_server_duration_us":9568981,"encode_duration_us":17826,"operation_id":"0x1236","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10006364},{"operation_name":"upsert","last_dispatch_duration_us":93779,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1755941,"total_server_duration_us":367689,"encode_duration_us":17410,"operation_id":"0x1217","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10005576},{"operation_name":"upsert","last_dispatch_duration_us":9182653,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10911589,"total_server_duration_us":9459633,"encode_duration_us":19745,"operation_id":"0x120b","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10005184},{"operation_name":"upsert","last_dispatch_duration_us":317424,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8434441,"total_server_duration_us":913843,"encode_duration_us":35958,"operation_id":"0x12de","timeout_ms":10000,"last_server_duration_us":65253,"total_duration_us":10002922},{"operation_name":"upsert","last_dispatch_duration_us":9098327,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10180607,"total_server_duration_us":9289443,"encode_duration_us":19217,"operation_id":"0x121f","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10002624},{"operation_name":"upsert","last_dispatch_duration_us":8502574,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10673164,"total_server_duration_us":608618,"encode_duration_us":22058,"operation_id":"0x11ef","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10002061}],"total_count":20}} WARN [2022-03-13 17:08:06,875] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8306170,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":8593635,"total_server_duration_us":102707,"encode_duration_us":17459,"operation_id":"0x11b8","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10164242},{"operation_name":"upsert","last_dispatch_duration_us":8469653,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":8877378,"total_server_duration_us":101210,"encode_duration_us":17837,"operation_id":"0x11d2","timeout_ms":10000,"last_server_duration_us":27492,"total_duration_us":10011047},{"operation_name":"upsert","last_dispatch_duration_us":64256,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1201189,"total_server_duration_us":257811,"encode_duration_us":20180,"operation_id":"0x128c","timeout_ms":10000,"last_server_duration_us":46739,"total_duration_us":10011002},{"operation_name":"upsert","last_dispatch_duration_us":65962,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":313535,"total_server_duration_us":44787,"encode_duration_us":16809,"operation_id":"0x11e0","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10010697},{"operation_name":"upsert","last_dispatch_duration_us":8776120,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9508562,"total_server_duration_us":296736,"encode_duration_us":18139,"operation_id":"0x12ac","timeout_ms":10000,"last_server_duration_us":42091,"total_duration_us":10006651},{"operation_name":"upsert","last_dispatch_duration_us":10012193,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10063804,"total_server_duration_us":9101321,"encode_duration_us":18409,"operation_id":"0x125f","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10006621},{"operation_name":"upsert","last_dispatch_duration_us":8600639,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":12942301,"total_server_duration_us":1598822,"encode_duration_us":17125,"operation_id":"0x11b2","timeout_ms":10000,"last_server_duration_us":120988,"total_duration_us":10006452},{"operation_name":"upsert","last_dispatch_duration_us":9315167,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9315167,"total_server_duration_us":9044797,"encode_duration_us":18468,"operation_id":"0x11ec","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10004420},{"operation_name":"upsert","last_dispatch_duration_us":8667588,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":12003230,"total_server_duration_us":1703260,"encode_duration_us":17174,"operation_id":"0x1222","timeout_ms":10000,"last_server_duration_us":83124,"total_duration_us":10003651},{"operation_name":"upsert","last_dispatch_duration_us":8611981,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13311675,"total_server_duration_us":1417633,"encode_duration_us":19550,"operation_id":"0x119a","timeout_ms":10000,"last_server_duration_us":51710,"total_duration_us":10001830}],"total_count":15}} WARN [2022-03-13 17:08:06,876] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":50359,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8854964,"total_server_duration_us":162288,"encode_duration_us":25322,"operation_id":"0x120c","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10010274},{"operation_name":"upsert","last_dispatch_duration_us":191412,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3047242,"total_server_duration_us":240256,"encode_duration_us":19274,"operation_id":"0x123e","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10010108},{"operation_name":"upsert","last_dispatch_duration_us":264903,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9972401,"total_server_duration_us":125465,"encode_duration_us":22896,"operation_id":"0x11ae","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10009187},{"operation_name":"upsert","last_dispatch_duration_us":35350,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1002151,"total_server_duration_us":204367,"encode_duration_us":16981,"operation_id":"0x1302","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10009085},{"operation_name":"upsert","last_dispatch_duration_us":74734,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":6465808,"total_server_duration_us":307732,"encode_duration_us":24069,"operation_id":"0x125e","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10008430},{"operation_name":"upsert","last_dispatch_duration_us":790187,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10327440,"total_server_duration_us":339230,"encode_duration_us":19153,"operation_id":"0x12a9","timeout_ms":10000,"last_server_duration_us":70556,"total_duration_us":10008091},{"operation_name":"upsert","last_dispatch_duration_us":185931,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":6719204,"total_server_duration_us":349924,"encode_duration_us":17807,"operation_id":"0x1237","timeout_ms":10000,"last_server_duration_us":39424,"total_duration_us":10006703},{"operation_name":"upsert","last_dispatch_duration_us":440562,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":7826270,"total_server_duration_us":228126,"encode_duration_us":24135,"operation_id":"0x11de","timeout_ms":10000,"last_server_duration_us":58999,"total_duration_us":10004919},{"operation_name":"upsert","last_dispatch_duration_us":175087,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2299484,"total_server_duration_us":200750,"encode_duration_us":18236,"operation_id":"0x123f","timeout_ms":10000,"last_server_duration_us":31489,"total_duration_us":10004264},{"operation_name":"upsert","last_dispatch_duration_us":7840490,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13522664,"total_server_duration_us":478779,"encode_duration_us":19370,"operation_id":"0x1191","timeout_ms":10000,"last_server_duration_us":33721,"total_duration_us":10003996}],"total_count":20}} WARN [2022-03-13 17:08:06,974] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9481292,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":17030637,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":9296445,"encode_duration_us":19849,"operation_id":"0x11d6","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10007923},{"operation_name":"upsert","last_dispatch_duration_us":16535405,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25976151,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9162853,"encode_duration_us":25029,"operation_id":"0x108b","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":10010150}],"total_count":2}} WARN [2022-03-13 17:08:06,975] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9315167,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9315167,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":9044797,"encode_duration_us":18468,"operation_id":"0x11ec","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10004420},{"operation_name":"upsert","last_dispatch_duration_us":10012193,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10063804,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9101321,"encode_duration_us":18409,"operation_id":"0x125f","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10006621},{"operation_name":"upsert","last_dispatch_duration_us":9274258,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13788551,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":10611769,"encode_duration_us":23228,"operation_id":"0x1170","timeout_ms":10000,"last_server_duration_us":9155503,"total_duration_us":10008347}],"total_count":3}} WARN [2022-03-13 17:08:06,976] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":58642,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":561677,"total_server_duration_us":167877,"encode_duration_us":22437,"operation_id":"0x11bb","timeout_ms":10000,"last_server_duration_us":40802,"total_duration_us":10127841},{"operation_name":"upsert","last_dispatch_duration_us":127619,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":613855,"total_server_duration_us":172964,"encode_duration_us":18372,"operation_id":"0x11b3","timeout_ms":10000,"last_server_duration_us":84138,"total_duration_us":10009396},{"operation_name":"upsert","last_dispatch_duration_us":1261041,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9613187,"total_server_duration_us":480010,"encode_duration_us":18372,"operation_id":"0x1286","timeout_ms":10000,"last_server_duration_us":62796,"total_duration_us":10008061},{"operation_name":"upsert","last_dispatch_duration_us":281434,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7557648,"total_server_duration_us":355135,"encode_duration_us":20548,"operation_id":"0x12e2","timeout_ms":10000,"last_server_duration_us":26064,"total_duration_us":10007938},{"operation_name":"upsert","last_dispatch_duration_us":1002112,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10292418,"total_server_duration_us":397422,"encode_duration_us":19445,"operation_id":"0x12dd","timeout_ms":10000,"last_server_duration_us":53494,"total_duration_us":10007646},{"operation_name":"upsert","last_dispatch_duration_us":9295982,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":9295982,"total_server_duration_us":9054355,"encode_duration_us":17957,"operation_id":"0x11db","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10007239},{"operation_name":"upsert","last_dispatch_duration_us":6738668,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7350779,"total_server_duration_us":225391,"encode_duration_us":31025,"operation_id":"0x1189","timeout_ms":10000,"last_server_duration_us":51003,"total_duration_us":10007228},{"operation_name":"upsert","last_dispatch_duration_us":6732596,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7350732,"total_server_duration_us":256140,"encode_duration_us":21293,"operation_id":"0x122e","timeout_ms":10000,"last_server_duration_us":65906,"total_duration_us":10006209},{"operation_name":"upsert","last_dispatch_duration_us":6574983,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11881051,"total_server_duration_us":6667310,"encode_duration_us":30093,"operation_id":"0x11e3","timeout_ms":10000,"last_server_duration_us":6435088,"total_duration_us":10005453},{"operation_name":"upsert","last_dispatch_duration_us":1573034,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9910474,"total_server_duration_us":501007,"encode_duration_us":17588,"operation_id":"0x1306","timeout_ms":10000,"last_server_duration_us":93954,"total_duration_us":10005102}],"total_count":20}} INFO [2022-03-13 17:08:07,021] jepsen worker 18 - jepsen.util 468 :ok :read [6 :nil] INFO [2022-03-13 17:08:07,023] jepsen worker 18 - jepsen.util 468 :invoke :read [6 nil] INFO [2022-03-13 17:08:07,034] jepsen worker nemesis - couchbase.util Rebalance status: {} INFO [2022-03-13 17:08:07,035] jepsen worker nemesis - couchbase.util Rebalance check count: 0 INFO [2022-03-13 17:08:07,035] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 WARN [2022-03-13 17:08:07,074] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7524237,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16124183,"total_server_duration_us":155733,"encode_duration_us":23373,"operation_id":"0x11e1","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10010433},{"operation_name":"upsert","last_dispatch_duration_us":7083161,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":13885124,"total_server_duration_us":307437,"encode_duration_us":19806,"operation_id":"0x1245","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10010005},{"operation_name":"upsert","last_dispatch_duration_us":7394572,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7494558,"total_server_duration_us":59449,"encode_duration_us":18084,"operation_id":"0x118d","timeout_ms":10000,"last_server_duration_us":36729,"total_duration_us":10009419},{"operation_name":"upsert","last_dispatch_duration_us":300230,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":742364,"total_server_duration_us":47999,"encode_duration_us":25925,"operation_id":"0x12bb","timeout_ms":10000,"last_server_duration_us":25362,"total_duration_us":10008516},{"operation_name":"upsert","last_dispatch_duration_us":9481292,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":17030637,"total_server_duration_us":9296445,"encode_duration_us":19849,"operation_id":"0x11d6","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10007923},{"operation_name":"upsert","last_dispatch_duration_us":88528,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":5277931,"total_server_duration_us":1449669,"encode_duration_us":25123,"operation_id":"0x11a8","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":10007650},{"operation_name":"upsert","last_dispatch_duration_us":7535954,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16337590,"total_server_duration_us":163938,"encode_duration_us":17727,"operation_id":"0x11f7","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10007169},{"operation_name":"upsert","last_dispatch_duration_us":7102526,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":13364496,"total_server_duration_us":350947,"encode_duration_us":21183,"operation_id":"0x1268","timeout_ms":10000,"last_server_duration_us":60632,"total_duration_us":10006092},{"operation_name":"upsert","last_dispatch_duration_us":461493,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":837456,"total_server_duration_us":60290,"encode_duration_us":19400,"operation_id":"0x1226","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10005981},{"operation_name":"upsert","last_dispatch_duration_us":249155,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9171352,"total_server_duration_us":377597,"encode_duration_us":20420,"operation_id":"0x11b0","timeout_ms":10000,"last_server_duration_us":24669,"total_duration_us":10004314}],"total_count":22}} INFO [2022-03-13 17:08:07,075] jepsen worker 57 - jepsen.util 507 :ok :read [19 35] INFO [2022-03-13 17:08:07,076] jepsen worker 57 - jepsen.util 507 :invoke :write [19 44] WARN [2022-03-13 17:08:07,077] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9109035,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9453427,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9103404,"encode_duration_us":18733,"operation_id":"0x121d","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10005020},{"operation_name":"upsert","last_dispatch_duration_us":6574983,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11881051,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":6667310,"encode_duration_us":30093,"operation_id":"0x11e3","timeout_ms":10000,"last_server_duration_us":6435088,"total_duration_us":10005453},{"operation_name":"upsert","last_dispatch_duration_us":9295982,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9295982,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9054355,"encode_duration_us":17957,"operation_id":"0x11db","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10007239},{"operation_name":"upsert","last_dispatch_duration_us":9101700,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17635280,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9174215,"encode_duration_us":20738,"operation_id":"0x1145","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10010869}],"total_count":4}} INFO [2022-03-13 17:08:07,077] jepsen worker 55 - jepsen.util 505 :ok :read [18 40] INFO [2022-03-13 17:08:07,078] jepsen worker 55 - jepsen.util 505 :invoke :write [18 10] INFO [2022-03-13 17:08:07,134] jepsen worker 17 - jepsen.util 557 :ok :read [5 32] INFO [2022-03-13 17:08:07,135] jepsen worker 17 - jepsen.util 557 :invoke :write [5 33] INFO [2022-03-13 17:08:07,171] jepsen worker 66 - jepsen.util 516 :ok :read [22 46] INFO [2022-03-13 17:08:07,172] jepsen worker 66 - jepsen.util 516 :invoke :write [22 13] INFO [2022-03-13 17:08:07,173] jepsen worker 15 - jepsen.util 555 :ok :read [5 32] INFO [2022-03-13 17:08:07,174] jepsen worker 15 - jepsen.util 555 :invoke :read [5 nil] INFO [2022-03-13 17:08:07,266] jepsen worker 74 - jepsen.util 524 :info :write [24 40] :RequestTimeoutException INFO [2022-03-13 17:08:07,268] jepsen worker 74 - jepsen.util 614 :invoke :write [24 22] INFO [2022-03-13 17:08:07,275] jepsen worker 15 - jepsen.util 555 :ok :read [5 32] INFO [2022-03-13 17:08:07,276] jepsen worker 15 - jepsen.util 555 :invoke :read [5 nil] INFO [2022-03-13 17:08:07,334] jepsen worker 73 - jepsen.util 523 :info :write [24 38] :RequestTimeoutException INFO [2022-03-13 17:08:07,334] jepsen worker 73 - jepsen.util 613 :invoke :write [24 7] INFO [2022-03-13 17:08:07,348] jepsen worker 83 - jepsen.util 353 :info :write [27 34] :RequestTimeoutException INFO [2022-03-13 17:08:07,349] jepsen worker 83 - jepsen.util 443 :invoke :write [27 25] INFO [2022-03-13 17:08:07,408] jepsen worker 15 - jepsen.util 555 :ok :read [5 32] INFO [2022-03-13 17:08:07,409] jepsen worker 15 - jepsen.util 555 :invoke :write [5 38] INFO [2022-03-13 17:08:07,432] jepsen worker 36 - jepsen.util 396 :info :write [12 10] :RequestTimeoutException INFO [2022-03-13 17:08:07,432] jepsen worker 36 - jepsen.util 486 :invoke :read [12 nil] INFO [2022-03-13 17:08:07,433] jepsen worker 37 - jepsen.util 397 :info :write [12 25] :RequestTimeoutException INFO [2022-03-13 17:08:07,434] jepsen worker 37 - jepsen.util 487 :invoke :read [12 nil] INFO [2022-03-13 17:08:07,459] jepsen worker 78 - jepsen.util 258 :info :write [26 21] :RequestTimeoutException INFO [2022-03-13 17:08:07,460] jepsen worker 78 - jepsen.util 348 :invoke :write [26 0] INFO [2022-03-13 17:08:07,534] jepsen worker 35 - jepsen.util 485 :info :write [11 38] :RequestTimeoutException INFO [2022-03-13 17:08:07,535] jepsen worker 35 - jepsen.util 575 :invoke :write [11 5] INFO [2022-03-13 17:08:07,547] jepsen worker 11 - jepsen.util 371 :info :write [3 11] :RequestTimeoutException INFO [2022-03-13 17:08:07,548] jepsen worker 62 - jepsen.util 242 :info :write [20 13] :RequestTimeoutException INFO [2022-03-13 17:08:07,548] jepsen worker 11 - jepsen.util 461 :invoke :write [3 49] INFO [2022-03-13 17:08:07,549] jepsen worker 62 - jepsen.util 332 :invoke :read [20 nil] INFO [2022-03-13 17:08:07,557] jepsen worker 36 - jepsen.util 486 :ok :read [12 46] INFO [2022-03-13 17:08:07,558] jepsen worker 36 - jepsen.util 486 :invoke :read [12 nil] INFO [2022-03-13 17:08:07,790] jepsen worker 36 - jepsen.util 486 :ok :read [12 46] INFO [2022-03-13 17:08:07,791] jepsen worker 36 - jepsen.util 486 :invoke :write [12 46] INFO [2022-03-13 17:08:07,801] jepsen worker 28 - jepsen.util 478 :info :write [9 34] :RequestTimeoutException INFO [2022-03-13 17:08:07,802] jepsen worker 28 - jepsen.util 568 :invoke :write [9 20] INFO [2022-03-13 17:08:08,014] jepsen worker 60 - jepsen.util 420 :info :write [20 10] :RequestTimeoutException INFO [2022-03-13 17:08:08,015] jepsen worker 60 - jepsen.util 510 :invoke :write [20 35] INFO [2022-03-13 17:08:08,024] jepsen worker 48 - jepsen.util 408 :info :write [16 33] :RequestTimeoutException INFO [2022-03-13 17:08:08,025] jepsen worker 48 - jepsen.util 498 :invoke :read [16 nil] INFO [2022-03-13 17:08:08,042] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:08,043] jepsen worker nemesis - couchbase.util Rebalance check count: 1 INFO [2022-03-13 17:08:08,043] jepsen worker nemesis - couchbase.util Rebalance stuck count: 0 INFO [2022-03-13 17:08:08,118] jepsen worker 2 - jepsen.util 542 :info :write [0 49] :RequestTimeoutException INFO [2022-03-13 17:08:08,119] jepsen worker 2 - jepsen.util 632 :invoke :read [0 nil] INFO [2022-03-13 17:08:08,315] jepsen worker 29 - jepsen.util 479 :info :write [9 22] :RequestTimeoutException INFO [2022-03-13 17:08:08,316] jepsen worker 29 - jepsen.util 569 :invoke :read [9 nil] INFO [2022-03-13 17:08:08,601] jepsen worker 80 - jepsen.util 440 :info :write [26 9] :RequestTimeoutException INFO [2022-03-13 17:08:08,602] jepsen worker 80 - jepsen.util 530 :invoke :write [26 16] INFO [2022-03-13 17:08:08,635] jepsen worker 29 - jepsen.util 569 :ok :read [9 30] INFO [2022-03-13 17:08:08,635] jepsen worker 29 - jepsen.util 569 :invoke :write [9 27] INFO [2022-03-13 17:08:08,711] jepsen worker 24 - jepsen.util 204 :info :write [8 45] :RequestTimeoutException INFO [2022-03-13 17:08:08,712] jepsen worker 24 - jepsen.util 294 :invoke :write [8 43] INFO [2022-03-13 17:08:08,981] jepsen worker 76 - jepsen.util 346 :info :write [25 39] :RequestTimeoutException INFO [2022-03-13 17:08:08,982] jepsen worker 76 - jepsen.util 436 :invoke :read [25 nil] INFO [2022-03-13 17:08:09,049] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:09,049] jepsen worker nemesis - couchbase.util Rebalance check count: 2 INFO [2022-03-13 17:08:09,049] jepsen worker nemesis - couchbase.util Rebalance stuck count: 1 INFO [2022-03-13 17:08:09,068] jepsen worker 54 - jepsen.util 594 :info :write [18 12] :RequestTimeoutException INFO [2022-03-13 17:08:09,069] jepsen worker 54 - jepsen.util 684 :invoke :read [18 nil] INFO [2022-03-13 17:08:09,168] jepsen worker 6 - jepsen.util 366 :info :write [2 16] :RequestTimeoutException INFO [2022-03-13 17:08:09,169] jepsen worker 6 - jepsen.util 456 :invoke :write [2 41] INFO [2022-03-13 17:08:09,271] jepsen worker 70 - jepsen.util 430 :info :write [23 21] :RequestTimeoutException INFO [2022-03-13 17:08:09,271] jepsen worker 70 - jepsen.util 520 :invoke :write [23 31] INFO [2022-03-13 17:08:09,444] jepsen worker 52 - jepsen.util 322 :info :write [17 47] :RequestTimeoutException INFO [2022-03-13 17:08:09,445] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:09,450] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:09,450] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:09,537] jepsen worker 82 - jepsen.util 532 :info :write [27 45] :RequestTimeoutException INFO [2022-03-13 17:08:09,538] jepsen worker 82 - jepsen.util 622 :invoke :write [27 8] INFO [2022-03-13 17:08:09,578] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:08:09,579] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:08:09,579] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:08:09,618] jepsen worker 61 - jepsen.util 421 :ok :read [20 10] INFO [2022-03-13 17:08:09,619] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:09,648] jepsen worker 79 - jepsen.util 529 :info :write [26 43] :RequestTimeoutException INFO [2022-03-13 17:08:09,649] jepsen worker 79 - jepsen.util 619 :invoke :read [26 nil] INFO [2022-03-13 17:08:09,679] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:08:09,680] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:08:09,803] jepsen worker 22 - jepsen.util 382 :ok :read [7 :nil] INFO [2022-03-13 17:08:09,804] jepsen worker 22 - jepsen.util 382 :invoke :read [7 nil] INFO [2022-03-13 17:08:09,878] cb-events - com.couchbase.node [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:08:09,888] jepsen worker 76 - jepsen.util 436 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:09,888] jepsen worker 79 - jepsen.util 619 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:09,889] jepsen worker 76 - jepsen.util 436 :invoke :write [25 21] INFO [2022-03-13 17:08:09,896] jepsen worker 79 - jepsen.util 619 :invoke :read [26 nil] INFO [2022-03-13 17:08:09,897] jepsen worker 22 - jepsen.util 382 :ok :read [7 :nil] INFO [2022-03-13 17:08:09,899] jepsen worker 22 - jepsen.util 382 :invoke :read [7 nil] INFO [2022-03-13 17:08:09,968] jepsen worker 88 - jepsen.util 538 :info :write [29 2] :RequestTimeoutException INFO [2022-03-13 17:08:09,969] jepsen worker 88 - jepsen.util 628 :invoke :write [29 7] INFO [2022-03-13 17:08:09,972] jepsen worker 22 - jepsen.util 382 :ok :read [7 :nil] INFO [2022-03-13 17:08:09,973] jepsen worker 22 - jepsen.util 382 :invoke :write [7 19] INFO [2022-03-13 17:08:10,041] jepsen worker 79 - jepsen.util 619 :ok :read [26 36] INFO [2022-03-13 17:08:10,042] jepsen worker 79 - jepsen.util 619 :invoke :read [26 nil] INFO [2022-03-13 17:08:10,055] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:10,055] jepsen worker nemesis - couchbase.util Rebalance check count: 3 INFO [2022-03-13 17:08:10,055] jepsen worker nemesis - couchbase.util Rebalance stuck count: 2 INFO [2022-03-13 17:08:10,159] jepsen worker 79 - jepsen.util 619 :ok :read [26 36] INFO [2022-03-13 17:08:10,159] jepsen worker 79 - jepsen.util 619 :invoke :write [26 17] INFO [2022-03-13 17:08:10,254] jepsen worker 1 - jepsen.util 451 :info :write [0 49] :RequestTimeoutException INFO [2022-03-13 17:08:10,254] jepsen worker 1 - jepsen.util 541 :invoke :write [0 22] INFO [2022-03-13 17:08:10,541] jepsen worker 50 - jepsen.util 500 :info :write [16 1] :RequestTimeoutException INFO [2022-03-13 17:08:10,542] jepsen worker 50 - jepsen.util 590 :invoke :write [16 38] INFO [2022-03-13 17:08:10,618] jepsen worker 39 - jepsen.util 579 :info :write [13 40] :RequestTimeoutException INFO [2022-03-13 17:08:10,619] jepsen worker 39 - jepsen.util 669 :invoke :write [13 37] INFO [2022-03-13 17:08:10,818] jepsen worker 30 - jepsen.util 390 :info :write [10 6] :RequestTimeoutException INFO [2022-03-13 17:08:10,819] jepsen worker 30 - jepsen.util 480 :invoke :read [10 nil] INFO [2022-03-13 17:08:11,021] jepsen worker 53 - jepsen.util 323 :info :write [17 7] :RequestTimeoutException INFO [2022-03-13 17:08:11,022] jepsen worker 53 - jepsen.util 413 :invoke :read [17 nil] INFO [2022-03-13 17:08:11,063] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:11,063] jepsen worker nemesis - couchbase.util Rebalance check count: 4 INFO [2022-03-13 17:08:11,063] jepsen worker nemesis - couchbase.util Rebalance stuck count: 3 INFO [2022-03-13 17:08:11,089] jepsen worker 53 - jepsen.util 413 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,089] jepsen worker 53 - jepsen.util 413 :invoke :read [17 nil] INFO [2022-03-13 17:08:11,091] jepsen worker 53 - jepsen.util 413 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,092] jepsen worker 53 - jepsen.util 413 :invoke :read [17 nil] INFO [2022-03-13 17:08:11,093] jepsen worker 53 - jepsen.util 413 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,093] jepsen worker 53 - jepsen.util 413 :invoke :read [17 nil] INFO [2022-03-13 17:08:11,096] jepsen worker 53 - jepsen.util 413 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,096] jepsen worker 53 - jepsen.util 413 :invoke :write [17 31] INFO [2022-03-13 17:08:11,178] jepsen worker 12 - jepsen.util 372 :info :write [4 36] :RequestTimeoutException INFO [2022-03-13 17:08:11,178] jepsen worker 12 - jepsen.util 462 :invoke :write [4 30] INFO [2022-03-13 17:08:11,741] jepsen worker 13 - jepsen.util 283 :info :write [4 3] :RequestTimeoutException INFO [2022-03-13 17:08:11,741] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:11,764] jepsen worker 45 - jepsen.util 585 :info :write [15 3] :RequestTimeoutException INFO [2022-03-13 17:08:11,765] jepsen worker 45 - jepsen.util 675 :invoke :write [15 37] INFO [2022-03-13 17:08:11,860] jepsen worker 31 - jepsen.util 391 :info :write [10 3] :RequestTimeoutException INFO [2022-03-13 17:08:11,861] jepsen worker 31 - jepsen.util 481 :invoke :write [10 40] INFO [2022-03-13 17:08:11,878] jepsen worker 33 - jepsen.util 393 :ok :read [11 :nil] INFO [2022-03-13 17:08:11,879] jepsen worker 7 - jepsen.util 277 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,879] jepsen worker 77 - jepsen.util 347 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,879] jepsen worker 51 - jepsen.util 231 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,879] jepsen worker 33 - jepsen.util 393 :invoke :write [11 43] INFO [2022-03-13 17:08:11,879] jepsen worker 23 - jepsen.util 293 :ok :read [7 :nil] INFO [2022-03-13 17:08:11,880] jepsen worker 77 - jepsen.util 347 :invoke :write [25 6] INFO [2022-03-13 17:08:11,880] jepsen worker 7 - jepsen.util 277 :invoke :read [2 nil] INFO [2022-03-13 17:08:11,881] jepsen worker 51 - jepsen.util 231 :invoke :read [17 nil] INFO [2022-03-13 17:08:11,886] jepsen worker 23 - jepsen.util 293 :invoke :write [7 3] INFO [2022-03-13 17:08:11,930] jepsen worker 7 - jepsen.util 277 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,930] jepsen worker 51 - jepsen.util 231 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:11,931] jepsen worker 7 - jepsen.util 277 :invoke :read [2 nil] INFO [2022-03-13 17:08:11,932] jepsen worker 51 - jepsen.util 231 :invoke :read [17 nil] INFO [2022-03-13 17:08:12,021] jepsen worker 7 - jepsen.util 277 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:12,021] jepsen worker 51 - jepsen.util 231 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:12,022] jepsen worker 51 - jepsen.util 231 :invoke :write [17 10] INFO [2022-03-13 17:08:12,022] jepsen worker 7 - jepsen.util 277 :invoke :write [2 47] INFO [2022-03-13 17:08:12,068] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:12,069] jepsen worker nemesis - couchbase.util Rebalance check count: 5 INFO [2022-03-13 17:08:12,069] jepsen worker nemesis - couchbase.util Rebalance stuck count: 4 INFO [2022-03-13 17:08:12,120] jepsen worker 47 - jepsen.util 497 :info :write [15 41] :RequestTimeoutException INFO [2022-03-13 17:08:12,121] jepsen worker 47 - jepsen.util 587 :invoke :write [15 11] INFO [2022-03-13 17:08:12,247] jepsen worker 8 - jepsen.util 458 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:12,247] jepsen worker 75 - jepsen.util 255 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:12,247] jepsen worker 43 - jepsen.util 313 :ok :read [14 :nil] INFO [2022-03-13 17:08:12,248] jepsen worker 8 - jepsen.util 458 :invoke :write [2 37] INFO [2022-03-13 17:08:12,248] jepsen worker 75 - jepsen.util 255 :invoke :write [25 27] INFO [2022-03-13 17:08:12,249] jepsen worker 43 - jepsen.util 313 :invoke :read [14 nil] INFO [2022-03-13 17:08:12,338] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:12,339] jepsen worker 25 - jepsen.util 385 :invoke :write [8 20] INFO [2022-03-13 17:08:12,384] jepsen worker 84 - jepsen.util 624 :info :write [28 19] :SyncWriteAmbiguous INFO [2022-03-13 17:08:12,385] jepsen worker 84 - jepsen.util 714 :invoke :read [28 nil] INFO [2022-03-13 17:08:12,504] jepsen worker 84 - jepsen.util 714 :ok :read [28 25] INFO [2022-03-13 17:08:12,505] jepsen worker 84 - jepsen.util 714 :invoke :write [28 35] INFO [2022-03-13 17:08:12,647] jepsen worker 37 - jepsen.util 487 :ok :read [12 46] INFO [2022-03-13 17:08:12,648] jepsen worker 37 - jepsen.util 487 :invoke :read [12 nil] INFO [2022-03-13 17:08:12,697] jepsen worker 49 - jepsen.util 499 :ok :read [16 :nil] INFO [2022-03-13 17:08:12,697] jepsen worker 49 - jepsen.util 499 :invoke :write [16 2] INFO [2022-03-13 17:08:12,703] jepsen worker 37 - jepsen.util 487 :ok :read [12 46] INFO [2022-03-13 17:08:12,703] jepsen worker 0 - jepsen.util 450 :ok :read [0 41] INFO [2022-03-13 17:08:12,704] jepsen worker 37 - jepsen.util 487 :invoke :read [12 nil] INFO [2022-03-13 17:08:12,704] jepsen worker 0 - jepsen.util 450 :invoke :read [0 nil] INFO [2022-03-13 17:08:12,710] jepsen worker 41 - jepsen.util 581 :ok :read [13 39] INFO [2022-03-13 17:08:12,711] jepsen worker 41 - jepsen.util 581 :invoke :read [13 nil] INFO [2022-03-13 17:08:12,754] jepsen worker 37 - jepsen.util 487 :ok :read [12 46] INFO [2022-03-13 17:08:12,755] jepsen worker 37 - jepsen.util 487 :invoke :write [12 23] INFO [2022-03-13 17:08:12,766] jepsen worker 40 - jepsen.util 490 :ok :read [13 39] INFO [2022-03-13 17:08:12,770] jepsen worker 40 - jepsen.util 490 :invoke :write [13 10] INFO [2022-03-13 17:08:12,778] jepsen worker 59 - jepsen.util 599 :ok :read [19 35] INFO [2022-03-13 17:08:12,779] jepsen worker 59 - jepsen.util 599 :invoke :read [19 nil] INFO [2022-03-13 17:08:12,811] jepsen worker 68 - jepsen.util 608 :ok :read [22 30] INFO [2022-03-13 17:08:12,813] jepsen worker 68 - jepsen.util 608 :invoke :write [22 40] INFO [2022-03-13 17:08:12,828] jepsen worker 2 - jepsen.util 632 :ok :read [0 41] INFO [2022-03-13 17:08:12,829] jepsen worker 2 - jepsen.util 632 :invoke :write [0 35] INFO [2022-03-13 17:08:12,875] jepsen worker 0 - jepsen.util 450 :ok :read [0 41] INFO [2022-03-13 17:08:12,876] jepsen worker 0 - jepsen.util 450 :invoke :read [0 nil] INFO [2022-03-13 17:08:12,887] jepsen worker 41 - jepsen.util 581 :ok :read [13 39] INFO [2022-03-13 17:08:12,888] jepsen worker 41 - jepsen.util 581 :invoke :read [13 nil] INFO [2022-03-13 17:08:12,911] jepsen worker 87 - jepsen.util 627 :info :write [29 14] :RequestTimeoutException INFO [2022-03-13 17:08:12,912] jepsen worker 87 - jepsen.util 717 :invoke :write [29 33] INFO [2022-03-13 17:08:12,929] jepsen worker 59 - jepsen.util 599 :ok :read [19 35] INFO [2022-03-13 17:08:12,930] jepsen worker 59 - jepsen.util 599 :invoke :write [19 26] INFO [2022-03-13 17:08:12,989] jepsen worker 43 - jepsen.util 313 :ok :read [14 :nil] INFO [2022-03-13 17:08:12,989] jepsen worker 43 - jepsen.util 313 :invoke :write [14 34] INFO [2022-03-13 17:08:13,065] jepsen worker 0 - jepsen.util 450 :ok :read [0 41] INFO [2022-03-13 17:08:13,066] jepsen worker 0 - jepsen.util 450 :invoke :write [0 15] INFO [2022-03-13 17:08:13,075] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:13,075] jepsen worker nemesis - couchbase.util Rebalance check count: 6 INFO [2022-03-13 17:08:13,075] jepsen worker nemesis - couchbase.util Rebalance stuck count: 5 INFO [2022-03-13 17:08:13,077] jepsen worker 41 - jepsen.util 581 :ok :read [13 39] INFO [2022-03-13 17:08:13,078] jepsen worker 41 - jepsen.util 581 :invoke :write [13 34] INFO [2022-03-13 17:08:13,281] jepsen worker 67 - jepsen.util 517 :info :write [22 34] :RequestTimeoutException INFO [2022-03-13 17:08:13,282] jepsen worker 67 - jepsen.util 607 :invoke :write [22 17] INFO [2022-03-13 17:08:13,301] jepsen worker 86 - jepsen.util 536 :info :write [28 12] :RequestTimeoutException INFO [2022-03-13 17:08:13,301] jepsen worker 86 - jepsen.util 626 :invoke :write [28 44] INFO [2022-03-13 17:08:13,530] jepsen worker 65 - jepsen.util 425 :info :write [21 25] :RequestTimeoutException INFO [2022-03-13 17:08:13,531] jepsen worker 65 - jepsen.util 515 :invoke :write [21 27] INFO [2022-03-13 17:08:13,615] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:13,616] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:13,736] jepsen worker 72 - jepsen.util 522 :info :write [24 31] :RequestTimeoutException INFO [2022-03-13 17:08:13,737] jepsen worker 72 - jepsen.util 612 :invoke :read [24 nil] INFO [2022-03-13 17:08:13,819] jepsen worker 16 - jepsen.util 466 :info :write [5 40] :RequestTimeoutException INFO [2022-03-13 17:08:13,819] jepsen worker 16 - jepsen.util 556 :invoke :write [5 7] INFO [2022-03-13 17:08:13,826] jepsen worker 58 - jepsen.util 508 :info :write [19 35] :RequestTimeoutException INFO [2022-03-13 17:08:13,826] jepsen worker 58 - jepsen.util 598 :invoke :read [19 nil] INFO [2022-03-13 17:08:13,857] jepsen worker 89 - jepsen.util 449 :info :write [29 23] :RequestTimeoutException INFO [2022-03-13 17:08:13,857] jepsen worker 89 - jepsen.util 539 :invoke :write [29 31] INFO [2022-03-13 17:08:13,872] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:13,872] jepsen worker 13 - jepsen.util 373 :invoke :write [4 28] INFO [2022-03-13 17:08:14,081] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:14,081] jepsen worker nemesis - couchbase.util Rebalance check count: 7 INFO [2022-03-13 17:08:14,082] jepsen worker nemesis - couchbase.util Rebalance stuck count: 6 INFO [2022-03-13 17:08:14,108] jepsen worker 20 - jepsen.util 560 :info :write [6 18] :RequestTimeoutException INFO [2022-03-13 17:08:14,109] jepsen worker 20 - jepsen.util 650 :invoke :write [6 49] INFO [2022-03-13 17:08:14,140] jepsen worker 9 - jepsen.util 459 :info :write [3 24] :RequestTimeoutException INFO [2022-03-13 17:08:14,141] jepsen worker 9 - jepsen.util 549 :invoke :write [3 49] INFO [2022-03-13 17:08:14,148] jepsen worker 4 - jepsen.util 544 :info :write [1 1] :RequestTimeoutException INFO [2022-03-13 17:08:14,148] jepsen worker 4 - jepsen.util 634 :invoke :read [1 nil] INFO [2022-03-13 17:08:14,234] jepsen worker 32 - jepsen.util 392 :info :write [10 23] :RequestTimeoutException INFO [2022-03-13 17:08:14,235] jepsen worker 32 - jepsen.util 482 :invoke :read [10 nil] INFO [2022-03-13 17:08:14,375] jepsen worker 26 - jepsen.util 296 :info :write [8 4] :SyncWriteAmbiguous INFO [2022-03-13 17:08:14,376] jepsen worker 81 - jepsen.util 621 :info :write [27 48] :RequestTimeoutException INFO [2022-03-13 17:08:14,376] jepsen worker 26 - jepsen.util 386 :invoke :write [8 0] INFO [2022-03-13 17:08:14,377] jepsen worker 81 - jepsen.util 711 :invoke :write [27 8] INFO [2022-03-13 17:08:14,406] jepsen worker 42 - jepsen.util 312 :info :write [14 30] :RequestTimeoutException INFO [2022-03-13 17:08:14,407] jepsen worker 42 - jepsen.util 402 :invoke :write [14 46] INFO [2022-03-13 17:08:14,477] jepsen worker 10 - jepsen.util 280 :info :write [3 27] :RequestTimeoutException INFO [2022-03-13 17:08:14,477] jepsen worker 10 - jepsen.util 370 :invoke :write [3 27] INFO [2022-03-13 17:08:14,597] jepsen worker 71 - jepsen.util 251 :info :write [23 26] :RequestTimeoutException INFO [2022-03-13 17:08:14,597] jepsen worker 71 - jepsen.util 341 :invoke :write [23 31] INFO [2022-03-13 17:08:14,646] jepsen worker 21 - jepsen.util 471 :info :write [7 13] :RequestTimeoutException INFO [2022-03-13 17:08:14,647] jepsen worker 21 - jepsen.util 561 :invoke :write [7 29] INFO [2022-03-13 17:08:14,657] jepsen worker 69 - jepsen.util 339 :info :write [23 37] :RequestTimeoutException INFO [2022-03-13 17:08:14,657] jepsen worker 69 - jepsen.util 429 :invoke :write [23 44] INFO [2022-03-13 17:08:14,831] jepsen worker 62 - jepsen.util 332 :ok :read [20 10] INFO [2022-03-13 17:08:14,831] jepsen worker 62 - jepsen.util 332 :invoke :read [20 nil] INFO [2022-03-13 17:08:14,946] jepsen worker 38 - jepsen.util 488 :info :write [12 45] :RequestTimeoutException INFO [2022-03-13 17:08:14,947] jepsen worker 38 - jepsen.util 578 :invoke :write [12 22] INFO [2022-03-13 17:08:15,091] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:15,091] jepsen worker nemesis - couchbase.util Rebalance check count: 8 INFO [2022-03-13 17:08:15,091] jepsen worker nemesis - couchbase.util Rebalance stuck count: 7 INFO [2022-03-13 17:08:15,137] jepsen worker 5 - jepsen.util 455 :info :write [1 33] :RequestTimeoutException INFO [2022-03-13 17:08:15,138] jepsen worker 5 - jepsen.util 545 :invoke :write [1 27] INFO [2022-03-13 17:08:15,287] jepsen worker 27 - jepsen.util 477 :info :write [9 28] :RequestTimeoutException WARN [2022-03-13 17:08:15,288] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":277213,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1358942,"total_server_duration_us":325998,"encode_duration_us":18993,"operation_id":"0x1410","timeout_ms":10000,"last_server_duration_us":48679,"total_duration_us":10010511},{"operation_name":"upsert","last_dispatch_duration_us":805337,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9265484,"total_server_duration_us":1056806,"encode_duration_us":17891,"operation_id":"0x1358","timeout_ms":10000,"last_server_duration_us":77992,"total_duration_us":10010187},{"operation_name":"upsert","last_dispatch_duration_us":7495546,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9166981,"total_server_duration_us":528382,"encode_duration_us":18813,"operation_id":"0x138b","timeout_ms":10000,"last_server_duration_us":62668,"total_duration_us":10010049},{"operation_name":"upsert","last_dispatch_duration_us":9130830,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10732425,"total_server_duration_us":9429151,"encode_duration_us":24797,"operation_id":"0x1379","timeout_ms":10000,"last_server_duration_us":9068168,"total_duration_us":10009330},{"operation_name":"upsert","last_dispatch_duration_us":7512995,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9377447,"total_server_duration_us":550894,"encode_duration_us":20918,"operation_id":"0x1383","timeout_ms":10000,"last_server_duration_us":47876,"total_duration_us":10008957},{"operation_name":"upsert","last_dispatch_duration_us":76126,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":8429423,"total_server_duration_us":397652,"encode_duration_us":23392,"operation_id":"0x13cb","timeout_ms":10000,"last_server_duration_us":26507,"total_duration_us":10007510},{"operation_name":"upsert","last_dispatch_duration_us":184229,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":5826987,"total_server_duration_us":406442,"encode_duration_us":17322,"operation_id":"0x1301","timeout_ms":10000,"last_server_duration_us":55790,"total_duration_us":10006847},{"operation_name":"upsert","last_dispatch_duration_us":9097562,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9097562,"total_server_duration_us":9049045,"encode_duration_us":18234,"operation_id":"0x13b6","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10006650},{"operation_name":"upsert","last_dispatch_duration_us":73071,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":5591089,"total_server_duration_us":449600,"encode_duration_us":18422,"operation_id":"0x1327","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10005707},{"operation_name":"upsert","last_dispatch_duration_us":7214053,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8411728,"total_server_duration_us":395176,"encode_duration_us":20138,"operation_id":"0x140a","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10004862}],"total_count":20}} INFO [2022-03-13 17:08:15,288] jepsen worker 27 - jepsen.util 567 :invoke :read [9 nil] WARN [2022-03-13 17:08:15,288] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13811227,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14948690,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9590722,"encode_duration_us":19953,"operation_id":"0x12a6","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10001398},{"operation_name":"upsert","last_dispatch_duration_us":9097562,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9097562,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9049045,"encode_duration_us":18234,"operation_id":"0x13b6","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10006650},{"operation_name":"upsert","last_dispatch_duration_us":13019797,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14058648,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":8587753,"encode_duration_us":17932,"operation_id":"0x12b9","timeout_ms":10000,"last_server_duration_us":8179878,"total_duration_us":10009037},{"operation_name":"upsert","last_dispatch_duration_us":9130830,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10732425,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":9429151,"encode_duration_us":24797,"operation_id":"0x1379","timeout_ms":10000,"last_server_duration_us":9068168,"total_duration_us":10009330}],"total_count":4}} INFO [2022-03-13 17:08:15,334] jepsen worker 63 - jepsen.util 513 :info :write [21 10] :RequestTimeoutException INFO [2022-03-13 17:08:15,334] jepsen worker 63 - jepsen.util 603 :invoke :write [21 48] INFO [2022-03-13 17:08:15,440] jepsen worker 64 - jepsen.util 334 :info :write [21 47] :RequestTimeoutException INFO [2022-03-13 17:08:15,441] jepsen worker 64 - jepsen.util 424 :invoke :read [21 nil] INFO [2022-03-13 17:08:15,688] jepsen worker 46 - jepsen.util 496 :info :write [15 37] :RequestTimeoutException INFO [2022-03-13 17:08:15,689] jepsen worker 46 - jepsen.util 586 :invoke :read [15 nil] INFO [2022-03-13 17:08:15,854] jepsen worker 85 - jepsen.util 625 :info :write [28 21] :RequestTimeoutException INFO [2022-03-13 17:08:15,855] jepsen worker 85 - jepsen.util 715 :invoke :read [28 nil] INFO [2022-03-13 17:08:15,984] jepsen worker 44 - jepsen.util 314 :info :write [14 33] :RequestTimeoutException INFO [2022-03-13 17:08:15,985] jepsen worker 44 - jepsen.util 404 :invoke :write [14 13] INFO [2022-03-13 17:08:15,990] jepsen worker 19 - jepsen.util 379 :info :write [6 40] :SyncWriteAmbiguous INFO [2022-03-13 17:08:15,991] jepsen worker 19 - jepsen.util 469 :invoke :read [6 nil] INFO [2022-03-13 17:08:16,015] jepsen worker 56 - jepsen.util 506 :ok :write [18 3] INFO [2022-03-13 17:08:16,016] jepsen worker 56 - jepsen.util 506 :invoke :write [18 29] INFO [2022-03-13 17:08:16,023] jepsen worker 34 - jepsen.util 304 :info :write [11 22] :RequestTimeoutException INFO [2022-03-13 17:08:16,024] jepsen worker 34 - jepsen.util 394 :invoke :read [11 nil] INFO [2022-03-13 17:08:16,095] jepsen worker 34 - jepsen.util 394 :ok :read [11 :nil] INFO [2022-03-13 17:08:16,095] jepsen worker 34 - jepsen.util 394 :invoke :read [11 nil] INFO [2022-03-13 17:08:16,097] jepsen worker 34 - jepsen.util 394 :ok :read [11 :nil] INFO [2022-03-13 17:08:16,098] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:16,098] jepsen worker nemesis - couchbase.util Rebalance check count: 9 INFO [2022-03-13 17:08:16,098] jepsen worker nemesis - couchbase.util Rebalance stuck count: 8 INFO [2022-03-13 17:08:16,098] jepsen worker 34 - jepsen.util 394 :invoke :read [11 nil] INFO [2022-03-13 17:08:16,099] jepsen worker 34 - jepsen.util 394 :ok :read [11 :nil] INFO [2022-03-13 17:08:16,100] jepsen worker 34 - jepsen.util 394 :invoke :write [11 34] WARN [2022-03-13 17:08:16,791] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9038724,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11123698,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9415208,"encode_duration_us":27039,"operation_id":"0x1336","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10002163},{"operation_name":"upsert","last_dispatch_duration_us":9303713,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10957905,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9691268,"encode_duration_us":20009,"operation_id":"0x1362","timeout_ms":10000,"last_server_duration_us":9244265,"total_duration_us":10008925},{"operation_name":"upsert","last_dispatch_duration_us":9172824,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9814947,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9399201,"encode_duration_us":19421,"operation_id":"0x13bd","timeout_ms":10000,"last_server_duration_us":9069231,"total_duration_us":10010157},{"operation_name":"upsert","last_dispatch_duration_us":8614261,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9616412,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":8805848,"encode_duration_us":16981,"operation_id":"0x1302","timeout_ms":10000,"last_server_duration_us":8601481,"total_duration_us":10009085}],"total_count":4}} INFO [2022-03-13 17:08:16,886] jepsen worker 18 - jepsen.util 468 :ok :read [6 :nil] INFO [2022-03-13 17:08:16,887] jepsen worker 18 - jepsen.util 468 :invoke :read [6 nil] WARN [2022-03-13 17:08:16,889] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16646233,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17736148,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9192529,"encode_duration_us":19946,"operation_id":"0x1338","timeout_ms":10000,"last_server_duration_us":9103272,"total_duration_us":10003419}],"total_count":1}} WARN [2022-03-13 17:08:16,889] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9376806,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11410787,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9418088,"encode_duration_us":25468,"operation_id":"0x1475","timeout_ms":10000,"last_server_duration_us":9144833,"total_duration_us":10003652},{"operation_name":"upsert","last_dispatch_duration_us":11155324,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12911266,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":4513214,"encode_duration_us":17410,"operation_id":"0x1217","timeout_ms":10000,"last_server_duration_us":4145525,"total_duration_us":10005576}],"total_count":2}} WARN [2022-03-13 17:08:16,889] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16582137,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16582137,"total_server_duration_us":158588,"operation_id":"0x1304","timeout_ms":30000,"last_server_duration_us":158588,"total_duration_us":16584672},{"operation_name":"get","last_dispatch_duration_us":14081582,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":14081582,"total_server_duration_us":56,"operation_id":"0x1363","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":14081971},{"operation_name":"get","last_dispatch_duration_us":13666783,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":13666783,"total_server_duration_us":84,"operation_id":"0x1374","timeout_ms":30000,"last_server_duration_us":84,"total_duration_us":13667815},{"operation_name":"get","last_dispatch_duration_us":13394223,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":13394223,"total_server_duration_us":108,"operation_id":"0x1388","timeout_ms":30000,"last_server_duration_us":108,"total_duration_us":13394539},{"operation_name":"get","last_dispatch_duration_us":12466647,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":12466647,"total_server_duration_us":32,"operation_id":"0x13ac","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":12468177},{"operation_name":"get","last_dispatch_duration_us":9047550,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9047550,"total_server_duration_us":23,"operation_id":"0x13c1","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":11667908},{"operation_name":"upsert","last_dispatch_duration_us":13072228,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":13072228,"total_server_duration_us":22141,"encode_duration_us":19268,"operation_id":"0x1392","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10009861},{"operation_name":"upsert","last_dispatch_duration_us":9074677,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9074677,"total_server_duration_us":26153,"encode_duration_us":24781,"operation_id":"0x13e3","timeout_ms":10000,"last_server_duration_us":26153,"total_duration_us":10005327},{"operation_name":"upsert","last_dispatch_duration_us":46778,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":3088638,"total_server_duration_us":1004750,"encode_duration_us":21658,"operation_id":"0x143b","timeout_ms":10000,"last_server_duration_us":27764,"total_duration_us":10002110},{"operation_name":"upsert","last_dispatch_duration_us":30322,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":3456924,"total_server_duration_us":1800641,"encode_duration_us":18567,"operation_id":"0x141a","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10001640}],"total_count":17}} WARN [2022-03-13 17:08:16,892] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":2825565,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9039136,"total_server_duration_us":487747,"encode_duration_us":18707,"operation_id":"0x1339","timeout_ms":10000,"last_server_duration_us":43618,"total_duration_us":10011198},{"operation_name":"upsert","last_dispatch_duration_us":194102,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5841530,"total_server_duration_us":452916,"encode_duration_us":23085,"operation_id":"0x1342","timeout_ms":10000,"last_server_duration_us":43948,"total_duration_us":10010294},{"operation_name":"upsert","last_dispatch_duration_us":9172824,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9814947,"total_server_duration_us":9399201,"encode_duration_us":19421,"operation_id":"0x13bd","timeout_ms":10000,"last_server_duration_us":9069231,"total_duration_us":10010157},{"operation_name":"upsert","last_dispatch_duration_us":1483351,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":1483351,"total_server_duration_us":51121,"encode_duration_us":10776,"operation_id":"0x148b","timeout_ms":10000,"last_server_duration_us":51121,"total_duration_us":10009795},{"operation_name":"upsert","last_dispatch_duration_us":9303713,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10957905,"total_server_duration_us":9691268,"encode_duration_us":20009,"operation_id":"0x1362","timeout_ms":10000,"last_server_duration_us":9244265,"total_duration_us":10008925},{"operation_name":"upsert","last_dispatch_duration_us":1043423,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5872780,"total_server_duration_us":505434,"encode_duration_us":17063,"operation_id":"0x1390","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10008193},{"operation_name":"upsert","last_dispatch_duration_us":11027145,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11348707,"total_server_duration_us":126630,"encode_duration_us":18631,"operation_id":"0x143c","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10006934},{"operation_name":"upsert","last_dispatch_duration_us":2874214,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":7541460,"total_server_duration_us":418478,"encode_duration_us":20250,"operation_id":"0x13a5","timeout_ms":10000,"last_server_duration_us":75894,"total_duration_us":10006153},{"operation_name":"upsert","last_dispatch_duration_us":4581300,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10404831,"total_server_duration_us":461833,"encode_duration_us":17970,"operation_id":"0x13c6","timeout_ms":10000,"last_server_duration_us":43618,"total_duration_us":10005079},{"operation_name":"upsert","last_dispatch_duration_us":38114,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":582161,"total_server_duration_us":204758,"encode_duration_us":18603,"operation_id":"0x1434","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10005077}],"total_count":16}} INFO [2022-03-13 17:08:16,932] jepsen worker 4 - jepsen.util 634 :ok :read [1 16] INFO [2022-03-13 17:08:16,933] jepsen worker 4 - jepsen.util 634 :invoke :read [1 nil] WARN [2022-03-13 17:08:16,992] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13311959,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":13311959,"total_server_duration_us":32,"operation_id":"0x1372","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":13312406},{"operation_name":"get","last_dispatch_duration_us":13192217,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":13192217,"total_server_duration_us":15,"operation_id":"0x1381","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":13192606},{"operation_name":"get","last_dispatch_duration_us":12140615,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":12140615,"total_server_duration_us":8,"operation_id":"0x13a6","timeout_ms":30000,"last_server_duration_us":8,"total_duration_us":12140870},{"operation_name":"get","last_dispatch_duration_us":10918367,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10918367,"total_server_duration_us":15,"operation_id":"0x13c8","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":10918619},{"operation_name":"upsert","last_dispatch_duration_us":8514447,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9480800,"total_server_duration_us":328648,"encode_duration_us":17026,"operation_id":"0x1474","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10010228},{"operation_name":"upsert","last_dispatch_duration_us":8558239,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10239117,"total_server_duration_us":547178,"encode_duration_us":22690,"operation_id":"0x1435","timeout_ms":10000,"last_server_duration_us":26064,"total_duration_us":10007136},{"operation_name":"upsert","last_dispatch_duration_us":9121704,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10587496,"total_server_duration_us":9446448,"encode_duration_us":24043,"operation_id":"0x1436","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10006000},{"operation_name":"upsert","last_dispatch_duration_us":8644993,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9566643,"total_server_duration_us":292813,"encode_duration_us":17047,"operation_id":"0x1471","timeout_ms":10000,"last_server_duration_us":29415,"total_duration_us":10003986},{"operation_name":"upsert","last_dispatch_duration_us":158995,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9675987,"total_server_duration_us":508519,"encode_duration_us":19748,"operation_id":"0x1346","timeout_ms":10000,"last_server_duration_us":30823,"total_duration_us":10003961},{"operation_name":"upsert","last_dispatch_duration_us":9648453,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9648453,"total_server_duration_us":9071357,"encode_duration_us":17191,"operation_id":"0x1497","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":9652729}],"total_count":12}} WARN [2022-03-13 17:08:16,993] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17291868,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":17605403,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":9088522,"encode_duration_us":16809,"operation_id":"0x11e0","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10010697},{"operation_name":"upsert","last_dispatch_duration_us":17912012,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19113202,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9467812,"encode_duration_us":20180,"operation_id":"0x128c","timeout_ms":10000,"last_server_duration_us":9210001,"total_duration_us":10011002}],"total_count":2}} WARN [2022-03-13 17:08:16,993] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10447402,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10447402,"total_server_duration_us":32,"operation_id":"0x1319","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":10450576},{"operation_name":"upsert","last_dispatch_duration_us":7924498,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9701260,"total_server_duration_us":335686,"encode_duration_us":18940,"operation_id":"0x1467","timeout_ms":10000,"last_server_duration_us":72862,"total_duration_us":10009860},{"operation_name":"upsert","last_dispatch_duration_us":127439,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1752708,"total_server_duration_us":293271,"encode_duration_us":17986,"operation_id":"0x1448","timeout_ms":10000,"last_server_duration_us":47533,"total_duration_us":10008941},{"operation_name":"upsert","last_dispatch_duration_us":7205539,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":11418546,"total_server_duration_us":1006120,"encode_duration_us":18703,"operation_id":"0x142d","timeout_ms":10000,"last_server_duration_us":29601,"total_duration_us":10008074},{"operation_name":"upsert","last_dispatch_duration_us":7862053,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9579404,"total_server_duration_us":409531,"encode_duration_us":19860,"operation_id":"0x145e","timeout_ms":10000,"last_server_duration_us":50769,"total_duration_us":10008026},{"operation_name":"upsert","last_dispatch_duration_us":7001353,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10707043,"total_server_duration_us":1409925,"encode_duration_us":17391,"operation_id":"0x144a","timeout_ms":10000,"last_server_duration_us":33230,"total_duration_us":10004498},{"operation_name":"upsert","last_dispatch_duration_us":9362235,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9953715,"total_server_duration_us":9251100,"encode_duration_us":30736,"operation_id":"0x1444","timeout_ms":10000,"last_server_duration_us":9111791,"total_duration_us":10004331},{"operation_name":"upsert","last_dispatch_duration_us":281426,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9274367,"total_server_duration_us":896712,"encode_duration_us":21570,"operation_id":"0x1332","timeout_ms":10000,"last_server_duration_us":49951,"total_duration_us":10004194},{"operation_name":"upsert","last_dispatch_duration_us":9376806,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":11410787,"total_server_duration_us":9418088,"encode_duration_us":25468,"operation_id":"0x1475","timeout_ms":10000,"last_server_duration_us":9144833,"total_duration_us":10003652},{"operation_name":"upsert","last_dispatch_duration_us":7792923,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9497749,"total_server_duration_us":379241,"encode_duration_us":19737,"operation_id":"0x1466","timeout_ms":10000,"last_server_duration_us":99346,"total_duration_us":10000927}],"total_count":15}} INFO [2022-03-13 17:08:17,010] jepsen worker 4 - jepsen.util 634 :ok :read [1 16] INFO [2022-03-13 17:08:17,011] jepsen worker 4 - jepsen.util 634 :invoke :read [1 nil] INFO [2022-03-13 17:08:17,078] jepsen worker 4 - jepsen.util 634 :ok :read [1 16] INFO [2022-03-13 17:08:17,078] jepsen worker 4 - jepsen.util 634 :invoke :read [1 nil] WARN [2022-03-13 17:08:17,090] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":492573,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":3764229,"total_server_duration_us":251124,"encode_duration_us":17771,"operation_id":"0x1440","timeout_ms":10000,"last_server_duration_us":58750,"total_duration_us":10008141},{"operation_name":"upsert","last_dispatch_duration_us":3300070,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":11620684,"total_server_duration_us":183014,"encode_duration_us":15104,"operation_id":"0x1360","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10008119},{"operation_name":"upsert","last_dispatch_duration_us":6839057,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6839057,"total_server_duration_us":37859,"encode_duration_us":17404,"operation_id":"0x1393","timeout_ms":10000,"last_server_duration_us":37859,"total_duration_us":10007924},{"operation_name":"upsert","last_dispatch_duration_us":8162192,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8387178,"total_server_duration_us":147424,"encode_duration_us":17035,"operation_id":"0x13ad","timeout_ms":10000,"last_server_duration_us":35815,"total_duration_us":10007513},{"operation_name":"upsert","last_dispatch_duration_us":3428012,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3428012,"total_server_duration_us":22306,"encode_duration_us":20919,"operation_id":"0x1495","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10006479},{"operation_name":"upsert","last_dispatch_duration_us":68480,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":245062,"total_server_duration_us":113592,"encode_duration_us":28735,"operation_id":"0x1445","timeout_ms":10000,"last_server_duration_us":41337,"total_duration_us":10006427},{"operation_name":"upsert","last_dispatch_duration_us":3375490,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3894632,"total_server_duration_us":90028,"encode_duration_us":18172,"operation_id":"0x1493","timeout_ms":10000,"last_server_duration_us":43289,"total_duration_us":10005531},{"operation_name":"upsert","last_dispatch_duration_us":7146062,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":8231680,"total_server_duration_us":131437,"encode_duration_us":19584,"operation_id":"0x1341","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10005398},{"operation_name":"upsert","last_dispatch_duration_us":1372906,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":4619613,"total_server_duration_us":778264,"encode_duration_us":20264,"operation_id":"0x1335","timeout_ms":10000,"last_server_duration_us":78414,"total_duration_us":10004368},{"operation_name":"upsert","last_dispatch_duration_us":149164,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":3276875,"total_server_duration_us":255120,"encode_duration_us":20304,"operation_id":"0x1476","timeout_ms":10000,"last_server_duration_us":42200,"total_duration_us":10004169}],"total_count":17}} WARN [2022-03-13 17:08:17,093] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15007434,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":15789488,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9429034,"encode_duration_us":17441,"operation_id":"0x12b6","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10003021},{"operation_name":"upsert","last_dispatch_duration_us":9121704,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10587496,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9446448,"encode_duration_us":24043,"operation_id":"0x1436","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10006000},{"operation_name":"upsert","last_dispatch_duration_us":11766978,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12380834,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":5834441,"encode_duration_us":18372,"operation_id":"0x11b3","timeout_ms":10000,"last_server_duration_us":5661477,"total_duration_us":10009396},{"operation_name":"upsert","last_dispatch_duration_us":15124815,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":15686493,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9204182,"encode_duration_us":22437,"operation_id":"0x11bb","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10127841}],"total_count":4}} INFO [2022-03-13 17:08:17,106] jepsen worker 57 - jepsen.util 507 :info :write [19 44] :RequestTimeoutException INFO [2022-03-13 17:08:17,106] jepsen worker 55 - jepsen.util 505 :info :write [18 10] :RequestTimeoutException INFO [2022-03-13 17:08:17,107] jepsen worker 57 - jepsen.util 597 :invoke :write [19 10] INFO [2022-03-13 17:08:17,108] jepsen worker 55 - jepsen.util 595 :invoke :write [18 18] INFO [2022-03-13 17:08:17,151] jepsen worker 4 - jepsen.util 634 :ok :read [1 16] INFO [2022-03-13 17:08:17,152] jepsen worker 4 - jepsen.util 634 :invoke :write [1 16] INFO [2022-03-13 17:08:17,156] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:17,156] jepsen worker nemesis - couchbase.util Rebalance check count: 10 INFO [2022-03-13 17:08:17,156] jepsen worker nemesis - couchbase.util Rebalance stuck count: 9 INFO [2022-03-13 17:08:17,157] jepsen worker 17 - jepsen.util 557 :info :write [5 33] :RequestTimeoutException INFO [2022-03-13 17:08:17,158] jepsen worker 17 - jepsen.util 647 :invoke :read [5 nil] INFO [2022-03-13 17:08:17,167] jepsen worker 19 - jepsen.util 469 :ok :read [6 :nil] INFO [2022-03-13 17:08:17,169] jepsen worker 19 - jepsen.util 469 :invoke :write [6 16] INFO [2022-03-13 17:08:17,196] jepsen worker 66 - jepsen.util 516 :info :write [22 13] :RequestTimeoutException INFO [2022-03-13 17:08:17,197] jepsen worker 66 - jepsen.util 606 :invoke :write [22 21] INFO [2022-03-13 17:08:17,311] jepsen worker 74 - jepsen.util 614 :info :write [24 22] :RequestTimeoutException INFO [2022-03-13 17:08:17,312] jepsen worker 74 - jepsen.util 704 :invoke :read [24 nil] INFO [2022-03-13 17:08:17,364] jepsen worker 73 - jepsen.util 613 :info :write [24 7] :RequestTimeoutException INFO [2022-03-13 17:08:17,365] jepsen worker 73 - jepsen.util 703 :invoke :read [24 nil] INFO [2022-03-13 17:08:17,376] jepsen worker 83 - jepsen.util 443 :info :write [27 25] :RequestTimeoutException INFO [2022-03-13 17:08:17,377] jepsen worker 83 - jepsen.util 533 :invoke :read [27 nil] INFO [2022-03-13 17:08:17,436] jepsen worker 15 - jepsen.util 555 :info :write [5 38] :RequestTimeoutException INFO [2022-03-13 17:08:17,437] jepsen worker 15 - jepsen.util 645 :invoke :read [5 nil] INFO [2022-03-13 17:08:17,482] jepsen worker 74 - jepsen.util 704 :ok :read [24 24] INFO [2022-03-13 17:08:17,483] jepsen worker 74 - jepsen.util 704 :invoke :write [24 10] INFO [2022-03-13 17:08:17,488] jepsen worker 78 - jepsen.util 348 :info :write [26 0] :RequestTimeoutException INFO [2022-03-13 17:08:17,489] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:17,561] jepsen worker 35 - jepsen.util 575 :info :write [11 5] :RequestTimeoutException INFO [2022-03-13 17:08:17,561] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:17,574] jepsen worker 11 - jepsen.util 461 :info :write [3 49] :RequestTimeoutException INFO [2022-03-13 17:08:17,575] jepsen worker 11 - jepsen.util 551 :invoke :write [3 31] INFO [2022-03-13 17:08:17,739] jepsen worker 27 - jepsen.util 567 :ok :read [9 30] INFO [2022-03-13 17:08:17,740] jepsen worker 27 - jepsen.util 567 :invoke :read [9 nil] INFO [2022-03-13 17:08:17,816] jepsen worker 36 - jepsen.util 486 :info :write [12 46] :RequestTimeoutException INFO [2022-03-13 17:08:17,817] jepsen worker 36 - jepsen.util 576 :invoke :read [12 nil] INFO [2022-03-13 17:08:17,826] jepsen worker 28 - jepsen.util 568 :info :write [9 20] :RequestTimeoutException INFO [2022-03-13 17:08:17,827] jepsen worker 28 - jepsen.util 658 :invoke :read [9 nil] INFO [2022-03-13 17:08:17,844] jepsen worker 36 - jepsen.util 576 :ok :read [12 46] INFO [2022-03-13 17:08:17,845] jepsen worker 36 - jepsen.util 576 :invoke :read [12 nil] INFO [2022-03-13 17:08:17,876] jepsen worker 36 - jepsen.util 576 :ok :read [12 46] INFO [2022-03-13 17:08:17,876] jepsen worker 36 - jepsen.util 576 :invoke :write [12 48] INFO [2022-03-13 17:08:17,940] jepsen worker 27 - jepsen.util 567 :ok :read [9 30] INFO [2022-03-13 17:08:17,941] jepsen worker 27 - jepsen.util 567 :invoke :write [9 27] INFO [2022-03-13 17:08:18,040] jepsen worker 60 - jepsen.util 510 :info :write [20 35] :RequestTimeoutException INFO [2022-03-13 17:08:18,041] jepsen worker 60 - jepsen.util 600 :invoke :write [20 12] INFO [2022-03-13 17:08:18,163] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:18,163] jepsen worker nemesis - couchbase.util Rebalance check count: 11 INFO [2022-03-13 17:08:18,163] jepsen worker nemesis - couchbase.util Rebalance stuck count: 10 INFO [2022-03-13 17:08:18,637] jepsen worker 80 - jepsen.util 530 :info :write [26 16] :RequestTimeoutException INFO [2022-03-13 17:08:18,637] jepsen worker 80 - jepsen.util 620 :invoke :write [26 49] INFO [2022-03-13 17:08:18,664] jepsen worker 29 - jepsen.util 569 :info :write [9 27] :RequestTimeoutException INFO [2022-03-13 17:08:18,673] jepsen worker 29 - jepsen.util 659 :invoke :read [9 nil] INFO [2022-03-13 17:08:18,696] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:18,697] jepsen worker 14 - jepsen.util 194 :invoke :write [4 42] INFO [2022-03-13 17:08:18,719] jepsen worker 29 - jepsen.util 659 :ok :read [9 30] INFO [2022-03-13 17:08:18,720] jepsen worker 29 - jepsen.util 659 :invoke :write [9 10] INFO [2022-03-13 17:08:18,736] jepsen worker 24 - jepsen.util 294 :info :write [8 43] :RequestTimeoutException INFO [2022-03-13 17:08:18,737] jepsen worker 24 - jepsen.util 384 :invoke :write [8 26] INFO [2022-03-13 17:08:18,833] jepsen worker 61 - jepsen.util 421 :ok :read [20 10] INFO [2022-03-13 17:08:18,834] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:19,105] jepsen worker 35 - jepsen.util 665 :ok :read [11 :nil] INFO [2022-03-13 17:08:19,106] jepsen worker 35 - jepsen.util 665 :invoke :write [11 17] INFO [2022-03-13 17:08:19,170] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:19,170] jepsen worker nemesis - couchbase.util Rebalance check count: 12 INFO [2022-03-13 17:08:19,170] jepsen worker nemesis - couchbase.util Rebalance stuck count: 11 INFO [2022-03-13 17:08:19,190] jepsen worker 6 - jepsen.util 456 :info :write [2 41] :RequestTimeoutException INFO [2022-03-13 17:08:19,191] jepsen worker 6 - jepsen.util 546 :invoke :write [2 31] INFO [2022-03-13 17:08:19,294] jepsen worker 70 - jepsen.util 520 :info :write [23 31] :RequestTimeoutException INFO [2022-03-13 17:08:19,295] jepsen worker 70 - jepsen.util 610 :invoke :write [23 48] INFO [2022-03-13 17:08:19,358] jepsen worker 61 - jepsen.util 421 :ok :read [20 10] INFO [2022-03-13 17:08:19,359] jepsen worker 61 - jepsen.util 421 :invoke :write [20 0] INFO [2022-03-13 17:08:19,401] jepsen worker 79 - jepsen.util 619 :info :write [26 17] :SyncWriteAmbiguous INFO [2022-03-13 17:08:19,402] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:19,567] jepsen worker 82 - jepsen.util 622 :info :write [27 8] :RequestTimeoutException INFO [2022-03-13 17:08:19,571] jepsen worker 82 - jepsen.util 712 :invoke :read [27 nil] INFO [2022-03-13 17:08:19,580] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:19,581] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:19,714] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:19,714] jepsen worker 79 - jepsen.util 709 :invoke :write [26 49] INFO [2022-03-13 17:08:19,825] jepsen worker 3 - jepsen.util 543 :ok :read [1 16] INFO [2022-03-13 17:08:19,826] jepsen worker 3 - jepsen.util 543 :invoke :read [1 nil] INFO [2022-03-13 17:08:19,875] jepsen worker 48 - jepsen.util 498 :ok :read [16 :nil] INFO [2022-03-13 17:08:19,876] jepsen worker 48 - jepsen.util 498 :invoke :read [16 nil] INFO [2022-03-13 17:08:19,921] jepsen worker 76 - jepsen.util 436 :info :write [25 21] :RequestTimeoutException INFO [2022-03-13 17:08:19,922] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:19,999] jepsen worker 88 - jepsen.util 628 :info :write [29 7] :RequestTimeoutException INFO [2022-03-13 17:08:19,999] jepsen worker 88 - jepsen.util 718 :invoke :write [29 6] INFO [2022-03-13 17:08:20,000] jepsen worker 22 - jepsen.util 382 :info :write [7 19] :RequestTimeoutException INFO [2022-03-13 17:08:20,002] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:20,043] jepsen worker 58 - jepsen.util 598 :ok :read [19 35] INFO [2022-03-13 17:08:20,043] jepsen worker 58 - jepsen.util 598 :invoke :read [19 nil] INFO [2022-03-13 17:08:20,176] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:20,176] jepsen worker nemesis - couchbase.util Rebalance check count: 13 INFO [2022-03-13 17:08:20,176] jepsen worker nemesis - couchbase.util Rebalance stuck count: 12 INFO [2022-03-13 17:08:20,280] jepsen worker 1 - jepsen.util 541 :info :write [0 22] :RequestTimeoutException INFO [2022-03-13 17:08:20,281] jepsen worker 1 - jepsen.util 631 :invoke :write [0 8] INFO [2022-03-13 17:08:20,385] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,386] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:20,564] jepsen worker 50 - jepsen.util 590 :info :write [16 38] :RequestTimeoutException INFO [2022-03-13 17:08:20,565] jepsen worker 50 - jepsen.util 680 :invoke :read [16 nil] INFO [2022-03-13 17:08:20,646] jepsen worker 39 - jepsen.util 669 :info :write [13 37] :RequestTimeoutException INFO [2022-03-13 17:08:20,647] jepsen worker 39 - jepsen.util 759 :invoke :write [13 2] INFO [2022-03-13 17:08:20,690] jepsen worker 12 - jepsen.util 462 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,690] jepsen worker 61 - jepsen.util 421 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,691] jepsen worker 12 - jepsen.util 462 :invoke :write [4 1] INFO [2022-03-13 17:08:20,691] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:20,780] jepsen worker 79 - jepsen.util 709 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,781] jepsen worker 42 - jepsen.util 402 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,782] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,782] jepsen worker 79 - jepsen.util 709 :invoke :write [26 8] INFO [2022-03-13 17:08:20,783] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:20,783] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:08:20,819] jepsen worker 44 - jepsen.util 404 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,820] jepsen worker 44 - jepsen.util 404 :invoke :write [14 26] INFO [2022-03-13 17:08:20,857] jepsen worker 35 - jepsen.util 665 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,858] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:20,895] jepsen worker 14 - jepsen.util 194 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,895] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:20,906] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,907] jepsen worker 76 - jepsen.util 526 :invoke :write [25 43] INFO [2022-03-13 17:08:20,921] jepsen worker 61 - jepsen.util 421 :ok :read [20 10] INFO [2022-03-13 17:08:20,922] jepsen worker 61 - jepsen.util 421 :invoke :write [20 47] INFO [2022-03-13 17:08:20,938] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,939] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] INFO [2022-03-13 17:08:20,976] jepsen worker 35 - jepsen.util 665 :ok :read [11 :nil] INFO [2022-03-13 17:08:20,976] jepsen worker 44 - jepsen.util 404 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:20,977] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:20,977] jepsen worker 44 - jepsen.util 404 :invoke :read [14 nil] INFO [2022-03-13 17:08:20,978] jepsen worker 35 - jepsen.util 665 :invoke :write [11 20] INFO [2022-03-13 17:08:20,978] jepsen worker 14 - jepsen.util 194 :invoke :write [4 7] INFO [2022-03-13 17:08:21,041] jepsen worker 76 - jepsen.util 526 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,042] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:21,101] jepsen worker 61 - jepsen.util 421 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,121] jepsen worker 53 - jepsen.util 413 :info :write [17 31] :RequestTimeoutException INFO [2022-03-13 17:08:21,140] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,144] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:21,140] jepsen worker 44 - jepsen.util 404 :ok :read [14 :nil] INFO [2022-03-13 17:08:21,144] jepsen worker 53 - jepsen.util 503 :invoke :write [17 43] INFO [2022-03-13 17:08:21,144] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:08:21,144] jepsen worker 44 - jepsen.util 404 :invoke :write [14 14] INFO [2022-03-13 17:08:21,182] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:21,182] jepsen worker nemesis - couchbase.util Rebalance check count: 14 INFO [2022-03-13 17:08:21,182] jepsen worker nemesis - couchbase.util Rebalance stuck count: 13 INFO [2022-03-13 17:08:21,190] jepsen worker 35 - jepsen.util 665 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,190] jepsen worker 35 - jepsen.util 665 :invoke :write [11 42] INFO [2022-03-13 17:08:21,221] jepsen worker 14 - jepsen.util 194 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,222] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:21,243] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,244] jepsen worker 76 - jepsen.util 526 :invoke :write [25 36] INFO [2022-03-13 17:08:21,257] jepsen worker 61 - jepsen.util 421 :ok :read [20 10] INFO [2022-03-13 17:08:21,257] jepsen worker 61 - jepsen.util 421 :invoke :write [20 14] INFO [2022-03-13 17:08:21,295] jepsen worker 44 - jepsen.util 404 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,296] jepsen worker 44 - jepsen.util 404 :invoke :write [14 6] INFO [2022-03-13 17:08:21,362] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,362] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:21,426] jepsen worker 53 - jepsen.util 503 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,427] jepsen worker 53 - jepsen.util 503 :invoke :write [17 28] INFO [2022-03-13 17:08:21,458] jepsen worker 35 - jepsen.util 665 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,459] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:21,459] jepsen worker 35 - jepsen.util 665 :invoke :write [11 23] INFO [2022-03-13 17:08:21,460] jepsen worker 14 - jepsen.util 194 :invoke :write [4 3] INFO [2022-03-13 17:08:21,502] jepsen worker 76 - jepsen.util 526 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,502] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:21,534] jepsen worker 61 - jepsen.util 421 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,536] jepsen worker 61 - jepsen.util 421 :invoke :write [20 31] INFO [2022-03-13 17:08:21,586] jepsen worker 44 - jepsen.util 404 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,587] jepsen worker 44 - jepsen.util 404 :invoke :write [14 39] INFO [2022-03-13 17:08:21,589] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:21,594] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:21,678] jepsen worker 53 - jepsen.util 503 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,678] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:21,715] jepsen worker 8 - jepsen.util 458 :info :write [2 37] :SyncWriteAmbiguous INFO [2022-03-13 17:08:21,715] jepsen worker 8 - jepsen.util 548 :invoke :write [2 5] INFO [2022-03-13 17:08:21,767] jepsen worker 35 - jepsen.util 665 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,768] jepsen worker 35 - jepsen.util 665 :invoke :write [11 38] INFO [2022-03-13 17:08:21,794] jepsen worker 45 - jepsen.util 675 :info :write [15 37] :RequestTimeoutException INFO [2022-03-13 17:08:21,794] jepsen worker 45 - jepsen.util 765 :invoke :write [15 16] INFO [2022-03-13 17:08:21,834] jepsen worker 14 - jepsen.util 194 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,835] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:21,835] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,835] jepsen worker 76 - jepsen.util 526 :invoke :write [25 4] INFO [2022-03-13 17:08:21,839] jepsen worker 25 - jepsen.util 385 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,840] jepsen worker 25 - jepsen.util 385 :invoke :write [8 38] INFO [2022-03-13 17:08:21,845] jepsen worker 49 - jepsen.util 499 :info :write [16 2] :SyncWriteAmbiguous INFO [2022-03-13 17:08:21,846] jepsen worker 49 - jepsen.util 589 :invoke :read [16 nil] INFO [2022-03-13 17:08:21,887] jepsen worker 31 - jepsen.util 481 :info :write [10 40] :RequestTimeoutException INFO [2022-03-13 17:08:21,887] jepsen worker 31 - jepsen.util 571 :invoke :read [10 nil] INFO [2022-03-13 17:08:21,897] jepsen worker 43 - jepsen.util 313 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,898] jepsen worker 43 - jepsen.util 313 :invoke :read [14 nil] INFO [2022-03-13 17:08:21,906] jepsen worker 77 - jepsen.util 347 :info :write [25 6] :RequestTimeoutException INFO [2022-03-13 17:08:21,906] jepsen worker 61 - jepsen.util 421 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,906] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:21,907] jepsen worker 33 - jepsen.util 393 :info :write [11 43] :RequestTimeoutException INFO [2022-03-13 17:08:21,909] jepsen worker 40 - jepsen.util 490 :info :write [13 10] :SyncWriteAmbiguous INFO [2022-03-13 17:08:21,909] jepsen worker 77 - jepsen.util 437 :invoke :write [25 35] INFO [2022-03-13 17:08:21,910] jepsen worker 40 - jepsen.util 580 :invoke :read [13 nil] INFO [2022-03-13 17:08:21,911] jepsen worker 42 - jepsen.util 402 :invoke :write [14 14] INFO [2022-03-13 17:08:21,912] jepsen worker 33 - jepsen.util 483 :invoke :read [11 nil] INFO [2022-03-13 17:08:21,913] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:21,917] jepsen worker 23 - jepsen.util 293 :info :write [7 3] :RequestTimeoutException INFO [2022-03-13 17:08:21,917] jepsen worker 23 - jepsen.util 383 :invoke :write [7 9] INFO [2022-03-13 17:08:21,941] jepsen worker 13 - jepsen.util 373 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,942] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:21,947] jepsen worker 40 - jepsen.util 580 :ok :read [13 39] INFO [2022-03-13 17:08:21,948] jepsen worker 40 - jepsen.util 580 :invoke :read [13 nil] INFO [2022-03-13 17:08:21,971] jepsen worker 40 - jepsen.util 580 :ok :read [13 39] INFO [2022-03-13 17:08:21,972] jepsen worker 40 - jepsen.util 580 :invoke :write [13 20] INFO [2022-03-13 17:08:21,974] jepsen worker 80 - jepsen.util 620 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,974] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:21,994] jepsen worker 44 - jepsen.util 404 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,995] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:21,996] jepsen worker 44 - jepsen.util 404 :invoke :read [14 nil] INFO [2022-03-13 17:08:21,997] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:22,047] jepsen worker 51 - jepsen.util 231 :info :write [17 10] :RequestTimeoutException INFO [2022-03-13 17:08:22,047] jepsen worker 7 - jepsen.util 277 :info :write [2 47] :RequestTimeoutException INFO [2022-03-13 17:08:22,047] jepsen worker 51 - jepsen.util 321 :invoke :write [17 12] INFO [2022-03-13 17:08:22,048] jepsen worker 25 - jepsen.util 385 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,048] jepsen worker 43 - jepsen.util 313 :ok :read [14 :nil] INFO [2022-03-13 17:08:22,048] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:22,049] jepsen worker 43 - jepsen.util 313 :invoke :write [14 16] INFO [2022-03-13 17:08:22,050] jepsen worker 25 - jepsen.util 385 :invoke :write [8 7] INFO [2022-03-13 17:08:22,051] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:22,052] jepsen worker 14 - jepsen.util 194 :invoke :write [4 18] INFO [2022-03-13 17:08:22,055] jepsen worker 35 - jepsen.util 665 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,056] jepsen worker 35 - jepsen.util 665 :invoke :write [11 40] INFO [2022-03-13 17:08:22,102] jepsen worker 76 - jepsen.util 526 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,102] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,103] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:22,103] jepsen worker 76 - jepsen.util 526 :invoke :write [25 23] INFO [2022-03-13 17:08:22,147] jepsen worker 47 - jepsen.util 587 :info :write [15 11] :RequestTimeoutException INFO [2022-03-13 17:08:22,147] jepsen worker 47 - jepsen.util 677 :invoke :write [15 38] INFO [2022-03-13 17:08:22,148] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,149] jepsen worker 77 - jepsen.util 437 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,149] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:22,149] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:22,150] jepsen worker 77 - jepsen.util 437 :invoke :read [25 nil] INFO [2022-03-13 17:08:22,150] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:22,158] jepsen worker 42 - jepsen.util 402 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,158] jepsen worker 44 - jepsen.util 404 :ok :read [14 :nil] INFO [2022-03-13 17:08:22,159] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,159] jepsen worker 42 - jepsen.util 402 :invoke :write [14 25] INFO [2022-03-13 17:08:22,159] jepsen worker 44 - jepsen.util 404 :invoke :read [14 nil] INFO [2022-03-13 17:08:22,159] jepsen worker 53 - jepsen.util 503 :invoke :write [17 13] INFO [2022-03-13 17:08:22,188] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0}} INFO [2022-03-13 17:08:22,188] jepsen worker nemesis - couchbase.util Rebalance check count: 15 INFO [2022-03-13 17:08:22,188] jepsen worker nemesis - couchbase.util Rebalance stuck count: 14 INFO [2022-03-13 17:08:22,205] jepsen worker 14 - jepsen.util 194 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,207] jepsen worker 14 - jepsen.util 194 :invoke :write [4 8] INFO [2022-03-13 17:08:22,280] jepsen worker 75 - jepsen.util 255 :info :write [25 27] :RequestTimeoutException INFO [2022-03-13 17:08:22,281] jepsen worker 75 - jepsen.util 345 :invoke :write [25 0] INFO [2022-03-13 17:08:22,531] jepsen worker 84 - jepsen.util 714 :info :write [28 35] :RequestTimeoutException INFO [2022-03-13 17:08:22,532] jepsen worker 84 - jepsen.util 804 :invoke :write [28 27] INFO [2022-03-13 17:08:22,616] jepsen worker 25 - jepsen.util 385 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,617] jepsen worker 25 - jepsen.util 385 :invoke :write [8 49] INFO [2022-03-13 17:08:22,667] jepsen worker 35 - jepsen.util 665 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,667] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,668] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:22,668] jepsen worker 61 - jepsen.util 421 :invoke :write [20 18] INFO [2022-03-13 17:08:22,725] jepsen worker 43 - jepsen.util 313 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,726] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,726] jepsen worker 77 - jepsen.util 437 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,726] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,726] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:22,727] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:22,728] jepsen worker 43 - jepsen.util 313 :invoke :write [14 2] INFO [2022-03-13 17:08:22,728] jepsen worker 77 - jepsen.util 437 :invoke :write [25 10] INFO [2022-03-13 17:08:22,730] jepsen worker 78 - jepsen.util 438 :invoke :write [26 10] INFO [2022-03-13 17:08:22,739] jepsen worker 13 - jepsen.util 373 :invoke :write [4 49] INFO [2022-03-13 17:08:22,740] jepsen worker 76 - jepsen.util 526 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,740] jepsen worker 44 - jepsen.util 404 :ok :read [14 :nil] INFO [2022-03-13 17:08:22,741] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:22,742] jepsen worker 44 - jepsen.util 404 :invoke :write [14 49] INFO [2022-03-13 17:08:22,780] jepsen worker 37 - jepsen.util 487 :info :write [12 23] :RequestTimeoutException INFO [2022-03-13 17:08:22,781] jepsen worker 37 - jepsen.util 577 :invoke :read [12 nil] INFO [2022-03-13 17:08:22,817] jepsen worker 37 - jepsen.util 577 :ok :read [12 46] INFO [2022-03-13 17:08:22,817] jepsen worker 37 - jepsen.util 577 :invoke :read [12 nil] INFO [2022-03-13 17:08:22,830] jepsen worker 53 - jepsen.util 503 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,831] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:22,840] jepsen worker 68 - jepsen.util 608 :info :write [22 40] :RequestTimeoutException INFO [2022-03-13 17:08:22,840] jepsen worker 68 - jepsen.util 698 :invoke :write [22 48] INFO [2022-03-13 17:08:22,842] jepsen worker 37 - jepsen.util 577 :ok :read [12 46] INFO [2022-03-13 17:08:22,843] jepsen worker 37 - jepsen.util 577 :invoke :read [12 nil] INFO [2022-03-13 17:08:22,854] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,854] jepsen worker 25 - jepsen.util 385 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,855] jepsen worker 25 - jepsen.util 385 :invoke :write [8 19] INFO [2022-03-13 17:08:22,855] jepsen worker 80 - jepsen.util 620 :invoke :write [26 9] INFO [2022-03-13 17:08:22,861] jepsen worker 2 - jepsen.util 632 :info :write [0 35] :RequestTimeoutException INFO [2022-03-13 17:08:22,862] jepsen worker 2 - jepsen.util 722 :invoke :read [0 nil] INFO [2022-03-13 17:08:22,885] jepsen worker 37 - jepsen.util 577 :ok :read [12 46] INFO [2022-03-13 17:08:22,886] jepsen worker 37 - jepsen.util 577 :invoke :write [12 4] INFO [2022-03-13 17:08:22,909] jepsen worker 42 - jepsen.util 402 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,910] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:22,914] jepsen worker 43 - jepsen.util 313 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,914] jepsen worker 43 - jepsen.util 313 :invoke :read [14 nil] INFO [2022-03-13 17:08:22,948] jepsen worker 87 - jepsen.util 717 :info :write [29 33] :RequestTimeoutException INFO [2022-03-13 17:08:22,948] jepsen worker 87 - jepsen.util 807 :invoke :write [29 33] INFO [2022-03-13 17:08:22,960] jepsen worker 59 - jepsen.util 599 :info :write [19 26] :RequestTimeoutException INFO [2022-03-13 17:08:22,961] jepsen worker 59 - jepsen.util 689 :invoke :read [19 nil] INFO [2022-03-13 17:08:22,997] jepsen worker 13 - jepsen.util 373 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,997] jepsen worker 14 - jepsen.util 194 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:22,997] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:22,998] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,078] jepsen worker 77 - jepsen.util 437 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,079] jepsen worker 77 - jepsen.util 437 :invoke :read [25 nil] INFO [2022-03-13 17:08:23,091] jepsen worker 0 - jepsen.util 450 :info :write [0 15] :RequestTimeoutException INFO [2022-03-13 17:08:23,092] jepsen worker 0 - jepsen.util 540 :invoke :read [0 nil] INFO [2022-03-13 17:08:23,111] jepsen worker 41 - jepsen.util 581 :info :write [13 34] :RequestTimeoutException INFO [2022-03-13 17:08:23,111] jepsen worker 41 - jepsen.util 671 :invoke :write [13 47] INFO [2022-03-13 17:08:23,123] jepsen worker 75 - jepsen.util 345 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,124] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:23,140] jepsen worker 78 - jepsen.util 438 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,141] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:23,192] jepsen worker 80 - jepsen.util 620 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,193] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:23,203] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0.01169590643274854}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0.0117647058823529}} INFO [2022-03-13 17:08:23,203] jepsen worker nemesis - couchbase.util Rebalance check count: 16 INFO [2022-03-13 17:08:23,203] jepsen worker nemesis - couchbase.util Rebalance stuck count: 14 INFO [2022-03-13 17:08:23,233] jepsen worker 61 - jepsen.util 421 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,233] jepsen worker 35 - jepsen.util 665 :ok :read [11 :nil] INFO [2022-03-13 17:08:23,233] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:23,234] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:23,237] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,237] jepsen worker 77 - jepsen.util 437 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,237] jepsen worker 43 - jepsen.util 313 :ok :read [14 :nil] INFO [2022-03-13 17:08:23,237] jepsen worker 25 - jepsen.util 385 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,237] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,238] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,238] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,239] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,239] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:23,240] jepsen worker 43 - jepsen.util 313 :invoke :write [14 0] INFO [2022-03-13 17:08:23,240] jepsen worker 77 - jepsen.util 437 :invoke :write [25 10] INFO [2022-03-13 17:08:23,240] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,241] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:23,242] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,242] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,243] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:23,244] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,244] jepsen worker 80 - jepsen.util 620 :invoke :write [26 26] INFO [2022-03-13 17:08:23,245] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,245] jepsen worker 78 - jepsen.util 438 :invoke :write [26 23] INFO [2022-03-13 17:08:23,250] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,251] jepsen worker 25 - jepsen.util 385 :invoke :write [8 43] INFO [2022-03-13 17:08:23,252] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,254] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,256] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,257] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,259] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,259] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,277] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,277] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:23,277] jepsen worker 35 - jepsen.util 665 :ok :read [11 :nil] INFO [2022-03-13 17:08:23,277] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,277] jepsen worker 44 - jepsen.util 404 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,277] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,278] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,282] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,284] jepsen worker 44 - jepsen.util 404 :invoke :write [14 44] INFO [2022-03-13 17:08:23,285] jepsen worker 14 - jepsen.util 194 :invoke :write [4 5] INFO [2022-03-13 17:08:23,287] jepsen worker 35 - jepsen.util 665 :invoke :write [11 47] INFO [2022-03-13 17:08:23,288] jepsen worker 42 - jepsen.util 402 :invoke :write [14 28] INFO [2022-03-13 17:08:23,290] jepsen worker 61 - jepsen.util 421 :invoke :write [20 21] INFO [2022-03-13 17:08:23,290] jepsen worker 53 - jepsen.util 503 :invoke :write [17 12] INFO [2022-03-13 17:08:23,290] jepsen worker 76 - jepsen.util 526 :invoke :write [25 1] INFO [2022-03-13 17:08:23,296] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:23,298] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,300] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:23,302] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,302] jepsen worker 75 - jepsen.util 345 :invoke :write [25 30] INFO [2022-03-13 17:08:23,311] jepsen worker 67 - jepsen.util 607 :info :write [22 17] :RequestTimeoutException INFO [2022-03-13 17:08:23,312] jepsen worker 67 - jepsen.util 697 :invoke :read [22 nil] INFO [2022-03-13 17:08:23,322] jepsen worker 43 - jepsen.util 313 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,322] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,324] jepsen worker 43 - jepsen.util 313 :invoke :read [14 nil] INFO [2022-03-13 17:08:23,325] jepsen worker 86 - jepsen.util 626 :info :write [28 44] :RequestTimeoutException INFO [2022-03-13 17:08:23,325] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,325] jepsen worker 86 - jepsen.util 716 :invoke :write [28 36] INFO [2022-03-13 17:08:23,375] jepsen worker 77 - jepsen.util 437 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,375] jepsen worker 77 - jepsen.util 437 :invoke :write [25 4] INFO [2022-03-13 17:08:23,393] jepsen worker 14 - jepsen.util 194 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,393] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,415] jepsen worker 61 - jepsen.util 421 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,415] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:23,476] jepsen worker 21 - jepsen.util 561 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,476] jepsen worker 76 - jepsen.util 526 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,476] jepsen worker 78 - jepsen.util 438 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,478] jepsen worker 78 - jepsen.util 438 :invoke :write [26 37] INFO [2022-03-13 17:08:23,483] jepsen worker 21 - jepsen.util 561 :invoke :write [7 29] INFO [2022-03-13 17:08:23,484] jepsen worker 76 - jepsen.util 526 :invoke :write [25 49] INFO [2022-03-13 17:08:23,547] jepsen worker 80 - jepsen.util 620 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,548] jepsen worker 80 - jepsen.util 620 :invoke :write [26 41] INFO [2022-03-13 17:08:23,556] jepsen worker 65 - jepsen.util 515 :info :write [21 27] :RequestTimeoutException INFO [2022-03-13 17:08:23,557] jepsen worker 65 - jepsen.util 605 :invoke :read [21 nil] INFO [2022-03-13 17:08:23,558] jepsen worker 65 - jepsen.util 605 :ok :read [21 :nil] INFO [2022-03-13 17:08:23,559] jepsen worker 65 - jepsen.util 605 :invoke :read [21 nil] INFO [2022-03-13 17:08:23,560] jepsen worker 65 - jepsen.util 605 :ok :read [21 :nil] INFO [2022-03-13 17:08:23,561] jepsen worker 65 - jepsen.util 605 :invoke :write [21 44] INFO [2022-03-13 17:08:23,571] jepsen worker 42 - jepsen.util 402 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,572] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:23,577] jepsen worker 81 - jepsen.util 711 :info :write [27 8] :SyncWriteAmbiguous INFO [2022-03-13 17:08:23,578] jepsen worker 81 - jepsen.util 801 :invoke :write [27 28] INFO [2022-03-13 17:08:23,618] jepsen worker 62 - jepsen.util 332 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,619] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,619] jepsen worker 62 - jepsen.util 332 :invoke :write [20 6] INFO [2022-03-13 17:08:23,620] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:23,642] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:23,642] jepsen worker 25 - jepsen.util 385 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,642] jepsen worker 43 - jepsen.util 313 :ok :read [14 :nil] INFO [2022-03-13 17:08:23,643] jepsen worker 25 - jepsen.util 385 :invoke :write [8 11] INFO [2022-03-13 17:08:23,647] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:23,648] jepsen worker 43 - jepsen.util 313 :invoke :read [14 nil] INFO [2022-03-13 17:08:23,688] jepsen worker 75 - jepsen.util 345 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,689] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:23,830] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:23,831] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:23,841] jepsen worker 16 - jepsen.util 556 :info :write [5 7] :RequestTimeoutException INFO [2022-03-13 17:08:23,842] jepsen worker 16 - jepsen.util 646 :invoke :write [5 43] INFO [2022-03-13 17:08:23,881] jepsen worker 89 - jepsen.util 539 :info :write [29 31] :RequestTimeoutException INFO [2022-03-13 17:08:23,881] jepsen worker 89 - jepsen.util 629 :invoke :write [29 38] INFO [2022-03-13 17:08:24,019] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:24,020] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:24,021] jepsen worker 33 - jepsen.util 483 :ok :read [11 :nil] INFO [2022-03-13 17:08:24,022] jepsen worker 33 - jepsen.util 483 :invoke :write [11 0] INFO [2022-03-13 17:08:24,125] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:24,125] jepsen worker 43 - jepsen.util 313 :ok :read [14 :nil] INFO [2022-03-13 17:08:24,125] jepsen worker 43 - jepsen.util 313 :invoke :read [14 nil] INFO [2022-03-13 17:08:24,126] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:24,134] jepsen worker 20 - jepsen.util 650 :info :write [6 49] :RequestTimeoutException INFO [2022-03-13 17:08:24,134] jepsen worker 20 - jepsen.util 740 :invoke :read [6 nil] INFO [2022-03-13 17:08:24,146] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:24,147] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:24,176] jepsen worker 9 - jepsen.util 549 :info :write [3 49] :RequestTimeoutException INFO [2022-03-13 17:08:24,177] jepsen worker 9 - jepsen.util 639 :invoke :write [3 33] INFO [2022-03-13 17:08:24,210] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0}, :ns_1@172.23.105.187 {:progress 0.01169590643274854}, :ns_1@172.23.107.227 {:progress 0}, :ns_1@172.23.107.228 {:progress 0.0117647058823529}} INFO [2022-03-13 17:08:24,210] jepsen worker nemesis - couchbase.util Rebalance check count: 17 INFO [2022-03-13 17:08:24,210] jepsen worker nemesis - couchbase.util Rebalance stuck count: 15 INFO [2022-03-13 17:08:24,258] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:24,259] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:24,271] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:24,271] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:24,272] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:24,273] jepsen worker 75 - jepsen.util 345 :invoke :write [25 49] INFO [2022-03-13 17:08:24,273] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:24,274] jepsen worker 42 - jepsen.util 402 :invoke :write [14 11] INFO [2022-03-13 17:08:24,318] jepsen worker 38 - jepsen.util 578 :info :write [12 22] :SyncWriteAmbiguous INFO [2022-03-13 17:08:24,321] jepsen worker 38 - jepsen.util 668 :invoke :read [12 nil] INFO [2022-03-13 17:08:24,398] jepsen worker 26 - jepsen.util 386 :info :write [8 0] :RequestTimeoutException INFO [2022-03-13 17:08:24,399] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:24,450] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:24,451] jepsen worker 14 - jepsen.util 194 :invoke :write [4 7] INFO [2022-03-13 17:08:24,473] jepsen worker 38 - jepsen.util 668 :ok :read [12 46] INFO [2022-03-13 17:08:24,474] jepsen worker 38 - jepsen.util 668 :invoke :write [12 14] INFO [2022-03-13 17:08:24,507] jepsen worker 10 - jepsen.util 370 :info :write [3 27] :RequestTimeoutException INFO [2022-03-13 17:08:24,514] jepsen worker 10 - jepsen.util 460 :invoke :write [3 21] INFO [2022-03-13 17:08:24,552] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:24,552] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:24,580] jepsen worker 43 - jepsen.util 313 :ok :read [14 :nil] INFO [2022-03-13 17:08:24,581] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:24,581] jepsen worker 43 - jepsen.util 313 :invoke :write [14 25] INFO [2022-03-13 17:08:24,589] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:24,626] jepsen worker 71 - jepsen.util 341 :info :write [23 31] :RequestTimeoutException INFO [2022-03-13 17:08:24,627] jepsen worker 71 - jepsen.util 431 :invoke :write [23 21] INFO [2022-03-13 17:08:24,678] jepsen worker 69 - jepsen.util 429 :info :write [23 44] :RequestTimeoutException INFO [2022-03-13 17:08:24,680] jepsen worker 69 - jepsen.util 519 :invoke :write [23 12] INFO [2022-03-13 17:08:24,692] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:24,693] jepsen worker 61 - jepsen.util 421 :invoke :write [20 28] INFO [2022-03-13 17:08:24,896] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:24,897] jepsen worker 52 - jepsen.util 412 :invoke :write [17 13] INFO [2022-03-13 17:08:24,984] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:24,985] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:25,046] jepsen worker 30 - jepsen.util 480 :ok :read [10 :nil] INFO [2022-03-13 17:08:25,047] jepsen worker 30 - jepsen.util 480 :invoke :read [10 nil] INFO [2022-03-13 17:08:25,125] jepsen worker 72 - jepsen.util 612 :ok :read [24 24] INFO [2022-03-13 17:08:25,126] jepsen worker 72 - jepsen.util 612 :invoke :read [24 nil] INFO [2022-03-13 17:08:25,141] jepsen worker 0 - jepsen.util 540 :ok :read [0 41] INFO [2022-03-13 17:08:25,142] jepsen worker 0 - jepsen.util 540 :invoke :write [0 12] INFO [2022-03-13 17:08:25,147] jepsen worker 17 - jepsen.util 647 :ok :read [5 32] INFO [2022-03-13 17:08:25,147] jepsen worker 17 - jepsen.util 647 :invoke :write [5 7] INFO [2022-03-13 17:08:25,164] jepsen worker 5 - jepsen.util 545 :info :write [1 27] :RequestTimeoutException INFO [2022-03-13 17:08:25,165] jepsen worker 5 - jepsen.util 635 :invoke :write [1 2] INFO [2022-03-13 17:08:25,191] jepsen worker 73 - jepsen.util 703 :ok :read [24 24] INFO [2022-03-13 17:08:25,192] jepsen worker 73 - jepsen.util 703 :invoke :write [24 39] INFO [2022-03-13 17:08:25,208] jepsen worker 30 - jepsen.util 480 :ok :read [10 :nil] INFO [2022-03-13 17:08:25,209] jepsen worker 30 - jepsen.util 480 :invoke :read [10 nil] INFO [2022-03-13 17:08:25,218] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.002932551319648091}, :ns_1@172.23.105.187 {:progress 0.01169590643274854}, :ns_1@172.23.107.227 {:progress 0.002932551319648091}, :ns_1@172.23.107.228 {:progress 0.01764705882352946}} INFO [2022-03-13 17:08:25,218] jepsen worker nemesis - couchbase.util Rebalance check count: 18 INFO [2022-03-13 17:08:25,218] jepsen worker nemesis - couchbase.util Rebalance stuck count: 15 INFO [2022-03-13 17:08:25,255] jepsen worker 30 - jepsen.util 480 :ok :read [10 :nil] INFO [2022-03-13 17:08:25,257] jepsen worker 30 - jepsen.util 480 :invoke :write [10 18] INFO [2022-03-13 17:08:25,273] jepsen worker 72 - jepsen.util 612 :ok :read [24 24] INFO [2022-03-13 17:08:25,274] jepsen worker 72 - jepsen.util 612 :invoke :read [24 nil] WARN [2022-03-13 17:08:25,303] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":1578740,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1578740,"total_server_duration_us":68547,"encode_duration_us":16325,"operation_id":"0x1478","timeout_ms":10000,"last_server_duration_us":68547,"total_duration_us":10007299},{"operation_name":"upsert","last_dispatch_duration_us":8474061,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9634003,"total_server_duration_us":454645,"encode_duration_us":23821,"operation_id":"0x154d","timeout_ms":10000,"last_server_duration_us":106787,"total_duration_us":10007144},{"operation_name":"upsert","last_dispatch_duration_us":8731850,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8764241,"total_server_duration_us":42489,"encode_duration_us":18262,"operation_id":"0x15b6","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10006494},{"operation_name":"upsert","last_dispatch_duration_us":7421155,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10358350,"total_server_duration_us":445365,"encode_duration_us":18513,"operation_id":"0x14de","timeout_ms":10000,"last_server_duration_us":60885,"total_duration_us":10005702},{"operation_name":"upsert","last_dispatch_duration_us":53523,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":242257,"total_server_duration_us":130565,"encode_duration_us":20017,"operation_id":"0x15a4","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10005117},{"operation_name":"upsert","last_dispatch_duration_us":3379083,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":11181409,"total_server_duration_us":482887,"encode_duration_us":19951,"operation_id":"0x1575","timeout_ms":10000,"last_server_duration_us":91683,"total_duration_us":10003627},{"operation_name":"upsert","last_dispatch_duration_us":9731275,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10959285,"total_server_duration_us":465109,"encode_duration_us":23250,"operation_id":"0x154f","timeout_ms":10000,"last_server_duration_us":82547,"total_duration_us":10003424},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17838,"operation_id":"0x15e6","timeout_ms":10000,"total_duration_us":10002856},{"operation_name":"get","last_dispatch_duration_us":9860777,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9860777,"total_server_duration_us":43,"operation_id":"0x14b4","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9862883},{"operation_name":"upsert","last_dispatch_duration_us":9326568,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9326568,"total_server_duration_us":4223552,"encode_duration_us":17204,"operation_id":"0x14af","timeout_ms":10000,"last_server_duration_us":4223552,"total_duration_us":9330417}],"total_count":12}} WARN [2022-03-13 17:08:25,304] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9731275,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10959285,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":465109,"encode_duration_us":23250,"operation_id":"0x154f","timeout_ms":10000,"last_server_duration_us":82547,"total_duration_us":10003424},{"operation_name":"upsert","last_dispatch_duration_us":16268641,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21859731,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9482721,"encode_duration_us":18422,"operation_id":"0x1327","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10005707},{"operation_name":"upsert","last_dispatch_duration_us":8474061,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9634003,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":454645,"encode_duration_us":23821,"operation_id":"0x154d","timeout_ms":10000,"last_server_duration_us":106787,"total_duration_us":10007144},{"operation_name":"upsert","last_dispatch_duration_us":9088838,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10447781,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9355936,"encode_duration_us":18993,"operation_id":"0x1410","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10010511}],"total_count":4}} INFO [2022-03-13 17:08:25,347] jepsen worker 60 - jepsen.util 600 :ok :write [20 12] INFO [2022-03-13 17:08:25,348] jepsen worker 60 - jepsen.util 600 :invoke :write [20 10] INFO [2022-03-13 17:08:25,357] jepsen worker 63 - jepsen.util 603 :info :write [21 48] :RequestTimeoutException INFO [2022-03-13 17:08:25,358] jepsen worker 63 - jepsen.util 693 :invoke :write [21 19] INFO [2022-03-13 17:08:25,382] jepsen worker 72 - jepsen.util 612 :ok :read [24 24] INFO [2022-03-13 17:08:25,382] jepsen worker 72 - jepsen.util 612 :invoke :read [24 nil] INFO [2022-03-13 17:08:25,464] jepsen worker 19 - jepsen.util 469 :ok :write [6 16] INFO [2022-03-13 17:08:25,465] jepsen worker 19 - jepsen.util 469 :invoke :read [6 nil] INFO [2022-03-13 17:08:25,488] jepsen worker 85 - jepsen.util 715 :ok :read [28 25] INFO [2022-03-13 17:08:25,489] jepsen worker 85 - jepsen.util 715 :invoke :write [28 7] INFO [2022-03-13 17:08:25,513] jepsen worker 72 - jepsen.util 612 :ok :read [24 24] INFO [2022-03-13 17:08:25,514] jepsen worker 72 - jepsen.util 612 :invoke :write [24 18] INFO [2022-03-13 17:08:25,524] jepsen worker 52 - jepsen.util 412 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:25,525] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:25,645] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:25,646] jepsen worker 13 - jepsen.util 373 :invoke :write [4 8] INFO [2022-03-13 17:08:25,773] jepsen worker 54 - jepsen.util 684 :ok :read [18 3] INFO [2022-03-13 17:08:25,774] jepsen worker 54 - jepsen.util 684 :invoke :write [18 47] INFO [2022-03-13 17:08:26,040] jepsen worker 56 - jepsen.util 506 :info :write [18 29] :RequestTimeoutException INFO [2022-03-13 17:08:26,041] jepsen worker 56 - jepsen.util 596 :invoke :write [18 27] INFO [2022-03-13 17:08:26,047] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:26,048] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:26,069] jepsen worker 34 - jepsen.util 394 :info :write [11 34] :SyncWriteAmbiguous INFO [2022-03-13 17:08:26,070] jepsen worker 34 - jepsen.util 484 :invoke :read [11 nil] INFO [2022-03-13 17:08:26,131] jepsen worker 32 - jepsen.util 482 :ok :read [10 :nil] INFO [2022-03-13 17:08:26,131] jepsen worker 32 - jepsen.util 482 :invoke :read [10 nil] INFO [2022-03-13 17:08:26,132] jepsen worker 64 - jepsen.util 424 :ok :read [21 :nil] INFO [2022-03-13 17:08:26,132] jepsen worker 64 - jepsen.util 424 :invoke :read [21 nil] INFO [2022-03-13 17:08:26,224] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.01173020527859236}, :ns_1@172.23.105.187 {:progress 0.02339181286549707}, :ns_1@172.23.107.227 {:progress 0.01173020527859236}, :ns_1@172.23.107.228 {:progress 0.02352941176470591}} INFO [2022-03-13 17:08:26,225] jepsen worker nemesis - couchbase.util Rebalance check count: 19 INFO [2022-03-13 17:08:26,225] jepsen worker nemesis - couchbase.util Rebalance stuck count: 15 INFO [2022-03-13 17:08:26,373] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:26,374] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:26,387] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:26,387] jepsen worker 7 - jepsen.util 367 :invoke :write [2 22] INFO [2022-03-13 17:08:26,489] jepsen worker 34 - jepsen.util 484 :ok :read [11 :nil] INFO [2022-03-13 17:08:26,490] jepsen worker 34 - jepsen.util 484 :invoke :write [11 46] INFO [2022-03-13 17:08:26,600] jepsen worker 46 - jepsen.util 586 :ok :read [15 48] INFO [2022-03-13 17:08:26,601] jepsen worker 46 - jepsen.util 586 :invoke :read [15 nil] INFO [2022-03-13 17:08:26,703] jepsen worker 18 - jepsen.util 468 :ok :read [6 16] INFO [2022-03-13 17:08:26,704] jepsen worker 18 - jepsen.util 468 :invoke :write [6 4] INFO [2022-03-13 17:08:26,705] jepsen worker 64 - jepsen.util 424 :ok :read [21 :nil] INFO [2022-03-13 17:08:26,705] jepsen worker 32 - jepsen.util 482 :ok :read [10 :nil] INFO [2022-03-13 17:08:26,706] jepsen worker 64 - jepsen.util 424 :invoke :write [21 36] INFO [2022-03-13 17:08:26,707] jepsen worker 32 - jepsen.util 482 :invoke :read [10 nil] INFO [2022-03-13 17:08:26,710] jepsen worker 28 - jepsen.util 658 :ok :read [9 27] INFO [2022-03-13 17:08:26,711] jepsen worker 28 - jepsen.util 658 :invoke :write [9 32] INFO [2022-03-13 17:08:26,712] jepsen worker 32 - jepsen.util 482 :ok :read [10 :nil] INFO [2022-03-13 17:08:26,713] jepsen worker 32 - jepsen.util 482 :invoke :read [10 nil] INFO [2022-03-13 17:08:26,715] jepsen worker 32 - jepsen.util 482 :ok :read [10 :nil] INFO [2022-03-13 17:08:26,716] jepsen worker 32 - jepsen.util 482 :invoke :read [10 nil] INFO [2022-03-13 17:08:26,718] jepsen worker 32 - jepsen.util 482 :ok :read [10 :nil] INFO [2022-03-13 17:08:26,719] jepsen worker 32 - jepsen.util 482 :invoke :read [10 nil] INFO [2022-03-13 17:08:26,719] jepsen worker 20 - jepsen.util 740 :ok :read [6 16] INFO [2022-03-13 17:08:26,720] jepsen worker 20 - jepsen.util 740 :invoke :write [6 13] INFO [2022-03-13 17:08:26,720] jepsen worker 32 - jepsen.util 482 :ok :read [10 :nil] INFO [2022-03-13 17:08:26,721] jepsen worker 32 - jepsen.util 482 :invoke :write [10 27] INFO [2022-03-13 17:08:26,768] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:26,768] jepsen worker 52 - jepsen.util 412 :invoke :write [17 37] INFO [2022-03-13 17:08:26,796] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:26,797] jepsen worker 26 - jepsen.util 476 :invoke :write [8 45] WARN [2022-03-13 17:08:26,805] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":12039194,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":16625575,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":9452079,"encode_duration_us":17832,"operation_id":"0x13cc","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10003483},{"operation_name":"upsert","last_dispatch_duration_us":18380612,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19863964,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9162912,"encode_duration_us":10776,"operation_id":"0x148b","timeout_ms":10000,"last_server_duration_us":9111791,"total_duration_us":10009795},{"operation_name":"upsert","last_dispatch_duration_us":19272530,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19854692,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9283558,"encode_duration_us":18603,"operation_id":"0x1434","timeout_ms":10000,"last_server_duration_us":9078800,"total_duration_us":10005077},{"operation_name":"upsert","last_dispatch_duration_us":11773626,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":17615156,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":9505147,"encode_duration_us":23085,"operation_id":"0x1342","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10010294}],"total_count":4}} WARN [2022-03-13 17:08:26,905] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16686316,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":16686316,"total_server_duration_us":8778,"operation_id":"0x1500","timeout_ms":30000,"last_server_duration_us":8778,"total_duration_us":16689217},{"operation_name":"get","last_dispatch_duration_us":4201092,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":4201092,"total_server_duration_us":23,"operation_id":"0x14b3","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":12938384},{"operation_name":"get","last_dispatch_duration_us":567310,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":567310,"total_server_duration_us":32,"operation_id":"0x14dd","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11849000},{"operation_name":"get","last_dispatch_duration_us":10893556,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10893556,"total_server_duration_us":190657,"operation_id":"0x162d","timeout_ms":30000,"last_server_duration_us":190657,"total_duration_us":10896747},{"operation_name":"upsert","last_dispatch_duration_us":8596256,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":11062526,"total_server_duration_us":67850,"encode_duration_us":20938,"operation_id":"0x14c1","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10006822},{"operation_name":"upsert","last_dispatch_duration_us":5740212,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5740212,"total_server_duration_us":21243,"encode_duration_us":28983,"operation_id":"0x15c3","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10006042},{"operation_name":"upsert","last_dispatch_duration_us":4203673,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":4203673,"total_server_duration_us":19809,"encode_duration_us":24697,"operation_id":"0x154e","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10004007},{"operation_name":"upsert","last_dispatch_duration_us":5079019,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":11097533,"total_server_duration_us":350514,"encode_duration_us":18598,"operation_id":"0x15ed","timeout_ms":10000,"last_server_duration_us":92892,"total_duration_us":10001949},{"operation_name":"upsert","last_dispatch_duration_us":5300107,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":11101370,"total_server_duration_us":252132,"encode_duration_us":17921,"operation_id":"0x1608","timeout_ms":10000,"last_server_duration_us":96557,"total_duration_us":10001547},{"operation_name":"upsert","last_dispatch_duration_us":347746,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9355379,"total_server_duration_us":241646,"encode_duration_us":17682,"operation_id":"0x1576","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":9492922}],"total_count":14}} WARN [2022-03-13 17:08:26,909] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9050248,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10553200,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9300308,"encode_duration_us":20191,"operation_id":"0x14b9","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10002936},{"operation_name":"upsert","last_dispatch_duration_us":9106966,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14537454,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9167805,"encode_duration_us":17428,"operation_id":"0x158e","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10007001},{"operation_name":"upsert","last_dispatch_duration_us":16857882,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":18610591,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9367817,"encode_duration_us":17986,"operation_id":"0x1448","timeout_ms":10000,"last_server_duration_us":9074546,"total_duration_us":10008941},{"operation_name":"upsert","last_dispatch_duration_us":8866444,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":15094451,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":139959,"encode_duration_us":23774,"operation_id":"0x14f9","timeout_ms":10000,"last_server_duration_us":103434,"total_duration_us":10009829},{"operation_name":"upsert","last_dispatch_duration_us":9082431,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10629264,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9367055,"encode_duration_us":19334,"operation_id":"0x14b5","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":10009748}],"total_count":5}} WARN [2022-03-13 17:08:26,909] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9001808,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9001808,"total_server_duration_us":32,"operation_id":"0x1503","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":14172710},{"operation_name":"upsert","last_dispatch_duration_us":41817,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":489166,"total_server_duration_us":139149,"encode_duration_us":18246,"operation_id":"0x15f3","timeout_ms":10000,"last_server_duration_us":18799,"total_duration_us":10010016},{"operation_name":"upsert","last_dispatch_duration_us":8866444,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":15094451,"total_server_duration_us":139959,"encode_duration_us":23774,"operation_id":"0x14f9","timeout_ms":10000,"last_server_duration_us":103434,"total_duration_us":10009829},{"operation_name":"upsert","last_dispatch_duration_us":9082431,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10629264,"total_server_duration_us":9367055,"encode_duration_us":19334,"operation_id":"0x14b5","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":10009748},{"operation_name":"upsert","last_dispatch_duration_us":8172863,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8702217,"total_server_duration_us":270992,"encode_duration_us":16991,"operation_id":"0x156c","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10009719},{"operation_name":"upsert","last_dispatch_duration_us":8336488,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9890039,"total_server_duration_us":319191,"encode_duration_us":18362,"operation_id":"0x14bd","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10007188},{"operation_name":"upsert","last_dispatch_duration_us":9106966,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":14537454,"total_server_duration_us":9167805,"encode_duration_us":17428,"operation_id":"0x158e","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10007001},{"operation_name":"upsert","last_dispatch_duration_us":8180913,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9894137,"total_server_duration_us":367089,"encode_duration_us":20250,"operation_id":"0x14be","timeout_ms":10000,"last_server_duration_us":23222,"total_duration_us":10005966},{"operation_name":"upsert","last_dispatch_duration_us":44061,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":4568716,"total_server_duration_us":326160,"encode_duration_us":19775,"operation_id":"0x15d3","timeout_ms":10000,"last_server_duration_us":24669,"total_duration_us":10004630},{"operation_name":"upsert","last_dispatch_duration_us":9050248,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10553200,"total_server_duration_us":9300308,"encode_duration_us":20191,"operation_id":"0x14b9","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10002936}],"total_count":20}} WARN [2022-03-13 17:08:26,911] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11013357,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19245037,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":9180482,"encode_duration_us":19584,"operation_id":"0x1341","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005398},{"operation_name":"upsert","last_dispatch_duration_us":9331365,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13095594,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9288490,"encode_duration_us":17771,"operation_id":"0x1440","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10008141},{"operation_name":"upsert","last_dispatch_duration_us":12329191,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19168248,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":9144325,"encode_duration_us":17404,"operation_id":"0x1393","timeout_ms":10000,"last_server_duration_us":9106466,"total_duration_us":10007924},{"operation_name":"upsert","last_dispatch_duration_us":14385100,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14385100,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":5538522,"encode_duration_us":17929,"operation_id":"0x14fa","timeout_ms":10000,"last_server_duration_us":5538522,"total_duration_us":10009813}],"total_count":4}} WARN [2022-03-13 17:08:26,911] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8495618,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9918016,"total_server_duration_us":239231,"encode_duration_us":19919,"operation_id":"0x15c0","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10011084},{"operation_name":"upsert","last_dispatch_duration_us":9115725,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10160151,"total_server_duration_us":9644680,"encode_duration_us":31416,"operation_id":"0x14bb","timeout_ms":10000,"last_server_duration_us":9092628,"total_duration_us":10010490},{"operation_name":"upsert","last_dispatch_duration_us":576510,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3790514,"total_server_duration_us":377327,"encode_duration_us":20801,"operation_id":"0x159b","timeout_ms":10000,"last_server_duration_us":46287,"total_duration_us":10010274},{"operation_name":"upsert","last_dispatch_duration_us":41990,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1088763,"total_server_duration_us":193581,"encode_duration_us":17334,"operation_id":"0x15c8","timeout_ms":10000,"last_server_duration_us":31203,"total_duration_us":10010094},{"operation_name":"upsert","last_dispatch_duration_us":8737534,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9888873,"total_server_duration_us":197348,"encode_duration_us":17957,"operation_id":"0x15bf","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10008350},{"operation_name":"upsert","last_dispatch_duration_us":9166610,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11829515,"total_server_duration_us":9083589,"encode_duration_us":16894,"operation_id":"0x1552","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007884},{"operation_name":"upsert","last_dispatch_duration_us":88011,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1385787,"total_server_duration_us":210654,"encode_duration_us":22511,"operation_id":"0x15c4","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10006618},{"operation_name":"upsert","last_dispatch_duration_us":8494827,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9712408,"total_server_duration_us":212590,"encode_duration_us":26430,"operation_id":"0x15c7","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10005344},{"operation_name":"upsert","last_dispatch_duration_us":151970,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1170000,"total_server_duration_us":190633,"encode_duration_us":19627,"operation_id":"0x15c6","timeout_ms":10000,"last_server_duration_us":35513,"total_duration_us":10004392},{"operation_name":"upsert","last_dispatch_duration_us":150923,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":8315722,"total_server_duration_us":307371,"encode_duration_us":20313,"operation_id":"0x14cc","timeout_ms":10000,"last_server_duration_us":40270,"total_duration_us":10004099}],"total_count":30}} INFO [2022-03-13 17:08:26,974] jepsen worker 25 - jepsen.util 385 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:26,974] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:26,980] jepsen worker 76 - jepsen.util 526 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:26,981] jepsen worker 76 - jepsen.util 526 :invoke :write [25 30] WARN [2022-03-13 17:08:27,009] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14224374,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":14224374,"total_server_duration_us":23,"operation_id":"0x1570","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":14224838},{"operation_name":"get","last_dispatch_duration_us":11363519,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11363519,"total_server_duration_us":6927,"operation_id":"0x15de","timeout_ms":30000,"last_server_duration_us":6927,"total_duration_us":11369589},{"operation_name":"upsert","last_dispatch_duration_us":44524,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1989479,"total_server_duration_us":227774,"encode_duration_us":20800,"operation_id":"0x1593","timeout_ms":10000,"last_server_duration_us":25888,"total_duration_us":10008597},{"operation_name":"upsert","last_dispatch_duration_us":17252528,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":17910486,"total_server_duration_us":9159890,"encode_duration_us":19314,"operation_id":"0x14b6","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10008069},{"operation_name":"upsert","last_dispatch_duration_us":9170453,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10484366,"total_server_duration_us":9166504,"encode_duration_us":20027,"operation_id":"0x1606","timeout_ms":10000,"last_server_duration_us":9088372,"total_duration_us":10007883},{"operation_name":"upsert","last_dispatch_duration_us":12941609,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12941609,"total_server_duration_us":22306,"encode_duration_us":17063,"operation_id":"0x1598","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10007394},{"operation_name":"upsert","last_dispatch_duration_us":9212723,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":12830099,"total_server_duration_us":9443891,"encode_duration_us":18882,"operation_id":"0x1591","timeout_ms":10000,"last_server_duration_us":9137368,"total_duration_us":10006915},{"operation_name":"upsert","last_dispatch_duration_us":8928632,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":12665560,"total_server_duration_us":360204,"encode_duration_us":17163,"operation_id":"0x1597","timeout_ms":10000,"last_server_duration_us":60002,"total_duration_us":10006844},{"operation_name":"upsert","last_dispatch_duration_us":8753216,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10954281,"total_server_duration_us":392021,"encode_duration_us":17535,"operation_id":"0x1596","timeout_ms":10000,"last_server_duration_us":108237,"total_duration_us":10006732},{"operation_name":"upsert","last_dispatch_duration_us":8659293,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10838355,"total_server_duration_us":259997,"encode_duration_us":21828,"operation_id":"0x1592","timeout_ms":10000,"last_server_duration_us":77010,"total_duration_us":10005387}],"total_count":18}} WARN [2022-03-13 17:08:27,011] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8867470,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10050716,"total_server_duration_us":128202,"encode_duration_us":17673,"operation_id":"0x14bc","timeout_ms":10000,"last_server_duration_us":30162,"total_duration_us":10010462},{"operation_name":"upsert","last_dispatch_duration_us":3337612,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":3633156,"total_server_duration_us":147270,"encode_duration_us":17379,"operation_id":"0x158a","timeout_ms":10000,"last_server_duration_us":32840,"total_duration_us":10010259},{"operation_name":"upsert","last_dispatch_duration_us":14385100,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":14385100,"total_server_duration_us":5538522,"encode_duration_us":17929,"operation_id":"0x14fa","timeout_ms":10000,"last_server_duration_us":5538522,"total_duration_us":10009813},{"operation_name":"upsert","last_dispatch_duration_us":137822,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10019798,"total_server_duration_us":180557,"encode_duration_us":17689,"operation_id":"0x14ce","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10006973},{"operation_name":"upsert","last_dispatch_duration_us":3146626,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":3374830,"total_server_duration_us":183662,"encode_duration_us":16978,"operation_id":"0x15e9","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10006927},{"operation_name":"upsert","last_dispatch_duration_us":32768,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":4301392,"total_server_duration_us":66329,"encode_duration_us":19926,"operation_id":"0x1617","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10004732},{"operation_name":"upsert","last_dispatch_duration_us":35357,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7391629,"total_server_duration_us":383939,"encode_duration_us":17981,"operation_id":"0x15c9","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10003962},{"operation_name":"upsert","last_dispatch_duration_us":94214,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9918417,"total_server_duration_us":233191,"encode_duration_us":16920,"operation_id":"0x1565","timeout_ms":10000,"last_server_duration_us":32840,"total_duration_us":10003798},{"operation_name":"upsert","last_dispatch_duration_us":409972,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":10317986,"total_server_duration_us":98423,"encode_duration_us":18343,"operation_id":"0x1502","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10003363},{"operation_name":"get","last_dispatch_duration_us":9243932,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9243932,"total_server_duration_us":56,"operation_id":"0x1504","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9244312}],"total_count":34}} WARN [2022-03-13 17:08:27,011] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":5181247,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9962021,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":198304,"encode_duration_us":18991,"operation_id":"0x1501","timeout_ms":10000,"last_server_duration_us":44722,"total_duration_us":10001658},{"operation_name":"upsert","last_dispatch_duration_us":9166610,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11829515,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9083589,"encode_duration_us":16894,"operation_id":"0x1552","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007884},{"operation_name":"upsert","last_dispatch_duration_us":9039573,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12128211,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":10033627,"encode_duration_us":21658,"operation_id":"0x143b","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10002110},{"operation_name":"upsert","last_dispatch_duration_us":9115725,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10160151,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9644680,"encode_duration_us":31416,"operation_id":"0x14bb","timeout_ms":10000,"last_server_duration_us":9092628,"total_duration_us":10010490},{"operation_name":"upsert","last_dispatch_duration_us":576510,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":3790514,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":377327,"encode_duration_us":20801,"operation_id":"0x159b","timeout_ms":10000,"last_server_duration_us":46287,"total_duration_us":10010274}],"total_count":5}} INFO [2022-03-13 17:08:27,036] jepsen worker 79 - jepsen.util 709 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,037] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:27,037] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:27,038] jepsen worker 22 - jepsen.util 472 :invoke :write [7 32] INFO [2022-03-13 17:08:27,060] jepsen worker 69 - jepsen.util 519 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,061] jepsen worker 69 - jepsen.util 519 :invoke :read [23 nil] INFO [2022-03-13 17:08:27,081] jepsen worker 33 - jepsen.util 483 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,081] jepsen worker 33 - jepsen.util 483 :invoke :read [11 nil] WARN [2022-03-13 17:08:27,110] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9212723,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12830099,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9443891,"encode_duration_us":18882,"operation_id":"0x1591","timeout_ms":10000,"last_server_duration_us":9137368,"total_duration_us":10006915},{"operation_name":"upsert","last_dispatch_duration_us":17252528,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17910486,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9159890,"encode_duration_us":19314,"operation_id":"0x14b6","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10008069},{"operation_name":"upsert","last_dispatch_duration_us":9170453,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10484366,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9166504,"encode_duration_us":20027,"operation_id":"0x1606","timeout_ms":10000,"last_server_duration_us":9088372,"total_duration_us":10007883}],"total_count":3}} INFO [2022-03-13 17:08:27,128] jepsen worker 55 - jepsen.util 595 :info :write [18 18] :RequestTimeoutException INFO [2022-03-13 17:08:27,129] jepsen worker 55 - jepsen.util 685 :invoke :write [18 22] INFO [2022-03-13 17:08:27,130] jepsen worker 57 - jepsen.util 597 :info :write [19 10] :RequestTimeoutException INFO [2022-03-13 17:08:27,131] jepsen worker 57 - jepsen.util 687 :invoke :write [19 2] INFO [2022-03-13 17:08:27,164] jepsen worker 13 - jepsen.util 373 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,165] jepsen worker 13 - jepsen.util 373 :invoke :read [4 nil] INFO [2022-03-13 17:08:27,166] jepsen worker 70 - jepsen.util 610 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,167] jepsen worker 70 - jepsen.util 610 :invoke :write [23 40] INFO [2022-03-13 17:08:27,171] jepsen worker 61 - jepsen.util 421 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,171] jepsen worker 61 - jepsen.util 421 :invoke :write [20 35] INFO [2022-03-13 17:08:27,177] jepsen worker 11 - jepsen.util 551 :info :write [3 31] :SyncWriteAmbiguous INFO [2022-03-13 17:08:27,187] jepsen worker 4 - jepsen.util 634 :info :write [1 16] :RequestTimeoutException INFO [2022-03-13 17:08:27,187] jepsen worker 11 - jepsen.util 641 :invoke :read [3 nil] INFO [2022-03-13 17:08:27,188] jepsen worker 4 - jepsen.util 724 :invoke :read [1 nil] INFO [2022-03-13 17:08:27,215] jepsen worker 34 - jepsen.util 484 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,216] jepsen worker 34 - jepsen.util 484 :invoke :read [11 nil] INFO [2022-03-13 17:08:27,224] jepsen worker 66 - jepsen.util 606 :info :write [22 21] :RequestTimeoutException INFO [2022-03-13 17:08:27,224] jepsen worker 66 - jepsen.util 696 :invoke :write [22 1] INFO [2022-03-13 17:08:27,232] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.01759530791788855}, :ns_1@172.23.105.187 {:progress 0.0292397660818714}, :ns_1@172.23.107.227 {:progress 0.01759530791788855}, :ns_1@172.23.107.228 {:progress 0.02941176470588236}} INFO [2022-03-13 17:08:27,232] jepsen worker nemesis - couchbase.util Rebalance check count: 20 INFO [2022-03-13 17:08:27,232] jepsen worker nemesis - couchbase.util Rebalance stuck count: 15 INFO [2022-03-13 17:08:27,242] jepsen worker 71 - jepsen.util 431 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,242] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:27,251] jepsen worker 51 - jepsen.util 321 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,251] jepsen worker 51 - jepsen.util 321 :invoke :read [17 nil] INFO [2022-03-13 17:08:27,255] jepsen worker 7 - jepsen.util 367 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,255] jepsen worker 7 - jepsen.util 367 :invoke :write [2 36] INFO [2022-03-13 17:08:27,381] jepsen worker 62 - jepsen.util 332 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,382] jepsen worker 62 - jepsen.util 332 :invoke :read [20 nil] INFO [2022-03-13 17:08:27,411] jepsen worker 21 - jepsen.util 561 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,411] jepsen worker 21 - jepsen.util 561 :invoke :read [7 nil] INFO [2022-03-13 17:08:27,423] jepsen worker 77 - jepsen.util 437 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,423] jepsen worker 77 - jepsen.util 437 :invoke :write [25 15] INFO [2022-03-13 17:08:27,429] jepsen worker 42 - jepsen.util 402 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,430] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:27,454] jepsen worker 69 - jepsen.util 519 :ok :read [23 30] INFO [2022-03-13 17:08:27,454] jepsen worker 69 - jepsen.util 519 :invoke :write [23 18] INFO [2022-03-13 17:08:27,506] jepsen worker 74 - jepsen.util 704 :info :write [24 10] :RequestTimeoutException INFO [2022-03-13 17:08:27,507] jepsen worker 74 - jepsen.util 794 :invoke :write [24 20] INFO [2022-03-13 17:08:27,540] jepsen worker 52 - jepsen.util 412 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,541] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:27,547] jepsen worker 6 - jepsen.util 546 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,547] jepsen worker 6 - jepsen.util 546 :invoke :write [2 6] INFO [2022-03-13 17:08:27,549] jepsen worker 60 - jepsen.util 600 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,550] jepsen worker 60 - jepsen.util 600 :invoke :write [20 42] INFO [2022-03-13 17:08:27,708] jepsen worker 70 - jepsen.util 610 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,708] jepsen worker 51 - jepsen.util 321 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,708] jepsen worker 21 - jepsen.util 561 :ok :read [7 :nil] INFO [2022-03-13 17:08:27,709] jepsen worker 51 - jepsen.util 321 :invoke :read [17 nil] INFO [2022-03-13 17:08:27,709] jepsen worker 70 - jepsen.util 610 :invoke :read [23 nil] INFO [2022-03-13 17:08:27,709] jepsen worker 21 - jepsen.util 561 :invoke :read [7 nil] INFO [2022-03-13 17:08:27,788] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:27,789] jepsen worker 25 - jepsen.util 385 :invoke :write [8 46] INFO [2022-03-13 17:08:27,791] jepsen worker 51 - jepsen.util 321 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,792] jepsen worker 51 - jepsen.util 321 :invoke :read [17 nil] INFO [2022-03-13 17:08:27,817] jepsen worker 21 - jepsen.util 561 :ok :read [7 :nil] INFO [2022-03-13 17:08:27,818] jepsen worker 21 - jepsen.util 561 :invoke :read [7 nil] INFO [2022-03-13 17:08:27,819] jepsen worker 51 - jepsen.util 321 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,821] jepsen worker 51 - jepsen.util 321 :invoke :read [17 nil] INFO [2022-03-13 17:08:27,831] jepsen worker 70 - jepsen.util 610 :ok :read [23 30] INFO [2022-03-13 17:08:27,831] jepsen worker 70 - jepsen.util 610 :invoke :read [23 nil] INFO [2022-03-13 17:08:27,845] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,845] jepsen worker 79 - jepsen.util 709 :invoke :write [26 11] INFO [2022-03-13 17:08:27,916] jepsen worker 36 - jepsen.util 576 :info :write [12 48] :RequestTimeoutException INFO [2022-03-13 17:08:27,917] jepsen worker 36 - jepsen.util 666 :invoke :write [12 37] INFO [2022-03-13 17:08:27,917] jepsen worker 33 - jepsen.util 483 :ok :read [11 :nil] INFO [2022-03-13 17:08:27,918] jepsen worker 34 - jepsen.util 484 :ok :read [11 :nil] INFO [2022-03-13 17:08:27,919] jepsen worker 51 - jepsen.util 321 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,919] jepsen worker 33 - jepsen.util 483 :invoke :read [11 nil] INFO [2022-03-13 17:08:27,919] jepsen worker 34 - jepsen.util 484 :invoke :read [11 nil] INFO [2022-03-13 17:08:27,920] jepsen worker 51 - jepsen.util 321 :invoke :read [17 nil] INFO [2022-03-13 17:08:27,921] jepsen worker 21 - jepsen.util 561 :ok :read [7 :nil] INFO [2022-03-13 17:08:27,921] jepsen worker 21 - jepsen.util 561 :invoke :read [7 nil] INFO [2022-03-13 17:08:27,941] jepsen worker 51 - jepsen.util 321 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,942] jepsen worker 51 - jepsen.util 321 :invoke :write [17 25] INFO [2022-03-13 17:08:27,956] jepsen worker 70 - jepsen.util 610 :ok :read [23 30] INFO [2022-03-13 17:08:27,957] jepsen worker 70 - jepsen.util 610 :invoke :write [23 20] INFO [2022-03-13 17:08:27,976] jepsen worker 27 - jepsen.util 567 :info :write [9 27] :RequestTimeoutException INFO [2022-03-13 17:08:27,983] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:27,983] jepsen worker 27 - jepsen.util 657 :invoke :write [9 9] INFO [2022-03-13 17:08:27,983] jepsen worker 33 - jepsen.util 483 :ok :read [11 :nil] INFO [2022-03-13 17:08:27,984] jepsen worker 34 - jepsen.util 484 :ok :read [11 :nil] INFO [2022-03-13 17:08:27,984] jepsen worker 33 - jepsen.util 483 :invoke :read [11 nil] INFO [2022-03-13 17:08:27,985] jepsen worker 34 - jepsen.util 484 :invoke :write [11 7] INFO [2022-03-13 17:08:27,986] jepsen worker 52 - jepsen.util 412 :invoke :read [17 nil] INFO [2022-03-13 17:08:28,001] jepsen worker 62 - jepsen.util 332 :ok :read [20 12] INFO [2022-03-13 17:08:28,002] jepsen worker 62 - jepsen.util 332 :invoke :read [20 nil] INFO [2022-03-13 17:08:28,037] jepsen worker 13 - jepsen.util 373 :ok :read [4 :nil] INFO [2022-03-13 17:08:28,038] jepsen worker 13 - jepsen.util 373 :invoke :write [4 27] INFO [2022-03-13 17:08:28,059] jepsen worker 21 - jepsen.util 561 :ok :read [7 :nil] INFO [2022-03-13 17:08:28,059] jepsen worker 21 - jepsen.util 561 :invoke :read [7 nil] INFO [2022-03-13 17:08:28,096] jepsen worker 52 - jepsen.util 412 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:28,096] jepsen worker 33 - jepsen.util 483 :ok :read [11 :nil] INFO [2022-03-13 17:08:28,098] jepsen worker 33 - jepsen.util 483 :invoke :write [11 43] INFO [2022-03-13 17:08:28,099] jepsen worker 52 - jepsen.util 412 :invoke :write [17 7] INFO [2022-03-13 17:08:28,147] jepsen worker 71 - jepsen.util 431 :ok :read [23 30] INFO [2022-03-13 17:08:28,148] jepsen worker 71 - jepsen.util 431 :invoke :write [23 48] INFO [2022-03-13 17:08:28,159] jepsen worker 62 - jepsen.util 332 :ok :read [20 12] INFO [2022-03-13 17:08:28,159] jepsen worker 62 - jepsen.util 332 :invoke :write [20 23] INFO [2022-03-13 17:08:28,204] jepsen worker 21 - jepsen.util 561 :ok :read [7 :nil] INFO [2022-03-13 17:08:28,205] jepsen worker 21 - jepsen.util 561 :invoke :write [7 29] INFO [2022-03-13 17:08:28,239] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.02346041055718473}, :ns_1@172.23.105.187 {:progress 0.04093567251461994}, :ns_1@172.23.107.227 {:progress 0.02346041055718473}, :ns_1@172.23.107.228 {:progress 0.03529411764705881}} INFO [2022-03-13 17:08:28,240] jepsen worker nemesis - couchbase.util Rebalance check count: 21 INFO [2022-03-13 17:08:28,240] jepsen worker nemesis - couchbase.util Rebalance stuck count: 15 INFO [2022-03-13 17:08:28,726] jepsen worker 48 - jepsen.util 498 :ok :read [16 :nil] INFO [2022-03-13 17:08:28,727] jepsen worker 48 - jepsen.util 498 :invoke :write [16 10] INFO [2022-03-13 17:08:28,735] jepsen worker 15 - jepsen.util 645 :ok :read [5 32] INFO [2022-03-13 17:08:28,735] jepsen worker 15 - jepsen.util 645 :invoke :write [5 5] INFO [2022-03-13 17:08:28,739] jepsen worker 3 - jepsen.util 543 :ok :read [1 16] INFO [2022-03-13 17:08:28,740] jepsen worker 3 - jepsen.util 543 :invoke :read [1 nil] INFO [2022-03-13 17:08:28,754] jepsen worker 29 - jepsen.util 659 :info :write [9 10] :RequestTimeoutException INFO [2022-03-13 17:08:28,755] jepsen worker 29 - jepsen.util 749 :invoke :read [9 nil] INFO [2022-03-13 17:08:28,756] jepsen worker 24 - jepsen.util 384 :info :write [8 26] :RequestTimeoutException INFO [2022-03-13 17:08:28,756] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:28,788] jepsen worker 58 - jepsen.util 598 :ok :read [19 35] INFO [2022-03-13 17:08:28,789] jepsen worker 58 - jepsen.util 598 :invoke :read [19 nil] INFO [2022-03-13 17:08:29,246] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.02346041055718473}, :ns_1@172.23.105.187 {:progress 0.04093567251461994}, :ns_1@172.23.107.227 {:progress 0.02346041055718473}, :ns_1@172.23.107.228 {:progress 0.03529411764705881}} INFO [2022-03-13 17:08:29,246] jepsen worker nemesis - couchbase.util Rebalance check count: 22 INFO [2022-03-13 17:08:29,246] jepsen worker nemesis - couchbase.util Rebalance stuck count: 16 INFO [2022-03-13 17:08:29,272] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:29,272] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:29,343] jepsen worker 31 - jepsen.util 571 :ok :read [10 :nil] INFO [2022-03-13 17:08:29,344] jepsen worker 31 - jepsen.util 571 :invoke :write [10 1] INFO [2022-03-13 17:08:29,963] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:29,963] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:30,026] jepsen worker 88 - jepsen.util 718 :info :write [29 6] :RequestTimeoutException INFO [2022-03-13 17:08:30,027] jepsen worker 88 - jepsen.util 808 :invoke :write [29 46] INFO [2022-03-13 17:08:30,252] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.02346041055718473}, :ns_1@172.23.105.187 {:progress 0.04093567251461994}, :ns_1@172.23.107.227 {:progress 0.02346041055718473}, :ns_1@172.23.107.228 {:progress 0.03529411764705881}} INFO [2022-03-13 17:08:30,252] jepsen worker nemesis - couchbase.util Rebalance check count: 23 INFO [2022-03-13 17:08:30,252] jepsen worker nemesis - couchbase.util Rebalance stuck count: 17 INFO [2022-03-13 17:08:30,310] jepsen worker 1 - jepsen.util 631 :info :write [0 8] :RequestTimeoutException INFO [2022-03-13 17:08:30,311] jepsen worker 1 - jepsen.util 721 :invoke :write [0 42] INFO [2022-03-13 17:08:30,548] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:30,549] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:30,670] jepsen worker 39 - jepsen.util 759 :info :write [13 2] :RequestTimeoutException INFO [2022-03-13 17:08:30,671] jepsen worker 39 - jepsen.util 849 :invoke :write [13 18] INFO [2022-03-13 17:08:30,712] jepsen worker 49 - jepsen.util 589 :ok :read [16 :nil] INFO [2022-03-13 17:08:30,713] jepsen worker 49 - jepsen.util 589 :invoke :write [16 40] INFO [2022-03-13 17:08:30,718] jepsen worker 12 - jepsen.util 462 :info :write [4 1] :RequestTimeoutException INFO [2022-03-13 17:08:30,718] jepsen worker 12 - jepsen.util 552 :invoke :read [4 nil] INFO [2022-03-13 17:08:30,818] jepsen worker 83 - jepsen.util 533 :ok :read [27 4] INFO [2022-03-13 17:08:30,819] jepsen worker 83 - jepsen.util 533 :invoke :read [27 nil] INFO [2022-03-13 17:08:30,868] jepsen worker 83 - jepsen.util 533 :ok :read [27 4] INFO [2022-03-13 17:08:30,869] jepsen worker 83 - jepsen.util 533 :invoke :write [27 4] INFO [2022-03-13 17:08:30,997] jepsen worker 12 - jepsen.util 552 :ok :read [4 :nil] INFO [2022-03-13 17:08:30,998] jepsen worker 12 - jepsen.util 552 :invoke :write [4 18] INFO [2022-03-13 17:08:31,034] jepsen worker 40 - jepsen.util 580 :info :write [13 20] :SyncWriteAmbiguous INFO [2022-03-13 17:08:31,035] jepsen worker 40 - jepsen.util 670 :invoke :read [13 nil] INFO [2022-03-13 17:08:31,063] jepsen worker 67 - jepsen.util 697 :ok :read [22 30] INFO [2022-03-13 17:08:31,064] jepsen worker 67 - jepsen.util 697 :invoke :write [22 47] INFO [2022-03-13 17:08:31,105] jepsen worker 59 - jepsen.util 689 :ok :read [19 35] INFO [2022-03-13 17:08:31,106] jepsen worker 59 - jepsen.util 689 :invoke :read [19 nil] INFO [2022-03-13 17:08:31,259] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.02932551319648091}, :ns_1@172.23.105.187 {:progress 0.04093567251461994}, :ns_1@172.23.107.227 {:progress 0.02932551319648091}, :ns_1@172.23.107.228 {:progress 0.03529411764705881}} INFO [2022-03-13 17:08:31,259] jepsen worker nemesis - couchbase.util Rebalance check count: 24 INFO [2022-03-13 17:08:31,259] jepsen worker nemesis - couchbase.util Rebalance stuck count: 17 INFO [2022-03-13 17:08:31,354] jepsen worker 16 - jepsen.util 646 :ok :write [5 43] INFO [2022-03-13 17:08:31,355] jepsen worker 16 - jepsen.util 646 :invoke :write [5 15] INFO [2022-03-13 17:08:31,736] jepsen worker 8 - jepsen.util 548 :info :write [2 5] :RequestTimeoutException INFO [2022-03-13 17:08:31,737] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:31,777] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:31,780] jepsen worker 24 - jepsen.util 474 :invoke :write [8 6] INFO [2022-03-13 17:08:31,806] jepsen worker 50 - jepsen.util 680 :ok :read [16 :nil] INFO [2022-03-13 17:08:31,807] jepsen worker 50 - jepsen.util 680 :invoke :write [16 37] INFO [2022-03-13 17:08:31,816] jepsen worker 45 - jepsen.util 765 :info :write [15 16] :RequestTimeoutException INFO [2022-03-13 17:08:31,817] jepsen worker 45 - jepsen.util 855 :invoke :read [15 nil] INFO [2022-03-13 17:08:31,838] jepsen worker 2 - jepsen.util 722 :ok :read [0 41] INFO [2022-03-13 17:08:31,839] jepsen worker 2 - jepsen.util 722 :invoke :write [0 2] INFO [2022-03-13 17:08:31,858] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:31,859] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:31,930] jepsen worker 60 - jepsen.util 600 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:31,931] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:08:31,936] jepsen worker 23 - jepsen.util 383 :info :write [7 9] :RequestTimeoutException INFO [2022-03-13 17:08:31,937] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:32,171] jepsen worker 47 - jepsen.util 677 :info :write [15 38] :RequestTimeoutException INFO [2022-03-13 17:08:32,172] jepsen worker 47 - jepsen.util 767 :invoke :write [15 10] INFO [2022-03-13 17:08:32,180] jepsen worker 12 - jepsen.util 552 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,181] jepsen worker 12 - jepsen.util 552 :invoke :write [4 4] INFO [2022-03-13 17:08:32,188] jepsen worker 75 - jepsen.util 345 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,188] jepsen worker 75 - jepsen.util 345 :invoke :write [25 11] INFO [2022-03-13 17:08:32,190] jepsen worker 45 - jepsen.util 855 :ok :read [15 48] INFO [2022-03-13 17:08:32,193] jepsen worker 45 - jepsen.util 855 :invoke :read [15 nil] INFO [2022-03-13 17:08:32,246] jepsen worker 79 - jepsen.util 709 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,247] jepsen worker 79 - jepsen.util 709 :invoke :write [26 38] INFO [2022-03-13 17:08:32,248] jepsen worker 45 - jepsen.util 855 :ok :read [15 48] INFO [2022-03-13 17:08:32,249] jepsen worker 45 - jepsen.util 855 :invoke :read [15 nil] INFO [2022-03-13 17:08:32,267] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.03519061583577709}, :ns_1@172.23.105.187 {:progress 0.04678362573099415}, :ns_1@172.23.107.227 {:progress 0.03519061583577709}, :ns_1@172.23.107.228 {:progress 0.03529411764705881}} INFO [2022-03-13 17:08:32,268] jepsen worker nemesis - couchbase.util Rebalance check count: 25 INFO [2022-03-13 17:08:32,268] jepsen worker nemesis - couchbase.util Rebalance stuck count: 17 INFO [2022-03-13 17:08:32,279] jepsen worker 80 - jepsen.util 620 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,280] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:32,296] jepsen worker 70 - jepsen.util 610 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,296] jepsen worker 45 - jepsen.util 855 :ok :read [15 48] INFO [2022-03-13 17:08:32,297] jepsen worker 70 - jepsen.util 610 :invoke :write [23 7] INFO [2022-03-13 17:08:32,297] jepsen worker 45 - jepsen.util 855 :invoke :write [15 23] INFO [2022-03-13 17:08:32,349] jepsen worker 14 - jepsen.util 194 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,349] jepsen worker 14 - jepsen.util 194 :invoke :write [4 14] INFO [2022-03-13 17:08:32,365] jepsen worker 71 - jepsen.util 431 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,366] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:32,374] jepsen worker 21 - jepsen.util 561 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,375] jepsen worker 21 - jepsen.util 561 :invoke :read [7 nil] INFO [2022-03-13 17:08:32,404] jepsen worker 7 - jepsen.util 367 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,405] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:32,502] jepsen worker 78 - jepsen.util 438 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,503] jepsen worker 78 - jepsen.util 438 :invoke :write [26 6] INFO [2022-03-13 17:08:32,506] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,506] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:32,508] jepsen worker 12 - jepsen.util 552 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,509] jepsen worker 12 - jepsen.util 552 :invoke :read [4 nil] INFO [2022-03-13 17:08:32,522] jepsen worker 35 - jepsen.util 665 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,523] jepsen worker 35 - jepsen.util 665 :invoke :write [11 26] INFO [2022-03-13 17:08:32,548] jepsen worker 6 - jepsen.util 546 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,549] jepsen worker 54 - jepsen.util 684 :ok :write [18 47] INFO [2022-03-13 17:08:32,550] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:32,552] jepsen worker 54 - jepsen.util 684 :invoke :read [18 nil] INFO [2022-03-13 17:08:32,556] jepsen worker 84 - jepsen.util 804 :info :write [28 27] :RequestTimeoutException INFO [2022-03-13 17:08:32,556] jepsen worker 84 - jepsen.util 894 :invoke :read [28 nil] INFO [2022-03-13 17:08:32,573] jepsen worker 11 - jepsen.util 641 :ok :read [3 :nil] INFO [2022-03-13 17:08:32,574] jepsen worker 11 - jepsen.util 641 :invoke :read [3 nil] INFO [2022-03-13 17:08:32,589] jepsen worker 46 - jepsen.util 586 :ok :read [15 48] INFO [2022-03-13 17:08:32,590] jepsen worker 46 - jepsen.util 586 :invoke :read [15 nil] INFO [2022-03-13 17:08:32,600] jepsen worker 29 - jepsen.util 749 :ok :read [9 27] INFO [2022-03-13 17:08:32,600] jepsen worker 29 - jepsen.util 749 :invoke :read [9 nil] INFO [2022-03-13 17:08:32,619] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,620] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:32,627] jepsen worker 70 - jepsen.util 610 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,628] jepsen worker 70 - jepsen.util 610 :invoke :write [23 11] INFO [2022-03-13 17:08:32,643] jepsen worker 24 - jepsen.util 474 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,644] jepsen worker 24 - jepsen.util 474 :invoke :write [8 4] INFO [2022-03-13 17:08:32,671] jepsen worker 11 - jepsen.util 641 :ok :read [3 :nil] INFO [2022-03-13 17:08:32,672] jepsen worker 11 - jepsen.util 641 :invoke :read [3 nil] INFO [2022-03-13 17:08:32,687] jepsen worker 54 - jepsen.util 684 :ok :read [18 47] INFO [2022-03-13 17:08:32,687] jepsen worker 54 - jepsen.util 684 :invoke :read [18 nil] INFO [2022-03-13 17:08:32,695] jepsen worker 46 - jepsen.util 586 :ok :read [15 48] INFO [2022-03-13 17:08:32,696] jepsen worker 46 - jepsen.util 586 :invoke :read [15 nil] INFO [2022-03-13 17:08:32,705] jepsen worker 21 - jepsen.util 561 :ok :read [7 :nil] INFO [2022-03-13 17:08:32,705] jepsen worker 51 - jepsen.util 321 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,706] jepsen worker 51 - jepsen.util 321 :invoke :read [17 nil] INFO [2022-03-13 17:08:32,706] jepsen worker 21 - jepsen.util 561 :invoke :read [7 nil] INFO [2022-03-13 17:08:32,707] jepsen worker 22 - jepsen.util 472 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,709] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:32,728] jepsen worker 29 - jepsen.util 749 :ok :read [9 27] INFO [2022-03-13 17:08:32,729] jepsen worker 29 - jepsen.util 749 :invoke :write [9 32] INFO [2022-03-13 17:08:32,730] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,730] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:32,730] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:32,731] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:32,732] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,732] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:32,733] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:32,733] jepsen worker 22 - jepsen.util 472 :invoke :write [7 3] INFO [2022-03-13 17:08:32,734] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,734] jepsen worker 6 - jepsen.util 546 :invoke :write [2 9] INFO [2022-03-13 17:08:32,741] jepsen worker 11 - jepsen.util 641 :ok :read [3 :nil] INFO [2022-03-13 17:08:32,742] jepsen worker 11 - jepsen.util 641 :invoke :read [3 nil] INFO [2022-03-13 17:08:32,745] jepsen worker 71 - jepsen.util 431 :ok :read [23 30] INFO [2022-03-13 17:08:32,746] jepsen worker 71 - jepsen.util 431 :invoke :write [23 8] INFO [2022-03-13 17:08:32,775] jepsen worker 11 - jepsen.util 641 :ok :read [3 :nil] INFO [2022-03-13 17:08:32,775] jepsen worker 11 - jepsen.util 641 :invoke :read [3 nil] INFO [2022-03-13 17:08:32,780] jepsen worker 53 - jepsen.util 503 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,781] jepsen worker 53 - jepsen.util 503 :invoke :write [17 26] INFO [2022-03-13 17:08:32,788] jepsen worker 54 - jepsen.util 684 :ok :read [18 47] INFO [2022-03-13 17:08:32,788] jepsen worker 54 - jepsen.util 684 :invoke :write [18 16] INFO [2022-03-13 17:08:32,789] jepsen worker 46 - jepsen.util 586 :ok :read [15 48] INFO [2022-03-13 17:08:32,790] jepsen worker 46 - jepsen.util 586 :invoke :read [15 nil] INFO [2022-03-13 17:08:32,793] jepsen worker 25 - jepsen.util 385 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,794] jepsen worker 25 - jepsen.util 385 :invoke :write [8 18] INFO [2022-03-13 17:08:32,819] jepsen worker 69 - jepsen.util 519 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,819] jepsen worker 12 - jepsen.util 552 :ok :read [4 :nil] INFO [2022-03-13 17:08:32,820] jepsen worker 12 - jepsen.util 552 :invoke :read [4 nil] INFO [2022-03-13 17:08:32,821] jepsen worker 69 - jepsen.util 519 :invoke :read [23 nil] INFO [2022-03-13 17:08:32,828] jepsen worker 11 - jepsen.util 641 :ok :read [3 :nil] INFO [2022-03-13 17:08:32,829] jepsen worker 11 - jepsen.util 641 :invoke :read [3 nil] INFO [2022-03-13 17:08:32,850] jepsen worker 79 - jepsen.util 709 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,850] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,852] jepsen worker 79 - jepsen.util 709 :invoke :write [26 35] INFO [2022-03-13 17:08:32,852] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:32,861] jepsen worker 68 - jepsen.util 698 :info :write [22 48] :RequestTimeoutException INFO [2022-03-13 17:08:32,861] jepsen worker 68 - jepsen.util 788 :invoke :read [22 nil] INFO [2022-03-13 17:08:32,873] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,874] jepsen worker 7 - jepsen.util 367 :invoke :write [2 1] INFO [2022-03-13 17:08:32,881] jepsen worker 51 - jepsen.util 321 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,881] jepsen worker 21 - jepsen.util 561 :ok :read [7 :nil] INFO [2022-03-13 17:08:32,881] jepsen worker 70 - jepsen.util 610 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,881] jepsen worker 12 - jepsen.util 552 :ok :read [4 :nil] INFO [2022-03-13 17:08:32,881] jepsen worker 69 - jepsen.util 519 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,882] jepsen worker 21 - jepsen.util 561 :invoke :write [7 18] INFO [2022-03-13 17:08:32,883] jepsen worker 70 - jepsen.util 610 :invoke :write [23 32] INFO [2022-03-13 17:08:32,884] jepsen worker 51 - jepsen.util 321 :invoke :read [17 nil] INFO [2022-03-13 17:08:32,885] jepsen worker 69 - jepsen.util 519 :invoke :write [23 24] INFO [2022-03-13 17:08:32,885] jepsen worker 12 - jepsen.util 552 :invoke :write [4 36] INFO [2022-03-13 17:08:32,891] jepsen worker 51 - jepsen.util 321 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,892] jepsen worker 51 - jepsen.util 321 :invoke :write [17 49] INFO [2022-03-13 17:08:32,895] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:32,896] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:32,908] jepsen worker 46 - jepsen.util 586 :ok :read [15 48] INFO [2022-03-13 17:08:32,909] jepsen worker 46 - jepsen.util 586 :invoke :write [15 17] INFO [2022-03-13 17:08:32,927] jepsen worker 37 - jepsen.util 577 :info :write [12 4] :RequestTimeoutException INFO [2022-03-13 17:08:32,928] jepsen worker 37 - jepsen.util 667 :invoke :read [12 nil] INFO [2022-03-13 17:08:32,928] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:32,928] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:32,937] jepsen worker 11 - jepsen.util 641 :ok :read [3 :nil] INFO [2022-03-13 17:08:32,938] jepsen worker 11 - jepsen.util 641 :invoke :read [3 nil] INFO [2022-03-13 17:08:32,970] jepsen worker 87 - jepsen.util 807 :info :write [29 33] :RequestTimeoutException INFO [2022-03-13 17:08:32,971] jepsen worker 87 - jepsen.util 897 :invoke :write [29 37] INFO [2022-03-13 17:08:33,027] jepsen worker 11 - jepsen.util 641 :ok :read [3 :nil] INFO [2022-03-13 17:08:33,029] jepsen worker 11 - jepsen.util 641 :invoke :write [3 6] INFO [2022-03-13 17:08:33,081] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:33,081] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:33,096] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:33,096] jepsen worker 80 - jepsen.util 620 :invoke :write [26 20] INFO [2022-03-13 17:08:33,136] jepsen worker 41 - jepsen.util 671 :info :write [13 47] :RequestTimeoutException INFO [2022-03-13 17:08:33,137] jepsen worker 41 - jepsen.util 761 :invoke :read [13 nil] INFO [2022-03-13 17:08:33,274] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.04105571847507328}, :ns_1@172.23.105.187 {:progress 0.05263157894736847}, :ns_1@172.23.107.227 {:progress 0.04105571847507328}, :ns_1@172.23.107.228 {:progress 0.04117647058823526}} INFO [2022-03-13 17:08:33,274] jepsen worker nemesis - couchbase.util Rebalance check count: 26 INFO [2022-03-13 17:08:33,274] jepsen worker nemesis - couchbase.util Rebalance stuck count: 17 INFO [2022-03-13 17:08:33,324] jepsen worker 44 - jepsen.util 404 :info :write [14 44] :RequestTimeoutException INFO [2022-03-13 17:08:33,325] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:33,358] jepsen worker 86 - jepsen.util 716 :info :write [28 36] :RequestTimeoutException INFO [2022-03-13 17:08:33,359] jepsen worker 86 - jepsen.util 806 :invoke :read [28 nil] INFO [2022-03-13 17:08:33,412] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:33,412] jepsen worker 60 - jepsen.util 600 :invoke :write [20 22] INFO [2022-03-13 17:08:33,425] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:33,426] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:33,544] jepsen worker 38 - jepsen.util 668 :info :write [12 14] :SyncWriteAmbiguous INFO [2022-03-13 17:08:33,545] jepsen worker 38 - jepsen.util 758 :invoke :write [12 23] INFO [2022-03-13 17:08:33,590] jepsen worker 65 - jepsen.util 605 :info :write [21 44] :RequestTimeoutException INFO [2022-03-13 17:08:33,591] jepsen worker 65 - jepsen.util 695 :invoke :write [21 4] INFO [2022-03-13 17:08:33,604] jepsen worker 81 - jepsen.util 801 :info :write [27 28] :RequestTimeoutException INFO [2022-03-13 17:08:33,605] jepsen worker 81 - jepsen.util 891 :invoke :read [27 nil] INFO [2022-03-13 17:08:33,611] jepsen worker 82 - jepsen.util 712 :ok :read [27 4] INFO [2022-03-13 17:08:33,615] jepsen worker 82 - jepsen.util 712 :invoke :write [27 16] INFO [2022-03-13 17:08:33,819] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:33,820] jepsen worker 44 - jepsen.util 494 :invoke :write [14 22] INFO [2022-03-13 17:08:33,906] jepsen worker 89 - jepsen.util 629 :info :write [29 38] :RequestTimeoutException INFO [2022-03-13 17:08:33,907] jepsen worker 89 - jepsen.util 719 :invoke :write [29 6] INFO [2022-03-13 17:08:34,184] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:34,185] jepsen worker 42 - jepsen.util 402 :invoke :write [14 46] INFO [2022-03-13 17:08:34,198] jepsen worker 9 - jepsen.util 639 :info :write [3 33] :RequestTimeoutException INFO [2022-03-13 17:08:34,199] jepsen worker 9 - jepsen.util 729 :invoke :read [3 nil] INFO [2022-03-13 17:08:34,280] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.04105571847507328}, :ns_1@172.23.105.187 {:progress 0.05263157894736847}, :ns_1@172.23.107.227 {:progress 0.04105571847507328}, :ns_1@172.23.107.228 {:progress 0.04117647058823526}} INFO [2022-03-13 17:08:34,280] jepsen worker nemesis - couchbase.util Rebalance check count: 27 INFO [2022-03-13 17:08:34,280] jepsen worker nemesis - couchbase.util Rebalance stuck count: 18 INFO [2022-03-13 17:08:34,424] jepsen worker 8 - jepsen.util 638 :ok :read [2 5] INFO [2022-03-13 17:08:34,425] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:34,536] jepsen worker 10 - jepsen.util 460 :info :write [3 21] :RequestTimeoutException INFO [2022-03-13 17:08:34,537] jepsen worker 10 - jepsen.util 550 :invoke :write [3 28] INFO [2022-03-13 17:08:34,611] jepsen worker 43 - jepsen.util 313 :info :write [14 25] :RequestTimeoutException INFO [2022-03-13 17:08:34,611] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:34,791] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:34,791] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:34,972] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:34,973] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:34,981] jepsen worker 63 - jepsen.util 693 :info :write [21 19] :SyncWriteAmbiguous INFO [2022-03-13 17:08:34,982] jepsen worker 63 - jepsen.util 783 :invoke :read [21 nil] INFO [2022-03-13 17:08:35,166] jepsen worker 0 - jepsen.util 540 :info :write [0 12] :RequestTimeoutException INFO [2022-03-13 17:08:35,166] jepsen worker 0 - jepsen.util 630 :invoke :write [0 40] INFO [2022-03-13 17:08:35,170] jepsen worker 17 - jepsen.util 647 :info :write [5 7] :RequestTimeoutException INFO [2022-03-13 17:08:35,171] jepsen worker 17 - jepsen.util 737 :invoke :write [5 33] INFO [2022-03-13 17:08:35,188] jepsen worker 5 - jepsen.util 635 :info :write [1 2] :RequestTimeoutException INFO [2022-03-13 17:08:35,189] jepsen worker 5 - jepsen.util 725 :invoke :read [1 nil] INFO [2022-03-13 17:08:35,216] jepsen worker 73 - jepsen.util 703 :info :write [24 39] :RequestTimeoutException INFO [2022-03-13 17:08:35,216] jepsen worker 73 - jepsen.util 793 :invoke :read [24 nil] INFO [2022-03-13 17:08:35,235] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:35,236] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:35,286] jepsen worker 30 - jepsen.util 480 :info :write [10 18] :RequestTimeoutException INFO [2022-03-13 17:08:35,287] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.04105571847507328}, :ns_1@172.23.105.187 {:progress 0.05847953216374269}, :ns_1@172.23.107.227 {:progress 0.04105571847507328}, :ns_1@172.23.107.228 {:progress 0.04117647058823526}} INFO [2022-03-13 17:08:35,287] jepsen worker nemesis - couchbase.util Rebalance check count: 28 INFO [2022-03-13 17:08:35,287] jepsen worker nemesis - couchbase.util Rebalance stuck count: 18 INFO [2022-03-13 17:08:35,287] jepsen worker 30 - jepsen.util 570 :invoke :read [10 nil] WARN [2022-03-13 17:08:35,322] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":98330,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":98330,"total_server_duration_us":18723,"operation_id":"0x1660","timeout_ms":30000,"last_server_duration_us":18723,"total_duration_us":13423196},{"operation_name":"get","last_dispatch_duration_us":11894409,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11894409,"total_server_duration_us":43,"operation_id":"0x15ec","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":11895231},{"operation_name":"get","last_dispatch_duration_us":10689003,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10689003,"total_server_duration_us":56,"operation_id":"0x161a","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":10689391},{"operation_name":"upsert","last_dispatch_duration_us":8844659,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8844659,"total_server_duration_us":24583,"encode_duration_us":21757,"operation_id":"0x16de","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10006389},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":16792,"operation_id":"0x1658","timeout_ms":10000,"total_duration_us":10005778},{"operation_name":"upsert","last_dispatch_duration_us":10633848,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10633848,"total_server_duration_us":26864,"encode_duration_us":19007,"operation_id":"0x1635","timeout_ms":10000,"last_server_duration_us":26864,"total_duration_us":10004019},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17669,"operation_id":"0x17b0","timeout_ms":10000,"total_duration_us":10001505},{"operation_name":"get","last_dispatch_duration_us":9798286,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9798286,"total_server_duration_us":76,"operation_id":"0x1654","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":9799930},{"operation_name":"get","last_dispatch_duration_us":8866840,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8866840,"total_server_duration_us":38,"operation_id":"0x1683","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":8868325},{"operation_name":"get","last_dispatch_duration_us":8864982,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8864982,"total_server_duration_us":62,"operation_id":"0x175b","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":8865324}],"total_count":24}} WARN [2022-03-13 17:08:35,322] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17745419,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17987676,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9170054,"encode_duration_us":20017,"operation_id":"0x15a4","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10005117},{"operation_name":"upsert","last_dispatch_duration_us":19104453,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20683193,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9373981,"encode_duration_us":16325,"operation_id":"0x1478","timeout_ms":10000,"last_server_duration_us":9305434,"total_duration_us":10007299}],"total_count":2}} INFO [2022-03-13 17:08:35,397] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:35,398] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:35,520] jepsen worker 85 - jepsen.util 715 :info :write [28 7] :RequestTimeoutException INFO [2022-03-13 17:08:35,521] jepsen worker 85 - jepsen.util 805 :invoke :write [28 7] INFO [2022-03-13 17:08:35,529] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:35,529] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:35,547] jepsen worker 72 - jepsen.util 612 :info :write [24 18] :RequestTimeoutException INFO [2022-03-13 17:08:35,548] jepsen worker 72 - jepsen.util 702 :invoke :write [24 0] INFO [2022-03-13 17:08:35,692] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:35,693] jepsen worker 43 - jepsen.util 403 :invoke :write [14 30] INFO [2022-03-13 17:08:35,808] jepsen worker 18 - jepsen.util 468 :info :write [6 4] :SyncWriteAmbiguous INFO [2022-03-13 17:08:35,810] jepsen worker 18 - jepsen.util 558 :invoke :read [6 nil] INFO [2022-03-13 17:08:35,960] jepsen worker 43 - jepsen.util 403 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:35,961] jepsen worker 43 - jepsen.util 403 :invoke :write [14 16] INFO [2022-03-13 17:08:36,001] jepsen worker 14 - jepsen.util 194 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,001] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:36,064] jepsen worker 56 - jepsen.util 596 :info :write [18 27] :RequestTimeoutException INFO [2022-03-13 17:08:36,065] jepsen worker 56 - jepsen.util 686 :invoke :read [18 nil] INFO [2022-03-13 17:08:36,085] jepsen worker 7 - jepsen.util 367 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,086] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:36,123] jepsen worker 26 - jepsen.util 476 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,124] jepsen worker 26 - jepsen.util 476 :invoke :write [8 24] INFO [2022-03-13 17:08:36,172] jepsen worker 60 - jepsen.util 600 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,172] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:36,202] jepsen worker 42 - jepsen.util 402 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,203] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:36,239] jepsen worker 79 - jepsen.util 709 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,239] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:36,253] jepsen worker 19 - jepsen.util 469 :ok :read [6 16] INFO [2022-03-13 17:08:36,254] jepsen worker 19 - jepsen.util 469 :invoke :write [6 28] INFO [2022-03-13 17:08:36,269] jepsen worker 35 - jepsen.util 665 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,269] jepsen worker 35 - jepsen.util 665 :invoke :write [11 0] INFO [2022-03-13 17:08:36,293] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.04398826979472137}, :ns_1@172.23.105.187 {:progress 0.05847953216374269}, :ns_1@172.23.107.227 {:progress 0.04398826979472137}, :ns_1@172.23.107.228 {:progress 0.04705882352941182}} INFO [2022-03-13 17:08:36,293] jepsen worker nemesis - couchbase.util Rebalance check count: 29 INFO [2022-03-13 17:08:36,293] jepsen worker nemesis - couchbase.util Rebalance stuck count: 18 INFO [2022-03-13 17:08:36,298] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,298] jepsen worker 43 - jepsen.util 403 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,299] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:36,299] jepsen worker 43 - jepsen.util 403 :invoke :write [14 14] INFO [2022-03-13 17:08:36,329] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,329] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,329] jepsen worker 79 - jepsen.util 709 :invoke :write [26 30] INFO [2022-03-13 17:08:36,330] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:36,342] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:36,343] jepsen worker 60 - jepsen.util 600 :invoke :write [20 18] INFO [2022-03-13 17:08:36,367] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:36,367] jepsen worker 61 - jepsen.util 421 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,367] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:36,368] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:36,423] jepsen worker 43 - jepsen.util 403 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,423] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,424] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:36,424] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:36,446] jepsen worker 53 - jepsen.util 503 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,447] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:36,493] jepsen worker 79 - jepsen.util 709 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,494] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:36,515] jepsen worker 26 - jepsen.util 476 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,515] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:36,515] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:36,516] jepsen worker 42 - jepsen.util 402 :invoke :write [14 26] INFO [2022-03-13 17:08:36,590] jepsen worker 60 - jepsen.util 600 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,590] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:36,591] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,591] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,591] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:36,591] jepsen worker 7 - jepsen.util 367 :invoke :write [2 15] INFO [2022-03-13 17:08:36,592] jepsen worker 43 - jepsen.util 403 :invoke :write [14 37] INFO [2022-03-13 17:08:36,592] jepsen worker 79 - jepsen.util 709 :invoke :write [26 48] INFO [2022-03-13 17:08:36,625] jepsen worker 76 - jepsen.util 526 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,625] jepsen worker 76 - jepsen.util 526 :invoke :write [25 22] INFO [2022-03-13 17:08:36,660] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:36,660] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:36,710] jepsen worker 35 - jepsen.util 665 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,710] jepsen worker 35 - jepsen.util 665 :invoke :write [11 23] INFO [2022-03-13 17:08:36,730] jepsen worker 80 - jepsen.util 620 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,731] jepsen worker 64 - jepsen.util 424 :info :write [21 36] :RequestTimeoutException INFO [2022-03-13 17:08:36,731] jepsen worker 80 - jepsen.util 620 :invoke :write [26 20] INFO [2022-03-13 17:08:36,732] jepsen worker 64 - jepsen.util 514 :invoke :read [21 nil] INFO [2022-03-13 17:08:36,741] jepsen worker 20 - jepsen.util 740 :info :write [6 13] :RequestTimeoutException INFO [2022-03-13 17:08:36,741] jepsen worker 20 - jepsen.util 830 :invoke :write [6 29] INFO [2022-03-13 17:08:36,750] jepsen worker 32 - jepsen.util 482 :info :write [10 27] :RequestTimeoutException INFO [2022-03-13 17:08:36,750] jepsen worker 28 - jepsen.util 658 :info :write [9 32] :RequestTimeoutException INFO [2022-03-13 17:08:36,752] jepsen worker 28 - jepsen.util 748 :invoke :write [9 5] INFO [2022-03-13 17:08:36,754] jepsen worker 32 - jepsen.util 572 :invoke :read [10 nil] INFO [2022-03-13 17:08:36,755] jepsen worker 7 - jepsen.util 367 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,756] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:36,780] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:36,781] jepsen worker 14 - jepsen.util 194 :invoke :write [4 17] INFO [2022-03-13 17:08:36,797] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:36,798] jepsen worker 61 - jepsen.util 421 :invoke :write [20 35] WARN [2022-03-13 17:08:36,820] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":12211290,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12211290,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":2744395,"encode_duration_us":18189,"operation_id":"0x1659","timeout_ms":10000,"last_server_duration_us":2744395,"total_duration_us":10001037},{"operation_name":"upsert","last_dispatch_duration_us":23606675,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24531578,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9548122,"encode_duration_us":18275,"operation_id":"0x13ab","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10001587},{"operation_name":"upsert","last_dispatch_duration_us":11520375,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17260587,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":3096537,"encode_duration_us":28983,"operation_id":"0x15c3","timeout_ms":10000,"last_server_duration_us":3075294,"total_duration_us":10006042}],"total_count":3}} INFO [2022-03-13 17:08:36,871] jepsen worker 22 - jepsen.util 472 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,872] jepsen worker 22 - jepsen.util 472 :invoke :write [7 4] INFO [2022-03-13 17:08:36,884] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,884] jepsen worker 79 - jepsen.util 709 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,884] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:36,884] jepsen worker 75 - jepsen.util 345 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,885] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:36,887] jepsen worker 26 - jepsen.util 476 :invoke :write [8 11] INFO [2022-03-13 17:08:36,887] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:36,887] jepsen worker 79 - jepsen.util 709 :invoke :write [26 33] WARN [2022-03-13 17:08:36,921] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9168339,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9168339,"total_server_duration_us":32,"operation_id":"0x1661","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11281216},{"operation_name":"get","last_dispatch_duration_us":11237741,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11237741,"total_server_duration_us":27,"operation_id":"0x16f2","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":11238154},{"operation_name":"upsert","last_dispatch_duration_us":2352190,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":11310982,"total_server_duration_us":393767,"encode_duration_us":17666,"operation_id":"0x1707","timeout_ms":10000,"last_server_duration_us":32548,"total_duration_us":10008367},{"operation_name":"upsert","last_dispatch_duration_us":8495864,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8495864,"total_server_duration_us":24841,"encode_duration_us":24351,"operation_id":"0x17de","timeout_ms":10000,"last_server_duration_us":24841,"total_duration_us":10007091},{"operation_name":"upsert","last_dispatch_duration_us":100826,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9154269,"total_server_duration_us":192183,"encode_duration_us":17174,"operation_id":"0x1821","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10003414},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":19574,"operation_id":"0x185b","timeout_ms":10000,"total_duration_us":10001570},{"operation_name":"upsert","last_dispatch_duration_us":12211290,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12211290,"total_server_duration_us":2744395,"encode_duration_us":18189,"operation_id":"0x1659","timeout_ms":10000,"last_server_duration_us":2744395,"total_duration_us":10001037},{"operation_name":"get","last_dispatch_duration_us":8953623,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8953623,"total_server_duration_us":32,"operation_id":"0x17b1","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8956111},{"operation_name":"get","last_dispatch_duration_us":8896070,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8896070,"total_server_duration_us":15,"operation_id":"0x16d8","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":8898724},{"operation_name":"get","last_dispatch_duration_us":8848901,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8848901,"total_server_duration_us":23,"operation_id":"0x16d9","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":8849317}],"total_count":28}} WARN [2022-03-13 17:08:36,925] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":8152341,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8152341,"total_server_duration_us":32,"operation_id":"0x1863","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":10773985},{"operation_name":"upsert","last_dispatch_duration_us":239321,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":6408385,"total_server_duration_us":172179,"encode_duration_us":17312,"operation_id":"0x1859","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10005791},{"operation_name":"upsert","last_dispatch_duration_us":5719596,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8806366,"total_server_duration_us":372830,"encode_duration_us":19641,"operation_id":"0x16dc","timeout_ms":10000,"last_server_duration_us":38899,"total_duration_us":10005770},{"operation_name":"upsert","last_dispatch_duration_us":5757334,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7744846,"total_server_duration_us":256432,"encode_duration_us":17583,"operation_id":"0x178f","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10005735},{"operation_name":"upsert","last_dispatch_duration_us":7162466,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8246971,"total_server_duration_us":388803,"encode_duration_us":28756,"operation_id":"0x1686","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10005409},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17207,"operation_id":"0x1836","timeout_ms":10000,"total_duration_us":10003955},{"operation_name":"upsert","last_dispatch_duration_us":9227571,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10083362,"total_server_duration_us":9302870,"encode_duration_us":20550,"operation_id":"0x1669","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10002232},{"operation_name":"upsert","last_dispatch_duration_us":325556,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":7975895,"total_server_duration_us":864768,"encode_duration_us":17624,"operation_id":"0x1751","timeout_ms":10000,"last_server_duration_us":64473,"total_duration_us":10001898},{"operation_name":"upsert","last_dispatch_duration_us":1014372,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10012181,"total_server_duration_us":946078,"encode_duration_us":17180,"operation_id":"0x16ad","timeout_ms":10000,"last_server_duration_us":69481,"total_duration_us":10000630},{"operation_name":"upsert","last_dispatch_duration_us":9115287,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9538887,"total_server_duration_us":9124190,"encode_duration_us":21634,"operation_id":"0x1677","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9578195}],"total_count":18}} WARN [2022-03-13 17:08:36,925] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9053192,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9656631,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9058552,"encode_duration_us":20915,"operation_id":"0x165d","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10004570},{"operation_name":"upsert","last_dispatch_duration_us":9042592,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13343985,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9097328,"encode_duration_us":19926,"operation_id":"0x1617","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10004732},{"operation_name":"upsert","last_dispatch_duration_us":12508470,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12986503,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9289183,"encode_duration_us":28185,"operation_id":"0x16ac","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10004916}],"total_count":3}} WARN [2022-03-13 17:08:36,925] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9227571,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10083362,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9302870,"encode_duration_us":20550,"operation_id":"0x1669","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10002232},{"operation_name":"upsert","last_dispatch_duration_us":15492668,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15981835,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":8480431,"encode_duration_us":18246,"operation_id":"0x15f3","timeout_ms":10000,"last_server_duration_us":8341282,"total_duration_us":10010016}],"total_count":2}} WARN [2022-03-13 17:08:36,926] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":8853137,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8853137,"total_server_duration_us":484,"operation_id":"0x16c9","timeout_ms":30000,"last_server_duration_us":484,"total_duration_us":14024057},{"operation_name":"upsert","last_dispatch_duration_us":29135,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":567811,"total_server_duration_us":315427,"encode_duration_us":18217,"operation_id":"0x17fb","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10010371},{"operation_name":"upsert","last_dispatch_duration_us":503795,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3793898,"total_server_duration_us":320315,"encode_duration_us":21432,"operation_id":"0x1845","timeout_ms":10000,"last_server_duration_us":67221,"total_duration_us":10007120},{"operation_name":"upsert","last_dispatch_duration_us":8490237,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8490237,"total_server_duration_us":21813,"encode_duration_us":17653,"operation_id":"0x185a","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10004298},{"operation_name":"upsert","last_dispatch_duration_us":61879,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":634201,"total_server_duration_us":422874,"encode_duration_us":18671,"operation_id":"0x1778","timeout_ms":10000,"last_server_duration_us":49951,"total_duration_us":10003748},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18772,"operation_id":"0x17b8","timeout_ms":10000,"total_duration_us":10001858},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":29063,"operation_id":"0x1870","timeout_ms":10000,"total_duration_us":10001791},{"operation_name":"upsert","last_dispatch_duration_us":8815292,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8815292,"total_server_duration_us":26774,"encode_duration_us":20560,"operation_id":"0x1705","timeout_ms":10000,"last_server_duration_us":26774,"total_duration_us":10001716},{"operation_name":"upsert","last_dispatch_duration_us":9074006,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9491159,"total_server_duration_us":9327758,"encode_duration_us":21526,"operation_id":"0x1861","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":9598880},{"operation_name":"upsert","last_dispatch_duration_us":9046020,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9046020,"total_server_duration_us":9034183,"encode_duration_us":18674,"operation_id":"0x182f","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":9050000}],"total_count":26}} INFO [2022-03-13 17:08:36,962] jepsen worker 42 - jepsen.util 402 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,962] jepsen worker 43 - jepsen.util 403 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,963] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:36,964] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:36,974] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:36,974] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:36,990] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:36,990] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] WARN [2022-03-13 17:08:37,025] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9090004,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9959814,"total_server_duration_us":9309386,"encode_duration_us":19461,"operation_id":"0x185f","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10009172},{"operation_name":"upsert","last_dispatch_duration_us":37286,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1367205,"total_server_duration_us":626886,"encode_duration_us":30501,"operation_id":"0x1685","timeout_ms":10000,"last_server_duration_us":27222,"total_duration_us":10008289},{"operation_name":"upsert","last_dispatch_duration_us":40233,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1135605,"total_server_duration_us":538484,"encode_duration_us":32039,"operation_id":"0x17b6","timeout_ms":10000,"last_server_duration_us":28768,"total_duration_us":10007177},{"operation_name":"upsert","last_dispatch_duration_us":118851,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1182759,"total_server_duration_us":520938,"encode_duration_us":17009,"operation_id":"0x17be","timeout_ms":10000,"last_server_duration_us":30256,"total_duration_us":10006351},{"operation_name":"upsert","last_dispatch_duration_us":8946235,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9893724,"total_server_duration_us":301686,"encode_duration_us":17607,"operation_id":"0x185d","timeout_ms":10000,"last_server_duration_us":46852,"total_duration_us":10005222},{"operation_name":"upsert","last_dispatch_duration_us":61803,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":795299,"total_server_duration_us":219579,"encode_duration_us":27769,"operation_id":"0x1878","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":10004233},{"operation_name":"upsert","last_dispatch_duration_us":9209543,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12834578,"total_server_duration_us":9133799,"encode_duration_us":17328,"operation_id":"0x1759","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10003153},{"operation_name":"upsert","last_dispatch_duration_us":80710,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":981018,"total_server_duration_us":509911,"encode_duration_us":20580,"operation_id":"0x1814","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10002900},{"operation_name":"upsert","last_dispatch_duration_us":34089,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1378191,"total_server_duration_us":677918,"encode_duration_us":20692,"operation_id":"0x165b","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10001886},{"operation_name":"upsert","last_dispatch_duration_us":838825,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3493827,"total_server_duration_us":252093,"encode_duration_us":16997,"operation_id":"0x1764","timeout_ms":10000,"last_server_duration_us":54576,"total_duration_us":10000678}],"total_count":18}} WARN [2022-03-13 17:08:37,025] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":18764,"operation_id":"0x1802","timeout_ms":10000,"total_duration_us":10006226},{"operation_name":"upsert","last_dispatch_duration_us":12508470,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":12986503,"total_server_duration_us":9289183,"encode_duration_us":28185,"operation_id":"0x16ac","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10004916},{"operation_name":"upsert","last_dispatch_duration_us":9053192,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9656631,"total_server_duration_us":9058552,"encode_duration_us":20915,"operation_id":"0x165d","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10004570},{"operation_name":"upsert","last_dispatch_duration_us":924556,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3299629,"total_server_duration_us":329554,"encode_duration_us":35859,"operation_id":"0x17d8","timeout_ms":10000,"last_server_duration_us":56401,"total_duration_us":10003151},{"operation_name":"upsert","last_dispatch_duration_us":41454,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6372716,"total_server_duration_us":174602,"encode_duration_us":20026,"operation_id":"0x188e","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10002262},{"operation_name":"upsert","last_dispatch_duration_us":387014,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4948197,"total_server_duration_us":198478,"encode_duration_us":22389,"operation_id":"0x18c6","timeout_ms":10000,"last_server_duration_us":33524,"total_duration_us":9620587},{"operation_name":"upsert","last_dispatch_duration_us":982745,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":7823762,"total_server_duration_us":566195,"encode_duration_us":41226,"operation_id":"0x17dd","timeout_ms":10000,"last_server_duration_us":90182,"total_duration_us":9448533},{"operation_name":"upsert","last_dispatch_duration_us":325212,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":7180411,"total_server_duration_us":301247,"encode_duration_us":17810,"operation_id":"0x18c4","timeout_ms":10000,"last_server_duration_us":28584,"total_duration_us":9307613},{"operation_name":"upsert","last_dispatch_duration_us":854514,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4021233,"total_server_duration_us":440204,"encode_duration_us":30033,"operation_id":"0x17d7","timeout_ms":10000,"last_server_duration_us":58004,"total_duration_us":9204648},{"operation_name":"upsert","last_dispatch_duration_us":367286,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":7059880,"total_server_duration_us":743820,"encode_duration_us":18122,"operation_id":"0x18d1","timeout_ms":10000,"last_server_duration_us":36831,"total_duration_us":9176176}],"total_count":25}} WARN [2022-03-13 17:08:37,026] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17530303,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18700304,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9241801,"encode_duration_us":19627,"operation_id":"0x15c6","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10004392}],"total_count":1}} INFO [2022-03-13 17:08:37,027] jepsen worker 76 - jepsen.util 526 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,028] jepsen worker 79 - jepsen.util 709 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,028] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:37,028] jepsen worker 76 - jepsen.util 526 :invoke :write [25 16] INFO [2022-03-13 17:08:37,030] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,032] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:37,034] jepsen worker 43 - jepsen.util 403 :invoke :write [14 46] INFO [2022-03-13 17:08:37,034] jepsen worker 79 - jepsen.util 709 :invoke :write [26 46] INFO [2022-03-13 17:08:37,044] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,045] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:37,046] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,047] jepsen worker 7 - jepsen.util 367 :invoke :write [2 45] INFO [2022-03-13 17:08:37,064] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:37,064] jepsen worker 60 - jepsen.util 600 :invoke :write [20 13] INFO [2022-03-13 17:08:37,085] jepsen worker 35 - jepsen.util 665 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,085] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] WARN [2022-03-13 17:08:37,125] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9209543,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12834578,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9133799,"encode_duration_us":17328,"operation_id":"0x1759","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10003153},{"operation_name":"upsert","last_dispatch_duration_us":8967263,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10956742,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":288533,"encode_duration_us":20800,"operation_id":"0x1593","timeout_ms":10000,"last_server_duration_us":60759,"total_duration_us":10008597},{"operation_name":"upsert","last_dispatch_duration_us":9090004,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9959814,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9309386,"encode_duration_us":19461,"operation_id":"0x185f","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10009172}],"total_count":3}} INFO [2022-03-13 17:08:37,166] jepsen worker 55 - jepsen.util 685 :info :write [18 22] :RequestTimeoutException INFO [2022-03-13 17:08:37,167] jepsen worker 55 - jepsen.util 775 :invoke :write [18 26] INFO [2022-03-13 17:08:37,168] jepsen worker 14 - jepsen.util 194 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,168] jepsen worker 43 - jepsen.util 403 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,169] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:37,170] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:37,171] jepsen worker 57 - jepsen.util 687 :info :write [19 2] :RequestTimeoutException INFO [2022-03-13 17:08:37,171] jepsen worker 57 - jepsen.util 777 :invoke :read [19 nil] INFO [2022-03-13 17:08:37,253] jepsen worker 66 - jepsen.util 696 :info :write [22 1] :RequestTimeoutException INFO [2022-03-13 17:08:37,254] jepsen worker 66 - jepsen.util 786 :invoke :write [22 19] INFO [2022-03-13 17:08:37,300] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.04398826979472137}, :ns_1@172.23.105.187 {:progress 0.05847953216374269}, :ns_1@172.23.107.227 {:progress 0.04398826979472137}, :ns_1@172.23.107.228 {:progress 0.04705882352941182}} INFO [2022-03-13 17:08:37,300] jepsen worker nemesis - couchbase.util Rebalance check count: 30 INFO [2022-03-13 17:08:37,300] jepsen worker nemesis - couchbase.util Rebalance stuck count: 19 INFO [2022-03-13 17:08:37,421] jepsen worker 61 - jepsen.util 421 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,421] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,421] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,422] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:37,423] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:37,423] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:37,451] jepsen worker 77 - jepsen.util 437 :info :write [25 15] :RequestTimeoutException INFO [2022-03-13 17:08:37,451] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:37,478] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:37,479] jepsen worker 42 - jepsen.util 402 :invoke :write [14 24] INFO [2022-03-13 17:08:37,536] jepsen worker 74 - jepsen.util 794 :info :write [24 20] :RequestTimeoutException INFO [2022-03-13 17:08:37,537] jepsen worker 74 - jepsen.util 884 :invoke :write [24 5] INFO [2022-03-13 17:08:37,537] jepsen worker 59 - jepsen.util 689 :ok :read [19 35] INFO [2022-03-13 17:08:37,538] jepsen worker 59 - jepsen.util 689 :invoke :read [19 nil] INFO [2022-03-13 17:08:37,544] jepsen worker 35 - jepsen.util 665 :ok :read [11 :nil] INFO [2022-03-13 17:08:37,545] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:37,545] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:37,546] jepsen worker 14 - jepsen.util 194 :invoke :write [4 44] INFO [2022-03-13 17:08:37,546] jepsen worker 43 - jepsen.util 403 :invoke :write [14 30] INFO [2022-03-13 17:08:37,547] jepsen worker 35 - jepsen.util 665 :invoke :write [11 13] INFO [2022-03-13 17:08:37,569] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,569] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,571] jepsen worker 53 - jepsen.util 503 :invoke :write [17 25] INFO [2022-03-13 17:08:37,576] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:37,590] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:37,591] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:37,592] jepsen worker 81 - jepsen.util 891 :ok :read [27 4] INFO [2022-03-13 17:08:37,593] jepsen worker 81 - jepsen.util 891 :invoke :read [27 nil] INFO [2022-03-13 17:08:37,628] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,629] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:37,677] jepsen worker 59 - jepsen.util 689 :ok :read [19 35] INFO [2022-03-13 17:08:37,678] jepsen worker 59 - jepsen.util 689 :invoke :write [19 35] INFO [2022-03-13 17:08:37,710] jepsen worker 81 - jepsen.util 891 :ok :read [27 4] INFO [2022-03-13 17:08:37,711] jepsen worker 81 - jepsen.util 891 :invoke :read [27 nil] INFO [2022-03-13 17:08:37,806] jepsen worker 32 - jepsen.util 572 :ok :read [10 :nil] INFO [2022-03-13 17:08:37,807] jepsen worker 32 - jepsen.util 572 :invoke :read [10 nil] INFO [2022-03-13 17:08:37,809] jepsen worker 81 - jepsen.util 891 :ok :read [27 4] INFO [2022-03-13 17:08:37,810] jepsen worker 81 - jepsen.util 891 :invoke :write [27 9] INFO [2022-03-13 17:08:37,811] jepsen worker 32 - jepsen.util 572 :ok :read [10 :nil] INFO [2022-03-13 17:08:37,811] jepsen worker 32 - jepsen.util 572 :invoke :write [10 33] INFO [2022-03-13 17:08:37,912] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,913] jepsen worker 77 - jepsen.util 527 :invoke :write [25 17] INFO [2022-03-13 17:08:37,946] jepsen worker 36 - jepsen.util 666 :info :write [12 37] :RequestTimeoutException INFO [2022-03-13 17:08:37,947] jepsen worker 36 - jepsen.util 756 :invoke :read [12 nil] INFO [2022-03-13 17:08:37,949] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:37,950] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:38,000] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:38,001] jepsen worker 61 - jepsen.util 421 :invoke :write [20 17] INFO [2022-03-13 17:08:38,011] jepsen worker 27 - jepsen.util 657 :info :write [9 9] :RequestTimeoutException INFO [2022-03-13 17:08:38,011] jepsen worker 27 - jepsen.util 747 :invoke :write [9 37] INFO [2022-03-13 17:08:38,016] jepsen worker 34 - jepsen.util 484 :info :write [11 7] :RequestTimeoutException INFO [2022-03-13 17:08:38,016] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:08:38,060] jepsen worker 13 - jepsen.util 373 :info :write [4 27] :RequestTimeoutException INFO [2022-03-13 17:08:38,061] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:38,157] jepsen worker 52 - jepsen.util 412 :info :write [17 7] :RequestTimeoutException INFO [2022-03-13 17:08:38,157] jepsen worker 33 - jepsen.util 483 :info :write [11 43] :RequestTimeoutException INFO [2022-03-13 17:08:38,159] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:38,159] jepsen worker 52 - jepsen.util 502 :invoke :write [17 22] INFO [2022-03-13 17:08:38,178] jepsen worker 36 - jepsen.util 756 :ok :read [12 46] INFO [2022-03-13 17:08:38,178] jepsen worker 36 - jepsen.util 756 :invoke :write [12 25] INFO [2022-03-13 17:08:38,196] jepsen worker 62 - jepsen.util 332 :info :write [20 23] :RequestTimeoutException INFO [2022-03-13 17:08:38,196] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:08:38,310] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.05865102639296182}, :ns_1@172.23.105.187 {:progress 0.07602339181286555}, :ns_1@172.23.107.227 {:progress 0.05865102639296182}, :ns_1@172.23.107.228 {:progress 0.05294117647058827}} INFO [2022-03-13 17:08:38,311] jepsen worker nemesis - couchbase.util Rebalance check count: 31 INFO [2022-03-13 17:08:38,311] jepsen worker nemesis - couchbase.util Rebalance stuck count: 19 INFO [2022-03-13 17:08:38,351] jepsen worker 36 - jepsen.util 756 :fail :write [12 25] :DurabilityImpossible INFO [2022-03-13 17:08:38,352] jepsen worker 36 - jepsen.util 756 :invoke :read [12 nil] INFO [2022-03-13 17:08:38,487] jepsen worker 36 - jepsen.util 756 :ok :read [12 46] INFO [2022-03-13 17:08:38,488] jepsen worker 36 - jepsen.util 756 :invoke :write [12 48] INFO [2022-03-13 17:08:38,648] jepsen worker 36 - jepsen.util 756 :fail :write [12 48] :DurabilityImpossible INFO [2022-03-13 17:08:38,649] jepsen worker 36 - jepsen.util 756 :invoke :write [12 17] INFO [2022-03-13 17:08:38,753] jepsen worker 33 - jepsen.util 573 :ok :read [11 :nil] INFO [2022-03-13 17:08:38,753] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:38,754] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:38,754] jepsen worker 75 - jepsen.util 345 :invoke :write [25 45] INFO [2022-03-13 17:08:38,768] jepsen worker 48 - jepsen.util 498 :info :write [16 10] :RequestTimeoutException INFO [2022-03-13 17:08:38,768] jepsen worker 15 - jepsen.util 645 :info :write [5 5] :RequestTimeoutException INFO [2022-03-13 17:08:38,769] jepsen worker 15 - jepsen.util 735 :invoke :read [5 nil] INFO [2022-03-13 17:08:38,770] jepsen worker 48 - jepsen.util 588 :invoke :read [16 nil] INFO [2022-03-13 17:08:39,026] jepsen worker 62 - jepsen.util 422 :ok :read [20 12] INFO [2022-03-13 17:08:39,027] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:08:39,114] jepsen worker 79 - jepsen.util 709 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:39,114] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:39,116] jepsen worker 33 - jepsen.util 573 :ok :read [11 :nil] INFO [2022-03-13 17:08:39,117] jepsen worker 33 - jepsen.util 573 :invoke :write [11 18] INFO [2022-03-13 17:08:39,164] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:39,165] jepsen worker 8 - jepsen.util 638 :invoke :write [2 9] INFO [2022-03-13 17:08:39,317] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.05865102639296182}, :ns_1@172.23.105.187 {:progress 0.07602339181286555}, :ns_1@172.23.107.227 {:progress 0.05865102639296182}, :ns_1@172.23.107.228 {:progress 0.06470588235294117}} INFO [2022-03-13 17:08:39,317] jepsen worker nemesis - couchbase.util Rebalance check count: 32 INFO [2022-03-13 17:08:39,317] jepsen worker nemesis - couchbase.util Rebalance stuck count: 19 INFO [2022-03-13 17:08:39,368] jepsen worker 31 - jepsen.util 571 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:08:39,368] jepsen worker 31 - jepsen.util 661 :invoke :write [10 26] INFO [2022-03-13 17:08:39,577] jepsen worker 34 - jepsen.util 574 :ok :read [11 :nil] INFO [2022-03-13 17:08:39,577] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:08:39,711] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:39,712] jepsen worker 79 - jepsen.util 709 :invoke :write [26 42] INFO [2022-03-13 17:08:39,749] jepsen worker 62 - jepsen.util 422 :ok :read [20 12] INFO [2022-03-13 17:08:39,749] jepsen worker 62 - jepsen.util 422 :invoke :write [20 22] INFO [2022-03-13 17:08:39,821] jepsen worker 49 - jepsen.util 589 :info :write [16 40] :SyncWriteAmbiguous INFO [2022-03-13 17:08:39,822] jepsen worker 49 - jepsen.util 679 :invoke :read [16 nil] INFO [2022-03-13 17:08:39,958] jepsen worker 83 - jepsen.util 533 :info :write [27 4] :SyncWriteAmbiguous INFO [2022-03-13 17:08:39,958] jepsen worker 83 - jepsen.util 623 :invoke :read [27 nil] INFO [2022-03-13 17:08:40,055] jepsen worker 88 - jepsen.util 808 :info :write [29 46] :RequestTimeoutException INFO [2022-03-13 17:08:40,055] jepsen worker 88 - jepsen.util 898 :invoke :read [29 nil] INFO [2022-03-13 17:08:40,082] jepsen worker 44 - jepsen.util 494 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,085] jepsen worker 39 - jepsen.util 849 :info :write [13 18] :SyncWriteAmbiguous INFO [2022-03-13 17:08:40,084] jepsen worker 44 - jepsen.util 494 :invoke :write [14 7] INFO [2022-03-13 17:08:40,088] jepsen worker 39 - jepsen.util 939 :invoke :read [13 nil] INFO [2022-03-13 17:08:40,127] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:08:40,128] jepsen worker 34 - jepsen.util 574 :invoke :write [11 38] INFO [2022-03-13 17:08:40,150] jepsen worker 26 - jepsen.util 476 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,151] jepsen worker 7 - jepsen.util 367 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,151] jepsen worker 26 - jepsen.util 476 :invoke :write [8 40] INFO [2022-03-13 17:08:40,156] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:40,229] jepsen worker 25 - jepsen.util 385 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,231] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:40,263] jepsen worker 84 - jepsen.util 894 :ok :read [28 44] INFO [2022-03-13 17:08:40,263] jepsen worker 84 - jepsen.util 894 :invoke :write [28 33] INFO [2022-03-13 17:08:40,271] jepsen worker 22 - jepsen.util 472 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,272] jepsen worker 22 - jepsen.util 472 :invoke :write [7 32] INFO [2022-03-13 17:08:40,290] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:40,291] jepsen worker 23 - jepsen.util 473 :invoke :write [7 8] INFO [2022-03-13 17:08:40,323] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.05865102639296182}, :ns_1@172.23.105.187 {:progress 0.07602339181286555}, :ns_1@172.23.107.227 {:progress 0.05865102639296182}, :ns_1@172.23.107.228 {:progress 0.06470588235294117}} INFO [2022-03-13 17:08:40,323] jepsen worker nemesis - couchbase.util Rebalance check count: 33 INFO [2022-03-13 17:08:40,323] jepsen worker nemesis - couchbase.util Rebalance stuck count: 20 INFO [2022-03-13 17:08:40,326] jepsen worker 35 - jepsen.util 665 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,327] jepsen worker 35 - jepsen.util 665 :invoke :write [11 23] INFO [2022-03-13 17:08:40,336] jepsen worker 1 - jepsen.util 721 :info :write [0 42] :RequestTimeoutException INFO [2022-03-13 17:08:40,337] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:40,400] jepsen worker 60 - jepsen.util 600 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,401] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:40,433] jepsen worker 23 - jepsen.util 473 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,434] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:40,438] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:40,439] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:40,440] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:40,441] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:40,442] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:40,442] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:40,443] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:40,444] jepsen worker 23 - jepsen.util 473 :invoke :write [7 28] INFO [2022-03-13 17:08:40,454] jepsen worker 8 - jepsen.util 638 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,458] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:40,638] jepsen worker 16 - jepsen.util 646 :info :write [5 15] :SyncWriteAmbiguous INFO [2022-03-13 17:08:40,639] jepsen worker 16 - jepsen.util 736 :invoke :read [5 nil] INFO [2022-03-13 17:08:40,730] jepsen worker 77 - jepsen.util 527 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,731] jepsen worker 77 - jepsen.util 527 :invoke :write [25 33] INFO [2022-03-13 17:08:40,737] jepsen worker 23 - jepsen.util 473 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,737] jepsen worker 23 - jepsen.util 473 :invoke :write [7 32] INFO [2022-03-13 17:08:40,814] jepsen worker 24 - jepsen.util 474 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,814] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:40,815] jepsen worker 25 - jepsen.util 385 :invoke :write [8 10] INFO [2022-03-13 17:08:40,815] jepsen worker 24 - jepsen.util 474 :invoke :write [8 45] INFO [2022-03-13 17:08:40,842] jepsen worker 23 - jepsen.util 473 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,843] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:40,848] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:40,849] jepsen worker 23 - jepsen.util 473 :invoke :write [7 0] INFO [2022-03-13 17:08:40,851] jepsen worker 44 - jepsen.util 494 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,852] jepsen worker 44 - jepsen.util 494 :invoke :write [14 1] INFO [2022-03-13 17:08:40,904] jepsen worker 79 - jepsen.util 709 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,904] jepsen worker 62 - jepsen.util 422 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,905] jepsen worker 79 - jepsen.util 709 :invoke :write [26 30] INFO [2022-03-13 17:08:40,907] jepsen worker 62 - jepsen.util 422 :invoke :write [20 41] INFO [2022-03-13 17:08:40,918] jepsen worker 3 - jepsen.util 543 :ok :read [1 16] INFO [2022-03-13 17:08:40,919] jepsen worker 3 - jepsen.util 543 :invoke :write [1 12] INFO [2022-03-13 17:08:40,968] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,968] jepsen worker 52 - jepsen.util 502 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:40,969] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:08:40,970] jepsen worker 8 - jepsen.util 638 :invoke :write [2 10] INFO [2022-03-13 17:08:40,989] jepsen worker 58 - jepsen.util 598 :ok :read [19 35] INFO [2022-03-13 17:08:40,991] jepsen worker 58 - jepsen.util 598 :invoke :write [19 32] INFO [2022-03-13 17:08:41,019] jepsen worker 34 - jepsen.util 574 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,020] jepsen worker 34 - jepsen.util 574 :invoke :write [11 26] INFO [2022-03-13 17:08:41,027] jepsen worker 23 - jepsen.util 473 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,028] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:41,029] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:41,030] jepsen worker 23 - jepsen.util 473 :invoke :write [7 31] INFO [2022-03-13 17:08:41,077] jepsen worker 42 - jepsen.util 402 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,077] jepsen worker 33 - jepsen.util 573 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,077] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:41,077] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,090] jepsen worker 67 - jepsen.util 697 :info :write [22 47] :RequestTimeoutException INFO [2022-03-13 17:08:41,108] jepsen worker 33 - jepsen.util 573 :invoke :write [11 16] INFO [2022-03-13 17:08:41,108] jepsen worker 7 - jepsen.util 367 :invoke :write [2 22] INFO [2022-03-13 17:08:41,108] jepsen worker 67 - jepsen.util 787 :invoke :read [22 nil] INFO [2022-03-13 17:08:41,113] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:41,114] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:41,135] jepsen worker 23 - jepsen.util 473 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,136] jepsen worker 23 - jepsen.util 473 :invoke :write [7 43] INFO [2022-03-13 17:08:41,146] jepsen worker 78 - jepsen.util 438 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,146] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:41,184] jepsen worker 22 - jepsen.util 472 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,184] jepsen worker 22 - jepsen.util 472 :invoke :write [7 49] INFO [2022-03-13 17:08:41,217] jepsen worker 23 - jepsen.util 473 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,218] jepsen worker 23 - jepsen.util 473 :invoke :write [7 48] INFO [2022-03-13 17:08:41,265] jepsen worker 43 - jepsen.util 403 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,265] jepsen worker 26 - jepsen.util 476 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,266] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:41,273] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:41,315] jepsen worker 24 - jepsen.util 474 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,316] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:41,328] jepsen worker 23 - jepsen.util 473 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,329] jepsen worker 23 - jepsen.util 473 :invoke :write [7 19] INFO [2022-03-13 17:08:41,330] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.06744868035190621}, :ns_1@172.23.105.187 {:progress 0.08771929824561409}, :ns_1@172.23.107.227 {:progress 0.06744868035190621}, :ns_1@172.23.107.228 {:progress 0.06470588235294117}} INFO [2022-03-13 17:08:41,330] jepsen worker nemesis - couchbase.util Rebalance check count: 34 INFO [2022-03-13 17:08:41,330] jepsen worker nemesis - couchbase.util Rebalance stuck count: 20 INFO [2022-03-13 17:08:41,376] jepsen worker 6 - jepsen.util 546 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,377] jepsen worker 6 - jepsen.util 546 :invoke :write [2 34] INFO [2022-03-13 17:08:41,421] jepsen worker 77 - jepsen.util 527 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,422] jepsen worker 77 - jepsen.util 527 :invoke :write [25 22] INFO [2022-03-13 17:08:41,422] jepsen worker 35 - jepsen.util 665 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,423] jepsen worker 35 - jepsen.util 665 :invoke :write [11 21] INFO [2022-03-13 17:08:41,434] jepsen worker 23 - jepsen.util 473 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,435] jepsen worker 23 - jepsen.util 473 :invoke :write [7 38] INFO [2022-03-13 17:08:41,532] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,532] jepsen worker 25 - jepsen.util 385 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,533] jepsen worker 52 - jepsen.util 502 :invoke :write [17 7] INFO [2022-03-13 17:08:41,533] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:41,544] jepsen worker 23 - jepsen.util 473 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,544] jepsen worker 23 - jepsen.util 473 :invoke :write [7 31] INFO [2022-03-13 17:08:41,551] jepsen worker 53 - jepsen.util 503 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,551] jepsen worker 79 - jepsen.util 709 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,552] jepsen worker 53 - jepsen.util 503 :invoke :write [17 46] INFO [2022-03-13 17:08:41,552] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:41,590] jepsen worker 41 - jepsen.util 761 :ok :read [13 39] INFO [2022-03-13 17:08:41,590] jepsen worker 41 - jepsen.util 761 :invoke :read [13 nil] INFO [2022-03-13 17:08:41,605] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:41,605] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:41,635] jepsen worker 71 - jepsen.util 431 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,635] jepsen worker 71 - jepsen.util 431 :invoke :write [23 47] INFO [2022-03-13 17:08:41,686] jepsen worker 23 - jepsen.util 473 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,687] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:41,692] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:41,693] jepsen worker 23 - jepsen.util 473 :invoke :write [7 4] INFO [2022-03-13 17:08:41,732] jepsen worker 8 - jepsen.util 638 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,732] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,733] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:41,734] jepsen worker 78 - jepsen.util 438 :invoke :write [26 48] INFO [2022-03-13 17:08:41,748] jepsen worker 7 - jepsen.util 367 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,748] jepsen worker 14 - jepsen.util 194 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,748] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:41,749] jepsen worker 14 - jepsen.util 194 :invoke :write [4 28] INFO [2022-03-13 17:08:41,759] jepsen worker 44 - jepsen.util 494 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,760] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:41,813] jepsen worker 23 - jepsen.util 473 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,813] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:41,818] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:41,819] jepsen worker 23 - jepsen.util 473 :invoke :write [7 14] INFO [2022-03-13 17:08:41,821] jepsen worker 62 - jepsen.util 422 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,821] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:41,822] jepsen worker 62 - jepsen.util 422 :invoke :write [20 40] INFO [2022-03-13 17:08:41,822] jepsen worker 42 - jepsen.util 402 :invoke :write [14 42] INFO [2022-03-13 17:08:41,838] jepsen worker 50 - jepsen.util 680 :info :write [16 37] :RequestTimeoutException INFO [2022-03-13 17:08:41,839] jepsen worker 50 - jepsen.util 770 :invoke :write [16 3] INFO [2022-03-13 17:08:41,854] jepsen worker 29 - jepsen.util 749 :info :write [9 32] :SyncWriteAmbiguous INFO [2022-03-13 17:08:41,855] jepsen worker 29 - jepsen.util 839 :invoke :write [9 23] INFO [2022-03-13 17:08:41,866] jepsen worker 33 - jepsen.util 573 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:41,867] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:41,867] jepsen worker 33 - jepsen.util 573 :invoke :write [11 13] INFO [2022-03-13 17:08:41,868] jepsen worker 2 - jepsen.util 722 :info :write [0 2] :RequestTimeoutException INFO [2022-03-13 17:08:41,868] jepsen worker 2 - jepsen.util 812 :invoke :write [0 26] INFO [2022-03-13 17:08:41,874] jepsen worker 43 - jepsen.util 403 :invoke :write [14 6] INFO [2022-03-13 17:08:42,097] jepsen worker 30 - jepsen.util 570 :ok :read [10 :nil] INFO [2022-03-13 17:08:42,097] jepsen worker 30 - jepsen.util 570 :invoke :read [10 nil] INFO [2022-03-13 17:08:42,126] jepsen worker 30 - jepsen.util 570 :ok :read [10 :nil] INFO [2022-03-13 17:08:42,126] jepsen worker 30 - jepsen.util 570 :invoke :read [10 nil] INFO [2022-03-13 17:08:42,153] jepsen worker 30 - jepsen.util 570 :ok :read [10 :nil] INFO [2022-03-13 17:08:42,154] jepsen worker 30 - jepsen.util 570 :invoke :read [10 nil] INFO [2022-03-13 17:08:42,155] jepsen worker 30 - jepsen.util 570 :ok :read [10 :nil] INFO [2022-03-13 17:08:42,155] jepsen worker 30 - jepsen.util 570 :invoke :read [10 nil] INFO [2022-03-13 17:08:42,157] jepsen worker 30 - jepsen.util 570 :ok :read [10 :nil] INFO [2022-03-13 17:08:42,157] jepsen worker 30 - jepsen.util 570 :invoke :write [10 46] INFO [2022-03-13 17:08:42,196] jepsen worker 47 - jepsen.util 767 :info :write [15 10] :RequestTimeoutException INFO [2022-03-13 17:08:42,197] jepsen worker 47 - jepsen.util 857 :invoke :write [15 7] INFO [2022-03-13 17:08:42,324] jepsen worker 45 - jepsen.util 855 :info :write [15 23] :RequestTimeoutException INFO [2022-03-13 17:08:42,324] jepsen worker 45 - jepsen.util 945 :invoke :write [15 4] INFO [2022-03-13 17:08:42,336] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.06744868035190621}, :ns_1@172.23.105.187 {:progress 0.08771929824561409}, :ns_1@172.23.107.227 {:progress 0.06744868035190621}, :ns_1@172.23.107.228 {:progress 0.06470588235294117}} INFO [2022-03-13 17:08:42,336] jepsen worker nemesis - couchbase.util Rebalance check count: 35 INFO [2022-03-13 17:08:42,336] jepsen worker nemesis - couchbase.util Rebalance stuck count: 21 INFO [2022-03-13 17:08:42,676] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:42,676] jepsen worker 80 - jepsen.util 620 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,676] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:42,677] jepsen worker 80 - jepsen.util 620 :invoke :write [26 35] INFO [2022-03-13 17:08:42,677] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:42,678] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:42,680] jepsen worker 23 - jepsen.util 473 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,681] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:42,683] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:42,684] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:42,685] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:42,686] jepsen worker 23 - jepsen.util 473 :invoke :write [7 24] INFO [2022-03-13 17:08:42,745] jepsen worker 34 - jepsen.util 574 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,746] jepsen worker 34 - jepsen.util 574 :invoke :write [11 9] INFO [2022-03-13 17:08:42,760] jepsen worker 76 - jepsen.util 526 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,760] jepsen worker 77 - jepsen.util 527 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,761] jepsen worker 76 - jepsen.util 526 :invoke :write [25 8] INFO [2022-03-13 17:08:42,762] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,762] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:42,763] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:42,763] jepsen worker 79 - jepsen.util 709 :invoke :write [26 38] INFO [2022-03-13 17:08:42,764] jepsen worker 26 - jepsen.util 476 :invoke :write [8 16] INFO [2022-03-13 17:08:42,792] jepsen worker 23 - jepsen.util 473 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,792] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:42,794] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:42,795] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:42,797] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,797] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:42,797] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:42,798] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:42,799] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:42,799] jepsen worker 23 - jepsen.util 473 :invoke :write [7 32] INFO [2022-03-13 17:08:42,816] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:42,817] jepsen worker 60 - jepsen.util 600 :invoke :write [20 29] INFO [2022-03-13 17:08:42,818] jepsen worker 54 - jepsen.util 684 :info :write [18 16] :RequestTimeoutException INFO [2022-03-13 17:08:42,818] jepsen worker 54 - jepsen.util 774 :invoke :read [18 nil] INFO [2022-03-13 17:08:42,831] jepsen worker 52 - jepsen.util 502 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,831] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,831] jepsen worker 52 - jepsen.util 502 :invoke :write [17 30] INFO [2022-03-13 17:08:42,832] jepsen worker 8 - jepsen.util 638 :invoke :write [2 11] INFO [2022-03-13 17:08:42,890] jepsen worker 22 - jepsen.util 472 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,891] jepsen worker 22 - jepsen.util 472 :invoke :write [7 42] INFO [2022-03-13 17:08:42,905] jepsen worker 23 - jepsen.util 473 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,906] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:42,907] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:42,908] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:42,909] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:42,909] jepsen worker 23 - jepsen.util 473 :invoke :write [7 2] INFO [2022-03-13 17:08:42,910] jepsen worker 21 - jepsen.util 561 :info :write [7 18] :RequestTimeoutException INFO [2022-03-13 17:08:42,911] jepsen worker 12 - jepsen.util 552 :info :write [4 36] :RequestTimeoutException INFO [2022-03-13 17:08:42,911] jepsen worker 69 - jepsen.util 519 :info :write [23 24] :RequestTimeoutException INFO [2022-03-13 17:08:42,911] jepsen worker 70 - jepsen.util 610 :info :write [23 32] :RequestTimeoutException INFO [2022-03-13 17:08:42,912] jepsen worker 35 - jepsen.util 665 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,912] jepsen worker 21 - jepsen.util 651 :invoke :write [7 1] INFO [2022-03-13 17:08:42,912] jepsen worker 12 - jepsen.util 642 :invoke :write [4 18] INFO [2022-03-13 17:08:42,913] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:42,913] jepsen worker 35 - jepsen.util 665 :invoke :write [11 38] INFO [2022-03-13 17:08:42,914] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:42,915] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,915] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:08:42,921] jepsen worker 51 - jepsen.util 321 :info :write [17 49] :RequestTimeoutException INFO [2022-03-13 17:08:42,922] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:42,938] jepsen worker 46 - jepsen.util 586 :info :write [15 17] :RequestTimeoutException INFO [2022-03-13 17:08:42,938] jepsen worker 46 - jepsen.util 676 :invoke :write [15 41] INFO [2022-03-13 17:08:42,968] jepsen worker 43 - jepsen.util 403 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:42,968] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:42,994] jepsen worker 87 - jepsen.util 897 :info :write [29 37] :RequestTimeoutException INFO [2022-03-13 17:08:42,995] jepsen worker 87 - jepsen.util 987 :invoke :write [29 16] INFO [2022-03-13 17:08:43,057] jepsen worker 11 - jepsen.util 641 :info :write [3 6] :RequestTimeoutException INFO [2022-03-13 17:08:43,058] jepsen worker 11 - jepsen.util 731 :invoke :write [3 31] INFO [2022-03-13 17:08:43,088] jepsen worker 23 - jepsen.util 473 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,089] jepsen worker 23 - jepsen.util 473 :invoke :write [7 40] INFO [2022-03-13 17:08:43,122] jepsen worker 53 - jepsen.util 503 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,123] jepsen worker 53 - jepsen.util 503 :invoke :write [17 18] INFO [2022-03-13 17:08:43,150] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,151] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:08:43,161] jepsen worker 6 - jepsen.util 546 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,162] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:43,162] jepsen worker 78 - jepsen.util 438 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,162] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:43,163] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:43,163] jepsen worker 78 - jepsen.util 438 :invoke :write [26 22] INFO [2022-03-13 17:08:43,163] jepsen worker 25 - jepsen.util 385 :invoke :write [8 9] INFO [2022-03-13 17:08:43,170] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:43,178] jepsen worker 33 - jepsen.util 573 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,178] jepsen worker 33 - jepsen.util 573 :invoke :write [11 5] INFO [2022-03-13 17:08:43,183] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,184] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:43,221] jepsen worker 23 - jepsen.util 473 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,222] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:43,309] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,309] jepsen worker 7 - jepsen.util 367 :invoke :write [2 2] INFO [2022-03-13 17:08:43,320] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:43,325] jepsen worker 23 - jepsen.util 473 :invoke :write [7 0] INFO [2022-03-13 17:08:43,342] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07331378299120239}, :ns_1@172.23.105.187 {:progress 0.08771929824561409}, :ns_1@172.23.107.227 {:progress 0.07331378299120239}, :ns_1@172.23.107.228 {:progress 0.07058823529411762}} INFO [2022-03-13 17:08:43,342] jepsen worker nemesis - couchbase.util Rebalance check count: 36 INFO [2022-03-13 17:08:43,342] jepsen worker nemesis - couchbase.util Rebalance stuck count: 21 INFO [2022-03-13 17:08:43,568] jepsen worker 38 - jepsen.util 758 :info :write [12 23] :RequestTimeoutException INFO [2022-03-13 17:08:43,568] jepsen worker 38 - jepsen.util 848 :invoke :read [12 nil] INFO [2022-03-13 17:08:43,600] jepsen worker 38 - jepsen.util 848 :ok :read [12 46] INFO [2022-03-13 17:08:43,600] jepsen worker 38 - jepsen.util 848 :invoke :write [12 37] INFO [2022-03-13 17:08:43,620] jepsen worker 65 - jepsen.util 695 :info :write [21 4] :RequestTimeoutException INFO [2022-03-13 17:08:43,621] jepsen worker 65 - jepsen.util 785 :invoke :write [21 22] INFO [2022-03-13 17:08:43,640] jepsen worker 82 - jepsen.util 712 :info :write [27 16] :RequestTimeoutException INFO [2022-03-13 17:08:43,641] jepsen worker 82 - jepsen.util 802 :invoke :write [27 11] INFO [2022-03-13 17:08:43,728] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,728] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:43,729] jepsen worker 43 - jepsen.util 403 :invoke :write [14 37] INFO [2022-03-13 17:08:43,729] jepsen worker 77 - jepsen.util 527 :invoke :write [25 1] INFO [2022-03-13 17:08:43,797] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:43,797] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,797] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:43,798] jepsen worker 13 - jepsen.util 463 :invoke :write [4 40] INFO [2022-03-13 17:08:43,801] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:43,802] jepsen worker 51 - jepsen.util 411 :invoke :write [17 26] INFO [2022-03-13 17:08:43,926] jepsen worker 89 - jepsen.util 719 :info :write [29 6] :RequestTimeoutException INFO [2022-03-13 17:08:43,927] jepsen worker 89 - jepsen.util 809 :invoke :write [29 37] INFO [2022-03-13 17:08:44,348] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07331378299120239}, :ns_1@172.23.105.187 {:progress 0.08771929824561409}, :ns_1@172.23.107.227 {:progress 0.07331378299120239}, :ns_1@172.23.107.228 {:progress 0.07058823529411762}} INFO [2022-03-13 17:08:44,348] jepsen worker nemesis - couchbase.util Rebalance check count: 37 INFO [2022-03-13 17:08:44,348] jepsen worker nemesis - couchbase.util Rebalance stuck count: 22 INFO [2022-03-13 17:08:44,539] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:44,539] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:44,560] jepsen worker 10 - jepsen.util 550 :info :write [3 28] :RequestTimeoutException INFO [2022-03-13 17:08:44,561] jepsen worker 10 - jepsen.util 640 :invoke :write [3 13] INFO [2022-03-13 17:08:45,051] jepsen worker 64 - jepsen.util 514 :ok :read [21 :nil] INFO [2022-03-13 17:08:45,052] jepsen worker 64 - jepsen.util 514 :invoke :write [21 10] INFO [2022-03-13 17:08:45,071] jepsen worker 73 - jepsen.util 793 :ok :read [24 24] INFO [2022-03-13 17:08:45,072] jepsen worker 73 - jepsen.util 793 :invoke :write [24 11] INFO [2022-03-13 17:08:45,194] jepsen worker 0 - jepsen.util 630 :info :write [0 40] :RequestTimeoutException INFO [2022-03-13 17:08:45,195] jepsen worker 0 - jepsen.util 720 :invoke :write [0 35] INFO [2022-03-13 17:08:45,196] jepsen worker 17 - jepsen.util 737 :info :write [5 33] :RequestTimeoutException INFO [2022-03-13 17:08:45,197] jepsen worker 17 - jepsen.util 827 :invoke :read [5 nil] INFO [2022-03-13 17:08:45,296] jepsen worker 51 - jepsen.util 411 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,297] jepsen worker 51 - jepsen.util 411 :invoke :write [17 37] INFO [2022-03-13 17:08:45,311] jepsen worker 26 - jepsen.util 476 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,312] jepsen worker 26 - jepsen.util 476 :invoke :write [8 47] INFO [2022-03-13 17:08:45,355] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07331378299120239}, :ns_1@172.23.105.187 {:progress 0.08771929824561409}, :ns_1@172.23.107.227 {:progress 0.07331378299120239}, :ns_1@172.23.107.228 {:progress 0.07058823529411762}} INFO [2022-03-13 17:08:45,355] jepsen worker nemesis - couchbase.util Rebalance check count: 38 INFO [2022-03-13 17:08:45,356] jepsen worker nemesis - couchbase.util Rebalance stuck count: 23 WARN [2022-03-13 17:08:45,365] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8588313,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10405786,"total_server_duration_us":368083,"encode_duration_us":17597,"operation_id":"0x1a28","timeout_ms":10000,"last_server_duration_us":60885,"total_duration_us":10010305},{"operation_name":"upsert","last_dispatch_duration_us":17996112,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":18138351,"total_server_duration_us":9061308,"encode_duration_us":18380,"operation_id":"0x18bb","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10010240},{"operation_name":"upsert","last_dispatch_duration_us":83860,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":123466,"total_server_duration_us":51940,"encode_duration_us":30444,"operation_id":"0x18cc","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10008043},{"operation_name":"upsert","last_dispatch_duration_us":8503170,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10168231,"total_server_duration_us":241380,"encode_duration_us":19894,"operation_id":"0x1a2c","timeout_ms":10000,"last_server_duration_us":37138,"total_duration_us":10007885},{"operation_name":"upsert","last_dispatch_duration_us":9373271,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10296455,"total_server_duration_us":9198335,"encode_duration_us":17921,"operation_id":"0x1a29","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10007516},{"operation_name":"upsert","last_dispatch_duration_us":9034995,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":18159288,"total_server_duration_us":62296,"encode_duration_us":31518,"operation_id":"0x18bc","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10007297},{"operation_name":"upsert","last_dispatch_duration_us":16916243,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16916243,"total_server_duration_us":19652,"encode_duration_us":19715,"operation_id":"0x1917","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10007003},{"operation_name":"upsert","last_dispatch_duration_us":9046570,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":18232904,"total_server_duration_us":105515,"encode_duration_us":17356,"operation_id":"0x18b8","timeout_ms":10000,"last_server_duration_us":29881,"total_duration_us":10006835},{"operation_name":"upsert","last_dispatch_duration_us":8537580,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10336661,"total_server_duration_us":268423,"encode_duration_us":18791,"operation_id":"0x1a2a","timeout_ms":10000,"last_server_duration_us":37653,"total_duration_us":10006152},{"operation_name":"upsert","last_dispatch_duration_us":8631992,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10297981,"total_server_duration_us":721623,"encode_duration_us":22989,"operation_id":"0x1a2b","timeout_ms":10000,"last_server_duration_us":337599,"total_duration_us":10004256}],"total_count":21}} INFO [2022-03-13 17:08:45,384] jepsen worker 51 - jepsen.util 411 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,384] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:45,389] jepsen worker 7 - jepsen.util 367 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,389] jepsen worker 7 - jepsen.util 367 :invoke :write [2 8] INFO [2022-03-13 17:08:45,433] jepsen worker 23 - jepsen.util 473 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,433] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:45,466] jepsen worker 13 - jepsen.util 463 :fail :write [4 40] :ServerOutOfMemoryException WARN [2022-03-13 17:08:45,466] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9063744,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9063744,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9047983,"encode_duration_us":17669,"operation_id":"0x17b0","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10001505},{"operation_name":"upsert","last_dispatch_duration_us":18004054,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18108132,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9063689,"encode_duration_us":17560,"operation_id":"0x18ba","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10002574},{"operation_name":"upsert","last_dispatch_duration_us":9373271,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10296455,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":9198335,"encode_duration_us":17921,"operation_id":"0x1a29","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10007516},{"operation_name":"upsert","last_dispatch_duration_us":17937230,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17937230,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9034183,"encode_duration_us":16792,"operation_id":"0x1658","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10005778},{"operation_name":"upsert","last_dispatch_duration_us":17996112,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18138351,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9061308,"encode_duration_us":18380,"operation_id":"0x18bb","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10010240}],"total_count":5}} INFO [2022-03-13 17:08:45,466] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,467] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:45,467] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:45,471] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,471] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:45,472] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:45,473] jepsen worker 13 - jepsen.util 463 :invoke :write [4 6] INFO [2022-03-13 17:08:45,473] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,474] jepsen worker 51 - jepsen.util 411 :invoke :write [17 32] INFO [2022-03-13 17:08:45,513] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:45,513] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,514] jepsen worker 23 - jepsen.util 473 :invoke :write [7 49] INFO [2022-03-13 17:08:45,515] jepsen worker 69 - jepsen.util 609 :invoke :write [23 30] INFO [2022-03-13 17:08:45,519] jepsen worker 53 - jepsen.util 503 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,520] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:45,533] jepsen worker 8 - jepsen.util 638 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,534] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:45,550] jepsen worker 85 - jepsen.util 805 :info :write [28 7] :RequestTimeoutException INFO [2022-03-13 17:08:45,550] jepsen worker 85 - jepsen.util 895 :invoke :write [28 13] INFO [2022-03-13 17:08:45,573] jepsen worker 72 - jepsen.util 702 :info :write [24 0] :RequestTimeoutException INFO [2022-03-13 17:08:45,574] jepsen worker 72 - jepsen.util 792 :invoke :write [24 44] INFO [2022-03-13 17:08:45,593] jepsen worker 13 - jepsen.util 463 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,593] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:45,616] jepsen worker 18 - jepsen.util 558 :ok :read [6 16] INFO [2022-03-13 17:08:45,617] jepsen worker 18 - jepsen.util 558 :invoke :write [6 13] INFO [2022-03-13 17:08:45,620] jepsen worker 60 - jepsen.util 600 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,620] jepsen worker 60 - jepsen.util 600 :invoke :write [20 35] INFO [2022-03-13 17:08:45,688] jepsen worker 23 - jepsen.util 473 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,688] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:45,710] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:45,710] jepsen worker 51 - jepsen.util 411 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,711] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:45,711] jepsen worker 13 - jepsen.util 463 :invoke :write [4 12] INFO [2022-03-13 17:08:45,712] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,713] jepsen worker 51 - jepsen.util 411 :invoke :write [17 7] INFO [2022-03-13 17:08:45,751] jepsen worker 61 - jepsen.util 421 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,751] jepsen worker 61 - jepsen.util 421 :invoke :write [20 45] INFO [2022-03-13 17:08:45,779] jepsen worker 69 - jepsen.util 609 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,780] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:08:45,785] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:45,786] jepsen worker 23 - jepsen.util 473 :invoke :write [7 9] INFO [2022-03-13 17:08:45,799] jepsen worker 77 - jepsen.util 527 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,800] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:45,850] jepsen worker 13 - jepsen.util 463 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,851] jepsen worker 13 - jepsen.util 463 :invoke :write [4 37] INFO [2022-03-13 17:08:45,922] jepsen worker 51 - jepsen.util 411 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,923] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:45,936] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,937] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:45,959] jepsen worker 13 - jepsen.util 463 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,959] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,960] jepsen worker 51 - jepsen.util 411 :invoke :write [17 10] INFO [2022-03-13 17:08:45,960] jepsen worker 13 - jepsen.util 463 :invoke :write [4 14] INFO [2022-03-13 17:08:45,968] jepsen worker 14 - jepsen.util 194 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:45,969] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:46,015] jepsen worker 34 - jepsen.util 574 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,015] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:08:46,045] jepsen worker 23 - jepsen.util 473 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,046] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,046] jepsen worker 23 - jepsen.util 473 :invoke :write [7 36] INFO [2022-03-13 17:08:46,047] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:46,047] jepsen worker 43 - jepsen.util 403 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,048] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,048] jepsen worker 43 - jepsen.util 403 :invoke :write [14 36] INFO [2022-03-13 17:08:46,048] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:46,050] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,050] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:46,052] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,052] jepsen worker 69 - jepsen.util 609 :invoke :write [23 3] INFO [2022-03-13 17:08:46,075] jepsen worker 51 - jepsen.util 411 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,076] jepsen worker 51 - jepsen.util 411 :invoke :write [17 35] INFO [2022-03-13 17:08:46,141] jepsen worker 13 - jepsen.util 463 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,158] jepsen worker 23 - jepsen.util 473 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,164] jepsen worker 13 - jepsen.util 463 :invoke :write [4 0] INFO [2022-03-13 17:08:46,164] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:46,215] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:46,215] jepsen worker 69 - jepsen.util 609 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,216] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:46,217] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:08:46,236] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:46,239] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:46,244] jepsen worker 33 - jepsen.util 573 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,245] jepsen worker 42 - jepsen.util 402 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,245] jepsen worker 33 - jepsen.util 573 :invoke :write [11 46] INFO [2022-03-13 17:08:46,250] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] INFO [2022-03-13 17:08:46,262] jepsen worker 13 - jepsen.util 463 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,262] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:46,276] jepsen worker 19 - jepsen.util 469 :info :write [6 28] :RequestTimeoutException INFO [2022-03-13 17:08:46,276] jepsen worker 19 - jepsen.util 559 :invoke :read [6 nil] INFO [2022-03-13 17:08:46,339] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:46,339] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,340] jepsen worker 23 - jepsen.util 473 :invoke :write [7 10] INFO [2022-03-13 17:08:46,340] jepsen worker 69 - jepsen.util 609 :invoke :write [23 2] INFO [2022-03-13 17:08:46,363] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07331378299120239}, :ns_1@172.23.105.187 {:progress 0.08771929824561409}, :ns_1@172.23.107.227 {:progress 0.07331378299120239}, :ns_1@172.23.107.228 {:progress 0.07058823529411762}} INFO [2022-03-13 17:08:46,363] jepsen worker nemesis - couchbase.util Rebalance check count: 39 INFO [2022-03-13 17:08:46,363] jepsen worker nemesis - couchbase.util Rebalance stuck count: 24 INFO [2022-03-13 17:08:46,375] jepsen worker 76 - jepsen.util 526 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,375] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:46,388] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:46,388] jepsen worker 51 - jepsen.util 411 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,389] jepsen worker 51 - jepsen.util 411 :invoke :write [17 4] INFO [2022-03-13 17:08:46,389] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:46,390] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:46,391] jepsen worker 13 - jepsen.util 463 :invoke :write [4 35] INFO [2022-03-13 17:08:46,410] jepsen worker 23 - jepsen.util 473 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,410] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:46,433] jepsen worker 79 - jepsen.util 709 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,434] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:46,509] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,509] jepsen worker 26 - jepsen.util 476 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,510] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:46,511] jepsen worker 26 - jepsen.util 476 :invoke :write [8 31] INFO [2022-03-13 17:08:46,523] jepsen worker 51 - jepsen.util 411 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,524] jepsen worker 51 - jepsen.util 411 :invoke :write [17 2] INFO [2022-03-13 17:08:46,536] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:46,536] jepsen worker 69 - jepsen.util 609 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,537] jepsen worker 23 - jepsen.util 473 :invoke :write [7 9] INFO [2022-03-13 17:08:46,537] jepsen worker 69 - jepsen.util 609 :invoke :write [23 21] INFO [2022-03-13 17:08:46,558] jepsen worker 13 - jepsen.util 463 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,559] jepsen worker 13 - jepsen.util 463 :invoke :write [4 7] INFO [2022-03-13 17:08:46,596] jepsen worker 7 - jepsen.util 367 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,597] jepsen worker 7 - jepsen.util 367 :invoke :write [2 28] INFO [2022-03-13 17:08:46,636] jepsen worker 51 - jepsen.util 411 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,637] jepsen worker 51 - jepsen.util 411 :invoke :write [17 44] INFO [2022-03-13 17:08:46,644] jepsen worker 23 - jepsen.util 473 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,645] jepsen worker 23 - jepsen.util 473 :invoke :write [7 25] INFO [2022-03-13 17:08:46,697] jepsen worker 61 - jepsen.util 421 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,697] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:46,697] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:46,698] jepsen worker 14 - jepsen.util 194 :invoke :write [4 13] INFO [2022-03-13 17:08:46,715] jepsen worker 69 - jepsen.util 609 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,716] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:08:46,737] jepsen worker 13 - jepsen.util 463 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,737] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:46,768] jepsen worker 20 - jepsen.util 830 :info :write [6 29] :RequestTimeoutException INFO [2022-03-13 17:08:46,769] jepsen worker 20 - jepsen.util 920 :invoke :write [6 12] INFO [2022-03-13 17:08:46,780] jepsen worker 28 - jepsen.util 748 :info :write [9 5] :RequestTimeoutException INFO [2022-03-13 17:08:46,781] jepsen worker 28 - jepsen.util 838 :invoke :write [9 7] INFO [2022-03-13 17:08:46,787] jepsen worker 23 - jepsen.util 473 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,787] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:46,801] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,801] jepsen worker 60 - jepsen.util 600 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,801] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:46,802] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:46,836] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:46,836] jepsen worker 51 - jepsen.util 411 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,836] jepsen worker 13 - jepsen.util 463 :invoke :write [4 4] INFO [2022-03-13 17:08:46,837] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:46,838] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,839] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:46,840] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,841] jepsen worker 51 - jepsen.util 411 :invoke :write [17 34] INFO [2022-03-13 17:08:46,871] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:46,872] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,874] jepsen worker 23 - jepsen.util 473 :invoke :write [7 4] WARN [2022-03-13 17:08:46,878] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15700759,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15700759,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9026756,"encode_duration_us":19574,"operation_id":"0x185b","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10001570}],"total_count":1}} INFO [2022-03-13 17:08:46,881] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:08:46,881] jepsen worker 88 - jepsen.util 898 :ok :read [29 33] INFO [2022-03-13 17:08:46,882] jepsen worker 88 - jepsen.util 898 :invoke :write [29 48] INFO [2022-03-13 17:08:46,918] jepsen worker 43 - jepsen.util 403 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,918] jepsen worker 12 - jepsen.util 642 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,919] jepsen worker 12 - jepsen.util 642 :invoke :write [4 0] INFO [2022-03-13 17:08:46,920] jepsen worker 43 - jepsen.util 403 :invoke :write [14 3] INFO [2022-03-13 17:08:46,953] jepsen worker 51 - jepsen.util 411 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:46,954] jepsen worker 51 - jepsen.util 411 :invoke :write [17 26] INFO [2022-03-13 17:08:46,960] jepsen worker 4 - jepsen.util 724 :ok :read [1 16] INFO [2022-03-13 17:08:46,961] jepsen worker 4 - jepsen.util 724 :invoke :read [1 nil] WARN [2022-03-13 17:08:46,966] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8378410,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":11890544,"total_server_duration_us":566334,"encode_duration_us":18206,"operation_id":"0x191b","timeout_ms":10000,"last_server_duration_us":81254,"total_duration_us":10038594},{"operation_name":"upsert","last_dispatch_duration_us":8399068,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":11630432,"total_server_duration_us":919002,"encode_duration_us":35189,"operation_id":"0x191c","timeout_ms":10000,"last_server_duration_us":43948,"total_duration_us":10023106},{"operation_name":"upsert","last_dispatch_duration_us":9827254,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":18219701,"total_server_duration_us":47193,"encode_duration_us":18969,"operation_id":"0x18e8","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10009542},{"operation_name":"upsert","last_dispatch_duration_us":9082753,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":11893390,"total_server_duration_us":9111201,"encode_duration_us":21356,"operation_id":"0x1918","timeout_ms":10000,"last_server_duration_us":8742269,"total_duration_us":10008791},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17992,"operation_id":"0x1a92","timeout_ms":10000,"total_duration_us":10006525},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18363,"operation_id":"0x1ac8","timeout_ms":10000,"total_duration_us":10002430},{"operation_name":"upsert","last_dispatch_duration_us":9796863,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11163052,"total_server_duration_us":71897,"encode_duration_us":21154,"operation_id":"0x1972","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10002364},{"operation_name":"upsert","last_dispatch_duration_us":8348888,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":11487598,"total_server_duration_us":508723,"encode_duration_us":33842,"operation_id":"0x191e","timeout_ms":10000,"last_server_duration_us":44168,"total_duration_us":10002234},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":16712,"operation_id":"0x1a63","timeout_ms":10000,"total_duration_us":10001557},{"operation_name":"get","last_dispatch_duration_us":9789112,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9789112,"total_server_duration_us":32,"operation_id":"0x1abf","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9790611}],"total_count":16}} WARN [2022-03-13 17:08:46,972] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":5178072,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11550789,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":5332766,"encode_duration_us":20026,"operation_id":"0x188e","timeout_ms":10000,"last_server_duration_us":5158164,"total_duration_us":10002262},{"operation_name":"upsert","last_dispatch_duration_us":14500340,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21891969,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":6715579,"encode_duration_us":17981,"operation_id":"0x15c9","timeout_ms":10000,"last_server_duration_us":6331640,"total_duration_us":10003962},{"operation_name":"upsert","last_dispatch_duration_us":11438367,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11438367,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9042674,"encode_duration_us":17536,"operation_id":"0x1a90","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10009330}],"total_count":3}} WARN [2022-03-13 17:08:46,979] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12180201,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12180201,"total_server_duration_us":62,"operation_id":"0x1946","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":12183949},{"operation_name":"get","last_dispatch_duration_us":12160737,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12160737,"total_server_duration_us":62,"operation_id":"0x193e","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":12162877},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":29679,"operation_id":"0x1940","timeout_ms":10000,"total_duration_us":10010878},{"operation_name":"upsert","last_dispatch_duration_us":135408,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":135408,"total_server_duration_us":20124,"encode_duration_us":17670,"operation_id":"0x1b00","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10008335},{"operation_name":"upsert","last_dispatch_duration_us":63559,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9229549,"total_server_duration_us":208438,"encode_duration_us":21675,"operation_id":"0x1a32","timeout_ms":10000,"last_server_duration_us":36423,"total_duration_us":10006444},{"operation_name":"upsert","last_dispatch_duration_us":72795,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9247902,"total_server_duration_us":201997,"encode_duration_us":21820,"operation_id":"0x1a2f","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10006176},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":23684,"operation_id":"0x19c9","timeout_ms":10000,"total_duration_us":10005898},{"operation_name":"upsert","last_dispatch_duration_us":8249429,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12293068,"total_server_duration_us":205090,"encode_duration_us":17560,"operation_id":"0x194b","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10005739},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":29337,"operation_id":"0x1942","timeout_ms":10000,"total_duration_us":10002954},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":25287,"operation_id":"0x19cb","timeout_ms":10000,"total_duration_us":10002606}],"total_count":48}} INFO [2022-03-13 17:08:46,987] jepsen worker 37 - jepsen.util 667 :ok :read [12 46] INFO [2022-03-13 17:08:46,988] jepsen worker 37 - jepsen.util 667 :invoke :write [12 30] INFO [2022-03-13 17:08:47,015] jepsen worker 52 - jepsen.util 502 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,015] jepsen worker 52 - jepsen.util 502 :invoke :write [17 14] INFO [2022-03-13 17:08:47,020] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,021] jepsen worker 42 - jepsen.util 402 :invoke :write [14 44] INFO [2022-03-13 17:08:47,025] jepsen worker 13 - jepsen.util 463 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,025] jepsen worker 13 - jepsen.util 463 :invoke :write [4 3] INFO [2022-03-13 17:08:47,030] jepsen worker 23 - jepsen.util 473 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,030] jepsen worker 23 - jepsen.util 473 :invoke :write [7 7] WARN [2022-03-13 17:08:47,066] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":18733,"operation_id":"0x1a54","timeout_ms":10000,"total_duration_us":10009964},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18278,"operation_id":"0x1aa4","timeout_ms":10000,"total_duration_us":10009690},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":19702,"operation_id":"0x1b01","timeout_ms":10000,"total_duration_us":10007430},{"operation_name":"upsert","last_dispatch_duration_us":995335,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5887116,"total_server_duration_us":394036,"encode_duration_us":19098,"operation_id":"0x18da","timeout_ms":10000,"last_server_duration_us":31776,"total_duration_us":10007236},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":19085,"operation_id":"0x1a5b","timeout_ms":10000,"total_duration_us":10005534},{"operation_name":"upsert","last_dispatch_duration_us":4899715,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":8472533,"total_server_duration_us":359328,"encode_duration_us":19190,"operation_id":"0x1919","timeout_ms":10000,"last_server_duration_us":53614,"total_duration_us":10002579},{"operation_name":"upsert","last_dispatch_duration_us":612297,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1922395,"total_server_duration_us":228865,"encode_duration_us":20920,"operation_id":"0x19e9","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":8621190},{"operation_name":"upsert","last_dispatch_duration_us":768678,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3981765,"total_server_duration_us":181717,"encode_duration_us":18291,"operation_id":"0x1a05","timeout_ms":10000,"last_server_duration_us":44500,"total_duration_us":8151210},{"operation_name":"upsert","last_dispatch_duration_us":664941,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":6835957,"total_server_duration_us":211885,"encode_duration_us":18894,"operation_id":"0x1a1d","timeout_ms":10000,"last_server_duration_us":81541,"total_duration_us":7415583},{"operation_name":"upsert","last_dispatch_duration_us":1448939,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2277244,"total_server_duration_us":118538,"encode_duration_us":19900,"operation_id":"0x1afd","timeout_ms":10000,"last_server_duration_us":82115,"total_duration_us":5924066}],"total_count":25}} WARN [2022-03-13 17:08:47,066] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8348888,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11487598,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":508723,"encode_duration_us":33842,"operation_id":"0x191e","timeout_ms":10000,"last_server_duration_us":44168,"total_duration_us":10002234},{"operation_name":"upsert","last_dispatch_duration_us":20777049,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20777049,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9138434,"encode_duration_us":17207,"operation_id":"0x1836","timeout_ms":10000,"last_server_duration_us":9138434,"total_duration_us":10003955},{"operation_name":"upsert","last_dispatch_duration_us":8378410,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11890544,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":566334,"encode_duration_us":18206,"operation_id":"0x191b","timeout_ms":10000,"last_server_duration_us":81254,"total_duration_us":10038594},{"operation_name":"upsert","last_dispatch_duration_us":8399068,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11630432,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":919002,"encode_duration_us":35189,"operation_id":"0x191c","timeout_ms":10000,"last_server_duration_us":43948,"total_duration_us":10023106},{"operation_name":"upsert","last_dispatch_duration_us":9082753,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11893390,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9111201,"encode_duration_us":21356,"operation_id":"0x1918","timeout_ms":10000,"last_server_duration_us":8742269,"total_duration_us":10008791}],"total_count":5}} WARN [2022-03-13 17:08:47,072] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11438367,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11438367,"total_server_duration_us":9042674,"encode_duration_us":17536,"operation_id":"0x1a90","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10009330},{"operation_name":"upsert","last_dispatch_duration_us":49755,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8239219,"total_server_duration_us":194017,"encode_duration_us":19330,"operation_id":"0x1971","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10006844},{"operation_name":"upsert","last_dispatch_duration_us":37916,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":2911191,"total_server_duration_us":387505,"encode_duration_us":18442,"operation_id":"0x1aab","timeout_ms":10000,"last_server_duration_us":25712,"total_duration_us":10006523},{"operation_name":"upsert","last_dispatch_duration_us":34494,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":4611450,"total_server_duration_us":474965,"encode_duration_us":19992,"operation_id":"0x19d5","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10005744},{"operation_name":"upsert","last_dispatch_duration_us":7915381,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13210820,"total_server_duration_us":215411,"encode_duration_us":19431,"operation_id":"0x1a31","timeout_ms":10000,"last_server_duration_us":29323,"total_duration_us":10002843},{"operation_name":"upsert","last_dispatch_duration_us":10221460,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10221460,"total_server_duration_us":18876,"encode_duration_us":25647,"operation_id":"0x18d4","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10002250},{"operation_name":"upsert","last_dispatch_duration_us":1003437,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4590771,"total_server_duration_us":154944,"encode_duration_us":17640,"operation_id":"0x1b5c","timeout_ms":10000,"last_server_duration_us":46175,"total_duration_us":7730748},{"operation_name":"get","last_dispatch_duration_us":106550,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":106550,"total_server_duration_us":69,"operation_id":"0x1bbc","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":6805299},{"operation_name":"get","last_dispatch_duration_us":6414504,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":6414504,"total_server_duration_us":32,"operation_id":"0x19a0","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6416810},{"operation_name":"upsert","last_dispatch_duration_us":1602994,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3571696,"total_server_duration_us":909786,"encode_duration_us":30827,"operation_id":"0x1b16","timeout_ms":10000,"last_server_duration_us":776387,"total_duration_us":5700128}],"total_count":42}} INFO [2022-03-13 17:08:47,087] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,087] jepsen worker 33 - jepsen.util 573 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,087] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:47,088] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:47,123] jepsen worker 51 - jepsen.util 411 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,124] jepsen worker 51 - jepsen.util 411 :invoke :write [17 27] INFO [2022-03-13 17:08:47,130] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,131] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] WARN [2022-03-13 17:08:47,166] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8456344,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9010666,"total_server_duration_us":356827,"encode_duration_us":18392,"operation_id":"0x1911","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10009524},{"operation_name":"upsert","last_dispatch_duration_us":8608265,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8608265,"total_server_duration_us":21163,"encode_duration_us":31109,"operation_id":"0x18cb","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10005343},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":21429,"operation_id":"0x19cf","timeout_ms":10000,"total_duration_us":10001133},{"operation_name":"upsert","last_dispatch_duration_us":9083243,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9318211,"total_server_duration_us":9190401,"encode_duration_us":17535,"operation_id":"0x1992","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":9394249},{"operation_name":"upsert","last_dispatch_duration_us":784633,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":8524900,"total_server_duration_us":415818,"encode_duration_us":20104,"operation_id":"0x1a13","timeout_ms":10000,"last_server_duration_us":101224,"total_duration_us":8867365},{"operation_name":"upsert","last_dispatch_duration_us":795496,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":8485246,"total_server_duration_us":335971,"encode_duration_us":20138,"operation_id":"0x1a12","timeout_ms":10000,"last_server_duration_us":47762,"total_duration_us":8620915},{"operation_name":"upsert","last_dispatch_duration_us":621737,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6219078,"total_server_duration_us":258257,"encode_duration_us":19678,"operation_id":"0x1a62","timeout_ms":10000,"last_server_duration_us":52896,"total_duration_us":6241411},{"operation_name":"upsert","last_dispatch_duration_us":679648,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3358544,"total_server_duration_us":78235,"encode_duration_us":21330,"operation_id":"0x1b0c","timeout_ms":10000,"last_server_duration_us":40376,"total_duration_us":3377287},{"operation_name":"upsert","last_dispatch_duration_us":3242041,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3242041,"total_server_duration_us":94107,"encode_duration_us":20921,"operation_id":"0x1c7d","timeout_ms":10000,"last_server_duration_us":94107,"total_duration_us":3247163},{"operation_name":"get","last_dispatch_duration_us":2777819,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2777819,"total_server_duration_us":76,"operation_id":"0x1c40","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":2778217}],"total_count":19}} WARN [2022-03-13 17:08:47,166] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18638321,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18638321,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9648738,"encode_duration_us":18772,"operation_id":"0x17b8","timeout_ms":10000,"last_server_duration_us":9648738,"total_duration_us":10001858},{"operation_name":"upsert","last_dispatch_duration_us":17198875,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17766686,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9344304,"encode_duration_us":18217,"operation_id":"0x17fb","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10010371},{"operation_name":"upsert","last_dispatch_duration_us":17059434,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17693636,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9448569,"encode_duration_us":18671,"operation_id":"0x1778","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10003748}],"total_count":3}} INFO [2022-03-13 17:08:47,186] jepsen worker 55 - jepsen.util 775 :info :write [18 26] :RequestTimeoutException INFO [2022-03-13 17:08:47,187] jepsen worker 55 - jepsen.util 865 :invoke :write [18 27] INFO [2022-03-13 17:08:47,204] jepsen worker 13 - jepsen.util 463 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,205] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:47,213] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,213] jepsen worker 23 - jepsen.util 473 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,214] jepsen worker 69 - jepsen.util 609 :invoke :write [23 44] INFO [2022-03-13 17:08:47,215] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,219] jepsen worker 75 - jepsen.util 345 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,219] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,219] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,220] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:47,221] jepsen worker 53 - jepsen.util 503 :invoke :write [17 3] INFO [2022-03-13 17:08:47,221] jepsen worker 76 - jepsen.util 526 :invoke :write [25 24] INFO [2022-03-13 17:08:47,230] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,231] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,232] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,233] jepsen worker 23 - jepsen.util 473 :invoke :write [7 25] WARN [2022-03-13 17:08:47,266] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17961370,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18756669,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9258007,"encode_duration_us":27769,"operation_id":"0x1878","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10004233}],"total_count":1}} INFO [2022-03-13 17:08:47,280] jepsen worker 7 - jepsen.util 367 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,280] jepsen worker 7 - jepsen.util 367 :invoke :write [2 14] INFO [2022-03-13 17:08:47,280] jepsen worker 66 - jepsen.util 786 :info :write [22 19] :RequestTimeoutException INFO [2022-03-13 17:08:47,282] jepsen worker 66 - jepsen.util 876 :invoke :write [22 24] INFO [2022-03-13 17:08:47,299] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:47,299] jepsen worker 51 - jepsen.util 411 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,301] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:47,302] jepsen worker 13 - jepsen.util 463 :invoke :write [4 38] INFO [2022-03-13 17:08:47,304] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,306] jepsen worker 51 - jepsen.util 411 :invoke :write [17 39] INFO [2022-03-13 17:08:47,319] jepsen worker 69 - jepsen.util 609 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,326] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:47,342] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,343] jepsen worker 77 - jepsen.util 527 :invoke :write [25 12] INFO [2022-03-13 17:08:47,357] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:47,358] jepsen worker 60 - jepsen.util 600 :invoke :write [20 46] INFO [2022-03-13 17:08:47,372] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07331378299120239}, :ns_1@172.23.105.187 {:progress 0.08771929824561409}, :ns_1@172.23.107.227 {:progress 0.07331378299120239}, :ns_1@172.23.107.228 {:progress 0.07058823529411762}} INFO [2022-03-13 17:08:47,372] jepsen worker nemesis - couchbase.util Rebalance check count: 40 INFO [2022-03-13 17:08:47,372] jepsen worker nemesis - couchbase.util Rebalance stuck count: 25 INFO [2022-03-13 17:08:47,374] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,375] jepsen worker 23 - jepsen.util 473 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,375] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:47,376] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,380] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,380] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,381] jepsen worker 69 - jepsen.util 609 :invoke :write [23 18] INFO [2022-03-13 17:08:47,381] jepsen worker 23 - jepsen.util 473 :invoke :write [7 37] INFO [2022-03-13 17:08:47,386] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,386] jepsen worker 22 - jepsen.util 472 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,387] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,387] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:47,388] jepsen worker 6 - jepsen.util 546 :invoke :write [2 10] INFO [2022-03-13 17:08:47,389] jepsen worker 44 - jepsen.util 494 :invoke :write [14 12] INFO [2022-03-13 17:08:47,389] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:47,390] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,428] jepsen worker 26 - jepsen.util 476 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,428] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:47,451] jepsen worker 13 - jepsen.util 463 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,452] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:47,465] jepsen worker 43 - jepsen.util 403 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,465] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,466] jepsen worker 79 - jepsen.util 709 :invoke :write [26 47] INFO [2022-03-13 17:08:47,467] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:47,522] jepsen worker 23 - jepsen.util 473 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,522] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,529] jepsen worker 51 - jepsen.util 411 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,529] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:47,530] jepsen worker 13 - jepsen.util 463 :invoke :write [4 29] INFO [2022-03-13 17:08:47,530] jepsen worker 51 - jepsen.util 411 :invoke :write [17 22] INFO [2022-03-13 17:08:47,549] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:08:47,550] jepsen worker 33 - jepsen.util 573 :invoke :write [11 25] INFO [2022-03-13 17:08:47,555] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:47,557] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:47,560] jepsen worker 74 - jepsen.util 884 :info :write [24 5] :RequestTimeoutException INFO [2022-03-13 17:08:47,561] jepsen worker 74 - jepsen.util 974 :invoke :write [24 31] INFO [2022-03-13 17:08:47,666] jepsen worker 14 - jepsen.util 194 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,666] jepsen worker 7 - jepsen.util 367 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,667] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:47,667] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:47,677] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,677] jepsen worker 69 - jepsen.util 609 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,677] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,678] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:47,684] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,684] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,684] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,685] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:08:47,685] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,686] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,687] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,688] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:47,688] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:47,689] jepsen worker 23 - jepsen.util 473 :invoke :write [7 34] INFO [2022-03-13 17:08:47,704] jepsen worker 59 - jepsen.util 689 :info :write [19 35] :RequestTimeoutException INFO [2022-03-13 17:08:47,705] jepsen worker 59 - jepsen.util 779 :invoke :read [19 nil] INFO [2022-03-13 17:08:47,759] jepsen worker 13 - jepsen.util 463 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,760] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:47,833] jepsen worker 32 - jepsen.util 572 :info :write [10 33] :RequestTimeoutException INFO [2022-03-13 17:08:47,834] jepsen worker 32 - jepsen.util 662 :invoke :write [10 31] INFO [2022-03-13 17:08:47,854] jepsen worker 81 - jepsen.util 891 :info :write [27 9] :RequestTimeoutException INFO [2022-03-13 17:08:47,855] jepsen worker 81 - jepsen.util 981 :invoke :write [27 16] INFO [2022-03-13 17:08:47,867] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,867] jepsen worker 12 - jepsen.util 642 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,869] jepsen worker 12 - jepsen.util 642 :invoke :write [4 10] INFO [2022-03-13 17:08:47,870] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:08:47,879] jepsen worker 83 - jepsen.util 623 :ok :read [27 4] INFO [2022-03-13 17:08:47,880] jepsen worker 83 - jepsen.util 623 :invoke :write [27 25] INFO [2022-03-13 17:08:47,906] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:47,906] jepsen worker 51 - jepsen.util 411 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,907] jepsen worker 51 - jepsen.util 411 :invoke :write [17 40] INFO [2022-03-13 17:08:47,908] jepsen worker 13 - jepsen.util 463 :invoke :write [4 23] INFO [2022-03-13 17:08:47,944] jepsen worker 4 - jepsen.util 724 :ok :read [1 16] INFO [2022-03-13 17:08:47,944] jepsen worker 4 - jepsen.util 724 :invoke :read [1 nil] INFO [2022-03-13 17:08:47,949] jepsen worker 77 - jepsen.util 527 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:47,950] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:48,033] jepsen worker 4 - jepsen.util 724 :ok :read [1 16] INFO [2022-03-13 17:08:48,034] jepsen worker 4 - jepsen.util 724 :invoke :read [1 nil] INFO [2022-03-13 17:08:48,036] jepsen worker 27 - jepsen.util 747 :info :write [9 37] :RequestTimeoutException INFO [2022-03-13 17:08:48,036] jepsen worker 27 - jepsen.util 837 :invoke :write [9 44] INFO [2022-03-13 17:08:48,092] jepsen worker 23 - jepsen.util 473 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,092] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:48,100] jepsen worker 4 - jepsen.util 724 :ok :read [1 16] INFO [2022-03-13 17:08:48,101] jepsen worker 4 - jepsen.util 724 :invoke :write [1 49] INFO [2022-03-13 17:08:48,160] jepsen worker 13 - jepsen.util 463 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,161] jepsen worker 13 - jepsen.util 463 :invoke :write [4 27] INFO [2022-03-13 17:08:48,190] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,190] jepsen worker 42 - jepsen.util 402 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,191] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:08:48,191] jepsen worker 75 - jepsen.util 345 :invoke :write [25 8] INFO [2022-03-13 17:08:48,192] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,192] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:48,193] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:48,194] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:48,194] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:48,195] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:48,196] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,196] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:48,197] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:48,197] jepsen worker 23 - jepsen.util 473 :invoke :write [7 1] INFO [2022-03-13 17:08:48,198] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,199] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:48,200] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,201] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:08:48,220] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:48,220] jepsen worker 60 - jepsen.util 600 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,221] jepsen worker 43 - jepsen.util 403 :invoke :write [14 38] INFO [2022-03-13 17:08:48,224] jepsen worker 60 - jepsen.util 600 :invoke :write [20 40] INFO [2022-03-13 17:08:48,241] jepsen worker 51 - jepsen.util 411 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,242] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:48,298] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,299] jepsen worker 69 - jepsen.util 609 :invoke :write [23 31] INFO [2022-03-13 17:08:48,321] jepsen worker 53 - jepsen.util 503 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,321] jepsen worker 13 - jepsen.util 463 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,321] jepsen worker 53 - jepsen.util 503 :invoke :write [17 30] INFO [2022-03-13 17:08:48,321] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,323] jepsen worker 51 - jepsen.util 411 :invoke :write [17 23] INFO [2022-03-13 17:08:48,334] jepsen worker 13 - jepsen.util 463 :invoke :write [4 24] INFO [2022-03-13 17:08:48,363] jepsen worker 23 - jepsen.util 473 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,364] jepsen worker 23 - jepsen.util 473 :invoke :write [7 6] INFO [2022-03-13 17:08:48,373] jepsen worker 79 - jepsen.util 709 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,374] jepsen worker 79 - jepsen.util 709 :invoke :write [26 47] INFO [2022-03-13 17:08:48,378] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07917888563049857}, :ns_1@172.23.105.187 {:progress 0.0935672514619883}, :ns_1@172.23.107.227 {:progress 0.07917888563049857}, :ns_1@172.23.107.228 {:progress 0.07647058823529407}} INFO [2022-03-13 17:08:48,378] jepsen worker nemesis - couchbase.util Rebalance check count: 41 INFO [2022-03-13 17:08:48,378] jepsen worker nemesis - couchbase.util Rebalance stuck count: 25 INFO [2022-03-13 17:08:48,438] jepsen worker 51 - jepsen.util 411 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,439] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:48,444] jepsen worker 69 - jepsen.util 609 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,445] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:48,474] jepsen worker 76 - jepsen.util 526 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,475] jepsen worker 76 - jepsen.util 526 :invoke :write [25 42] INFO [2022-03-13 17:08:48,507] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,507] jepsen worker 23 - jepsen.util 473 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,508] jepsen worker 69 - jepsen.util 609 :invoke :write [23 47] INFO [2022-03-13 17:08:48,508] jepsen worker 23 - jepsen.util 473 :invoke :write [7 38] INFO [2022-03-13 17:08:48,532] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,532] jepsen worker 13 - jepsen.util 463 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,534] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:48,534] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:48,535] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:48,535] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,536] jepsen worker 13 - jepsen.util 463 :invoke :write [4 31] INFO [2022-03-13 17:08:48,536] jepsen worker 51 - jepsen.util 411 :invoke :write [17 44] INFO [2022-03-13 17:08:48,553] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,553] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,553] jepsen worker 33 - jepsen.util 573 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,553] jepsen worker 77 - jepsen.util 527 :invoke :write [25 32] INFO [2022-03-13 17:08:48,554] jepsen worker 7 - jepsen.util 367 :invoke :write [2 42] INFO [2022-03-13 17:08:48,555] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:48,616] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:48,617] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:48,638] jepsen worker 21 - jepsen.util 651 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,638] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:48,638] jepsen worker 71 - jepsen.util 431 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,639] jepsen worker 24 - jepsen.util 474 :invoke :write [8 24] INFO [2022-03-13 17:08:48,639] jepsen worker 21 - jepsen.util 651 :invoke :write [7 16] INFO [2022-03-13 17:08:48,640] jepsen worker 71 - jepsen.util 431 :invoke :write [23 3] INFO [2022-03-13 17:08:48,652] jepsen worker 69 - jepsen.util 609 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,653] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:08:48,656] jepsen worker 51 - jepsen.util 411 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,657] jepsen worker 51 - jepsen.util 411 :invoke :write [17 33] INFO [2022-03-13 17:08:48,675] jepsen worker 36 - jepsen.util 756 :info :write [12 17] :RequestTimeoutException INFO [2022-03-13 17:08:48,675] jepsen worker 36 - jepsen.util 846 :invoke :write [12 2] INFO [2022-03-13 17:08:48,712] jepsen worker 13 - jepsen.util 463 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,713] jepsen worker 13 - jepsen.util 463 :invoke :write [4 3] INFO [2022-03-13 17:08:48,726] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:48,726] jepsen worker 35 - jepsen.util 665 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,726] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:48,727] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:48,728] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:48,728] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:48,746] jepsen worker 23 - jepsen.util 473 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,747] jepsen worker 23 - jepsen.util 473 :invoke :write [7 21] INFO [2022-03-13 17:08:48,779] jepsen worker 43 - jepsen.util 403 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,780] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:48,791] jepsen worker 51 - jepsen.util 411 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,792] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:48,831] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,832] jepsen worker 69 - jepsen.util 609 :invoke :write [23 12] INFO [2022-03-13 17:08:48,875] jepsen worker 12 - jepsen.util 642 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,876] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:08:48,890] jepsen worker 13 - jepsen.util 463 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,890] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,890] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:48,891] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:48,892] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:48,892] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,892] jepsen worker 13 - jepsen.util 463 :invoke :write [4 49] INFO [2022-03-13 17:08:48,893] jepsen worker 51 - jepsen.util 411 :invoke :write [17 13] INFO [2022-03-13 17:08:48,914] jepsen worker 23 - jepsen.util 473 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,914] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:48,929] jepsen worker 60 - jepsen.util 600 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:48,930] jepsen worker 60 - jepsen.util 600 :invoke :write [20 34] INFO [2022-03-13 17:08:49,000] jepsen worker 13 - jepsen.util 463 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,001] jepsen worker 13 - jepsen.util 463 :invoke :write [4 3] INFO [2022-03-13 17:08:49,027] jepsen worker 69 - jepsen.util 609 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,027] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:49,028] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:49,029] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:49,031] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:49,031] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,032] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:49,032] jepsen worker 23 - jepsen.util 473 :invoke :write [7 3] INFO [2022-03-13 17:08:49,033] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,034] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:49,035] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,035] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:08:49,046] jepsen worker 62 - jepsen.util 422 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,051] jepsen worker 62 - jepsen.util 422 :invoke :write [20 30] INFO [2022-03-13 17:08:49,055] jepsen worker 51 - jepsen.util 411 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,056] jepsen worker 51 - jepsen.util 411 :invoke :write [17 17] INFO [2022-03-13 17:08:49,083] jepsen worker 79 - jepsen.util 709 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,085] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:49,112] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:08:49,113] jepsen worker 33 - jepsen.util 573 :invoke :write [11 21] INFO [2022-03-13 17:08:49,151] jepsen worker 23 - jepsen.util 473 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,151] jepsen worker 23 - jepsen.util 473 :invoke :write [7 42] INFO [2022-03-13 17:08:49,176] jepsen worker 13 - jepsen.util 463 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,177] jepsen worker 13 - jepsen.util 463 :invoke :write [4 1] INFO [2022-03-13 17:08:49,221] jepsen worker 75 - jepsen.util 345 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,222] jepsen worker 75 - jepsen.util 345 :invoke :write [25 48] INFO [2022-03-13 17:08:49,228] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,229] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:08:49,239] jepsen worker 86 - jepsen.util 806 :ok :read [28 44] INFO [2022-03-13 17:08:49,239] jepsen worker 86 - jepsen.util 806 :invoke :read [28 nil] INFO [2022-03-13 17:08:49,245] jepsen worker 5 - jepsen.util 725 :ok :read [1 16] INFO [2022-03-13 17:08:49,246] jepsen worker 5 - jepsen.util 725 :invoke :read [1 nil] INFO [2022-03-13 17:08:49,263] jepsen worker 48 - jepsen.util 588 :ok :read [16 :nil] INFO [2022-03-13 17:08:49,264] jepsen worker 48 - jepsen.util 588 :invoke :write [16 6] INFO [2022-03-13 17:08:49,281] jepsen worker 57 - jepsen.util 777 :ok :read [19 2] INFO [2022-03-13 17:08:49,282] jepsen worker 57 - jepsen.util 777 :invoke :write [19 12] INFO [2022-03-13 17:08:49,286] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:49,287] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:49,344] jepsen worker 5 - jepsen.util 725 :ok :read [1 16] INFO [2022-03-13 17:08:49,345] jepsen worker 5 - jepsen.util 725 :invoke :write [1 42] INFO [2022-03-13 17:08:49,385] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07917888563049857}, :ns_1@172.23.105.187 {:progress 0.0935672514619883}, :ns_1@172.23.107.227 {:progress 0.07917888563049857}, :ns_1@172.23.107.228 {:progress 0.07647058823529407}} INFO [2022-03-13 17:08:49,385] jepsen worker nemesis - couchbase.util Rebalance check count: 42 INFO [2022-03-13 17:08:49,385] jepsen worker nemesis - couchbase.util Rebalance stuck count: 26 INFO [2022-03-13 17:08:49,391] jepsen worker 31 - jepsen.util 661 :info :write [10 26] :RequestTimeoutException INFO [2022-03-13 17:08:49,391] jepsen worker 31 - jepsen.util 751 :invoke :write [10 35] INFO [2022-03-13 17:08:49,439] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:49,439] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:49,477] jepsen worker 29 - jepsen.util 839 :ok :write [9 23] INFO [2022-03-13 17:08:49,479] jepsen worker 29 - jepsen.util 839 :invoke :write [9 28] INFO [2022-03-13 17:08:49,489] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:49,490] jepsen worker 43 - jepsen.util 403 :invoke :write [14 5] INFO [2022-03-13 17:08:49,523] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:49,524] jepsen worker 61 - jepsen.util 421 :invoke :write [20 4] INFO [2022-03-13 17:08:49,623] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:49,624] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:49,704] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:49,705] jepsen worker 79 - jepsen.util 709 :invoke :write [26 37] INFO [2022-03-13 17:08:49,738] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:49,738] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:49,738] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:08:49,740] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:49,740] jepsen worker 14 - jepsen.util 194 :invoke :write [4 46] INFO [2022-03-13 17:08:49,741] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:08:49,751] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:08:49,752] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:49,789] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:49,789] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:49,953] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:49,953] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:50,020] jepsen worker 40 - jepsen.util 670 :ok :read [13 39] INFO [2022-03-13 17:08:50,020] jepsen worker 40 - jepsen.util 670 :invoke :read [13 nil] INFO [2022-03-13 17:08:50,131] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:50,132] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:50,140] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:50,141] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:50,142] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,146] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,161] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:08:50,162] jepsen worker 34 - jepsen.util 574 :invoke :write [11 48] INFO [2022-03-13 17:08:50,180] jepsen worker 49 - jepsen.util 679 :ok :read [16 :nil] INFO [2022-03-13 17:08:50,181] jepsen worker 49 - jepsen.util 679 :invoke :write [16 11] INFO [2022-03-13 17:08:50,214] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,215] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,269] jepsen worker 51 - jepsen.util 411 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,270] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:50,286] jepsen worker 40 - jepsen.util 670 :ok :read [13 39] INFO [2022-03-13 17:08:50,286] jepsen worker 40 - jepsen.util 670 :invoke :read [13 nil] INFO [2022-03-13 17:08:50,290] jepsen worker 84 - jepsen.util 894 :info :write [28 33] :RequestTimeoutException INFO [2022-03-13 17:08:50,291] jepsen worker 84 - jepsen.util 984 :invoke :write [28 43] INFO [2022-03-13 17:08:50,307] jepsen worker 78 - jepsen.util 438 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,307] jepsen worker 78 - jepsen.util 438 :invoke :write [26 7] INFO [2022-03-13 17:08:50,316] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:50,323] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:50,372] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,372] jepsen worker 23 - jepsen.util 473 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,373] jepsen worker 23 - jepsen.util 473 :invoke :write [7 44] INFO [2022-03-13 17:08:50,374] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,375] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,375] jepsen worker 13 - jepsen.util 463 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,375] jepsen worker 6 - jepsen.util 546 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,376] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:50,376] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:50,377] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,378] jepsen worker 7 - jepsen.util 367 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,378] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,378] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:50,379] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:50,379] jepsen worker 75 - jepsen.util 345 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,381] jepsen worker 7 - jepsen.util 367 :invoke :write [2 14] INFO [2022-03-13 17:08:50,382] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:50,384] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:50,384] jepsen worker 51 - jepsen.util 411 :invoke :write [17 48] INFO [2022-03-13 17:08:50,386] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:50,386] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,387] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:50,388] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,390] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:50,392] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:08:50,393] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.07917888563049857}, :ns_1@172.23.105.187 {:progress 0.1052631578947368}, :ns_1@172.23.107.227 {:progress 0.07917888563049857}, :ns_1@172.23.107.228 {:progress 0.07647058823529407}} INFO [2022-03-13 17:08:50,393] jepsen worker nemesis - couchbase.util Rebalance check count: 43 INFO [2022-03-13 17:08:50,393] jepsen worker nemesis - couchbase.util Rebalance stuck count: 26 INFO [2022-03-13 17:08:50,394] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:50,395] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:50,396] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:50,398] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:50,399] jepsen worker 13 - jepsen.util 463 :invoke :write [4 37] INFO [2022-03-13 17:08:50,493] jepsen worker 68 - jepsen.util 788 :ok :read [22 30] INFO [2022-03-13 17:08:50,494] jepsen worker 68 - jepsen.util 788 :invoke :read [22 nil] INFO [2022-03-13 17:08:50,498] jepsen worker 40 - jepsen.util 670 :ok :read [13 39] INFO [2022-03-13 17:08:50,498] jepsen worker 40 - jepsen.util 670 :invoke :read [13 nil] INFO [2022-03-13 17:08:50,507] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:50,507] jepsen worker 1 - jepsen.util 811 :invoke :read [0 nil] INFO [2022-03-13 17:08:50,507] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,508] jepsen worker 42 - jepsen.util 402 :invoke :write [14 29] INFO [2022-03-13 17:08:50,535] jepsen worker 23 - jepsen.util 473 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,537] jepsen worker 25 - jepsen.util 385 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,537] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,537] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:50,538] jepsen worker 59 - jepsen.util 779 :ok :read [19 2] INFO [2022-03-13 17:08:50,538] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:50,539] jepsen worker 25 - jepsen.util 385 :invoke :write [8 19] INFO [2022-03-13 17:08:50,541] jepsen worker 59 - jepsen.util 779 :invoke :read [19 nil] INFO [2022-03-13 17:08:50,545] jepsen worker 67 - jepsen.util 787 :ok :read [22 30] INFO [2022-03-13 17:08:50,546] jepsen worker 67 - jepsen.util 787 :invoke :write [22 30] INFO [2022-03-13 17:08:50,564] jepsen worker 51 - jepsen.util 411 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,565] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:50,580] jepsen worker 43 - jepsen.util 403 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,581] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:50,591] jepsen worker 44 - jepsen.util 494 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,592] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:50,621] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,621] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:50,622] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:08:50,623] jepsen worker 23 - jepsen.util 473 :invoke :write [7 37] INFO [2022-03-13 17:08:50,630] jepsen worker 1 - jepsen.util 811 :ok :read [0 41] INFO [2022-03-13 17:08:50,632] jepsen worker 1 - jepsen.util 811 :invoke :write [0 34] INFO [2022-03-13 17:08:50,662] jepsen worker 13 - jepsen.util 463 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,663] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,663] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:50,664] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:50,755] jepsen worker 39 - jepsen.util 939 :ok :read [13 39] INFO [2022-03-13 17:08:50,759] jepsen worker 39 - jepsen.util 939 :invoke :write [13 34] INFO [2022-03-13 17:08:50,764] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:50,764] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,767] jepsen worker 51 - jepsen.util 411 :invoke :write [17 28] INFO [2022-03-13 17:08:50,771] jepsen worker 13 - jepsen.util 463 :invoke :write [4 5] INFO [2022-03-13 17:08:50,781] jepsen worker 15 - jepsen.util 735 :ok :read [5 43] INFO [2022-03-13 17:08:50,781] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,781] jepsen worker 15 - jepsen.util 735 :invoke :write [5 23] INFO [2022-03-13 17:08:50,786] jepsen worker 61 - jepsen.util 421 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,787] jepsen worker 61 - jepsen.util 421 :invoke :write [20 33] INFO [2022-03-13 17:08:50,790] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,814] jepsen worker 80 - jepsen.util 620 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,815] jepsen worker 80 - jepsen.util 620 :invoke :write [26 11] INFO [2022-03-13 17:08:50,835] jepsen worker 23 - jepsen.util 473 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,835] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,837] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:50,838] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:08:50,843] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:50,843] jepsen worker 23 - jepsen.util 473 :invoke :write [7 48] INFO [2022-03-13 17:08:50,849] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:50,849] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,849] jepsen worker 71 - jepsen.util 431 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,851] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,851] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:50,852] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,887] jepsen worker 60 - jepsen.util 600 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,888] jepsen worker 60 - jepsen.util 600 :invoke :write [20 25] INFO [2022-03-13 17:08:50,945] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,945] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:50,948] jepsen worker 3 - jepsen.util 543 :info :write [1 12] :RequestTimeoutException INFO [2022-03-13 17:08:50,949] jepsen worker 3 - jepsen.util 633 :invoke :read [1 nil] INFO [2022-03-13 17:08:50,969] jepsen worker 13 - jepsen.util 463 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:08:50,970] jepsen worker 13 - jepsen.util 463 :invoke :write [4 21] INFO [2022-03-13 17:08:50,978] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:50,979] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:08:50,983] jepsen worker 26 - jepsen.util 476 :invoke :write [8 19] INFO [2022-03-13 17:08:50,989] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:08:50,997] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:08:50,998] jepsen worker 35 - jepsen.util 665 :invoke :write [11 42] INFO [2022-03-13 17:08:51,018] jepsen worker 58 - jepsen.util 598 :info :write [19 32] :RequestTimeoutException INFO [2022-03-13 17:08:51,020] jepsen worker 58 - jepsen.util 688 :invoke :write [19 19] INFO [2022-03-13 17:08:51,069] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,069] jepsen worker 23 - jepsen.util 473 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,070] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:08:51,071] jepsen worker 23 - jepsen.util 473 :invoke :write [7 47] INFO [2022-03-13 17:08:51,084] jepsen worker 34 - jepsen.util 574 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,084] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,084] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:51,085] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:08:51,086] jepsen worker 6 - jepsen.util 546 :invoke :write [2 25] INFO [2022-03-13 17:08:51,086] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,087] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,087] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:51,087] jepsen worker 52 - jepsen.util 502 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,087] jepsen worker 54 - jepsen.util 774 :ok :read [18 16] INFO [2022-03-13 17:08:51,088] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,089] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:51,089] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:51,090] jepsen worker 54 - jepsen.util 774 :invoke :write [18 49] INFO [2022-03-13 17:08:51,091] jepsen worker 52 - jepsen.util 502 :invoke :write [17 40] INFO [2022-03-13 17:08:51,092] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,109] jepsen worker 51 - jepsen.util 411 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,110] jepsen worker 51 - jepsen.util 411 :invoke :write [17 40] INFO [2022-03-13 17:08:51,133] jepsen worker 14 - jepsen.util 194 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,134] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:51,137] jepsen worker 19 - jepsen.util 559 :ok :read [6 16] INFO [2022-03-13 17:08:51,137] jepsen worker 19 - jepsen.util 559 :invoke :read [6 nil] INFO [2022-03-13 17:08:51,182] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,183] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:51,183] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,183] jepsen worker 22 - jepsen.util 472 :invoke :write [7 41] INFO [2022-03-13 17:08:51,190] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:51,191] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,191] jepsen worker 71 - jepsen.util 431 :invoke :write [23 29] INFO [2022-03-13 17:08:51,200] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:51,202] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:08:51,203] jepsen worker 34 - jepsen.util 574 :invoke :write [11 32] INFO [2022-03-13 17:08:51,213] jepsen worker 79 - jepsen.util 709 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,214] jepsen worker 79 - jepsen.util 709 :invoke :write [26 21] INFO [2022-03-13 17:08:51,246] jepsen worker 19 - jepsen.util 559 :ok :read [6 16] INFO [2022-03-13 17:08:51,247] jepsen worker 19 - jepsen.util 559 :invoke :read [6 nil] INFO [2022-03-13 17:08:51,294] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,295] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,302] jepsen worker 13 - jepsen.util 463 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,303] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:51,304] jepsen worker 19 - jepsen.util 559 :ok :read [6 16] INFO [2022-03-13 17:08:51,305] jepsen worker 19 - jepsen.util 559 :invoke :write [6 9] INFO [2022-03-13 17:08:51,372] jepsen worker 24 - jepsen.util 474 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,372] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:51,399] jepsen worker 23 - jepsen.util 473 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,399] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,400] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.09090909090909094}, :ns_1@172.23.105.187 {:progress 0.1111111111111112}, :ns_1@172.23.107.227 {:progress 0.09090909090909094}, :ns_1@172.23.107.228 {:progress 0.08823529411764708}} INFO [2022-03-13 17:08:51,400] jepsen worker nemesis - couchbase.util Rebalance check count: 44 INFO [2022-03-13 17:08:51,400] jepsen worker nemesis - couchbase.util Rebalance stuck count: 26 INFO [2022-03-13 17:08:51,401] jepsen worker 23 - jepsen.util 473 :invoke :write [7 8] INFO [2022-03-13 17:08:51,401] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:08:51,431] jepsen worker 51 - jepsen.util 411 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,431] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:51,432] jepsen worker 51 - jepsen.util 411 :invoke :write [17 1] INFO [2022-03-13 17:08:51,432] jepsen worker 13 - jepsen.util 463 :invoke :write [4 34] INFO [2022-03-13 17:08:51,456] jepsen worker 62 - jepsen.util 422 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,457] jepsen worker 33 - jepsen.util 573 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,457] jepsen worker 62 - jepsen.util 422 :invoke :write [20 11] INFO [2022-03-13 17:08:51,458] jepsen worker 33 - jepsen.util 573 :invoke :write [11 16] INFO [2022-03-13 17:08:51,504] jepsen worker 6 - jepsen.util 546 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,505] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,505] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:51,505] jepsen worker 44 - jepsen.util 494 :invoke :write [14 16] INFO [2022-03-13 17:08:51,506] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,506] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:51,539] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,540] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:08:51,548] jepsen worker 13 - jepsen.util 463 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,548] jepsen worker 13 - jepsen.util 463 :invoke :write [4 44] INFO [2022-03-13 17:08:51,596] jepsen worker 76 - jepsen.util 526 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,596] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,597] jepsen worker 75 - jepsen.util 345 :invoke :write [25 9] INFO [2022-03-13 17:08:51,597] jepsen worker 76 - jepsen.util 526 :invoke :write [25 4] INFO [2022-03-13 17:08:51,612] jepsen worker 21 - jepsen.util 651 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,612] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,613] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:08:51,613] jepsen worker 8 - jepsen.util 638 :invoke :write [2 6] INFO [2022-03-13 17:08:51,616] jepsen worker 23 - jepsen.util 473 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,617] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:51,636] jepsen worker 51 - jepsen.util 411 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,637] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:51,674] jepsen worker 22 - jepsen.util 472 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,675] jepsen worker 22 - jepsen.util 472 :invoke :write [7 46] INFO [2022-03-13 17:08:51,691] jepsen worker 13 - jepsen.util 463 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,691] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,695] jepsen worker 13 - jepsen.util 463 :invoke :write [4 21] INFO [2022-03-13 17:08:51,696] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:51,698] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,698] jepsen worker 51 - jepsen.util 411 :invoke :write [17 2] INFO [2022-03-13 17:08:51,703] jepsen worker 7 - jepsen.util 367 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,703] jepsen worker 7 - jepsen.util 367 :invoke :write [2 36] INFO [2022-03-13 17:08:51,714] jepsen worker 25 - jepsen.util 385 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,715] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:51,752] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,752] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:51,753] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:08:51,753] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:51,754] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:51,755] jepsen worker 23 - jepsen.util 473 :invoke :write [7 9] INFO [2022-03-13 17:08:51,780] jepsen worker 71 - jepsen.util 431 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,781] jepsen worker 71 - jepsen.util 431 :invoke :write [23 37] INFO [2022-03-13 17:08:51,825] jepsen worker 13 - jepsen.util 463 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,825] jepsen worker 13 - jepsen.util 463 :invoke :write [4 43] INFO [2022-03-13 17:08:51,831] jepsen worker 42 - jepsen.util 402 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,832] jepsen worker 42 - jepsen.util 402 :invoke :write [14 9] INFO [2022-03-13 17:08:51,870] jepsen worker 50 - jepsen.util 770 :info :write [16 3] :RequestTimeoutException INFO [2022-03-13 17:08:51,870] jepsen worker 50 - jepsen.util 860 :invoke :write [16 48] INFO [2022-03-13 17:08:51,888] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,889] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,896] jepsen worker 2 - jepsen.util 812 :info :write [0 26] :RequestTimeoutException INFO [2022-03-13 17:08:51,897] jepsen worker 2 - jepsen.util 902 :invoke :write [0 29] INFO [2022-03-13 17:08:51,904] jepsen worker 80 - jepsen.util 620 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,905] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:51,917] jepsen worker 51 - jepsen.util 411 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,917] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:51,921] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:51,921] jepsen worker 77 - jepsen.util 527 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,922] jepsen worker 77 - jepsen.util 527 :invoke :write [25 23] INFO [2022-03-13 17:08:51,923] jepsen worker 43 - jepsen.util 403 :invoke :write [14 13] INFO [2022-03-13 17:08:51,957] jepsen worker 23 - jepsen.util 473 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,957] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,958] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:51,958] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,984] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,984] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:51,985] jepsen worker 13 - jepsen.util 463 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,985] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,985] jepsen worker 51 - jepsen.util 411 :invoke :write [17 41] INFO [2022-03-13 17:08:51,986] jepsen worker 13 - jepsen.util 463 :invoke :write [4 42] INFO [2022-03-13 17:08:51,988] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:51,990] jepsen worker 34 - jepsen.util 574 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,990] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,990] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,990] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,990] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:51,991] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:51,991] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:51,992] jepsen worker 6 - jepsen.util 546 :invoke :write [2 5] INFO [2022-03-13 17:08:51,992] jepsen worker 34 - jepsen.util 574 :invoke :write [11 13] INFO [2022-03-13 17:08:51,993] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:08:51,994] jepsen worker 23 - jepsen.util 473 :invoke :write [7 11] INFO [2022-03-13 17:08:52,013] jepsen worker 4 - jepsen.util 724 :info :write [1 49] :SyncWriteAmbiguous INFO [2022-03-13 17:08:52,015] jepsen worker 4 - jepsen.util 814 :invoke :read [1 nil] INFO [2022-03-13 17:08:52,050] jepsen worker 52 - jepsen.util 502 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,050] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:08:52,051] jepsen worker 21 - jepsen.util 651 :invoke :write [7 24] INFO [2022-03-13 17:08:52,052] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:08:52,132] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:08:52,132] jepsen worker 61 - jepsen.util 421 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,133] jepsen worker 61 - jepsen.util 421 :invoke :write [20 10] INFO [2022-03-13 17:08:52,135] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:08:52,165] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,165] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:52,173] jepsen worker 44 - jepsen.util 494 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,173] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:52,178] jepsen worker 30 - jepsen.util 570 :info :write [10 46] :RequestTimeoutException INFO [2022-03-13 17:08:52,179] jepsen worker 30 - jepsen.util 660 :invoke :write [10 14] INFO [2022-03-13 17:08:52,180] jepsen worker 13 - jepsen.util 463 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,180] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:52,217] jepsen worker 47 - jepsen.util 857 :info :write [15 7] :RequestTimeoutException INFO [2022-03-13 17:08:52,217] jepsen worker 47 - jepsen.util 947 :invoke :read [15 nil] INFO [2022-03-13 17:08:52,264] jepsen worker 60 - jepsen.util 600 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,264] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:08:52,303] jepsen worker 23 - jepsen.util 473 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,303] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:52,303] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,304] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:08:52,304] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:52,305] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:52,306] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:52,306] jepsen worker 51 - jepsen.util 411 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,306] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:52,307] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:52,308] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:52,308] jepsen worker 13 - jepsen.util 463 :invoke :write [4 7] INFO [2022-03-13 17:08:52,309] jepsen worker 51 - jepsen.util 411 :invoke :write [17 7] INFO [2022-03-13 17:08:52,310] jepsen worker 23 - jepsen.util 473 :invoke :write [7 34] INFO [2022-03-13 17:08:52,311] jepsen worker 8 - jepsen.util 638 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,311] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:52,312] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,312] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:52,313] jepsen worker 80 - jepsen.util 620 :invoke :write [26 36] INFO [2022-03-13 17:08:52,313] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:52,351] jepsen worker 45 - jepsen.util 945 :info :write [15 4] :RequestTimeoutException INFO [2022-03-13 17:08:52,351] jepsen worker 45 - jepsen.util 1035 :invoke :read [15 nil] INFO [2022-03-13 17:08:52,394] jepsen worker 22 - jepsen.util 472 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,395] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:52,406] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.09677419354838712}, :ns_1@172.23.105.187 {:progress 0.1111111111111112}, :ns_1@172.23.107.227 {:progress 0.09677419354838712}, :ns_1@172.23.107.228 {:progress 0.09411764705882353}} INFO [2022-03-13 17:08:52,407] jepsen worker nemesis - couchbase.util Rebalance check count: 45 INFO [2022-03-13 17:08:52,407] jepsen worker nemesis - couchbase.util Rebalance stuck count: 26 INFO [2022-03-13 17:08:52,475] jepsen worker 51 - jepsen.util 411 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,476] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:52,501] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,501] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:08:52,542] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,542] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:52,542] jepsen worker 78 - jepsen.util 438 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,543] jepsen worker 24 - jepsen.util 474 :invoke :write [8 20] INFO [2022-03-13 17:08:52,543] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:52,543] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:08:52,595] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,595] jepsen worker 13 - jepsen.util 463 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,595] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:52,596] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:52,608] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,608] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:52,608] jepsen worker 23 - jepsen.util 473 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,608] jepsen worker 51 - jepsen.util 411 :invoke :write [17 34] INFO [2022-03-13 17:08:52,609] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:52,609] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:52,610] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:52,611] jepsen worker 13 - jepsen.util 463 :invoke :write [4 16] INFO [2022-03-13 17:08:52,622] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,622] jepsen worker 71 - jepsen.util 431 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,623] jepsen worker 70 - jepsen.util 700 :invoke :write [23 41] INFO [2022-03-13 17:08:52,623] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:52,629] jepsen worker 79 - jepsen.util 709 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,630] jepsen worker 79 - jepsen.util 709 :invoke :write [26 7] INFO [2022-03-13 17:08:52,633] jepsen worker 26 - jepsen.util 476 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,634] jepsen worker 26 - jepsen.util 476 :invoke :write [8 36] INFO [2022-03-13 17:08:52,638] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:52,639] jepsen worker 14 - jepsen.util 194 :invoke :write [4 45] INFO [2022-03-13 17:08:52,721] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:52,721] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,722] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:52,723] jepsen worker 23 - jepsen.util 473 :invoke :write [7 44] INFO [2022-03-13 17:08:52,724] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,726] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:08:52,750] jepsen worker 51 - jepsen.util 411 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,751] jepsen worker 51 - jepsen.util 411 :invoke :write [17 31] INFO [2022-03-13 17:08:52,787] jepsen worker 21 - jepsen.util 651 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,787] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:52,787] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,787] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:08:52,788] jepsen worker 25 - jepsen.util 385 :invoke :write [8 40] INFO [2022-03-13 17:08:52,788] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:52,795] jepsen worker 23 - jepsen.util 473 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,795] jepsen worker 23 - jepsen.util 473 :invoke :write [7 42] INFO [2022-03-13 17:08:52,802] jepsen worker 35 - jepsen.util 665 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,802] jepsen worker 33 - jepsen.util 573 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,803] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:52,803] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:52,872] jepsen worker 13 - jepsen.util 463 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,873] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:52,880] jepsen worker 6 - jepsen.util 546 :fail :write [2 5] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,881] jepsen worker 6 - jepsen.util 546 :invoke :write [2 44] INFO [2022-03-13 17:08:52,882] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,882] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:52,916] jepsen worker 62 - jepsen.util 422 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,916] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:08:52,958] jepsen worker 46 - jepsen.util 676 :info :write [15 41] :RequestTimeoutException INFO [2022-03-13 17:08:52,959] jepsen worker 46 - jepsen.util 766 :invoke :write [15 3] INFO [2022-03-13 17:08:52,976] jepsen worker 23 - jepsen.util 473 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,976] jepsen worker 51 - jepsen.util 411 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,976] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,976] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:52,976] jepsen worker 51 - jepsen.util 411 :invoke :write [17 36] INFO [2022-03-13 17:08:52,977] jepsen worker 23 - jepsen.util 473 :invoke :write [7 14] INFO [2022-03-13 17:08:52,978] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:52,978] jepsen worker 69 - jepsen.util 609 :invoke :write [23 44] INFO [2022-03-13 17:08:52,982] jepsen worker 80 - jepsen.util 620 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,982] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,982] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,983] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:08:52,984] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:52,984] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:52,985] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:52,985] jepsen worker 13 - jepsen.util 463 :invoke :write [4 46] INFO [2022-03-13 17:08:52,988] jepsen worker 7 - jepsen.util 367 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:52,989] jepsen worker 7 - jepsen.util 367 :invoke :write [2 17] INFO [2022-03-13 17:08:52,993] jepsen worker 36 - jepsen.util 846 :ok :write [12 2] INFO [2022-03-13 17:08:52,994] jepsen worker 36 - jepsen.util 846 :invoke :write [12 35] INFO [2022-03-13 17:08:53,020] jepsen worker 87 - jepsen.util 987 :info :write [29 16] :RequestTimeoutException INFO [2022-03-13 17:08:53,021] jepsen worker 87 - jepsen.util 1077 :invoke :write [29 15] INFO [2022-03-13 17:08:53,073] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:53,073] jepsen worker 34 - jepsen.util 574 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,074] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:53,074] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,074] jepsen worker 34 - jepsen.util 574 :invoke :write [11 36] INFO [2022-03-13 17:08:53,076] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:53,077] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:53,077] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:53,084] jepsen worker 11 - jepsen.util 731 :info :write [3 31] :RequestTimeoutException INFO [2022-03-13 17:08:53,084] jepsen worker 11 - jepsen.util 821 :invoke :write [3 44] INFO [2022-03-13 17:08:53,105] jepsen worker 75 - jepsen.util 345 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,106] jepsen worker 75 - jepsen.util 345 :invoke :write [25 1] INFO [2022-03-13 17:08:53,117] jepsen worker 13 - jepsen.util 463 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,117] jepsen worker 13 - jepsen.util 463 :invoke :write [4 41] INFO [2022-03-13 17:08:53,155] jepsen worker 69 - jepsen.util 609 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,156] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:08:53,179] jepsen worker 43 - jepsen.util 403 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,180] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:53,191] jepsen worker 51 - jepsen.util 411 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,192] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:53,206] jepsen worker 63 - jepsen.util 783 :ok :read [21 :nil] INFO [2022-03-13 17:08:53,206] jepsen worker 63 - jepsen.util 783 :invoke :write [21 39] INFO [2022-03-13 17:08:53,229] jepsen worker 24 - jepsen.util 474 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,230] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,230] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:08:53,230] jepsen worker 24 - jepsen.util 474 :invoke :write [8 25] INFO [2022-03-13 17:08:53,231] jepsen worker 8 - jepsen.util 638 :invoke :write [2 23] INFO [2022-03-13 17:08:53,233] jepsen worker 21 - jepsen.util 651 :invoke :write [7 9] INFO [2022-03-13 17:08:53,295] jepsen worker 23 - jepsen.util 473 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,296] jepsen worker 23 - jepsen.util 473 :invoke :write [7 45] INFO [2022-03-13 17:08:53,299] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,299] jepsen worker 13 - jepsen.util 463 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,299] jepsen worker 51 - jepsen.util 411 :invoke :write [17 16] INFO [2022-03-13 17:08:53,306] jepsen worker 70 - jepsen.util 700 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,307] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:53,310] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:53,312] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:08:53,312] jepsen worker 76 - jepsen.util 526 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,312] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:53,313] jepsen worker 12 - jepsen.util 642 :invoke :write [4 28] INFO [2022-03-13 17:08:53,323] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:53,325] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:53,344] jepsen worker 25 - jepsen.util 385 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,344] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,345] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,345] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,345] jepsen worker 52 - jepsen.util 502 :invoke :write [17 8] INFO [2022-03-13 17:08:53,346] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:53,346] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:53,347] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:53,369] jepsen worker 6 - jepsen.util 546 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,370] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:53,370] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,370] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:53,371] jepsen worker 6 - jepsen.util 546 :invoke :write [2 36] INFO [2022-03-13 17:08:53,375] jepsen worker 22 - jepsen.util 472 :invoke :write [7 4] INFO [2022-03-13 17:08:53,376] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:53,376] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:53,404] jepsen worker 24 - jepsen.util 474 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,405] jepsen worker 24 - jepsen.util 474 :invoke :write [8 40] INFO [2022-03-13 17:08:53,423] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.09677419354838712}, :ns_1@172.23.105.187 {:progress 0.1169590643274854}, :ns_1@172.23.107.227 {:progress 0.09677419354838712}, :ns_1@172.23.107.228 {:progress 0.09411764705882353}} INFO [2022-03-13 17:08:53,423] jepsen worker nemesis - couchbase.util Rebalance check count: 46 INFO [2022-03-13 17:08:53,423] jepsen worker nemesis - couchbase.util Rebalance stuck count: 26 INFO [2022-03-13 17:08:53,426] jepsen worker 77 - jepsen.util 527 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,427] jepsen worker 77 - jepsen.util 527 :invoke :write [25 47] INFO [2022-03-13 17:08:53,438] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,439] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:08:53,472] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:53,472] jepsen worker 51 - jepsen.util 411 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,474] jepsen worker 13 - jepsen.util 463 :invoke :write [4 14] INFO [2022-03-13 17:08:53,475] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:53,480] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,481] jepsen worker 51 - jepsen.util 411 :invoke :write [17 43] INFO [2022-03-13 17:08:53,496] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,496] jepsen worker 34 - jepsen.util 574 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,497] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,497] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:53,497] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:53,498] jepsen worker 34 - jepsen.util 574 :invoke :write [11 24] INFO [2022-03-13 17:08:53,498] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:53,499] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:53,550] jepsen worker 23 - jepsen.util 473 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,551] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:53,585] jepsen worker 13 - jepsen.util 463 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,586] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:53,626] jepsen worker 38 - jepsen.util 848 :info :write [12 37] :RequestTimeoutException INFO [2022-03-13 17:08:53,627] jepsen worker 38 - jepsen.util 938 :invoke :write [12 30] INFO [2022-03-13 17:08:53,640] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:53,640] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,641] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:08:53,642] jepsen worker 23 - jepsen.util 473 :invoke :write [7 10] INFO [2022-03-13 17:08:53,643] jepsen worker 8 - jepsen.util 638 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,644] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:53,646] jepsen worker 65 - jepsen.util 785 :info :write [21 22] :RequestTimeoutException INFO [2022-03-13 17:08:53,646] jepsen worker 65 - jepsen.util 875 :invoke :read [21 nil] INFO [2022-03-13 17:08:53,656] jepsen worker 51 - jepsen.util 411 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,656] jepsen worker 51 - jepsen.util 411 :invoke :write [17 13] INFO [2022-03-13 17:08:53,656] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:53,657] jepsen worker 13 - jepsen.util 463 :invoke :write [4 29] INFO [2022-03-13 17:08:53,661] jepsen worker 82 - jepsen.util 802 :info :write [27 11] :RequestTimeoutException INFO [2022-03-13 17:08:53,662] jepsen worker 82 - jepsen.util 892 :invoke :read [27 nil] INFO [2022-03-13 17:08:53,668] jepsen worker 61 - jepsen.util 421 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,668] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,669] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:53,669] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:53,737] jepsen worker 6 - jepsen.util 546 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,738] jepsen worker 6 - jepsen.util 546 :invoke :write [2 12] INFO [2022-03-13 17:08:53,742] jepsen worker 9 - jepsen.util 729 :ok :read [3 27] INFO [2022-03-13 17:08:53,743] jepsen worker 9 - jepsen.util 729 :invoke :read [3 nil] INFO [2022-03-13 17:08:53,776] jepsen worker 13 - jepsen.util 463 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,777] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:53,783] jepsen worker 26 - jepsen.util 476 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,783] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:53,788] jepsen worker 56 - jepsen.util 686 :ok :read [18 16] INFO [2022-03-13 17:08:53,789] jepsen worker 56 - jepsen.util 686 :invoke :write [18 44] INFO [2022-03-13 17:08:53,829] jepsen worker 23 - jepsen.util 473 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,830] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:53,840] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,841] jepsen worker 21 - jepsen.util 651 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,841] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,841] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:53,841] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:53,842] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:53,843] jepsen worker 25 - jepsen.util 385 :invoke :write [8 10] INFO [2022-03-13 17:08:53,844] jepsen worker 21 - jepsen.util 651 :invoke :write [7 36] INFO [2022-03-13 17:08:53,860] jepsen worker 51 - jepsen.util 411 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,860] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:53,860] jepsen worker 51 - jepsen.util 411 :invoke :write [17 46] INFO [2022-03-13 17:08:53,861] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:53,908] jepsen worker 22 - jepsen.util 472 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,908] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,909] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:53,908] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,909] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:53,910] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:08:53,915] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:53,915] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:53,916] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:53,916] jepsen worker 13 - jepsen.util 463 :invoke :write [4 10] INFO [2022-03-13 17:08:53,924] jepsen worker 79 - jepsen.util 709 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,925] jepsen worker 79 - jepsen.util 709 :invoke :write [26 17] INFO [2022-03-13 17:08:53,958] jepsen worker 89 - jepsen.util 809 :info :write [29 37] :RequestTimeoutException INFO [2022-03-13 17:08:53,960] jepsen worker 89 - jepsen.util 899 :invoke :write [29 34] INFO [2022-03-13 17:08:53,967] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:53,967] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:08:53,968] jepsen worker 23 - jepsen.util 473 :invoke :write [7 13] INFO [2022-03-13 17:08:53,969] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:08:53,990] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:08:53,990] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:54,008] jepsen worker 52 - jepsen.util 502 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,009] jepsen worker 52 - jepsen.util 502 :invoke :write [17 45] INFO [2022-03-13 17:08:54,022] jepsen worker 51 - jepsen.util 411 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,023] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:54,041] jepsen worker 14 - jepsen.util 194 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,043] jepsen worker 14 - jepsen.util 194 :invoke :write [4 8] INFO [2022-03-13 17:08:54,081] jepsen worker 34 - jepsen.util 574 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,081] jepsen worker 34 - jepsen.util 574 :invoke :write [11 19] INFO [2022-03-13 17:08:54,089] jepsen worker 23 - jepsen.util 473 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,090] jepsen worker 23 - jepsen.util 473 :invoke :write [7 46] INFO [2022-03-13 17:08:54,156] jepsen worker 7 - jepsen.util 367 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,157] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:54,157] jepsen worker 7 - jepsen.util 367 :invoke :write [2 17] INFO [2022-03-13 17:08:54,160] jepsen worker 43 - jepsen.util 403 :invoke :write [14 26] INFO [2022-03-13 17:08:54,171] jepsen worker 13 - jepsen.util 463 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,171] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,172] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:54,173] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,173] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:54,173] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:54,174] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:54,175] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,175] jepsen worker 13 - jepsen.util 463 :invoke :write [4 21] INFO [2022-03-13 17:08:54,176] jepsen worker 51 - jepsen.util 411 :invoke :write [17 48] INFO [2022-03-13 17:08:54,184] jepsen worker 62 - jepsen.util 422 :ok :read [20 12] INFO [2022-03-13 17:08:54,184] jepsen worker 62 - jepsen.util 422 :invoke :write [20 27] INFO [2022-03-13 17:08:54,196] jepsen worker 73 - jepsen.util 793 :info :write [24 11] :SyncWriteAmbiguous INFO [2022-03-13 17:08:54,197] jepsen worker 73 - jepsen.util 883 :invoke :read [24 nil] INFO [2022-03-13 17:08:54,208] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,208] jepsen worker 24 - jepsen.util 474 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,208] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,208] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,209] jepsen worker 8 - jepsen.util 638 :invoke :write [2 14] INFO [2022-03-13 17:08:54,210] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:54,210] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:54,211] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:54,223] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:08:54,223] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:08:54,236] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,237] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:54,265] jepsen worker 6 - jepsen.util 546 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,265] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:08:54,265] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,265] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:54,266] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:54,267] jepsen worker 71 - jepsen.util 431 :invoke :write [23 2] INFO [2022-03-13 17:08:54,267] jepsen worker 22 - jepsen.util 472 :invoke :write [7 26] INFO [2022-03-13 17:08:54,268] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:54,268] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,268] jepsen worker 75 - jepsen.util 345 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,269] jepsen worker 76 - jepsen.util 526 :invoke :write [25 3] INFO [2022-03-13 17:08:54,270] jepsen worker 75 - jepsen.util 345 :invoke :write [25 1] INFO [2022-03-13 17:08:54,321] jepsen worker 51 - jepsen.util 411 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,321] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:54,369] jepsen worker 77 - jepsen.util 527 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,370] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:54,394] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:54,395] jepsen worker 60 - jepsen.util 600 :invoke :write [20 33] INFO [2022-03-13 17:08:54,412] jepsen worker 25 - jepsen.util 385 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,412] jepsen worker 25 - jepsen.util 385 :invoke :write [8 41] INFO [2022-03-13 17:08:54,415] jepsen worker 23 - jepsen.util 473 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,415] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,415] jepsen worker 23 - jepsen.util 473 :invoke :write [7 16] INFO [2022-03-13 17:08:54,416] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:08:54,431] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.09970674486803521}, :ns_1@172.23.105.187 {:progress 0.1169590643274854}, :ns_1@172.23.107.227 {:progress 0.09970674486803521}, :ns_1@172.23.107.228 {:progress 0.09999999999999998}} INFO [2022-03-13 17:08:54,431] jepsen worker nemesis - couchbase.util Rebalance check count: 47 INFO [2022-03-13 17:08:54,431] jepsen worker nemesis - couchbase.util Rebalance stuck count: 26 INFO [2022-03-13 17:08:54,590] jepsen worker 10 - jepsen.util 640 :info :write [3 13] :RequestTimeoutException INFO [2022-03-13 17:08:54,591] jepsen worker 10 - jepsen.util 730 :invoke :read [3 nil] INFO [2022-03-13 17:08:54,884] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,885] jepsen worker 13 - jepsen.util 463 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,885] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:54,886] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:54,893] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:54,893] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,894] jepsen worker 13 - jepsen.util 463 :invoke :write [4 33] INFO [2022-03-13 17:08:54,895] jepsen worker 51 - jepsen.util 411 :invoke :write [17 27] INFO [2022-03-13 17:08:54,942] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,943] jepsen worker 70 - jepsen.util 700 :invoke :write [23 13] INFO [2022-03-13 17:08:54,971] jepsen worker 23 - jepsen.util 473 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:54,972] jepsen worker 23 - jepsen.util 473 :invoke :write [7 22] INFO [2022-03-13 17:08:55,000] jepsen worker 13 - jepsen.util 463 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,000] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:55,031] jepsen worker 79 - jepsen.util 709 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,031] jepsen worker 12 - jepsen.util 642 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,032] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:55,033] jepsen worker 12 - jepsen.util 642 :invoke :write [4 48] INFO [2022-03-13 17:08:55,080] jepsen worker 21 - jepsen.util 651 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,081] jepsen worker 64 - jepsen.util 514 :info :write [21 10] :RequestTimeoutException INFO [2022-03-13 17:08:55,081] jepsen worker 21 - jepsen.util 651 :invoke :write [7 30] INFO [2022-03-13 17:08:55,081] jepsen worker 64 - jepsen.util 604 :invoke :read [21 nil] INFO [2022-03-13 17:08:55,089] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:08:55,091] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:55,092] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,092] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:55,126] jepsen worker 51 - jepsen.util 411 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,126] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:55,127] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:55,128] jepsen worker 13 - jepsen.util 463 :invoke :write [4 3] INFO [2022-03-13 17:08:55,128] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,129] jepsen worker 51 - jepsen.util 411 :invoke :write [17 7] INFO [2022-03-13 17:08:55,176] jepsen worker 23 - jepsen.util 473 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,176] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,176] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,177] jepsen worker 34 - jepsen.util 574 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,177] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:55,178] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:55,179] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:55,180] jepsen worker 44 - jepsen.util 494 :invoke :write [14 12] INFO [2022-03-13 17:08:55,184] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,187] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:55,188] jepsen worker 23 - jepsen.util 473 :invoke :write [7 10] INFO [2022-03-13 17:08:55,189] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,189] jepsen worker 34 - jepsen.util 574 :invoke :write [11 33] INFO [2022-03-13 17:08:55,195] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:08:55,216] jepsen worker 53 - jepsen.util 503 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,217] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:08:55,224] jepsen worker 0 - jepsen.util 720 :info :write [0 35] :RequestTimeoutException INFO [2022-03-13 17:08:55,225] jepsen worker 0 - jepsen.util 810 :invoke :write [0 26] INFO [2022-03-13 17:08:55,265] jepsen worker 51 - jepsen.util 411 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,266] jepsen worker 7 - jepsen.util 367 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,267] jepsen worker 51 - jepsen.util 411 :invoke :write [17 14] INFO [2022-03-13 17:08:55,267] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:55,276] jepsen worker 52 - jepsen.util 502 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,276] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:08:55,276] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,276] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,277] jepsen worker 24 - jepsen.util 474 :invoke :write [8 16] INFO [2022-03-13 17:08:55,278] jepsen worker 52 - jepsen.util 502 :invoke :write [17 40] INFO [2022-03-13 17:08:55,279] jepsen worker 78 - jepsen.util 438 :invoke :write [26 25] INFO [2022-03-13 17:08:55,279] jepsen worker 80 - jepsen.util 620 :invoke :write [26 21] INFO [2022-03-13 17:08:55,286] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:55,287] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:55,307] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:55,308] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:55,351] jepsen worker 22 - jepsen.util 472 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,352] jepsen worker 22 - jepsen.util 472 :invoke :write [7 6] WARN [2022-03-13 17:08:55,386] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15862287,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":15862287,"total_server_duration_us":32,"operation_id":"0x1a37","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":15863625},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17582,"operation_id":"0x1b1f","timeout_ms":10000,"total_duration_us":10008154},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":18588,"operation_id":"0x1bc3","timeout_ms":10000,"total_duration_us":10007647},{"operation_name":"upsert","last_dispatch_duration_us":7009741,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7297577,"total_server_duration_us":109266,"encode_duration_us":17950,"operation_id":"0x1c57","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10007430},{"operation_name":"upsert","last_dispatch_duration_us":41484,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":468868,"total_server_duration_us":260774,"encode_duration_us":21382,"operation_id":"0x1cff","timeout_ms":10000,"last_server_duration_us":24325,"total_duration_us":10006270},{"operation_name":"upsert","last_dispatch_duration_us":14709783,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":14709783,"total_server_duration_us":9042674,"encode_duration_us":17239,"operation_id":"0x1b92","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10004274},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17231,"operation_id":"0x1ccc","timeout_ms":10000,"total_duration_us":10001454},{"operation_name":"upsert","last_dispatch_duration_us":9094957,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9094957,"total_server_duration_us":9032060,"encode_duration_us":21186,"operation_id":"0x1d00","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":9100278},{"operation_name":"upsert","last_dispatch_duration_us":131620,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1436411,"total_server_duration_us":513342,"encode_duration_us":17151,"operation_id":"0x1cd6","timeout_ms":10000,"last_server_duration_us":63182,"total_duration_us":1649911},{"operation_name":"upsert","last_dispatch_duration_us":100663,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1344574,"total_server_duration_us":542715,"encode_duration_us":17392,"operation_id":"0x1cd7","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":1475997}],"total_count":14}} WARN [2022-03-13 17:08:55,386] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17321403,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17769681,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9266883,"encode_duration_us":23979,"operation_id":"0x199f","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10001019},{"operation_name":"upsert","last_dispatch_duration_us":14507553,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14631020,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":6292629,"encode_duration_us":30444,"operation_id":"0x18cc","timeout_ms":10000,"last_server_duration_us":6240689,"total_duration_us":10008043},{"operation_name":"upsert","last_dispatch_duration_us":14709783,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14709783,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9042674,"encode_duration_us":17239,"operation_id":"0x1b92","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10004274}],"total_count":3}} INFO [2022-03-13 17:08:55,403] jepsen worker 23 - jepsen.util 473 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,403] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:55,423] jepsen worker 42 - jepsen.util 402 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,424] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:08:55,428] jepsen worker 13 - jepsen.util 463 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,428] jepsen worker 13 - jepsen.util 463 :invoke :write [4 5] INFO [2022-03-13 17:08:55,436] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.09970674486803521}, :ns_1@172.23.105.187 {:progress 0.1169590643274854}, :ns_1@172.23.107.227 {:progress 0.09970674486803521}, :ns_1@172.23.107.228 {:progress 0.09999999999999998}} INFO [2022-03-13 17:08:55,436] jepsen worker nemesis - couchbase.util Rebalance check count: 48 INFO [2022-03-13 17:08:55,436] jepsen worker nemesis - couchbase.util Rebalance stuck count: 27 INFO [2022-03-13 17:08:55,532] jepsen worker 8 - jepsen.util 638 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,533] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:55,545] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,545] jepsen worker 43 - jepsen.util 403 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,546] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:55,547] jepsen worker 43 - jepsen.util 403 :invoke :write [14 8] INFO [2022-03-13 17:08:55,561] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:55,561] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,561] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:08:55,562] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:55,564] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:55,564] jepsen worker 23 - jepsen.util 473 :invoke :write [7 37] INFO [2022-03-13 17:08:55,576] jepsen worker 85 - jepsen.util 895 :info :write [28 13] :RequestTimeoutException INFO [2022-03-13 17:08:55,577] jepsen worker 85 - jepsen.util 985 :invoke :read [28 nil] INFO [2022-03-13 17:08:55,584] jepsen worker 51 - jepsen.util 411 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,585] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:55,600] jepsen worker 72 - jepsen.util 792 :info :write [24 44] :RequestTimeoutException INFO [2022-03-13 17:08:55,601] jepsen worker 72 - jepsen.util 882 :invoke :read [24 nil] INFO [2022-03-13 17:08:55,645] jepsen worker 18 - jepsen.util 558 :info :write [6 13] :RequestTimeoutException INFO [2022-03-13 17:08:55,647] jepsen worker 18 - jepsen.util 648 :invoke :write [6 48] INFO [2022-03-13 17:08:55,670] jepsen worker 13 - jepsen.util 463 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,670] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,670] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:55,671] jepsen worker 51 - jepsen.util 411 :invoke :write [17 46] INFO [2022-03-13 17:08:55,672] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:55,672] jepsen worker 13 - jepsen.util 463 :invoke :write [4 19] INFO [2022-03-13 17:08:55,724] jepsen worker 71 - jepsen.util 431 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,725] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:55,753] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,753] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:55,808] jepsen worker 14 - jepsen.util 194 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,809] jepsen worker 14 - jepsen.util 194 :invoke :write [4 42] INFO [2022-03-13 17:08:55,852] jepsen worker 25 - jepsen.util 385 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,853] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:55,869] jepsen worker 51 - jepsen.util 411 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,870] jepsen worker 51 - jepsen.util 411 :invoke :write [17 42] INFO [2022-03-13 17:08:55,888] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,888] jepsen worker 23 - jepsen.util 473 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,888] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:55,889] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:55,890] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:55,890] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,890] jepsen worker 23 - jepsen.util 473 :invoke :write [7 0] INFO [2022-03-13 17:08:55,891] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:08:55,899] jepsen worker 60 - jepsen.util 600 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,899] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:55,901] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,901] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:55,920] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,920] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,921] jepsen worker 7 - jepsen.util 367 :invoke :write [2 3] INFO [2022-03-13 17:08:55,921] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:08:55,933] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:08:55,934] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:55,950] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,950] jepsen worker 70 - jepsen.util 700 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:08:55,951] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:55,951] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:08:56,443] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1055718475073314}, :ns_1@172.23.105.187 {:progress 0.1228070175438597}, :ns_1@172.23.107.227 {:progress 0.1055718475073314}, :ns_1@172.23.107.228 {:progress 0.1058823529411764}} INFO [2022-03-13 17:08:56,443] jepsen worker nemesis - couchbase.util Rebalance check count: 49 INFO [2022-03-13 17:08:56,448] jepsen worker nemesis - couchbase.util Rebalance stuck count: 27 INFO [2022-03-13 17:08:56,455] jepsen worker 13 - jepsen.util 463 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,455] jepsen worker 13 - jepsen.util 463 :invoke :write [4 32] INFO [2022-03-13 17:08:56,462] jepsen worker 23 - jepsen.util 473 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,463] jepsen worker 23 - jepsen.util 473 :invoke :write [7 30] INFO [2022-03-13 17:08:56,528] jepsen worker 21 - jepsen.util 651 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,529] jepsen worker 21 - jepsen.util 651 :invoke :write [7 12] INFO [2022-03-13 17:08:56,598] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,599] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:08:56,615] jepsen worker 44 - jepsen.util 494 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,615] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:08:56,626] jepsen worker 62 - jepsen.util 422 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,626] jepsen worker 43 - jepsen.util 403 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,626] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:08:56,627] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:08:56,647] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,648] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,648] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:56,649] jepsen worker 77 - jepsen.util 527 :invoke :write [25 28] INFO [2022-03-13 17:08:56,657] jepsen worker 51 - jepsen.util 411 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,658] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:56,666] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:56,667] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:56,679] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:08:56,680] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:08:56,725] jepsen worker 23 - jepsen.util 473 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,726] jepsen worker 23 - jepsen.util 473 :invoke :write [7 41] INFO [2022-03-13 17:08:56,767] jepsen worker 52 - jepsen.util 502 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,768] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:08:56,790] jepsen worker 20 - jepsen.util 920 :info :write [6 12] :RequestTimeoutException INFO [2022-03-13 17:08:56,791] jepsen worker 20 - jepsen.util 1010 :invoke :write [6 24] INFO [2022-03-13 17:08:56,804] jepsen worker 28 - jepsen.util 838 :info :write [9 7] :RequestTimeoutException INFO [2022-03-13 17:08:56,805] jepsen worker 28 - jepsen.util 928 :invoke :write [9 11] INFO [2022-03-13 17:08:56,811] jepsen worker 13 - jepsen.util 463 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,811] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,814] jepsen worker 13 - jepsen.util 463 :invoke :write [4 10] INFO [2022-03-13 17:08:56,815] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:56,821] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,821] jepsen worker 51 - jepsen.util 411 :invoke :write [17 10] INFO [2022-03-13 17:08:56,824] jepsen worker 76 - jepsen.util 526 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,826] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:56,853] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,854] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:56,867] jepsen worker 7 - jepsen.util 367 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,867] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:08:56,867] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,867] jepsen worker 7 - jepsen.util 367 :invoke :write [2 43] INFO [2022-03-13 17:08:56,868] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:56,869] jepsen worker 43 - jepsen.util 403 :invoke :write [14 31] WARN [2022-03-13 17:08:56,897] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17303171,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17990183,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9127200,"encode_duration_us":27022,"operation_id":"0x1a1e","timeout_ms":10000,"last_server_duration_us":8988600,"total_duration_us":10001652},{"operation_name":"upsert","last_dispatch_duration_us":9065950,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9065950,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9044797,"encode_duration_us":19640,"operation_id":"0x1a4c","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10001685},{"operation_name":"upsert","last_dispatch_duration_us":21204340,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21204340,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9033121,"encode_duration_us":29337,"operation_id":"0x1942","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10002954},{"operation_name":"upsert","last_dispatch_duration_us":21180719,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21180719,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9033121,"encode_duration_us":29679,"operation_id":"0x1940","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10010878}],"total_count":4}} INFO [2022-03-13 17:08:56,899] jepsen worker 13 - jepsen.util 463 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,899] jepsen worker 13 - jepsen.util 463 :invoke :write [4 6] INFO [2022-03-13 17:08:56,903] jepsen worker 88 - jepsen.util 898 :info :write [29 48] :RequestTimeoutException INFO [2022-03-13 17:08:56,904] jepsen worker 88 - jepsen.util 988 :invoke :read [29 nil] INFO [2022-03-13 17:08:56,946] jepsen worker 75 - jepsen.util 345 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,947] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:08:56,953] jepsen worker 23 - jepsen.util 473 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,953] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,954] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:56,954] jepsen worker 69 - jepsen.util 609 :invoke :write [23 43] INFO [2022-03-13 17:08:56,958] jepsen worker 51 - jepsen.util 411 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:08:56,959] jepsen worker 51 - jepsen.util 411 :invoke :write [17 34] INFO [2022-03-13 17:08:56,963] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:56,964] jepsen worker 23 - jepsen.util 473 :invoke :write [7 12] WARN [2022-03-13 17:08:56,987] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9887988,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9887988,"total_server_duration_us":15,"operation_id":"0x1b85","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":11995082},{"operation_name":"get","last_dispatch_duration_us":10079666,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10079666,"total_server_duration_us":56,"operation_id":"0x1bbd","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":10647435},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18297,"operation_id":"0x1d32","timeout_ms":10000,"total_duration_us":10009560},{"operation_name":"upsert","last_dispatch_duration_us":54273,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1079115,"total_server_duration_us":622378,"encode_duration_us":18012,"operation_id":"0x1cca","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10006890},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18106,"operation_id":"0x1b5d","timeout_ms":10000,"total_duration_us":10005552},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":21911,"operation_id":"0x1c4e","timeout_ms":10000,"total_duration_us":10005230},{"operation_name":"upsert","last_dispatch_duration_us":17485291,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":17485291,"total_server_duration_us":25625,"encode_duration_us":17257,"operation_id":"0x1b1e","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10001125},{"operation_name":"upsert","last_dispatch_duration_us":270383,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1143364,"total_server_duration_us":523746,"encode_duration_us":21440,"operation_id":"0x1e6e","timeout_ms":10000,"last_server_duration_us":129414,"total_duration_us":1198675},{"operation_name":"upsert","last_dispatch_duration_us":221264,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":916751,"total_server_duration_us":303033,"encode_duration_us":32925,"operation_id":"0x1e73","timeout_ms":10000,"last_server_duration_us":88098,"total_duration_us":951716},{"operation_name":"upsert","last_dispatch_duration_us":675156,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":675156,"total_server_duration_us":84138,"encode_duration_us":22868,"operation_id":"0x20f0","timeout_ms":10000,"last_server_duration_us":84138,"total_duration_us":683165}],"total_count":13}} WARN [2022-03-13 17:08:56,987] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18358941,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18358941,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9027816,"encode_duration_us":18363,"operation_id":"0x1ac8","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10002430},{"operation_name":"upsert","last_dispatch_duration_us":18652288,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26397134,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9328852,"encode_duration_us":17583,"operation_id":"0x178f","timeout_ms":10000,"last_server_duration_us":9072420,"total_duration_us":10005735}],"total_count":2}} WARN [2022-03-13 17:08:56,997] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18963652,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18963652,"total_server_duration_us":27,"operation_id":"0x199e","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":18966207},{"operation_name":"get","last_dispatch_duration_us":197674,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":197674,"total_server_duration_us":43,"operation_id":"0x1a23","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":17604779},{"operation_name":"get","last_dispatch_duration_us":9272399,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9272399,"total_server_duration_us":56,"operation_id":"0x1bba","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":10357521},{"operation_name":"upsert","last_dispatch_duration_us":166460,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9632657,"total_server_duration_us":79231,"encode_duration_us":17724,"operation_id":"0x1bfd","timeout_ms":10000,"last_server_duration_us":28768,"total_duration_us":10010502},{"operation_name":"upsert","last_dispatch_duration_us":183751,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":183751,"total_server_duration_us":45390,"encode_duration_us":22041,"operation_id":"0x1cd8","timeout_ms":10000,"last_server_duration_us":45390,"total_duration_us":10007534},{"operation_name":"upsert","last_dispatch_duration_us":151138,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9586053,"total_server_duration_us":69332,"encode_duration_us":20753,"operation_id":"0x1c00","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10005553},{"operation_name":"upsert","last_dispatch_duration_us":35446,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8957422,"total_server_duration_us":202152,"encode_duration_us":17712,"operation_id":"0x1c55","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10002594},{"operation_name":"upsert","last_dispatch_duration_us":50543,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8465413,"total_server_duration_us":133669,"encode_duration_us":16975,"operation_id":"0x1c9b","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10001805},{"operation_name":"get","last_dispatch_duration_us":170573,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":170573,"total_server_duration_us":32,"operation_id":"0x1c05","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9420827},{"operation_name":"get","last_dispatch_duration_us":8248700,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8248700,"total_server_duration_us":49,"operation_id":"0x1c8c","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8249912}],"total_count":68}} INFO [2022-03-13 17:08:57,003] jepsen worker 13 - jepsen.util 463 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,004] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:57,009] jepsen worker 37 - jepsen.util 667 :info :write [12 30] :RequestTimeoutException INFO [2022-03-13 17:08:57,010] jepsen worker 37 - jepsen.util 757 :invoke :read [12 nil] INFO [2022-03-13 17:08:57,013] jepsen worker 77 - jepsen.util 527 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,014] jepsen worker 77 - jepsen.util 527 :invoke :write [25 29] INFO [2022-03-13 17:08:57,047] jepsen worker 37 - jepsen.util 757 :ok :read [12 2] INFO [2022-03-13 17:08:57,047] jepsen worker 37 - jepsen.util 757 :invoke :write [12 18] INFO [2022-03-13 17:08:57,055] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:57,056] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:57,066] jepsen worker 69 - jepsen.util 609 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,067] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:08:57,069] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,070] jepsen worker 79 - jepsen.util 709 :invoke :write [26 24] INFO [2022-03-13 17:08:57,083] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:08:57,084] jepsen worker 33 - jepsen.util 573 :invoke :write [11 9] WARN [2022-03-13 17:08:57,085] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":19746933,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":19746933,"total_server_duration_us":32,"operation_id":"0x18d0","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":19749454},{"operation_name":"get","last_dispatch_duration_us":18222520,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":18222520,"total_server_duration_us":32,"operation_id":"0x1a8f","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":18222857},{"operation_name":"get","last_dispatch_duration_us":16547,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":14026447,"total_server_duration_us":188,"operation_id":"0x1a2d","timeout_ms":30000,"last_server_duration_us":145,"total_duration_us":14040029},{"operation_name":"upsert","last_dispatch_duration_us":10556828,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10556828,"total_server_duration_us":37859,"encode_duration_us":17257,"operation_id":"0x1cec","timeout_ms":10000,"last_server_duration_us":37859,"total_duration_us":10011042},{"operation_name":"upsert","last_dispatch_duration_us":4433249,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10891531,"total_server_duration_us":233305,"encode_duration_us":20406,"operation_id":"0x1c4c","timeout_ms":10000,"last_server_duration_us":33918,"total_duration_us":10009651},{"operation_name":"upsert","last_dispatch_duration_us":4172029,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4485726,"total_server_duration_us":78328,"encode_duration_us":17554,"operation_id":"0x1c9d","timeout_ms":10000,"last_server_duration_us":34116,"total_duration_us":10006767},{"operation_name":"upsert","last_dispatch_duration_us":17514139,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":17514139,"total_server_duration_us":50184,"encode_duration_us":20308,"operation_id":"0x1b38","timeout_ms":10000,"last_server_duration_us":50184,"total_duration_us":10001625},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":18543,"operation_id":"0x1d90","timeout_ms":10000,"total_duration_us":10001201},{"operation_name":"upsert","last_dispatch_duration_us":3949235,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5489804,"total_server_duration_us":206628,"encode_duration_us":27446,"operation_id":"0x1c7b","timeout_ms":10000,"last_server_duration_us":138081,"total_duration_us":8109307},{"operation_name":"get","last_dispatch_duration_us":7901971,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7901971,"total_server_duration_us":27,"operation_id":"0x1bbb","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":7904095}],"total_count":47}} WARN [2022-03-13 17:08:57,085] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19588945,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19588945,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9136301,"encode_duration_us":18733,"operation_id":"0x1a54","timeout_ms":10000,"last_server_duration_us":9136301,"total_duration_us":10009964}],"total_count":1}} WARN [2022-03-13 17:08:57,087] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17152772,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17861769,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9213546,"encode_duration_us":22747,"operation_id":"0x1b4d","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10002730},{"operation_name":"upsert","last_dispatch_duration_us":9072443,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9072443,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9025695,"encode_duration_us":18717,"operation_id":"0x1d02","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009342},{"operation_name":"upsert","last_dispatch_duration_us":17176870,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25416089,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9249434,"encode_duration_us":19330,"operation_id":"0x1971","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10006844},{"operation_name":"upsert","last_dispatch_duration_us":17142679,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17787745,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9171454,"encode_duration_us":32257,"operation_id":"0x1b57","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10010885}],"total_count":4}} INFO [2022-03-13 17:08:57,113] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:57,113] jepsen worker 51 - jepsen.util 411 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,114] jepsen worker 51 - jepsen.util 411 :invoke :write [17 22] INFO [2022-03-13 17:08:57,115] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:57,116] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:57,117] jepsen worker 13 - jepsen.util 463 :invoke :write [4 19] INFO [2022-03-13 17:08:57,125] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:57,125] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,125] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:08:57,126] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:57,132] jepsen worker 23 - jepsen.util 473 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,132] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] WARN [2022-03-13 17:08:57,187] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":19524750,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":19524750,"total_server_duration_us":38,"operation_id":"0x1a64","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":19525814},{"operation_name":"get","last_dispatch_duration_us":17706157,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":17706157,"total_server_duration_us":32,"operation_id":"0x1ac2","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":17707304},{"operation_name":"get","last_dispatch_duration_us":14038078,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":14038078,"total_server_duration_us":32,"operation_id":"0x1a91","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":14041342},{"operation_name":"get","last_dispatch_duration_us":12088594,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12088594,"total_server_duration_us":15,"operation_id":"0x1b1d","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":12090189},{"operation_name":"get","last_dispatch_duration_us":10491919,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10491919,"total_server_duration_us":27,"operation_id":"0x1b86","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":10492282},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":17924,"operation_id":"0x1ccb","timeout_ms":10000,"total_duration_us":10006327},{"operation_name":"upsert","last_dispatch_duration_us":133153,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4797074,"total_server_duration_us":222200,"encode_duration_us":19536,"operation_id":"0x1d23","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10005590},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":16943,"operation_id":"0x1c56","timeout_ms":10000,"total_duration_us":10002159},{"operation_name":"get","last_dispatch_duration_us":8930904,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8930904,"total_server_duration_us":32,"operation_id":"0x1bc6","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8933340},{"operation_name":"upsert","last_dispatch_duration_us":4088776,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6969040,"total_server_duration_us":137002,"encode_duration_us":18338,"operation_id":"0x1c3a","timeout_ms":10000,"last_server_duration_us":71096,"total_duration_us":6983460}],"total_count":48}} WARN [2022-03-13 17:08:57,187] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21515135,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21515135,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9026756,"encode_duration_us":21429,"operation_id":"0x19cf","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10001133},{"operation_name":"upsert","last_dispatch_duration_us":17116532,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18299291,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9561489,"encode_duration_us":17009,"operation_id":"0x17be","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10006351}],"total_count":2}} WARN [2022-03-13 17:08:57,188] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17142679,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17787745,"total_server_duration_us":9171454,"encode_duration_us":32257,"operation_id":"0x1b57","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10010885},{"operation_name":"upsert","last_dispatch_duration_us":9072443,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9072443,"total_server_duration_us":9025695,"encode_duration_us":18717,"operation_id":"0x1d02","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009342},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":16922,"operation_id":"0x1c9e","timeout_ms":10000,"total_duration_us":10007823},{"operation_name":"upsert","last_dispatch_duration_us":33601,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":8645155,"total_server_duration_us":319309,"encode_duration_us":18546,"operation_id":"0x1b83","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10006347},{"operation_name":"upsert","last_dispatch_duration_us":6670510,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7536761,"total_server_duration_us":424284,"encode_duration_us":18388,"operation_id":"0x1b56","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10003024},{"operation_name":"upsert","last_dispatch_duration_us":17152772,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17861769,"total_server_duration_us":9213546,"encode_duration_us":22747,"operation_id":"0x1b4d","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10002730},{"operation_name":"upsert","last_dispatch_duration_us":31953,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":338879,"total_server_duration_us":205984,"encode_duration_us":18103,"operation_id":"0x1d9c","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10002166},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":20608,"operation_id":"0x1d96","timeout_ms":10000,"total_duration_us":10001153},{"operation_name":"upsert","last_dispatch_duration_us":911461,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4293278,"total_server_duration_us":126378,"encode_duration_us":18999,"operation_id":"0x1b87","timeout_ms":10000,"last_server_duration_us":53015,"total_duration_us":8445312},{"operation_name":"upsert","last_dispatch_duration_us":7596053,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7596053,"total_server_duration_us":7576268,"encode_duration_us":21394,"operation_id":"0x1c4d","timeout_ms":10000,"last_server_duration_us":7576268,"total_duration_us":7599616}],"total_count":69}} INFO [2022-03-13 17:08:57,201] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:57,201] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,201] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:57,202] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:57,206] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:57,206] jepsen worker 7 - jepsen.util 367 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,206] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:57,207] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,207] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:08:57,207] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:57,210] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:57,210] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,210] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:57,211] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:57,212] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:57,212] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,213] jepsen worker 23 - jepsen.util 473 :invoke :write [7 18] INFO [2022-03-13 17:08:57,214] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:57,216] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,216] jepsen worker 55 - jepsen.util 865 :info :write [18 27] :RequestTimeoutException INFO [2022-03-13 17:08:57,217] jepsen worker 55 - jepsen.util 955 :invoke :read [18 nil] INFO [2022-03-13 17:08:57,218] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:57,219] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,220] jepsen worker 69 - jepsen.util 609 :invoke :write [23 36] INFO [2022-03-13 17:08:57,244] jepsen worker 51 - jepsen.util 411 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,244] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:57,256] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:57,256] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:57,287] jepsen worker 12 - jepsen.util 642 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,288] jepsen worker 12 - jepsen.util 642 :invoke :write [4 40] INFO [2022-03-13 17:08:57,307] jepsen worker 66 - jepsen.util 876 :info :write [22 24] :RequestTimeoutException INFO [2022-03-13 17:08:57,307] jepsen worker 66 - jepsen.util 966 :invoke :write [22 22] INFO [2022-03-13 17:08:57,330] jepsen worker 43 - jepsen.util 403 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,331] jepsen worker 43 - jepsen.util 403 :invoke :write [14 16] INFO [2022-03-13 17:08:57,352] jepsen worker 23 - jepsen.util 473 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,353] jepsen worker 23 - jepsen.util 473 :invoke :write [7 17] INFO [2022-03-13 17:08:57,382] jepsen worker 13 - jepsen.util 463 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,382] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,382] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:57,383] jepsen worker 51 - jepsen.util 411 :invoke :write [17 40] INFO [2022-03-13 17:08:57,386] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:57,387] jepsen worker 13 - jepsen.util 463 :invoke :write [4 2] INFO [2022-03-13 17:08:57,389] jepsen worker 34 - jepsen.util 574 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,390] jepsen worker 34 - jepsen.util 574 :invoke :write [11 47] INFO [2022-03-13 17:08:57,411] jepsen worker 69 - jepsen.util 609 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,412] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:08:57,455] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1114369501466276}, :ns_1@172.23.105.187 {:progress 0.1345029239766082}, :ns_1@172.23.107.227 {:progress 0.1114369501466276}, :ns_1@172.23.107.228 {:progress 0.111764705882353}} INFO [2022-03-13 17:08:57,455] jepsen worker nemesis - couchbase.util Rebalance check count: 50 INFO [2022-03-13 17:08:57,455] jepsen worker nemesis - couchbase.util Rebalance stuck count: 27 INFO [2022-03-13 17:08:57,477] jepsen worker 23 - jepsen.util 473 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,477] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:08:57,526] jepsen worker 51 - jepsen.util 411 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,527] jepsen worker 51 - jepsen.util 411 :invoke :write [17 29] INFO [2022-03-13 17:08:57,538] jepsen worker 80 - jepsen.util 620 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,539] jepsen worker 80 - jepsen.util 620 :invoke :write [26 33] INFO [2022-03-13 17:08:57,544] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,544] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:08:57,545] jepsen worker 23 - jepsen.util 473 :invoke :write [7 34] INFO [2022-03-13 17:08:57,546] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] INFO [2022-03-13 17:08:57,589] jepsen worker 22 - jepsen.util 472 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,589] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,589] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,589] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,589] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:08:57,590] jepsen worker 71 - jepsen.util 431 :invoke :write [23 42] INFO [2022-03-13 17:08:57,591] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:08:57,591] jepsen worker 74 - jepsen.util 974 :info :write [24 31] :RequestTimeoutException INFO [2022-03-13 17:08:57,592] jepsen worker 22 - jepsen.util 472 :invoke :write [7 28] INFO [2022-03-13 17:08:57,593] jepsen worker 44 - jepsen.util 494 :invoke :write [14 29] INFO [2022-03-13 17:08:57,593] jepsen worker 74 - jepsen.util 1064 :invoke :read [24 nil] INFO [2022-03-13 17:08:57,593] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:08:57,606] jepsen worker 13 - jepsen.util 463 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,607] jepsen worker 13 - jepsen.util 463 :invoke :write [4 15] INFO [2022-03-13 17:08:57,663] jepsen worker 51 - jepsen.util 411 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,664] jepsen worker 51 - jepsen.util 411 :invoke :write [17 14] INFO [2022-03-13 17:08:57,672] jepsen worker 23 - jepsen.util 473 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,672] jepsen worker 23 - jepsen.util 473 :invoke :write [7 42] INFO [2022-03-13 17:08:57,717] jepsen worker 78 - jepsen.util 438 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,717] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:08:57,730] jepsen worker 13 - jepsen.util 463 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,731] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:57,752] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,753] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:08:57,859] jepsen worker 32 - jepsen.util 662 :info :write [10 31] :RequestTimeoutException INFO [2022-03-13 17:08:57,860] jepsen worker 32 - jepsen.util 752 :invoke :read [10 nil] INFO [2022-03-13 17:08:57,882] jepsen worker 34 - jepsen.util 574 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,883] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,884] jepsen worker 51 - jepsen.util 411 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:57,884] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:57,884] jepsen worker 6 - jepsen.util 546 :invoke :write [2 15] INFO [2022-03-13 17:08:57,885] jepsen worker 51 - jepsen.util 411 :invoke :write [17 46] INFO [2022-03-13 17:08:57,885] jepsen worker 34 - jepsen.util 574 :invoke :write [11 25] INFO [2022-03-13 17:08:57,888] jepsen worker 13 - jepsen.util 463 :invoke :write [4 14] INFO [2022-03-13 17:08:57,894] jepsen worker 81 - jepsen.util 981 :info :write [27 16] :RequestTimeoutException INFO [2022-03-13 17:08:57,895] jepsen worker 81 - jepsen.util 1071 :invoke :write [27 29] INFO [2022-03-13 17:08:57,911] jepsen worker 83 - jepsen.util 623 :info :write [27 25] :RequestTimeoutException INFO [2022-03-13 17:08:57,912] jepsen worker 83 - jepsen.util 713 :invoke :read [27 nil] INFO [2022-03-13 17:08:58,056] jepsen worker 27 - jepsen.util 837 :info :write [9 44] :RequestTimeoutException INFO [2022-03-13 17:08:58,056] jepsen worker 27 - jepsen.util 927 :invoke :write [9 2] INFO [2022-03-13 17:08:58,318] jepsen worker 16 - jepsen.util 736 :ok :read [5 43] INFO [2022-03-13 17:08:58,318] jepsen worker 16 - jepsen.util 736 :invoke :read [5 nil] INFO [2022-03-13 17:08:58,353] jepsen worker 86 - jepsen.util 806 :ok :read [28 44] INFO [2022-03-13 17:08:58,354] jepsen worker 86 - jepsen.util 806 :invoke :write [28 17] INFO [2022-03-13 17:08:58,367] jepsen worker 16 - jepsen.util 736 :ok :read [5 43] INFO [2022-03-13 17:08:58,367] jepsen worker 16 - jepsen.util 736 :invoke :read [5 nil] INFO [2022-03-13 17:08:58,371] jepsen worker 4 - jepsen.util 814 :ok :read [1 49] INFO [2022-03-13 17:08:58,371] jepsen worker 4 - jepsen.util 814 :invoke :read [1 nil] INFO [2022-03-13 17:08:58,395] jepsen worker 4 - jepsen.util 814 :ok :read [1 49] INFO [2022-03-13 17:08:58,396] jepsen worker 4 - jepsen.util 814 :invoke :write [1 6] INFO [2022-03-13 17:08:58,400] jepsen worker 16 - jepsen.util 736 :ok :read [5 43] INFO [2022-03-13 17:08:58,400] jepsen worker 16 - jepsen.util 736 :invoke :write [5 32] INFO [2022-03-13 17:08:58,461] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1114369501466276}, :ns_1@172.23.105.187 {:progress 0.1345029239766082}, :ns_1@172.23.107.227 {:progress 0.1114369501466276}, :ns_1@172.23.107.228 {:progress 0.111764705882353}} INFO [2022-03-13 17:08:58,461] jepsen worker nemesis - couchbase.util Rebalance check count: 51 INFO [2022-03-13 17:08:58,461] jepsen worker nemesis - couchbase.util Rebalance stuck count: 28 INFO [2022-03-13 17:08:58,569] jepsen worker 23 - jepsen.util 473 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,570] jepsen worker 23 - jepsen.util 473 :invoke :write [7 18] INFO [2022-03-13 17:08:58,622] jepsen worker 24 - jepsen.util 474 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,622] jepsen worker 24 - jepsen.util 474 :invoke :write [8 25] INFO [2022-03-13 17:08:58,651] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,651] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:08:58,654] jepsen worker 13 - jepsen.util 463 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,655] jepsen worker 13 - jepsen.util 463 :invoke :write [4 17] INFO [2022-03-13 17:08:58,687] jepsen worker 71 - jepsen.util 431 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,687] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:08:58,697] jepsen worker 45 - jepsen.util 1035 :ok :read [15 48] INFO [2022-03-13 17:08:58,698] jepsen worker 45 - jepsen.util 1035 :invoke :read [15 nil] INFO [2022-03-13 17:08:58,738] jepsen worker 51 - jepsen.util 411 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,739] jepsen worker 51 - jepsen.util 411 :invoke :write [17 17] INFO [2022-03-13 17:08:58,742] jepsen worker 45 - jepsen.util 1035 :ok :read [15 48] INFO [2022-03-13 17:08:58,743] jepsen worker 45 - jepsen.util 1035 :invoke :write [15 19] INFO [2022-03-13 17:08:58,755] jepsen worker 23 - jepsen.util 473 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,756] jepsen worker 23 - jepsen.util 473 :invoke :write [7 7] INFO [2022-03-13 17:08:58,793] jepsen worker 21 - jepsen.util 651 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,793] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:58,794] jepsen worker 21 - jepsen.util 651 :invoke :write [7 2] INFO [2022-03-13 17:08:58,796] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:08:58,867] jepsen worker 22 - jepsen.util 472 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,867] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:08:58,873] jepsen worker 54 - jepsen.util 774 :ok :write [18 49] INFO [2022-03-13 17:08:58,874] jepsen worker 54 - jepsen.util 774 :invoke :read [18 nil] INFO [2022-03-13 17:08:58,879] jepsen worker 13 - jepsen.util 463 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,881] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:58,911] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,912] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:58,925] jepsen worker 14 - jepsen.util 194 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,925] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:08:58,964] jepsen worker 80 - jepsen.util 620 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,964] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,964] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,964] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,965] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:08:58,966] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:08:58,966] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:08:58,967] jepsen worker 78 - jepsen.util 438 :invoke :write [26 48] INFO [2022-03-13 17:08:58,993] jepsen worker 51 - jepsen.util 411 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:08:58,993] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:08:58,994] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:08:58,994] jepsen worker 13 - jepsen.util 463 :invoke :write [4 5] INFO [2022-03-13 17:08:59,000] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,001] jepsen worker 51 - jepsen.util 411 :invoke :write [17 24] INFO [2022-03-13 17:08:59,006] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,006] jepsen worker 23 - jepsen.util 473 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,007] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:59,008] jepsen worker 23 - jepsen.util 473 :invoke :write [7 44] INFO [2022-03-13 17:08:59,011] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,011] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:08:59,038] jepsen worker 77 - jepsen.util 527 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,039] jepsen worker 77 - jepsen.util 527 :invoke :write [25 7] INFO [2022-03-13 17:08:59,086] jepsen worker 68 - jepsen.util 788 :ok :read [22 30] INFO [2022-03-13 17:08:59,086] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,087] jepsen worker 68 - jepsen.util 788 :invoke :write [22 45] INFO [2022-03-13 17:08:59,087] jepsen worker 70 - jepsen.util 700 :invoke :write [23 5] INFO [2022-03-13 17:08:59,098] jepsen worker 40 - jepsen.util 670 :ok :read [13 39] INFO [2022-03-13 17:08:59,099] jepsen worker 40 - jepsen.util 670 :invoke :read [13 nil] INFO [2022-03-13 17:08:59,103] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,103] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,104] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:59,105] jepsen worker 53 - jepsen.util 503 :invoke :write [17 33] INFO [2022-03-13 17:08:59,128] jepsen worker 62 - jepsen.util 422 :ok :read [20 12] INFO [2022-03-13 17:08:59,130] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,130] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:08:59,130] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:08:59,130] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:08:59,131] jepsen worker 75 - jepsen.util 345 :invoke :write [25 37] INFO [2022-03-13 17:08:59,132] jepsen worker 26 - jepsen.util 476 :invoke :write [8 12] INFO [2022-03-13 17:08:59,135] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:08:59,139] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:08:59,140] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:08:59,143] jepsen worker 59 - jepsen.util 779 :ok :read [19 2] INFO [2022-03-13 17:08:59,144] jepsen worker 59 - jepsen.util 779 :invoke :read [19 nil] INFO [2022-03-13 17:08:59,147] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:08:59,148] jepsen worker 35 - jepsen.util 665 :invoke :write [11 33] INFO [2022-03-13 17:08:59,174] jepsen worker 13 - jepsen.util 463 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,175] jepsen worker 13 - jepsen.util 463 :invoke :write [4 47] INFO [2022-03-13 17:08:59,179] jepsen worker 23 - jepsen.util 473 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,180] jepsen worker 23 - jepsen.util 473 :invoke :write [7 28] INFO [2022-03-13 17:08:59,240] jepsen worker 24 - jepsen.util 474 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,241] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:08:59,241] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:08:59,241] jepsen worker 25 - jepsen.util 385 :invoke :write [8 31] INFO [2022-03-13 17:08:59,287] jepsen worker 48 - jepsen.util 588 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:08:59,288] jepsen worker 48 - jepsen.util 678 :invoke :write [16 9] INFO [2022-03-13 17:08:59,306] jepsen worker 57 - jepsen.util 777 :info :write [19 12] :RequestTimeoutException INFO [2022-03-13 17:08:59,307] jepsen worker 57 - jepsen.util 867 :invoke :write [19 32] INFO [2022-03-13 17:08:59,309] jepsen worker 40 - jepsen.util 670 :ok :read [13 39] INFO [2022-03-13 17:08:59,309] jepsen worker 40 - jepsen.util 670 :invoke :write [13 12] INFO [2022-03-13 17:08:59,366] jepsen worker 5 - jepsen.util 725 :info :write [1 42] :RequestTimeoutException INFO [2022-03-13 17:08:59,368] jepsen worker 5 - jepsen.util 815 :invoke :write [1 18] INFO [2022-03-13 17:08:59,373] jepsen worker 51 - jepsen.util 411 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,374] jepsen worker 51 - jepsen.util 411 :invoke :write [17 36] INFO [2022-03-13 17:08:59,382] jepsen worker 59 - jepsen.util 779 :ok :read [19 2] INFO [2022-03-13 17:08:59,383] jepsen worker 59 - jepsen.util 779 :invoke :write [19 18] INFO [2022-03-13 17:08:59,392] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,392] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:08:59,416] jepsen worker 13 - jepsen.util 463 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,417] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:08:59,420] jepsen worker 31 - jepsen.util 751 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:08:59,421] jepsen worker 31 - jepsen.util 841 :invoke :write [10 20] INFO [2022-03-13 17:08:59,469] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1173020527859238}, :ns_1@172.23.105.187 {:progress 0.1345029239766082}, :ns_1@172.23.107.227 {:progress 0.1173020527859238}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:08:59,469] jepsen worker nemesis - couchbase.util Rebalance check count: 52 INFO [2022-03-13 17:08:59,469] jepsen worker nemesis - couchbase.util Rebalance stuck count: 28 INFO [2022-03-13 17:08:59,503] jepsen worker 29 - jepsen.util 839 :info :write [9 28] :RequestTimeoutException INFO [2022-03-13 17:08:59,504] jepsen worker 29 - jepsen.util 929 :invoke :read [9 nil] INFO [2022-03-13 17:08:59,536] jepsen worker 29 - jepsen.util 929 :ok :read [9 23] INFO [2022-03-13 17:08:59,537] jepsen worker 29 - jepsen.util 929 :invoke :write [9 29] INFO [2022-03-13 17:08:59,548] jepsen worker 79 - jepsen.util 709 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,549] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:08:59,560] jepsen worker 12 - jepsen.util 642 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,561] jepsen worker 12 - jepsen.util 642 :invoke :write [4 34] INFO [2022-03-13 17:08:59,561] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:08:59,561] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:08:59,562] jepsen worker 14 - jepsen.util 194 :invoke :write [4 32] INFO [2022-03-13 17:08:59,563] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:08:59,575] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:08:59,576] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:08:59,592] jepsen worker 62 - jepsen.util 422 :ok :read [20 12] INFO [2022-03-13 17:08:59,593] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:08:59,776] jepsen worker 41 - jepsen.util 761 :ok :read [13 39] INFO [2022-03-13 17:08:59,776] jepsen worker 41 - jepsen.util 761 :invoke :write [13 2] INFO [2022-03-13 17:08:59,811] jepsen worker 17 - jepsen.util 827 :ok :read [5 43] INFO [2022-03-13 17:08:59,812] jepsen worker 17 - jepsen.util 827 :invoke :write [5 3] INFO [2022-03-13 17:09:00,031] jepsen worker 44 - jepsen.util 494 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,031] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,031] jepsen worker 23 - jepsen.util 473 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,032] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:00,033] jepsen worker 23 - jepsen.util 473 :invoke :write [7 33] INFO [2022-03-13 17:09:00,033] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:00,039] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,040] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:00,053] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,054] jepsen worker 69 - jepsen.util 609 :invoke :write [23 6] INFO [2022-03-13 17:09:00,108] jepsen worker 51 - jepsen.util 411 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,109] jepsen worker 51 - jepsen.util 411 :invoke :write [17 38] INFO [2022-03-13 17:09:00,110] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:00,111] jepsen worker 13 - jepsen.util 463 :invoke :write [4 20] INFO [2022-03-13 17:09:00,137] jepsen worker 23 - jepsen.util 473 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,137] jepsen worker 23 - jepsen.util 473 :invoke :write [7 27] INFO [2022-03-13 17:09:00,147] jepsen worker 21 - jepsen.util 651 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,147] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,148] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,148] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,148] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:00,149] jepsen worker 8 - jepsen.util 638 :invoke :write [2 15] INFO [2022-03-13 17:09:00,149] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:00,150] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:00,208] jepsen worker 49 - jepsen.util 679 :info :write [16 11] :RequestTimeoutException INFO [2022-03-13 17:09:00,208] jepsen worker 49 - jepsen.util 769 :invoke :write [16 8] INFO [2022-03-13 17:09:00,219] jepsen worker 13 - jepsen.util 463 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,219] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:00,243] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:00,243] jepsen worker 33 - jepsen.util 573 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,243] jepsen worker 53 - jepsen.util 503 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,244] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,244] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:09:00,245] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:09:00,245] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:09:00,246] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:00,253] jepsen worker 6 - jepsen.util 546 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,254] jepsen worker 6 - jepsen.util 546 :invoke :write [2 3] INFO [2022-03-13 17:09:00,257] jepsen worker 69 - jepsen.util 609 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,258] jepsen worker 69 - jepsen.util 609 :invoke :write [23 0] INFO [2022-03-13 17:09:00,317] jepsen worker 84 - jepsen.util 984 :info :write [28 43] :RequestTimeoutException INFO [2022-03-13 17:09:00,318] jepsen worker 84 - jepsen.util 1074 :invoke :read [28 nil] INFO [2022-03-13 17:09:00,327] jepsen worker 51 - jepsen.util 411 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,327] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:00,327] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:00,328] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:00,332] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,333] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:00,334] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,334] jepsen worker 51 - jepsen.util 411 :invoke :write [17 7] INFO [2022-03-13 17:09:00,345] jepsen worker 84 - jepsen.util 1074 :ok :read [28 44] INFO [2022-03-13 17:09:00,345] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:00,346] jepsen worker 84 - jepsen.util 1074 :invoke :read [28 nil] INFO [2022-03-13 17:09:00,346] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:00,348] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:00,349] jepsen worker 13 - jepsen.util 463 :invoke :write [4 12] INFO [2022-03-13 17:09:00,351] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:00,351] jepsen worker 78 - jepsen.util 438 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,352] jepsen worker 78 - jepsen.util 438 :invoke :write [26 31] INFO [2022-03-13 17:09:00,352] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:00,369] jepsen worker 84 - jepsen.util 1074 :ok :read [28 44] INFO [2022-03-13 17:09:00,370] jepsen worker 84 - jepsen.util 1074 :invoke :write [28 49] INFO [2022-03-13 17:09:00,371] jepsen worker 23 - jepsen.util 473 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,371] jepsen worker 23 - jepsen.util 473 :invoke :write [7 20] INFO [2022-03-13 17:09:00,475] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1173020527859238}, :ns_1@172.23.105.187 {:progress 0.1345029239766082}, :ns_1@172.23.107.227 {:progress 0.1173020527859238}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:00,475] jepsen worker nemesis - couchbase.util Rebalance check count: 53 INFO [2022-03-13 17:09:00,475] jepsen worker nemesis - couchbase.util Rebalance stuck count: 29 INFO [2022-03-13 17:09:00,568] jepsen worker 67 - jepsen.util 787 :info :write [22 30] :RequestTimeoutException INFO [2022-03-13 17:09:00,569] jepsen worker 67 - jepsen.util 877 :invoke :write [22 22] INFO [2022-03-13 17:09:00,666] jepsen worker 1 - jepsen.util 811 :info :write [0 34] :RequestTimeoutException INFO [2022-03-13 17:09:00,667] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] INFO [2022-03-13 17:09:00,688] jepsen worker 51 - jepsen.util 411 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,688] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:00,689] jepsen worker 34 - jepsen.util 574 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,689] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:00,689] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,690] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:00,691] jepsen worker 71 - jepsen.util 431 :invoke :write [23 41] INFO [2022-03-13 17:09:00,691] jepsen worker 22 - jepsen.util 472 :invoke :write [7 10] INFO [2022-03-13 17:09:00,731] jepsen worker 26 - jepsen.util 476 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,731] jepsen worker 43 - jepsen.util 403 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,731] jepsen worker 43 - jepsen.util 403 :invoke :write [14 25] INFO [2022-03-13 17:09:00,732] jepsen worker 26 - jepsen.util 476 :invoke :write [8 33] INFO [2022-03-13 17:09:00,733] jepsen worker 69 - jepsen.util 609 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,734] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:00,796] jepsen worker 39 - jepsen.util 939 :info :write [13 34] :RequestTimeoutException INFO [2022-03-13 17:09:00,797] jepsen worker 39 - jepsen.util 1029 :invoke :read [13 nil] INFO [2022-03-13 17:09:00,806] jepsen worker 15 - jepsen.util 735 :info :write [5 23] :RequestTimeoutException INFO [2022-03-13 17:09:00,807] jepsen worker 15 - jepsen.util 825 :invoke :write [5 26] INFO [2022-03-13 17:09:00,811] jepsen worker 25 - jepsen.util 385 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,812] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:00,814] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,814] jepsen worker 13 - jepsen.util 463 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,814] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,814] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:00,815] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,815] jepsen worker 52 - jepsen.util 502 :invoke :write [17 24] INFO [2022-03-13 17:09:00,816] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:00,816] jepsen worker 13 - jepsen.util 463 :invoke :write [4 48] INFO [2022-03-13 17:09:00,817] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:00,819] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,819] jepsen worker 80 - jepsen.util 620 :invoke :write [26 49] INFO [2022-03-13 17:09:00,821] jepsen worker 51 - jepsen.util 411 :invoke :write [17 35] INFO [2022-03-13 17:09:00,857] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,857] jepsen worker 23 - jepsen.util 473 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,858] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:09:00,858] jepsen worker 23 - jepsen.util 473 :invoke :write [7 6] INFO [2022-03-13 17:09:00,874] jepsen worker 75 - jepsen.util 345 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,875] jepsen worker 75 - jepsen.util 345 :invoke :write [25 3] INFO [2022-03-13 17:09:00,884] jepsen worker 70 - jepsen.util 700 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,890] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:00,890] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:00,890] jepsen worker 44 - jepsen.util 494 :invoke :write [14 45] INFO [2022-03-13 17:09:00,936] jepsen worker 13 - jepsen.util 463 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,937] jepsen worker 13 - jepsen.util 463 :invoke :write [4 5] INFO [2022-03-13 17:09:00,955] jepsen worker 77 - jepsen.util 527 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,956] jepsen worker 77 - jepsen.util 527 :invoke :write [25 49] INFO [2022-03-13 17:09:00,980] jepsen worker 8 - jepsen.util 638 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,980] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:00,981] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:00,982] jepsen worker 24 - jepsen.util 474 :invoke :write [8 3] INFO [2022-03-13 17:09:00,991] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:00,992] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:01,010] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,011] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:09:01,021] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:09:01,021] jepsen worker 60 - jepsen.util 600 :invoke :write [20 18] INFO [2022-03-13 17:09:01,048] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,049] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:09:01,049] jepsen worker 51 - jepsen.util 411 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,050] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:01,053] jepsen worker 58 - jepsen.util 688 :info :write [19 19] :RequestTimeoutException INFO [2022-03-13 17:09:01,054] jepsen worker 58 - jepsen.util 778 :invoke :write [19 9] INFO [2022-03-13 17:09:01,061] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:09:01,062] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:09:01,068] jepsen worker 6 - jepsen.util 546 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,070] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:01,099] jepsen worker 23 - jepsen.util 473 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,100] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,100] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:01,101] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:09:01,103] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:01,104] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:01,106] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:01,107] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:01,108] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:01,108] jepsen worker 23 - jepsen.util 473 :invoke :write [7 10] INFO [2022-03-13 17:09:01,327] jepsen worker 19 - jepsen.util 559 :info :write [6 9] :RequestTimeoutException INFO [2022-03-13 17:09:01,329] jepsen worker 19 - jepsen.util 649 :invoke :read [6 nil] INFO [2022-03-13 17:09:01,354] jepsen worker 35 - jepsen.util 665 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,354] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,355] jepsen worker 35 - jepsen.util 665 :invoke :write [11 33] INFO [2022-03-13 17:09:01,356] jepsen worker 76 - jepsen.util 526 :invoke :write [25 39] INFO [2022-03-13 17:09:01,381] jepsen worker 13 - jepsen.util 463 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,381] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,382] jepsen worker 13 - jepsen.util 463 :invoke :write [4 6] INFO [2022-03-13 17:09:01,383] jepsen worker 51 - jepsen.util 411 :invoke :write [17 33] INFO [2022-03-13 17:09:01,402] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,403] jepsen worker 69 - jepsen.util 609 :invoke :write [23 3] INFO [2022-03-13 17:09:01,405] jepsen worker 43 - jepsen.util 403 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,406] jepsen worker 43 - jepsen.util 403 :invoke :write [14 24] INFO [2022-03-13 17:09:01,431] jepsen worker 78 - jepsen.util 438 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,431] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:01,432] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:01,432] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:01,433] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:01,433] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:01,462] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:09:01,462] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:01,469] jepsen worker 23 - jepsen.util 473 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,483] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1231671554252199}, :ns_1@172.23.105.187 {:progress 0.1403508771929824}, :ns_1@172.23.107.227 {:progress 0.1231671554252199}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:01,516] jepsen worker 13 - jepsen.util 463 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,553] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:01,533] jepsen worker 69 - jepsen.util 609 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,553] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:01,537] jepsen worker 71 - jepsen.util 431 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,553] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:09:01,553] jepsen worker nemesis - couchbase.util Rebalance check count: 54 INFO [2022-03-13 17:09:01,553] jepsen worker nemesis - couchbase.util Rebalance stuck count: 29 INFO [2022-03-13 17:09:01,553] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:01,555] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:01,555] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:01,557] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:01,557] jepsen worker 23 - jepsen.util 473 :invoke :write [7 28] INFO [2022-03-13 17:09:01,577] jepsen worker 77 - jepsen.util 527 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,577] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,578] jepsen worker 12 - jepsen.util 642 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,578] jepsen worker 79 - jepsen.util 709 :invoke :write [26 1] INFO [2022-03-13 17:09:01,579] jepsen worker 12 - jepsen.util 642 :invoke :write [4 25] INFO [2022-03-13 17:09:01,579] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:09:01,613] jepsen worker 51 - jepsen.util 411 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,613] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:01,614] jepsen worker 51 - jepsen.util 411 :invoke :write [17 31] INFO [2022-03-13 17:09:01,614] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:01,615] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:01,616] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:01,617] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:01,618] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:01,620] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:01,621] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:01,625] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:01,626] jepsen worker 13 - jepsen.util 463 :invoke :write [4 40] INFO [2022-03-13 17:09:01,665] jepsen worker 52 - jepsen.util 502 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,666] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:01,678] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,679] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:09:01,744] jepsen worker 22 - jepsen.util 472 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,744] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,745] jepsen worker 22 - jepsen.util 472 :invoke :write [7 29] INFO [2022-03-13 17:09:01,746] jepsen worker 70 - jepsen.util 700 :invoke :write [23 21] INFO [2022-03-13 17:09:01,768] jepsen worker 51 - jepsen.util 411 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,769] jepsen worker 23 - jepsen.util 473 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,770] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:01,770] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:01,788] jepsen worker 14 - jepsen.util 194 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,788] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,788] jepsen worker 60 - jepsen.util 600 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,789] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:01,790] jepsen worker 14 - jepsen.util 194 :invoke :write [4 36] INFO [2022-03-13 17:09:01,791] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:09:01,839] jepsen worker 80 - jepsen.util 620 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,839] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,840] jepsen worker 8 - jepsen.util 638 :invoke :write [2 4] INFO [2022-03-13 17:09:01,841] jepsen worker 80 - jepsen.util 620 :invoke :write [26 28] INFO [2022-03-13 17:09:01,845] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:09:01,846] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,846] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:01,846] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:09:01,847] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:09:01,848] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:01,865] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:01,866] jepsen worker 23 - jepsen.util 473 :invoke :write [7 42] INFO [2022-03-13 17:09:01,896] jepsen worker 50 - jepsen.util 860 :info :write [16 48] :RequestTimeoutException INFO [2022-03-13 17:09:01,896] jepsen worker 50 - jepsen.util 950 :invoke :write [16 21] INFO [2022-03-13 17:09:01,900] jepsen worker 13 - jepsen.util 463 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,900] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,900] jepsen worker 13 - jepsen.util 463 :invoke :write [4 15] INFO [2022-03-13 17:09:01,901] jepsen worker 51 - jepsen.util 411 :invoke :write [17 10] INFO [2022-03-13 17:09:01,926] jepsen worker 2 - jepsen.util 902 :info :write [0 29] :RequestTimeoutException INFO [2022-03-13 17:09:01,927] jepsen worker 2 - jepsen.util 992 :invoke :write [0 18] INFO [2022-03-13 17:09:01,947] jepsen worker 44 - jepsen.util 494 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,947] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,947] jepsen worker 44 - jepsen.util 494 :invoke :write [14 38] INFO [2022-03-13 17:09:01,948] jepsen worker 6 - jepsen.util 546 :invoke :write [2 24] INFO [2022-03-13 17:09:01,990] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:01,990] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:02,004] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:02,005] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:02,008] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,009] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:09:02,026] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,027] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:09:02,050] jepsen worker 24 - jepsen.util 474 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,050] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:02,050] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,051] jepsen worker 78 - jepsen.util 438 :invoke :write [26 49] INFO [2022-03-13 17:09:02,052] jepsen worker 25 - jepsen.util 385 :invoke :write [8 32] INFO [2022-03-13 17:09:02,052] jepsen worker 24 - jepsen.util 474 :invoke :write [8 26] INFO [2022-03-13 17:09:02,053] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:02,053] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,054] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:02,054] jepsen worker 52 - jepsen.util 502 :invoke :write [17 17] INFO [2022-03-13 17:09:02,090] jepsen worker 51 - jepsen.util 411 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,090] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:02,098] jepsen worker 43 - jepsen.util 403 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,098] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,098] jepsen worker 77 - jepsen.util 527 :invoke :write [25 35] INFO [2022-03-13 17:09:02,099] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:02,115] jepsen worker 3 - jepsen.util 633 :ok :read [1 49] INFO [2022-03-13 17:09:02,116] jepsen worker 3 - jepsen.util 633 :invoke :write [1 2] INFO [2022-03-13 17:09:02,180] jepsen worker 70 - jepsen.util 700 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,180] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:09:02,185] jepsen worker 23 - jepsen.util 473 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,186] jepsen worker 23 - jepsen.util 473 :invoke :write [7 1] INFO [2022-03-13 17:09:02,192] jepsen worker 13 - jepsen.util 463 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,192] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,193] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:02,194] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:02,194] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:02,195] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,195] jepsen worker 13 - jepsen.util 463 :invoke :write [4 15] INFO [2022-03-13 17:09:02,197] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:02,201] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,202] jepsen worker 51 - jepsen.util 411 :invoke :write [17 4] INFO [2022-03-13 17:09:02,211] jepsen worker 30 - jepsen.util 660 :info :write [10 14] :RequestTimeoutException INFO [2022-03-13 17:09:02,213] jepsen worker 26 - jepsen.util 476 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,214] jepsen worker 30 - jepsen.util 750 :invoke :write [10 33] INFO [2022-03-13 17:09:02,219] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:02,276] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,276] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:02,311] jepsen worker 80 - jepsen.util 620 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,312] jepsen worker 80 - jepsen.util 620 :invoke :write [26 32] INFO [2022-03-13 17:09:02,344] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,344] jepsen worker 79 - jepsen.util 709 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,345] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,345] jepsen worker 79 - jepsen.util 709 :invoke :write [26 27] INFO [2022-03-13 17:09:02,346] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:09:02,347] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:02,370] jepsen worker 13 - jepsen.util 463 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,371] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:02,381] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:02,382] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:02,401] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:09:02,402] jepsen worker 33 - jepsen.util 573 :invoke :write [11 14] INFO [2022-03-13 17:09:02,448] jepsen worker 22 - jepsen.util 472 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,449] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:02,456] jepsen worker 23 - jepsen.util 473 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,456] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,456] jepsen worker 23 - jepsen.util 473 :invoke :write [7 4] INFO [2022-03-13 17:09:02,457] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:09:02,478] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:02,479] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:02,487] jepsen worker 51 - jepsen.util 411 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,487] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:02,487] jepsen worker 51 - jepsen.util 411 :invoke :write [17 38] INFO [2022-03-13 17:09:02,488] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:02,490] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:02,490] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:02,492] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:02,492] jepsen worker 13 - jepsen.util 463 :invoke :write [4 38] INFO [2022-03-13 17:09:02,498] jepsen worker 72 - jepsen.util 882 :ok :read [24 24] INFO [2022-03-13 17:09:02,499] jepsen worker 72 - jepsen.util 882 :invoke :read [24 nil] INFO [2022-03-13 17:09:02,554] jepsen worker 51 - jepsen.util 411 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,555] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:02,559] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1231671554252199}, :ns_1@172.23.105.187 {:progress 0.1403508771929824}, :ns_1@172.23.107.227 {:progress 0.1231671554252199}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:02,559] jepsen worker nemesis - couchbase.util Rebalance check count: 55 INFO [2022-03-13 17:09:02,559] jepsen worker nemesis - couchbase.util Rebalance stuck count: 30 INFO [2022-03-13 17:09:02,572] jepsen worker 77 - jepsen.util 527 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,572] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,572] jepsen worker 75 - jepsen.util 345 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,572] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,572] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:09:02,573] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:02,574] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:09:02,574] jepsen worker 60 - jepsen.util 600 :invoke :write [20 2] INFO [2022-03-13 17:09:02,588] jepsen worker 13 - jepsen.util 463 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,588] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,588] jepsen worker 51 - jepsen.util 411 :invoke :write [17 23] INFO [2022-03-13 17:09:02,589] jepsen worker 13 - jepsen.util 463 :invoke :write [4 30] INFO [2022-03-13 17:09:02,621] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,621] jepsen worker 69 - jepsen.util 609 :invoke :write [23 44] INFO [2022-03-13 17:09:02,622] jepsen worker 8 - jepsen.util 638 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,622] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:02,628] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:02,628] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:02,708] jepsen worker 44 - jepsen.util 494 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,708] jepsen worker 23 - jepsen.util 473 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,709] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:02,709] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:02,720] jepsen worker 76 - jepsen.util 526 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,720] jepsen worker 79 - jepsen.util 709 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,720] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:02,721] jepsen worker 76 - jepsen.util 526 :invoke :write [25 10] INFO [2022-03-13 17:09:02,723] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:02,726] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:09:02,741] jepsen worker 13 - jepsen.util 463 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,741] jepsen worker 13 - jepsen.util 463 :invoke :write [4 32] INFO [2022-03-13 17:09:02,777] jepsen worker 69 - jepsen.util 609 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,777] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:02,778] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:09:02,779] jepsen worker 23 - jepsen.util 473 :invoke :write [7 28] INFO [2022-03-13 17:09:02,809] jepsen worker 35 - jepsen.util 665 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,810] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:02,812] jepsen worker 24 - jepsen.util 474 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,812] jepsen worker 24 - jepsen.util 474 :invoke :write [8 2] INFO [2022-03-13 17:09:02,813] jepsen worker 51 - jepsen.util 411 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,814] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:02,855] jepsen worker 6 - jepsen.util 546 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,855] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,856] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:02,856] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:02,861] jepsen worker 9 - jepsen.util 729 :ok :read [3 27] INFO [2022-03-13 17:09:02,861] jepsen worker 9 - jepsen.util 729 :invoke :read [3 nil] INFO [2022-03-13 17:09:02,885] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:02,886] jepsen worker 34 - jepsen.util 574 :invoke :write [11 38] INFO [2022-03-13 17:09:02,917] jepsen worker 73 - jepsen.util 883 :ok :read [24 24] INFO [2022-03-13 17:09:02,917] jepsen worker 73 - jepsen.util 883 :invoke :read [24 nil] INFO [2022-03-13 17:09:02,920] jepsen worker 33 - jepsen.util 573 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,921] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:09:02,923] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,923] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,924] jepsen worker 77 - jepsen.util 527 :invoke :write [25 21] INFO [2022-03-13 17:09:02,924] jepsen worker 7 - jepsen.util 367 :invoke :write [2 10] INFO [2022-03-13 17:09:02,931] jepsen worker 52 - jepsen.util 502 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,932] jepsen worker 52 - jepsen.util 502 :invoke :write [17 44] INFO [2022-03-13 17:09:02,967] jepsen worker 13 - jepsen.util 463 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,968] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,969] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:02,969] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:02,976] jepsen worker 23 - jepsen.util 473 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,977] jepsen worker 23 - jepsen.util 473 :invoke :write [7 8] INFO [2022-03-13 17:09:02,977] jepsen worker 19 - jepsen.util 649 :ok :read [6 16] INFO [2022-03-13 17:09:02,978] jepsen worker 19 - jepsen.util 649 :invoke :read [6 nil] INFO [2022-03-13 17:09:02,980] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:02,980] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,980] jepsen worker 13 - jepsen.util 463 :invoke :write [4 25] INFO [2022-03-13 17:09:02,984] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:02,986] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,986] jepsen worker 46 - jepsen.util 766 :info :write [15 3] :RequestTimeoutException INFO [2022-03-13 17:09:02,987] jepsen worker 46 - jepsen.util 856 :invoke :write [15 21] INFO [2022-03-13 17:09:02,988] jepsen worker 51 - jepsen.util 411 :invoke :write [17 5] INFO [2022-03-13 17:09:02,989] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,990] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:02,990] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:02,990] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:02,991] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:02,992] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:09:02,994] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:02,994] jepsen worker 44 - jepsen.util 494 :invoke :write [14 13] INFO [2022-03-13 17:09:02,998] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,005] jepsen worker 71 - jepsen.util 431 :invoke :write [23 14] INFO [2022-03-13 17:09:03,020] jepsen worker 9 - jepsen.util 729 :ok :read [3 27] INFO [2022-03-13 17:09:03,020] jepsen worker 9 - jepsen.util 729 :invoke :write [3 1] INFO [2022-03-13 17:09:03,026] jepsen worker 36 - jepsen.util 846 :info :write [12 35] :RequestTimeoutException INFO [2022-03-13 17:09:03,028] jepsen worker 36 - jepsen.util 936 :invoke :read [12 nil] INFO [2022-03-13 17:09:03,053] jepsen worker 87 - jepsen.util 1077 :info :write [29 15] :RequestTimeoutException INFO [2022-03-13 17:09:03,055] jepsen worker 87 - jepsen.util 1167 :invoke :write [29 12] INFO [2022-03-13 17:09:03,061] jepsen worker 73 - jepsen.util 883 :ok :read [24 24] INFO [2022-03-13 17:09:03,062] jepsen worker 73 - jepsen.util 883 :invoke :write [24 25] INFO [2022-03-13 17:09:03,066] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,067] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:09:03,074] jepsen worker 36 - jepsen.util 936 :ok :read [12 2] INFO [2022-03-13 17:09:03,075] jepsen worker 36 - jepsen.util 936 :invoke :write [12 7] INFO [2022-03-13 17:09:03,109] jepsen worker 23 - jepsen.util 473 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,110] jepsen worker 23 - jepsen.util 473 :invoke :write [7 46] INFO [2022-03-13 17:09:03,111] jepsen worker 11 - jepsen.util 821 :info :write [3 44] :RequestTimeoutException INFO [2022-03-13 17:09:03,111] jepsen worker 11 - jepsen.util 911 :invoke :read [3 nil] INFO [2022-03-13 17:09:03,126] jepsen worker 19 - jepsen.util 649 :ok :read [6 16] INFO [2022-03-13 17:09:03,127] jepsen worker 19 - jepsen.util 649 :invoke :read [6 nil] INFO [2022-03-13 17:09:03,151] jepsen worker 14 - jepsen.util 194 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,151] jepsen worker 14 - jepsen.util 194 :invoke :write [4 35] INFO [2022-03-13 17:09:03,164] jepsen worker 25 - jepsen.util 385 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,165] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:03,171] jepsen worker 13 - jepsen.util 463 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,171] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:03,199] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,200] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,200] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:03,200] jepsen worker 60 - jepsen.util 600 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,201] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,201] jepsen worker 43 - jepsen.util 403 :invoke :write [14 19] INFO [2022-03-13 17:09:03,201] jepsen worker 60 - jepsen.util 600 :invoke :write [20 6] INFO [2022-03-13 17:09:03,202] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:09:03,230] jepsen worker 63 - jepsen.util 783 :info :write [21 39] :RequestTimeoutException INFO [2022-03-13 17:09:03,231] jepsen worker 63 - jepsen.util 873 :invoke :read [21 nil] INFO [2022-03-13 17:09:03,255] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:09:03,255] jepsen worker 33 - jepsen.util 573 :invoke :write [11 2] INFO [2022-03-13 17:09:03,281] jepsen worker 34 - jepsen.util 574 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,281] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,282] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,282] jepsen worker 34 - jepsen.util 574 :invoke :write [11 41] INFO [2022-03-13 17:09:03,283] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,283] jepsen worker 6 - jepsen.util 546 :invoke :write [2 8] INFO [2022-03-13 17:09:03,293] jepsen worker 51 - jepsen.util 411 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,293] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:03,294] jepsen worker 51 - jepsen.util 411 :invoke :write [17 49] INFO [2022-03-13 17:09:03,296] jepsen worker 13 - jepsen.util 463 :invoke :write [4 21] INFO [2022-03-13 17:09:03,313] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,313] jepsen worker 23 - jepsen.util 473 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,315] jepsen worker 69 - jepsen.util 609 :invoke :write [23 6] INFO [2022-03-13 17:09:03,315] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,332] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,333] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,338] jepsen worker 19 - jepsen.util 649 :ok :read [6 16] INFO [2022-03-13 17:09:03,339] jepsen worker 19 - jepsen.util 649 :invoke :write [6 21] INFO [2022-03-13 17:09:03,416] jepsen worker 78 - jepsen.util 438 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,416] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,416] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,418] jepsen worker 8 - jepsen.util 638 :invoke :write [2 33] INFO [2022-03-13 17:09:03,418] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,419] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:03,421] jepsen worker 12 - jepsen.util 642 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,421] jepsen worker 77 - jepsen.util 527 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,421] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,422] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:03,423] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:03,423] jepsen worker 26 - jepsen.util 476 :invoke :write [8 18] INFO [2022-03-13 17:09:03,424] jepsen worker 75 - jepsen.util 345 :invoke :write [25 44] INFO [2022-03-13 17:09:03,425] jepsen worker 77 - jepsen.util 527 :invoke :read [25 nil] INFO [2022-03-13 17:09:03,477] jepsen worker 51 - jepsen.util 411 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,478] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:03,489] jepsen worker 10 - jepsen.util 730 :ok :read [3 27] INFO [2022-03-13 17:09:03,490] jepsen worker 10 - jepsen.util 730 :invoke :read [3 nil] INFO [2022-03-13 17:09:03,495] jepsen worker 69 - jepsen.util 609 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,495] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,495] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,496] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,496] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,497] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,497] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,498] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,498] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,499] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,505] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,505] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,506] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,506] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,511] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,512] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,513] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:03,514] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:09:03,514] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,515] jepsen worker 23 - jepsen.util 473 :invoke :write [7 26] INFO [2022-03-13 17:09:03,524] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,524] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,547] jepsen worker 13 - jepsen.util 463 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,548] jepsen worker 13 - jepsen.util 463 :invoke :write [4 17] INFO [2022-03-13 17:09:03,547] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,549] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:03,575] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.126099706744868}, :ns_1@172.23.105.187 {:progress 0.1461988304093568}, :ns_1@172.23.107.227 {:progress 0.126099706744868}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:03,575] jepsen worker nemesis - couchbase.util Rebalance check count: 56 INFO [2022-03-13 17:09:03,575] jepsen worker nemesis - couchbase.util Rebalance stuck count: 30 INFO [2022-03-13 17:09:03,576] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,577] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:03,592] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,592] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,624] jepsen worker 24 - jepsen.util 474 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,624] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:03,648] jepsen worker 38 - jepsen.util 938 :info :write [12 30] :RequestTimeoutException INFO [2022-03-13 17:09:03,649] jepsen worker 38 - jepsen.util 1028 :invoke :write [12 44] INFO [2022-03-13 17:09:03,667] jepsen worker 7 - jepsen.util 367 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,667] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,667] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,667] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,667] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,667] jepsen worker 44 - jepsen.util 494 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,668] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:09:03,669] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:09:03,670] jepsen worker 62 - jepsen.util 422 :invoke :write [20 48] INFO [2022-03-13 17:09:03,671] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:09:03,672] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:03,673] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,674] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,675] jepsen worker 61 - jepsen.util 421 :invoke :write [20 12] INFO [2022-03-13 17:09:03,676] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,677] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:09:03,682] jepsen worker 23 - jepsen.util 473 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,683] jepsen worker 23 - jepsen.util 473 :invoke :write [7 41] INFO [2022-03-13 17:09:03,688] jepsen worker 13 - jepsen.util 463 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,688] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,689] jepsen worker 13 - jepsen.util 463 :invoke :write [4 28] INFO [2022-03-13 17:09:03,689] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:03,691] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,692] jepsen worker 51 - jepsen.util 411 :invoke :write [17 41] INFO [2022-03-13 17:09:03,779] jepsen worker 80 - jepsen.util 620 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,780] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:03,816] jepsen worker 56 - jepsen.util 686 :info :write [18 44] :RequestTimeoutException INFO [2022-03-13 17:09:03,818] jepsen worker 56 - jepsen.util 776 :invoke :read [18 nil] INFO [2022-03-13 17:09:03,844] jepsen worker 23 - jepsen.util 473 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,844] jepsen worker 23 - jepsen.util 473 :invoke :write [7 0] INFO [2022-03-13 17:09:03,847] jepsen worker 13 - jepsen.util 463 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,847] jepsen worker 43 - jepsen.util 403 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,848] jepsen worker 76 - jepsen.util 526 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,848] jepsen worker 43 - jepsen.util 403 :invoke :write [14 21] INFO [2022-03-13 17:09:03,849] jepsen worker 76 - jepsen.util 526 :invoke :write [25 45] INFO [2022-03-13 17:09:03,849] jepsen worker 13 - jepsen.util 463 :invoke :write [4 7] INFO [2022-03-13 17:09:03,878] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,879] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:03,892] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:03,893] jepsen worker 35 - jepsen.util 665 :invoke :write [11 15] INFO [2022-03-13 17:09:03,937] jepsen worker 71 - jepsen.util 431 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,937] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:03,937] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:03,938] jepsen worker 22 - jepsen.util 472 :invoke :write [7 41] INFO [2022-03-13 17:09:03,945] jepsen worker 51 - jepsen.util 411 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,945] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:03,961] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,962] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:09:03,978] jepsen worker 14 - jepsen.util 194 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:03,978] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:03,979] jepsen worker 12 - jepsen.util 642 :invoke :write [4 18] INFO [2022-03-13 17:09:03,979] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:03,986] jepsen worker 89 - jepsen.util 899 :info :write [29 34] :RequestTimeoutException INFO [2022-03-13 17:09:03,987] jepsen worker 89 - jepsen.util 989 :invoke :write [29 30] INFO [2022-03-13 17:09:04,049] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:04,049] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,049] jepsen worker 60 - jepsen.util 600 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,049] jepsen worker 52 - jepsen.util 502 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,050] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:04,051] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,052] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:04,053] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:04,054] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:04,054] jepsen worker 23 - jepsen.util 473 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,055] jepsen worker 23 - jepsen.util 473 :invoke :write [7 23] INFO [2022-03-13 17:09:04,058] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:04,064] jepsen worker 13 - jepsen.util 463 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,064] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,065] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,065] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,066] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,066] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,067] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,067] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,068] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,068] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,069] jepsen worker 13 - jepsen.util 463 :invoke :write [4 18] INFO [2022-03-13 17:09:04,070] jepsen worker 51 - jepsen.util 411 :invoke :write [17 47] INFO [2022-03-13 17:09:04,097] jepsen worker 34 - jepsen.util 574 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,098] jepsen worker 34 - jepsen.util 574 :invoke :write [11 31] INFO [2022-03-13 17:09:04,120] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,121] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,169] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:04,169] jepsen worker 8 - jepsen.util 638 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,169] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,170] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,170] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:04,170] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,170] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:04,177] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:04,178] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:04,179] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,179] jepsen worker 21 - jepsen.util 651 :invoke :write [7 15] INFO [2022-03-13 17:09:04,180] jepsen worker 25 - jepsen.util 385 :invoke :write [8 18] INFO [2022-03-13 17:09:04,181] jepsen worker 24 - jepsen.util 474 :invoke :write [8 8] INFO [2022-03-13 17:09:04,181] jepsen worker 78 - jepsen.util 438 :invoke :write [26 14] INFO [2022-03-13 17:09:04,195] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,195] jepsen worker 23 - jepsen.util 473 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,196] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:09:04,197] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:04,198] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:04,199] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:04,202] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:04,202] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:04,203] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:04,204] jepsen worker 23 - jepsen.util 473 :invoke :write [7 3] INFO [2022-03-13 17:09:04,209] jepsen worker 13 - jepsen.util 463 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,209] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,213] jepsen worker 6 - jepsen.util 546 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,213] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,213] jepsen worker 6 - jepsen.util 546 :invoke :write [2 15] INFO [2022-03-13 17:09:04,214] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,216] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:04,216] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,217] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:04,218] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,221] jepsen worker 65 - jepsen.util 875 :ok :read [21 :nil] INFO [2022-03-13 17:09:04,223] jepsen worker 65 - jepsen.util 875 :invoke :write [21 36] INFO [2022-03-13 17:09:04,227] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,227] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,227] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,229] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:04,230] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,230] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:04,281] jepsen worker 75 - jepsen.util 345 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,281] jepsen worker 33 - jepsen.util 573 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,281] jepsen worker 77 - jepsen.util 527 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,281] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,282] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,282] jepsen worker 77 - jepsen.util 527 :invoke :write [25 29] INFO [2022-03-13 17:09:04,283] jepsen worker 75 - jepsen.util 345 :invoke :write [25 22] INFO [2022-03-13 17:09:04,284] jepsen worker 7 - jepsen.util 367 :invoke :read [2 nil] INFO [2022-03-13 17:09:04,285] jepsen worker 33 - jepsen.util 573 :invoke :read [11 nil] INFO [2022-03-13 17:09:04,286] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:09:04,291] jepsen worker 51 - jepsen.util 411 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,291] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,292] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,292] jepsen worker 13 - jepsen.util 463 :invoke :write [4 28] INFO [2022-03-13 17:09:04,293] jepsen worker 22 - jepsen.util 472 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,293] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:04,295] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,296] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,297] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,298] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:04,299] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,299] jepsen worker 51 - jepsen.util 411 :invoke :write [17 4] INFO [2022-03-13 17:09:04,322] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,322] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,369] jepsen worker 21 - jepsen.util 651 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,370] jepsen worker 21 - jepsen.util 651 :invoke :write [7 27] INFO [2022-03-13 17:09:04,390] jepsen worker 13 - jepsen.util 463 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,391] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,436] jepsen worker 43 - jepsen.util 403 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,436] jepsen worker 7 - jepsen.util 367 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,436] jepsen worker 26 - jepsen.util 476 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,436] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,437] jepsen worker 60 - jepsen.util 600 :invoke :write [20 25] INFO [2022-03-13 17:09:04,438] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:04,438] jepsen worker 7 - jepsen.util 367 :invoke :write [2 33] INFO [2022-03-13 17:09:04,439] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:04,445] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,446] jepsen worker 79 - jepsen.util 709 :invoke :read [26 nil] INFO [2022-03-13 17:09:04,461] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,461] jepsen worker 23 - jepsen.util 473 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,461] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,462] jepsen worker 23 - jepsen.util 473 :invoke :write [7 35] INFO [2022-03-13 17:09:04,462] jepsen worker 33 - jepsen.util 573 :ok :read [11 7] INFO [2022-03-13 17:09:04,463] jepsen worker 33 - jepsen.util 573 :invoke :write [11 3] INFO [2022-03-13 17:09:04,463] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,464] jepsen worker 69 - jepsen.util 609 :invoke :write [23 7] INFO [2022-03-13 17:09:04,515] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,516] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:04,516] jepsen worker 34 - jepsen.util 574 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,516] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,517] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,517] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:04,518] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:04,518] jepsen worker 71 - jepsen.util 431 :invoke :write [23 30] INFO [2022-03-13 17:09:04,525] jepsen worker 51 - jepsen.util 411 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,525] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,526] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,527] jepsen worker 51 - jepsen.util 411 :invoke :write [17 3] INFO [2022-03-13 17:09:04,540] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,541] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,543] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,544] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,545] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,546] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,547] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,548] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:04,549] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,550] jepsen worker 13 - jepsen.util 463 :invoke :write [4 27] INFO [2022-03-13 17:09:04,556] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:04,556] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,556] jepsen worker 77 - jepsen.util 527 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,556] jepsen worker 79 - jepsen.util 709 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,559] jepsen worker 79 - jepsen.util 709 :invoke :write [26 7] INFO [2022-03-13 17:09:04,560] jepsen worker 77 - jepsen.util 527 :invoke :write [25 14] INFO [2022-03-13 17:09:04,561] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:04,562] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:04,584] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1290322580645161}, :ns_1@172.23.105.187 {:progress 0.1461988304093568}, :ns_1@172.23.107.227 {:progress 0.1290322580645161}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:04,584] jepsen worker nemesis - couchbase.util Rebalance check count: 57 INFO [2022-03-13 17:09:04,584] jepsen worker nemesis - couchbase.util Rebalance stuck count: 30 INFO [2022-03-13 17:09:04,600] jepsen worker 23 - jepsen.util 473 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,600] jepsen worker 23 - jepsen.util 473 :invoke :read [7 nil] INFO [2022-03-13 17:09:04,611] jepsen worker 47 - jepsen.util 947 :ok :read [15 48] INFO [2022-03-13 17:09:04,612] jepsen worker 47 - jepsen.util 947 :invoke :read [15 nil] INFO [2022-03-13 17:09:04,634] jepsen worker 78 - jepsen.util 438 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,635] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:04,676] jepsen worker 51 - jepsen.util 411 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,677] jepsen worker 51 - jepsen.util 411 :invoke :write [17 11] INFO [2022-03-13 17:09:04,687] jepsen worker 61 - jepsen.util 421 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,688] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:04,723] jepsen worker 69 - jepsen.util 609 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,723] jepsen worker 23 - jepsen.util 473 :ok :read [7 :nil] INFO [2022-03-13 17:09:04,723] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,724] jepsen worker 23 - jepsen.util 473 :invoke :write [7 34] INFO [2022-03-13 17:09:04,737] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,738] jepsen worker 69 - jepsen.util 609 :invoke :write [23 30] INFO [2022-03-13 17:09:04,738] jepsen worker 60 - jepsen.util 600 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,739] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:04,761] jepsen worker 6 - jepsen.util 546 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,762] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,762] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:04,762] jepsen worker 6 - jepsen.util 546 :invoke :write [2 24] INFO [2022-03-13 17:09:04,763] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:04,764] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:04,795] jepsen worker 13 - jepsen.util 463 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,796] jepsen worker 13 - jepsen.util 463 :invoke :write [4 35] INFO [2022-03-13 17:09:04,814] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:04,815] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:04,827] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:04,827] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:04,836] jepsen worker 23 - jepsen.util 473 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,836] jepsen worker 23 - jepsen.util 473 :invoke :write [7 35] INFO [2022-03-13 17:09:04,886] jepsen worker 62 - jepsen.util 422 :fail :write [20 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,886] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:04,887] jepsen worker 14 - jepsen.util 194 :invoke :write [4 23] INFO [2022-03-13 17:09:04,887] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:04,916] jepsen worker 24 - jepsen.util 474 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,916] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:04,917] jepsen worker 80 - jepsen.util 620 :invoke :write [26 4] INFO [2022-03-13 17:09:04,918] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:05,049] jepsen worker 82 - jepsen.util 892 :ok :read [27 4] INFO [2022-03-13 17:09:05,050] jepsen worker 82 - jepsen.util 892 :invoke :read [27 nil] INFO [2022-03-13 17:09:05,089] jepsen worker 85 - jepsen.util 985 :ok :read [28 44] INFO [2022-03-13 17:09:05,090] jepsen worker 85 - jepsen.util 985 :invoke :read [28 nil] INFO [2022-03-13 17:09:05,095] jepsen worker 83 - jepsen.util 713 :ok :read [27 4] INFO [2022-03-13 17:09:05,096] jepsen worker 83 - jepsen.util 713 :invoke :write [27 9] INFO [2022-03-13 17:09:05,113] jepsen worker 82 - jepsen.util 892 :ok :read [27 4] INFO [2022-03-13 17:09:05,113] jepsen worker 82 - jepsen.util 892 :invoke :read [27 nil] INFO [2022-03-13 17:09:05,142] jepsen worker 69 - jepsen.util 609 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,143] jepsen worker 69 - jepsen.util 609 :invoke :write [23 47] INFO [2022-03-13 17:09:05,153] jepsen worker 7 - jepsen.util 367 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,155] jepsen worker 7 - jepsen.util 367 :invoke :write [2 34] INFO [2022-03-13 17:09:05,158] jepsen worker 51 - jepsen.util 411 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,158] jepsen worker 51 - jepsen.util 411 :invoke :write [17 43] INFO [2022-03-13 17:09:05,170] jepsen worker 85 - jepsen.util 985 :ok :read [28 44] INFO [2022-03-13 17:09:05,170] jepsen worker 85 - jepsen.util 985 :invoke :read [28 nil] INFO [2022-03-13 17:09:05,213] jepsen worker 82 - jepsen.util 892 :ok :read [27 4] INFO [2022-03-13 17:09:05,214] jepsen worker 82 - jepsen.util 892 :invoke :write [27 33] INFO [2022-03-13 17:09:05,231] jepsen worker 23 - jepsen.util 473 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,236] jepsen worker 23 - jepsen.util 473 :invoke :write [7 27] INFO [2022-03-13 17:09:05,254] jepsen worker 0 - jepsen.util 810 :info :write [0 26] :RequestTimeoutException INFO [2022-03-13 17:09:05,255] jepsen worker 0 - jepsen.util 900 :invoke :write [0 11] INFO [2022-03-13 17:09:05,263] jepsen worker 85 - jepsen.util 985 :ok :read [28 44] INFO [2022-03-13 17:09:05,263] jepsen worker 85 - jepsen.util 985 :invoke :write [28 30] WARN [2022-03-13 17:09:05,406] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17663478,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":17663478,"total_server_duration_us":27,"operation_id":"0x1bed","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":17664486},{"operation_name":"upsert","last_dispatch_duration_us":6755966,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6755966,"total_server_duration_us":27043,"encode_duration_us":18908,"operation_id":"0x1de3","timeout_ms":10000,"last_server_duration_us":27043,"total_duration_us":10009653},{"operation_name":"upsert","last_dispatch_duration_us":13947925,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13947925,"total_server_duration_us":33622,"encode_duration_us":18939,"operation_id":"0x1e78","timeout_ms":10000,"last_server_duration_us":33622,"total_duration_us":10009160},{"operation_name":"upsert","last_dispatch_duration_us":16109310,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16539837,"total_server_duration_us":9284303,"encode_duration_us":17386,"operation_id":"0x1d2a","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10007700},{"operation_name":"upsert","last_dispatch_duration_us":8853836,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8853836,"total_server_duration_us":72181,"encode_duration_us":19137,"operation_id":"0x1fe2","timeout_ms":10000,"last_server_duration_us":72181,"total_duration_us":10006311},{"operation_name":"get","last_dispatch_duration_us":9096529,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9096529,"total_server_duration_us":27,"operation_id":"0x1e70","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9098269},{"operation_name":"get","last_dispatch_duration_us":8880802,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8880802,"total_server_duration_us":2448,"operation_id":"0x2083","timeout_ms":30000,"last_server_duration_us":2448,"total_duration_us":8882344},{"operation_name":"get","last_dispatch_duration_us":13593,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":6331224,"total_server_duration_us":119,"operation_id":"0x1f77","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":6339789},{"operation_name":"upsert","last_dispatch_duration_us":812854,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":812854,"total_server_duration_us":67353,"encode_duration_us":34153,"operation_id":"0x2198","timeout_ms":10000,"last_server_duration_us":67353,"total_duration_us":818159},{"operation_name":"upsert","last_dispatch_duration_us":764895,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":764895,"total_server_duration_us":110674,"encode_duration_us":18310,"operation_id":"0x20ea","timeout_ms":10000,"last_server_duration_us":110674,"total_duration_us":768907}],"total_count":14}} WARN [2022-03-13 17:09:05,406] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16120715,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16589583,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9307695,"encode_duration_us":21382,"operation_id":"0x1cff","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10006270},{"operation_name":"upsert","last_dispatch_duration_us":20986058,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20986058,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9026756,"encode_duration_us":17582,"operation_id":"0x1b1f","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008154},{"operation_name":"upsert","last_dispatch_duration_us":16109310,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16539837,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9284303,"encode_duration_us":17386,"operation_id":"0x1d2a","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10007700}],"total_count":3}} INFO [2022-03-13 17:09:05,475] jepsen worker 71 - jepsen.util 431 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,475] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:05,475] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,476] jepsen worker 22 - jepsen.util 472 :invoke :write [7 27] INFO [2022-03-13 17:09:05,476] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:05,477] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:05,522] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:05,522] jepsen worker 75 - jepsen.util 345 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,523] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:05,524] jepsen worker 75 - jepsen.util 345 :invoke :write [25 7] INFO [2022-03-13 17:09:05,525] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,525] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,526] jepsen worker 8 - jepsen.util 638 :invoke :write [2 15] INFO [2022-03-13 17:09:05,526] jepsen worker 52 - jepsen.util 502 :invoke :write [17 1] INFO [2022-03-13 17:09:05,590] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1290322580645161}, :ns_1@172.23.105.187 {:progress 0.1461988304093568}, :ns_1@172.23.107.227 {:progress 0.1290322580645161}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:05,590] jepsen worker nemesis - couchbase.util Rebalance check count: 58 INFO [2022-03-13 17:09:05,590] jepsen worker nemesis - couchbase.util Rebalance stuck count: 31 INFO [2022-03-13 17:09:05,613] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:05,614] jepsen worker 44 - jepsen.util 494 :invoke :write [14 15] INFO [2022-03-13 17:09:05,676] jepsen worker 18 - jepsen.util 648 :info :write [6 48] :RequestTimeoutException INFO [2022-03-13 17:09:05,677] jepsen worker 18 - jepsen.util 738 :invoke :read [6 nil] INFO [2022-03-13 17:09:05,701] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:05,702] jepsen worker 26 - jepsen.util 476 :invoke :write [8 12] INFO [2022-03-13 17:09:05,754] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:05,754] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:05,754] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:05,755] jepsen worker 78 - jepsen.util 438 :invoke :write [26 5] INFO [2022-03-13 17:09:05,760] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:05,761] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:05,773] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:05,773] jepsen worker 34 - jepsen.util 574 :invoke :write [11 30] INFO [2022-03-13 17:09:05,883] jepsen worker 70 - jepsen.util 700 :ok :read [23 30] INFO [2022-03-13 17:09:05,884] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:05,938] jepsen worker 61 - jepsen.util 421 :ok :read [20 12] INFO [2022-03-13 17:09:05,939] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:05,943] jepsen worker 60 - jepsen.util 600 :ok :read [20 12] INFO [2022-03-13 17:09:05,943] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:05,980] jepsen worker 62 - jepsen.util 422 :ok :read [20 12] INFO [2022-03-13 17:09:05,980] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:05,983] jepsen worker 71 - jepsen.util 431 :ok :read [23 30] INFO [2022-03-13 17:09:05,984] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:06,135] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,136] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:06,320] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:06,320] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,322] jepsen worker 61 - jepsen.util 421 :invoke :write [20 21] INFO [2022-03-13 17:09:06,323] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:06,362] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,362] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:06,397] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,399] jepsen worker 53 - jepsen.util 503 :invoke :write [17 36] INFO [2022-03-13 17:09:06,517] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:06,518] jepsen worker 42 - jepsen.util 402 :invoke :write [14 17] INFO [2022-03-13 17:09:06,550] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,550] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,550] jepsen worker 70 - jepsen.util 700 :invoke :write [23 2] INFO [2022-03-13 17:09:06,551] jepsen worker 71 - jepsen.util 431 :invoke :write [23 46] INFO [2022-03-13 17:09:06,597] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1290322580645161}, :ns_1@172.23.105.187 {:progress 0.1578947368421053}, :ns_1@172.23.107.227 {:progress 0.1290322580645161}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:06,597] jepsen worker nemesis - couchbase.util Rebalance check count: 59 INFO [2022-03-13 17:09:06,597] jepsen worker nemesis - couchbase.util Rebalance stuck count: 31 INFO [2022-03-13 17:09:06,630] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,631] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:06,820] jepsen worker 20 - jepsen.util 1010 :info :write [6 24] :RequestTimeoutException INFO [2022-03-13 17:09:06,821] jepsen worker 20 - jepsen.util 1100 :invoke :write [6 40] INFO [2022-03-13 17:09:06,828] jepsen worker 28 - jepsen.util 928 :info :write [9 11] :RequestTimeoutException INFO [2022-03-13 17:09:06,828] jepsen worker 28 - jepsen.util 1018 :invoke :read [9 nil] INFO [2022-03-13 17:09:06,900] jepsen worker 80 - jepsen.util 620 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,901] jepsen worker 53 - jepsen.util 503 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,902] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:06,902] jepsen worker 53 - jepsen.util 503 :invoke :write [17 12] WARN [2022-03-13 17:09:06,924] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":27222310,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27222310,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9039489,"encode_duration_us":23684,"operation_id":"0x19c9","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10005898}],"total_count":1}} INFO [2022-03-13 17:09:06,968] jepsen worker 6 - jepsen.util 546 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:06,969] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:07,003] jepsen worker 75 - jepsen.util 345 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,004] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] WARN [2022-03-13 17:09:07,008] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9245245,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9245245,"total_server_duration_us":324854,"operation_id":"0x1f82","timeout_ms":30000,"last_server_duration_us":324854,"total_duration_us":12370396},{"operation_name":"get","last_dispatch_duration_us":8359896,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":10597241,"total_server_duration_us":755,"operation_id":"0x1f08","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":11151834},{"operation_name":"get","last_dispatch_duration_us":8981030,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8981030,"total_server_duration_us":92,"operation_id":"0x2027","timeout_ms":30000,"last_server_duration_us":92,"total_duration_us":10573567},{"operation_name":"upsert","last_dispatch_duration_us":10505438,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10505438,"total_server_duration_us":9060729,"encode_duration_us":17043,"operation_id":"0x20a8","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10007838},{"operation_name":"upsert","last_dispatch_duration_us":476333,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5541779,"total_server_duration_us":300936,"encode_duration_us":22326,"operation_id":"0x1e00","timeout_ms":10000,"last_server_duration_us":28584,"total_duration_us":10007746},{"operation_name":"upsert","last_dispatch_duration_us":9074829,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9844657,"total_server_duration_us":9480746,"encode_duration_us":17418,"operation_id":"0x1da2","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10003114},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":19849,"operation_id":"0x1fe8","timeout_ms":10000,"total_duration_us":10002900},{"operation_name":"upsert","last_dispatch_duration_us":16356644,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":16356644,"total_server_duration_us":21895,"encode_duration_us":23711,"operation_id":"0x1dfd","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10000639},{"operation_name":"get","last_dispatch_duration_us":1324180,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1324180,"total_server_duration_us":38,"operation_id":"0x20c1","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":3430487},{"operation_name":"upsert","last_dispatch_duration_us":3200201,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3200201,"total_server_duration_us":714329,"encode_duration_us":27418,"operation_id":"0x20af","timeout_ms":10000,"last_server_duration_us":714329,"total_duration_us":3316964}],"total_count":63}} WARN [2022-03-13 17:09:07,024] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":6840586,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":6840586,"total_server_duration_us":22389,"encode_duration_us":18392,"operation_id":"0x1f81","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10012661},{"operation_name":"upsert","last_dispatch_duration_us":163600,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":337888,"total_server_duration_us":54898,"encode_duration_us":17529,"operation_id":"0x1eae","timeout_ms":10000,"last_server_duration_us":26953,"total_duration_us":10008293},{"operation_name":"upsert","last_dispatch_duration_us":29744,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":905006,"total_server_duration_us":514850,"encode_duration_us":17284,"operation_id":"0x2026","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10003460},{"operation_name":"upsert","last_dispatch_duration_us":7457660,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7640387,"total_server_duration_us":89485,"encode_duration_us":17945,"operation_id":"0x1f24","timeout_ms":10000,"last_server_duration_us":25188,"total_duration_us":10003452},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":18042,"operation_id":"0x1ede","timeout_ms":10000,"total_duration_us":10003243},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":19364,"operation_id":"0x212a","timeout_ms":10000,"total_duration_us":10003119},{"operation_name":"upsert","last_dispatch_duration_us":89663,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9083810,"total_server_duration_us":93043,"encode_duration_us":22717,"operation_id":"0x1eb2","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10003020},{"operation_name":"get","last_dispatch_duration_us":8581265,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8581265,"total_server_duration_us":32,"operation_id":"0x1edb","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8586593},{"operation_name":"get","last_dispatch_duration_us":8577026,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8577026,"total_server_duration_us":15,"operation_id":"0x1ecb","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":8579522},{"operation_name":"get","last_dispatch_duration_us":8570358,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8570358,"total_server_duration_us":27,"operation_id":"0x1eca","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":8573984}],"total_count":46}} WARN [2022-03-13 17:09:07,024] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18165894,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18165894,"total_server_duration_us":32,"operation_id":"0x1c32","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":18168061},{"operation_name":"get","last_dispatch_duration_us":14594621,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14594621,"total_server_duration_us":43,"operation_id":"0x1d01","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":14595752},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":19024,"operation_id":"0x1dac","timeout_ms":10000,"total_duration_us":10009141},{"operation_name":"upsert","last_dispatch_duration_us":7372963,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7372963,"total_server_duration_us":20045,"encode_duration_us":17561,"operation_id":"0x1fd2","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10008347},{"operation_name":"upsert","last_dispatch_duration_us":9027219,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9027219,"total_server_duration_us":20601,"encode_duration_us":28608,"operation_id":"0x1ef6","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10007779},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17723,"operation_id":"0x2047","timeout_ms":10000,"total_duration_us":10007514},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":16948,"operation_id":"0x1f67","timeout_ms":10000,"total_duration_us":10007482},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":19878,"operation_id":"0x1efd","timeout_ms":10000,"total_duration_us":10004576},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17514,"operation_id":"0x1e07","timeout_ms":10000,"total_duration_us":10000995},{"operation_name":"get","last_dispatch_duration_us":6877863,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6877863,"total_server_duration_us":7803,"operation_id":"0x20d5","timeout_ms":30000,"last_server_duration_us":7803,"total_duration_us":6881330}],"total_count":14}} WARN [2022-03-13 17:09:07,025] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":24248681,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24248681,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":7594001,"encode_duration_us":18106,"operation_id":"0x1b5d","timeout_ms":10000,"last_server_duration_us":7594001,"total_duration_us":10005552},{"operation_name":"upsert","last_dispatch_duration_us":24553782,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24553782,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9039489,"encode_duration_us":17992,"operation_id":"0x1a92","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10006525}],"total_count":2}} INFO [2022-03-13 17:09:07,041] jepsen worker 13 - jepsen.util 463 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,042] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:07,074] jepsen worker 37 - jepsen.util 757 :info :write [12 18] :RequestTimeoutException INFO [2022-03-13 17:09:07,075] jepsen worker 37 - jepsen.util 847 :invoke :write [12 19] INFO [2022-03-13 17:09:07,092] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:07,092] jepsen worker 21 - jepsen.util 651 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,094] jepsen worker 21 - jepsen.util 651 :invoke :write [7 34] INFO [2022-03-13 17:09:07,099] jepsen worker 24 - jepsen.util 474 :invoke :write [8 14] WARN [2022-03-13 17:09:07,103] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17320444,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17659323,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9231679,"encode_duration_us":18103,"operation_id":"0x1d9c","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10002166},{"operation_name":"upsert","last_dispatch_duration_us":15526818,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15526818,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9036305,"encode_duration_us":16922,"operation_id":"0x1c9e","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10007823},{"operation_name":"upsert","last_dispatch_duration_us":17101246,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17101246,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9047983,"encode_duration_us":29329,"operation_id":"0x1dfe","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10008074}],"total_count":3}} INFO [2022-03-13 17:09:07,106] jepsen worker 42 - jepsen.util 402 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,107] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] WARN [2022-03-13 17:09:07,109] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17395837,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17395837,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9032060,"encode_duration_us":18543,"operation_id":"0x1d90","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10001201},{"operation_name":"upsert","last_dispatch_duration_us":9074829,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9844657,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9480746,"encode_duration_us":17418,"operation_id":"0x1da2","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10003114},{"operation_name":"upsert","last_dispatch_duration_us":10505438,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10505438,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9060729,"encode_duration_us":17043,"operation_id":"0x20a8","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10007838}],"total_count":3}} WARN [2022-03-13 17:09:07,110] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":365316,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8027550,"total_server_duration_us":134061,"encode_duration_us":18096,"operation_id":"0x20dc","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10010868},{"operation_name":"upsert","last_dispatch_duration_us":29076,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8937325,"total_server_duration_us":238595,"encode_duration_us":24656,"operation_id":"0x1ef0","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10009160},{"operation_name":"upsert","last_dispatch_duration_us":34743,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":34743,"total_server_duration_us":23644,"encode_duration_us":17376,"operation_id":"0x1fdd","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10007409},{"operation_name":"upsert","last_dispatch_duration_us":49983,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":6351710,"total_server_duration_us":421658,"encode_duration_us":23793,"operation_id":"0x1fdc","timeout_ms":10000,"last_server_duration_us":31681,"total_duration_us":10007214},{"operation_name":"upsert","last_dispatch_duration_us":56840,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8128400,"total_server_duration_us":80275,"encode_duration_us":21243,"operation_id":"0x1f6b","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10006965},{"operation_name":"upsert","last_dispatch_duration_us":344286,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9793662,"total_server_duration_us":132827,"encode_duration_us":19932,"operation_id":"0x2039","timeout_ms":10000,"last_server_duration_us":31393,"total_duration_us":10006811},{"operation_name":"upsert","last_dispatch_duration_us":8134062,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":11478415,"total_server_duration_us":271946,"encode_duration_us":19340,"operation_id":"0x1db5","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10004888},{"operation_name":"upsert","last_dispatch_duration_us":7366593,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":8906196,"total_server_duration_us":7640083,"encode_duration_us":16671,"operation_id":"0x1e77","timeout_ms":10000,"last_server_duration_us":7348320,"total_duration_us":10004191},{"operation_name":"upsert","last_dispatch_duration_us":8106301,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9533290,"total_server_duration_us":262907,"encode_duration_us":18732,"operation_id":"0x1e74","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10003565},{"operation_name":"upsert","last_dispatch_duration_us":8107570,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9559730,"total_server_duration_us":252342,"encode_duration_us":21654,"operation_id":"0x1e76","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10002231}],"total_count":60}} INFO [2022-03-13 17:09:07,187] jepsen worker 44 - jepsen.util 494 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,188] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:07,190] jepsen worker 26 - jepsen.util 476 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,190] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] WARN [2022-03-13 17:09:07,204] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11368951,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11368951,"total_server_duration_us":49,"operation_id":"0x2029","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":11370936},{"operation_name":"upsert","last_dispatch_duration_us":13979061,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13979061,"total_server_duration_us":24069,"encode_duration_us":22377,"operation_id":"0x1f0d","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10010363},{"operation_name":"upsert","last_dispatch_duration_us":17101246,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17101246,"total_server_duration_us":9047983,"encode_duration_us":29329,"operation_id":"0x1dfe","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10008074},{"operation_name":"upsert","last_dispatch_duration_us":40205,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9625809,"total_server_duration_us":239006,"encode_duration_us":22714,"operation_id":"0x1e80","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10001388},{"operation_name":"get","last_dispatch_duration_us":9489051,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9489051,"total_server_duration_us":32,"operation_id":"0x20cb","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9490391},{"operation_name":"get","last_dispatch_duration_us":7162567,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7162567,"total_server_duration_us":32,"operation_id":"0x2197","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":7166300},{"operation_name":"get","last_dispatch_duration_us":6327123,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6327123,"total_server_duration_us":43,"operation_id":"0x1f8e","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":6329507},{"operation_name":"get","last_dispatch_duration_us":2283356,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2283356,"total_server_duration_us":15,"operation_id":"0x205c","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":4905007},{"operation_name":"get","last_dispatch_duration_us":939863,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":939863,"total_server_duration_us":19,"operation_id":"0x2218","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":4073634},{"operation_name":"get","last_dispatch_duration_us":2289384,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2289384,"total_server_duration_us":23,"operation_id":"0x20a6","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":3885617}],"total_count":62}} WARN [2022-03-13 17:09:07,210] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":20581676,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20581676,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9025695,"encode_duration_us":16943,"operation_id":"0x1c56","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10002159},{"operation_name":"upsert","last_dispatch_duration_us":9164994,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13962068,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9258505,"encode_duration_us":19536,"operation_id":"0x1d23","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10005590},{"operation_name":"upsert","last_dispatch_duration_us":7366593,"last_local_id":"A312307A00000006/000000005DE126B4","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":8906196,"last_local_socket":"172.23.104.79:33480","total_server_duration_us":7640083,"encode_duration_us":16671,"operation_id":"0x1e77","timeout_ms":10000,"last_server_duration_us":7348320,"total_duration_us":10004191}],"total_count":3}} INFO [2022-03-13 17:09:07,272] jepsen worker 25 - jepsen.util 385 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,273] jepsen worker 25 - jepsen.util 385 :invoke :write [8 35] INFO [2022-03-13 17:09:07,296] jepsen worker 76 - jepsen.util 526 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,297] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:07,328] jepsen worker 34 - jepsen.util 574 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,329] jepsen worker 34 - jepsen.util 574 :invoke :write [11 40] INFO [2022-03-13 17:09:07,336] jepsen worker 66 - jepsen.util 966 :info :write [22 22] :RequestTimeoutException INFO [2022-03-13 17:09:07,337] jepsen worker 66 - jepsen.util 1056 :invoke :read [22 nil] INFO [2022-03-13 17:09:07,384] jepsen worker 8 - jepsen.util 638 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,385] jepsen worker 8 - jepsen.util 638 :invoke :write [2 49] INFO [2022-03-13 17:09:07,431] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,431] jepsen worker 61 - jepsen.util 421 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,431] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,431] jepsen worker 62 - jepsen.util 422 :invoke :write [20 25] INFO [2022-03-13 17:09:07,432] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:07,432] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:07,470] jepsen worker 35 - jepsen.util 665 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,471] jepsen worker 71 - jepsen.util 431 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,471] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:07,472] jepsen worker 71 - jepsen.util 431 :invoke :write [23 15] INFO [2022-03-13 17:09:07,547] jepsen worker 53 - jepsen.util 503 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,547] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:07,547] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:07,548] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:07,549] jepsen worker 42 - jepsen.util 402 :invoke :write [14 22] INFO [2022-03-13 17:09:07,549] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:07,560] jepsen worker 78 - jepsen.util 438 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,560] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,561] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:07,561] jepsen worker 78 - jepsen.util 438 :invoke :write [26 2] INFO [2022-03-13 17:09:07,590] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,590] jepsen worker 14 - jepsen.util 194 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,591] jepsen worker 14 - jepsen.util 194 :invoke :write [4 43] INFO [2022-03-13 17:09:07,592] jepsen worker 76 - jepsen.util 526 :invoke :write [25 7] INFO [2022-03-13 17:09:07,604] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1290322580645161}, :ns_1@172.23.105.187 {:progress 0.1578947368421053}, :ns_1@172.23.107.227 {:progress 0.1290322580645161}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:07,604] jepsen worker nemesis - couchbase.util Rebalance check count: 60 INFO [2022-03-13 17:09:07,604] jepsen worker nemesis - couchbase.util Rebalance stuck count: 32 INFO [2022-03-13 17:09:07,638] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,639] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,639] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:07,639] jepsen worker 12 - jepsen.util 642 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,639] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:07,640] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:07,648] jepsen worker 70 - jepsen.util 700 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,648] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,649] jepsen worker 6 - jepsen.util 546 :invoke :write [2 33] INFO [2022-03-13 17:09:07,651] jepsen worker 70 - jepsen.util 700 :invoke :write [23 23] INFO [2022-03-13 17:09:07,714] jepsen worker 62 - jepsen.util 422 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,714] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:07,723] jepsen worker 52 - jepsen.util 502 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,724] jepsen worker 52 - jepsen.util 502 :invoke :write [17 25] INFO [2022-03-13 17:09:07,734] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:07,734] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,734] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:07,735] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:07,755] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:07,755] jepsen worker 35 - jepsen.util 665 :invoke :write [11 0] INFO [2022-03-13 17:09:07,770] jepsen worker 88 - jepsen.util 988 :ok :read [29 33] INFO [2022-03-13 17:09:07,771] jepsen worker 88 - jepsen.util 988 :invoke :write [29 14] INFO [2022-03-13 17:09:07,795] jepsen worker 42 - jepsen.util 402 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,795] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:09:07,833] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:07,833] jepsen worker 22 - jepsen.util 472 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,834] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:07,834] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:07,860] jepsen worker 76 - jepsen.util 526 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,860] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:07,892] jepsen worker 21 - jepsen.util 651 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,893] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:07,929] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,929] jepsen worker 14 - jepsen.util 194 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,929] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,929] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,930] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:07,930] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,929] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:07,930] jepsen worker 81 - jepsen.util 1071 :info :write [27 29] :RequestTimeoutException INFO [2022-03-13 17:09:07,931] jepsen worker 14 - jepsen.util 194 :invoke :write [4 48] INFO [2022-03-13 17:09:07,932] jepsen worker 75 - jepsen.util 345 :invoke :write [25 39] INFO [2022-03-13 17:09:07,934] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:07,935] jepsen worker 61 - jepsen.util 421 :invoke :write [20 36] INFO [2022-03-13 17:09:07,935] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:07,935] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:07,936] jepsen worker 53 - jepsen.util 503 :invoke :write [17 23] INFO [2022-03-13 17:09:07,937] jepsen worker 81 - jepsen.util 1161 :invoke :read [27 nil] INFO [2022-03-13 17:09:07,960] jepsen worker 64 - jepsen.util 604 :ok :read [21 :nil] INFO [2022-03-13 17:09:07,964] jepsen worker 64 - jepsen.util 604 :invoke :write [21 25] INFO [2022-03-13 17:09:07,994] jepsen worker 34 - jepsen.util 574 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:07,995] jepsen worker 34 - jepsen.util 574 :invoke :write [11 48] INFO [2022-03-13 17:09:08,020] jepsen worker 32 - jepsen.util 752 :ok :read [10 :nil] INFO [2022-03-13 17:09:08,020] jepsen worker 32 - jepsen.util 752 :invoke :read [10 nil] INFO [2022-03-13 17:09:08,034] jepsen worker 54 - jepsen.util 774 :ok :read [18 49] INFO [2022-03-13 17:09:08,034] jepsen worker 54 - jepsen.util 774 :invoke :read [18 nil] INFO [2022-03-13 17:09:08,086] jepsen worker 27 - jepsen.util 927 :info :write [9 2] :RequestTimeoutException INFO [2022-03-13 17:09:08,087] jepsen worker 27 - jepsen.util 1017 :invoke :read [9 nil] INFO [2022-03-13 17:09:08,140] jepsen worker 35 - jepsen.util 665 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,140] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:08,153] jepsen worker 25 - jepsen.util 385 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,154] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:08,216] jepsen worker 71 - jepsen.util 431 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,216] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:08,224] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,225] jepsen worker 42 - jepsen.util 402 :invoke :write [14 48] INFO [2022-03-13 17:09:08,225] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,225] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:08,225] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,225] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:08,226] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:08,226] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:08,227] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:08,228] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:08,259] jepsen worker 75 - jepsen.util 345 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,259] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:08,295] jepsen worker 8 - jepsen.util 638 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,295] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,296] jepsen worker 80 - jepsen.util 620 :invoke :write [26 4] INFO [2022-03-13 17:09:08,299] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:08,319] jepsen worker 53 - jepsen.util 503 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,320] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:08,368] jepsen worker 6 - jepsen.util 546 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,368] jepsen worker 6 - jepsen.util 546 :invoke :write [2 25] INFO [2022-03-13 17:09:08,374] jepsen worker 14 - jepsen.util 194 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,375] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:08,380] jepsen worker 86 - jepsen.util 806 :info :write [28 17] :RequestTimeoutException INFO [2022-03-13 17:09:08,380] jepsen worker 86 - jepsen.util 896 :invoke :read [28 nil] INFO [2022-03-13 17:09:08,421] jepsen worker 16 - jepsen.util 736 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:09:08,421] jepsen worker 4 - jepsen.util 814 :info :write [1 6] :RequestTimeoutException INFO [2022-03-13 17:09:08,422] jepsen worker 4 - jepsen.util 904 :invoke :read [1 nil] INFO [2022-03-13 17:09:08,423] jepsen worker 16 - jepsen.util 826 :invoke :write [5 21] INFO [2022-03-13 17:09:08,433] jepsen worker 61 - jepsen.util 421 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,433] jepsen worker 61 - jepsen.util 421 :invoke :write [20 33] INFO [2022-03-13 17:09:08,447] jepsen worker 40 - jepsen.util 670 :info :write [13 12] :SyncWriteAmbiguous INFO [2022-03-13 17:09:08,448] jepsen worker 40 - jepsen.util 760 :invoke :write [13 32] INFO [2022-03-13 17:09:08,449] jepsen worker 51 - jepsen.util 411 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,449] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:08,450] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:08,450] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,450] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:08,450] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,450] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:08,451] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:08,452] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:08,453] jepsen worker 13 - jepsen.util 463 :invoke :write [4 24] INFO [2022-03-13 17:09:08,454] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:08,454] jepsen worker 12 - jepsen.util 642 :invoke :write [4 16] INFO [2022-03-13 17:09:08,466] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:08,466] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:08,472] jepsen worker 52 - jepsen.util 502 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,473] jepsen worker 52 - jepsen.util 502 :invoke :write [17 18] INFO [2022-03-13 17:09:08,518] jepsen worker 42 - jepsen.util 402 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,518] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,518] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,519] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:08,519] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,519] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:08,519] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,519] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:08,521] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:08,521] jepsen worker 14 - jepsen.util 194 :invoke :write [4 24] INFO [2022-03-13 17:09:08,522] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:08,523] jepsen worker 42 - jepsen.util 402 :invoke :write [14 26] INFO [2022-03-13 17:09:08,524] jepsen worker 26 - jepsen.util 476 :invoke :write [8 0] INFO [2022-03-13 17:09:08,524] jepsen worker 62 - jepsen.util 422 :invoke :write [20 47] INFO [2022-03-13 17:09:08,575] jepsen worker 70 - jepsen.util 700 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,576] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:08,576] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:08,576] jepsen worker 70 - jepsen.util 700 :invoke :write [23 40] INFO [2022-03-13 17:09:08,577] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:08,577] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:08,580] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,582] jepsen worker 51 - jepsen.util 411 :invoke :write [17 44] INFO [2022-03-13 17:09:08,588] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] INFO [2022-03-13 17:09:08,589] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] INFO [2022-03-13 17:09:08,602] jepsen worker 61 - jepsen.util 421 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,603] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:08,611] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1290322580645161}, :ns_1@172.23.105.187 {:progress 0.1578947368421053}, :ns_1@172.23.107.227 {:progress 0.1290322580645161}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:08,611] jepsen worker nemesis - couchbase.util Rebalance check count: 61 INFO [2022-03-13 17:09:08,611] jepsen worker nemesis - couchbase.util Rebalance stuck count: 33 INFO [2022-03-13 17:09:08,654] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:08,655] jepsen worker 35 - jepsen.util 665 :invoke :write [11 37] INFO [2022-03-13 17:09:08,689] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:08,689] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:08,689] jepsen worker 78 - jepsen.util 438 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,690] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:08,691] jepsen worker 21 - jepsen.util 651 :invoke :write [7 20] INFO [2022-03-13 17:09:08,692] jepsen worker 25 - jepsen.util 385 :invoke :write [8 25] INFO [2022-03-13 17:09:08,719] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,719] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,719] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,719] jepsen worker 12 - jepsen.util 642 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,719] jepsen worker 75 - jepsen.util 345 :invoke :write [25 6] INFO [2022-03-13 17:09:08,720] jepsen worker 76 - jepsen.util 526 :invoke :write [25 12] INFO [2022-03-13 17:09:08,721] jepsen worker 53 - jepsen.util 503 :invoke :write [17 27] INFO [2022-03-13 17:09:08,721] jepsen worker 12 - jepsen.util 642 :invoke :write [4 3] INFO [2022-03-13 17:09:08,734] jepsen worker 34 - jepsen.util 574 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,734] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,736] jepsen worker 71 - jepsen.util 431 :invoke :write [23 16] INFO [2022-03-13 17:09:08,736] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:08,764] jepsen worker 45 - jepsen.util 1035 :info :write [15 19] :RequestTimeoutException INFO [2022-03-13 17:09:08,765] jepsen worker 45 - jepsen.util 1125 :invoke :write [15 47] INFO [2022-03-13 17:09:08,774] jepsen worker 24 - jepsen.util 474 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,774] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,776] jepsen worker 8 - jepsen.util 638 :invoke :write [2 37] INFO [2022-03-13 17:09:08,779] jepsen worker 24 - jepsen.util 474 :invoke :write [8 19] INFO [2022-03-13 17:09:08,834] jepsen worker 14 - jepsen.util 194 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,835] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:08,877] jepsen worker 13 - jepsen.util 463 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,878] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:08,883] jepsen worker 6 - jepsen.util 546 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,884] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:08,884] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:08,884] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:08,885] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:08,886] jepsen worker 6 - jepsen.util 546 :invoke :write [2 6] INFO [2022-03-13 17:09:08,920] jepsen worker 80 - jepsen.util 620 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,921] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:08,947] jepsen worker 70 - jepsen.util 700 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,948] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:08,959] jepsen worker 42 - jepsen.util 402 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:08,960] jepsen worker 42 - jepsen.util 402 :invoke :write [14 44] INFO [2022-03-13 17:09:08,977] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:08,978] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:09,008] jepsen worker 52 - jepsen.util 502 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,008] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,009] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,010] jepsen worker 78 - jepsen.util 438 :invoke :write [26 7] INFO [2022-03-13 17:09:09,012] jepsen worker 39 - jepsen.util 1029 :ok :read [13 39] INFO [2022-03-13 17:09:09,013] jepsen worker 39 - jepsen.util 1029 :invoke :write [13 21] INFO [2022-03-13 17:09:09,032] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,032] jepsen worker 62 - jepsen.util 422 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,033] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:09,033] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:09,073] jepsen worker 66 - jepsen.util 1056 :ok :read [22 30] INFO [2022-03-13 17:09:09,074] jepsen worker 66 - jepsen.util 1056 :invoke :read [22 nil] INFO [2022-03-13 17:09:09,092] jepsen worker 71 - jepsen.util 431 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,092] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:09,092] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:09,093] jepsen worker 22 - jepsen.util 472 :invoke :write [7 44] INFO [2022-03-13 17:09:09,095] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:09,095] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:09,106] jepsen worker 26 - jepsen.util 476 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,107] jepsen worker 68 - jepsen.util 788 :info :write [22 45] :RequestTimeoutException INFO [2022-03-13 17:09:09,109] jepsen worker 68 - jepsen.util 878 :invoke :write [22 16] INFO [2022-03-13 17:09:09,111] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:09,147] jepsen worker 66 - jepsen.util 1056 :ok :read [22 30] INFO [2022-03-13 17:09:09,147] jepsen worker 66 - jepsen.util 1056 :invoke :read [22 nil] INFO [2022-03-13 17:09:09,175] jepsen worker 66 - jepsen.util 1056 :ok :read [22 30] INFO [2022-03-13 17:09:09,175] jepsen worker 66 - jepsen.util 1056 :invoke :write [22 18] INFO [2022-03-13 17:09:09,178] jepsen worker 35 - jepsen.util 665 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,178] jepsen worker 35 - jepsen.util 665 :invoke :write [11 7] INFO [2022-03-13 17:09:09,191] jepsen worker 21 - jepsen.util 651 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,191] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:09,248] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,248] jepsen worker 6 - jepsen.util 546 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,249] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:09,250] jepsen worker 6 - jepsen.util 546 :invoke :write [2 34] INFO [2022-03-13 17:09:09,262] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,263] jepsen worker 71 - jepsen.util 431 :invoke :write [23 22] INFO [2022-03-13 17:09:09,264] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:09,264] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:09,275] jepsen worker 76 - jepsen.util 526 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,276] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:09,279] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:09,280] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:09,281] jepsen worker 25 - jepsen.util 385 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,282] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:09,316] jepsen worker 48 - jepsen.util 678 :info :write [16 9] :RequestTimeoutException INFO [2022-03-13 17:09:09,320] jepsen worker 48 - jepsen.util 768 :invoke :write [16 6] INFO [2022-03-13 17:09:09,331] jepsen worker 57 - jepsen.util 867 :info :write [19 32] :RequestTimeoutException INFO [2022-03-13 17:09:09,331] jepsen worker 57 - jepsen.util 957 :invoke :write [19 23] INFO [2022-03-13 17:09:09,336] jepsen worker 75 - jepsen.util 345 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,337] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:09,358] jepsen worker 22 - jepsen.util 472 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,359] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,359] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:09,359] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:09,360] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:09,360] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:09,382] jepsen worker 53 - jepsen.util 503 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,383] jepsen worker 53 - jepsen.util 503 :invoke :write [17 29] INFO [2022-03-13 17:09:09,398] jepsen worker 5 - jepsen.util 815 :info :write [1 18] :RequestTimeoutException INFO [2022-03-13 17:09:09,399] jepsen worker 5 - jepsen.util 905 :invoke :read [1 nil] INFO [2022-03-13 17:09:09,408] jepsen worker 59 - jepsen.util 779 :info :write [19 18] :RequestTimeoutException INFO [2022-03-13 17:09:09,409] jepsen worker 59 - jepsen.util 869 :invoke :write [19 7] INFO [2022-03-13 17:09:09,429] jepsen worker 5 - jepsen.util 905 :ok :read [1 42] INFO [2022-03-13 17:09:09,430] jepsen worker 5 - jepsen.util 905 :invoke :read [1 nil] INFO [2022-03-13 17:09:09,431] jepsen worker 8 - jepsen.util 638 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,433] jepsen worker 8 - jepsen.util 638 :invoke :write [2 8] INFO [2022-03-13 17:09:09,438] jepsen worker 84 - jepsen.util 1074 :info :write [28 49] :SyncWriteAmbiguous INFO [2022-03-13 17:09:09,440] jepsen worker 84 - jepsen.util 1164 :invoke :write [28 21] INFO [2022-03-13 17:09:09,450] jepsen worker 31 - jepsen.util 841 :info :write [10 20] :RequestTimeoutException INFO [2022-03-13 17:09:09,453] jepsen worker 31 - jepsen.util 931 :invoke :read [10 nil] INFO [2022-03-13 17:09:09,459] jepsen worker 5 - jepsen.util 905 :ok :read [1 42] INFO [2022-03-13 17:09:09,460] jepsen worker 5 - jepsen.util 905 :invoke :read [1 nil] INFO [2022-03-13 17:09:09,468] jepsen worker 12 - jepsen.util 642 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,468] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:09,469] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:09,469] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,470] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,471] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:09,472] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:09,473] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:09,473] jepsen worker 62 - jepsen.util 422 :invoke :write [20 48] INFO [2022-03-13 17:09:09,485] jepsen worker 51 - jepsen.util 411 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,485] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:09,485] jepsen worker 14 - jepsen.util 194 :invoke :write [4 21] INFO [2022-03-13 17:09:09,486] jepsen worker 5 - jepsen.util 905 :ok :read [1 42] INFO [2022-03-13 17:09:09,487] jepsen worker 5 - jepsen.util 905 :invoke :read [1 nil] INFO [2022-03-13 17:09:09,487] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,487] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:09,489] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,490] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:09,490] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,491] jepsen worker 13 - jepsen.util 463 :invoke :write [4 26] INFO [2022-03-13 17:09:09,498] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,499] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,500] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,500] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,502] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,502] jepsen worker 51 - jepsen.util 411 :invoke :write [17 18] INFO [2022-03-13 17:09:09,514] jepsen worker 5 - jepsen.util 905 :ok :read [1 42] INFO [2022-03-13 17:09:09,514] jepsen worker 5 - jepsen.util 905 :invoke :read [1 nil] INFO [2022-03-13 17:09:09,523] jepsen worker 6 - jepsen.util 546 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,523] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:09,533] jepsen worker 42 - jepsen.util 402 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,534] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:09:09,545] jepsen worker 5 - jepsen.util 905 :ok :read [1 42] INFO [2022-03-13 17:09:09,546] jepsen worker 5 - jepsen.util 905 :invoke :write [1 36] INFO [2022-03-13 17:09:09,563] jepsen worker 29 - jepsen.util 929 :info :write [9 29] :RequestTimeoutException INFO [2022-03-13 17:09:09,564] jepsen worker 29 - jepsen.util 1019 :invoke :read [9 nil] INFO [2022-03-13 17:09:09,585] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:09,586] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:09,618] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1348973607038123}, :ns_1@172.23.105.187 {:progress 0.1578947368421053}, :ns_1@172.23.107.227 {:progress 0.1348973607038123}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:09,619] jepsen worker nemesis - couchbase.util Rebalance check count: 62 INFO [2022-03-13 17:09:09,619] jepsen worker nemesis - couchbase.util Rebalance stuck count: 33 INFO [2022-03-13 17:09:09,626] jepsen worker 13 - jepsen.util 463 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,627] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:09,628] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,628] jepsen worker 35 - jepsen.util 665 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,629] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,629] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:09,630] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:09,630] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:09,642] jepsen worker 24 - jepsen.util 474 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,642] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:09,678] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:09,678] jepsen worker 51 - jepsen.util 411 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,679] jepsen worker 13 - jepsen.util 463 :invoke :write [4 42] INFO [2022-03-13 17:09:09,679] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,685] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,686] jepsen worker 51 - jepsen.util 411 :invoke :write [17 45] INFO [2022-03-13 17:09:09,758] jepsen worker 53 - jepsen.util 503 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,759] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:09,758] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,758] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:09,759] jepsen worker 12 - jepsen.util 642 :invoke :write [4 14] INFO [2022-03-13 17:09:09,760] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:09,761] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,761] jepsen worker 61 - jepsen.util 421 :invoke :write [20 20] INFO [2022-03-13 17:09:09,780] jepsen worker 71 - jepsen.util 431 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,780] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,780] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,780] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:09,781] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:09,781] jepsen worker 22 - jepsen.util 472 :invoke :write [7 43] INFO [2022-03-13 17:09:09,782] jepsen worker 70 - jepsen.util 700 :invoke :write [23 20] INFO [2022-03-13 17:09:09,784] jepsen worker 71 - jepsen.util 431 :invoke :write [23 1] INFO [2022-03-13 17:09:09,785] jepsen worker 44 - jepsen.util 494 :invoke :write [14 12] INFO [2022-03-13 17:09:09,787] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:09,793] jepsen worker 81 - jepsen.util 1161 :ok :read [27 4] INFO [2022-03-13 17:09:09,794] jepsen worker 81 - jepsen.util 1161 :invoke :read [27 nil] INFO [2022-03-13 17:09:09,816] jepsen worker 41 - jepsen.util 761 :info :write [13 2] :RequestTimeoutException INFO [2022-03-13 17:09:09,817] jepsen worker 41 - jepsen.util 851 :invoke :write [13 40] INFO [2022-03-13 17:09:09,836] jepsen worker 17 - jepsen.util 827 :info :write [5 3] :RequestTimeoutException INFO [2022-03-13 17:09:09,836] jepsen worker 13 - jepsen.util 463 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,837] jepsen worker 13 - jepsen.util 463 :invoke :write [4 37] INFO [2022-03-13 17:09:09,838] jepsen worker 17 - jepsen.util 917 :invoke :read [5 nil] INFO [2022-03-13 17:09:09,864] jepsen worker 78 - jepsen.util 438 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,864] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,865] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:09,865] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:09,866] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:09,867] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:09,867] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:09,868] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:09,888] jepsen worker 62 - jepsen.util 422 :fail :write [20 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,889] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:09,890] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,891] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:09,903] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:09,903] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:09,908] jepsen worker 51 - jepsen.util 411 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,908] jepsen worker 51 - jepsen.util 411 :invoke :write [17 32] INFO [2022-03-13 17:09:09,981] jepsen worker 13 - jepsen.util 463 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,982] jepsen worker 13 - jepsen.util 463 :invoke :write [4 9] INFO [2022-03-13 17:09:09,996] jepsen worker 14 - jepsen.util 194 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:09,997] jepsen worker 14 - jepsen.util 194 :invoke :write [4 20] INFO [2022-03-13 17:09:10,077] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,079] jepsen worker 42 - jepsen.util 402 :invoke :write [14 22] INFO [2022-03-13 17:09:10,085] jepsen worker 51 - jepsen.util 411 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,085] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:10,111] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,111] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:10,111] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,111] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,112] jepsen worker 76 - jepsen.util 526 :invoke :write [25 42] INFO [2022-03-13 17:09:10,113] jepsen worker 44 - jepsen.util 494 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,113] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:10,114] jepsen worker 8 - jepsen.util 638 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,115] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,115] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:10,115] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:10,115] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:10,115] jepsen worker 75 - jepsen.util 345 :invoke :write [25 49] INFO [2022-03-13 17:09:10,115] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,115] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,115] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:10,117] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,119] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:10,120] jepsen worker 8 - jepsen.util 638 :invoke :write [2 38] INFO [2022-03-13 17:09:10,121] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:10,124] jepsen worker 44 - jepsen.util 494 :invoke :write [14 39] INFO [2022-03-13 17:09:10,125] jepsen worker 78 - jepsen.util 438 :invoke :write [26 45] INFO [2022-03-13 17:09:10,127] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:10,128] jepsen worker 21 - jepsen.util 651 :invoke :write [7 16] INFO [2022-03-13 17:09:10,129] jepsen worker 80 - jepsen.util 620 :invoke :write [26 16] INFO [2022-03-13 17:09:10,129] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:10,132] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:10,132] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:10,133] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:10,133] jepsen worker 25 - jepsen.util 385 :invoke :write [8 25] INFO [2022-03-13 17:09:10,134] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:10,134] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,135] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:10,136] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:10,190] jepsen worker 13 - jepsen.util 463 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,190] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,190] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:10,191] jepsen worker 51 - jepsen.util 411 :invoke :write [17 49] INFO [2022-03-13 17:09:10,198] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:10,200] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:10,201] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:10,202] jepsen worker 61 - jepsen.util 421 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,202] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:10,203] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:10,203] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:10,204] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:10,205] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:10,206] jepsen worker 13 - jepsen.util 463 :invoke :write [4 29] INFO [2022-03-13 17:09:10,212] jepsen worker 22 - jepsen.util 472 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,213] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,214] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:10,215] jepsen worker 22 - jepsen.util 472 :invoke :write [7 40] INFO [2022-03-13 17:09:10,215] jepsen worker 52 - jepsen.util 502 :invoke :write [17 13] INFO [2022-03-13 17:09:10,216] jepsen worker 24 - jepsen.util 474 :invoke :write [8 36] INFO [2022-03-13 17:09:10,236] jepsen worker 49 - jepsen.util 769 :info :write [16 8] :RequestTimeoutException INFO [2022-03-13 17:09:10,237] jepsen worker 49 - jepsen.util 859 :invoke :read [16 nil] INFO [2022-03-13 17:09:10,287] jepsen worker 51 - jepsen.util 411 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,288] jepsen worker 51 - jepsen.util 411 :invoke :write [17 18] INFO [2022-03-13 17:09:10,310] jepsen worker 70 - jepsen.util 700 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,310] jepsen worker 70 - jepsen.util 700 :invoke :write [23 9] INFO [2022-03-13 17:09:10,311] jepsen worker 12 - jepsen.util 642 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,312] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,312] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:10,312] jepsen worker 62 - jepsen.util 422 :invoke :write [20 20] INFO [2022-03-13 17:09:10,600] jepsen worker 67 - jepsen.util 877 :info :write [22 22] :RequestTimeoutException INFO [2022-03-13 17:09:10,601] jepsen worker 67 - jepsen.util 967 :invoke :write [22 14] INFO [2022-03-13 17:09:10,624] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1407624633431085}, :ns_1@172.23.105.187 {:progress 0.1637426900584795}, :ns_1@172.23.107.227 {:progress 0.1407624633431085}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:10,624] jepsen worker nemesis - couchbase.util Rebalance check count: 63 INFO [2022-03-13 17:09:10,624] jepsen worker nemesis - couchbase.util Rebalance stuck count: 33 INFO [2022-03-13 17:09:10,739] jepsen worker 78 - jepsen.util 438 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,740] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:10,799] jepsen worker 13 - jepsen.util 463 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,800] jepsen worker 13 - jepsen.util 463 :invoke :write [4 38] INFO [2022-03-13 17:09:10,853] jepsen worker 14 - jepsen.util 194 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,853] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,853] jepsen worker 15 - jepsen.util 825 :info :write [5 26] :RequestTimeoutException INFO [2022-03-13 17:09:10,854] jepsen worker 53 - jepsen.util 503 :invoke :write [17 26] INFO [2022-03-13 17:09:10,854] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:10,855] jepsen worker 15 - jepsen.util 915 :invoke :write [5 9] INFO [2022-03-13 17:09:10,870] jepsen worker 51 - jepsen.util 411 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,871] jepsen worker 51 - jepsen.util 411 :invoke :write [17 17] INFO [2022-03-13 17:09:10,871] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,871] jepsen worker 71 - jepsen.util 431 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,872] jepsen worker 6 - jepsen.util 546 :invoke :write [2 36] INFO [2022-03-13 17:09:10,873] jepsen worker 71 - jepsen.util 431 :invoke :write [23 49] INFO [2022-03-13 17:09:10,926] jepsen worker 8 - jepsen.util 638 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,927] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:10,930] jepsen worker 42 - jepsen.util 402 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,930] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:10,932] jepsen worker 26 - jepsen.util 476 :invoke :write [8 0] INFO [2022-03-13 17:09:10,932] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:10,968] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:10,968] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:10,983] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:10,984] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:10,985] jepsen worker 13 - jepsen.util 463 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:10,986] jepsen worker 13 - jepsen.util 463 :invoke :write [4 31] INFO [2022-03-13 17:09:11,007] jepsen worker 76 - jepsen.util 526 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,008] jepsen worker 76 - jepsen.util 526 :invoke :write [25 46] INFO [2022-03-13 17:09:11,086] jepsen worker 58 - jepsen.util 778 :info :write [19 9] :RequestTimeoutException INFO [2022-03-13 17:09:11,087] jepsen worker 58 - jepsen.util 868 :invoke :read [19 nil] INFO [2022-03-13 17:09:11,093] jepsen worker 75 - jepsen.util 345 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,093] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,094] jepsen worker 75 - jepsen.util 345 :invoke :write [25 43] INFO [2022-03-13 17:09:11,094] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,094] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:11,095] jepsen worker 12 - jepsen.util 642 :invoke :write [4 5] INFO [2022-03-13 17:09:11,102] jepsen worker 51 - jepsen.util 411 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,102] jepsen worker 51 - jepsen.util 411 :invoke :write [17 29] INFO [2022-03-13 17:09:11,164] jepsen worker 44 - jepsen.util 494 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,164] jepsen worker 25 - jepsen.util 385 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,164] jepsen worker 25 - jepsen.util 385 :invoke :write [8 20] INFO [2022-03-13 17:09:11,165] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:11,176] jepsen worker 13 - jepsen.util 463 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,176] jepsen worker 62 - jepsen.util 422 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,177] jepsen worker 62 - jepsen.util 422 :invoke :write [20 6] INFO [2022-03-13 17:09:11,178] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,180] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,180] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,215] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,215] jepsen worker 51 - jepsen.util 411 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,216] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:11,216] jepsen worker 13 - jepsen.util 463 :invoke :write [4 2] INFO [2022-03-13 17:09:11,218] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,218] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:11,220] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,220] jepsen worker 51 - jepsen.util 411 :invoke :write [17 10] INFO [2022-03-13 17:09:11,262] jepsen worker 53 - jepsen.util 503 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,262] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:11,263] jepsen worker 53 - jepsen.util 503 :invoke :write [17 0] INFO [2022-03-13 17:09:11,263] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:11,323] jepsen worker 13 - jepsen.util 463 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,324] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,353] jepsen worker 26 - jepsen.util 476 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,354] jepsen worker 26 - jepsen.util 476 :invoke :write [8 44] INFO [2022-03-13 17:09:11,368] jepsen worker 22 - jepsen.util 472 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,368] jepsen worker 80 - jepsen.util 620 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,368] jepsen worker 22 - jepsen.util 472 :invoke :write [7 22] INFO [2022-03-13 17:09:11,369] jepsen worker 80 - jepsen.util 620 :invoke :write [26 19] INFO [2022-03-13 17:09:11,409] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:11,410] jepsen worker 35 - jepsen.util 665 :invoke :write [11 5] INFO [2022-03-13 17:09:11,432] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,433] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,434] jepsen worker 51 - jepsen.util 411 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,435] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,436] jepsen worker 13 - jepsen.util 463 :invoke :write [4 29] INFO [2022-03-13 17:09:11,436] jepsen worker 51 - jepsen.util 411 :invoke :write [17 29] INFO [2022-03-13 17:09:11,461] jepsen worker 70 - jepsen.util 700 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,462] jepsen worker 70 - jepsen.util 700 :invoke :write [23 2] INFO [2022-03-13 17:09:11,486] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,486] jepsen worker 76 - jepsen.util 526 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,487] jepsen worker 61 - jepsen.util 421 :invoke :write [20 45] INFO [2022-03-13 17:09:11,487] jepsen worker 76 - jepsen.util 526 :invoke :write [25 15] INFO [2022-03-13 17:09:11,526] jepsen worker 21 - jepsen.util 651 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,526] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,526] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:11,528] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:11,536] jepsen worker 3 - jepsen.util 633 :info :write [1 2] :SyncWriteAmbiguous INFO [2022-03-13 17:09:11,537] jepsen worker 3 - jepsen.util 723 :invoke :write [1 49] INFO [2022-03-13 17:09:11,577] jepsen worker 13 - jepsen.util 463 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,578] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,592] jepsen worker 75 - jepsen.util 345 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,593] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:11,596] jepsen worker 4 - jepsen.util 904 :ok :read [1 42] INFO [2022-03-13 17:09:11,598] jepsen worker 4 - jepsen.util 904 :invoke :write [1 2] INFO [2022-03-13 17:09:11,616] jepsen worker 71 - jepsen.util 431 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,617] jepsen worker 71 - jepsen.util 431 :invoke :write [23 44] INFO [2022-03-13 17:09:11,631] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1407624633431085}, :ns_1@172.23.105.187 {:progress 0.1637426900584795}, :ns_1@172.23.107.227 {:progress 0.1407624633431085}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:11,632] jepsen worker nemesis - couchbase.util Rebalance check count: 64 INFO [2022-03-13 17:09:11,632] jepsen worker nemesis - couchbase.util Rebalance stuck count: 34 INFO [2022-03-13 17:09:11,661] jepsen worker 52 - jepsen.util 502 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,662] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:11,671] jepsen worker 51 - jepsen.util 411 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,671] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,672] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:11,672] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,674] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,675] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,675] jepsen worker 51 - jepsen.util 411 :invoke :write [17 29] INFO [2022-03-13 17:09:11,676] jepsen worker 13 - jepsen.util 463 :invoke :write [4 26] INFO [2022-03-13 17:09:11,676] jepsen worker 12 - jepsen.util 642 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,676] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,677] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,677] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,776] jepsen worker 62 - jepsen.util 422 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,776] jepsen worker 62 - jepsen.util 422 :invoke :write [20 8] INFO [2022-03-13 17:09:11,791] jepsen worker 13 - jepsen.util 463 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,792] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,853] jepsen worker 6 - jepsen.util 546 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,854] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:11,857] jepsen worker 24 - jepsen.util 474 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,858] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:11,892] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:11,893] jepsen worker 44 - jepsen.util 494 :invoke :write [14 19] INFO [2022-03-13 17:09:11,910] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:11,911] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:11,921] jepsen worker 50 - jepsen.util 950 :info :write [16 21] :RequestTimeoutException INFO [2022-03-13 17:09:11,922] jepsen worker 50 - jepsen.util 1040 :invoke :write [16 3] INFO [2022-03-13 17:09:11,958] jepsen worker 2 - jepsen.util 992 :info :write [0 18] :RequestTimeoutException INFO [2022-03-13 17:09:11,958] jepsen worker 2 - jepsen.util 1082 :invoke :read [0 nil] INFO [2022-03-13 17:09:11,976] jepsen worker 51 - jepsen.util 411 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,977] jepsen worker 51 - jepsen.util 411 :invoke :write [17 25] INFO [2022-03-13 17:09:11,978] jepsen worker 53 - jepsen.util 503 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:11,978] jepsen worker 53 - jepsen.util 503 :invoke :write [17 27] INFO [2022-03-13 17:09:11,979] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,980] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:11,982] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:11,982] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:11,983] jepsen worker 42 - jepsen.util 402 :invoke :write [14 7] INFO [2022-03-13 17:09:11,984] jepsen worker 13 - jepsen.util 463 :invoke :write [4 38] INFO [2022-03-13 17:09:12,083] jepsen worker 35 - jepsen.util 665 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,084] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:12,110] jepsen worker 51 - jepsen.util 411 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,125] jepsen worker 22 - jepsen.util 472 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,125] jepsen worker 46 - jepsen.util 856 :info :write [15 21] :SyncWriteAmbiguous INFO [2022-03-13 17:09:12,129] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:12,130] jepsen worker 46 - jepsen.util 946 :invoke :read [15 nil] INFO [2022-03-13 17:09:12,131] jepsen worker 22 - jepsen.util 472 :invoke :write [7 26] INFO [2022-03-13 17:09:12,132] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,132] jepsen worker 80 - jepsen.util 620 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,133] jepsen worker 8 - jepsen.util 638 :invoke :write [2 21] INFO [2022-03-13 17:09:12,146] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:12,178] jepsen worker 56 - jepsen.util 776 :ok :read [18 49] INFO [2022-03-13 17:09:12,178] jepsen worker 56 - jepsen.util 776 :invoke :write [18 19] INFO [2022-03-13 17:09:12,228] jepsen worker 26 - jepsen.util 476 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,228] jepsen worker 26 - jepsen.util 476 :invoke :write [8 35] INFO [2022-03-13 17:09:12,240] jepsen worker 30 - jepsen.util 750 :info :write [10 33] :RequestTimeoutException INFO [2022-03-13 17:09:12,241] jepsen worker 30 - jepsen.util 840 :invoke :write [10 17] INFO [2022-03-13 17:09:12,272] jepsen worker 70 - jepsen.util 700 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,273] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:12,332] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,332] jepsen worker 13 - jepsen.util 463 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,333] jepsen worker 76 - jepsen.util 526 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,333] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:12,334] jepsen worker 13 - jepsen.util 463 :invoke :write [4 5] INFO [2022-03-13 17:09:12,334] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:12,334] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,335] jepsen worker 51 - jepsen.util 411 :invoke :write [17 11] INFO [2022-03-13 17:09:12,350] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:12,350] jepsen worker 25 - jepsen.util 385 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,351] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,351] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:12,350] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,351] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,352] jepsen worker 78 - jepsen.util 438 :invoke :write [26 25] INFO [2022-03-13 17:09:12,353] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:12,354] jepsen worker 24 - jepsen.util 474 :invoke :write [8 5] INFO [2022-03-13 17:09:12,355] jepsen worker 80 - jepsen.util 620 :invoke :write [26 12] INFO [2022-03-13 17:09:12,355] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:12,356] jepsen worker 52 - jepsen.util 502 :invoke :write [17 39] INFO [2022-03-13 17:09:12,398] jepsen worker 55 - jepsen.util 955 :ok :read [18 49] INFO [2022-03-13 17:09:12,399] jepsen worker 55 - jepsen.util 955 :invoke :read [18 nil] INFO [2022-03-13 17:09:12,404] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,404] jepsen worker 61 - jepsen.util 421 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,405] jepsen worker 75 - jepsen.util 345 :invoke :write [25 12] INFO [2022-03-13 17:09:12,406] jepsen worker 61 - jepsen.util 421 :invoke :write [20 40] INFO [2022-03-13 17:09:12,438] jepsen worker 51 - jepsen.util 411 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,438] jepsen worker 51 - jepsen.util 411 :invoke :write [17 0] INFO [2022-03-13 17:09:12,476] jepsen worker 62 - jepsen.util 422 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,477] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:12,507] jepsen worker 11 - jepsen.util 911 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,507] jepsen worker 10 - jepsen.util 730 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,508] jepsen worker 18 - jepsen.util 738 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,509] jepsen worker 18 - jepsen.util 738 :invoke :write [6 26] INFO [2022-03-13 17:09:12,509] jepsen worker 11 - jepsen.util 911 :invoke :read [3 nil] INFO [2022-03-13 17:09:12,510] jepsen worker 10 - jepsen.util 730 :invoke :read [3 nil] INFO [2022-03-13 17:09:12,514] jepsen worker 13 - jepsen.util 463 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,515] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:12,515] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:12,516] jepsen worker 35 - jepsen.util 665 :invoke :write [11 0] INFO [2022-03-13 17:09:12,525] jepsen worker 74 - jepsen.util 1064 :ok :read [24 24] INFO [2022-03-13 17:09:12,526] jepsen worker 74 - jepsen.util 1064 :invoke :write [24 8] INFO [2022-03-13 17:09:12,536] jepsen worker 9 - jepsen.util 729 :info :write [3 1] :SyncWriteAmbiguous INFO [2022-03-13 17:09:12,538] jepsen worker 9 - jepsen.util 819 :invoke :write [3 48] INFO [2022-03-13 17:09:12,556] jepsen worker 31 - jepsen.util 931 :ok :read [10 :nil] INFO [2022-03-13 17:09:12,557] jepsen worker 31 - jepsen.util 931 :invoke :write [10 21] INFO [2022-03-13 17:09:12,573] jepsen worker 27 - jepsen.util 1017 :ok :read [9 37] INFO [2022-03-13 17:09:12,574] jepsen worker 27 - jepsen.util 1017 :invoke :write [9 47] INFO [2022-03-13 17:09:12,574] jepsen worker 53 - jepsen.util 503 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,575] jepsen worker 53 - jepsen.util 503 :invoke :write [17 2] INFO [2022-03-13 17:09:12,577] jepsen worker 71 - jepsen.util 431 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,577] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,579] jepsen worker 6 - jepsen.util 546 :invoke :write [2 8] INFO [2022-03-13 17:09:12,585] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:12,619] jepsen worker 11 - jepsen.util 911 :ok :read [3 27] INFO [2022-03-13 17:09:12,620] jepsen worker 11 - jepsen.util 911 :invoke :read [3 nil] INFO [2022-03-13 17:09:12,621] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:12,621] jepsen worker 51 - jepsen.util 411 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,622] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:12,623] jepsen worker 51 - jepsen.util 411 :invoke :write [17 39] INFO [2022-03-13 17:09:12,623] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:12,624] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:12,625] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:12,625] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:12,626] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:12,627] jepsen worker 13 - jepsen.util 463 :invoke :write [4 40] INFO [2022-03-13 17:09:12,637] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1407624633431085}, :ns_1@172.23.105.187 {:progress 0.1637426900584795}, :ns_1@172.23.107.227 {:progress 0.1407624633431085}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:12,637] jepsen worker nemesis - couchbase.util Rebalance check count: 65 INFO [2022-03-13 17:09:12,638] jepsen worker nemesis - couchbase.util Rebalance stuck count: 35 INFO [2022-03-13 17:09:12,665] jepsen worker 10 - jepsen.util 730 :ok :read [3 27] INFO [2022-03-13 17:09:12,665] jepsen worker 10 - jepsen.util 730 :invoke :write [3 44] INFO [2022-03-13 17:09:12,672] jepsen worker 42 - jepsen.util 402 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,672] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:12,673] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:12,673] jepsen worker 42 - jepsen.util 402 :invoke :write [14 8] INFO [2022-03-13 17:09:12,674] jepsen worker 14 - jepsen.util 194 :invoke :write [4 31] INFO [2022-03-13 17:09:12,675] jepsen worker 12 - jepsen.util 642 :invoke :write [4 33] INFO [2022-03-13 17:09:12,680] jepsen worker 8 - jepsen.util 638 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,680] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:12,680] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:12,682] jepsen worker 21 - jepsen.util 651 :invoke :write [7 16] INFO [2022-03-13 17:09:12,684] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:12,685] jepsen worker 25 - jepsen.util 385 :invoke :write [8 48] INFO [2022-03-13 17:09:12,724] jepsen worker 26 - jepsen.util 476 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,724] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,725] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:12,726] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:12,727] jepsen worker 34 - jepsen.util 574 :invoke :write [11 8] INFO [2022-03-13 17:09:12,727] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:12,753] jepsen worker 51 - jepsen.util 411 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,754] jepsen worker 51 - jepsen.util 411 :invoke :write [17 32] INFO [2022-03-13 17:09:12,796] jepsen worker 75 - jepsen.util 345 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,797] jepsen worker 75 - jepsen.util 345 :invoke :write [25 1] INFO [2022-03-13 17:09:12,880] jepsen worker 13 - jepsen.util 463 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:12,881] jepsen worker 13 - jepsen.util 463 :invoke :write [4 31] INFO [2022-03-13 17:09:12,934] jepsen worker 46 - jepsen.util 946 :ok :read [15 48] INFO [2022-03-13 17:09:12,934] jepsen worker 46 - jepsen.util 946 :invoke :read [15 nil] INFO [2022-03-13 17:09:12,959] jepsen worker 11 - jepsen.util 911 :ok :read [3 27] INFO [2022-03-13 17:09:12,960] jepsen worker 11 - jepsen.util 911 :invoke :write [3 44] INFO [2022-03-13 17:09:13,046] jepsen worker 72 - jepsen.util 882 :ok :read [24 24] INFO [2022-03-13 17:09:13,047] jepsen worker 72 - jepsen.util 882 :invoke :read [24 nil] INFO [2022-03-13 17:09:13,086] jepsen worker 73 - jepsen.util 883 :info :write [24 25] :RequestTimeoutException INFO [2022-03-13 17:09:13,086] jepsen worker 87 - jepsen.util 1167 :info :write [29 12] :RequestTimeoutException INFO [2022-03-13 17:09:13,088] jepsen worker 87 - jepsen.util 1257 :invoke :write [29 7] INFO [2022-03-13 17:09:13,088] jepsen worker 73 - jepsen.util 973 :invoke :write [24 2] INFO [2022-03-13 17:09:13,092] jepsen worker 61 - jepsen.util 421 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,092] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,092] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:13,093] jepsen worker 62 - jepsen.util 422 :invoke :write [20 31] INFO [2022-03-13 17:09:13,096] jepsen worker 80 - jepsen.util 620 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,097] jepsen worker 80 - jepsen.util 620 :invoke :write [26 49] INFO [2022-03-13 17:09:13,098] jepsen worker 44 - jepsen.util 494 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,099] jepsen worker 44 - jepsen.util 494 :invoke :write [14 23] INFO [2022-03-13 17:09:13,106] jepsen worker 36 - jepsen.util 936 :info :write [12 7] :RequestTimeoutException INFO [2022-03-13 17:09:13,108] jepsen worker 36 - jepsen.util 1026 :invoke :read [12 nil] INFO [2022-03-13 17:09:13,109] jepsen worker 51 - jepsen.util 411 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,110] jepsen worker 51 - jepsen.util 411 :invoke :write [17 38] INFO [2022-03-13 17:09:13,142] jepsen worker 36 - jepsen.util 1026 :ok :read [12 2] INFO [2022-03-13 17:09:13,142] jepsen worker 36 - jepsen.util 1026 :invoke :read [12 nil] INFO [2022-03-13 17:09:13,159] jepsen worker 55 - jepsen.util 955 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,159] jepsen worker 72 - jepsen.util 882 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,159] jepsen worker 72 - jepsen.util 882 :invoke :write [24 30] INFO [2022-03-13 17:09:13,160] jepsen worker 55 - jepsen.util 955 :invoke :read [18 nil] INFO [2022-03-13 17:09:13,161] jepsen worker 55 - jepsen.util 955 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,162] jepsen worker 55 - jepsen.util 955 :invoke :read [18 nil] INFO [2022-03-13 17:09:13,163] jepsen worker 55 - jepsen.util 955 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,164] jepsen worker 55 - jepsen.util 955 :invoke :write [18 30] INFO [2022-03-13 17:09:13,173] jepsen worker 36 - jepsen.util 1026 :ok :read [12 2] INFO [2022-03-13 17:09:13,173] jepsen worker 36 - jepsen.util 1026 :invoke :read [12 nil] INFO [2022-03-13 17:09:13,199] jepsen worker 46 - jepsen.util 946 :ok :read [15 48] INFO [2022-03-13 17:09:13,200] jepsen worker 46 - jepsen.util 946 :invoke :read [15 nil] INFO [2022-03-13 17:09:13,200] jepsen worker 13 - jepsen.util 463 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,201] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,210] jepsen worker 36 - jepsen.util 1026 :ok :read [12 2] INFO [2022-03-13 17:09:13,211] jepsen worker 36 - jepsen.util 1026 :invoke :read [12 nil] INFO [2022-03-13 17:09:13,215] jepsen worker 35 - jepsen.util 665 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,216] jepsen worker 35 - jepsen.util 665 :invoke :write [11 33] INFO [2022-03-13 17:09:13,247] jepsen worker 36 - jepsen.util 1026 :ok :read [12 2] INFO [2022-03-13 17:09:13,248] jepsen worker 36 - jepsen.util 1026 :invoke :write [12 20] INFO [2022-03-13 17:09:13,284] jepsen worker 52 - jepsen.util 502 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,285] jepsen worker 22 - jepsen.util 472 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,285] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:13,285] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:13,285] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,286] jepsen worker 70 - jepsen.util 700 :invoke :write [23 18] INFO [2022-03-13 17:09:13,286] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,286] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:13,313] jepsen worker 51 - jepsen.util 411 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,313] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:13,314] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:13,314] jepsen worker 13 - jepsen.util 463 :invoke :write [4 11] INFO [2022-03-13 17:09:13,316] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,316] jepsen worker 51 - jepsen.util 411 :invoke :write [17 36] INFO [2022-03-13 17:09:13,331] jepsen worker 53 - jepsen.util 503 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,331] jepsen worker 53 - jepsen.util 503 :invoke :write [17 11] INFO [2022-03-13 17:09:13,367] jepsen worker 19 - jepsen.util 649 :info :write [6 21] :RequestTimeoutException INFO [2022-03-13 17:09:13,368] jepsen worker 19 - jepsen.util 739 :invoke :write [6 48] INFO [2022-03-13 17:09:13,406] jepsen worker 6 - jepsen.util 546 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,407] jepsen worker 6 - jepsen.util 546 :invoke :write [2 20] INFO [2022-03-13 17:09:13,409] jepsen worker 46 - jepsen.util 946 :ok :read [15 48] INFO [2022-03-13 17:09:13,410] jepsen worker 46 - jepsen.util 946 :invoke :write [15 27] INFO [2022-03-13 17:09:13,414] jepsen worker 13 - jepsen.util 463 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,415] jepsen worker 13 - jepsen.util 463 :invoke :write [4 48] INFO [2022-03-13 17:09:13,426] jepsen worker 14 - jepsen.util 194 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,426] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,465] jepsen worker 78 - jepsen.util 438 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,466] jepsen worker 78 - jepsen.util 438 :invoke :write [26 2] INFO [2022-03-13 17:09:13,484] jepsen worker 51 - jepsen.util 411 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,484] jepsen worker 51 - jepsen.util 411 :invoke :write [17 23] INFO [2022-03-13 17:09:13,509] jepsen worker 12 - jepsen.util 642 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,509] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,509] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,510] jepsen worker 61 - jepsen.util 421 :invoke :write [20 12] INFO [2022-03-13 17:09:13,568] jepsen worker 34 - jepsen.util 574 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,569] jepsen worker 34 - jepsen.util 574 :invoke :write [11 28] INFO [2022-03-13 17:09:13,579] jepsen worker 62 - jepsen.util 422 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,580] jepsen worker 62 - jepsen.util 422 :invoke :write [20 40] INFO [2022-03-13 17:09:13,589] jepsen worker 13 - jepsen.util 463 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,591] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,608] jepsen worker 24 - jepsen.util 474 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,608] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:13,644] jepsen worker 75 - jepsen.util 345 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,644] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1466275659824047}, :ns_1@172.23.105.187 {:progress 0.1812865497076024}, :ns_1@172.23.107.227 {:progress 0.1466275659824047}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:13,644] jepsen worker nemesis - couchbase.util Rebalance check count: 66 INFO [2022-03-13 17:09:13,644] jepsen worker nemesis - couchbase.util Rebalance stuck count: 35 INFO [2022-03-13 17:09:13,644] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:13,680] jepsen worker 38 - jepsen.util 1028 :info :write [12 44] :RequestTimeoutException INFO [2022-03-13 17:09:13,682] jepsen worker 38 - jepsen.util 1118 :invoke :read [12 nil] INFO [2022-03-13 17:09:13,683] jepsen worker 44 - jepsen.util 494 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,684] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:13,702] jepsen worker 35 - jepsen.util 665 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,704] jepsen worker 35 - jepsen.util 665 :invoke :write [11 37] INFO [2022-03-13 17:09:13,707] jepsen worker 38 - jepsen.util 1118 :ok :read [12 2] INFO [2022-03-13 17:09:13,708] jepsen worker 38 - jepsen.util 1118 :invoke :write [12 20] INFO [2022-03-13 17:09:13,740] jepsen worker 53 - jepsen.util 503 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,741] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:13,741] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:13,741] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:13,741] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,742] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,800] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:13,800] jepsen worker 21 - jepsen.util 651 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,800] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,801] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:13,802] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:13,803] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:13,841] jepsen worker 61 - jepsen.util 421 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,842] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:13,899] jepsen worker 62 - jepsen.util 422 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,899] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,900] jepsen worker 75 - jepsen.util 345 :invoke :write [25 43] INFO [2022-03-13 17:09:13,900] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:13,945] jepsen worker 25 - jepsen.util 385 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,946] jepsen worker 70 - jepsen.util 700 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,946] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,946] jepsen worker 35 - jepsen.util 665 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,946] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:13,947] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:13,947] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:13,947] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:13,948] jepsen worker 53 - jepsen.util 503 :invoke :write [17 49] INFO [2022-03-13 17:09:13,948] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:13,948] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:13,949] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,950] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:13,950] jepsen worker 35 - jepsen.util 665 :invoke :write [11 47] INFO [2022-03-13 17:09:14,010] jepsen worker 42 - jepsen.util 402 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,010] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,011] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:14,011] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,011] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,011] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:14,012] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:14,013] jepsen worker 76 - jepsen.util 526 :invoke :write [25 18] INFO [2022-03-13 17:09:14,014] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:14,014] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:14,016] jepsen worker 89 - jepsen.util 989 :info :write [29 30] :RequestTimeoutException INFO [2022-03-13 17:09:14,017] jepsen worker 89 - jepsen.util 1079 :invoke :write [29 3] INFO [2022-03-13 17:09:14,071] jepsen worker 6 - jepsen.util 546 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,071] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:14,076] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:14,076] jepsen worker 75 - jepsen.util 345 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,076] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:14,077] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:14,077] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:14,078] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:14,127] jepsen worker 80 - jepsen.util 620 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,127] jepsen worker 51 - jepsen.util 411 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,127] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:14,128] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:14,129] jepsen worker 13 - jepsen.util 463 :invoke :write [4 30] INFO [2022-03-13 17:09:14,131] jepsen worker 80 - jepsen.util 620 :invoke :write [26 21] INFO [2022-03-13 17:09:14,148] jepsen worker 53 - jepsen.util 503 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,149] jepsen worker 53 - jepsen.util 503 :invoke :write [17 1] INFO [2022-03-13 17:09:14,186] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:14,186] jepsen worker 34 - jepsen.util 574 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,186] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,186] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:14,186] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,187] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:14,187] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,188] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:14,188] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:14,189] jepsen worker 22 - jepsen.util 472 :invoke :write [7 39] INFO [2022-03-13 17:09:14,189] jepsen worker 6 - jepsen.util 546 :invoke :write [2 21] INFO [2022-03-13 17:09:14,190] jepsen worker 71 - jepsen.util 431 :invoke :write [23 31] INFO [2022-03-13 17:09:14,205] jepsen worker 83 - jepsen.util 713 :info :write [27 9] :SyncWriteAmbiguous INFO [2022-03-13 17:09:14,205] jepsen worker 83 - jepsen.util 803 :invoke :read [27 nil] INFO [2022-03-13 17:09:14,240] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,241] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:14,248] jepsen worker 35 - jepsen.util 665 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,248] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,248] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:14,248] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:14,248] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,250] jepsen worker 26 - jepsen.util 476 :invoke :write [8 33] INFO [2022-03-13 17:09:14,250] jepsen worker 62 - jepsen.util 422 :invoke :write [20 47] INFO [2022-03-13 17:09:14,250] jepsen worker 65 - jepsen.util 875 :info :write [21 36] :RequestTimeoutException INFO [2022-03-13 17:09:14,252] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:14,253] jepsen worker 65 - jepsen.util 965 :invoke :write [21 42] INFO [2022-03-13 17:09:14,254] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:09:14,255] jepsen worker 61 - jepsen.util 421 :invoke :write [20 39] INFO [2022-03-13 17:09:14,280] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:14,280] jepsen worker 76 - jepsen.util 526 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,280] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,280] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:14,281] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:14,282] jepsen worker 12 - jepsen.util 642 :invoke :write [4 30] INFO [2022-03-13 17:09:14,283] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:14,283] jepsen worker 76 - jepsen.util 526 :invoke :write [25 19] INFO [2022-03-13 17:09:14,339] jepsen worker 78 - jepsen.util 438 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,339] jepsen worker 78 - jepsen.util 438 :invoke :write [26 41] INFO [2022-03-13 17:09:14,341] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:14,341] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,341] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:14,341] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:14,342] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,342] jepsen worker 24 - jepsen.util 474 :invoke :write [8 45] INFO [2022-03-13 17:09:14,342] jepsen worker 21 - jepsen.util 651 :invoke :write [7 14] INFO [2022-03-13 17:09:14,343] jepsen worker 25 - jepsen.util 385 :invoke :write [8 16] INFO [2022-03-13 17:09:14,344] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:14,346] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:14,349] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:14,350] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:14,354] jepsen worker 53 - jepsen.util 503 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,355] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:14,381] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:14,381] jepsen worker 35 - jepsen.util 665 :invoke :write [11 49] INFO [2022-03-13 17:09:14,389] jepsen worker 62 - jepsen.util 422 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,389] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:14,396] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,403] jepsen worker 70 - jepsen.util 700 :invoke :write [23 0] INFO [2022-03-13 17:09:14,414] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:14,415] jepsen worker 34 - jepsen.util 574 :invoke :write [11 5] INFO [2022-03-13 17:09:14,440] jepsen worker 26 - jepsen.util 476 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,441] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:14,498] jepsen worker 33 - jepsen.util 573 :info :write [11 3] :RequestTimeoutException INFO [2022-03-13 17:09:14,498] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:14,540] jepsen worker 80 - jepsen.util 620 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,541] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,541] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,541] jepsen worker 80 - jepsen.util 620 :invoke :write [26 5] INFO [2022-03-13 17:09:14,542] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:14,542] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:14,559] jepsen worker 13 - jepsen.util 463 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,560] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,561] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:14,561] jepsen worker 13 - jepsen.util 463 :invoke :write [4 19] INFO [2022-03-13 17:09:14,588] jepsen worker 79 - jepsen.util 709 :info :write [26 7] :RequestTimeoutException INFO [2022-03-13 17:09:14,589] jepsen worker 79 - jepsen.util 799 :invoke :write [26 17] INFO [2022-03-13 17:09:14,597] jepsen worker 77 - jepsen.util 527 :info :write [25 14] :RequestTimeoutException INFO [2022-03-13 17:09:14,598] jepsen worker 77 - jepsen.util 617 :invoke :write [25 35] INFO [2022-03-13 17:09:14,615] jepsen worker 6 - jepsen.util 546 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,616] jepsen worker 6 - jepsen.util 546 :invoke :write [2 42] INFO [2022-03-13 17:09:14,617] jepsen worker 61 - jepsen.util 421 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,617] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:14,618] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:14,619] jepsen worker 14 - jepsen.util 194 :invoke :write [4 1] INFO [2022-03-13 17:09:14,620] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,621] jepsen worker 75 - jepsen.util 345 :invoke :write [25 18] INFO [2022-03-13 17:09:14,651] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1466275659824047}, :ns_1@172.23.105.187 {:progress 0.1812865497076024}, :ns_1@172.23.107.227 {:progress 0.1466275659824047}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:14,651] jepsen worker nemesis - couchbase.util Rebalance check count: 67 INFO [2022-03-13 17:09:14,651] jepsen worker nemesis - couchbase.util Rebalance stuck count: 36 INFO [2022-03-13 17:09:14,707] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,708] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,709] jepsen worker 42 - jepsen.util 402 :invoke :write [14 5] INFO [2022-03-13 17:09:14,709] jepsen worker 53 - jepsen.util 503 :invoke :write [17 8] INFO [2022-03-13 17:09:14,723] jepsen worker 23 - jepsen.util 473 :info :write [7 27] :SyncWriteAmbiguous INFO [2022-03-13 17:09:14,724] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:14,724] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,725] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:14,728] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:14,729] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:14,756] jepsen worker 21 - jepsen.util 651 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,756] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:14,795] jepsen worker 12 - jepsen.util 642 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,796] jepsen worker 12 - jepsen.util 642 :invoke :write [4 43] INFO [2022-03-13 17:09:14,812] jepsen worker 22 - jepsen.util 472 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,813] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:14,823] jepsen worker 69 - jepsen.util 609 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,823] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:14,847] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:14,847] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:09:14,858] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:14,859] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:14,887] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:14,887] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:15,011] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,011] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:09:15,016] jepsen worker 76 - jepsen.util 526 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,017] jepsen worker 76 - jepsen.util 526 :invoke :write [25 8] INFO [2022-03-13 17:09:15,019] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:15,019] jepsen worker 78 - jepsen.util 438 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,019] jepsen worker 71 - jepsen.util 431 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,020] jepsen worker 44 - jepsen.util 494 :invoke :write [14 21] INFO [2022-03-13 17:09:15,021] jepsen worker 71 - jepsen.util 431 :invoke :write [23 44] INFO [2022-03-13 17:09:15,021] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:15,058] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:15,059] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:15,112] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,112] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:15,115] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,116] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:15,133] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,134] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:15,135] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,136] jepsen worker 69 - jepsen.util 609 :invoke :write [23 36] INFO [2022-03-13 17:09:15,146] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:15,147] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:09:15,148] jepsen worker 70 - jepsen.util 700 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,149] jepsen worker 24 - jepsen.util 474 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,149] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:15,150] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:15,178] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,178] jepsen worker 35 - jepsen.util 665 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,178] jepsen worker 7 - jepsen.util 367 :info :write [2 34] :RequestTimeoutException INFO [2022-03-13 17:09:15,178] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:15,179] jepsen worker 62 - jepsen.util 422 :invoke :write [20 28] INFO [2022-03-13 17:09:15,179] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:15,180] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,180] jepsen worker 7 - jepsen.util 457 :invoke :write [2 28] INFO [2022-03-13 17:09:15,181] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:15,181] jepsen worker 35 - jepsen.util 665 :invoke :write [11 8] INFO [2022-03-13 17:09:15,191] jepsen worker 25 - jepsen.util 385 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,192] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:15,214] jepsen worker 34 - jepsen.util 574 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,215] jepsen worker 34 - jepsen.util 574 :invoke :write [11 31] INFO [2022-03-13 17:09:15,227] jepsen worker 69 - jepsen.util 609 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,228] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:15,244] jepsen worker 82 - jepsen.util 892 :info :write [27 33] :RequestTimeoutException INFO [2022-03-13 17:09:15,245] jepsen worker 82 - jepsen.util 982 :invoke :write [27 15] INFO [2022-03-13 17:09:15,277] jepsen worker 0 - jepsen.util 900 :info :write [0 11] :RequestTimeoutException INFO [2022-03-13 17:09:15,277] jepsen worker 0 - jepsen.util 990 :invoke :read [0 nil] INFO [2022-03-13 17:09:15,278] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:15,278] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:15,278] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,279] jepsen worker 60 - jepsen.util 600 :invoke :write [20 47] INFO [2022-03-13 17:09:15,279] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:15,280] jepsen worker 23 - jepsen.util 563 :invoke :write [7 1] INFO [2022-03-13 17:09:15,281] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:15,282] jepsen worker 43 - jepsen.util 403 :invoke :write [14 26] INFO [2022-03-13 17:09:15,284] jepsen worker 85 - jepsen.util 985 :info :write [28 30] :RequestTimeoutException INFO [2022-03-13 17:09:15,284] jepsen worker 85 - jepsen.util 1075 :invoke :read [28 nil] INFO [2022-03-13 17:09:15,322] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,322] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:15,322] jepsen worker 80 - jepsen.util 620 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,322] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,322] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:15,322] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:15,322] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,323] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:15,324] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:15,324] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:15,325] jepsen worker 8 - jepsen.util 638 :invoke :write [2 1] INFO [2022-03-13 17:09:15,325] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:15,326] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:15,327] jepsen worker 80 - jepsen.util 620 :invoke :write [26 30] INFO [2022-03-13 17:09:15,336] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,336] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,337] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:09:15,337] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:09:15,340] jepsen worker 14 - jepsen.util 194 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,341] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:15,393] jepsen worker 79 - jepsen.util 799 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,393] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] WARN [2022-03-13 17:09:15,430] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18096699,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24852666,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9092023,"encode_duration_us":18908,"operation_id":"0x1de3","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10009653}],"total_count":1}} WARN [2022-03-13 17:09:15,430] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14903826,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":14903826,"total_server_duration_us":8667,"operation_id":"0x217c","timeout_ms":30000,"last_server_duration_us":8667,"total_duration_us":14908325},{"operation_name":"upsert","last_dispatch_duration_us":28234,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":506688,"total_server_duration_us":287470,"encode_duration_us":20296,"operation_id":"0x225f","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10010665},{"operation_name":"upsert","last_dispatch_duration_us":15664360,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15664360,"total_server_duration_us":20841,"encode_duration_us":19897,"operation_id":"0x2128","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10008441},{"operation_name":"upsert","last_dispatch_duration_us":31792,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2992240,"total_server_duration_us":534502,"encode_duration_us":17567,"operation_id":"0x219e","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10007507},{"operation_name":"upsert","last_dispatch_duration_us":30929,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1057802,"total_server_duration_us":643754,"encode_duration_us":17007,"operation_id":"0x219f","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10006838},{"operation_name":"upsert","last_dispatch_duration_us":34839,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":862186,"total_server_duration_us":448782,"encode_duration_us":16939,"operation_id":"0x22e0","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10006686},{"operation_name":"upsert","last_dispatch_duration_us":28432,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2807113,"total_server_duration_us":467928,"encode_duration_us":17611,"operation_id":"0x21f1","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10004903},{"operation_name":"upsert","last_dispatch_duration_us":35277,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2983599,"total_server_duration_us":510316,"encode_duration_us":17774,"operation_id":"0x21a0","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10002272},{"operation_name":"get","last_dispatch_duration_us":9394304,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9394304,"total_server_duration_us":49,"operation_id":"0x2362","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9394787},{"operation_name":"get","last_dispatch_duration_us":9016917,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9016917,"total_server_duration_us":69,"operation_id":"0x2385","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":9017148}],"total_count":22}} INFO [2022-03-13 17:09:15,452] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,453] jepsen worker 69 - jepsen.util 609 :invoke :write [23 37] INFO [2022-03-13 17:09:15,530] jepsen worker 7 - jepsen.util 457 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,530] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:15,531] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,531] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:15,531] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:15,532] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:15,532] jepsen worker 7 - jepsen.util 457 :invoke :write [2 16] INFO [2022-03-13 17:09:15,533] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:15,600] jepsen worker 75 - jepsen.util 345 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,600] jepsen worker 60 - jepsen.util 600 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,601] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:15,602] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:15,624] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,625] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:15,652] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:15,653] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,652] jepsen worker 6 - jepsen.util 546 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,653] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:15,655] jepsen worker 6 - jepsen.util 546 :invoke :write [2 17] INFO [2022-03-13 17:09:15,656] jepsen worker 22 - jepsen.util 472 :invoke :write [7 46] INFO [2022-03-13 17:09:15,657] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1524926686217009}, :ns_1@172.23.105.187 {:progress 0.1812865497076024}, :ns_1@172.23.107.227 {:progress 0.1524926686217009}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:15,657] jepsen worker nemesis - couchbase.util Rebalance check count: 68 INFO [2022-03-13 17:09:15,657] jepsen worker nemesis - couchbase.util Rebalance stuck count: 36 INFO [2022-03-13 17:09:15,658] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:15,658] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:15,696] jepsen worker 13 - jepsen.util 463 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,697] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:15,755] jepsen worker 69 - jepsen.util 609 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,755] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:09:15,776] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:15,777] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:09:15,832] jepsen worker 44 - jepsen.util 494 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,833] jepsen worker 80 - jepsen.util 620 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,833] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:15,834] jepsen worker 80 - jepsen.util 620 :invoke :write [26 21] INFO [2022-03-13 17:09:15,848] jepsen worker 53 - jepsen.util 503 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,848] jepsen worker 23 - jepsen.util 563 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,849] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:15,850] jepsen worker 23 - jepsen.util 563 :invoke :write [7 8] INFO [2022-03-13 17:09:15,958] jepsen worker 8 - jepsen.util 638 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,959] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,961] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,961] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:09:15,962] jepsen worker 8 - jepsen.util 638 :invoke :write [2 6] INFO [2022-03-13 17:09:15,962] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:15,961] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:15,963] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:15,964] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:15,964] jepsen worker 25 - jepsen.util 385 :invoke :write [8 7] INFO [2022-03-13 17:09:15,995] jepsen worker 71 - jepsen.util 431 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:15,996] jepsen worker 71 - jepsen.util 431 :invoke :write [23 15] INFO [2022-03-13 17:09:16,020] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,020] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:16,028] jepsen worker 43 - jepsen.util 403 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,028] jepsen worker 42 - jepsen.util 402 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,028] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,029] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:16,030] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:16,030] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:16,061] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:16,061] jepsen worker 7 - jepsen.util 457 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,061] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,062] jepsen worker 7 - jepsen.util 457 :invoke :write [2 3] INFO [2022-03-13 17:09:16,062] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:16,063] jepsen worker 78 - jepsen.util 438 :invoke :write [26 5] INFO [2022-03-13 17:09:16,210] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,211] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,213] jepsen worker 77 - jepsen.util 617 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,213] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,213] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:16,214] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:16,214] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:16,215] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:16,216] jepsen worker 34 - jepsen.util 574 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,216] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,216] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,216] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:16,217] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,217] jepsen worker 79 - jepsen.util 799 :invoke :write [26 2] INFO [2022-03-13 17:09:16,236] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,236] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:09:16,247] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:16,248] jepsen worker 33 - jepsen.util 663 :invoke :write [11 1] INFO [2022-03-13 17:09:16,259] jepsen worker 76 - jepsen.util 526 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,259] jepsen worker 76 - jepsen.util 526 :invoke :write [25 46] INFO [2022-03-13 17:09:16,303] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:16,303] jepsen worker 80 - jepsen.util 620 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,304] jepsen worker 80 - jepsen.util 620 :invoke :write [26 21] INFO [2022-03-13 17:09:16,304] jepsen worker 21 - jepsen.util 651 :invoke :write [7 20] INFO [2022-03-13 17:09:16,318] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:16,318] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,318] jepsen worker 23 - jepsen.util 563 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,319] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:16,319] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:16,320] jepsen worker 43 - jepsen.util 403 :invoke :write [14 18] INFO [2022-03-13 17:09:16,326] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,327] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,329] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,329] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,329] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:16,330] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:16,330] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,330] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:16,330] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:16,331] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:16,331] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:16,332] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:16,332] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,332] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:16,333] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:16,333] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,334] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:16,334] jepsen worker 60 - jepsen.util 600 :invoke :write [20 5] INFO [2022-03-13 17:09:16,336] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,337] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:16,337] jepsen worker 51 - jepsen.util 411 :invoke :write [17 41] INFO [2022-03-13 17:09:16,338] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,338] jepsen worker 13 - jepsen.util 463 :invoke :write [4 13] INFO [2022-03-13 17:09:16,338] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:16,339] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:16,339] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:16,340] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,341] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:16,342] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,342] jepsen worker 77 - jepsen.util 617 :invoke :write [25 37] INFO [2022-03-13 17:09:16,349] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:16,350] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:09:16,357] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,358] jepsen worker 6 - jepsen.util 546 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,358] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:16,358] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:16,428] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,428] jepsen worker 33 - jepsen.util 663 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,429] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,429] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:16,431] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,431] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,437] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,437] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:09:16,457] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:16,457] jepsen worker 33 - jepsen.util 663 :invoke :write [11 15] INFO [2022-03-13 17:09:16,459] jepsen worker 51 - jepsen.util 411 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,460] jepsen worker 51 - jepsen.util 411 :invoke :write [17 0] INFO [2022-03-13 17:09:16,540] jepsen worker 62 - jepsen.util 422 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,541] jepsen worker 62 - jepsen.util 422 :invoke :write [20 36] INFO [2022-03-13 17:09:16,559] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,578] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,630] jepsen worker 13 - jepsen.util 463 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,630] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:16,640] jepsen worker 22 - jepsen.util 472 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,640] jepsen worker 8 - jepsen.util 638 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,640] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:16,640] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:16,641] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:16,642] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:16,667] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1524926686217009}, :ns_1@172.23.105.187 {:progress 0.1812865497076024}, :ns_1@172.23.107.227 {:progress 0.1524926686217009}, :ns_1@172.23.107.228 {:progress 0.1176470588235294}} INFO [2022-03-13 17:09:16,667] jepsen worker nemesis - couchbase.util Rebalance check count: 69 INFO [2022-03-13 17:09:16,667] jepsen worker nemesis - couchbase.util Rebalance stuck count: 37 INFO [2022-03-13 17:09:16,692] jepsen worker 77 - jepsen.util 617 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,693] jepsen worker 77 - jepsen.util 617 :invoke :write [25 25] INFO [2022-03-13 17:09:16,701] jepsen worker 33 - jepsen.util 663 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,701] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:16,706] jepsen worker 43 - jepsen.util 403 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,706] jepsen worker 43 - jepsen.util 403 :invoke :write [14 33] INFO [2022-03-13 17:09:16,714] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,714] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,718] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:16,718] jepsen worker 25 - jepsen.util 385 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,718] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:16,719] jepsen worker 24 - jepsen.util 474 :invoke :write [8 29] INFO [2022-03-13 17:09:16,733] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,734] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:09:16,752] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:16,753] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:09:16,772] jepsen worker 51 - jepsen.util 411 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,772] jepsen worker 51 - jepsen.util 411 :invoke :write [17 11] INFO [2022-03-13 17:09:16,773] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:16,773] jepsen worker 13 - jepsen.util 463 :invoke :write [4 33] INFO [2022-03-13 17:09:16,809] jepsen worker 71 - jepsen.util 431 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,810] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,824] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,824] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,838] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:16,838] jepsen worker 34 - jepsen.util 574 :invoke :write [11 5] INFO [2022-03-13 17:09:16,843] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,843] jepsen worker 35 - jepsen.util 665 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,843] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,843] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:16,843] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:16,843] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,844] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,844] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:16,844] jepsen worker 75 - jepsen.util 345 :invoke :write [25 49] INFO [2022-03-13 17:09:16,845] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,845] jepsen worker 53 - jepsen.util 503 :invoke :write [17 39] INFO [2022-03-13 17:09:16,851] jepsen worker 20 - jepsen.util 1100 :info :write [6 40] :RequestTimeoutException INFO [2022-03-13 17:09:16,851] jepsen worker 77 - jepsen.util 617 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,851] jepsen worker 61 - jepsen.util 421 :invoke :write [20 13] INFO [2022-03-13 17:09:16,852] jepsen worker 77 - jepsen.util 617 :invoke :write [25 22] INFO [2022-03-13 17:09:16,857] jepsen worker 20 - jepsen.util 1190 :invoke :read [6 nil] INFO [2022-03-13 17:09:16,927] jepsen worker 60 - jepsen.util 600 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,927] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:16,935] jepsen worker 78 - jepsen.util 438 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,936] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:16,939] jepsen worker 13 - jepsen.util 463 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,940] jepsen worker 13 - jepsen.util 463 :invoke :write [4 40] INFO [2022-03-13 17:09:16,954] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,954] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,955] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:16,956] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:16,974] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:16,975] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:09:16,990] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:16,991] jepsen worker 83 - jepsen.util 803 :ok :read [27 4] INFO [2022-03-13 17:09:16,991] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:16,992] jepsen worker 83 - jepsen.util 803 :invoke :read [27 nil] INFO [2022-03-13 17:09:17,020] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:17,020] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:17,029] jepsen worker 51 - jepsen.util 411 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,029] jepsen worker 51 - jepsen.util 411 :invoke :write [17 11] INFO [2022-03-13 17:09:17,049] jepsen worker 12 - jepsen.util 642 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,050] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:17,050] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:17,051] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:17,054] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:17,056] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] WARN [2022-03-13 17:09:17,059] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9178107,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10078083,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":77483,"encode_duration_us":20445,"operation_id":"0x2457","timeout_ms":10000,"last_server_duration_us":51946,"total_duration_us":10001540},{"operation_name":"upsert","last_dispatch_duration_us":16467378,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25424800,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9263944,"encode_duration_us":17712,"operation_id":"0x1c55","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10002594},{"operation_name":"upsert","last_dispatch_duration_us":10140452,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10140452,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":9088372,"encode_duration_us":19862,"operation_id":"0x2429","timeout_ms":10000,"last_server_duration_us":9088372,"total_duration_us":10007867},{"operation_name":"upsert","last_dispatch_duration_us":17619365,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17619365,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9055417,"encode_duration_us":18042,"operation_id":"0x1ede","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10003243},{"operation_name":"upsert","last_dispatch_duration_us":9083550,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9988556,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9577704,"encode_duration_us":17284,"operation_id":"0x2026","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10003460},{"operation_name":"upsert","last_dispatch_duration_us":17646933,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17984822,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9106066,"encode_duration_us":17529,"operation_id":"0x1eae","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008293},{"operation_name":"upsert","last_dispatch_duration_us":9344291,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10371966,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":96456,"encode_duration_us":26386,"operation_id":"0x240f","timeout_ms":10000,"last_server_duration_us":59625,"total_duration_us":10007874},{"operation_name":"upsert","last_dispatch_duration_us":9229699,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10499630,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":148294,"encode_duration_us":27180,"operation_id":"0x2430","timeout_ms":10000,"last_server_duration_us":59751,"total_duration_us":10009484},{"operation_name":"upsert","last_dispatch_duration_us":17630880,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27216934,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9091845,"encode_duration_us":20753,"operation_id":"0x1c00","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10005553}],"total_count":9}} INFO [2022-03-13 17:09:17,095] jepsen worker 63 - jepsen.util 873 :ok :read [21 :nil] INFO [2022-03-13 17:09:17,096] jepsen worker 63 - jepsen.util 873 :invoke :write [21 35] INFO [2022-03-13 17:09:17,103] jepsen worker 37 - jepsen.util 847 :info :write [12 19] :RequestTimeoutException INFO [2022-03-13 17:09:17,104] jepsen worker 37 - jepsen.util 937 :invoke :read [12 nil] INFO [2022-03-13 17:09:17,133] jepsen worker 37 - jepsen.util 937 :ok :read [12 2] INFO [2022-03-13 17:09:17,134] jepsen worker 37 - jepsen.util 937 :invoke :read [12 nil] INFO [2022-03-13 17:09:17,155] jepsen worker 79 - jepsen.util 799 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,156] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,156] jepsen worker 79 - jepsen.util 799 :invoke :write [26 8] INFO [2022-03-13 17:09:17,156] jepsen worker 32 - jepsen.util 752 :ok :read [10 :nil] INFO [2022-03-13 17:09:17,156] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:17,156] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:17,157] jepsen worker 37 - jepsen.util 937 :ok :read [12 2] INFO [2022-03-13 17:09:17,157] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:17,157] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:17,158] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,158] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,158] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] WARN [2022-03-13 17:09:17,159] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10846730,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10846730,"total_server_duration_us":62,"operation_id":"0x2132","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":10848019},{"operation_name":"upsert","last_dispatch_duration_us":34739,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":8418835,"total_server_duration_us":242512,"encode_duration_us":19527,"operation_id":"0x23ae","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10010499},{"operation_name":"upsert","last_dispatch_duration_us":10754178,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10754178,"total_server_duration_us":29788,"encode_duration_us":25593,"operation_id":"0x2199","timeout_ms":10000,"last_server_duration_us":29788,"total_duration_us":10009201},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":17811,"operation_id":"0x23f2","timeout_ms":10000,"total_duration_us":10008727},{"operation_name":"upsert","last_dispatch_duration_us":15051167,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":15051167,"total_server_duration_us":34116,"encode_duration_us":18878,"operation_id":"0x21f8","timeout_ms":10000,"last_server_duration_us":34116,"total_duration_us":10008416},{"operation_name":"upsert","last_dispatch_duration_us":12992089,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":12992089,"total_server_duration_us":20521,"encode_duration_us":19108,"operation_id":"0x22fb","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10006390},{"operation_name":"upsert","last_dispatch_duration_us":9037113,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9290900,"total_server_duration_us":9180250,"encode_duration_us":19566,"operation_id":"0x22f2","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9398610},{"operation_name":"get","last_dispatch_duration_us":8195510,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8195510,"total_server_duration_us":38,"operation_id":"0x2274","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":8197704},{"operation_name":"get","last_dispatch_duration_us":3158798,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":3158798,"total_server_duration_us":27,"operation_id":"0x252f","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":3159990},{"operation_name":"upsert","last_dispatch_duration_us":942356,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3001061,"total_server_duration_us":140347,"encode_duration_us":29420,"operation_id":"0x23e8","timeout_ms":10000,"last_server_duration_us":52777,"total_duration_us":3062081}],"total_count":76}} WARN [2022-03-13 17:09:17,159] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12876573,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12876573,"total_server_duration_us":69,"operation_id":"0x2099","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":12876977},{"operation_name":"get","last_dispatch_duration_us":9071230,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9071230,"total_server_duration_us":43,"operation_id":"0x2193","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":10158550},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":18735,"operation_id":"0x24c2","timeout_ms":10000,"total_duration_us":10010717},{"operation_name":"upsert","last_dispatch_duration_us":9229699,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":10499630,"total_server_duration_us":148294,"encode_duration_us":27180,"operation_id":"0x2430","timeout_ms":10000,"last_server_duration_us":59751,"total_duration_us":10009484},{"operation_name":"upsert","last_dispatch_duration_us":9344291,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":10371966,"total_server_duration_us":96456,"encode_duration_us":26386,"operation_id":"0x240f","timeout_ms":10000,"last_server_duration_us":59625,"total_duration_us":10007874},{"operation_name":"upsert","last_dispatch_duration_us":10140452,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":10140452,"total_server_duration_us":9088372,"encode_duration_us":19862,"operation_id":"0x2429","timeout_ms":10000,"last_server_duration_us":9088372,"total_duration_us":10007867},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":22839,"operation_id":"0x22e3","timeout_ms":10000,"total_duration_us":10007169},{"operation_name":"upsert","last_dispatch_duration_us":9022475,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9140715,"total_server_duration_us":47300,"encode_duration_us":18271,"operation_id":"0x21f5","timeout_ms":10000,"last_server_duration_us":25976,"total_duration_us":10005836},{"operation_name":"upsert","last_dispatch_duration_us":9178107,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":10078083,"total_server_duration_us":77483,"encode_duration_us":20445,"operation_id":"0x2457","timeout_ms":10000,"last_server_duration_us":51946,"total_duration_us":10001540},{"operation_name":"upsert","last_dispatch_duration_us":7753861,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":8328555,"total_server_duration_us":368143,"encode_duration_us":27886,"operation_id":"0x21f4","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10001470}],"total_count":21}} INFO [2022-03-13 17:09:17,159] jepsen worker 37 - jepsen.util 937 :invoke :write [12 13] INFO [2022-03-13 17:09:17,160] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] WARN [2022-03-13 17:09:17,160] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":23405881,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23405881,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9077736,"encode_duration_us":17514,"operation_id":"0x1e07","timeout_ms":10000,"last_server_duration_us":9077736,"total_duration_us":10000995},{"operation_name":"upsert","last_dispatch_duration_us":24204988,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24204988,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9076673,"encode_duration_us":19024,"operation_id":"0x1dac","timeout_ms":10000,"last_server_duration_us":9076673,"total_duration_us":10009141},{"operation_name":"upsert","last_dispatch_duration_us":18053628,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18053628,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9029938,"encode_duration_us":19878,"operation_id":"0x1efd","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10004576}],"total_count":3}} WARN [2022-03-13 17:09:17,160] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15162857,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":15162857,"total_server_duration_us":53974,"operation_id":"0x214f","timeout_ms":30000,"last_server_duration_us":53974,"total_duration_us":15165340},{"operation_name":"get","last_dispatch_duration_us":10530460,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10530460,"total_server_duration_us":49,"operation_id":"0x2310","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":10532940},{"operation_name":"upsert","last_dispatch_duration_us":14751898,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":14751898,"total_server_duration_us":79262,"encode_duration_us":17915,"operation_id":"0x219a","timeout_ms":10000,"last_server_duration_us":79262,"total_duration_us":10010949},{"operation_name":"upsert","last_dispatch_duration_us":37354,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":37354,"total_server_duration_us":21163,"encode_duration_us":20430,"operation_id":"0x228d","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10010915},{"operation_name":"upsert","last_dispatch_duration_us":103685,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":103685,"total_server_duration_us":22389,"encode_duration_us":17430,"operation_id":"0x2234","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10009327},{"operation_name":"upsert","last_dispatch_duration_us":33743,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1068589,"total_server_duration_us":631424,"encode_duration_us":21314,"operation_id":"0x2361","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10009176},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":30729,"operation_id":"0x2225","timeout_ms":10000,"total_duration_us":10008370},{"operation_name":"upsert","last_dispatch_duration_us":30426,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":30426,"total_server_duration_us":20045,"encode_duration_us":20420,"operation_id":"0x23cf","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10007349},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17135,"operation_id":"0x2226","timeout_ms":10000,"total_duration_us":10006256},{"operation_name":"upsert","last_dispatch_duration_us":9231078,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9636132,"total_server_duration_us":401018,"encode_duration_us":26243,"operation_id":"0x2455","timeout_ms":10000,"last_server_duration_us":65514,"total_duration_us":9653029}],"total_count":13}} WARN [2022-03-13 17:09:17,160] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":43993,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":718189,"total_server_duration_us":248160,"encode_duration_us":18558,"operation_id":"0x2459","timeout_ms":10000,"last_server_duration_us":24325,"total_duration_us":10010331},{"operation_name":"upsert","last_dispatch_duration_us":7425292,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12309399,"total_server_duration_us":264862,"encode_duration_us":36392,"operation_id":"0x227b","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10009318},{"operation_name":"upsert","last_dispatch_duration_us":29937,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":977180,"total_server_duration_us":573175,"encode_duration_us":18911,"operation_id":"0x213d","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10007152},{"operation_name":"upsert","last_dispatch_duration_us":8283455,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8962858,"total_server_duration_us":405174,"encode_duration_us":18605,"operation_id":"0x220d","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":10006993},{"operation_name":"upsert","last_dispatch_duration_us":78281,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":813583,"total_server_duration_us":406403,"encode_duration_us":16882,"operation_id":"0x21c6","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10003332},{"operation_name":"upsert","last_dispatch_duration_us":7610973,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8346716,"total_server_duration_us":248613,"encode_duration_us":16582,"operation_id":"0x245c","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10002963},{"operation_name":"upsert","last_dispatch_duration_us":9095110,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9095110,"total_server_duration_us":9043735,"encode_duration_us":29563,"operation_id":"0x2359","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":9102985},{"operation_name":"upsert","last_dispatch_duration_us":9084551,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9084551,"total_server_duration_us":9050106,"encode_duration_us":18263,"operation_id":"0x2454","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9088672},{"operation_name":"get","last_dispatch_duration_us":8340970,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8340970,"total_server_duration_us":56,"operation_id":"0x23c2","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8344056},{"operation_name":"upsert","last_dispatch_duration_us":238564,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2014944,"total_server_duration_us":131213,"encode_duration_us":25830,"operation_id":"0x23ce","timeout_ms":10000,"last_server_duration_us":35211,"total_duration_us":3632391}],"total_count":81}} INFO [2022-03-13 17:09:17,160] jepsen worker 32 - jepsen.util 752 :invoke :read [10 nil] INFO [2022-03-13 17:09:17,161] jepsen worker 70 - jepsen.util 700 :invoke :write [23 18] INFO [2022-03-13 17:09:17,180] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,181] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:17,194] jepsen worker 54 - jepsen.util 774 :ok :read [18 49] INFO [2022-03-13 17:09:17,195] jepsen worker 54 - jepsen.util 774 :invoke :write [18 44] INFO [2022-03-13 17:09:17,200] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,201] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:17,203] jepsen worker 29 - jepsen.util 1019 :ok :read [9 37] INFO [2022-03-13 17:09:17,203] jepsen worker 29 - jepsen.util 1019 :invoke :read [9 nil] INFO [2022-03-13 17:09:17,218] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,219] jepsen worker 69 - jepsen.util 609 :invoke :write [23 12] INFO [2022-03-13 17:09:17,234] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:17,235] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:17,251] jepsen worker 76 - jepsen.util 526 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,251] jepsen worker 76 - jepsen.util 526 :invoke :write [25 44] WARN [2022-03-13 17:09:17,259] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8508178,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8666052,"total_server_duration_us":63673,"encode_duration_us":17921,"operation_id":"0x2156","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10009995},{"operation_name":"upsert","last_dispatch_duration_us":8590473,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8650901,"total_server_duration_us":46730,"encode_duration_us":17483,"operation_id":"0x21f0","timeout_ms":10000,"last_server_duration_us":26685,"total_duration_us":10009966},{"operation_name":"upsert","last_dispatch_duration_us":223304,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9964774,"total_server_duration_us":214742,"encode_duration_us":17455,"operation_id":"0x2376","timeout_ms":10000,"last_server_duration_us":40163,"total_duration_us":10009110},{"operation_name":"upsert","last_dispatch_duration_us":266371,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10094014,"total_server_duration_us":240868,"encode_duration_us":19669,"operation_id":"0x235e","timeout_ms":10000,"last_server_duration_us":39846,"total_duration_us":10003512},{"operation_name":"upsert","last_dispatch_duration_us":6636915,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":6636915,"total_server_duration_us":21001,"encode_duration_us":27474,"operation_id":"0x235f","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10003376},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":17080,"operation_id":"0x245b","timeout_ms":10000,"total_duration_us":10003373},{"operation_name":"upsert","last_dispatch_duration_us":9331649,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9451411,"total_server_duration_us":9134499,"encode_duration_us":34896,"operation_id":"0x235d","timeout_ms":10000,"last_server_duration_us":9104337,"total_duration_us":9477201},{"operation_name":"upsert","last_dispatch_duration_us":9039274,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9039274,"total_server_duration_us":9023573,"encode_duration_us":21733,"operation_id":"0x2245","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9043660},{"operation_name":"upsert","last_dispatch_duration_us":693052,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2027828,"total_server_duration_us":315832,"encode_duration_us":18255,"operation_id":"0x2465","timeout_ms":10000,"last_server_duration_us":77290,"total_duration_us":2338237},{"operation_name":"get","last_dispatch_duration_us":236181,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":236181,"total_server_duration_us":62,"operation_id":"0x2516","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":1838007}],"total_count":72}} WARN [2022-03-13 17:09:17,259] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18226629,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18226629,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9330164,"encode_duration_us":19849,"operation_id":"0x1fe8","timeout_ms":10000,"last_server_duration_us":9330164,"total_duration_us":10002900}],"total_count":1}} INFO [2022-03-13 17:09:17,263] jepsen worker 77 - jepsen.util 617 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,263] jepsen worker 77 - jepsen.util 617 :invoke :write [25 47] INFO [2022-03-13 17:09:17,268] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:17,269] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:17,290] jepsen worker 32 - jepsen.util 752 :ok :read [10 :nil] INFO [2022-03-13 17:09:17,291] jepsen worker 32 - jepsen.util 752 :invoke :read [10 nil] INFO [2022-03-13 17:09:17,307] jepsen worker 43 - jepsen.util 403 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,307] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:17,307] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,308] jepsen worker 23 - jepsen.util 563 :invoke :write [7 19] INFO [2022-03-13 17:09:17,308] jepsen worker 60 - jepsen.util 600 :invoke :write [20 14] INFO [2022-03-13 17:09:17,308] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:17,323] jepsen worker 29 - jepsen.util 1019 :ok :read [9 37] INFO [2022-03-13 17:09:17,323] jepsen worker 29 - jepsen.util 1019 :invoke :read [9 nil] INFO [2022-03-13 17:09:17,339] jepsen worker 7 - jepsen.util 457 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,339] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,339] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,340] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:17,340] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:17,341] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:17,342] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:17,343] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:17,344] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:17,344] jepsen worker 43 - jepsen.util 403 :invoke :write [14 35] INFO [2022-03-13 17:09:17,393] jepsen worker 69 - jepsen.util 609 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,393] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:09:17,397] jepsen worker 13 - jepsen.util 463 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,398] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:17,404] jepsen worker 32 - jepsen.util 752 :ok :read [10 :nil] INFO [2022-03-13 17:09:17,405] jepsen worker 32 - jepsen.util 752 :invoke :write [10 37] INFO [2022-03-13 17:09:17,413] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:17,413] jepsen worker 34 - jepsen.util 574 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,413] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,413] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:17,414] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,414] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:17,415] jepsen worker 6 - jepsen.util 546 :invoke :write [2 14] INFO [2022-03-13 17:09:17,416] jepsen worker 44 - jepsen.util 494 :invoke :write [14 32] INFO [2022-03-13 17:09:17,417] jepsen worker 34 - jepsen.util 574 :invoke :write [11 49] INFO [2022-03-13 17:09:17,417] jepsen worker 62 - jepsen.util 422 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,417] jepsen worker 71 - jepsen.util 431 :invoke :write [23 38] INFO [2022-03-13 17:09:17,418] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:17,443] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:17,445] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:17,462] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:17,465] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:17,476] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:17,477] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:17,481] jepsen worker 29 - jepsen.util 1019 :ok :read [9 37] INFO [2022-03-13 17:09:17,481] jepsen worker 29 - jepsen.util 1019 :invoke :read [9 nil] INFO [2022-03-13 17:09:17,513] jepsen worker 51 - jepsen.util 411 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,514] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:17,538] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,539] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:09:17,578] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:17,578] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:09:17,622] jepsen worker 79 - jepsen.util 799 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,622] jepsen worker 24 - jepsen.util 474 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,623] jepsen worker 24 - jepsen.util 474 :invoke :write [8 41] INFO [2022-03-13 17:09:17,623] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:17,624] jepsen worker 77 - jepsen.util 617 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,624] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:17,624] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,625] jepsen worker 77 - jepsen.util 617 :invoke :write [25 42] INFO [2022-03-13 17:09:17,625] jepsen worker 75 - jepsen.util 345 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,627] jepsen worker 51 - jepsen.util 411 :invoke :write [17 38] INFO [2022-03-13 17:09:17,628] jepsen worker 75 - jepsen.util 345 :invoke :write [25 43] INFO [2022-03-13 17:09:17,628] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:17,632] jepsen worker 23 - jepsen.util 563 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,633] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:17,655] jepsen worker 86 - jepsen.util 896 :ok :read [28 44] INFO [2022-03-13 17:09:17,656] jepsen worker 86 - jepsen.util 896 :invoke :read [28 nil] INFO [2022-03-13 17:09:17,672] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.158357771260997}, :ns_1@172.23.105.187 {:progress 0.1871345029239766}, :ns_1@172.23.107.227 {:progress 0.158357771260997}, :ns_1@172.23.107.228 {:progress 0.1294117647058823}} INFO [2022-03-13 17:09:17,672] jepsen worker nemesis - couchbase.util Rebalance check count: 70 INFO [2022-03-13 17:09:17,672] jepsen worker nemesis - couchbase.util Rebalance stuck count: 37 INFO [2022-03-13 17:09:17,700] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:17,700] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,701] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:17,701] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:09:17,740] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] INFO [2022-03-13 17:09:17,740] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] INFO [2022-03-13 17:09:17,767] jepsen worker 21 - jepsen.util 651 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,768] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:17,799] jepsen worker 88 - jepsen.util 988 :info :write [29 14] :RequestTimeoutException INFO [2022-03-13 17:09:17,800] jepsen worker 88 - jepsen.util 1078 :invoke :read [29 nil] INFO [2022-03-13 17:09:17,820] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,821] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:17,822] jepsen worker 77 - jepsen.util 617 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,822] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:17,862] jepsen worker 70 - jepsen.util 700 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:17,862] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:17,863] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:17,863] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:17,987] jepsen worker 64 - jepsen.util 604 :info :write [21 25] :RequestTimeoutException INFO [2022-03-13 17:09:17,989] jepsen worker 64 - jepsen.util 694 :invoke :write [21 17] INFO [2022-03-13 17:09:18,035] jepsen worker 60 - jepsen.util 600 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,035] jepsen worker 61 - jepsen.util 421 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,036] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:18,036] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,037] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:09:18,037] jepsen worker 61 - jepsen.util 421 :invoke :write [20 13] INFO [2022-03-13 17:09:18,048] jepsen worker 51 - jepsen.util 411 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,048] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,048] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,050] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:18,050] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:18,050] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,051] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,051] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,051] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,052] jepsen worker 51 - jepsen.util 411 :invoke :write [17 14] INFO [2022-03-13 17:09:18,052] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,052] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:18,055] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,055] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,056] jepsen worker 77 - jepsen.util 617 :invoke :write [25 27] INFO [2022-03-13 17:09:18,059] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,062] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,063] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,065] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:18,066] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:09:18,069] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,069] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,075] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,076] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,112] jepsen worker 80 - jepsen.util 620 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,112] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,112] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,112] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,112] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,112] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:18,113] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:18,114] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:18,114] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:18,115] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:18,116] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:18,116] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:18,151] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,151] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:18,206] jepsen worker 6 - jepsen.util 546 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,207] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:18,233] jepsen worker 51 - jepsen.util 411 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,234] jepsen worker 51 - jepsen.util 411 :invoke :write [17 9] INFO [2022-03-13 17:09:18,258] jepsen worker 53 - jepsen.util 503 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,258] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:18,259] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,259] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,259] jepsen worker 43 - jepsen.util 403 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,259] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,259] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:18,260] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:09:18,261] jepsen worker 43 - jepsen.util 403 :invoke :write [14 43] INFO [2022-03-13 17:09:18,261] jepsen worker 60 - jepsen.util 600 :invoke :write [20 46] INFO [2022-03-13 17:09:18,262] jepsen worker 12 - jepsen.util 642 :invoke :write [4 49] INFO [2022-03-13 17:09:18,264] jepsen worker 14 - jepsen.util 194 :invoke :write [4 41] INFO [2022-03-13 17:09:18,270] jepsen worker 53 - jepsen.util 503 :invoke :write [17 11] INFO [2022-03-13 17:09:18,271] jepsen worker 42 - jepsen.util 402 :invoke :write [14 3] INFO [2022-03-13 17:09:18,285] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,286] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,289] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:09:18,291] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:09:18,313] jepsen worker 2 - jepsen.util 1082 :ok :read [0 41] INFO [2022-03-13 17:09:18,314] jepsen worker 2 - jepsen.util 1082 :invoke :read [0 nil] INFO [2022-03-13 17:09:18,324] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:18,324] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,324] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:18,324] jepsen worker 24 - jepsen.util 474 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,324] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,324] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,324] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,325] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,325] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:18,325] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:18,325] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:18,326] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:18,327] jepsen worker 8 - jepsen.util 638 :invoke :write [2 35] INFO [2022-03-13 17:09:18,327] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:18,327] jepsen worker 24 - jepsen.util 474 :invoke :write [8 27] INFO [2022-03-13 17:09:18,328] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:18,338] jepsen worker 76 - jepsen.util 526 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,338] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:18,338] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,338] jepsen worker 77 - jepsen.util 617 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,338] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,340] jepsen worker 26 - jepsen.util 476 :invoke :write [8 36] INFO [2022-03-13 17:09:18,341] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:18,342] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,342] jepsen worker 62 - jepsen.util 422 :invoke :write [20 30] INFO [2022-03-13 17:09:18,343] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:18,388] jepsen worker 2 - jepsen.util 1082 :ok :read [0 41] INFO [2022-03-13 17:09:18,388] jepsen worker 2 - jepsen.util 1082 :invoke :read [0 nil] INFO [2022-03-13 17:09:18,396] jepsen worker 35 - jepsen.util 665 :ok :read [11 7] INFO [2022-03-13 17:09:18,397] jepsen worker 35 - jepsen.util 665 :invoke :write [11 28] INFO [2022-03-13 17:09:18,421] jepsen worker 44 - jepsen.util 494 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,421] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:18,421] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,422] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,422] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,422] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:18,422] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,423] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:18,424] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:18,424] jepsen worker 52 - jepsen.util 502 :invoke :write [17 29] INFO [2022-03-13 17:09:18,426] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:18,426] jepsen worker 25 - jepsen.util 385 :invoke :write [8 12] INFO [2022-03-13 17:09:18,427] jepsen worker 21 - jepsen.util 651 :invoke :write [7 22] INFO [2022-03-13 17:09:18,428] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,428] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:09:18,433] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:18,444] jepsen worker 16 - jepsen.util 826 :info :write [5 21] :RequestTimeoutException INFO [2022-03-13 17:09:18,445] jepsen worker 16 - jepsen.util 916 :invoke :write [5 45] INFO [2022-03-13 17:09:18,453] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,453] jepsen worker 51 - jepsen.util 411 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,453] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,454] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,455] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:18,456] jepsen worker 77 - jepsen.util 617 :invoke :write [25 37] INFO [2022-03-13 17:09:18,456] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,456] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,457] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,458] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,458] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:18,458] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,459] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,459] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,462] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:18,463] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:18,464] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,465] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:18,465] jepsen worker 51 - jepsen.util 411 :invoke :write [17 35] INFO [2022-03-13 17:09:18,466] jepsen worker 2 - jepsen.util 1082 :ok :read [0 41] INFO [2022-03-13 17:09:18,467] jepsen worker 13 - jepsen.util 463 :invoke :write [4 26] INFO [2022-03-13 17:09:18,467] jepsen worker 2 - jepsen.util 1082 :invoke :read [0 nil] INFO [2022-03-13 17:09:18,474] jepsen worker 0 - jepsen.util 990 :ok :read [0 41] INFO [2022-03-13 17:09:18,476] jepsen worker 0 - jepsen.util 990 :invoke :write [0 45] INFO [2022-03-13 17:09:18,498] jepsen worker 40 - jepsen.util 760 :info :write [13 32] :RequestTimeoutException INFO [2022-03-13 17:09:18,500] jepsen worker 40 - jepsen.util 850 :invoke :write [13 41] INFO [2022-03-13 17:09:18,522] jepsen worker 2 - jepsen.util 1082 :ok :read [0 41] INFO [2022-03-13 17:09:18,523] jepsen worker 2 - jepsen.util 1082 :invoke :write [0 10] INFO [2022-03-13 17:09:18,561] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,562] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:09:18,568] jepsen worker 49 - jepsen.util 859 :ok :read [16 :nil] INFO [2022-03-13 17:09:18,569] jepsen worker 49 - jepsen.util 859 :invoke :read [16 nil] INFO [2022-03-13 17:09:18,596] jepsen worker 75 - jepsen.util 345 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,596] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,597] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:18,597] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:18,599] jepsen worker 77 - jepsen.util 617 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,600] jepsen worker 77 - jepsen.util 617 :invoke :write [25 12] INFO [2022-03-13 17:09:18,606] jepsen worker 34 - jepsen.util 574 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,606] jepsen worker 34 - jepsen.util 574 :invoke :write [11 41] INFO [2022-03-13 17:09:18,650] jepsen worker 8 - jepsen.util 638 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,650] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:18,663] jepsen worker 49 - jepsen.util 859 :ok :read [16 :nil] INFO [2022-03-13 17:09:18,664] jepsen worker 49 - jepsen.util 859 :invoke :write [16 15] INFO [2022-03-13 17:09:18,678] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.158357771260997}, :ns_1@172.23.105.187 {:progress 0.1871345029239766}, :ns_1@172.23.107.227 {:progress 0.158357771260997}, :ns_1@172.23.107.228 {:progress 0.1294117647058823}} INFO [2022-03-13 17:09:18,679] jepsen worker nemesis - couchbase.util Rebalance check count: 71 INFO [2022-03-13 17:09:18,679] jepsen worker nemesis - couchbase.util Rebalance stuck count: 38 INFO [2022-03-13 17:09:18,702] jepsen worker 51 - jepsen.util 411 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,703] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:18,704] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:18,705] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:18,780] jepsen worker 84 - jepsen.util 1164 :info :write [28 21] :SyncWriteAmbiguous INFO [2022-03-13 17:09:18,781] jepsen worker 84 - jepsen.util 1254 :invoke :read [28 nil] INFO [2022-03-13 17:09:18,786] jepsen worker 45 - jepsen.util 1125 :info :write [15 47] :RequestTimeoutException INFO [2022-03-13 17:09:18,789] jepsen worker 45 - jepsen.util 1215 :invoke :read [15 nil] INFO [2022-03-13 17:09:18,834] jepsen worker 81 - jepsen.util 1161 :ok :read [27 4] INFO [2022-03-13 17:09:18,835] jepsen worker 81 - jepsen.util 1161 :invoke :write [27 5] INFO [2022-03-13 17:09:18,909] jepsen worker 85 - jepsen.util 1075 :ok :read [28 44] INFO [2022-03-13 17:09:18,910] jepsen worker 85 - jepsen.util 1075 :invoke :write [28 11] INFO [2022-03-13 17:09:19,040] jepsen worker 39 - jepsen.util 1029 :info :write [13 21] :RequestTimeoutException INFO [2022-03-13 17:09:19,041] jepsen worker 39 - jepsen.util 1119 :invoke :read [13 nil] INFO [2022-03-13 17:09:19,106] jepsen worker 24 - jepsen.util 474 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 71 - jepsen.util 431 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,106] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:19,107] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:19,107] jepsen worker 24 - jepsen.util 474 :invoke :write [8 19] INFO [2022-03-13 17:09:19,108] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:19,108] jepsen worker 71 - jepsen.util 431 :invoke :write [23 24] INFO [2022-03-13 17:09:19,109] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:19,110] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:19,111] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,112] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:19,112] jepsen worker 22 - jepsen.util 472 :invoke :write [7 43] INFO [2022-03-13 17:09:19,113] jepsen worker 7 - jepsen.util 457 :invoke :write [2 31] INFO [2022-03-13 17:09:19,115] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,116] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,118] jepsen worker 78 - jepsen.util 438 :invoke :write [26 22] INFO [2022-03-13 17:09:19,119] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,119] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:09:19,120] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,120] jepsen worker 13 - jepsen.util 463 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,123] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,125] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,125] jepsen worker 13 - jepsen.util 463 :invoke :write [4 29] INFO [2022-03-13 17:09:19,128] jepsen worker 61 - jepsen.util 421 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,132] jepsen worker 60 - jepsen.util 600 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,135] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:19,136] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:19,138] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,140] jepsen worker 68 - jepsen.util 878 :info :write [22 16] :RequestTimeoutException INFO [2022-03-13 17:09:19,140] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,141] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,141] jepsen worker 68 - jepsen.util 968 :invoke :read [22 nil] INFO [2022-03-13 17:09:19,142] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,143] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,144] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,145] jepsen worker 69 - jepsen.util 609 :invoke :write [23 37] INFO [2022-03-13 17:09:19,205] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,205] jepsen worker 77 - jepsen.util 617 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,206] jepsen worker 51 - jepsen.util 411 :invoke :write [17 0] INFO [2022-03-13 17:09:19,206] jepsen worker 66 - jepsen.util 1056 :info :write [22 18] :RequestTimeoutException INFO [2022-03-13 17:09:19,206] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:19,207] jepsen worker 66 - jepsen.util 1146 :invoke :read [22 nil] INFO [2022-03-13 17:09:19,233] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,236] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:19,271] jepsen worker 13 - jepsen.util 463 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,271] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,272] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:19,273] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:19,292] jepsen worker 69 - jepsen.util 609 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,293] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,313] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,313] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,315] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,315] jepsen worker 69 - jepsen.util 609 :invoke :write [23 12] INFO [2022-03-13 17:09:19,327] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:19,328] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:19,348] jepsen worker 48 - jepsen.util 768 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:09:19,349] jepsen worker 48 - jepsen.util 858 :invoke :write [16 23] INFO [2022-03-13 17:09:19,358] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:19,359] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:19,360] jepsen worker 57 - jepsen.util 957 :info :write [19 23] :RequestTimeoutException INFO [2022-03-13 17:09:19,361] jepsen worker 57 - jepsen.util 1047 :invoke :read [19 nil] INFO [2022-03-13 17:09:19,381] jepsen worker 34 - jepsen.util 574 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,381] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:19,381] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:19,381] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,381] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,381] jepsen worker 52 - jepsen.util 502 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,381] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,381] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,381] jepsen worker 51 - jepsen.util 411 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,382] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:19,383] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:19,384] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:19,384] jepsen worker 14 - jepsen.util 194 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,384] jepsen worker 79 - jepsen.util 799 :invoke :write [26 32] INFO [2022-03-13 17:09:19,385] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:19,385] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:19,386] jepsen worker 51 - jepsen.util 411 :invoke :write [17 47] INFO [2022-03-13 17:09:19,387] jepsen worker 14 - jepsen.util 194 :invoke :write [4 45] INFO [2022-03-13 17:09:19,387] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,387] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:19,388] jepsen worker 13 - jepsen.util 463 :invoke :write [4 11] INFO [2022-03-13 17:09:19,389] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,389] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:19,390] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,391] jepsen worker 77 - jepsen.util 617 :invoke :write [25 48] INFO [2022-03-13 17:09:19,393] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:19,394] jepsen worker 43 - jepsen.util 403 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,395] jepsen worker 43 - jepsen.util 403 :invoke :write [14 11] INFO [2022-03-13 17:09:19,400] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:19,400] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,400] jepsen worker 23 - jepsen.util 563 :invoke :write [7 19] INFO [2022-03-13 17:09:19,401] jepsen worker 60 - jepsen.util 600 :invoke :write [20 12] INFO [2022-03-13 17:09:19,436] jepsen worker 59 - jepsen.util 869 :info :write [19 7] :RequestTimeoutException INFO [2022-03-13 17:09:19,437] jepsen worker 59 - jepsen.util 959 :invoke :read [19 nil] INFO [2022-03-13 17:09:19,440] jepsen worker 69 - jepsen.util 609 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,441] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:09:19,471] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:19,472] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:09:19,505] jepsen worker 77 - jepsen.util 617 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,506] jepsen worker 77 - jepsen.util 617 :invoke :write [25 2] INFO [2022-03-13 17:09:19,525] jepsen worker 12 - jepsen.util 642 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,525] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:19,527] jepsen worker 12 - jepsen.util 642 :invoke :write [4 3] INFO [2022-03-13 17:09:19,527] jepsen worker 76 - jepsen.util 526 :invoke :write [25 48] INFO [2022-03-13 17:09:19,573] jepsen worker 5 - jepsen.util 905 :info :write [1 36] :RequestTimeoutException INFO [2022-03-13 17:09:19,574] jepsen worker 5 - jepsen.util 995 :invoke :write [1 4] INFO [2022-03-13 17:09:19,684] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1642228739002932}, :ns_1@172.23.105.187 {:progress 0.1929824561403509}, :ns_1@172.23.107.227 {:progress 0.1642228739002932}, :ns_1@172.23.107.228 {:progress 0.1352941176470588}} INFO [2022-03-13 17:09:19,684] jepsen worker nemesis - couchbase.util Rebalance check count: 72 INFO [2022-03-13 17:09:19,684] jepsen worker nemesis - couchbase.util Rebalance stuck count: 38 INFO [2022-03-13 17:09:19,840] jepsen worker 41 - jepsen.util 851 :info :write [13 40] :RequestTimeoutException INFO [2022-03-13 17:09:19,841] jepsen worker 41 - jepsen.util 941 :invoke :write [13 3] INFO [2022-03-13 17:09:20,096] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,096] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:20,124] jepsen worker 25 - jepsen.util 385 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,124] jepsen worker 22 - jepsen.util 472 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,124] jepsen worker 25 - jepsen.util 385 :invoke :write [8 46] INFO [2022-03-13 17:09:20,125] jepsen worker 13 - jepsen.util 463 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,125] jepsen worker 22 - jepsen.util 472 :invoke :write [7 31] INFO [2022-03-13 17:09:20,125] jepsen worker 13 - jepsen.util 463 :invoke :write [4 2] INFO [2022-03-13 17:09:20,177] jepsen worker 23 - jepsen.util 563 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,177] jepsen worker 42 - jepsen.util 402 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,177] jepsen worker 42 - jepsen.util 402 :invoke :write [14 15] INFO [2022-03-13 17:09:20,178] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:20,216] jepsen worker 51 - jepsen.util 411 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,216] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:20,221] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,221] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,222] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:20,223] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:20,242] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,242] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:20,244] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,244] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:09:20,255] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:20,256] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:20,287] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:09:20,287] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:09:20,292] jepsen worker 77 - jepsen.util 617 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,292] jepsen worker 77 - jepsen.util 617 :invoke :write [25 26] INFO [2022-03-13 17:09:20,303] jepsen worker 71 - jepsen.util 431 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,303] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:20,303] jepsen worker 44 - jepsen.util 494 :invoke :write [14 33] INFO [2022-03-13 17:09:20,304] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:20,362] jepsen worker 35 - jepsen.util 665 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,362] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,362] jepsen worker 60 - jepsen.util 600 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,362] jepsen worker 60 - jepsen.util 600 :invoke :write [20 1] INFO [2022-03-13 17:09:20,363] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:20,363] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,372] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,372] jepsen worker 21 - jepsen.util 651 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,372] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,372] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:20,373] jepsen worker 8 - jepsen.util 638 :invoke :write [2 19] INFO [2022-03-13 17:09:20,373] jepsen worker 80 - jepsen.util 620 :invoke :write [26 19] INFO [2022-03-13 17:09:20,389] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,390] jepsen worker 69 - jepsen.util 609 :invoke :write [23 31] INFO [2022-03-13 17:09:20,401] jepsen worker 13 - jepsen.util 463 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,401] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,402] jepsen worker 13 - jepsen.util 463 :invoke :write [4 14] INFO [2022-03-13 17:09:20,402] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:20,415] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,415] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,416] jepsen worker 70 - jepsen.util 700 :invoke :write [23 26] INFO [2022-03-13 17:09:20,417] jepsen worker 6 - jepsen.util 546 :invoke :write [2 1] INFO [2022-03-13 17:09:20,432] jepsen worker 34 - jepsen.util 574 :ok :read [11 7] INFO [2022-03-13 17:09:20,433] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:20,492] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,493] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:09:20,507] jepsen worker 77 - jepsen.util 617 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,507] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,508] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,509] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:20,585] jepsen worker 26 - jepsen.util 476 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,585] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:20,585] jepsen worker 43 - jepsen.util 403 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,585] jepsen worker 43 - jepsen.util 403 :invoke :write [14 1] INFO [2022-03-13 17:09:20,586] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:20,586] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:20,605] jepsen worker 79 - jepsen.util 799 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,606] jepsen worker 69 - jepsen.util 609 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,606] jepsen worker 7 - jepsen.util 457 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,606] jepsen worker 79 - jepsen.util 799 :invoke :write [26 45] INFO [2022-03-13 17:09:20,607] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:20,607] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:09:20,616] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,616] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,616] jepsen worker 13 - jepsen.util 463 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,617] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:20,618] jepsen worker 51 - jepsen.util 411 :invoke :write [17 44] INFO [2022-03-13 17:09:20,618] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,619] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:20,619] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:20,619] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,620] jepsen worker 77 - jepsen.util 617 :invoke :write [25 11] INFO [2022-03-13 17:09:20,621] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:20,621] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:20,623] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:20,625] jepsen worker 3 - jepsen.util 723 :info :write [1 49] :SyncWriteAmbiguous INFO [2022-03-13 17:09:20,626] jepsen worker 67 - jepsen.util 967 :info :write [22 14] :RequestTimeoutException INFO [2022-03-13 17:09:20,626] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:20,627] jepsen worker 67 - jepsen.util 1057 :invoke :read [22 nil] INFO [2022-03-13 17:09:20,627] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:20,628] jepsen worker 3 - jepsen.util 813 :invoke :write [1 6] INFO [2022-03-13 17:09:20,629] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:20,630] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:20,631] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:20,632] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:20,633] jepsen worker 13 - jepsen.util 463 :invoke :write [4 22] INFO [2022-03-13 17:09:20,691] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1642228739002932}, :ns_1@172.23.105.187 {:progress 0.1929824561403509}, :ns_1@172.23.107.227 {:progress 0.1642228739002932}, :ns_1@172.23.107.228 {:progress 0.1352941176470588}} INFO [2022-03-13 17:09:20,691] jepsen worker nemesis - couchbase.util Rebalance check count: 73 INFO [2022-03-13 17:09:20,691] jepsen worker nemesis - couchbase.util Rebalance stuck count: 39 INFO [2022-03-13 17:09:20,692] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,693] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:20,733] jepsen worker 51 - jepsen.util 411 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,733] jepsen worker 51 - jepsen.util 411 :invoke :write [17 29] INFO [2022-03-13 17:09:20,764] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,764] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,766] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:09:20,766] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:09:20,788] jepsen worker 77 - jepsen.util 617 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,788] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,810] jepsen worker 60 - jepsen.util 600 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,810] jepsen worker 62 - jepsen.util 422 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,810] jepsen worker 60 - jepsen.util 600 :invoke :write [20 35] INFO [2022-03-13 17:09:20,811] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:20,812] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:20,812] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:20,851] jepsen worker 13 - jepsen.util 463 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,851] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:20,860] jepsen worker 22 - jepsen.util 472 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,860] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,860] jepsen worker 24 - jepsen.util 474 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,861] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,861] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:20,862] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:20,863] jepsen worker 22 - jepsen.util 472 :invoke :write [7 2] INFO [2022-03-13 17:09:20,864] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:20,881] jepsen worker 15 - jepsen.util 915 :info :write [5 9] :RequestTimeoutException INFO [2022-03-13 17:09:20,882] jepsen worker 15 - jepsen.util 1005 :invoke :write [5 10] INFO [2022-03-13 17:09:20,918] jepsen worker 51 - jepsen.util 411 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,918] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,918] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:20,919] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,920] jepsen worker 13 - jepsen.util 463 :invoke :write [4 40] INFO [2022-03-13 17:09:20,920] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,920] jepsen worker 51 - jepsen.util 411 :invoke :write [17 44] INFO [2022-03-13 17:09:20,921] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,922] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:09:20,922] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,923] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,924] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,925] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,926] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:20,927] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,927] jepsen worker 77 - jepsen.util 617 :invoke :write [25 39] INFO [2022-03-13 17:09:20,938] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,938] jepsen worker 69 - jepsen.util 609 :invoke :write [23 5] INFO [2022-03-13 17:09:20,982] jepsen worker 25 - jepsen.util 385 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,982] jepsen worker 44 - jepsen.util 494 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,982] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:20,983] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:20,984] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:20,985] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:20,985] jepsen worker 44 - jepsen.util 494 :invoke :write [14 46] INFO [2022-03-13 17:09:20,986] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:21,254] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,255] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:21,529] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:21,529] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,529] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:21,529] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,530] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:21,531] jepsen worker 52 - jepsen.util 502 :invoke :write [17 12] INFO [2022-03-13 17:09:21,532] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:21,533] jepsen worker 25 - jepsen.util 385 :invoke :write [8 33] INFO [2022-03-13 17:09:21,534] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:21,534] jepsen worker 21 - jepsen.util 651 :invoke :write [7 26] INFO [2022-03-13 17:09:21,620] jepsen worker 4 - jepsen.util 904 :info :write [1 2] :RequestTimeoutException INFO [2022-03-13 17:09:21,621] jepsen worker 4 - jepsen.util 994 :invoke :write [1 49] INFO [2022-03-13 17:09:21,689] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,689] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,689] jepsen worker 34 - jepsen.util 574 :invoke :write [11 36] INFO [2022-03-13 17:09:21,690] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:21,698] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1700879765395894}, :ns_1@172.23.105.187 {:progress 0.2046783625730995}, :ns_1@172.23.107.227 {:progress 0.1700879765395894}, :ns_1@172.23.107.228 {:progress 0.1411764705882353}} INFO [2022-03-13 17:09:21,698] jepsen worker nemesis - couchbase.util Rebalance check count: 74 INFO [2022-03-13 17:09:21,698] jepsen worker nemesis - couchbase.util Rebalance stuck count: 39 INFO [2022-03-13 17:09:21,764] jepsen worker 69 - jepsen.util 609 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,764] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:21,817] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,817] jepsen worker 13 - jepsen.util 463 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,818] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,818] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:21,819] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:21,820] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:09:21,827] jepsen worker 44 - jepsen.util 494 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,828] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:21,831] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,832] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:21,833] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,834] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:21,835] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,836] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:09:21,882] jepsen worker 8 - jepsen.util 638 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,883] jepsen worker 8 - jepsen.util 638 :invoke :write [2 30] INFO [2022-03-13 17:09:21,885] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:21,886] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:21,890] jepsen worker 77 - jepsen.util 617 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,891] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:21,944] jepsen worker 50 - jepsen.util 1040 :info :write [16 3] :RequestTimeoutException INFO [2022-03-13 17:09:21,944] jepsen worker 50 - jepsen.util 1130 :invoke :read [16 nil] INFO [2022-03-13 17:09:21,969] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:21,970] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:22,053] jepsen worker 51 - jepsen.util 411 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,053] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,053] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:22,054] jepsen worker 51 - jepsen.util 411 :invoke :write [17 23] INFO [2022-03-13 17:09:22,054] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:22,055] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:22,055] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:22,056] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,056] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:22,057] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:22,057] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:22,058] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,058] jepsen worker 13 - jepsen.util 463 :invoke :write [4 9] INFO [2022-03-13 17:09:22,058] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:22,060] jepsen worker 70 - jepsen.util 700 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,060] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,060] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:22,061] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:22,062] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,062] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:22,064] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,064] jepsen worker 77 - jepsen.util 617 :invoke :write [25 0] INFO [2022-03-13 17:09:22,064] jepsen worker 14 - jepsen.util 194 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,065] jepsen worker 14 - jepsen.util 194 :invoke :write [4 3] INFO [2022-03-13 17:09:22,076] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,076] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,077] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:22,077] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,078] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,079] jepsen worker 69 - jepsen.util 609 :invoke :write [23 37] INFO [2022-03-13 17:09:22,091] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,091] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,092] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,093] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:09:22,115] jepsen worker 52 - jepsen.util 502 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,117] jepsen worker 10 - jepsen.util 730 :info :write [3 44] :SyncWriteAmbiguous INFO [2022-03-13 17:09:22,117] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:22,118] jepsen worker 10 - jepsen.util 820 :invoke :write [3 7] INFO [2022-03-13 17:09:22,161] jepsen worker 13 - jepsen.util 463 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,162] jepsen worker 13 - jepsen.util 463 :invoke :write [4 10] INFO [2022-03-13 17:09:22,198] jepsen worker 69 - jepsen.util 609 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,198] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:22,204] jepsen worker 56 - jepsen.util 776 :info :write [18 19] :RequestTimeoutException INFO [2022-03-13 17:09:22,204] jepsen worker 56 - jepsen.util 866 :invoke :read [18 nil] INFO [2022-03-13 17:09:22,227] jepsen worker 51 - jepsen.util 411 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,227] jepsen worker 51 - jepsen.util 411 :invoke :write [17 19] INFO [2022-03-13 17:09:22,239] jepsen worker 43 - jepsen.util 403 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,239] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:22,240] jepsen worker 43 - jepsen.util 403 :invoke :write [14 20] INFO [2022-03-13 17:09:22,240] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:22,271] jepsen worker 30 - jepsen.util 840 :info :write [10 17] :RequestTimeoutException INFO [2022-03-13 17:09:22,272] jepsen worker 30 - jepsen.util 930 :invoke :read [10 nil] INFO [2022-03-13 17:09:22,274] jepsen worker 6 - jepsen.util 546 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,274] jepsen worker 25 - jepsen.util 385 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,274] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,275] jepsen worker 6 - jepsen.util 546 :invoke :write [2 30] INFO [2022-03-13 17:09:22,276] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:22,277] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:22,290] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,290] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,291] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,291] jepsen worker 69 - jepsen.util 609 :invoke :write [23 43] INFO [2022-03-13 17:09:22,293] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,294] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,295] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,295] jepsen worker 77 - jepsen.util 617 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,296] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,296] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:22,297] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,298] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:09:22,394] jepsen worker 69 - jepsen.util 609 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,395] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:22,417] jepsen worker 13 - jepsen.util 463 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,417] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:22,478] jepsen worker 51 - jepsen.util 411 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,478] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:22,478] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,478] jepsen worker 51 - jepsen.util 411 :invoke :write [17 49] INFO [2022-03-13 17:09:22,479] jepsen worker 13 - jepsen.util 463 :invoke :write [4 13] INFO [2022-03-13 17:09:22,480] jepsen worker 77 - jepsen.util 617 :invoke :write [25 49] INFO [2022-03-13 17:09:22,483] jepsen worker 21 - jepsen.util 651 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,483] jepsen worker 79 - jepsen.util 799 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,484] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:22,485] jepsen worker 21 - jepsen.util 651 :invoke :write [7 20] INFO [2022-03-13 17:09:22,488] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,488] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,489] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:09:22,490] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:09:22,551] jepsen worker 18 - jepsen.util 738 :info :write [6 26] :RequestTimeoutException INFO [2022-03-13 17:09:22,552] jepsen worker 18 - jepsen.util 828 :invoke :read [6 nil] INFO [2022-03-13 17:09:22,560] jepsen worker 74 - jepsen.util 1064 :info :write [24 8] :RequestTimeoutException INFO [2022-03-13 17:09:22,561] jepsen worker 74 - jepsen.util 1154 :invoke :write [24 32] INFO [2022-03-13 17:09:22,575] jepsen worker 22 - jepsen.util 472 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,575] jepsen worker 13 - jepsen.util 463 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,575] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:22,576] jepsen worker 13 - jepsen.util 463 :invoke :write [4 43] INFO [2022-03-13 17:09:22,578] jepsen worker 9 - jepsen.util 819 :info :write [3 48] :RequestTimeoutException INFO [2022-03-13 17:09:22,579] jepsen worker 9 - jepsen.util 909 :invoke :read [3 nil] INFO [2022-03-13 17:09:22,581] jepsen worker 31 - jepsen.util 931 :info :write [10 21] :RequestTimeoutException INFO [2022-03-13 17:09:22,581] jepsen worker 31 - jepsen.util 1021 :invoke :read [10 nil] INFO [2022-03-13 17:09:22,600] jepsen worker 27 - jepsen.util 1017 :info :write [9 47] :RequestTimeoutException INFO [2022-03-13 17:09:22,601] jepsen worker 27 - jepsen.util 1107 :invoke :read [9 nil] INFO [2022-03-13 17:09:22,614] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,614] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:09:22,619] jepsen worker 72 - jepsen.util 882 :info :write [24 30] :SyncWriteAmbiguous INFO [2022-03-13 17:09:22,620] jepsen worker 72 - jepsen.util 972 :invoke :read [24 nil] INFO [2022-03-13 17:09:22,622] jepsen worker 80 - jepsen.util 620 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,623] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,623] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:22,624] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:22,624] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:22,625] jepsen worker 7 - jepsen.util 457 :invoke :write [2 29] INFO [2022-03-13 17:09:22,644] jepsen worker 51 - jepsen.util 411 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,645] jepsen worker 51 - jepsen.util 411 :invoke :write [17 2] INFO [2022-03-13 17:09:22,704] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1700879765395894}, :ns_1@172.23.105.187 {:progress 0.2046783625730995}, :ns_1@172.23.107.227 {:progress 0.1700879765395894}, :ns_1@172.23.107.228 {:progress 0.1411764705882353}} INFO [2022-03-13 17:09:22,704] jepsen worker nemesis - couchbase.util Rebalance check count: 75 INFO [2022-03-13 17:09:22,704] jepsen worker nemesis - couchbase.util Rebalance stuck count: 40 INFO [2022-03-13 17:09:22,704] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,704] jepsen worker 34 - jepsen.util 574 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,704] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,705] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:22,705] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,706] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:22,706] jepsen worker 71 - jepsen.util 431 :invoke :write [23 16] INFO [2022-03-13 17:09:22,707] jepsen worker 77 - jepsen.util 617 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,707] jepsen worker 70 - jepsen.util 700 :invoke :write [23 41] INFO [2022-03-13 17:09:22,707] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,711] jepsen worker 77 - jepsen.util 617 :invoke :write [25 3] INFO [2022-03-13 17:09:22,712] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,765] jepsen worker 56 - jepsen.util 866 :ok :read [18 49] INFO [2022-03-13 17:09:22,766] jepsen worker 56 - jepsen.util 866 :invoke :read [18 nil] INFO [2022-03-13 17:09:22,775] jepsen worker 20 - jepsen.util 1190 :ok :read [6 16] INFO [2022-03-13 17:09:22,776] jepsen worker 20 - jepsen.util 1190 :invoke :read [6 nil] INFO [2022-03-13 17:09:22,782] jepsen worker 18 - jepsen.util 828 :ok :read [6 16] INFO [2022-03-13 17:09:22,783] jepsen worker 18 - jepsen.util 828 :invoke :read [6 nil] INFO [2022-03-13 17:09:22,787] jepsen worker 8 - jepsen.util 638 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,787] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:22,787] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:22,788] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:22,800] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,801] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:22,802] jepsen worker 27 - jepsen.util 1107 :ok :read [9 37] INFO [2022-03-13 17:09:22,803] jepsen worker 27 - jepsen.util 1107 :invoke :read [9 nil] INFO [2022-03-13 17:09:22,809] jepsen worker 13 - jepsen.util 463 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,809] jepsen worker 56 - jepsen.util 866 :ok :read [18 49] INFO [2022-03-13 17:09:22,810] jepsen worker 13 - jepsen.util 463 :invoke :write [4 36] INFO [2022-03-13 17:09:22,810] jepsen worker 56 - jepsen.util 866 :invoke :write [18 45] INFO [2022-03-13 17:09:22,842] jepsen worker 12 - jepsen.util 642 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,842] jepsen worker 60 - jepsen.util 600 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,843] jepsen worker 60 - jepsen.util 600 :invoke :write [20 11] INFO [2022-03-13 17:09:22,843] jepsen worker 12 - jepsen.util 642 :invoke :write [4 23] INFO [2022-03-13 17:09:22,846] jepsen worker 20 - jepsen.util 1190 :ok :read [6 16] INFO [2022-03-13 17:09:22,847] jepsen worker 20 - jepsen.util 1190 :invoke :write [6 33] INFO [2022-03-13 17:09:22,848] jepsen worker 18 - jepsen.util 828 :ok :read [6 16] INFO [2022-03-13 17:09:22,849] jepsen worker 18 - jepsen.util 828 :invoke :write [6 46] INFO [2022-03-13 17:09:22,854] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:22,859] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:22,863] jepsen worker 27 - jepsen.util 1107 :ok :read [9 37] INFO [2022-03-13 17:09:22,864] jepsen worker 27 - jepsen.util 1107 :invoke :write [9 9] INFO [2022-03-13 17:09:22,874] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,874] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,874] jepsen worker 69 - jepsen.util 609 :invoke :write [23 47] INFO [2022-03-13 17:09:22,875] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,879] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,880] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:09:22,886] jepsen worker 51 - jepsen.util 411 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,888] jepsen worker 51 - jepsen.util 411 :invoke :write [17 40] INFO [2022-03-13 17:09:22,956] jepsen worker 6 - jepsen.util 546 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,956] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,956] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,956] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:22,956] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:22,956] jepsen worker 21 - jepsen.util 651 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,957] jepsen worker 79 - jepsen.util 799 :invoke :write [26 31] INFO [2022-03-13 17:09:22,958] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:22,958] jepsen worker 21 - jepsen.util 651 :invoke :write [7 21] INFO [2022-03-13 17:09:22,959] jepsen worker 22 - jepsen.util 472 :invoke :write [7 31] INFO [2022-03-13 17:09:22,960] jepsen worker 6 - jepsen.util 546 :invoke :write [2 4] INFO [2022-03-13 17:09:22,963] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,969] jepsen worker 77 - jepsen.util 617 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,970] jepsen worker 77 - jepsen.util 617 :invoke :write [25 40] INFO [2022-03-13 17:09:22,978] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:22,979] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:22,980] jepsen worker 11 - jepsen.util 911 :info :write [3 44] :RequestTimeoutException INFO [2022-03-13 17:09:22,981] jepsen worker 11 - jepsen.util 1001 :invoke :read [3 nil] INFO [2022-03-13 17:09:23,030] jepsen worker 13 - jepsen.util 463 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,031] jepsen worker 13 - jepsen.util 463 :invoke :write [4 37] INFO [2022-03-13 17:09:23,083] jepsen worker 78 - jepsen.util 438 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,084] jepsen worker 71 - jepsen.util 431 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,084] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,084] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:23,084] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:23,085] jepsen worker 71 - jepsen.util 431 :invoke :write [23 31] INFO [2022-03-13 17:09:23,085] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:23,086] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:23,097] jepsen worker 51 - jepsen.util 411 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,097] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:23,110] jepsen worker 87 - jepsen.util 1257 :info :write [29 7] :RequestTimeoutException INFO [2022-03-13 17:09:23,110] jepsen worker 87 - jepsen.util 1347 :invoke :write [29 41] INFO [2022-03-13 17:09:23,114] jepsen worker 73 - jepsen.util 973 :info :write [24 2] :RequestTimeoutException INFO [2022-03-13 17:09:23,115] jepsen worker 73 - jepsen.util 1063 :invoke :read [24 nil] INFO [2022-03-13 17:09:23,134] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,135] jepsen worker 69 - jepsen.util 609 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,135] jepsen worker 33 - jepsen.util 663 :invoke :write [11 11] INFO [2022-03-13 17:09:23,136] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:09:23,196] jepsen worker 55 - jepsen.util 955 :info :write [18 30] :RequestTimeoutException INFO [2022-03-13 17:09:23,197] jepsen worker 55 - jepsen.util 1045 :invoke :read [18 nil] INFO [2022-03-13 17:09:23,276] jepsen worker 36 - jepsen.util 1026 :info :write [12 20] :RequestTimeoutException INFO [2022-03-13 17:09:23,277] jepsen worker 36 - jepsen.util 1116 :invoke :read [12 nil] INFO [2022-03-13 17:09:23,302] jepsen worker 36 - jepsen.util 1116 :ok :read [12 2] INFO [2022-03-13 17:09:23,303] jepsen worker 36 - jepsen.util 1116 :invoke :read [12 nil] INFO [2022-03-13 17:09:23,324] jepsen worker 36 - jepsen.util 1116 :ok :read [12 2] INFO [2022-03-13 17:09:23,325] jepsen worker 36 - jepsen.util 1116 :invoke :read [12 nil] INFO [2022-03-13 17:09:23,347] jepsen worker 36 - jepsen.util 1116 :ok :read [12 2] INFO [2022-03-13 17:09:23,348] jepsen worker 36 - jepsen.util 1116 :invoke :write [12 29] INFO [2022-03-13 17:09:23,378] jepsen worker 17 - jepsen.util 917 :ok :read [5 43] INFO [2022-03-13 17:09:23,379] jepsen worker 17 - jepsen.util 917 :invoke :write [5 21] INFO [2022-03-13 17:09:23,396] jepsen worker 19 - jepsen.util 739 :info :write [6 48] :RequestTimeoutException INFO [2022-03-13 17:09:23,397] jepsen worker 19 - jepsen.util 829 :invoke :read [6 nil] INFO [2022-03-13 17:09:23,428] jepsen worker 77 - jepsen.util 617 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,429] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:23,432] jepsen worker 70 - jepsen.util 700 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,432] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:23,433] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,433] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:23,434] jepsen worker 34 - jepsen.util 574 :invoke :write [11 43] INFO [2022-03-13 17:09:23,434] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:23,436] jepsen worker 46 - jepsen.util 946 :info :write [15 27] :RequestTimeoutException INFO [2022-03-13 17:09:23,437] jepsen worker 46 - jepsen.util 1036 :invoke :write [15 37] INFO [2022-03-13 17:09:23,466] jepsen worker 33 - jepsen.util 663 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,466] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:09:23,485] jepsen worker 13 - jepsen.util 463 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,486] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,485] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,487] jepsen worker 77 - jepsen.util 617 :invoke :write [25 46] INFO [2022-03-13 17:09:23,487] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:23,488] jepsen worker 51 - jepsen.util 411 :invoke :write [17 37] INFO [2022-03-13 17:09:23,489] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:23,490] jepsen worker 13 - jepsen.util 463 :invoke :write [4 42] INFO [2022-03-13 17:09:23,514] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,515] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:09:23,534] jepsen worker 7 - jepsen.util 457 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,534] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,534] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:23,534] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,535] jepsen worker 7 - jepsen.util 457 :invoke :write [2 34] INFO [2022-03-13 17:09:23,535] jepsen worker 52 - jepsen.util 502 :invoke :write [17 30] INFO [2022-03-13 17:09:23,536] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:23,537] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:23,572] jepsen worker 77 - jepsen.util 617 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,573] jepsen worker 77 - jepsen.util 617 :invoke :write [25 37] INFO [2022-03-13 17:09:23,576] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,577] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:09:23,642] jepsen worker 51 - jepsen.util 411 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,643] jepsen worker 51 - jepsen.util 411 :invoke :write [17 37] INFO [2022-03-13 17:09:23,644] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,644] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:23,712] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1700879765395894}, :ns_1@172.23.105.187 {:progress 0.2046783625730995}, :ns_1@172.23.107.227 {:progress 0.1700879765395894}, :ns_1@172.23.107.228 {:progress 0.1411764705882353}} INFO [2022-03-13 17:09:23,712] jepsen worker nemesis - couchbase.util Rebalance check count: 76 INFO [2022-03-13 17:09:23,712] jepsen worker nemesis - couchbase.util Rebalance stuck count: 41 INFO [2022-03-13 17:09:23,724] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,724] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,725] jepsen worker 13 - jepsen.util 463 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,725] jepsen worker 69 - jepsen.util 609 :invoke :write [23 36] INFO [2022-03-13 17:09:23,725] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:23,726] jepsen worker 13 - jepsen.util 463 :invoke :write [4 17] INFO [2022-03-13 17:09:23,727] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,727] jepsen worker 33 - jepsen.util 663 :invoke :write [11 15] INFO [2022-03-13 17:09:23,730] jepsen worker 38 - jepsen.util 1118 :info :write [12 20] :RequestTimeoutException INFO [2022-03-13 17:09:23,731] jepsen worker 38 - jepsen.util 1208 :invoke :read [12 nil] INFO [2022-03-13 17:09:23,732] jepsen worker 79 - jepsen.util 799 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,733] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:23,769] jepsen worker 21 - jepsen.util 651 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,769] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:23,769] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,770] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,770] jepsen worker 21 - jepsen.util 651 :invoke :write [7 6] INFO [2022-03-13 17:09:23,770] jepsen worker 78 - jepsen.util 438 :invoke :write [26 8] INFO [2022-03-13 17:09:23,771] jepsen worker 24 - jepsen.util 474 :invoke :write [8 15] INFO [2022-03-13 17:09:23,772] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:23,772] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:23,772] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,773] jepsen worker 8 - jepsen.util 638 :invoke :write [2 44] INFO [2022-03-13 17:09:23,774] jepsen worker 25 - jepsen.util 385 :invoke :write [8 43] INFO [2022-03-13 17:09:23,831] jepsen worker 77 - jepsen.util 617 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,831] jepsen worker 77 - jepsen.util 617 :invoke :write [25 8] INFO [2022-03-13 17:09:23,862] jepsen worker 33 - jepsen.util 663 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,863] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:09:23,905] jepsen worker 22 - jepsen.util 472 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,905] jepsen worker 7 - jepsen.util 457 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,906] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:23,906] jepsen worker 22 - jepsen.util 472 :invoke :write [7 44] INFO [2022-03-13 17:09:23,941] jepsen worker 51 - jepsen.util 411 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,941] jepsen worker 51 - jepsen.util 411 :invoke :write [17 8] INFO [2022-03-13 17:09:23,981] jepsen worker 69 - jepsen.util 609 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:23,982] jepsen worker 69 - jepsen.util 609 :invoke :write [23 9] INFO [2022-03-13 17:09:24,022] jepsen worker 13 - jepsen.util 463 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,022] jepsen worker 13 - jepsen.util 463 :invoke :write [4 14] INFO [2022-03-13 17:09:24,024] jepsen worker 52 - jepsen.util 502 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,024] jepsen worker 6 - jepsen.util 546 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,025] jepsen worker 6 - jepsen.util 546 :invoke :write [2 26] INFO [2022-03-13 17:09:24,025] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:24,026] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,026] jepsen worker 80 - jepsen.util 620 :invoke :write [26 13] INFO [2022-03-13 17:09:24,048] jepsen worker 89 - jepsen.util 1079 :info :write [29 3] :RequestTimeoutException INFO [2022-03-13 17:09:24,049] jepsen worker 89 - jepsen.util 1169 :invoke :write [29 25] INFO [2022-03-13 17:09:24,141] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:24,141] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,141] jepsen worker 71 - jepsen.util 431 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,141] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,143] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:24,143] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:24,144] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:24,144] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:24,163] jepsen worker 24 - jepsen.util 474 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,163] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:24,205] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,205] jepsen worker 34 - jepsen.util 574 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,206] jepsen worker 34 - jepsen.util 574 :invoke :write [11 8] INFO [2022-03-13 17:09:24,206] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:24,221] jepsen worker 77 - jepsen.util 617 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,221] jepsen worker 77 - jepsen.util 617 :invoke :write [25 2] INFO [2022-03-13 17:09:24,229] jepsen worker 21 - jepsen.util 651 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,229] jepsen worker 21 - jepsen.util 651 :invoke :write [7 11] INFO [2022-03-13 17:09:24,258] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,258] jepsen worker 69 - jepsen.util 609 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,259] jepsen worker 33 - jepsen.util 663 :invoke :write [11 9] INFO [2022-03-13 17:09:24,260] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:24,267] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,268] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:09:24,283] jepsen worker 65 - jepsen.util 965 :info :write [21 42] :RequestTimeoutException INFO [2022-03-13 17:09:24,284] jepsen worker 65 - jepsen.util 1055 :invoke :read [21 nil] INFO [2022-03-13 17:09:24,311] jepsen worker 43 - jepsen.util 403 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,311] jepsen worker 76 - jepsen.util 526 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,313] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:24,314] jepsen worker 76 - jepsen.util 526 :invoke :write [25 35] INFO [2022-03-13 17:09:24,314] jepsen worker 43 - jepsen.util 403 :invoke :write [14 20] INFO [2022-03-13 17:09:24,314] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:24,317] jepsen worker 51 - jepsen.util 411 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,317] jepsen worker 51 - jepsen.util 411 :invoke :write [17 8] INFO [2022-03-13 17:09:24,360] jepsen worker 8 - jepsen.util 638 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,360] jepsen worker 22 - jepsen.util 472 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,361] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:24,362] jepsen worker 22 - jepsen.util 472 :invoke :write [7 30] INFO [2022-03-13 17:09:24,412] jepsen worker 33 - jepsen.util 663 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,413] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:24,424] jepsen worker 13 - jepsen.util 463 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,424] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:24,516] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,516] jepsen worker 77 - jepsen.util 617 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,516] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,517] jepsen worker 77 - jepsen.util 617 :invoke :write [25 33] INFO [2022-03-13 17:09:24,518] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:24,519] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:09:24,519] jepsen worker 6 - jepsen.util 546 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,519] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,519] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:24,519] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,519] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,519] jepsen worker 78 - jepsen.util 438 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,519] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,521] jepsen worker 69 - jepsen.util 609 :invoke :write [23 9] INFO [2022-03-13 17:09:24,522] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:24,523] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:24,524] jepsen worker 79 - jepsen.util 799 :invoke :write [26 37] INFO [2022-03-13 17:09:24,525] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:24,526] jepsen worker 71 - jepsen.util 431 :invoke :write [23 12] INFO [2022-03-13 17:09:24,526] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,530] jepsen worker 78 - jepsen.util 438 :invoke :write [26 38] INFO [2022-03-13 17:09:24,532] jepsen worker 70 - jepsen.util 700 :invoke :write [23 20] INFO [2022-03-13 17:09:24,622] jepsen worker 80 - jepsen.util 620 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,622] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:24,622] jepsen worker 34 - jepsen.util 574 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,622] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,623] jepsen worker 7 - jepsen.util 457 :invoke :write [2 1] INFO [2022-03-13 17:09:24,624] jepsen worker 80 - jepsen.util 620 :invoke :write [26 23] INFO [2022-03-13 17:09:24,624] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:24,625] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:24,661] jepsen worker 22 - jepsen.util 472 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,661] jepsen worker 22 - jepsen.util 472 :invoke :write [7 22] INFO [2022-03-13 17:09:24,662] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,662] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:24,662] jepsen worker 6 - jepsen.util 546 :invoke :write [2 28] INFO [2022-03-13 17:09:24,663] jepsen worker 44 - jepsen.util 494 :invoke :write [14 47] INFO [2022-03-13 17:09:24,682] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,682] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,682] jepsen worker 21 - jepsen.util 651 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,683] jepsen worker 21 - jepsen.util 651 :invoke :write [7 28] INFO [2022-03-13 17:09:24,684] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:24,684] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:24,719] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1759530791788856}, :ns_1@172.23.105.187 {:progress 0.216374269005848}, :ns_1@172.23.107.227 {:progress 0.1759530791788856}, :ns_1@172.23.107.228 {:progress 0.1470588235294118}} INFO [2022-03-13 17:09:24,719] jepsen worker nemesis - couchbase.util Rebalance check count: 77 INFO [2022-03-13 17:09:24,719] jepsen worker nemesis - couchbase.util Rebalance stuck count: 41 INFO [2022-03-13 17:09:24,732] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:24,732] jepsen worker 51 - jepsen.util 411 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,732] jepsen worker 13 - jepsen.util 463 :invoke :write [4 11] INFO [2022-03-13 17:09:24,733] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:24,735] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,735] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:24,813] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,813] jepsen worker 77 - jepsen.util 617 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,814] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:24,814] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:24,839] jepsen worker 71 - jepsen.util 431 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,839] jepsen worker 78 - jepsen.util 438 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,839] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:24,840] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:24,840] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:24,841] jepsen worker 24 - jepsen.util 474 :invoke :write [8 18] INFO [2022-03-13 17:09:24,841] jepsen worker 69 - jepsen.util 609 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,842] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,843] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:09:24,846] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:24,847] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,848] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:09:24,877] jepsen worker 13 - jepsen.util 463 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,877] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,877] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,877] jepsen worker 13 - jepsen.util 463 :invoke :write [4 4] INFO [2022-03-13 17:09:24,878] jepsen worker 77 - jepsen.util 617 :invoke :write [25 36] INFO [2022-03-13 17:09:24,878] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:24,948] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:24,949] jepsen worker 51 - jepsen.util 411 :invoke :write [17 33] INFO [2022-03-13 17:09:25,006] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,006] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:09:25,048] jepsen worker 13 - jepsen.util 463 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,049] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:25,062] jepsen worker 70 - jepsen.util 700 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,062] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,063] jepsen worker 80 - jepsen.util 620 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,063] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,064] jepsen worker 80 - jepsen.util 620 :invoke :write [26 35] INFO [2022-03-13 17:09:25,064] jepsen worker 34 - jepsen.util 574 :invoke :write [11 28] INFO [2022-03-13 17:09:25,067] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,068] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,131] jepsen worker 77 - jepsen.util 617 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,131] jepsen worker 77 - jepsen.util 617 :invoke :write [25 19] INFO [2022-03-13 17:09:25,169] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,169] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,170] jepsen worker 33 - jepsen.util 663 :invoke :write [11 42] INFO [2022-03-13 17:09:25,170] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:09:25,172] jepsen worker 47 - jepsen.util 947 :ok :read [15 48] INFO [2022-03-13 17:09:25,172] jepsen worker 47 - jepsen.util 947 :invoke :read [15 nil] INFO [2022-03-13 17:09:25,206] jepsen worker 79 - jepsen.util 799 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,207] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,207] jepsen worker 79 - jepsen.util 799 :invoke :write [26 2] INFO [2022-03-13 17:09:25,207] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,207] jepsen worker 7 - jepsen.util 457 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,208] jepsen worker 52 - jepsen.util 502 :invoke :write [17 37] INFO [2022-03-13 17:09:25,208] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:25,208] jepsen worker 7 - jepsen.util 457 :invoke :write [2 27] INFO [2022-03-13 17:09:25,271] jepsen worker 51 - jepsen.util 411 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,271] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:25,272] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:25,273] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:25,274] jepsen worker 82 - jepsen.util 982 :info :write [27 15] :RequestTimeoutException INFO [2022-03-13 17:09:25,274] jepsen worker 82 - jepsen.util 1072 :invoke :read [27 nil] INFO [2022-03-13 17:09:25,287] jepsen worker 6 - jepsen.util 546 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,288] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:25,308] jepsen worker 21 - jepsen.util 651 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,308] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,309] jepsen worker 78 - jepsen.util 438 :invoke :write [26 20] INFO [2022-03-13 17:09:25,310] jepsen worker 21 - jepsen.util 651 :invoke :write [7 41] INFO [2022-03-13 17:09:25,376] jepsen worker 53 - jepsen.util 503 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,376] jepsen worker 60 - jepsen.util 600 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,376] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,376] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,376] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:25,376] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,376] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:25,378] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:25,378] jepsen worker 33 - jepsen.util 663 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,379] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:25,379] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:25,380] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:25,380] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:25,381] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:25,381] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,381] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,392] jepsen worker 77 - jepsen.util 617 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,392] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,392] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:25,393] jepsen worker 51 - jepsen.util 411 :invoke :write [17 29] INFO [2022-03-13 17:09:25,394] jepsen worker 13 - jepsen.util 463 :invoke :write [4 4] INFO [2022-03-13 17:09:25,394] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:25,396] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,396] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:25,397] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,398] jepsen worker 77 - jepsen.util 617 :invoke :write [25 18] INFO [2022-03-13 17:09:25,399] jepsen worker 44 - jepsen.util 494 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,399] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:25,426] jepsen worker 24 - jepsen.util 474 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,427] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:25,500] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,500] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,501] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:09:25,501] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] WARN [2022-03-13 17:09:25,571] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9039187,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11846301,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9496805,"encode_duration_us":17611,"operation_id":"0x21f1","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10004903},{"operation_name":"upsert","last_dispatch_duration_us":9056683,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9563371,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9334391,"encode_duration_us":20296,"operation_id":"0x225f","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10010665},{"operation_name":"upsert","last_dispatch_duration_us":9036880,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9899067,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9474477,"encode_duration_us":16939,"operation_id":"0x22e0","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10006686}],"total_count":3}} WARN [2022-03-13 17:09:25,571] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":19782,"operation_id":"0x258c","timeout_ms":10000,"total_duration_us":10010365},{"operation_name":"upsert","last_dispatch_duration_us":260184,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10110246,"total_server_duration_us":143095,"encode_duration_us":18667,"operation_id":"0x26b4","timeout_ms":10000,"last_server_duration_us":50886,"total_duration_us":10010053},{"operation_name":"upsert","last_dispatch_duration_us":251836,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10007665,"total_server_duration_us":146238,"encode_duration_us":31815,"operation_id":"0x26d4","timeout_ms":10000,"last_server_duration_us":31681,"total_duration_us":10009665},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":19848,"operation_id":"0x2647","timeout_ms":10000,"total_duration_us":10005193},{"operation_name":"upsert","last_dispatch_duration_us":491793,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":635167,"total_server_duration_us":111184,"encode_duration_us":29391,"operation_id":"0x26dc","timeout_ms":10000,"last_server_duration_us":76870,"total_duration_us":10004329},{"operation_name":"upsert","last_dispatch_duration_us":35809,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":981550,"total_server_duration_us":612548,"encode_duration_us":17911,"operation_id":"0x2755","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10004143},{"operation_name":"upsert","last_dispatch_duration_us":322896,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":322896,"total_server_duration_us":45950,"encode_duration_us":22772,"operation_id":"0x26e7","timeout_ms":10000,"last_server_duration_us":45950,"total_duration_us":10003029},{"operation_name":"upsert","last_dispatch_duration_us":37534,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9711055,"total_server_duration_us":100567,"encode_duration_us":17369,"operation_id":"0x2709","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10002117},{"operation_name":"upsert","last_dispatch_duration_us":130355,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10059596,"total_server_duration_us":165628,"encode_duration_us":21721,"operation_id":"0x26de","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10001505},{"operation_name":"upsert","last_dispatch_duration_us":9425620,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9425620,"total_server_duration_us":9104337,"encode_duration_us":18983,"operation_id":"0x26f6","timeout_ms":10000,"last_server_duration_us":9104337,"total_duration_us":9430626}],"total_count":24}} INFO [2022-03-13 17:09:25,602] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,602] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,603] jepsen worker 80 - jepsen.util 620 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,603] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,602] jepsen worker 22 - jepsen.util 472 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,603] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,605] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:25,605] jepsen worker 22 - jepsen.util 472 :invoke :write [7 34] INFO [2022-03-13 17:09:25,606] jepsen worker 8 - jepsen.util 638 :invoke :write [2 27] INFO [2022-03-13 17:09:25,608] jepsen worker 13 - jepsen.util 463 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,609] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:25,610] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,665] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,666] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:09:25,724] jepsen worker 51 - jepsen.util 411 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,724] jepsen worker 51 - jepsen.util 411 :invoke :write [17 41] INFO [2022-03-13 17:09:25,725] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1759530791788856}, :ns_1@172.23.105.187 {:progress 0.216374269005848}, :ns_1@172.23.107.227 {:progress 0.1759530791788856}, :ns_1@172.23.107.228 {:progress 0.1470588235294118}} INFO [2022-03-13 17:09:25,725] jepsen worker nemesis - couchbase.util Rebalance check count: 78 INFO [2022-03-13 17:09:25,725] jepsen worker nemesis - couchbase.util Rebalance stuck count: 42 INFO [2022-03-13 17:09:25,802] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,803] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,817] jepsen worker 34 - jepsen.util 574 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,817] jepsen worker 52 - jepsen.util 502 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,818] jepsen worker 52 - jepsen.util 502 :invoke :write [17 13] INFO [2022-03-13 17:09:25,818] jepsen worker 34 - jepsen.util 574 :invoke :write [11 23] INFO [2022-03-13 17:09:25,833] jepsen worker 77 - jepsen.util 617 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,833] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:25,834] jepsen worker 13 - jepsen.util 463 :invoke :write [4 28] INFO [2022-03-13 17:09:25,835] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:25,959] jepsen worker 51 - jepsen.util 411 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,959] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,959] jepsen worker 51 - jepsen.util 411 :invoke :write [17 16] INFO [2022-03-13 17:09:25,960] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:25,963] jepsen worker 79 - jepsen.util 799 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,963] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,963] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:25,963] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,964] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,964] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:25,964] jepsen worker 44 - jepsen.util 494 :invoke :write [14 5] INFO [2022-03-13 17:09:25,965] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,965] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,965] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,966] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:25,966] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,967] jepsen worker 71 - jepsen.util 431 :invoke :write [23 48] INFO [2022-03-13 17:09:25,967] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,967] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,968] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,968] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,969] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,969] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,969] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,970] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,970] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,970] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,971] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,972] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,972] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,972] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:25,973] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,974] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,974] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:09:25,974] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,975] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:25,976] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:25,976] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:09:26,056] jepsen worker 7 - jepsen.util 457 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,057] jepsen worker 13 - jepsen.util 463 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,057] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,058] jepsen worker 7 - jepsen.util 457 :invoke :write [2 47] INFO [2022-03-13 17:09:26,058] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:26,059] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,072] jepsen worker 28 - jepsen.util 1018 :ok :read [9 37] INFO [2022-03-13 17:09:26,073] jepsen worker 28 - jepsen.util 1018 :invoke :read [9 nil] INFO [2022-03-13 17:09:26,079] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,080] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:26,118] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,118] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,119] jepsen worker 77 - jepsen.util 617 :invoke :write [25 46] INFO [2022-03-13 17:09:26,119] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,121] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,121] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,122] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,123] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,123] jepsen worker 51 - jepsen.util 411 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,123] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:26,124] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,124] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,124] jepsen worker 13 - jepsen.util 463 :invoke :write [4 8] INFO [2022-03-13 17:09:26,125] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:26,126] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,126] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:26,127] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,128] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,128] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,128] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:26,129] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:09:26,129] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,129] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:26,130] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:26,131] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,131] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:09:26,131] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,132] jepsen worker 51 - jepsen.util 411 :invoke :write [17 16] INFO [2022-03-13 17:09:26,173] jepsen worker 22 - jepsen.util 472 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,173] jepsen worker 22 - jepsen.util 472 :invoke :write [7 45] INFO [2022-03-13 17:09:26,194] jepsen worker 9 - jepsen.util 909 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,195] jepsen worker 9 - jepsen.util 909 :invoke :read [3 nil] INFO [2022-03-13 17:09:26,197] jepsen worker 73 - jepsen.util 1063 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,197] jepsen worker 73 - jepsen.util 1063 :invoke :read [24 nil] INFO [2022-03-13 17:09:26,211] jepsen worker 77 - jepsen.util 617 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,212] jepsen worker 77 - jepsen.util 617 :invoke :write [25 15] INFO [2022-03-13 17:09:26,241] jepsen worker 78 - jepsen.util 438 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,241] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:26,318] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,319] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:26,355] jepsen worker 54 - jepsen.util 774 :info :write [18 44] :SyncWriteAmbiguous INFO [2022-03-13 17:09:26,355] jepsen worker 54 - jepsen.util 864 :invoke :write [18 0] INFO [2022-03-13 17:09:26,370] jepsen worker 13 - jepsen.util 463 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,371] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,446] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,446] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,447] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:09:26,451] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:26,453] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,453] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:26,460] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,461] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:26,462] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,463] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:09:26,503] jepsen worker 51 - jepsen.util 411 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,503] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:26,529] jepsen worker 21 - jepsen.util 651 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,529] jepsen worker 34 - jepsen.util 574 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,530] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,530] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,529] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,531] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:26,532] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:26,534] jepsen worker 21 - jepsen.util 651 :invoke :write [7 35] INFO [2022-03-13 17:09:26,535] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:26,535] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:26,580] jepsen worker 32 - jepsen.util 752 :info :write [10 37] :SyncWriteAmbiguous INFO [2022-03-13 17:09:26,580] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,581] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,581] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,582] jepsen worker 32 - jepsen.util 842 :invoke :write [10 35] INFO [2022-03-13 17:09:26,582] jepsen worker 77 - jepsen.util 617 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,584] jepsen worker 33 - jepsen.util 663 :invoke :write [11 45] INFO [2022-03-13 17:09:26,584] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,585] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:26,585] jepsen worker 77 - jepsen.util 617 :invoke :write [25 3] INFO [2022-03-13 17:09:26,586] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,586] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,587] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,587] jepsen worker 51 - jepsen.util 411 :invoke :write [17 27] INFO [2022-03-13 17:09:26,588] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,589] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,590] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,591] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,592] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,593] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,595] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,596] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,597] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,597] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,598] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,599] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,600] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,601] jepsen worker 13 - jepsen.util 463 :invoke :write [4 8] INFO [2022-03-13 17:09:26,605] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,605] jepsen worker 42 - jepsen.util 402 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,607] jepsen worker 62 - jepsen.util 422 :invoke :write [20 31] INFO [2022-03-13 17:09:26,607] jepsen worker 42 - jepsen.util 402 :invoke :write [14 49] INFO [2022-03-13 17:09:26,618] jepsen worker 29 - jepsen.util 1019 :ok :read [9 37] INFO [2022-03-13 17:09:26,619] jepsen worker 29 - jepsen.util 1019 :invoke :read [9 nil] INFO [2022-03-13 17:09:26,671] jepsen worker 84 - jepsen.util 1254 :ok :read [28 44] INFO [2022-03-13 17:09:26,671] jepsen worker 84 - jepsen.util 1254 :invoke :write [28 6] INFO [2022-03-13 17:09:26,676] jepsen worker 57 - jepsen.util 1047 :ok :read [19 2] INFO [2022-03-13 17:09:26,676] jepsen worker 57 - jepsen.util 1047 :invoke :read [19 nil] INFO [2022-03-13 17:09:26,696] jepsen worker 77 - jepsen.util 617 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,696] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:26,704] jepsen worker 57 - jepsen.util 1047 :ok :read [19 2] INFO [2022-03-13 17:09:26,705] jepsen worker 57 - jepsen.util 1047 :invoke :write [19 40] INFO [2022-03-13 17:09:26,709] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,710] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:26,719] jepsen worker 43 - jepsen.util 403 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,719] jepsen worker 43 - jepsen.util 403 :invoke :write [14 42] INFO [2022-03-13 17:09:26,721] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:26,721] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,721] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:26,722] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:26,731] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1759530791788856}, :ns_1@172.23.105.187 {:progress 0.216374269005848}, :ns_1@172.23.107.227 {:progress 0.1759530791788856}, :ns_1@172.23.107.228 {:progress 0.1529411764705882}} INFO [2022-03-13 17:09:26,731] jepsen worker nemesis - couchbase.util Rebalance check count: 79 INFO [2022-03-13 17:09:26,731] jepsen worker nemesis - couchbase.util Rebalance stuck count: 42 INFO [2022-03-13 17:09:26,738] jepsen worker 45 - jepsen.util 1215 :ok :read [15 48] INFO [2022-03-13 17:09:26,739] jepsen worker 45 - jepsen.util 1215 :invoke :read [15 nil] INFO [2022-03-13 17:09:26,790] jepsen worker 29 - jepsen.util 1019 :ok :read [9 37] INFO [2022-03-13 17:09:26,791] jepsen worker 29 - jepsen.util 1019 :invoke :read [9 nil] INFO [2022-03-13 17:09:26,800] jepsen worker 58 - jepsen.util 868 :ok :read [19 2] INFO [2022-03-13 17:09:26,801] jepsen worker 58 - jepsen.util 868 :invoke :write [19 5] INFO [2022-03-13 17:09:26,830] jepsen worker 51 - jepsen.util 411 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,830] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,831] jepsen worker 60 - jepsen.util 600 :invoke :write [20 23] INFO [2022-03-13 17:09:26,835] jepsen worker 51 - jepsen.util 411 :invoke :write [17 37] INFO [2022-03-13 17:09:26,842] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:26,843] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:09:26,860] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] INFO [2022-03-13 17:09:26,861] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] INFO [2022-03-13 17:09:26,864] jepsen worker 86 - jepsen.util 896 :ok :read [28 44] INFO [2022-03-13 17:09:26,864] jepsen worker 45 - jepsen.util 1215 :ok :read [15 48] INFO [2022-03-13 17:09:26,864] jepsen worker 86 - jepsen.util 896 :invoke :write [28 47] INFO [2022-03-13 17:09:26,865] jepsen worker 45 - jepsen.util 1215 :invoke :read [15 nil] INFO [2022-03-13 17:09:26,896] jepsen worker 59 - jepsen.util 959 :ok :read [19 2] INFO [2022-03-13 17:09:26,896] jepsen worker 59 - jepsen.util 959 :invoke :write [19 27] INFO [2022-03-13 17:09:26,909] jepsen worker 13 - jepsen.util 463 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,909] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,911] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:26,911] jepsen worker 33 - jepsen.util 663 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,911] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,912] jepsen worker 44 - jepsen.util 494 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,912] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,912] jepsen worker 52 - jepsen.util 502 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,913] jepsen worker 33 - jepsen.util 663 :invoke :write [11 49] INFO [2022-03-13 17:09:26,914] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:09:26,916] jepsen worker 44 - jepsen.util 494 :invoke :write [14 29] INFO [2022-03-13 17:09:26,917] jepsen worker 52 - jepsen.util 502 :invoke :write [17 18] INFO [2022-03-13 17:09:26,919] jepsen worker 28 - jepsen.util 1018 :ok :read [9 37] INFO [2022-03-13 17:09:26,923] jepsen worker 47 - jepsen.util 947 :ok :read [15 48] INFO [2022-03-13 17:09:26,925] jepsen worker 28 - jepsen.util 1018 :invoke :read [9 nil] INFO [2022-03-13 17:09:26,925] jepsen worker 47 - jepsen.util 947 :invoke :read [15 nil] INFO [2022-03-13 17:09:26,954] jepsen worker 9 - jepsen.util 909 :ok :read [3 27] INFO [2022-03-13 17:09:26,956] jepsen worker 9 - jepsen.util 909 :invoke :read [3 nil] INFO [2022-03-13 17:09:26,958] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,958] jepsen worker 51 - jepsen.util 411 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:26,958] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,959] jepsen worker 51 - jepsen.util 411 :invoke :write [17 24] INFO [2022-03-13 17:09:26,960] jepsen worker 77 - jepsen.util 617 :invoke :write [25 1] INFO [2022-03-13 17:09:26,969] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:26,971] jepsen worker 29 - jepsen.util 1019 :ok :read [9 37] INFO [2022-03-13 17:09:26,971] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:26,972] jepsen worker 13 - jepsen.util 463 :invoke :write [4 3] INFO [2022-03-13 17:09:26,976] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] WARN [2022-03-13 17:09:26,978] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18029224,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25669612,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9125790,"encode_duration_us":17945,"operation_id":"0x1f24","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10003452},{"operation_name":"upsert","last_dispatch_duration_us":18141900,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27282615,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9071934,"encode_duration_us":18271,"operation_id":"0x21f5","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10005836},{"operation_name":"upsert","last_dispatch_duration_us":15671817,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15671817,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9052231,"encode_duration_us":22839,"operation_id":"0x22e3","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10007169},{"operation_name":"upsert","last_dispatch_duration_us":18041013,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24881600,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9053388,"encode_duration_us":18392,"operation_id":"0x1f81","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10012661},{"operation_name":"upsert","last_dispatch_duration_us":17426285,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17426285,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9030999,"encode_duration_us":17542,"operation_id":"0x2599","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10009163}],"total_count":5}} INFO [2022-03-13 17:09:26,980] jepsen worker 73 - jepsen.util 1063 :ok :read [24 24] INFO [2022-03-13 17:09:26,980] jepsen worker 29 - jepsen.util 1019 :invoke :write [9 36] INFO [2022-03-13 17:09:26,981] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] INFO [2022-03-13 17:09:26,984] jepsen worker 73 - jepsen.util 1063 :invoke :read [24 nil] INFO [2022-03-13 17:09:27,005] jepsen worker 45 - jepsen.util 1215 :ok :read [15 48] INFO [2022-03-13 17:09:27,006] jepsen worker 45 - jepsen.util 1215 :invoke :read [15 nil] INFO [2022-03-13 17:09:27,025] jepsen worker 28 - jepsen.util 1018 :ok :read [9 37] INFO [2022-03-13 17:09:27,025] jepsen worker 28 - jepsen.util 1018 :invoke :read [9 nil] INFO [2022-03-13 17:09:27,028] jepsen worker 47 - jepsen.util 947 :ok :read [15 48] INFO [2022-03-13 17:09:27,029] jepsen worker 47 - jepsen.util 947 :invoke :write [15 8] INFO [2022-03-13 17:09:27,034] jepsen worker 33 - jepsen.util 663 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,034] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:09:27,040] jepsen worker 9 - jepsen.util 909 :ok :read [3 27] INFO [2022-03-13 17:09:27,040] jepsen worker 9 - jepsen.util 909 :invoke :read [3 nil] INFO [2022-03-13 17:09:27,058] jepsen worker 45 - jepsen.util 1215 :ok :read [15 48] INFO [2022-03-13 17:09:27,059] jepsen worker 45 - jepsen.util 1215 :invoke :write [15 48] INFO [2022-03-13 17:09:27,064] jepsen worker 77 - jepsen.util 617 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,065] jepsen worker 77 - jepsen.util 617 :invoke :write [25 27] INFO [2022-03-13 17:09:27,075] jepsen worker 9 - jepsen.util 909 :ok :read [3 27] INFO [2022-03-13 17:09:27,075] jepsen worker 71 - jepsen.util 431 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,076] jepsen worker 9 - jepsen.util 909 :invoke :write [3 11] INFO [2022-03-13 17:09:27,076] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] WARN [2022-03-13 17:09:27,078] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17400549,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18214132,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9443769,"encode_duration_us":16882,"operation_id":"0x21c6","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10003332},{"operation_name":"upsert","last_dispatch_duration_us":9053814,"last_local_id":"A312307A00000004/0000000095638474","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9633035,"last_local_socket":"172.23.104.79:33498","total_server_duration_us":9400136,"encode_duration_us":17363,"operation_id":"0x24c8","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10010320},{"operation_name":"upsert","last_dispatch_duration_us":16215074,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16933264,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9283404,"encode_duration_us":18558,"operation_id":"0x2459","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10010331}],"total_count":3}} WARN [2022-03-13 17:09:27,078] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15695789,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":15695789,"total_server_duration_us":62,"operation_id":"0x2652","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":15696788},{"operation_name":"get","last_dispatch_duration_us":13862498,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13862498,"total_server_duration_us":23,"operation_id":"0x236c","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":13862916},{"operation_name":"upsert","last_dispatch_duration_us":17426285,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":17426285,"total_server_duration_us":9030999,"encode_duration_us":17542,"operation_id":"0x2599","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10009163},{"operation_name":"upsert","last_dispatch_duration_us":9127482,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":14008774,"total_server_duration_us":87228,"encode_duration_us":30564,"operation_id":"0x26dd","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10008521},{"operation_name":"upsert","last_dispatch_duration_us":12766517,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12766517,"total_server_duration_us":19966,"encode_duration_us":25217,"operation_id":"0x2771","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10005159},{"operation_name":"upsert","last_dispatch_duration_us":9095765,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18662651,"total_server_duration_us":115405,"encode_duration_us":18668,"operation_id":"0x251b","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10004464},{"operation_name":"get","last_dispatch_duration_us":9258015,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9258015,"total_server_duration_us":56,"operation_id":"0x252d","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9259185},{"operation_name":"get","last_dispatch_duration_us":9168911,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9168911,"total_server_duration_us":56,"operation_id":"0x2905","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9170236},{"operation_name":"get","last_dispatch_duration_us":9139915,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9139915,"total_server_duration_us":7180,"operation_id":"0x251e","timeout_ms":30000,"last_server_duration_us":7180,"total_duration_us":9142570},{"operation_name":"upsert","last_dispatch_duration_us":9133202,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9133202,"total_server_duration_us":9050106,"encode_duration_us":20886,"operation_id":"0x28c1","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9137385}],"total_count":40}} WARN [2022-03-13 17:09:27,085] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":87831,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9151515,"total_server_duration_us":210248,"encode_duration_us":19684,"operation_id":"0x258f","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10010287},{"operation_name":"upsert","last_dispatch_duration_us":32821,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":973245,"total_server_duration_us":626117,"encode_duration_us":18969,"operation_id":"0x271d","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10008753},{"operation_name":"upsert","last_dispatch_duration_us":82246,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9667243,"total_server_duration_us":240699,"encode_duration_us":23029,"operation_id":"0x2718","timeout_ms":10000,"last_server_duration_us":32065,"total_duration_us":10008580},{"operation_name":"upsert","last_dispatch_duration_us":112318,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9085712,"total_server_duration_us":156345,"encode_duration_us":21168,"operation_id":"0x25a1","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10005566},{"operation_name":"upsert","last_dispatch_duration_us":57751,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":57751,"total_server_duration_us":23475,"encode_duration_us":42229,"operation_id":"0x253f","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10003989},{"operation_name":"upsert","last_dispatch_duration_us":9117830,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9402247,"total_server_duration_us":9208976,"encode_duration_us":21755,"operation_id":"0x2717","timeout_ms":10000,"last_server_duration_us":9078800,"total_duration_us":9435319},{"operation_name":"get","last_dispatch_duration_us":60945,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":60945,"total_server_duration_us":43,"operation_id":"0x2617","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8330740},{"operation_name":"get","last_dispatch_duration_us":139978,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":139978,"total_server_duration_us":49,"operation_id":"0x26a6","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":6339127},{"operation_name":"get","last_dispatch_duration_us":48696,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":48696,"total_server_duration_us":32,"operation_id":"0x27eb","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3179416},{"operation_name":"upsert","last_dispatch_duration_us":114029,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":821163,"total_server_duration_us":277017,"encode_duration_us":19820,"operation_id":"0x2a58","timeout_ms":10000,"last_server_duration_us":42200,"total_duration_us":875469}],"total_count":15}} INFO [2022-03-13 17:09:27,091] jepsen worker 73 - jepsen.util 1063 :ok :read [24 24] INFO [2022-03-13 17:09:27,091] jepsen worker 73 - jepsen.util 1063 :invoke :write [24 15] INFO [2022-03-13 17:09:27,101] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,101] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:09:27,118] jepsen worker 63 - jepsen.util 873 :info :write [21 35] :RequestTimeoutException INFO [2022-03-13 17:09:27,119] jepsen worker 63 - jepsen.util 963 :invoke :read [21 nil] INFO [2022-03-13 17:09:27,148] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] INFO [2022-03-13 17:09:27,148] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] WARN [2022-03-13 17:09:27,177] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":20542313,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":20542313,"total_server_duration_us":80968,"operation_id":"0x2432","timeout_ms":30000,"last_server_duration_us":80968,"total_duration_us":20544230},{"operation_name":"get","last_dispatch_duration_us":19226088,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":19226088,"total_server_duration_us":23,"operation_id":"0x24c1","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":19228308},{"operation_name":"upsert","last_dispatch_duration_us":2894958,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4890234,"total_server_duration_us":375103,"encode_duration_us":17376,"operation_id":"0x250a","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10010211},{"operation_name":"upsert","last_dispatch_duration_us":2949814,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4744260,"total_server_duration_us":382597,"encode_duration_us":20240,"operation_id":"0x259a","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10007924},{"operation_name":"upsert","last_dispatch_duration_us":3361535,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4011415,"total_server_duration_us":310919,"encode_duration_us":16815,"operation_id":"0x25ef","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10005238},{"operation_name":"upsert","last_dispatch_duration_us":3344240,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4188971,"total_server_duration_us":381060,"encode_duration_us":22467,"operation_id":"0x2586","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10004196},{"operation_name":"upsert","last_dispatch_duration_us":3906940,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7241728,"total_server_duration_us":49138,"encode_duration_us":18353,"operation_id":"0x270c","timeout_ms":10000,"last_server_duration_us":22720,"total_duration_us":10002482},{"operation_name":"upsert","last_dispatch_duration_us":43630,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":556288,"total_server_duration_us":212484,"encode_duration_us":20118,"operation_id":"0x2690","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10000932},{"operation_name":"upsert","last_dispatch_duration_us":9063379,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9063379,"total_server_duration_us":9037366,"encode_duration_us":18771,"operation_id":"0x2681","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":9066911},{"operation_name":"upsert","last_dispatch_duration_us":539767,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1324601,"total_server_duration_us":119212,"encode_duration_us":29056,"operation_id":"0x29a4","timeout_ms":10000,"last_server_duration_us":43179,"total_duration_us":3935939}],"total_count":82}} WARN [2022-03-13 17:09:27,177] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":20471311,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20471311,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9233552,"encode_duration_us":17811,"operation_id":"0x23f2","timeout_ms":10000,"last_server_duration_us":9233552,"total_duration_us":10008727}],"total_count":1}} WARN [2022-03-13 17:09:27,179] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9046906,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9977956,"total_server_duration_us":9369240,"encode_duration_us":20648,"operation_id":"0x2726","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007071},{"operation_name":"upsert","last_dispatch_duration_us":84376,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8900248,"total_server_duration_us":177995,"encode_duration_us":17586,"operation_id":"0x263c","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10006118},{"operation_name":"upsert","last_dispatch_duration_us":54385,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1103411,"total_server_duration_us":398131,"encode_duration_us":22924,"operation_id":"0x2729","timeout_ms":10000,"last_server_duration_us":29695,"total_duration_us":10002549},{"operation_name":"upsert","last_dispatch_duration_us":39353,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":5439288,"total_server_duration_us":541049,"encode_duration_us":18883,"operation_id":"0x2570","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10001276},{"operation_name":"upsert","last_dispatch_duration_us":9308167,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9308167,"total_server_duration_us":9039489,"encode_duration_us":23833,"operation_id":"0x25a5","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":9314419},{"operation_name":"get","last_dispatch_duration_us":9022633,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9022633,"total_server_duration_us":43,"operation_id":"0x25e7","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9025547},{"operation_name":"get","last_dispatch_duration_us":3609712,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":3609712,"total_server_duration_us":43,"operation_id":"0x27ed","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3611019},{"operation_name":"upsert","last_dispatch_duration_us":941865,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1829786,"total_server_duration_us":138562,"encode_duration_us":29129,"operation_id":"0x2a3a","timeout_ms":10000,"last_server_duration_us":68813,"total_duration_us":1846915},{"operation_name":"upsert","last_dispatch_duration_us":912244,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1645674,"total_server_duration_us":98734,"encode_duration_us":20437,"operation_id":"0x2a0b","timeout_ms":10000,"last_server_duration_us":45838,"total_duration_us":1835899},{"operation_name":"upsert","last_dispatch_duration_us":874748,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1678758,"total_server_duration_us":104682,"encode_duration_us":17477,"operation_id":"0x2a51","timeout_ms":10000,"last_server_duration_us":62157,"total_duration_us":1693707}],"total_count":71}} WARN [2022-03-13 17:09:27,185] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18242131,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18242131,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9051168,"encode_duration_us":17135,"operation_id":"0x2226","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10006256},{"operation_name":"upsert","last_dispatch_duration_us":18164725,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18164725,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9039489,"encode_duration_us":30729,"operation_id":"0x2225","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10008370},{"operation_name":"upsert","last_dispatch_duration_us":23960724,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":23960724,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9045859,"encode_duration_us":17723,"operation_id":"0x2047","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007514}],"total_count":3}} INFO [2022-03-13 17:09:27,198] jepsen worker 37 - jepsen.util 937 :info :write [12 13] :RequestTimeoutException INFO [2022-03-13 17:09:27,199] jepsen worker 37 - jepsen.util 1027 :invoke :write [12 0] INFO [2022-03-13 17:09:27,219] jepsen worker 51 - jepsen.util 411 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,220] jepsen worker 51 - jepsen.util 411 :invoke :write [17 5] INFO [2022-03-13 17:09:27,235] jepsen worker 25 - jepsen.util 385 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,236] jepsen worker 25 - jepsen.util 385 :invoke :write [8 40] INFO [2022-03-13 17:09:27,241] jepsen worker 63 - jepsen.util 963 :ok :read [21 :nil] INFO [2022-03-13 17:09:27,242] jepsen worker 63 - jepsen.util 963 :invoke :write [21 41] INFO [2022-03-13 17:09:27,265] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,265] jepsen worker 13 - jepsen.util 463 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:27,265] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:09:27,266] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] WARN [2022-03-13 17:09:27,279] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13523939,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13523939,"total_server_duration_us":43,"operation_id":"0x25f0","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":13525113},{"operation_name":"upsert","last_dispatch_duration_us":9053814,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9633035,"total_server_duration_us":9400136,"encode_duration_us":17363,"operation_id":"0x24c8","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10010320},{"operation_name":"upsert","last_dispatch_duration_us":82997,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9779267,"total_server_duration_us":170561,"encode_duration_us":16961,"operation_id":"0x26b2","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10007553},{"operation_name":"upsert","last_dispatch_duration_us":155278,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7963750,"total_server_duration_us":256484,"encode_duration_us":23362,"operation_id":"0x2786","timeout_ms":10000,"last_server_duration_us":72999,"total_duration_us":10006309},{"operation_name":"upsert","last_dispatch_duration_us":8737718,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9234440,"total_server_duration_us":334348,"encode_duration_us":20142,"operation_id":"0x25c9","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10006185},{"operation_name":"upsert","last_dispatch_duration_us":82213,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9124858,"total_server_duration_us":247265,"encode_duration_us":18763,"operation_id":"0x270d","timeout_ms":10000,"last_server_duration_us":34612,"total_duration_us":10006090},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":21815,"operation_id":"0x27ea","timeout_ms":10000,"total_duration_us":10006086},{"operation_name":"upsert","last_dispatch_duration_us":14911815,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":14911815,"total_server_duration_us":22554,"encode_duration_us":17333,"operation_id":"0x2530","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10003333},{"operation_name":"upsert","last_dispatch_duration_us":11446914,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11446914,"total_server_duration_us":22059,"encode_duration_us":19048,"operation_id":"0x26a5","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10001902},{"operation_name":"upsert","last_dispatch_duration_us":4976667,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4976667,"total_server_duration_us":112475,"encode_duration_us":20183,"operation_id":"0x2933","timeout_ms":10000,"last_server_duration_us":112475,"total_duration_us":7085055}],"total_count":43}} WARN [2022-03-13 17:09:27,280] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9046906,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9977956,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9369240,"encode_duration_us":20648,"operation_id":"0x2726","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007071}],"total_count":1}} INFO [2022-03-13 17:09:27,321] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] INFO [2022-03-13 17:09:27,322] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] INFO [2022-03-13 17:09:27,328] jepsen worker 83 - jepsen.util 803 :ok :read [27 4] INFO [2022-03-13 17:09:27,328] jepsen worker 83 - jepsen.util 803 :invoke :read [27 nil] INFO [2022-03-13 17:09:27,485] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] INFO [2022-03-13 17:09:27,486] jepsen worker 1 - jepsen.util 901 :invoke :read [0 nil] INFO [2022-03-13 17:09:27,560] jepsen worker 16 - jepsen.util 916 :info :write [5 45] :SyncWriteAmbiguous INFO [2022-03-13 17:09:27,561] jepsen worker 16 - jepsen.util 1006 :invoke :write [5 6] INFO [2022-03-13 17:09:27,626] jepsen worker 0 - jepsen.util 990 :info :write [0 45] :SyncWriteAmbiguous INFO [2022-03-13 17:09:27,627] jepsen worker 0 - jepsen.util 1080 :invoke :write [0 24] INFO [2022-03-13 17:09:27,738] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1759530791788856}, :ns_1@172.23.105.187 {:progress 0.216374269005848}, :ns_1@172.23.107.227 {:progress 0.1759530791788856}, :ns_1@172.23.107.228 {:progress 0.1529411764705882}} INFO [2022-03-13 17:09:27,738] jepsen worker nemesis - couchbase.util Rebalance check count: 80 INFO [2022-03-13 17:09:27,739] jepsen worker nemesis - couchbase.util Rebalance stuck count: 43 INFO [2022-03-13 17:09:27,785] jepsen worker 66 - jepsen.util 1146 :ok :read [22 30] INFO [2022-03-13 17:09:27,786] jepsen worker 66 - jepsen.util 1146 :invoke :write [22 36] INFO [2022-03-13 17:09:28,014] jepsen worker 64 - jepsen.util 694 :info :write [21 17] :RequestTimeoutException INFO [2022-03-13 17:09:28,015] jepsen worker 64 - jepsen.util 784 :invoke :write [21 16] INFO [2022-03-13 17:09:28,073] jepsen worker 85 - jepsen.util 1075 :info :write [28 11] :SyncWriteAmbiguous INFO [2022-03-13 17:09:28,073] jepsen worker 85 - jepsen.util 1165 :invoke :write [28 20] INFO [2022-03-13 17:09:28,121] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,122] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,124] jepsen worker 77 - jepsen.util 617 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,124] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,124] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,124] jepsen worker 22 - jepsen.util 472 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,124] jepsen worker 77 - jepsen.util 617 :invoke :write [25 13] INFO [2022-03-13 17:09:28,124] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,125] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,125] jepsen worker 34 - jepsen.util 574 :invoke :write [11 35] INFO [2022-03-13 17:09:28,126] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,127] jepsen worker 6 - jepsen.util 546 :invoke :write [2 41] INFO [2022-03-13 17:09:28,127] jepsen worker 22 - jepsen.util 472 :invoke :write [7 36] INFO [2022-03-13 17:09:28,134] jepsen worker 79 - jepsen.util 799 :invoke :write [26 33] INFO [2022-03-13 17:09:28,183] jepsen worker 7 - jepsen.util 457 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,183] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,184] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:28,185] jepsen worker 78 - jepsen.util 438 :invoke :write [26 10] INFO [2022-03-13 17:09:28,223] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:28,224] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,224] jepsen worker 51 - jepsen.util 411 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,224] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,226] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,226] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:28,227] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:28,228] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:28,232] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,233] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,233] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,233] jepsen worker 33 - jepsen.util 663 :invoke :write [11 15] INFO [2022-03-13 17:09:28,234] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,235] jepsen worker 69 - jepsen.util 609 :invoke :write [23 31] INFO [2022-03-13 17:09:28,299] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:28,299] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,299] jepsen worker 77 - jepsen.util 617 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,300] jepsen worker 51 - jepsen.util 411 :invoke :write [17 26] INFO [2022-03-13 17:09:28,303] jepsen worker 13 - jepsen.util 463 :invoke :write [4 2] INFO [2022-03-13 17:09:28,305] jepsen worker 28 - jepsen.util 1018 :ok :read [9 37] INFO [2022-03-13 17:09:28,305] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:28,306] jepsen worker 28 - jepsen.util 1018 :invoke :read [9 nil] INFO [2022-03-13 17:09:28,307] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,307] jepsen worker 77 - jepsen.util 617 :invoke :write [25 22] INFO [2022-03-13 17:09:28,327] jepsen worker 14 - jepsen.util 194 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,328] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:28,349] jepsen worker 33 - jepsen.util 663 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,350] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:09:28,372] jepsen worker 44 - jepsen.util 494 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,372] jepsen worker 21 - jepsen.util 651 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,372] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,373] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,373] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:28,374] jepsen worker 71 - jepsen.util 431 :invoke :write [23 45] INFO [2022-03-13 17:09:28,375] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:28,375] jepsen worker 80 - jepsen.util 620 :invoke :write [26 18] INFO [2022-03-13 17:09:28,379] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,380] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,397] jepsen worker 51 - jepsen.util 411 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,398] jepsen worker 51 - jepsen.util 411 :invoke :write [17 45] INFO [2022-03-13 17:09:28,410] jepsen worker 28 - jepsen.util 1018 :ok :read [9 37] INFO [2022-03-13 17:09:28,410] jepsen worker 28 - jepsen.util 1018 :invoke :write [9 38] INFO [2022-03-13 17:09:28,412] jepsen worker 69 - jepsen.util 609 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,413] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,455] jepsen worker 13 - jepsen.util 463 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,456] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:28,480] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,480] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,481] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,482] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:09:28,483] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,484] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:09:28,504] jepsen worker 77 - jepsen.util 617 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,505] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:28,526] jepsen worker 40 - jepsen.util 850 :info :write [13 41] :RequestTimeoutException INFO [2022-03-13 17:09:28,527] jepsen worker 40 - jepsen.util 940 :invoke :write [13 19] INFO [2022-03-13 17:09:28,546] jepsen worker 2 - jepsen.util 1082 :info :write [0 10] :RequestTimeoutException INFO [2022-03-13 17:09:28,547] jepsen worker 2 - jepsen.util 1172 :invoke :write [0 20] INFO [2022-03-13 17:09:28,566] jepsen worker 34 - jepsen.util 574 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,566] jepsen worker 8 - jepsen.util 638 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,567] jepsen worker 34 - jepsen.util 574 :invoke :write [11 17] INFO [2022-03-13 17:09:28,567] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:28,597] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,597] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:28,597] jepsen worker 51 - jepsen.util 411 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,598] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:28,599] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,599] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:28,599] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:28,600] jepsen worker 51 - jepsen.util 411 :invoke :write [17 4] INFO [2022-03-13 17:09:28,601] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,601] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:09:28,601] jepsen worker 77 - jepsen.util 617 :invoke :write [25 15] INFO [2022-03-13 17:09:28,602] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:28,603] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:28,604] jepsen worker 13 - jepsen.util 463 :invoke :write [4 3] INFO [2022-03-13 17:09:28,658] jepsen worker 6 - jepsen.util 546 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,659] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:28,663] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,664] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:28,686] jepsen worker 49 - jepsen.util 859 :info :write [16 15] :RequestTimeoutException INFO [2022-03-13 17:09:28,686] jepsen worker 49 - jepsen.util 949 :invoke :read [16 nil] INFO [2022-03-13 17:09:28,706] jepsen worker 13 - jepsen.util 463 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,706] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:28,730] jepsen worker 43 - jepsen.util 403 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,730] jepsen worker 43 - jepsen.util 403 :invoke :write [14 9] INFO [2022-03-13 17:09:28,738] jepsen worker 52 - jepsen.util 502 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,738] jepsen worker 52 - jepsen.util 502 :invoke :write [17 24] INFO [2022-03-13 17:09:28,744] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1759530791788856}, :ns_1@172.23.105.187 {:progress 0.216374269005848}, :ns_1@172.23.107.227 {:progress 0.1759530791788856}, :ns_1@172.23.107.228 {:progress 0.1529411764705882}} INFO [2022-03-13 17:09:28,744] jepsen worker nemesis - couchbase.util Rebalance check count: 81 INFO [2022-03-13 17:09:28,744] jepsen worker nemesis - couchbase.util Rebalance stuck count: 44 INFO [2022-03-13 17:09:28,779] jepsen worker 77 - jepsen.util 617 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,780] jepsen worker 77 - jepsen.util 617 :invoke :write [25 15] INFO [2022-03-13 17:09:28,782] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,782] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:28,785] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,785] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:09:28,788] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,789] jepsen worker 69 - jepsen.util 609 :invoke :write [23 44] INFO [2022-03-13 17:09:28,856] jepsen worker 81 - jepsen.util 1161 :info :write [27 5] :RequestTimeoutException INFO [2022-03-13 17:09:28,856] jepsen worker 81 - jepsen.util 1251 :invoke :read [27 nil] INFO [2022-03-13 17:09:28,879] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,880] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:09:28,894] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:28,894] jepsen worker 51 - jepsen.util 411 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,895] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:28,895] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:28,923] jepsen worker 22 - jepsen.util 472 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,923] jepsen worker 22 - jepsen.util 472 :invoke :write [7 12] INFO [2022-03-13 17:09:28,983] jepsen worker 69 - jepsen.util 609 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:28,984] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,074] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,074] jepsen worker 25 - jepsen.util 385 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,075] jepsen worker 25 - jepsen.util 385 :invoke :write [8 3] INFO [2022-03-13 17:09:29,076] jepsen worker 7 - jepsen.util 457 :invoke :write [2 11] INFO [2022-03-13 17:09:29,083] jepsen worker 77 - jepsen.util 617 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,083] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:29,083] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,084] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:29,084] jepsen worker 13 - jepsen.util 463 :invoke :write [4 37] INFO [2022-03-13 17:09:29,085] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:29,085] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,085] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:29,086] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,086] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,087] jepsen worker 51 - jepsen.util 411 :invoke :write [17 36] INFO [2022-03-13 17:09:29,087] jepsen worker 77 - jepsen.util 617 :invoke :write [25 22] INFO [2022-03-13 17:09:29,124] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,124] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,125] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:09:29,125] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,126] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,127] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,128] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,128] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,129] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,130] jepsen worker 69 - jepsen.util 609 :invoke :write [23 23] INFO [2022-03-13 17:09:29,312] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,312] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,316] jepsen worker 13 - jepsen.util 463 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,316] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:29,324] jepsen worker 79 - jepsen.util 799 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,324] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:29,324] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,324] jepsen worker 78 - jepsen.util 438 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,324] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:29,325] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,326] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:29,326] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:29,327] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:29,327] jepsen worker 44 - jepsen.util 494 :invoke :write [14 31] INFO [2022-03-13 17:09:29,376] jepsen worker 48 - jepsen.util 858 :info :write [16 23] :RequestTimeoutException INFO [2022-03-13 17:09:29,376] jepsen worker 48 - jepsen.util 948 :invoke :read [16 nil] INFO [2022-03-13 17:09:29,378] jepsen worker 48 - jepsen.util 948 :ok :read [16 :nil] INFO [2022-03-13 17:09:29,378] jepsen worker 48 - jepsen.util 948 :invoke :read [16 nil] INFO [2022-03-13 17:09:29,379] jepsen worker 48 - jepsen.util 948 :ok :read [16 :nil] INFO [2022-03-13 17:09:29,380] jepsen worker 48 - jepsen.util 948 :invoke :write [16 23] INFO [2022-03-13 17:09:29,481] jepsen worker 38 - jepsen.util 1208 :ok :read [12 2] INFO [2022-03-13 17:09:29,481] jepsen worker 38 - jepsen.util 1208 :invoke :write [12 21] INFO [2022-03-13 17:09:29,593] jepsen worker 71 - jepsen.util 431 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,593] jepsen worker 80 - jepsen.util 620 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,594] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,595] jepsen worker 8 - jepsen.util 638 :invoke :write [2 19] INFO [2022-03-13 17:09:29,596] jepsen worker 71 - jepsen.util 431 :invoke :write [23 7] INFO [2022-03-13 17:09:29,597] jepsen worker 80 - jepsen.util 620 :invoke :write [26 49] INFO [2022-03-13 17:09:29,603] jepsen worker 5 - jepsen.util 995 :info :write [1 4] :RequestTimeoutException INFO [2022-03-13 17:09:29,604] jepsen worker 5 - jepsen.util 1085 :invoke :write [1 8] INFO [2022-03-13 17:09:29,612] jepsen worker 51 - jepsen.util 411 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,614] jepsen worker 51 - jepsen.util 411 :invoke :write [17 22] INFO [2022-03-13 17:09:29,636] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,640] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,640] jepsen worker 69 - jepsen.util 609 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,640] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,641] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,642] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,642] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,643] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:09:29,643] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,644] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,645] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,646] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,647] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,648] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,650] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,652] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,653] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,654] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,655] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,656] jepsen worker 33 - jepsen.util 663 :invoke :write [11 0] INFO [2022-03-13 17:09:29,667] jepsen worker 52 - jepsen.util 502 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,667] jepsen worker 34 - jepsen.util 574 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,667] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,676] jepsen worker 52 - jepsen.util 502 :invoke :write [17 35] INFO [2022-03-13 17:09:29,677] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:29,677] jepsen worker 34 - jepsen.util 574 :invoke :write [11 26] INFO [2022-03-13 17:09:29,719] jepsen worker 22 - jepsen.util 472 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,719] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,719] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,720] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:29,721] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,721] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:29,728] jepsen worker 77 - jepsen.util 617 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,728] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:29,729] jepsen worker 13 - jepsen.util 463 :invoke :write [4 24] INFO [2022-03-13 17:09:29,730] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:29,751] jepsen worker 12 - jepsen.util 642 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,752] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:29,756] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1759530791788856}, :ns_1@172.23.105.187 {:progress 0.216374269005848}, :ns_1@172.23.107.227 {:progress 0.1759530791788856}, :ns_1@172.23.107.228 {:progress 0.1529411764705882}} INFO [2022-03-13 17:09:29,756] jepsen worker nemesis - couchbase.util Rebalance check count: 82 INFO [2022-03-13 17:09:29,756] jepsen worker nemesis - couchbase.util Rebalance stuck count: 45 INFO [2022-03-13 17:09:29,787] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,788] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:29,795] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,795] jepsen worker 51 - jepsen.util 411 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,796] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:29,797] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:29,802] jepsen worker 7 - jepsen.util 457 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,802] jepsen worker 7 - jepsen.util 457 :invoke :write [2 24] INFO [2022-03-13 17:09:29,851] jepsen worker 13 - jepsen.util 463 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,851] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,851] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,852] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:29,852] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:29,853] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:29,864] jepsen worker 41 - jepsen.util 941 :info :write [13 3] :RequestTimeoutException INFO [2022-03-13 17:09:29,864] jepsen worker 41 - jepsen.util 1031 :invoke :write [13 26] INFO [2022-03-13 17:09:29,876] jepsen worker 33 - jepsen.util 663 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,876] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,877] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:29,878] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:29,878] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:09:29,878] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,879] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,880] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:29,881] jepsen worker 77 - jepsen.util 617 :invoke :write [25 33] INFO [2022-03-13 17:09:29,881] jepsen worker 13 - jepsen.util 463 :invoke :write [4 21] INFO [2022-03-13 17:09:29,883] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,884] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:29,885] jepsen worker 51 - jepsen.util 411 :invoke :write [17 48] INFO [2022-03-13 17:09:29,886] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:09:30,051] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,052] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:09:30,063] jepsen worker 13 - jepsen.util 463 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,063] jepsen worker 13 - jepsen.util 463 :invoke :write [4 24] INFO [2022-03-13 17:09:30,069] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,069] jepsen worker 25 - jepsen.util 385 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,069] jepsen worker 44 - jepsen.util 494 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,070] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:30,070] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:30,071] jepsen worker 25 - jepsen.util 385 :invoke :write [8 2] INFO [2022-03-13 17:09:30,072] jepsen worker 44 - jepsen.util 494 :invoke :write [14 30] INFO [2022-03-13 17:09:30,072] jepsen worker 21 - jepsen.util 651 :invoke :write [7 30] INFO [2022-03-13 17:09:30,073] jepsen worker 60 - jepsen.util 600 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,074] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:30,074] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:30,075] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:30,142] jepsen worker 51 - jepsen.util 411 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,143] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:30,536] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,536] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:09:30,564] jepsen worker 77 - jepsen.util 617 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,565] jepsen worker 77 - jepsen.util 617 :invoke :write [25 7] INFO [2022-03-13 17:09:30,598] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,598] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:30,630] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,630] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,631] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:30,631] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:30,632] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,633] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,633] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:09:30,633] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:09:30,656] jepsen worker 3 - jepsen.util 813 :info :write [1 6] :RequestTimeoutException INFO [2022-03-13 17:09:30,657] jepsen worker 3 - jepsen.util 903 :invoke :read [1 nil] INFO [2022-03-13 17:09:30,671] jepsen worker 71 - jepsen.util 431 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,671] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,672] jepsen worker 80 - jepsen.util 620 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,673] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:30,673] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:30,674] jepsen worker 71 - jepsen.util 431 :invoke :write [23 27] INFO [2022-03-13 17:09:30,674] jepsen worker 13 - jepsen.util 463 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,674] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,675] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:30,675] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:30,690] jepsen worker 3 - jepsen.util 903 :ok :read [1 42] INFO [2022-03-13 17:09:30,691] jepsen worker 3 - jepsen.util 903 :invoke :read [1 nil] INFO [2022-03-13 17:09:30,718] jepsen worker 3 - jepsen.util 903 :ok :read [1 42] INFO [2022-03-13 17:09:30,719] jepsen worker 3 - jepsen.util 903 :invoke :write [1 47] INFO [2022-03-13 17:09:30,762] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2222222222222222}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.1588235294117647}} INFO [2022-03-13 17:09:30,762] jepsen worker nemesis - couchbase.util Rebalance check count: 83 INFO [2022-03-13 17:09:30,762] jepsen worker nemesis - couchbase.util Rebalance stuck count: 45 INFO [2022-03-13 17:09:30,781] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:30,781] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,782] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:30,783] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:30,790] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,791] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:30,791] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:30,792] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:30,852] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,852] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:30,852] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,853] jepsen worker 22 - jepsen.util 472 :invoke :write [7 22] INFO [2022-03-13 17:09:30,855] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:30,855] jepsen worker 70 - jepsen.util 700 :invoke :write [23 22] INFO [2022-03-13 17:09:30,890] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:30,890] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,891] jepsen worker 13 - jepsen.util 463 :invoke :write [4 45] INFO [2022-03-13 17:09:30,891] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:30,893] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,894] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:30,897] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,897] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,898] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,897] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:30,898] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,898] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:30,899] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:30,900] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:30,902] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:30,903] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:30,906] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:30,910] jepsen worker 51 - jepsen.util 411 :invoke :write [17 9] INFO [2022-03-13 17:09:30,910] jepsen worker 15 - jepsen.util 1005 :info :write [5 10] :RequestTimeoutException INFO [2022-03-13 17:09:30,911] jepsen worker 15 - jepsen.util 1095 :invoke :read [5 nil] INFO [2022-03-13 17:09:31,109] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,109] jepsen worker 6 - jepsen.util 546 :invoke :write [2 6] INFO [2022-03-13 17:09:31,143] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,144] jepsen worker 60 - jepsen.util 600 :invoke :write [20 43] INFO [2022-03-13 17:09:31,329] jepsen worker 14 - jepsen.util 194 :ok :read [4 :nil] INFO [2022-03-13 17:09:31,329] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:31,329] jepsen worker 14 - jepsen.util 194 :invoke :write [4 12] INFO [2022-03-13 17:09:31,330] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:31,347] jepsen worker 10 - jepsen.util 820 :info :write [3 7] :SyncWriteAmbiguous INFO [2022-03-13 17:09:31,347] jepsen worker 31 - jepsen.util 1021 :ok :read [10 :nil] INFO [2022-03-13 17:09:31,348] jepsen worker 10 - jepsen.util 910 :invoke :read [3 nil] INFO [2022-03-13 17:09:31,348] jepsen worker 31 - jepsen.util 1021 :invoke :write [10 16] INFO [2022-03-13 17:09:31,419] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,419] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,419] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:31,423] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:31,424] jepsen worker 53 - jepsen.util 503 :invoke :write [17 0] INFO [2022-03-13 17:09:31,424] jepsen worker 35 - jepsen.util 665 :invoke :write [11 46] INFO [2022-03-13 17:09:31,464] jepsen worker 65 - jepsen.util 1055 :ok :read [21 :nil] INFO [2022-03-13 17:09:31,465] jepsen worker 65 - jepsen.util 1055 :invoke :write [21 25] INFO [2022-03-13 17:09:31,479] jepsen worker 55 - jepsen.util 1045 :ok :read [18 49] INFO [2022-03-13 17:09:31,479] jepsen worker 55 - jepsen.util 1045 :invoke :write [18 33] INFO [2022-03-13 17:09:31,655] jepsen worker 76 - jepsen.util 526 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,655] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,655] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,656] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:31,656] jepsen worker 75 - jepsen.util 345 :invoke :write [25 31] INFO [2022-03-13 17:09:31,657] jepsen worker 76 - jepsen.util 526 :invoke :write [25 40] INFO [2022-03-13 17:09:31,668] jepsen worker 4 - jepsen.util 994 :info :write [1 49] :RequestTimeoutException INFO [2022-03-13 17:09:31,669] jepsen worker 4 - jepsen.util 1084 :invoke :read [1 nil] INFO [2022-03-13 17:09:31,680] jepsen worker 43 - jepsen.util 403 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,680] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:31,704] jepsen worker 4 - jepsen.util 1084 :ok :read [1 42] INFO [2022-03-13 17:09:31,705] jepsen worker 4 - jepsen.util 1084 :invoke :write [1 3] INFO [2022-03-13 17:09:31,714] jepsen worker 13 - jepsen.util 463 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,715] jepsen worker 13 - jepsen.util 463 :invoke :write [4 48] INFO [2022-03-13 17:09:31,721] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:31,721] jepsen worker 42 - jepsen.util 402 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,722] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:31,723] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] INFO [2022-03-13 17:09:31,752] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,753] jepsen worker 79 - jepsen.util 799 :invoke :write [26 3] INFO [2022-03-13 17:09:31,760] jepsen worker 7 - jepsen.util 457 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,760] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:31,760] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,761] jepsen worker 7 - jepsen.util 457 :invoke :write [2 4] INFO [2022-03-13 17:09:31,762] jepsen worker 78 - jepsen.util 438 :invoke :write [26 17] INFO [2022-03-13 17:09:31,763] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:31,775] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2339181286549707}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.1705882352941176}} INFO [2022-03-13 17:09:31,775] jepsen worker nemesis - couchbase.util Rebalance check count: 84 INFO [2022-03-13 17:09:31,775] jepsen worker nemesis - couchbase.util Rebalance stuck count: 45 INFO [2022-03-13 17:09:31,794] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,795] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:09:31,818] jepsen worker 60 - jepsen.util 600 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,822] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:31,868] jepsen worker 11 - jepsen.util 1001 :ok :read [3 27] INFO [2022-03-13 17:09:31,869] jepsen worker 11 - jepsen.util 1001 :invoke :read [3 nil] INFO [2022-03-13 17:09:31,876] jepsen worker 51 - jepsen.util 411 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,876] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:31,943] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,943] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:09:31,973] jepsen worker 77 - jepsen.util 617 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:31,974] jepsen worker 77 - jepsen.util 617 :invoke :write [25 4] INFO [2022-03-13 17:09:32,020] jepsen worker 62 - jepsen.util 422 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,020] jepsen worker 62 - jepsen.util 422 :invoke :write [20 40] INFO [2022-03-13 17:09:32,031] jepsen worker 34 - jepsen.util 574 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,031] jepsen worker 25 - jepsen.util 385 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,032] jepsen worker 25 - jepsen.util 385 :invoke :write [8 20] INFO [2022-03-13 17:09:32,033] jepsen worker 34 - jepsen.util 574 :invoke :write [11 15] INFO [2022-03-13 17:09:32,072] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,072] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:32,072] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,073] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:32,073] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:32,074] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:32,075] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:32,075] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,075] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:32,075] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:32,076] jepsen worker 43 - jepsen.util 403 :invoke :write [14 20] INFO [2022-03-13 17:09:32,083] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,084] jepsen worker 23 - jepsen.util 563 :invoke :write [7 22] INFO [2022-03-13 17:09:32,084] jepsen worker 60 - jepsen.util 600 :invoke :write [20 43] INFO [2022-03-13 17:09:32,492] jepsen worker 68 - jepsen.util 968 :ok :read [22 30] INFO [2022-03-13 17:09:32,493] jepsen worker 68 - jepsen.util 968 :invoke :read [22 nil] INFO [2022-03-13 17:09:32,527] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,528] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:32,532] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:32,532] jepsen worker 14 - jepsen.util 194 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,533] jepsen worker 14 - jepsen.util 194 :invoke :write [4 48] INFO [2022-03-13 17:09:32,533] jepsen worker 26 - jepsen.util 476 :invoke :write [8 20] INFO [2022-03-13 17:09:32,534] jepsen worker 13 - jepsen.util 463 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,535] jepsen worker 13 - jepsen.util 463 :invoke :write [4 47] INFO [2022-03-13 17:09:32,537] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,538] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:32,576] jepsen worker 49 - jepsen.util 949 :ok :read [16 :nil] INFO [2022-03-13 17:09:32,576] jepsen worker 49 - jepsen.util 949 :invoke :read [16 nil] INFO [2022-03-13 17:09:32,598] jepsen worker 74 - jepsen.util 1154 :info :write [24 32] :RequestTimeoutException INFO [2022-03-13 17:09:32,599] jepsen worker 74 - jepsen.util 1244 :invoke :write [24 8] INFO [2022-03-13 17:09:32,611] jepsen worker 15 - jepsen.util 1095 :ok :read [5 43] INFO [2022-03-13 17:09:32,612] jepsen worker 15 - jepsen.util 1095 :invoke :write [5 3] INFO [2022-03-13 17:09:32,615] jepsen worker 68 - jepsen.util 968 :ok :read [22 30] INFO [2022-03-13 17:09:32,615] jepsen worker 68 - jepsen.util 968 :invoke :write [22 41] INFO [2022-03-13 17:09:32,617] jepsen worker 77 - jepsen.util 617 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,617] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,618] jepsen worker 51 - jepsen.util 411 :invoke :write [17 18] INFO [2022-03-13 17:09:32,623] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:32,623] jepsen worker 49 - jepsen.util 949 :ok :read [16 :nil] INFO [2022-03-13 17:09:32,624] jepsen worker 49 - jepsen.util 949 :invoke :read [16 nil] INFO [2022-03-13 17:09:32,642] jepsen worker 53 - jepsen.util 503 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,643] jepsen worker 53 - jepsen.util 503 :invoke :write [17 43] INFO [2022-03-13 17:09:32,647] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,647] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,648] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:32,649] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:32,654] jepsen worker 49 - jepsen.util 949 :ok :read [16 :nil] INFO [2022-03-13 17:09:32,655] jepsen worker 49 - jepsen.util 949 :invoke :read [16 nil] INFO [2022-03-13 17:09:32,664] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,664] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,665] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:32,665] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:09:32,672] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,672] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:32,673] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,674] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:32,675] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,676] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:09:32,681] jepsen worker 21 - jepsen.util 651 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,682] jepsen worker 22 - jepsen.util 472 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,682] jepsen worker 21 - jepsen.util 651 :invoke :write [7 15] INFO [2022-03-13 17:09:32,683] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:32,687] jepsen worker 43 - jepsen.util 403 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,688] jepsen worker 43 - jepsen.util 403 :invoke :write [14 10] INFO [2022-03-13 17:09:32,712] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,713] jepsen worker 13 - jepsen.util 463 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,714] jepsen worker 77 - jepsen.util 617 :invoke :write [25 8] INFO [2022-03-13 17:09:32,715] jepsen worker 13 - jepsen.util 463 :invoke :write [4 1] INFO [2022-03-13 17:09:32,781] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2339181286549707}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.1705882352941176}} INFO [2022-03-13 17:09:32,781] jepsen worker nemesis - couchbase.util Rebalance check count: 85 INFO [2022-03-13 17:09:32,781] jepsen worker nemesis - couchbase.util Rebalance stuck count: 46 INFO [2022-03-13 17:09:32,795] jepsen worker 49 - jepsen.util 949 :ok :read [16 :nil] INFO [2022-03-13 17:09:32,796] jepsen worker 49 - jepsen.util 949 :invoke :read [16 nil] INFO [2022-03-13 17:09:32,813] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,814] jepsen worker 69 - jepsen.util 609 :invoke :write [23 21] INFO [2022-03-13 17:09:32,837] jepsen worker 56 - jepsen.util 866 :info :write [18 45] :RequestTimeoutException INFO [2022-03-13 17:09:32,837] jepsen worker 56 - jepsen.util 956 :invoke :read [18 nil] INFO [2022-03-13 17:09:32,864] jepsen worker 35 - jepsen.util 665 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,865] jepsen worker 35 - jepsen.util 665 :invoke :write [11 29] INFO [2022-03-13 17:09:32,867] jepsen worker 51 - jepsen.util 411 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,868] jepsen worker 51 - jepsen.util 411 :invoke :write [17 15] INFO [2022-03-13 17:09:32,877] jepsen worker 70 - jepsen.util 700 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,878] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:32,881] jepsen worker 18 - jepsen.util 828 :info :write [6 46] :RequestTimeoutException INFO [2022-03-13 17:09:32,881] jepsen worker 18 - jepsen.util 918 :invoke :read [6 nil] INFO [2022-03-13 17:09:32,890] jepsen worker 20 - jepsen.util 1190 :info :write [6 33] :RequestTimeoutException INFO [2022-03-13 17:09:32,891] jepsen worker 20 - jepsen.util 1280 :invoke :write [6 25] INFO [2022-03-13 17:09:32,892] jepsen worker 49 - jepsen.util 949 :ok :read [16 :nil] INFO [2022-03-13 17:09:32,892] jepsen worker 49 - jepsen.util 949 :invoke :write [16 5] INFO [2022-03-13 17:09:32,896] jepsen worker 27 - jepsen.util 1107 :info :write [9 9] :RequestTimeoutException INFO [2022-03-13 17:09:32,896] jepsen worker 27 - jepsen.util 1197 :invoke :write [9 9] INFO [2022-03-13 17:09:32,909] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,910] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:32,937] jepsen worker 8 - jepsen.util 638 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,937] jepsen worker 23 - jepsen.util 563 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,938] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:32,938] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:32,962] jepsen worker 77 - jepsen.util 617 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,962] jepsen worker 77 - jepsen.util 617 :invoke :write [25 3] INFO [2022-03-13 17:09:32,998] jepsen worker 69 - jepsen.util 609 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:32,999] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:09:32,999] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,000] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:09:33,137] jepsen worker 13 - jepsen.util 463 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,137] jepsen worker 87 - jepsen.util 1347 :info :write [29 41] :RequestTimeoutException INFO [2022-03-13 17:09:33,138] jepsen worker 13 - jepsen.util 463 :invoke :write [4 30] INFO [2022-03-13 17:09:33,138] jepsen worker 87 - jepsen.util 1437 :invoke :write [29 29] INFO [2022-03-13 17:09:33,141] jepsen worker 44 - jepsen.util 494 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,141] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:33,150] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,150] jepsen worker 75 - jepsen.util 345 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,151] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:33,151] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:33,155] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,156] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:33,215] jepsen worker 52 - jepsen.util 502 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,216] jepsen worker 52 - jepsen.util 502 :invoke :write [17 25] INFO [2022-03-13 17:09:33,242] jepsen worker 60 - jepsen.util 600 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,243] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:33,255] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,256] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:33,256] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,257] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:33,258] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,258] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,259] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:33,259] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:09:33,260] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,261] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:09:33,266] jepsen worker 51 - jepsen.util 411 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,267] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,376] jepsen worker 36 - jepsen.util 1116 :info :write [12 29] :RequestTimeoutException INFO [2022-03-13 17:09:33,377] jepsen worker 36 - jepsen.util 1206 :invoke :read [12 nil] INFO [2022-03-13 17:09:33,404] jepsen worker 17 - jepsen.util 917 :info :write [5 21] :RequestTimeoutException INFO [2022-03-13 17:09:33,404] jepsen worker 17 - jepsen.util 1007 :invoke :read [5 nil] INFO [2022-03-13 17:09:33,441] jepsen worker 77 - jepsen.util 617 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,441] jepsen worker 77 - jepsen.util 617 :invoke :write [25 18] INFO [2022-03-13 17:09:33,448] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,448] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:33,452] jepsen worker 78 - jepsen.util 438 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,452] jepsen worker 6 - jepsen.util 546 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,453] jepsen worker 6 - jepsen.util 546 :invoke :write [2 17] INFO [2022-03-13 17:09:33,453] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:33,465] jepsen worker 46 - jepsen.util 1036 :info :write [15 37] :RequestTimeoutException INFO [2022-03-13 17:09:33,466] jepsen worker 46 - jepsen.util 1126 :invoke :write [15 11] INFO [2022-03-13 17:09:33,488] jepsen worker 76 - jepsen.util 526 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,488] jepsen worker 43 - jepsen.util 403 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,488] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:33,488] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,489] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:33,490] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:33,490] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:33,491] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:33,492] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,492] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:33,492] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:33,494] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:33,494] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:33,495] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:09:33,505] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:33,505] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,506] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:33,507] jepsen worker 43 - jepsen.util 403 :invoke :write [14 44] INFO [2022-03-13 17:09:33,508] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,510] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:33,511] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,512] jepsen worker 13 - jepsen.util 463 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,513] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:33,513] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,513] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:33,514] jepsen worker 51 - jepsen.util 411 :invoke :write [17 0] INFO [2022-03-13 17:09:33,580] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,580] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,581] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:09:33,581] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:09:33,585] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:33,585] jepsen worker 7 - jepsen.util 457 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,585] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,585] jepsen worker 79 - jepsen.util 799 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,585] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:33,586] jepsen worker 22 - jepsen.util 472 :invoke :write [7 44] INFO [2022-03-13 17:09:33,586] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:33,587] jepsen worker 80 - jepsen.util 620 :invoke :write [26 28] INFO [2022-03-13 17:09:33,623] jepsen worker 26 - jepsen.util 476 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,623] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,624] jepsen worker 60 - jepsen.util 600 :invoke :write [20 42] INFO [2022-03-13 17:09:33,628] jepsen worker 26 - jepsen.util 476 :invoke :write [8 9] INFO [2022-03-13 17:09:33,660] jepsen worker 77 - jepsen.util 617 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,660] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:33,661] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:33,662] jepsen worker 77 - jepsen.util 617 :invoke :write [25 27] INFO [2022-03-13 17:09:33,740] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,741] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:09:33,755] jepsen worker 51 - jepsen.util 411 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,755] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:33,756] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:33,756] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,760] jepsen worker 14 - jepsen.util 194 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,760] jepsen worker 14 - jepsen.util 194 :invoke :write [4 20] INFO [2022-03-13 17:09:33,780] jepsen worker 71 - jepsen.util 431 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,781] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,781] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,780] jepsen worker 21 - jepsen.util 651 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,781] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:33,781] jepsen worker 21 - jepsen.util 651 :invoke :write [7 18] INFO [2022-03-13 17:09:33,785] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:33,786] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:33,786] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:09:33,786] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:33,788] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2339181286549707}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.1705882352941176}} INFO [2022-03-13 17:09:33,788] jepsen worker nemesis - couchbase.util Rebalance check count: 86 INFO [2022-03-13 17:09:33,788] jepsen worker nemesis - couchbase.util Rebalance stuck count: 47 INFO [2022-03-13 17:09:33,850] jepsen worker 77 - jepsen.util 617 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,850] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:33,850] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,851] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:33,851] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,852] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:33,852] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,852] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,853] jepsen worker 33 - jepsen.util 663 :invoke :write [11 1] INFO [2022-03-13 17:09:33,853] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:33,872] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,872] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:33,873] jepsen worker 77 - jepsen.util 617 :invoke :write [25 44] INFO [2022-03-13 17:09:33,873] jepsen worker 13 - jepsen.util 463 :invoke :write [4 29] INFO [2022-03-13 17:09:33,875] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,876] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,877] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,877] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,879] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,879] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,880] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,880] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,881] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,882] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,883] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,883] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:33,884] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,885] jepsen worker 51 - jepsen.util 411 :invoke :write [17 21] INFO [2022-03-13 17:09:33,891] jepsen worker 34 - jepsen.util 574 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,892] jepsen worker 34 - jepsen.util 574 :invoke :write [11 26] INFO [2022-03-13 17:09:33,932] jepsen worker 25 - jepsen.util 385 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,933] jepsen worker 25 - jepsen.util 385 :invoke :write [8 16] INFO [2022-03-13 17:09:33,953] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,953] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:09:33,984] jepsen worker 53 - jepsen.util 503 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:33,984] jepsen worker 53 - jepsen.util 503 :invoke :write [17 16] INFO [2022-03-13 17:09:34,021] jepsen worker 13 - jepsen.util 463 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,022] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:34,070] jepsen worker 89 - jepsen.util 1169 :info :write [29 25] :RequestTimeoutException INFO [2022-03-13 17:09:34,070] jepsen worker 89 - jepsen.util 1259 :invoke :read [29 nil] INFO [2022-03-13 17:09:34,104] jepsen worker 43 - jepsen.util 403 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,105] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:34,121] jepsen worker 33 - jepsen.util 663 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,121] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:09:34,124] jepsen worker 89 - jepsen.util 1259 :ok :read [29 33] INFO [2022-03-13 17:09:34,125] jepsen worker 89 - jepsen.util 1259 :invoke :write [29 6] INFO [2022-03-13 17:09:34,173] jepsen worker 77 - jepsen.util 617 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,173] jepsen worker 77 - jepsen.util 617 :invoke :write [25 41] INFO [2022-03-13 17:09:34,181] jepsen worker 6 - jepsen.util 546 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,181] jepsen worker 52 - jepsen.util 502 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,181] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,181] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,181] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:34,181] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,183] jepsen worker 52 - jepsen.util 502 :invoke :write [17 16] INFO [2022-03-13 17:09:34,183] jepsen worker 79 - jepsen.util 799 :invoke :read [26 nil] INFO [2022-03-13 17:09:34,184] jepsen worker 78 - jepsen.util 438 :invoke :write [26 30] INFO [2022-03-13 17:09:34,191] jepsen worker 7 - jepsen.util 457 :invoke :write [2 45] INFO [2022-03-13 17:09:34,423] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,423] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:34,456] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,456] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,456] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:34,457] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:09:34,467] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,468] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:09:34,504] jepsen worker 35 - jepsen.util 665 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,504] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,504] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:34,504] jepsen worker 60 - jepsen.util 600 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,505] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,505] jepsen worker 35 - jepsen.util 665 :invoke :write [11 22] INFO [2022-03-13 17:09:34,506] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:34,506] jepsen worker 43 - jepsen.util 403 :invoke :write [14 46] INFO [2022-03-13 17:09:34,507] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:34,506] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:34,507] jepsen worker 51 - jepsen.util 411 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,507] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:34,507] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:34,507] jepsen worker 61 - jepsen.util 421 :invoke :write [20 20] INFO [2022-03-13 17:09:34,508] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:34,509] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:34,509] jepsen worker 51 - jepsen.util 411 :invoke :write [17 30] INFO [2022-03-13 17:09:34,509] jepsen worker 23 - jepsen.util 563 :invoke :write [7 8] INFO [2022-03-13 17:09:34,518] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,519] jepsen worker 60 - jepsen.util 600 :invoke :write [20 26] INFO [2022-03-13 17:09:34,591] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,592] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:34,597] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:34,597] jepsen worker 77 - jepsen.util 617 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,597] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:34,598] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:34,600] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,600] jepsen worker 80 - jepsen.util 620 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,600] jepsen worker 22 - jepsen.util 472 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,600] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,601] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:34,602] jepsen worker 22 - jepsen.util 472 :invoke :write [7 11] INFO [2022-03-13 17:09:34,603] jepsen worker 8 - jepsen.util 638 :invoke :write [2 27] INFO [2022-03-13 17:09:34,603] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:34,605] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:34,605] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:34,614] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,614] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,615] jepsen worker 76 - jepsen.util 526 :invoke :write [25 0] INFO [2022-03-13 17:09:34,616] jepsen worker 42 - jepsen.util 402 :invoke :write [14 28] INFO [2022-03-13 17:09:34,695] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,695] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,695] jepsen worker 51 - jepsen.util 411 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,695] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,695] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:34,696] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:34,697] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:34,697] jepsen worker 77 - jepsen.util 617 :invoke :write [25 12] INFO [2022-03-13 17:09:34,698] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:34,698] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:34,698] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,698] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:34,699] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:34,699] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,699] jepsen worker 13 - jepsen.util 463 :invoke :write [4 4] INFO [2022-03-13 17:09:34,699] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,700] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,700] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:34,701] jepsen worker 69 - jepsen.util 609 :invoke :write [23 1] INFO [2022-03-13 17:09:34,701] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:09:34,702] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,703] jepsen worker 51 - jepsen.util 411 :invoke :write [17 11] INFO [2022-03-13 17:09:34,725] jepsen worker 43 - jepsen.util 403 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,726] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:34,770] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,770] jepsen worker 21 - jepsen.util 651 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,770] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:34,771] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:34,771] jepsen worker 21 - jepsen.util 651 :invoke :write [7 21] INFO [2022-03-13 17:09:34,771] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:34,791] jepsen worker 62 - jepsen.util 422 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,791] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:34,795] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2456140350877193}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.1823529411764706}} INFO [2022-03-13 17:09:34,795] jepsen worker nemesis - couchbase.util Rebalance check count: 87 INFO [2022-03-13 17:09:34,795] jepsen worker nemesis - couchbase.util Rebalance stuck count: 47 INFO [2022-03-13 17:09:34,828] jepsen worker 13 - jepsen.util 463 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,829] jepsen worker 13 - jepsen.util 463 :invoke :write [4 36] INFO [2022-03-13 17:09:34,830] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,831] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:09:34,887] jepsen worker 60 - jepsen.util 600 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,888] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:34,913] jepsen worker 34 - jepsen.util 574 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,913] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,913] jepsen worker 79 - jepsen.util 799 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,913] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,913] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:34,914] jepsen worker 34 - jepsen.util 574 :invoke :write [11 42] INFO [2022-03-13 17:09:34,914] jepsen worker 79 - jepsen.util 799 :invoke :write [26 19] INFO [2022-03-13 17:09:34,915] jepsen worker 71 - jepsen.util 431 :invoke :write [23 39] INFO [2022-03-13 17:09:34,916] jepsen worker 6 - jepsen.util 546 :invoke :write [2 7] INFO [2022-03-13 17:09:34,922] jepsen worker 44 - jepsen.util 494 :invoke :write [14 14] INFO [2022-03-13 17:09:34,931] jepsen worker 69 - jepsen.util 609 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,932] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:09:34,942] jepsen worker 77 - jepsen.util 617 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:34,942] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:35,004] jepsen worker 26 - jepsen.util 476 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,005] jepsen worker 26 - jepsen.util 476 :invoke :write [8 32] INFO [2022-03-13 17:09:35,008] jepsen worker 25 - jepsen.util 385 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,008] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:35,073] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,073] jepsen worker 51 - jepsen.util 411 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,073] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:35,074] jepsen worker 51 - jepsen.util 411 :invoke :write [17 45] INFO [2022-03-13 17:09:35,129] jepsen worker 23 - jepsen.util 563 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,129] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:35,129] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,129] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:09:35,130] jepsen worker 60 - jepsen.util 600 :invoke :write [20 29] INFO [2022-03-13 17:09:35,131] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:35,159] jepsen worker 30 - jepsen.util 930 :ok :read [10 :nil] INFO [2022-03-13 17:09:35,160] jepsen worker 30 - jepsen.util 930 :invoke :read [10 nil] INFO [2022-03-13 17:09:35,161] jepsen worker 13 - jepsen.util 463 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,161] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,162] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:35,162] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:35,175] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,176] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:09:35,182] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,184] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:35,190] jepsen worker 72 - jepsen.util 972 :ok :read [24 24] INFO [2022-03-13 17:09:35,190] jepsen worker 72 - jepsen.util 972 :invoke :read [24 nil] INFO [2022-03-13 17:09:35,212] jepsen worker 19 - jepsen.util 829 :ok :read [6 16] INFO [2022-03-13 17:09:35,213] jepsen worker 19 - jepsen.util 829 :invoke :write [6 41] INFO [2022-03-13 17:09:35,253] jepsen worker 30 - jepsen.util 930 :ok :read [10 :nil] INFO [2022-03-13 17:09:35,253] jepsen worker 30 - jepsen.util 930 :invoke :read [10 nil] INFO [2022-03-13 17:09:35,286] jepsen worker 72 - jepsen.util 972 :ok :read [24 24] INFO [2022-03-13 17:09:35,287] jepsen worker 72 - jepsen.util 972 :invoke :write [24 13] INFO [2022-03-13 17:09:35,359] jepsen worker 30 - jepsen.util 930 :ok :read [10 :nil] INFO [2022-03-13 17:09:35,360] jepsen worker 30 - jepsen.util 930 :invoke :read [10 nil] INFO [2022-03-13 17:09:35,451] jepsen worker 30 - jepsen.util 930 :ok :read [10 :nil] INFO [2022-03-13 17:09:35,452] jepsen worker 30 - jepsen.util 930 :invoke :read [10 nil] INFO [2022-03-13 17:09:35,512] jepsen worker 30 - jepsen.util 930 :ok :read [10 :nil] INFO [2022-03-13 17:09:35,513] jepsen worker 30 - jepsen.util 930 :invoke :read [10 nil] INFO [2022-03-13 17:09:35,570] jepsen worker 30 - jepsen.util 930 :ok :read [10 :nil] INFO [2022-03-13 17:09:35,571] jepsen worker 30 - jepsen.util 930 :invoke :read [10 nil] WARN [2022-03-13 17:09:35,584] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8912806,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8912806,"total_server_duration_us":23475,"encode_duration_us":32291,"operation_id":"0x2b0a","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10009626},{"operation_name":"upsert","last_dispatch_duration_us":8392789,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8392789,"total_server_duration_us":22306,"encode_duration_us":17703,"operation_id":"0x2b2e","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10005390},{"operation_name":"upsert","last_dispatch_duration_us":47325,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6203349,"total_server_duration_us":214854,"encode_duration_us":22233,"operation_id":"0x2a52","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10005120},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":21143,"operation_id":"0x2b09","timeout_ms":10000,"total_duration_us":10003757},{"operation_name":"get","last_dispatch_duration_us":8759317,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8759317,"total_server_duration_us":10571,"operation_id":"0x2b17","timeout_ms":30000,"last_server_duration_us":10571,"total_duration_us":8869992},{"operation_name":"get","last_dispatch_duration_us":7869892,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7869892,"total_server_duration_us":27,"operation_id":"0x2987","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":7871293},{"operation_name":"get","last_dispatch_duration_us":7295890,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7295890,"total_server_duration_us":15,"operation_id":"0x29b2","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":7297178},{"operation_name":"upsert","last_dispatch_duration_us":579553,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1329004,"total_server_duration_us":369767,"encode_duration_us":17311,"operation_id":"0x2de2","timeout_ms":10000,"last_server_duration_us":59751,"total_duration_us":1390604},{"operation_name":"upsert","last_dispatch_duration_us":1022122,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1022122,"total_server_duration_us":818720,"encode_duration_us":31057,"operation_id":"0x2cc2","timeout_ms":10000,"last_server_duration_us":818720,"total_duration_us":1027351},{"operation_name":"upsert","last_dispatch_duration_us":976167,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":976167,"total_server_duration_us":55182,"encode_duration_us":20611,"operation_id":"0x2cc9","timeout_ms":10000,"last_server_duration_us":55182,"total_duration_us":982017}],"total_count":22}} WARN [2022-03-13 17:09:35,584] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18318999,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18641896,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9091809,"encode_duration_us":22772,"operation_id":"0x26e7","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10003029},{"operation_name":"upsert","last_dispatch_duration_us":17474896,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17474896,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9067105,"encode_duration_us":19782,"operation_id":"0x258c","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":10010365},{"operation_name":"upsert","last_dispatch_duration_us":9215822,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9850990,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9145367,"encode_duration_us":29391,"operation_id":"0x26dc","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10004329}],"total_count":3}} INFO [2022-03-13 17:09:35,631] jepsen worker 30 - jepsen.util 930 :ok :read [10 :nil] INFO [2022-03-13 17:09:35,632] jepsen worker 30 - jepsen.util 930 :invoke :write [10 4] INFO [2022-03-13 17:09:35,800] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2456140350877193}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.1823529411764706}} INFO [2022-03-13 17:09:35,801] jepsen worker nemesis - couchbase.util Rebalance check count: 88 INFO [2022-03-13 17:09:35,801] jepsen worker nemesis - couchbase.util Rebalance stuck count: 48 INFO [2022-03-13 17:09:35,831] jepsen worker 57 - jepsen.util 1047 :info :write [19 40] :SyncWriteAmbiguous INFO [2022-03-13 17:09:35,831] jepsen worker 57 - jepsen.util 1137 :invoke :write [19 6] INFO [2022-03-13 17:09:35,838] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,838] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:09:35,840] jepsen worker 14 - jepsen.util 194 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,840] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:35,841] jepsen worker 14 - jepsen.util 194 :invoke :write [4 4] INFO [2022-03-13 17:09:35,841] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:35,851] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,851] jepsen worker 22 - jepsen.util 472 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,851] jepsen worker 52 - jepsen.util 502 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,852] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:35,852] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:35,853] jepsen worker 52 - jepsen.util 502 :invoke :write [17 35] INFO [2022-03-13 17:09:35,917] jepsen worker 51 - jepsen.util 411 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,917] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,917] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:35,917] jepsen worker 51 - jepsen.util 411 :invoke :write [17 6] INFO [2022-03-13 17:09:35,918] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:35,920] jepsen worker 13 - jepsen.util 463 :invoke :write [4 23] INFO [2022-03-13 17:09:35,925] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,926] jepsen worker 77 - jepsen.util 617 :invoke :write [25 27] INFO [2022-03-13 17:09:35,987] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:35,988] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:09:36,036] jepsen worker 51 - jepsen.util 411 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,037] jepsen worker 51 - jepsen.util 411 :invoke :write [17 40] INFO [2022-03-13 17:09:36,086] jepsen worker 71 - jepsen.util 431 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,086] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,086] jepsen worker 78 - jepsen.util 438 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,087] jepsen worker 53 - jepsen.util 503 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,087] jepsen worker 80 - jepsen.util 620 :invoke :write [26 4] INFO [2022-03-13 17:09:36,087] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,087] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,087] jepsen worker 12 - jepsen.util 642 :ok :read [4 :nil] INFO [2022-03-13 17:09:36,087] jepsen worker 71 - jepsen.util 431 :invoke :write [23 38] INFO [2022-03-13 17:09:36,088] jepsen worker 75 - jepsen.util 345 :invoke :write [25 44] INFO [2022-03-13 17:09:36,088] jepsen worker 12 - jepsen.util 642 :invoke :write [4 31] INFO [2022-03-13 17:09:36,090] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:36,090] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:09:36,093] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:36,100] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,102] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:09:36,218] jepsen worker 13 - jepsen.util 463 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,219] jepsen worker 13 - jepsen.util 463 :invoke :write [4 33] INFO [2022-03-13 17:09:36,289] jepsen worker 88 - jepsen.util 1078 :ok :read [29 33] INFO [2022-03-13 17:09:36,289] jepsen worker 88 - jepsen.util 1078 :invoke :read [29 nil] INFO [2022-03-13 17:09:36,292] jepsen worker 60 - jepsen.util 600 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,292] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:36,293] jepsen worker 61 - jepsen.util 421 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,293] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:36,294] jepsen worker 61 - jepsen.util 421 :invoke :write [20 11] INFO [2022-03-13 17:09:36,296] jepsen worker 60 - jepsen.util 600 :invoke :write [20 4] INFO [2022-03-13 17:09:36,297] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,298] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:36,336] jepsen worker 39 - jepsen.util 1119 :ok :read [13 39] INFO [2022-03-13 17:09:36,337] jepsen worker 39 - jepsen.util 1119 :invoke :write [13 13] INFO [2022-03-13 17:09:36,361] jepsen worker 77 - jepsen.util 617 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,362] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:36,376] jepsen worker 54 - jepsen.util 864 :info :write [18 0] :RequestTimeoutException INFO [2022-03-13 17:09:36,377] jepsen worker 54 - jepsen.util 954 :invoke :read [18 nil] INFO [2022-03-13 17:09:36,383] jepsen worker 1 - jepsen.util 901 :ok :read [0 41] INFO [2022-03-13 17:09:36,384] jepsen worker 1 - jepsen.util 901 :invoke :write [0 32] INFO [2022-03-13 17:09:36,441] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:36,441] jepsen worker 34 - jepsen.util 574 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,441] jepsen worker 8 - jepsen.util 638 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,442] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:36,443] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:36,443] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:36,453] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,453] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,454] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:36,455] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:09:36,456] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,456] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:36,457] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,458] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:09:36,476] jepsen worker 51 - jepsen.util 411 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,477] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:36,509] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,509] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,509] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:36,509] jepsen worker 35 - jepsen.util 665 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,510] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:36,510] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:36,511] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:09:36,512] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:36,565] jepsen worker 79 - jepsen.util 799 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,565] jepsen worker 79 - jepsen.util 799 :invoke :write [26 35] INFO [2022-03-13 17:09:36,575] jepsen worker 13 - jepsen.util 463 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,575] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,575] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,575] jepsen worker 13 - jepsen.util 463 :invoke :write [4 25] INFO [2022-03-13 17:09:36,576] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:36,577] jepsen worker 51 - jepsen.util 411 :invoke :write [17 49] INFO [2022-03-13 17:09:36,584] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,585] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:36,586] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,586] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:36,587] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,587] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,588] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:09:36,591] jepsen worker 77 - jepsen.util 617 :invoke :write [25 9] INFO [2022-03-13 17:09:36,603] jepsen worker 21 - jepsen.util 651 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,603] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:36,603] jepsen worker 25 - jepsen.util 385 :invoke :write [8 25] INFO [2022-03-13 17:09:36,604] jepsen worker 21 - jepsen.util 651 :invoke :write [7 40] INFO [2022-03-13 17:09:36,610] jepsen worker 32 - jepsen.util 842 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:09:36,611] jepsen worker 32 - jepsen.util 932 :invoke :read [10 nil] INFO [2022-03-13 17:09:36,679] jepsen worker 26 - jepsen.util 476 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,679] jepsen worker 60 - jepsen.util 600 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,680] jepsen worker 60 - jepsen.util 600 :invoke :write [20 40] INFO [2022-03-13 17:09:36,681] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:36,682] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:36,683] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:36,698] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,699] jepsen worker 33 - jepsen.util 663 :invoke :write [11 19] INFO [2022-03-13 17:09:36,709] jepsen worker 84 - jepsen.util 1254 :info :write [28 6] :RequestTimeoutException INFO [2022-03-13 17:09:36,710] jepsen worker 84 - jepsen.util 1344 :invoke :write [28 38] INFO [2022-03-13 17:09:36,722] jepsen worker 13 - jepsen.util 463 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,723] jepsen worker 13 - jepsen.util 463 :invoke :write [4 46] INFO [2022-03-13 17:09:36,783] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,784] jepsen worker 69 - jepsen.util 609 :invoke :write [23 0] INFO [2022-03-13 17:09:36,806] jepsen worker 67 - jepsen.util 1057 :ok :read [22 30] INFO [2022-03-13 17:09:36,806] jepsen worker 67 - jepsen.util 1057 :invoke :read [22 nil] INFO [2022-03-13 17:09:36,808] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2456140350877193}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.1823529411764706}} INFO [2022-03-13 17:09:36,808] jepsen worker nemesis - couchbase.util Rebalance check count: 89 INFO [2022-03-13 17:09:36,808] jepsen worker nemesis - couchbase.util Rebalance stuck count: 49 INFO [2022-03-13 17:09:36,821] jepsen worker 82 - jepsen.util 1072 :ok :read [27 4] INFO [2022-03-13 17:09:36,822] jepsen worker 82 - jepsen.util 1072 :invoke :write [27 1] INFO [2022-03-13 17:09:36,827] jepsen worker 58 - jepsen.util 868 :info :write [19 5] :RequestTimeoutException INFO [2022-03-13 17:09:36,828] jepsen worker 58 - jepsen.util 958 :invoke :write [19 6] INFO [2022-03-13 17:09:36,830] jepsen worker 51 - jepsen.util 411 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,832] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:36,853] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:36,853] jepsen worker 76 - jepsen.util 526 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,853] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,854] jepsen worker 43 - jepsen.util 403 :invoke :write [14 36] INFO [2022-03-13 17:09:36,855] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:09:36,857] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:36,858] jepsen worker 7 - jepsen.util 457 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,859] jepsen worker 7 - jepsen.util 457 :invoke :write [2 39] INFO [2022-03-13 17:09:36,860] jepsen worker 33 - jepsen.util 663 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,860] jepsen worker 6 - jepsen.util 546 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,861] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:36,862] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:36,898] jepsen worker 86 - jepsen.util 896 :info :write [28 47] :RequestTimeoutException INFO [2022-03-13 17:09:36,899] jepsen worker 86 - jepsen.util 986 :invoke :read [28 nil] INFO [2022-03-13 17:09:36,903] jepsen worker 81 - jepsen.util 1251 :ok :read [27 4] INFO [2022-03-13 17:09:36,905] jepsen worker 81 - jepsen.util 1251 :invoke :write [27 40] INFO [2022-03-13 17:09:36,918] jepsen worker 59 - jepsen.util 959 :info :write [19 27] :RequestTimeoutException INFO [2022-03-13 17:09:36,919] jepsen worker 59 - jepsen.util 1049 :invoke :read [19 nil] INFO [2022-03-13 17:09:36,925] jepsen worker 77 - jepsen.util 617 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,926] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:36,929] jepsen worker 86 - jepsen.util 986 :ok :read [28 44] INFO [2022-03-13 17:09:36,930] jepsen worker 86 - jepsen.util 986 :invoke :read [28 nil] INFO [2022-03-13 17:09:36,960] jepsen worker 86 - jepsen.util 986 :ok :read [28 44] INFO [2022-03-13 17:09:36,961] jepsen worker 86 - jepsen.util 986 :invoke :write [28 48] INFO [2022-03-13 17:09:36,983] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,984] jepsen worker 69 - jepsen.util 609 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,984] jepsen worker 69 - jepsen.util 609 :invoke :write [23 44] INFO [2022-03-13 17:09:36,984] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] WARN [2022-03-13 17:09:36,992] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9134995,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18648419,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9126878,"encode_duration_us":16702,"operation_id":"0x28b4","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10004137},{"operation_name":"upsert","last_dispatch_duration_us":9051404,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9578815,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9355901,"encode_duration_us":28818,"operation_id":"0x28be","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10009338},{"operation_name":"upsert","last_dispatch_duration_us":9196461,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13180326,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9128170,"encode_duration_us":18093,"operation_id":"0x2b20","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008113}],"total_count":3}} INFO [2022-03-13 17:09:36,998] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:36,999] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:09:37,018] jepsen worker 29 - jepsen.util 1019 :info :write [9 36] :RequestTimeoutException INFO [2022-03-13 17:09:37,019] jepsen worker 29 - jepsen.util 1109 :invoke :write [9 5] INFO [2022-03-13 17:09:37,035] jepsen worker 10 - jepsen.util 910 :ok :read [3 27] INFO [2022-03-13 17:09:37,037] jepsen worker 10 - jepsen.util 910 :invoke :write [3 42] INFO [2022-03-13 17:09:37,048] jepsen worker 13 - jepsen.util 463 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,049] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,049] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,049] jepsen worker 13 - jepsen.util 463 :invoke :write [4 22] INFO [2022-03-13 17:09:37,050] jepsen worker 51 - jepsen.util 411 :invoke :write [17 48] INFO [2022-03-13 17:09:37,058] jepsen worker 77 - jepsen.util 617 :invoke :write [25 1] INFO [2022-03-13 17:09:37,060] jepsen worker 47 - jepsen.util 947 :info :write [15 8] :RequestTimeoutException INFO [2022-03-13 17:09:37,061] jepsen worker 47 - jepsen.util 1037 :invoke :write [15 15] INFO [2022-03-13 17:09:37,088] jepsen worker 45 - jepsen.util 1215 :info :write [15 48] :RequestTimeoutException INFO [2022-03-13 17:09:37,089] jepsen worker 45 - jepsen.util 1305 :invoke :read [15 nil] INFO [2022-03-13 17:09:37,092] jepsen worker 60 - jepsen.util 600 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,092] jepsen worker 42 - jepsen.util 402 :fail :write [14 28] :ServerOutOfMemoryException WARN [2022-03-13 17:09:37,092] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8296783,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13191883,"total_server_duration_us":248379,"encode_duration_us":26429,"operation_id":"0x2ae2","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10009436},{"operation_name":"upsert","last_dispatch_duration_us":9051404,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9578815,"total_server_duration_us":9355901,"encode_duration_us":28818,"operation_id":"0x28be","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10009338},{"operation_name":"upsert","last_dispatch_duration_us":9196461,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":13180326,"total_server_duration_us":9128170,"encode_duration_us":18093,"operation_id":"0x2b20","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008113},{"operation_name":"upsert","last_dispatch_duration_us":56089,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":8254889,"total_server_duration_us":275980,"encode_duration_us":18313,"operation_id":"0x2a84","timeout_ms":10000,"last_server_duration_us":37344,"total_duration_us":10007041},{"operation_name":"upsert","last_dispatch_duration_us":9134995,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18648419,"total_server_duration_us":9126878,"encode_duration_us":16702,"operation_id":"0x28b4","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10004137},{"operation_name":"upsert","last_dispatch_duration_us":8873810,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9382219,"total_server_duration_us":113054,"encode_duration_us":18535,"operation_id":"0x2cac","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10002939},{"operation_name":"upsert","last_dispatch_duration_us":8918158,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9567278,"total_server_duration_us":165942,"encode_duration_us":23871,"operation_id":"0x2ca3","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10002166},{"operation_name":"upsert","last_dispatch_duration_us":85693,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":543204,"total_server_duration_us":110280,"encode_duration_us":31368,"operation_id":"0x2ca9","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10001122},{"operation_name":"get","last_dispatch_duration_us":8880678,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8880678,"total_server_duration_us":62,"operation_id":"0x2cd6","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":8882886},{"operation_name":"get","last_dispatch_duration_us":5732670,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":5732670,"total_server_duration_us":43,"operation_id":"0x2b5c","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5733984}],"total_count":28}} INFO [2022-03-13 17:09:37,093] jepsen worker 42 - jepsen.util 402 :invoke :write [14 21] INFO [2022-03-13 17:09:37,093] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] WARN [2022-03-13 17:09:37,095] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17190618,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17190618,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9023573,"encode_duration_us":21815,"operation_id":"0x27ea","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10006086}],"total_count":1}} WARN [2022-03-13 17:09:37,103] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16161931,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":16161931,"total_server_duration_us":32,"operation_id":"0x2a36","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":16162953},{"operation_name":"get","last_dispatch_duration_us":11522260,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11522260,"total_server_duration_us":186,"operation_id":"0x2be5","timeout_ms":30000,"last_server_duration_us":186,"total_duration_us":11524338},{"operation_name":"upsert","last_dispatch_duration_us":7044575,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9903258,"total_server_duration_us":446341,"encode_duration_us":21012,"operation_id":"0x2b0c","timeout_ms":10000,"last_server_duration_us":33426,"total_duration_us":10010224},{"operation_name":"upsert","last_dispatch_duration_us":31790,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1137644,"total_server_duration_us":643578,"encode_duration_us":17748,"operation_id":"0x2b27","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10010004},{"operation_name":"upsert","last_dispatch_duration_us":9633621,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10416475,"total_server_duration_us":9394069,"encode_duration_us":18814,"operation_id":"0x2b04","timeout_ms":10000,"last_server_duration_us":9020392,"total_duration_us":10006926},{"operation_name":"upsert","last_dispatch_duration_us":17955189,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17955189,"total_server_duration_us":9037366,"encode_duration_us":17731,"operation_id":"0x2984","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10003495},{"operation_name":"upsert","last_dispatch_duration_us":18034275,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18149867,"total_server_duration_us":9059996,"encode_duration_us":19838,"operation_id":"0x2970","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10002354},{"operation_name":"upsert","last_dispatch_duration_us":9112286,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9112286,"total_server_duration_us":9043735,"encode_duration_us":18019,"operation_id":"0x2962","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":9129830},{"operation_name":"upsert","last_dispatch_duration_us":9084415,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9084415,"total_server_duration_us":9049045,"encode_duration_us":23957,"operation_id":"0x2958","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":9089134},{"operation_name":"get","last_dispatch_duration_us":8560381,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8560381,"total_server_duration_us":56,"operation_id":"0x29a8","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8561343}],"total_count":21}} INFO [2022-03-13 17:09:37,110] jepsen worker 9 - jepsen.util 909 :info :write [3 11] :RequestTimeoutException INFO [2022-03-13 17:09:37,110] jepsen worker 52 - jepsen.util 502 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,111] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,111] jepsen worker 44 - jepsen.util 494 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,112] jepsen worker 9 - jepsen.util 999 :invoke :write [3 25] INFO [2022-03-13 17:09:37,112] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:37,113] jepsen worker 52 - jepsen.util 502 :invoke :write [17 43] INFO [2022-03-13 17:09:37,115] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:37,120] jepsen worker 73 - jepsen.util 1063 :info :write [24 15] :RequestTimeoutException INFO [2022-03-13 17:09:37,121] jepsen worker 73 - jepsen.util 1153 :invoke :write [24 20] INFO [2022-03-13 17:09:37,128] jepsen worker 69 - jepsen.util 609 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,129] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] WARN [2022-03-13 17:09:37,193] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18471952,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":18471952,"total_server_duration_us":38,"operation_id":"0x2911","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":18473833},{"operation_name":"get","last_dispatch_duration_us":17276479,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17276479,"total_server_duration_us":62,"operation_id":"0x2991","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":17279262},{"operation_name":"get","last_dispatch_duration_us":10313235,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10313235,"total_server_duration_us":49,"operation_id":"0x28ac","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":10319064},{"operation_name":"upsert","last_dispatch_duration_us":10123170,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":10123170,"total_server_duration_us":22720,"encode_duration_us":19053,"operation_id":"0x29e2","timeout_ms":10000,"last_server_duration_us":22720,"total_duration_us":10008529},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18114,"operation_id":"0x2b77","timeout_ms":10000,"total_duration_us":10001562},{"operation_name":"get","last_dispatch_duration_us":499866,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":499866,"total_server_duration_us":11,"operation_id":"0x2bf7","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":4642404},{"operation_name":"upsert","last_dispatch_duration_us":1840434,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3332978,"total_server_duration_us":158908,"encode_duration_us":21417,"operation_id":"0x2da3","timeout_ms":10000,"last_server_duration_us":72726,"total_duration_us":3516704},{"operation_name":"upsert","last_dispatch_duration_us":1331811,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1331811,"total_server_duration_us":152179,"encode_duration_us":21012,"operation_id":"0x2b69","timeout_ms":10000,"last_server_duration_us":152179,"total_duration_us":3439785},{"operation_name":"upsert","last_dispatch_duration_us":1973229,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3205650,"total_server_duration_us":98433,"encode_duration_us":26719,"operation_id":"0x2d95","timeout_ms":10000,"last_server_duration_us":58376,"total_duration_us":3314808},{"operation_name":"get","last_dispatch_duration_us":900090,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":900090,"total_server_duration_us":23,"operation_id":"0x2e3f","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":3006613}],"total_count":55}} WARN [2022-03-13 17:09:37,195] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12886032,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12886032,"total_server_duration_us":32,"operation_id":"0x2ab3","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":12886340},{"operation_name":"get","last_dispatch_duration_us":12552653,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12552653,"total_server_duration_us":9058,"operation_id":"0x2aeb","timeout_ms":30000,"last_server_duration_us":9058,"total_duration_us":12554928},{"operation_name":"get","last_dispatch_duration_us":11796169,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11796169,"total_server_duration_us":24325,"operation_id":"0x2b29","timeout_ms":30000,"last_server_duration_us":24325,"total_duration_us":11798791},{"operation_name":"upsert","last_dispatch_duration_us":37984,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":1051490,"total_server_duration_us":657653,"encode_duration_us":17906,"operation_id":"0x2a3d","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10009572},{"operation_name":"upsert","last_dispatch_duration_us":31719,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9040384,"total_server_duration_us":234536,"encode_duration_us":19585,"operation_id":"0x29b3","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10006000},{"operation_name":"upsert","last_dispatch_duration_us":31431,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9381815,"total_server_duration_us":219877,"encode_duration_us":20316,"operation_id":"0x2969","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10004479},{"operation_name":"upsert","last_dispatch_duration_us":31481,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9237903,"total_server_duration_us":229457,"encode_duration_us":17340,"operation_id":"0x298f","timeout_ms":10000,"last_server_duration_us":18646,"total_duration_us":10002872},{"operation_name":"upsert","last_dispatch_duration_us":70215,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9171849,"total_server_duration_us":99812,"encode_duration_us":17700,"operation_id":"0x2c58","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10002331},{"operation_name":"upsert","last_dispatch_duration_us":9137767,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9137767,"total_server_duration_us":9035244,"encode_duration_us":17044,"operation_id":"0x2990","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":9144080},{"operation_name":"upsert","last_dispatch_duration_us":938571,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1995772,"total_server_duration_us":108915,"encode_duration_us":17445,"operation_id":"0x2df3","timeout_ms":10000,"last_server_duration_us":55061,"total_duration_us":3088475}],"total_count":59}} INFO [2022-03-13 17:09:37,196] jepsen worker 51 - jepsen.util 411 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,197] jepsen worker 51 - jepsen.util 411 :invoke :write [17 6] WARN [2022-03-13 17:09:37,204] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18034275,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18149867,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9059996,"encode_duration_us":19838,"operation_id":"0x2970","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10002354},{"operation_name":"upsert","last_dispatch_duration_us":17575862,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17633614,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9059780,"encode_duration_us":42229,"operation_id":"0x253f","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10003989},{"operation_name":"upsert","last_dispatch_duration_us":17955189,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17955189,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9037366,"encode_duration_us":17731,"operation_id":"0x2984","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10003495},{"operation_name":"upsert","last_dispatch_duration_us":17809981,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26961497,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9250799,"encode_duration_us":19684,"operation_id":"0x258f","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10010287},{"operation_name":"upsert","last_dispatch_duration_us":9633621,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10416475,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9394069,"encode_duration_us":18814,"operation_id":"0x2b04","timeout_ms":10000,"last_server_duration_us":9020392,"total_duration_us":10006926}],"total_count":5}} INFO [2022-03-13 17:09:37,228] jepsen worker 43 - jepsen.util 403 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,228] jepsen worker 75 - jepsen.util 345 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,229] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:37,229] jepsen worker 43 - jepsen.util 403 :invoke :write [14 22] INFO [2022-03-13 17:09:37,230] jepsen worker 37 - jepsen.util 1027 :info :write [12 0] :RequestTimeoutException INFO [2022-03-13 17:09:37,231] jepsen worker 37 - jepsen.util 1117 :invoke :read [12 nil] INFO [2022-03-13 17:09:37,256] jepsen worker 37 - jepsen.util 1117 :ok :read [12 2] INFO [2022-03-13 17:09:37,257] jepsen worker 37 - jepsen.util 1117 :invoke :read [12 nil] INFO [2022-03-13 17:09:37,268] jepsen worker 63 - jepsen.util 963 :info :write [21 41] :RequestTimeoutException INFO [2022-03-13 17:09:37,269] jepsen worker 63 - jepsen.util 1053 :invoke :write [21 31] INFO [2022-03-13 17:09:37,284] jepsen worker 37 - jepsen.util 1117 :ok :read [12 2] INFO [2022-03-13 17:09:37,284] jepsen worker 37 - jepsen.util 1117 :invoke :read [12 nil] WARN [2022-03-13 17:09:37,293] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17115340,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":17671629,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9260467,"encode_duration_us":20118,"operation_id":"0x2690","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10000932},{"operation_name":"upsert","last_dispatch_duration_us":10827539,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15717773,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9424148,"encode_duration_us":17376,"operation_id":"0x250a","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10010211},{"operation_name":"upsert","last_dispatch_duration_us":19349495,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":23538467,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9443914,"encode_duration_us":22467,"operation_id":"0x2586","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10004196}],"total_count":3}} WARN [2022-03-13 17:09:37,295] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13335814,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13335814,"total_server_duration_us":43,"operation_id":"0x29a0","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":13337223},{"operation_name":"upsert","last_dispatch_duration_us":115200,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9907437,"total_server_duration_us":177553,"encode_duration_us":17064,"operation_id":"0x2b28","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10007277},{"operation_name":"upsert","last_dispatch_duration_us":8731420,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":13054578,"total_server_duration_us":269231,"encode_duration_us":19259,"operation_id":"0x2916","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10004905},{"operation_name":"upsert","last_dispatch_duration_us":12646831,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12646831,"total_server_duration_us":21487,"encode_duration_us":18744,"operation_id":"0x29ef","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10003159},{"operation_name":"upsert","last_dispatch_duration_us":9202057,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9202057,"total_server_duration_us":9057542,"encode_duration_us":18781,"operation_id":"0x2aac","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":9207672},{"operation_name":"get","last_dispatch_duration_us":8763594,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8763594,"total_server_duration_us":43,"operation_id":"0x2ae1","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8763840},{"operation_name":"get","last_dispatch_duration_us":8263999,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8263999,"total_server_duration_us":6927,"operation_id":"0x2b23","timeout_ms":30000,"last_server_duration_us":6927,"total_duration_us":8265918},{"operation_name":"upsert","last_dispatch_duration_us":746204,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":7307478,"total_server_duration_us":315729,"encode_duration_us":17378,"operation_id":"0x2b86","timeout_ms":10000,"last_server_duration_us":217316,"total_duration_us":7323116},{"operation_name":"get","last_dispatch_duration_us":7179240,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7179240,"total_server_duration_us":23,"operation_id":"0x2b83","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":7179537},{"operation_name":"upsert","last_dispatch_duration_us":6863558,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6863558,"total_server_duration_us":66037,"encode_duration_us":37612,"operation_id":"0x2b0b","timeout_ms":10000,"last_server_duration_us":66037,"total_duration_us":6869542}],"total_count":55}} INFO [2022-03-13 17:09:37,307] jepsen worker 37 - jepsen.util 1117 :ok :read [12 2] INFO [2022-03-13 17:09:37,308] jepsen worker 37 - jepsen.util 1117 :invoke :read [12 nil] INFO [2022-03-13 17:09:37,309] jepsen worker 13 - jepsen.util 463 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,310] jepsen worker 13 - jepsen.util 463 :invoke :write [4 47] INFO [2022-03-13 17:09:37,331] jepsen worker 37 - jepsen.util 1117 :ok :read [12 2] INFO [2022-03-13 17:09:37,332] jepsen worker 37 - jepsen.util 1117 :invoke :write [12 22] INFO [2022-03-13 17:09:37,341] jepsen worker 71 - jepsen.util 431 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,341] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:37,341] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,342] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,341] jepsen worker 80 - jepsen.util 620 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,343] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:37,343] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:37,344] jepsen worker 24 - jepsen.util 474 :invoke :write [8 46] INFO [2022-03-13 17:09:37,351] jepsen worker 8 - jepsen.util 638 :invoke :write [2 9] INFO [2022-03-13 17:09:37,352] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:37,375] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,375] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,376] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:09:37,376] jepsen worker 12 - jepsen.util 642 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,376] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,376] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,376] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,377] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:37,378] jepsen worker 12 - jepsen.util 642 :invoke :write [4 16] INFO [2022-03-13 17:09:37,379] jepsen worker 53 - jepsen.util 503 :invoke :write [17 7] INFO [2022-03-13 17:09:37,379] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:37,380] jepsen worker 60 - jepsen.util 600 :invoke :write [20 32] INFO [2022-03-13 17:09:37,393] jepsen worker 77 - jepsen.util 617 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,394] jepsen worker 77 - jepsen.util 617 :invoke :write [25 20] WARN [2022-03-13 17:09:37,395] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16419622,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17523034,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9450362,"encode_duration_us":22924,"operation_id":"0x2729","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002549}],"total_count":1}} INFO [2022-03-13 17:09:37,455] jepsen worker 51 - jepsen.util 411 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,456] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:37,473] jepsen worker 79 - jepsen.util 799 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,474] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,474] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:37,474] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,474] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:37,474] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,474] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,475] jepsen worker 70 - jepsen.util 700 :invoke :write [23 34] INFO [2022-03-13 17:09:37,476] jepsen worker 22 - jepsen.util 472 :invoke :read [7 nil] INFO [2022-03-13 17:09:37,477] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:37,478] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:37,480] jepsen worker 34 - jepsen.util 574 :invoke :read [11 nil] INFO [2022-03-13 17:09:37,480] jepsen worker 79 - jepsen.util 799 :invoke :write [26 25] INFO [2022-03-13 17:09:37,481] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:37,497] jepsen worker 43 - jepsen.util 403 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,497] jepsen worker 61 - jepsen.util 421 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,498] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:37,498] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,499] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:37,500] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:37,500] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:37,501] jepsen worker 76 - jepsen.util 526 :invoke :write [25 0] INFO [2022-03-13 17:09:37,520] jepsen worker 13 - jepsen.util 463 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,521] jepsen worker 13 - jepsen.util 463 :invoke :write [4 14] INFO [2022-03-13 17:09:37,549] jepsen worker 22 - jepsen.util 472 :ok :read [7 :nil] INFO [2022-03-13 17:09:37,549] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,549] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,549] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:37,549] jepsen worker 25 - jepsen.util 385 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,549] jepsen worker 34 - jepsen.util 574 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,551] jepsen worker 6 - jepsen.util 546 :invoke :write [2 34] INFO [2022-03-13 17:09:37,552] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:37,553] jepsen worker 22 - jepsen.util 472 :invoke :write [7 33] INFO [2022-03-13 17:09:37,554] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:37,555] jepsen worker 34 - jepsen.util 574 :invoke :write [11 6] INFO [2022-03-13 17:09:37,555] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:37,584] jepsen worker 16 - jepsen.util 1006 :info :write [5 6] :RequestTimeoutException INFO [2022-03-13 17:09:37,584] jepsen worker 16 - jepsen.util 1096 :invoke :read [5 nil] INFO [2022-03-13 17:09:37,613] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,614] jepsen worker 77 - jepsen.util 617 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,614] jepsen worker 51 - jepsen.util 411 :invoke :write [17 48] INFO [2022-03-13 17:09:37,615] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:37,623] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,623] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:37,658] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,658] jepsen worker 14 - jepsen.util 194 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,659] jepsen worker 14 - jepsen.util 194 :invoke :write [4 12] INFO [2022-03-13 17:09:37,659] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:09:37,666] jepsen worker 0 - jepsen.util 1080 :info :write [0 24] :RequestTimeoutException INFO [2022-03-13 17:09:37,666] jepsen worker 0 - jepsen.util 1170 :invoke :read [0 nil] INFO [2022-03-13 17:09:37,702] jepsen worker 21 - jepsen.util 651 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,702] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:37,711] jepsen worker 13 - jepsen.util 463 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,711] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,712] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:37,712] jepsen worker 13 - jepsen.util 463 :invoke :write [4 35] INFO [2022-03-13 17:09:37,721] jepsen worker 70 - jepsen.util 700 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,722] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:37,769] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,769] jepsen worker 7 - jepsen.util 457 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,770] jepsen worker 78 - jepsen.util 438 :invoke :write [26 20] INFO [2022-03-13 17:09:37,770] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:37,797] jepsen worker 51 - jepsen.util 411 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,797] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,797] jepsen worker 51 - jepsen.util 411 :invoke :read [17 nil] INFO [2022-03-13 17:09:37,797] jepsen worker 77 - jepsen.util 617 :invoke :read [25 nil] INFO [2022-03-13 17:09:37,812] jepsen worker 79 - jepsen.util 799 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,812] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,812] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:37,812] jepsen worker 79 - jepsen.util 799 :invoke :write [26 43] INFO [2022-03-13 17:09:37,813] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:37,814] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:37,814] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2573099415204678}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.2}} INFO [2022-03-13 17:09:37,814] jepsen worker nemesis - couchbase.util Rebalance check count: 90 INFO [2022-03-13 17:09:37,814] jepsen worker nemesis - couchbase.util Rebalance stuck count: 49 INFO [2022-03-13 17:09:37,816] jepsen worker 66 - jepsen.util 1146 :info :write [22 36] :RequestTimeoutException INFO [2022-03-13 17:09:37,816] jepsen worker 66 - jepsen.util 1236 :invoke :read [22 nil] INFO [2022-03-13 17:09:37,863] jepsen worker 13 - jepsen.util 463 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,863] jepsen worker 13 - jepsen.util 463 :invoke :read [4 nil] INFO [2022-03-13 17:09:37,866] jepsen worker 51 - jepsen.util 411 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,866] jepsen worker 77 - jepsen.util 617 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,867] jepsen worker 51 - jepsen.util 411 :invoke :write [17 16] INFO [2022-03-13 17:09:37,867] jepsen worker 77 - jepsen.util 617 :invoke :write [25 42] INFO [2022-03-13 17:09:37,868] jepsen worker 13 - jepsen.util 463 :ok :read [4 :nil] INFO [2022-03-13 17:09:37,868] jepsen worker 13 - jepsen.util 463 :invoke :write [4 46] INFO [2022-03-13 17:09:37,901] jepsen worker 42 - jepsen.util 402 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,901] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,901] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,901] jepsen worker 42 - jepsen.util 402 :invoke :write [14 30] INFO [2022-03-13 17:09:37,902] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,902] jepsen worker 75 - jepsen.util 345 :invoke :write [25 22] INFO [2022-03-13 17:09:37,904] jepsen worker 62 - jepsen.util 422 :invoke :write [20 19] INFO [2022-03-13 17:09:37,910] jepsen worker 35 - jepsen.util 665 :invoke :write [11 29] INFO [2022-03-13 17:09:37,912] jepsen worker 52 - jepsen.util 502 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,912] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:37,915] jepsen worker 60 - jepsen.util 600 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:37,916] jepsen worker 60 - jepsen.util 600 :invoke :write [20 32] INFO [2022-03-13 17:09:37,920] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:37,921] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:38,025] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:38,026] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:09:38,036] jepsen worker 64 - jepsen.util 784 :info :write [21 16] :RequestTimeoutException INFO [2022-03-13 17:09:38,037] jepsen worker 64 - jepsen.util 874 :invoke :write [21 33] INFO [2022-03-13 17:09:38,100] jepsen worker 85 - jepsen.util 1165 :info :write [28 20] :RequestTimeoutException INFO [2022-03-13 17:09:38,101] jepsen worker 85 - jepsen.util 1255 :invoke :read [28 nil] INFO [2022-03-13 17:09:38,127] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:38,128] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:38,156] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:38,156] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:38,157] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:38,157] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:38,228] jepsen worker 16 - jepsen.util 1096 :ok :read [5 43] INFO [2022-03-13 17:09:38,228] jepsen worker 16 - jepsen.util 1096 :invoke :read [5 nil] INFO [2022-03-13 17:09:38,256] jepsen worker 16 - jepsen.util 1096 :ok :read [5 43] INFO [2022-03-13 17:09:38,256] jepsen worker 16 - jepsen.util 1096 :invoke :write [5 32] INFO [2022-03-13 17:09:38,430] jepsen worker 28 - jepsen.util 1018 :info :write [9 38] :RequestTimeoutException INFO [2022-03-13 17:09:38,431] jepsen worker 28 - jepsen.util 1108 :invoke :read [9 nil] INFO [2022-03-13 17:09:38,568] jepsen worker 40 - jepsen.util 940 :info :write [13 19] :RequestTimeoutException INFO [2022-03-13 17:09:38,569] jepsen worker 40 - jepsen.util 1030 :invoke :write [13 46] INFO [2022-03-13 17:09:38,570] jepsen worker 2 - jepsen.util 1172 :info :write [0 20] :RequestTimeoutException INFO [2022-03-13 17:09:38,570] jepsen worker 2 - jepsen.util 1262 :invoke :read [0 nil] INFO [2022-03-13 17:09:38,583] jepsen worker 38 - jepsen.util 1208 :info :write [12 21] :SyncWriteAmbiguous INFO [2022-03-13 17:09:38,583] jepsen worker 38 - jepsen.util 1298 :invoke :read [12 nil] INFO [2022-03-13 17:09:38,610] jepsen worker 38 - jepsen.util 1298 :ok :read [12 2] INFO [2022-03-13 17:09:38,610] jepsen worker 38 - jepsen.util 1298 :invoke :read [12 nil] INFO [2022-03-13 17:09:38,634] jepsen worker 38 - jepsen.util 1298 :ok :read [12 2] INFO [2022-03-13 17:09:38,634] jepsen worker 38 - jepsen.util 1298 :invoke :write [12 1] INFO [2022-03-13 17:09:38,714] jepsen worker 5 - jepsen.util 1085 :info :write [1 8] :SyncWriteAmbiguous INFO [2022-03-13 17:09:38,715] jepsen worker 5 - jepsen.util 1175 :invoke :write [1 44] INFO [2022-03-13 17:09:38,737] jepsen worker 36 - jepsen.util 1206 :ok :read [12 2] INFO [2022-03-13 17:09:38,737] jepsen worker 36 - jepsen.util 1206 :invoke :write [12 4] INFO [2022-03-13 17:09:38,820] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2573099415204678}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.2}} INFO [2022-03-13 17:09:38,820] jepsen worker nemesis - couchbase.util Rebalance check count: 91 INFO [2022-03-13 17:09:38,820] jepsen worker nemesis - couchbase.util Rebalance stuck count: 50 INFO [2022-03-13 17:09:38,875] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:38,875] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:38,875] jepsen worker 21 - jepsen.util 651 :invoke :write [7 18] INFO [2022-03-13 17:09:38,876] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:38,893] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:38,893] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:38,894] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:38,895] jepsen worker 26 - jepsen.util 476 :invoke :write [8 27] INFO [2022-03-13 17:09:38,979] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:38,979] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:39,080] jepsen worker 70 - jepsen.util 700 :ok :read [23 30] INFO [2022-03-13 17:09:39,080] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,112] jepsen worker 71 - jepsen.util 431 :ok :read [23 30] INFO [2022-03-13 17:09:39,114] jepsen worker 71 - jepsen.util 431 :invoke :write [23 20] INFO [2022-03-13 17:09:39,118] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:39,119] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:39,200] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,200] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:39,200] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,202] jepsen worker 25 - jepsen.util 385 :invoke :write [8 18] INFO [2022-03-13 17:09:39,202] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:39,203] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:39,240] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:39,241] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:39,346] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,346] jepsen worker 69 - jepsen.util 609 :invoke :write [23 47] INFO [2022-03-13 17:09:39,406] jepsen worker 48 - jepsen.util 948 :info :write [16 23] :RequestTimeoutException INFO [2022-03-13 17:09:39,407] jepsen worker 48 - jepsen.util 1038 :invoke :write [16 23] INFO [2022-03-13 17:09:39,471] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,472] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:39,508] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:39,508] jepsen worker 43 - jepsen.util 403 :invoke :write [14 2] INFO [2022-03-13 17:09:39,562] jepsen worker 69 - jepsen.util 609 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,563] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,645] jepsen worker 60 - jepsen.util 600 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,645] jepsen worker 75 - jepsen.util 345 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,645] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:39,645] jepsen worker 75 - jepsen.util 345 :invoke :write [25 33] INFO [2022-03-13 17:09:39,655] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,655] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,655] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:39,656] jepsen worker 69 - jepsen.util 609 :invoke :write [23 21] INFO [2022-03-13 17:09:39,656] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,657] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:39,658] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,659] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:39,660] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,660] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:09:39,724] jepsen worker 6 - jepsen.util 546 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,724] jepsen worker 6 - jepsen.util 546 :invoke :write [2 45] INFO [2022-03-13 17:09:39,796] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,796] jepsen worker 12 - jepsen.util 642 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,797] jepsen worker 12 - jepsen.util 642 :invoke :write [4 3] INFO [2022-03-13 17:09:39,798] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:39,800] jepsen worker 69 - jepsen.util 609 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,800] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,827] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1876832844574781}, :ns_1@172.23.105.187 {:progress 0.2573099415204678}, :ns_1@172.23.107.227 {:progress 0.1876832844574781}, :ns_1@172.23.107.228 {:progress 0.2}} INFO [2022-03-13 17:09:39,827] jepsen worker nemesis - couchbase.util Rebalance check count: 92 INFO [2022-03-13 17:09:39,827] jepsen worker nemesis - couchbase.util Rebalance stuck count: 51 INFO [2022-03-13 17:09:39,868] jepsen worker 24 - jepsen.util 474 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,868] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,868] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,869] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:39,870] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:39,870] jepsen worker 24 - jepsen.util 474 :invoke :write [8 42] INFO [2022-03-13 17:09:39,890] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,890] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,891] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:39,891] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,892] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,892] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,893] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:09:39,893] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,894] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,895] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,896] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,896] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,897] jepsen worker 41 - jepsen.util 1031 :info :write [13 26] :RequestTimeoutException INFO [2022-03-13 17:09:39,898] jepsen worker 41 - jepsen.util 1121 :invoke :read [13 nil] INFO [2022-03-13 17:09:39,900] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,900] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,904] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,905] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,906] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,906] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:39,907] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,908] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:09:39,924] jepsen worker 41 - jepsen.util 1121 :ok :read [13 39] INFO [2022-03-13 17:09:39,925] jepsen worker 78 - jepsen.util 438 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,926] jepsen worker 41 - jepsen.util 1121 :invoke :read [13 nil] INFO [2022-03-13 17:09:39,926] jepsen worker 78 - jepsen.util 438 :invoke :write [26 3] INFO [2022-03-13 17:09:39,955] jepsen worker 41 - jepsen.util 1121 :ok :read [13 39] INFO [2022-03-13 17:09:39,956] jepsen worker 41 - jepsen.util 1121 :invoke :read [13 nil] INFO [2022-03-13 17:09:39,967] jepsen worker 53 - jepsen.util 503 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,967] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:39,967] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,967] jepsen worker 43 - jepsen.util 403 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,967] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,967] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:39,968] jepsen worker 61 - jepsen.util 421 :invoke :write [20 43] INFO [2022-03-13 17:09:39,969] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:39,969] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:39,970] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:39,972] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,972] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:39,972] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:39,973] jepsen worker 60 - jepsen.util 600 :invoke :write [20 29] INFO [2022-03-13 17:09:39,973] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:39,973] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:09:39,975] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:39,979] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:39,980] jepsen worker 41 - jepsen.util 1121 :ok :read [13 39] INFO [2022-03-13 17:09:39,980] jepsen worker 41 - jepsen.util 1121 :invoke :read [13 nil] INFO [2022-03-13 17:09:39,982] jepsen worker 25 - jepsen.util 385 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,982] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:39,984] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:39,986] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:39,988] jepsen worker 7 - jepsen.util 457 :invoke :write [2 9] INFO [2022-03-13 17:09:39,988] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:39,992] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:39,992] jepsen worker 43 - jepsen.util 403 :invoke :write [14 4] INFO [2022-03-13 17:09:40,018] jepsen worker 41 - jepsen.util 1121 :ok :read [13 39] INFO [2022-03-13 17:09:40,018] jepsen worker 41 - jepsen.util 1121 :invoke :read [13 nil] INFO [2022-03-13 17:09:40,022] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,022] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:40,045] jepsen worker 41 - jepsen.util 1121 :ok :read [13 39] INFO [2022-03-13 17:09:40,046] jepsen worker 41 - jepsen.util 1121 :invoke :write [13 21] INFO [2022-03-13 17:09:40,074] jepsen worker 26 - jepsen.util 476 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,075] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:40,087] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,087] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,088] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:40,089] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:40,093] jepsen worker 8 - jepsen.util 638 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,094] jepsen worker 8 - jepsen.util 638 :invoke :write [2 19] INFO [2022-03-13 17:09:40,096] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,096] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,096] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:40,097] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:09:40,098] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,098] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:40,100] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,100] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:09:40,138] jepsen worker 21 - jepsen.util 651 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,139] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,139] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,139] jepsen worker 21 - jepsen.util 651 :invoke :write [7 7] INFO [2022-03-13 17:09:40,140] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:40,141] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:40,331] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,331] jepsen worker 33 - jepsen.util 663 :invoke :write [11 37] INFO [2022-03-13 17:09:40,373] jepsen worker 14 - jepsen.util 194 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,374] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:40,392] jepsen worker 24 - jepsen.util 474 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,396] jepsen worker 24 - jepsen.util 474 :invoke :write [8 16] INFO [2022-03-13 17:09:40,415] jepsen worker 60 - jepsen.util 600 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,415] jepsen worker 60 - jepsen.util 600 :invoke :write [20 14] INFO [2022-03-13 17:09:40,471] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,471] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:40,481] jepsen worker 35 - jepsen.util 665 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,482] jepsen worker 35 - jepsen.util 665 :invoke :write [11 23] INFO [2022-03-13 17:09:40,484] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,485] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:40,513] jepsen worker 31 - jepsen.util 1021 :info :write [10 16] :SyncWriteAmbiguous INFO [2022-03-13 17:09:40,514] jepsen worker 31 - jepsen.util 1111 :invoke :write [10 18] INFO [2022-03-13 17:09:40,539] jepsen worker 43 - jepsen.util 403 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,539] jepsen worker 62 - jepsen.util 422 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,540] jepsen worker 43 - jepsen.util 403 :invoke :write [14 30] INFO [2022-03-13 17:09:40,540] jepsen worker 62 - jepsen.util 422 :invoke :write [20 9] INFO [2022-03-13 17:09:40,573] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,574] jepsen worker 33 - jepsen.util 663 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,574] jepsen worker 33 - jepsen.util 663 :invoke :write [11 19] INFO [2022-03-13 17:09:40,575] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:40,580] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,581] jepsen worker 69 - jepsen.util 609 :invoke :write [23 0] INFO [2022-03-13 17:09:40,600] jepsen worker 60 - jepsen.util 600 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,600] jepsen worker 12 - jepsen.util 642 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,600] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:40,600] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,600] jepsen worker 12 - jepsen.util 642 :invoke :write [4 7] INFO [2022-03-13 17:09:40,601] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:40,601] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:40,602] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:40,636] jepsen worker 18 - jepsen.util 918 :ok :read [6 16] INFO [2022-03-13 17:09:40,636] jepsen worker 18 - jepsen.util 918 :invoke :write [6 47] INFO [2022-03-13 17:09:40,740] jepsen worker 3 - jepsen.util 903 :info :write [1 47] :RequestTimeoutException INFO [2022-03-13 17:09:40,741] jepsen worker 3 - jepsen.util 993 :invoke :write [1 39] INFO [2022-03-13 17:09:40,832] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1935483870967742}, :ns_1@172.23.105.187 {:progress 0.2631578947368421}, :ns_1@172.23.107.227 {:progress 0.1935483870967742}, :ns_1@172.23.107.228 {:progress 0.2}} INFO [2022-03-13 17:09:40,832] jepsen worker nemesis - couchbase.util Rebalance check count: 93 INFO [2022-03-13 17:09:40,832] jepsen worker nemesis - couchbase.util Rebalance stuck count: 51 INFO [2022-03-13 17:09:40,948] jepsen worker 61 - jepsen.util 421 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,949] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:40,949] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:40,949] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:40,969] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:40,969] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,969] jepsen worker 43 - jepsen.util 403 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,970] jepsen worker 23 - jepsen.util 563 :invoke :write [7 47] INFO [2022-03-13 17:09:40,970] jepsen worker 60 - jepsen.util 600 :invoke :write [20 44] INFO [2022-03-13 17:09:40,974] jepsen worker 43 - jepsen.util 403 :invoke :write [14 21] INFO [2022-03-13 17:09:40,983] jepsen worker 33 - jepsen.util 663 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:40,984] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:09:41,009] jepsen worker 76 - jepsen.util 526 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,010] jepsen worker 76 - jepsen.util 526 :invoke :write [25 17] INFO [2022-03-13 17:09:41,014] jepsen worker 14 - jepsen.util 194 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,015] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:41,050] jepsen worker 35 - jepsen.util 665 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,051] jepsen worker 35 - jepsen.util 665 :invoke :write [11 22] INFO [2022-03-13 17:09:41,091] jepsen worker 23 - jepsen.util 563 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,092] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:41,109] jepsen worker 69 - jepsen.util 609 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,109] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:41,155] jepsen worker 42 - jepsen.util 402 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,155] jepsen worker 42 - jepsen.util 402 :invoke :write [14 41] INFO [2022-03-13 17:09:41,203] jepsen worker 60 - jepsen.util 600 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,203] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:41,213] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,213] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,214] jepsen worker 69 - jepsen.util 609 :invoke :write [23 18] INFO [2022-03-13 17:09:41,215] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:09:41,294] jepsen worker 62 - jepsen.util 422 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,294] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,295] jepsen worker 62 - jepsen.util 422 :invoke :write [20 1] INFO [2022-03-13 17:09:41,295] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:41,302] jepsen worker 69 - jepsen.util 609 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,303] jepsen worker 69 - jepsen.util 609 :invoke :write [23 9] INFO [2022-03-13 17:09:41,345] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:41,345] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,345] jepsen worker 43 - jepsen.util 403 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,345] jepsen worker 43 - jepsen.util 403 :invoke :write [14 12] INFO [2022-03-13 17:09:41,346] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:41,346] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:41,351] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:41,351] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,351] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:41,352] jepsen worker 60 - jepsen.util 600 :invoke :write [20 46] INFO [2022-03-13 17:09:41,352] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:41,353] jepsen worker 23 - jepsen.util 563 :invoke :write [7 12] INFO [2022-03-13 17:09:41,364] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,365] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:09:41,416] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,416] jepsen worker 12 - jepsen.util 642 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,417] jepsen worker 12 - jepsen.util 642 :invoke :write [4 13] INFO [2022-03-13 17:09:41,417] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:41,417] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:41,417] jepsen worker 14 - jepsen.util 194 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,417] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:41,418] jepsen worker 14 - jepsen.util 194 :invoke :write [4 48] INFO [2022-03-13 17:09:41,452] jepsen worker 69 - jepsen.util 609 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,453] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:09:41,483] jepsen worker 43 - jepsen.util 403 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,484] jepsen worker 43 - jepsen.util 403 :invoke :write [14 46] INFO [2022-03-13 17:09:41,494] jepsen worker 65 - jepsen.util 1055 :info :write [21 25] :RequestTimeoutException INFO [2022-03-13 17:09:41,495] jepsen worker 65 - jepsen.util 1145 :invoke :write [21 6] INFO [2022-03-13 17:09:41,505] jepsen worker 55 - jepsen.util 1045 :info :write [18 33] :RequestTimeoutException INFO [2022-03-13 17:09:41,505] jepsen worker 55 - jepsen.util 1135 :invoke :read [18 nil] INFO [2022-03-13 17:09:41,531] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,531] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:41,550] jepsen worker 76 - jepsen.util 526 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,550] jepsen worker 76 - jepsen.util 526 :invoke :write [25 29] INFO [2022-03-13 17:09:41,640] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,640] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,641] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:09:41,641] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:09:41,657] jepsen worker 60 - jepsen.util 600 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,658] jepsen worker 60 - jepsen.util 600 :invoke :write [20 3] INFO [2022-03-13 17:09:41,726] jepsen worker 4 - jepsen.util 1084 :info :write [1 3] :RequestTimeoutException INFO [2022-03-13 17:09:41,726] jepsen worker 4 - jepsen.util 1174 :invoke :write [1 27] INFO [2022-03-13 17:09:41,757] jepsen worker 35 - jepsen.util 665 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,758] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:41,793] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,794] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:41,816] jepsen worker 23 - jepsen.util 563 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:41,817] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:41,838] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1935483870967742}, :ns_1@172.23.105.187 {:progress 0.2631578947368421}, :ns_1@172.23.107.227 {:progress 0.1935483870967742}, :ns_1@172.23.107.228 {:progress 0.2}} INFO [2022-03-13 17:09:41,838] jepsen worker nemesis - couchbase.util Rebalance check count: 94 INFO [2022-03-13 17:09:41,838] jepsen worker nemesis - couchbase.util Rebalance stuck count: 52 INFO [2022-03-13 17:09:42,228] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,228] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,228] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:42,229] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:42,233] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,233] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,234] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:09:42,234] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:42,236] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,237] jepsen worker 33 - jepsen.util 663 :invoke :write [11 44] INFO [2022-03-13 17:09:42,305] jepsen worker 42 - jepsen.util 402 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,305] jepsen worker 78 - jepsen.util 438 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,306] jepsen worker 42 - jepsen.util 402 :invoke :write [14 14] INFO [2022-03-13 17:09:42,306] jepsen worker 78 - jepsen.util 438 :invoke :write [26 11] INFO [2022-03-13 17:09:42,337] jepsen worker 43 - jepsen.util 403 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,337] jepsen worker 43 - jepsen.util 403 :invoke :write [14 19] INFO [2022-03-13 17:09:42,356] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,356] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:09:42,366] jepsen worker 75 - jepsen.util 345 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,367] jepsen worker 75 - jepsen.util 345 :invoke :write [25 45] INFO [2022-03-13 17:09:42,430] jepsen worker 33 - jepsen.util 663 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,431] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:09:42,458] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:42,458] jepsen worker 60 - jepsen.util 600 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,459] jepsen worker 60 - jepsen.util 600 :invoke :write [20 45] INFO [2022-03-13 17:09:42,459] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:42,481] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,481] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:42,529] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,529] jepsen worker 62 - jepsen.util 422 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,530] jepsen worker 61 - jepsen.util 421 :invoke :write [20 24] INFO [2022-03-13 17:09:42,530] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:42,531] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:42,531] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:42,550] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,550] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,550] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:09:42,551] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:09:42,594] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:42,594] jepsen worker 43 - jepsen.util 403 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,595] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:42,595] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:42,631] jepsen worker 74 - jepsen.util 1244 :info :write [24 8] :RequestTimeoutException INFO [2022-03-13 17:09:42,632] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,632] jepsen worker 74 - jepsen.util 1334 :invoke :read [24 nil] INFO [2022-03-13 17:09:42,633] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:42,644] jepsen worker 15 - jepsen.util 1095 :info :write [5 3] :RequestTimeoutException INFO [2022-03-13 17:09:42,644] jepsen worker 68 - jepsen.util 968 :info :write [22 41] :RequestTimeoutException INFO [2022-03-13 17:09:42,645] jepsen worker 68 - jepsen.util 1058 :invoke :write [22 1] INFO [2022-03-13 17:09:42,645] jepsen worker 15 - jepsen.util 1185 :invoke :read [5 nil] INFO [2022-03-13 17:09:42,651] jepsen worker 14 - jepsen.util 194 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,652] jepsen worker 14 - jepsen.util 194 :invoke :write [4 7] INFO [2022-03-13 17:09:42,690] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,690] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,691] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:09:42,691] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:42,696] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,697] jepsen worker 33 - jepsen.util 663 :invoke :write [11 44] INFO [2022-03-13 17:09:42,703] jepsen worker 60 - jepsen.util 600 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,706] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:42,706] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:42,707] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:42,708] jepsen worker 43 - jepsen.util 403 :invoke :write [14 33] INFO [2022-03-13 17:09:42,709] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:42,734] jepsen worker 12 - jepsen.util 642 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,735] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,735] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,735] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:42,735] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,736] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:42,736] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:42,737] jepsen worker 23 - jepsen.util 563 :invoke :write [7 32] INFO [2022-03-13 17:09:42,737] jepsen worker 53 - jepsen.util 503 :invoke :write [17 31] INFO [2022-03-13 17:09:42,737] jepsen worker 60 - jepsen.util 600 :invoke :write [20 0] INFO [2022-03-13 17:09:42,800] jepsen worker 76 - jepsen.util 526 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,801] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:42,803] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,804] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:42,845] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.1935483870967742}, :ns_1@172.23.105.187 {:progress 0.2631578947368421}, :ns_1@172.23.107.227 {:progress 0.1935483870967742}, :ns_1@172.23.107.228 {:progress 0.2}} INFO [2022-03-13 17:09:42,845] jepsen worker nemesis - couchbase.util Rebalance check count: 95 INFO [2022-03-13 17:09:42,845] jepsen worker nemesis - couchbase.util Rebalance stuck count: 53 INFO [2022-03-13 17:09:42,852] jepsen worker 43 - jepsen.util 403 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,853] jepsen worker 43 - jepsen.util 403 :invoke :write [14 30] INFO [2022-03-13 17:09:42,870] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,871] jepsen worker 33 - jepsen.util 663 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,871] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:42,871] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:42,872] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,873] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,873] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:09:42,878] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:42,881] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,882] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:09:42,913] jepsen worker 20 - jepsen.util 1280 :info :write [6 25] :RequestTimeoutException INFO [2022-03-13 17:09:42,914] jepsen worker 49 - jepsen.util 949 :info :write [16 5] :RequestTimeoutException INFO [2022-03-13 17:09:42,914] jepsen worker 20 - jepsen.util 1370 :invoke :write [6 23] INFO [2022-03-13 17:09:42,915] jepsen worker 49 - jepsen.util 1039 :invoke :read [16 nil] INFO [2022-03-13 17:09:42,916] jepsen worker 27 - jepsen.util 1197 :info :write [9 9] :RequestTimeoutException INFO [2022-03-13 17:09:42,917] jepsen worker 27 - jepsen.util 1287 :invoke :write [9 31] INFO [2022-03-13 17:09:42,954] jepsen worker 42 - jepsen.util 402 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,955] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:42,956] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:42,956] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:43,007] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,007] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,008] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,009] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:09:43,013] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,014] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:09:43,167] jepsen worker 87 - jepsen.util 1437 :info :write [29 29] :RequestTimeoutException INFO [2022-03-13 17:09:43,167] jepsen worker 87 - jepsen.util 1527 :invoke :read [29 nil] INFO [2022-03-13 17:09:43,172] jepsen worker 60 - jepsen.util 600 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,173] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:43,195] jepsen worker 87 - jepsen.util 1527 :ok :read [29 33] INFO [2022-03-13 17:09:43,195] jepsen worker 87 - jepsen.util 1527 :invoke :write [29 28] INFO [2022-03-13 17:09:43,215] jepsen worker 7 - jepsen.util 457 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,216] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:43,265] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,266] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:43,295] jepsen worker 75 - jepsen.util 345 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,296] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:43,325] jepsen worker 8 - jepsen.util 638 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,325] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:43,326] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,326] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,326] jepsen worker 8 - jepsen.util 638 :invoke :write [2 23] INFO [2022-03-13 17:09:43,327] jepsen worker 25 - jepsen.util 385 :invoke :write [8 36] INFO [2022-03-13 17:09:43,327] jepsen worker 52 - jepsen.util 502 :invoke :write [17 40] INFO [2022-03-13 17:09:43,333] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:43,342] jepsen worker 23 - jepsen.util 563 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,343] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:09:43,355] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,356] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,356] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,356] jepsen worker 69 - jepsen.util 609 :invoke :write [23 1] INFO [2022-03-13 17:09:43,369] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,370] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,371] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,372] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,373] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,374] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:09:43,420] jepsen worker 21 - jepsen.util 651 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,421] jepsen worker 21 - jepsen.util 651 :invoke :write [7 3] INFO [2022-03-13 17:09:43,427] jepsen worker 61 - jepsen.util 421 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,428] jepsen worker 61 - jepsen.util 421 :invoke :write [20 14] INFO [2022-03-13 17:09:43,463] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,463] jepsen worker 43 - jepsen.util 403 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,464] jepsen worker 43 - jepsen.util 403 :invoke :write [14 41] INFO [2022-03-13 17:09:43,464] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:09:43,490] jepsen worker 46 - jepsen.util 1126 :info :write [15 11] :RequestTimeoutException INFO [2022-03-13 17:09:43,491] jepsen worker 46 - jepsen.util 1216 :invoke :read [15 nil] INFO [2022-03-13 17:09:43,492] jepsen worker 69 - jepsen.util 609 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,492] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:43,546] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,547] jepsen worker 14 - jepsen.util 194 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,547] jepsen worker 12 - jepsen.util 642 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,547] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,548] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:43,548] jepsen worker 35 - jepsen.util 665 :invoke :write [11 48] INFO [2022-03-13 17:09:43,549] jepsen worker 14 - jepsen.util 194 :invoke :write [4 36] INFO [2022-03-13 17:09:43,551] jepsen worker 12 - jepsen.util 642 :invoke :write [4 11] INFO [2022-03-13 17:09:43,580] jepsen worker 24 - jepsen.util 474 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,580] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,580] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,581] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:43,582] jepsen worker 7 - jepsen.util 457 :invoke :write [2 39] INFO [2022-03-13 17:09:43,582] jepsen worker 80 - jepsen.util 620 :invoke :write [26 1] INFO [2022-03-13 17:09:43,633] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,633] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,634] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,634] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:09:43,638] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,639] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,643] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,644] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,646] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,647] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:09:43,648] jepsen worker 25 - jepsen.util 385 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,650] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:43,681] jepsen worker 8 - jepsen.util 638 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,682] jepsen worker 8 - jepsen.util 638 :invoke :write [2 5] INFO [2022-03-13 17:09:43,714] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,714] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:43,719] jepsen worker 52 - jepsen.util 502 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,720] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:43,753] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,754] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:09:43,766] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:43,766] jepsen worker 21 - jepsen.util 651 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,767] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:43,768] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:43,810] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:43,810] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,811] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:43,810] jepsen worker 53 - jepsen.util 503 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,810] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,811] jepsen worker 53 - jepsen.util 503 :invoke :write [17 16] INFO [2022-03-13 17:09:43,812] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:43,812] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:43,813] jepsen worker 42 - jepsen.util 402 :invoke :write [14 13] INFO [2022-03-13 17:09:43,817] jepsen worker 26 - jepsen.util 476 :invoke :write [8 26] INFO [2022-03-13 17:09:43,850] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2023460410557185}, :ns_1@172.23.105.187 {:progress 0.2748538011695907}, :ns_1@172.23.107.227 {:progress 0.2023460410557185}, :ns_1@172.23.107.228 {:progress 0.211764705882353}} INFO [2022-03-13 17:09:43,850] jepsen worker nemesis - couchbase.util Rebalance check count: 96 INFO [2022-03-13 17:09:43,850] jepsen worker nemesis - couchbase.util Rebalance stuck count: 53 INFO [2022-03-13 17:09:43,852] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,853] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,859] jepsen worker 80 - jepsen.util 620 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,859] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:43,943] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,944] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,948] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:43,949] jepsen worker 25 - jepsen.util 385 :invoke :write [8 16] INFO [2022-03-13 17:09:43,952] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,953] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:43,983] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:43,984] jepsen worker 76 - jepsen.util 526 :invoke :write [25 16] INFO [2022-03-13 17:09:44,038] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,038] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:44,038] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:44,039] jepsen worker 24 - jepsen.util 474 :invoke :write [8 29] INFO [2022-03-13 17:09:44,039] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,043] jepsen worker 21 - jepsen.util 651 :invoke :write [7 25] INFO [2022-03-13 17:09:44,043] jepsen worker 80 - jepsen.util 620 :invoke :write [26 44] INFO [2022-03-13 17:09:44,044] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:44,052] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,052] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:44,054] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,054] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:44,055] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,056] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:44,078] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:44,078] jepsen worker 23 - jepsen.util 563 :invoke :write [7 11] INFO [2022-03-13 17:09:44,142] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,142] jepsen worker 33 - jepsen.util 663 :invoke :write [11 19] INFO [2022-03-13 17:09:44,154] jepsen worker 89 - jepsen.util 1259 :info :write [29 6] :RequestTimeoutException INFO [2022-03-13 17:09:44,155] jepsen worker 89 - jepsen.util 1349 :invoke :read [29 nil] INFO [2022-03-13 17:09:44,264] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,265] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:44,424] jepsen worker 72 - jepsen.util 972 :info :write [24 13] :SyncWriteAmbiguous INFO [2022-03-13 17:09:44,425] jepsen worker 72 - jepsen.util 1062 :invoke :read [24 nil] INFO [2022-03-13 17:09:44,585] jepsen worker 54 - jepsen.util 954 :ok :read [18 49] INFO [2022-03-13 17:09:44,586] jepsen worker 54 - jepsen.util 954 :invoke :read [18 nil] INFO [2022-03-13 17:09:44,756] jepsen worker 54 - jepsen.util 954 :ok :read [18 49] INFO [2022-03-13 17:09:44,757] jepsen worker 54 - jepsen.util 954 :invoke :read [18 nil] INFO [2022-03-13 17:09:44,792] jepsen worker 28 - jepsen.util 1108 :ok :read [9 37] INFO [2022-03-13 17:09:44,793] jepsen worker 28 - jepsen.util 1108 :invoke :read [9 nil] INFO [2022-03-13 17:09:44,857] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2023460410557185}, :ns_1@172.23.105.187 {:progress 0.2748538011695907}, :ns_1@172.23.107.227 {:progress 0.2023460410557185}, :ns_1@172.23.107.228 {:progress 0.211764705882353}} INFO [2022-03-13 17:09:44,858] jepsen worker nemesis - couchbase.util Rebalance check count: 97 INFO [2022-03-13 17:09:44,858] jepsen worker nemesis - couchbase.util Rebalance stuck count: 54 INFO [2022-03-13 17:09:44,876] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,876] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:44,877] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:44,877] jepsen worker 75 - jepsen.util 345 :invoke :write [25 49] INFO [2022-03-13 17:09:44,877] jepsen worker 74 - jepsen.util 1334 :ok :read [24 24] INFO [2022-03-13 17:09:44,878] jepsen worker 74 - jepsen.util 1334 :invoke :read [24 nil] INFO [2022-03-13 17:09:44,917] jepsen worker 54 - jepsen.util 954 :ok :read [18 49] INFO [2022-03-13 17:09:44,917] jepsen worker 54 - jepsen.util 954 :invoke :write [18 32] INFO [2022-03-13 17:09:44,929] jepsen worker 28 - jepsen.util 1108 :ok :read [9 37] INFO [2022-03-13 17:09:44,930] jepsen worker 28 - jepsen.util 1108 :invoke :read [9 nil] INFO [2022-03-13 17:09:44,995] jepsen worker 74 - jepsen.util 1334 :ok :read [24 24] INFO [2022-03-13 17:09:44,997] jepsen worker 74 - jepsen.util 1334 :invoke :write [24 24] INFO [2022-03-13 17:09:45,032] jepsen worker 56 - jepsen.util 956 :ok :read [18 49] INFO [2022-03-13 17:09:45,033] jepsen worker 56 - jepsen.util 956 :invoke :read [18 nil] INFO [2022-03-13 17:09:45,079] jepsen worker 28 - jepsen.util 1108 :ok :read [9 37] INFO [2022-03-13 17:09:45,080] jepsen worker 28 - jepsen.util 1108 :invoke :read [9 nil] INFO [2022-03-13 17:09:45,123] jepsen worker 45 - jepsen.util 1305 :ok :read [15 48] INFO [2022-03-13 17:09:45,124] jepsen worker 45 - jepsen.util 1305 :invoke :write [15 40] INFO [2022-03-13 17:09:45,133] jepsen worker 56 - jepsen.util 956 :ok :read [18 49] INFO [2022-03-13 17:09:45,133] jepsen worker 56 - jepsen.util 956 :invoke :read [18 nil] INFO [2022-03-13 17:09:45,158] jepsen worker 56 - jepsen.util 956 :ok :read [18 49] INFO [2022-03-13 17:09:45,158] jepsen worker 56 - jepsen.util 956 :invoke :write [18 35] INFO [2022-03-13 17:09:45,236] jepsen worker 19 - jepsen.util 829 :info :write [6 41] :RequestTimeoutException INFO [2022-03-13 17:09:45,237] jepsen worker 19 - jepsen.util 919 :invoke :read [6 nil] INFO [2022-03-13 17:09:45,293] jepsen worker 28 - jepsen.util 1108 :ok :read [9 37] INFO [2022-03-13 17:09:45,293] jepsen worker 28 - jepsen.util 1108 :invoke :write [9 22] INFO [2022-03-13 17:09:45,362] jepsen worker 50 - jepsen.util 1130 :ok :read [16 :nil] INFO [2022-03-13 17:09:45,362] jepsen worker 50 - jepsen.util 1130 :invoke :write [16 13] INFO [2022-03-13 17:09:45,439] jepsen worker 83 - jepsen.util 803 :ok :read [27 4] INFO [2022-03-13 17:09:45,440] jepsen worker 83 - jepsen.util 803 :invoke :read [27 nil] INFO [2022-03-13 17:09:45,447] jepsen worker 26 - jepsen.util 476 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,448] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:45,454] jepsen worker 33 - jepsen.util 663 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,454] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:45,456] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,457] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:45,459] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,459] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:45,461] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,461] jepsen worker 33 - jepsen.util 663 :invoke :write [11 15] INFO [2022-03-13 17:09:45,497] jepsen worker 80 - jepsen.util 620 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,498] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] WARN [2022-03-13 17:09:45,502] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":17606,"operation_id":"0x2c85","timeout_ms":10000,"total_duration_us":10010115},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":21866,"operation_id":"0x2e7e","timeout_ms":10000,"total_duration_us":10009403},{"operation_name":"upsert","last_dispatch_duration_us":10107424,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10107424,"total_server_duration_us":20521,"encode_duration_us":17440,"operation_id":"0x2cf9","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10008272},{"operation_name":"upsert","last_dispatch_duration_us":10178495,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11097594,"total_server_duration_us":248259,"encode_duration_us":21439,"operation_id":"0x2c94","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10007203},{"operation_name":"upsert","last_dispatch_duration_us":9097241,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9097241,"total_server_duration_us":9042674,"encode_duration_us":20275,"operation_id":"0x2c99","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9101131},{"operation_name":"upsert","last_dispatch_duration_us":9073823,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9073823,"total_server_duration_us":9051168,"encode_duration_us":29073,"operation_id":"0x2d96","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9078216},{"operation_name":"get","last_dispatch_duration_us":5344077,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":5344077,"total_server_duration_us":43,"operation_id":"0x2eb6","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5345179},{"operation_name":"upsert","last_dispatch_duration_us":817003,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":817003,"total_server_duration_us":733555,"encode_duration_us":19811,"operation_id":"0x2f77","timeout_ms":10000,"last_server_duration_us":733555,"total_duration_us":822222},{"operation_name":"get","last_dispatch_duration_us":754555,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":754555,"total_server_duration_us":27,"operation_id":"0x2f7c","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":754744},{"operation_name":"get","last_dispatch_duration_us":754145,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":754145,"total_server_duration_us":23,"operation_id":"0x2f7d","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":754347}],"total_count":11}} INFO [2022-03-13 17:09:45,514] jepsen worker 23 - jepsen.util 563 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,515] jepsen worker 23 - jepsen.util 563 :invoke :write [7 23] INFO [2022-03-13 17:09:45,563] jepsen worker 33 - jepsen.util 663 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,564] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] WARN [2022-03-13 17:09:45,602] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17884384,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27595440,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9126262,"encode_duration_us":17369,"operation_id":"0x2709","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10002117},{"operation_name":"upsert","last_dispatch_duration_us":10297148,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16500497,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9315997,"encode_duration_us":22233,"operation_id":"0x2a52","timeout_ms":10000,"last_server_duration_us":9101143,"total_duration_us":10005120},{"operation_name":"upsert","last_dispatch_duration_us":17856350,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17856350,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9055417,"encode_duration_us":21143,"operation_id":"0x2b09","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10003757}],"total_count":3}} INFO [2022-03-13 17:09:45,606] jepsen worker 21 - jepsen.util 651 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,607] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:45,622] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,623] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:09:45,625] jepsen worker 83 - jepsen.util 803 :ok :read [27 4] INFO [2022-03-13 17:09:45,625] jepsen worker 83 - jepsen.util 803 :invoke :read [27 nil] INFO [2022-03-13 17:09:45,646] jepsen worker 35 - jepsen.util 665 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,647] jepsen worker 35 - jepsen.util 665 :invoke :write [11 15] INFO [2022-03-13 17:09:45,656] jepsen worker 30 - jepsen.util 930 :info :write [10 4] :RequestTimeoutException INFO [2022-03-13 17:09:45,657] jepsen worker 30 - jepsen.util 1020 :invoke :read [10 nil] INFO [2022-03-13 17:09:45,735] jepsen worker 25 - jepsen.util 385 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,736] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,736] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:45,736] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] INFO [2022-03-13 17:09:45,739] jepsen worker 17 - jepsen.util 1007 :ok :read [5 43] INFO [2022-03-13 17:09:45,740] jepsen worker 17 - jepsen.util 1007 :invoke :write [5 45] INFO [2022-03-13 17:09:45,756] jepsen worker 43 - jepsen.util 403 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,757] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:45,800] jepsen worker 8 - jepsen.util 638 :fail :write [2 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,801] jepsen worker 8 - jepsen.util 638 :invoke :write [2 22] INFO [2022-03-13 17:09:45,843] jepsen worker 15 - jepsen.util 1185 :ok :read [5 43] INFO [2022-03-13 17:09:45,843] jepsen worker 24 - jepsen.util 474 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,844] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,847] jepsen worker 15 - jepsen.util 1185 :invoke :read [5 nil] INFO [2022-03-13 17:09:45,848] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:45,849] jepsen worker 24 - jepsen.util 474 :invoke :write [8 18] INFO [2022-03-13 17:09:45,853] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,854] jepsen worker 69 - jepsen.util 609 :invoke :write [23 18] INFO [2022-03-13 17:09:45,856] jepsen worker 57 - jepsen.util 1137 :info :write [19 6] :RequestTimeoutException INFO [2022-03-13 17:09:45,857] jepsen worker 57 - jepsen.util 1227 :invoke :write [19 19] INFO [2022-03-13 17:09:45,864] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2023460410557185}, :ns_1@172.23.105.187 {:progress 0.2748538011695907}, :ns_1@172.23.107.227 {:progress 0.2023460410557185}, :ns_1@172.23.107.228 {:progress 0.211764705882353}} INFO [2022-03-13 17:09:45,864] jepsen worker nemesis - couchbase.util Rebalance check count: 98 INFO [2022-03-13 17:09:45,864] jepsen worker nemesis - couchbase.util Rebalance stuck count: 55 INFO [2022-03-13 17:09:45,892] jepsen worker 2 - jepsen.util 1262 :ok :read [0 41] INFO [2022-03-13 17:09:45,893] jepsen worker 2 - jepsen.util 1262 :invoke :write [0 38] INFO [2022-03-13 17:09:45,898] jepsen worker 76 - jepsen.util 526 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,899] jepsen worker 76 - jepsen.util 526 :invoke :write [25 22] INFO [2022-03-13 17:09:45,931] jepsen worker 15 - jepsen.util 1185 :ok :read [5 43] INFO [2022-03-13 17:09:45,931] jepsen worker 15 - jepsen.util 1185 :invoke :write [5 23] INFO [2022-03-13 17:09:45,941] jepsen worker 7 - jepsen.util 457 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,941] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:45,942] jepsen worker 7 - jepsen.util 457 :invoke :write [2 11] INFO [2022-03-13 17:09:45,942] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:45,945] jepsen worker 0 - jepsen.util 1170 :ok :read [0 41] INFO [2022-03-13 17:09:45,947] jepsen worker 0 - jepsen.util 1170 :invoke :write [0 9] INFO [2022-03-13 17:09:45,948] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,948] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:09:45,985] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:45,986] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:46,008] jepsen worker 67 - jepsen.util 1057 :ok :read [22 30] INFO [2022-03-13 17:09:46,009] jepsen worker 67 - jepsen.util 1057 :invoke :read [22 nil] INFO [2022-03-13 17:09:46,025] jepsen worker 69 - jepsen.util 609 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,025] jepsen worker 69 - jepsen.util 609 :invoke :write [23 10] INFO [2022-03-13 17:09:46,088] jepsen worker 59 - jepsen.util 1049 :ok :read [19 2] INFO [2022-03-13 17:09:46,088] jepsen worker 59 - jepsen.util 1049 :invoke :write [19 13] INFO [2022-03-13 17:09:46,117] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,118] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:46,121] jepsen worker 14 - jepsen.util 194 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,122] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:46,136] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:46,136] jepsen worker 78 - jepsen.util 438 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,137] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:46,137] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:46,360] jepsen worker 39 - jepsen.util 1119 :info :write [13 13] :RequestTimeoutException INFO [2022-03-13 17:09:46,361] jepsen worker 39 - jepsen.util 1209 :invoke :read [13 nil] INFO [2022-03-13 17:09:46,408] jepsen worker 1 - jepsen.util 901 :info :write [0 32] :RequestTimeoutException INFO [2022-03-13 17:09:46,408] jepsen worker 1 - jepsen.util 991 :invoke :write [0 5] INFO [2022-03-13 17:09:46,515] jepsen worker 23 - jepsen.util 563 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,515] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,515] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:46,516] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:46,516] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:46,517] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:46,518] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,518] jepsen worker 69 - jepsen.util 609 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,519] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:09:46,519] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:09:46,521] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:46,521] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,521] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:46,522] jepsen worker 23 - jepsen.util 563 :invoke :write [7 11] INFO [2022-03-13 17:09:46,522] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:46,522] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:46,528] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:46,528] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,529] jepsen worker 60 - jepsen.util 600 :invoke :write [20 3] INFO [2022-03-13 17:09:46,529] jepsen worker 43 - jepsen.util 403 :invoke :write [14 38] INFO [2022-03-13 17:09:46,534] jepsen worker 8 - jepsen.util 638 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,534] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,534] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,535] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:46,535] jepsen worker 8 - jepsen.util 638 :invoke :write [2 13] INFO [2022-03-13 17:09:46,538] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:46,566] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:46,566] jepsen worker 24 - jepsen.util 474 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,566] jepsen worker 24 - jepsen.util 474 :invoke :write [8 48] INFO [2022-03-13 17:09:46,570] jepsen worker 53 - jepsen.util 503 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,570] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:46,571] jepsen worker 53 - jepsen.util 503 :invoke :write [17 42] INFO [2022-03-13 17:09:46,602] jepsen worker 23 - jepsen.util 563 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,602] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:46,615] jepsen worker 7 - jepsen.util 457 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,616] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:46,616] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,617] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:46,617] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,617] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,617] jepsen worker 7 - jepsen.util 457 :invoke :write [2 7] INFO [2022-03-13 17:09:46,619] jepsen worker 78 - jepsen.util 438 :invoke :write [26 3] INFO [2022-03-13 17:09:46,619] jepsen worker 52 - jepsen.util 502 :invoke :write [17 45] INFO [2022-03-13 17:09:46,621] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:46,637] jepsen worker 42 - jepsen.util 402 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,638] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:46,639] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:46,639] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:46,660] jepsen worker 8 - jepsen.util 638 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,660] jepsen worker 8 - jepsen.util 638 :invoke :write [2 12] INFO [2022-03-13 17:09:46,661] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:46,662] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:46,680] jepsen worker 60 - jepsen.util 600 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,680] jepsen worker 60 - jepsen.util 600 :invoke :write [20 18] INFO [2022-03-13 17:09:46,702] jepsen worker 61 - jepsen.util 421 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,702] jepsen worker 61 - jepsen.util 421 :invoke :write [20 44] INFO [2022-03-13 17:09:46,706] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,706] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:46,706] jepsen worker 24 - jepsen.util 474 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,706] jepsen worker 25 - jepsen.util 385 :invoke :write [8 15] INFO [2022-03-13 17:09:46,707] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:46,707] jepsen worker 80 - jepsen.util 620 :invoke :write [26 46] INFO [2022-03-13 17:09:46,738] jepsen worker 84 - jepsen.util 1344 :info :write [28 38] :RequestTimeoutException INFO [2022-03-13 17:09:46,739] jepsen worker 43 - jepsen.util 403 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,739] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:46,759] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:46,739] jepsen worker 84 - jepsen.util 1434 :invoke :read [28 nil] INFO [2022-03-13 17:09:46,759] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:46,780] jepsen worker 35 - jepsen.util 665 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,781] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:46,782] jepsen worker 78 - jepsen.util 438 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,782] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:46,826] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:46,827] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:46,827] jepsen worker 7 - jepsen.util 457 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,827] jepsen worker 7 - jepsen.util 457 :invoke :write [2 3] INFO [2022-03-13 17:09:46,850] jepsen worker 58 - jepsen.util 958 :info :write [19 6] :RequestTimeoutException INFO [2022-03-13 17:09:46,851] jepsen worker 58 - jepsen.util 1048 :invoke :write [19 46] INFO [2022-03-13 17:09:46,856] jepsen worker 82 - jepsen.util 1072 :info :write [27 1] :RequestTimeoutException INFO [2022-03-13 17:09:46,857] jepsen worker 82 - jepsen.util 1162 :invoke :write [27 40] INFO [2022-03-13 17:09:46,868] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2023460410557185}, :ns_1@172.23.105.187 {:progress 0.2748538011695907}, :ns_1@172.23.107.227 {:progress 0.2023460410557185}, :ns_1@172.23.107.228 {:progress 0.211764705882353}} INFO [2022-03-13 17:09:46,868] jepsen worker nemesis - couchbase.util Rebalance check count: 99 INFO [2022-03-13 17:09:46,869] jepsen worker nemesis - couchbase.util Rebalance stuck count: 56 INFO [2022-03-13 17:09:46,893] jepsen worker 52 - jepsen.util 502 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,894] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:46,900] jepsen worker 60 - jepsen.util 600 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,900] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:46,900] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:46,901] jepsen worker 60 - jepsen.util 600 :invoke :write [20 12] INFO [2022-03-13 17:09:46,901] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:46,902] jepsen worker 23 - jepsen.util 563 :invoke :write [7 29] INFO [2022-03-13 17:09:46,917] jepsen worker 8 - jepsen.util 638 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,917] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:46,918] jepsen worker 24 - jepsen.util 474 :invoke :write [8 14] INFO [2022-03-13 17:09:46,918] jepsen worker 8 - jepsen.util 638 :invoke :write [2 31] INFO [2022-03-13 17:09:46,929] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:46,929] jepsen worker 76 - jepsen.util 526 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,930] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:46,931] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:46,935] jepsen worker 81 - jepsen.util 1251 :info :write [27 40] :RequestTimeoutException INFO [2022-03-13 17:09:46,936] jepsen worker 81 - jepsen.util 1341 :invoke :write [27 35] INFO [2022-03-13 17:09:46,980] jepsen worker 25 - jepsen.util 385 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:46,981] jepsen worker 25 - jepsen.util 385 :invoke :write [8 19] INFO [2022-03-13 17:09:46,993] jepsen worker 86 - jepsen.util 986 :info :write [28 48] :RequestTimeoutException INFO [2022-03-13 17:09:46,995] jepsen worker 86 - jepsen.util 1076 :invoke :read [28 nil] WARN [2022-03-13 17:09:47,013] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17922345,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18465550,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9159325,"encode_duration_us":31368,"operation_id":"0x2ca9","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10001122},{"operation_name":"upsert","last_dispatch_duration_us":17342589,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17601595,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9077096,"encode_duration_us":17965,"operation_id":"0x2ccb","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007402},{"operation_name":"upsert","last_dispatch_duration_us":17349918,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17860687,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9179603,"encode_duration_us":22146,"operation_id":"0x2cc1","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10006631}],"total_count":3}} INFO [2022-03-13 17:09:47,033] jepsen worker 80 - jepsen.util 620 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,033] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,034] jepsen worker 80 - jepsen.util 620 :invoke :write [26 39] INFO [2022-03-13 17:09:47,035] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:47,046] jepsen worker 29 - jepsen.util 1109 :info :write [9 5] :RequestTimeoutException INFO [2022-03-13 17:09:47,047] jepsen worker 53 - jepsen.util 503 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,047] jepsen worker 29 - jepsen.util 1199 :invoke :read [9 nil] INFO [2022-03-13 17:09:47,047] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:47,047] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:47,048] jepsen worker 14 - jepsen.util 194 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,048] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:47,048] jepsen worker 42 - jepsen.util 402 :invoke :write [14 49] INFO [2022-03-13 17:09:47,049] jepsen worker 14 - jepsen.util 194 :invoke :write [4 0] INFO [2022-03-13 17:09:47,049] jepsen worker 26 - jepsen.util 476 :invoke :write [8 20] INFO [2022-03-13 17:09:47,076] jepsen worker 10 - jepsen.util 910 :info :write [3 42] :RequestTimeoutException INFO [2022-03-13 17:09:47,077] jepsen worker 10 - jepsen.util 1000 :invoke :read [3 nil] INFO [2022-03-13 17:09:47,087] jepsen worker 7 - jepsen.util 457 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,088] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:47,088] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:47,089] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:47,090] jepsen worker 47 - jepsen.util 1037 :info :write [15 15] :RequestTimeoutException INFO [2022-03-13 17:09:47,092] jepsen worker 47 - jepsen.util 1127 :invoke :write [15 2] WARN [2022-03-13 17:09:47,108] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9059163,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10216159,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9341944,"encode_duration_us":17449,"operation_id":"0x2ea0","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10003449},{"operation_name":"upsert","last_dispatch_duration_us":8014577,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":8542066,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":8282683,"encode_duration_us":19605,"operation_id":"0x2f17","timeout_ms":10000,"last_server_duration_us":8000661,"total_duration_us":10008829}],"total_count":2}} WARN [2022-03-13 17:09:47,113] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15831,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":15831,"total_server_duration_us":43,"operation_id":"0x2eb7","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":12318882},{"operation_name":"get","last_dispatch_duration_us":12173530,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12173530,"total_server_duration_us":43,"operation_id":"0x2e9a","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":12175969},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":17831,"operation_id":"0x3005","timeout_ms":10000,"total_duration_us":10009619},{"operation_name":"upsert","last_dispatch_duration_us":16934201,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":16934201,"total_server_duration_us":55303,"encode_duration_us":32256,"operation_id":"0x2d30","timeout_ms":10000,"last_server_duration_us":55303,"total_duration_us":10007750},{"operation_name":"upsert","last_dispatch_duration_us":17342589,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17601595,"total_server_duration_us":9077096,"encode_duration_us":17965,"operation_id":"0x2ccb","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007402},{"operation_name":"upsert","last_dispatch_duration_us":15637071,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":15637071,"total_server_duration_us":21405,"encode_duration_us":25170,"operation_id":"0x2dba","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10007117},{"operation_name":"upsert","last_dispatch_duration_us":17349918,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17860687,"total_server_duration_us":9179603,"encode_duration_us":22146,"operation_id":"0x2cc1","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10006631},{"operation_name":"upsert","last_dispatch_duration_us":39952,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9334244,"total_server_duration_us":65027,"encode_duration_us":19341,"operation_id":"0x2fcb","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10003895},{"operation_name":"upsert","last_dispatch_duration_us":9211348,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17986985,"total_server_duration_us":179519,"encode_duration_us":34091,"operation_id":"0x2cbb","timeout_ms":10000,"last_server_duration_us":47192,"total_duration_us":10002610},{"operation_name":"upsert","last_dispatch_duration_us":9076729,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9076729,"total_server_duration_us":9034183,"encode_duration_us":16802,"operation_id":"0x2d7a","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":9081651}],"total_count":30}} WARN [2022-03-13 17:09:47,118] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9173489,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9173489,"total_server_duration_us":21813,"encode_duration_us":20063,"operation_id":"0x301a","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10009803},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":23992,"operation_id":"0x3010","timeout_ms":10000,"total_duration_us":10009408},{"operation_name":"upsert","last_dispatch_duration_us":18178702,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18178702,"total_server_duration_us":9037366,"encode_duration_us":29736,"operation_id":"0x2cf0","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10008432},{"operation_name":"upsert","last_dispatch_duration_us":10123008,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10123008,"total_server_duration_us":28493,"encode_duration_us":18350,"operation_id":"0x2fae","timeout_ms":10000,"last_server_duration_us":28493,"total_duration_us":10005636},{"operation_name":"upsert","last_dispatch_duration_us":18120056,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18120056,"total_server_duration_us":9092628,"encode_duration_us":26561,"operation_id":"0x2cf7","timeout_ms":10000,"last_server_duration_us":9092628,"total_duration_us":10002599},{"operation_name":"upsert","last_dispatch_duration_us":981409,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5601977,"total_server_duration_us":298696,"encode_duration_us":17658,"operation_id":"0x2ea2","timeout_ms":10000,"last_server_duration_us":39424,"total_duration_us":10001378},{"operation_name":"get","last_dispatch_duration_us":9182552,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9182552,"total_server_duration_us":32,"operation_id":"0x300d","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9183875},{"operation_name":"get","last_dispatch_duration_us":9150814,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9150814,"total_server_duration_us":27132,"operation_id":"0x3019","timeout_ms":30000,"last_server_duration_us":27132,"total_duration_us":9153282},{"operation_name":"get","last_dispatch_duration_us":5668577,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5668577,"total_server_duration_us":8557,"operation_id":"0x2e10","timeout_ms":30000,"last_server_duration_us":8557,"total_duration_us":5670908},{"operation_name":"upsert","last_dispatch_duration_us":186529,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2179163,"total_server_duration_us":202669,"encode_duration_us":19677,"operation_id":"0x302d","timeout_ms":10000,"last_server_duration_us":63439,"total_duration_us":2196998}],"total_count":16}} INFO [2022-03-13 17:09:47,154] jepsen worker 73 - jepsen.util 1153 :info :write [24 20] :RequestTimeoutException INFO [2022-03-13 17:09:47,155] jepsen worker 73 - jepsen.util 1243 :invoke :read [24 nil] INFO [2022-03-13 17:09:47,159] jepsen worker 9 - jepsen.util 999 :info :write [3 25] :RequestTimeoutException INFO [2022-03-13 17:09:47,160] jepsen worker 9 - jepsen.util 1089 :invoke :write [3 42] WARN [2022-03-13 17:09:47,206] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":23416043,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":23416043,"total_server_duration_us":49,"operation_id":"0x2a98","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":23416321},{"operation_name":"get","last_dispatch_duration_us":18081901,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":18081901,"total_server_duration_us":56,"operation_id":"0x2ccf","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":18084709},{"operation_name":"upsert","last_dispatch_duration_us":8989033,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10743811,"total_server_duration_us":341786,"encode_duration_us":22932,"operation_id":"0x2cc3","timeout_ms":10000,"last_server_duration_us":25362,"total_duration_us":10011097},{"operation_name":"upsert","last_dispatch_duration_us":8575961,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10371808,"total_server_duration_us":412430,"encode_duration_us":20857,"operation_id":"0x2cbe","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10009787},{"operation_name":"upsert","last_dispatch_duration_us":8993160,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10802150,"total_server_duration_us":424756,"encode_duration_us":21893,"operation_id":"0x2cc4","timeout_ms":10000,"last_server_duration_us":26153,"total_duration_us":10006083},{"operation_name":"upsert","last_dispatch_duration_us":32691,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7652892,"total_server_duration_us":263527,"encode_duration_us":17551,"operation_id":"0x2ec2","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10005548},{"operation_name":"upsert","last_dispatch_duration_us":16887743,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16887743,"total_server_duration_us":35211,"encode_duration_us":17632,"operation_id":"0x2d31","timeout_ms":10000,"last_server_duration_us":35211,"total_duration_us":10004711},{"operation_name":"upsert","last_dispatch_duration_us":9175445,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9175445,"total_server_duration_us":42634,"encode_duration_us":18292,"operation_id":"0x2fc8","timeout_ms":10000,"last_server_duration_us":42634,"total_duration_us":10004608},{"operation_name":"upsert","last_dispatch_duration_us":34029,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":3305631,"total_server_duration_us":507509,"encode_duration_us":28442,"operation_id":"0x2cc8","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10001391},{"operation_name":"upsert","last_dispatch_duration_us":33700,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1020891,"total_server_duration_us":636973,"encode_duration_us":18884,"operation_id":"0x2df5","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10001184}],"total_count":61}} WARN [2022-03-13 17:09:47,215] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":53526,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1000837,"total_server_duration_us":623485,"encode_duration_us":16802,"operation_id":"0x2eab","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10010916},{"operation_name":"upsert","last_dispatch_duration_us":52957,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1017244,"total_server_duration_us":633083,"encode_duration_us":17342,"operation_id":"0x2d73","timeout_ms":10000,"last_server_duration_us":39214,"total_duration_us":10008535},{"operation_name":"upsert","last_dispatch_duration_us":35074,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8643183,"total_server_duration_us":262935,"encode_duration_us":21081,"operation_id":"0x3021","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10005806},{"operation_name":"upsert","last_dispatch_duration_us":157098,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9664597,"total_server_duration_us":198744,"encode_duration_us":17153,"operation_id":"0x2f81","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10005708},{"operation_name":"upsert","last_dispatch_duration_us":8872788,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16461079,"total_server_duration_us":122828,"encode_duration_us":17430,"operation_id":"0x2cf8","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10003615},{"operation_name":"upsert","last_dispatch_duration_us":111743,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9812501,"total_server_duration_us":234834,"encode_duration_us":20195,"operation_id":"0x2fa6","timeout_ms":10000,"last_server_duration_us":33524,"total_duration_us":10003258},{"operation_name":"upsert","last_dispatch_duration_us":34736,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":746482,"total_server_duration_us":477332,"encode_duration_us":19559,"operation_id":"0x2e3c","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10000724},{"operation_name":"upsert","last_dispatch_duration_us":9109798,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9109798,"total_server_duration_us":9023573,"encode_duration_us":21034,"operation_id":"0x2f83","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9114898},{"operation_name":"get","last_dispatch_duration_us":8189357,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8189357,"total_server_duration_us":10154,"operation_id":"0x2fca","timeout_ms":30000,"last_server_duration_us":10154,"total_duration_us":8191787},{"operation_name":"get","last_dispatch_duration_us":161140,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":161140,"total_server_duration_us":7750,"operation_id":"0x30a3","timeout_ms":30000,"last_server_duration_us":7750,"total_duration_us":6347616}],"total_count":22}} WARN [2022-03-13 17:09:47,218] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18120056,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18120056,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9092628,"encode_duration_us":26561,"operation_id":"0x2cf7","timeout_ms":10000,"last_server_duration_us":9092628,"total_duration_us":10002599},{"operation_name":"upsert","last_dispatch_duration_us":18178702,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18178702,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9037366,"encode_duration_us":29736,"operation_id":"0x2cf0","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10008432}],"total_count":2}} INFO [2022-03-13 17:09:47,296] jepsen worker 63 - jepsen.util 1053 :info :write [21 31] :RequestTimeoutException INFO [2022-03-13 17:09:47,297] jepsen worker 63 - jepsen.util 1143 :invoke :read [21 nil] WARN [2022-03-13 17:09:47,307] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9062341,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9886667,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9318064,"encode_duration_us":18059,"operation_id":"0x2d18","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10000660},{"operation_name":"upsert","last_dispatch_duration_us":27510196,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":32254456,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9423148,"encode_duration_us":20240,"operation_id":"0x259a","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10007924}],"total_count":2}} WARN [2022-03-13 17:09:47,308] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":1059777,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":2296257,"total_server_duration_us":342371,"encode_duration_us":17678,"operation_id":"0x2e81","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10009850},{"operation_name":"upsert","last_dispatch_duration_us":10048458,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11389088,"total_server_duration_us":9350313,"encode_duration_us":21670,"operation_id":"0x2e82","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10009487},{"operation_name":"upsert","last_dispatch_duration_us":42099,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":1439869,"total_server_duration_us":195959,"encode_duration_us":22096,"operation_id":"0x301d","timeout_ms":10000,"last_server_duration_us":29045,"total_duration_us":10009438},{"operation_name":"upsert","last_dispatch_duration_us":8014577,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8542066,"total_server_duration_us":8282683,"encode_duration_us":19605,"operation_id":"0x2f17","timeout_ms":10000,"last_server_duration_us":8000661,"total_duration_us":10008829},{"operation_name":"upsert","last_dispatch_duration_us":8996679,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9042337,"total_server_duration_us":49878,"encode_duration_us":19457,"operation_id":"0x2e1c","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10008574},{"operation_name":"upsert","last_dispatch_duration_us":64941,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":64941,"total_server_duration_us":21731,"encode_duration_us":18912,"operation_id":"0x2e1d","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10006077},{"operation_name":"upsert","last_dispatch_duration_us":706922,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":6950635,"total_server_duration_us":371958,"encode_duration_us":16901,"operation_id":"0x2ce3","timeout_ms":10000,"last_server_duration_us":31012,"total_duration_us":10004418},{"operation_name":"upsert","last_dispatch_duration_us":7695545,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7695545,"total_server_duration_us":20441,"encode_duration_us":18283,"operation_id":"0x2e9f","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10003664},{"operation_name":"upsert","last_dispatch_duration_us":9059163,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10216159,"total_server_duration_us":9341944,"encode_duration_us":17449,"operation_id":"0x2ea0","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10003449},{"operation_name":"upsert","last_dispatch_duration_us":9138682,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9138682,"total_server_duration_us":9042674,"encode_duration_us":18278,"operation_id":"0x2e12","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9143250}],"total_count":81}} INFO [2022-03-13 17:09:47,356] jepsen worker 37 - jepsen.util 1117 :info :write [12 22] :RequestTimeoutException INFO [2022-03-13 17:09:47,357] jepsen worker 37 - jepsen.util 1207 :invoke :read [12 nil] INFO [2022-03-13 17:09:47,379] jepsen worker 37 - jepsen.util 1207 :ok :read [12 2] INFO [2022-03-13 17:09:47,380] jepsen worker 37 - jepsen.util 1207 :invoke :write [12 28] INFO [2022-03-13 17:09:47,476] jepsen worker 60 - jepsen.util 600 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,477] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:47,523] jepsen worker 24 - jepsen.util 474 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,523] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:47,553] jepsen worker 23 - jepsen.util 563 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,553] jepsen worker 61 - jepsen.util 421 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,553] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,553] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:47,554] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,554] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,554] jepsen worker 61 - jepsen.util 421 :invoke :write [20 4] INFO [2022-03-13 17:09:47,556] jepsen worker 43 - jepsen.util 403 :invoke :write [14 25] INFO [2022-03-13 17:09:47,557] jepsen worker 76 - jepsen.util 526 :invoke :write [25 13] INFO [2022-03-13 17:09:47,559] jepsen worker 60 - jepsen.util 600 :invoke :write [20 34] INFO [2022-03-13 17:09:47,560] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:09:47,566] jepsen worker 35 - jepsen.util 665 :invoke :write [11 36] INFO [2022-03-13 17:09:47,576] jepsen worker 34 - jepsen.util 574 :info :write [11 6] :RequestTimeoutException INFO [2022-03-13 17:09:47,576] jepsen worker 22 - jepsen.util 472 :info :write [7 33] :RequestTimeoutException INFO [2022-03-13 17:09:47,577] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:09:47,578] jepsen worker 34 - jepsen.util 664 :invoke :read [11 nil] INFO [2022-03-13 17:09:47,593] jepsen worker 12 - jepsen.util 642 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,594] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,594] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:47,594] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:47,643] jepsen worker 40 - jepsen.util 1030 :info :write [13 46] :SyncWriteAmbiguous INFO [2022-03-13 17:09:47,644] jepsen worker 40 - jepsen.util 1120 :invoke :write [13 42] INFO [2022-03-13 17:09:47,651] jepsen worker 8 - jepsen.util 638 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,652] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:47,674] jepsen worker 75 - jepsen.util 345 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,675] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,675] jepsen worker 75 - jepsen.util 345 :invoke :read [25 nil] INFO [2022-03-13 17:09:47,676] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:09:47,708] jepsen worker 60 - jepsen.util 600 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,709] jepsen worker 60 - jepsen.util 600 :invoke :write [20 45] INFO [2022-03-13 17:09:47,717] jepsen worker 38 - jepsen.util 1298 :info :write [12 1] :SyncWriteAmbiguous INFO [2022-03-13 17:09:47,718] jepsen worker 38 - jepsen.util 1388 :invoke :write [12 20] INFO [2022-03-13 17:09:47,724] jepsen worker 25 - jepsen.util 385 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,725] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:47,753] jepsen worker 88 - jepsen.util 1078 :ok :read [29 6] INFO [2022-03-13 17:09:47,754] jepsen worker 88 - jepsen.util 1078 :invoke :read [29 nil] INFO [2022-03-13 17:09:47,767] jepsen worker 80 - jepsen.util 620 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,768] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:47,768] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,768] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,768] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,768] jepsen worker 80 - jepsen.util 620 :invoke :write [26 11] INFO [2022-03-13 17:09:47,770] jepsen worker 7 - jepsen.util 457 :invoke :write [2 27] INFO [2022-03-13 17:09:47,770] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:47,770] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:09:47,771] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:47,771] jepsen worker 21 - jepsen.util 651 :invoke :write [7 37] INFO [2022-03-13 17:09:47,771] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,772] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:09:47,772] jepsen worker 25 - jepsen.util 385 :invoke :write [8 0] INFO [2022-03-13 17:09:47,773] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:09:47,773] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,773] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:47,774] jepsen worker 78 - jepsen.util 438 :invoke :write [26 10] INFO [2022-03-13 17:09:47,774] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:47,775] jepsen worker 52 - jepsen.util 502 :invoke :write [17 25] INFO [2022-03-13 17:09:47,776] jepsen worker 22 - jepsen.util 562 :invoke :write [7 3] INFO [2022-03-13 17:09:47,777] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:47,786] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:47,786] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,787] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:47,787] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:47,789] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:47,789] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:47,796] jepsen worker 42 - jepsen.util 402 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,797] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:47,836] jepsen worker 79 - jepsen.util 799 :info :write [26 43] :RequestTimeoutException INFO [2022-03-13 17:09:47,837] jepsen worker 79 - jepsen.util 889 :invoke :write [26 45] INFO [2022-03-13 17:09:47,866] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,866] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:47,866] jepsen worker 7 - jepsen.util 457 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,867] jepsen worker 7 - jepsen.util 457 :invoke :write [2 24] INFO [2022-03-13 17:09:47,867] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:09:47,867] jepsen worker 24 - jepsen.util 474 :invoke :write [8 36] INFO [2022-03-13 17:09:47,876] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2052785923753666}, :ns_1@172.23.105.187 {:progress 0.2865497076023392}, :ns_1@172.23.107.227 {:progress 0.2052785923753666}, :ns_1@172.23.107.228 {:progress 0.2176470588235294}} INFO [2022-03-13 17:09:47,876] jepsen worker nemesis - couchbase.util Rebalance check count: 100 INFO [2022-03-13 17:09:47,876] jepsen worker nemesis - couchbase.util Rebalance stuck count: 56 INFO [2022-03-13 17:09:47,890] jepsen worker 13 - jepsen.util 463 :info :write [4 46] :RequestTimeoutException INFO [2022-03-13 17:09:47,890] jepsen worker 77 - jepsen.util 617 :info :write [25 42] :RequestTimeoutException INFO [2022-03-13 17:09:47,891] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:09:47,891] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:09:47,900] jepsen worker 51 - jepsen.util 411 :info :write [17 16] :RequestTimeoutException INFO [2022-03-13 17:09:47,901] jepsen worker 51 - jepsen.util 501 :invoke :write [17 29] INFO [2022-03-13 17:09:47,912] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,912] jepsen worker 23 - jepsen.util 563 :invoke :write [7 25] INFO [2022-03-13 17:09:47,924] jepsen worker 21 - jepsen.util 651 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:47,924] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:47,927] jepsen worker 89 - jepsen.util 1349 :ok :read [29 6] INFO [2022-03-13 17:09:47,928] jepsen worker 89 - jepsen.util 1349 :invoke :write [29 0] INFO [2022-03-13 17:09:48,026] jepsen worker 25 - jepsen.util 385 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,026] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:48,027] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,027] jepsen worker 44 - jepsen.util 494 :invoke :write [14 11] INFO [2022-03-13 17:09:48,028] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:48,029] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:48,064] jepsen worker 64 - jepsen.util 874 :info :write [21 33] :RequestTimeoutException INFO [2022-03-13 17:09:48,064] jepsen worker 52 - jepsen.util 502 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,065] jepsen worker 52 - jepsen.util 502 :invoke :write [17 26] INFO [2022-03-13 17:09:48,065] jepsen worker 64 - jepsen.util 964 :invoke :write [21 28] INFO [2022-03-13 17:09:48,084] jepsen worker 71 - jepsen.util 431 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,085] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:48,117] jepsen worker 43 - jepsen.util 403 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,117] jepsen worker 14 - jepsen.util 194 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,118] jepsen worker 43 - jepsen.util 403 :invoke :write [14 43] INFO [2022-03-13 17:09:48,118] jepsen worker 14 - jepsen.util 194 :invoke :write [4 23] INFO [2022-03-13 17:09:48,121] jepsen worker 80 - jepsen.util 620 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,122] jepsen worker 80 - jepsen.util 620 :invoke :write [26 47] INFO [2022-03-13 17:09:48,281] jepsen worker 16 - jepsen.util 1096 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:09:48,281] jepsen worker 16 - jepsen.util 1186 :invoke :read [5 nil] INFO [2022-03-13 17:09:48,305] jepsen worker 16 - jepsen.util 1186 :ok :read [5 43] INFO [2022-03-13 17:09:48,305] jepsen worker 16 - jepsen.util 1186 :invoke :write [5 27] INFO [2022-03-13 17:09:48,662] jepsen worker 26 - jepsen.util 476 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,662] jepsen worker 60 - jepsen.util 600 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,663] jepsen worker 26 - jepsen.util 476 :invoke :write [8 38] INFO [2022-03-13 17:09:48,663] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:48,683] jepsen worker 78 - jepsen.util 438 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,683] jepsen worker 78 - jepsen.util 438 :invoke :write [26 15] INFO [2022-03-13 17:09:48,731] jepsen worker 61 - jepsen.util 421 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,731] jepsen worker 34 - jepsen.util 664 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,732] jepsen worker 34 - jepsen.util 664 :invoke :read [11 nil] INFO [2022-03-13 17:09:48,732] jepsen worker 61 - jepsen.util 421 :invoke :write [20 42] INFO [2022-03-13 17:09:48,744] jepsen worker 5 - jepsen.util 1175 :info :write [1 44] :RequestTimeoutException INFO [2022-03-13 17:09:48,745] jepsen worker 5 - jepsen.util 1265 :invoke :read [1 nil] INFO [2022-03-13 17:09:48,760] jepsen worker 36 - jepsen.util 1206 :info :write [12 4] :RequestTimeoutException INFO [2022-03-13 17:09:48,761] jepsen worker 36 - jepsen.util 1296 :invoke :write [12 36] INFO [2022-03-13 17:09:48,772] jepsen worker 79 - jepsen.util 889 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,772] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:09:48,775] jepsen worker 5 - jepsen.util 1265 :ok :read [1 42] INFO [2022-03-13 17:09:48,775] jepsen worker 5 - jepsen.util 1265 :invoke :read [1 nil] INFO [2022-03-13 17:09:48,786] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,787] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:48,800] jepsen worker 5 - jepsen.util 1265 :ok :read [1 42] INFO [2022-03-13 17:09:48,801] jepsen worker 5 - jepsen.util 1265 :invoke :read [1 nil] INFO [2022-03-13 17:09:48,826] jepsen worker 5 - jepsen.util 1265 :ok :read [1 42] INFO [2022-03-13 17:09:48,826] jepsen worker 5 - jepsen.util 1265 :invoke :write [1 9] INFO [2022-03-13 17:09:48,855] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,856] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:48,859] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,860] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:48,861] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,862] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:48,863] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,864] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:48,865] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,866] jepsen worker 79 - jepsen.util 889 :invoke :write [26 14] INFO [2022-03-13 17:09:48,866] jepsen worker 22 - jepsen.util 562 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,866] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:48,867] jepsen worker 25 - jepsen.util 385 :invoke :write [8 48] INFO [2022-03-13 17:09:48,867] jepsen worker 22 - jepsen.util 562 :invoke :write [7 38] INFO [2022-03-13 17:09:48,881] jepsen worker 76 - jepsen.util 526 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,882] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:48,882] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2052785923753666}, :ns_1@172.23.105.187 {:progress 0.2865497076023392}, :ns_1@172.23.107.227 {:progress 0.2052785923753666}, :ns_1@172.23.107.228 {:progress 0.2176470588235294}} INFO [2022-03-13 17:09:48,882] jepsen worker nemesis - couchbase.util Rebalance check count: 101 INFO [2022-03-13 17:09:48,882] jepsen worker nemesis - couchbase.util Rebalance stuck count: 57 INFO [2022-03-13 17:09:48,919] jepsen worker 23 - jepsen.util 563 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,920] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:48,937] jepsen worker 6 - jepsen.util 546 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,937] jepsen worker 6 - jepsen.util 546 :invoke :write [2 40] INFO [2022-03-13 17:09:48,940] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,940] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:48,949] jepsen worker 79 - jepsen.util 889 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:48,949] jepsen worker 79 - jepsen.util 889 :invoke :write [26 6] INFO [2022-03-13 17:09:49,029] jepsen worker 43 - jepsen.util 403 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,029] jepsen worker 35 - jepsen.util 665 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,029] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:49,030] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:49,030] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,030] jepsen worker 12 - jepsen.util 642 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,031] jepsen worker 12 - jepsen.util 642 :invoke :read [4 nil] INFO [2022-03-13 17:09:49,031] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:49,033] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,033] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,033] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:49,033] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:49,033] jepsen worker 60 - jepsen.util 600 :invoke :write [20 11] INFO [2022-03-13 17:09:49,034] jepsen worker 43 - jepsen.util 403 :invoke :write [14 10] INFO [2022-03-13 17:09:49,035] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:09:49,039] jepsen worker 23 - jepsen.util 563 :invoke :write [7 29] INFO [2022-03-13 17:09:49,057] jepsen worker 24 - jepsen.util 474 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,057] jepsen worker 24 - jepsen.util 474 :invoke :write [8 47] INFO [2022-03-13 17:09:49,076] jepsen worker 79 - jepsen.util 889 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,077] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:09:49,098] jepsen worker 14 - jepsen.util 194 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,099] jepsen worker 14 - jepsen.util 194 :invoke :write [4 35] INFO [2022-03-13 17:09:49,115] jepsen worker 44 - jepsen.util 494 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,115] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,116] jepsen worker 44 - jepsen.util 494 :invoke :write [14 33] INFO [2022-03-13 17:09:49,116] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,116] jepsen worker 71 - jepsen.util 431 :invoke :write [23 24] INFO [2022-03-13 17:09:49,117] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:09:49,191] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,192] jepsen worker 79 - jepsen.util 889 :invoke :write [26 24] INFO [2022-03-13 17:09:49,230] jepsen worker 7 - jepsen.util 457 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,230] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,231] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:49,231] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:09:49,242] jepsen worker 51 - jepsen.util 501 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,243] jepsen worker 51 - jepsen.util 501 :invoke :write [17 20] INFO [2022-03-13 17:09:49,265] jepsen worker 6 - jepsen.util 546 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,266] jepsen worker 6 - jepsen.util 546 :invoke :write [2 30] INFO [2022-03-13 17:09:49,297] jepsen worker 79 - jepsen.util 889 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,298] jepsen worker 79 - jepsen.util 889 :invoke :write [26 9] INFO [2022-03-13 17:09:49,316] jepsen worker 60 - jepsen.util 600 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,316] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:49,381] jepsen worker 52 - jepsen.util 502 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,381] jepsen worker 52 - jepsen.util 502 :invoke :write [17 36] INFO [2022-03-13 17:09:49,412] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,412] jepsen worker 79 - jepsen.util 889 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,412] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:09:49,413] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:49,415] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,415] jepsen worker 79 - jepsen.util 889 :invoke :write [26 2] INFO [2022-03-13 17:09:49,417] jepsen worker 26 - jepsen.util 476 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,418] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:49,430] jepsen worker 71 - jepsen.util 431 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,431] jepsen worker 71 - jepsen.util 431 :invoke :write [23 40] INFO [2022-03-13 17:09:49,436] jepsen worker 48 - jepsen.util 1038 :info :write [16 23] :RequestTimeoutException INFO [2022-03-13 17:09:49,438] jepsen worker 48 - jepsen.util 1128 :invoke :read [16 nil] INFO [2022-03-13 17:09:49,470] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,471] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,525] jepsen worker 79 - jepsen.util 889 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,525] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:49,526] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,527] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:49,528] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,528] jepsen worker 79 - jepsen.util 889 :invoke :write [26 48] INFO [2022-03-13 17:09:49,547] jepsen worker 78 - jepsen.util 438 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,547] jepsen worker 78 - jepsen.util 438 :invoke :write [26 18] INFO [2022-03-13 17:09:49,612] jepsen worker 61 - jepsen.util 421 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,612] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,612] jepsen worker 76 - jepsen.util 526 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,614] jepsen worker 61 - jepsen.util 421 :invoke :read [20 nil] INFO [2022-03-13 17:09:49,628] jepsen worker 44 - jepsen.util 494 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,628] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:49,636] jepsen worker 43 - jepsen.util 403 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,636] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,636] jepsen worker 12 - jepsen.util 642 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,636] jepsen worker 34 - jepsen.util 664 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,636] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,637] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:09:49,637] jepsen worker 62 - jepsen.util 422 :invoke :write [20 4] INFO [2022-03-13 17:09:49,638] jepsen worker 34 - jepsen.util 664 :invoke :write [11 42] INFO [2022-03-13 17:09:49,640] jepsen worker 43 - jepsen.util 403 :invoke :write [14 37] INFO [2022-03-13 17:09:49,642] jepsen worker 12 - jepsen.util 642 :invoke :write [4 21] INFO [2022-03-13 17:09:49,724] jepsen worker 79 - jepsen.util 889 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,724] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:49,726] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,727] jepsen worker 79 - jepsen.util 889 :invoke :write [26 45] INFO [2022-03-13 17:09:49,744] jepsen worker 22 - jepsen.util 562 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,745] jepsen worker 22 - jepsen.util 562 :invoke :write [7 12] INFO [2022-03-13 17:09:49,749] jepsen worker 55 - jepsen.util 1135 :ok :read [18 49] INFO [2022-03-13 17:09:49,749] jepsen worker 55 - jepsen.util 1135 :invoke :write [18 21] INFO [2022-03-13 17:09:49,759] jepsen worker 19 - jepsen.util 919 :ok :read [6 16] INFO [2022-03-13 17:09:49,759] jepsen worker 19 - jepsen.util 919 :invoke :write [6 42] INFO [2022-03-13 17:09:49,815] jepsen worker 23 - jepsen.util 563 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,815] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,815] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,816] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,817] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:49,817] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:49,818] jepsen worker 14 - jepsen.util 194 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,818] jepsen worker 14 - jepsen.util 194 :invoke :read [4 nil] INFO [2022-03-13 17:09:49,819] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,820] jepsen worker 70 - jepsen.util 700 :invoke :write [23 18] INFO [2022-03-13 17:09:49,829] jepsen worker 79 - jepsen.util 889 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,830] jepsen worker 79 - jepsen.util 889 :invoke :write [26 27] INFO [2022-03-13 17:09:49,842] jepsen worker 18 - jepsen.util 918 :info :write [6 47] :SyncWriteAmbiguous INFO [2022-03-13 17:09:49,843] jepsen worker 18 - jepsen.util 1008 :invoke :write [6 22] INFO [2022-03-13 17:09:49,856] jepsen worker 51 - jepsen.util 501 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,856] jepsen worker 75 - jepsen.util 345 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,856] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,856] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:09:49,857] jepsen worker 76 - jepsen.util 526 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,857] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:09:49,857] jepsen worker 61 - jepsen.util 421 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,857] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,858] jepsen worker 26 - jepsen.util 476 :invoke :write [8 40] INFO [2022-03-13 17:09:49,858] jepsen worker 53 - jepsen.util 503 :invoke :write [17 40] INFO [2022-03-13 17:09:49,861] jepsen worker 42 - jepsen.util 402 :invoke :write [14 12] INFO [2022-03-13 17:09:49,863] jepsen worker 35 - jepsen.util 665 :invoke :write [11 1] INFO [2022-03-13 17:09:49,863] jepsen worker 61 - jepsen.util 421 :invoke :write [20 37] INFO [2022-03-13 17:09:49,865] jepsen worker 75 - jepsen.util 345 :invoke :write [25 14] INFO [2022-03-13 17:09:49,865] jepsen worker 51 - jepsen.util 501 :invoke :write [17 36] INFO [2022-03-13 17:09:49,865] jepsen worker 76 - jepsen.util 526 :invoke :write [25 13] INFO [2022-03-13 17:09:49,889] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2052785923753666}, :ns_1@172.23.105.187 {:progress 0.2865497076023392}, :ns_1@172.23.107.227 {:progress 0.2052785923753666}, :ns_1@172.23.107.228 {:progress 0.2176470588235294}} INFO [2022-03-13 17:09:49,890] jepsen worker nemesis - couchbase.util Rebalance check count: 102 INFO [2022-03-13 17:09:49,890] jepsen worker nemesis - couchbase.util Rebalance stuck count: 58 INFO [2022-03-13 17:09:49,908] jepsen worker 43 - jepsen.util 403 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,908] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,908] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:49,908] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,909] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,911] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,911] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:49,912] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:49,912] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:49,914] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,914] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,915] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,915] jepsen worker 43 - jepsen.util 403 :invoke :write [14 33] INFO [2022-03-13 17:09:49,916] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:49,917] jepsen worker 60 - jepsen.util 600 :invoke :write [20 28] INFO [2022-03-13 17:09:49,922] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,922] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:09:49,923] jepsen worker 23 - jepsen.util 563 :invoke :write [7 25] INFO [2022-03-13 17:09:49,923] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,923] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,924] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,925] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,957] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,957] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:09:49,959] jepsen worker 34 - jepsen.util 664 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:49,960] jepsen worker 34 - jepsen.util 664 :invoke :write [11 31] INFO [2022-03-13 17:09:50,000] jepsen worker 72 - jepsen.util 1062 :ok :read [24 24] INFO [2022-03-13 17:09:50,001] jepsen worker 72 - jepsen.util 1062 :invoke :write [24 46] INFO [2022-03-13 17:09:50,012] jepsen worker 79 - jepsen.util 889 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,014] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,015] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,016] jepsen worker 79 - jepsen.util 889 :invoke :write [26 34] INFO [2022-03-13 17:09:50,021] jepsen worker 10 - jepsen.util 1000 :ok :read [3 25] INFO [2022-03-13 17:09:50,022] jepsen worker 10 - jepsen.util 1000 :invoke :read [3 nil] INFO [2022-03-13 17:09:50,024] jepsen worker 24 - jepsen.util 474 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,025] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:50,076] jepsen worker 41 - jepsen.util 1121 :info :write [13 21] :RequestTimeoutException INFO [2022-03-13 17:09:50,077] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:09:50,115] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,116] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:50,136] jepsen worker 79 - jepsen.util 889 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,136] jepsen worker 6 - jepsen.util 546 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,136] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,137] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:50,143] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,144] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,144] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,145] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,146] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,146] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,147] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,148] jepsen worker 79 - jepsen.util 889 :invoke :write [26 32] INFO [2022-03-13 17:09:50,202] jepsen worker 12 - jepsen.util 642 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,202] jepsen worker 43 - jepsen.util 403 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,203] jepsen worker 12 - jepsen.util 642 :invoke :write [4 11] INFO [2022-03-13 17:09:50,204] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:09:50,235] jepsen worker 79 - jepsen.util 889 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,236] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:09:50,243] jepsen worker 10 - jepsen.util 1000 :ok :read [3 25] INFO [2022-03-13 17:09:50,244] jepsen worker 10 - jepsen.util 1000 :invoke :write [3 38] INFO [2022-03-13 17:09:50,271] jepsen worker 11 - jepsen.util 1001 :ok :read [3 25] INFO [2022-03-13 17:09:50,272] jepsen worker 11 - jepsen.util 1001 :invoke :write [3 1] INFO [2022-03-13 17:09:50,295] jepsen worker 32 - jepsen.util 932 :ok :read [10 :nil] INFO [2022-03-13 17:09:50,296] jepsen worker 32 - jepsen.util 932 :invoke :write [10 34] INFO [2022-03-13 17:09:50,332] jepsen worker 52 - jepsen.util 502 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,332] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,332] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:50,332] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,333] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:50,333] jepsen worker 21 - jepsen.util 651 :invoke :write [7 48] INFO [2022-03-13 17:09:50,334] jepsen worker 7 - jepsen.util 457 :invoke :write [2 46] INFO [2022-03-13 17:09:50,334] jepsen worker 13 - jepsen.util 553 :invoke :write [4 11] INFO [2022-03-13 17:09:50,370] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,371] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,373] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,374] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:09:50,374] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,375] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:50,376] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,377] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:09:50,377] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,378] jepsen worker 79 - jepsen.util 889 :invoke :write [26 28] INFO [2022-03-13 17:09:50,401] jepsen worker 71 - jepsen.util 431 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,401] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:50,405] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:50,406] jepsen worker 44 - jepsen.util 494 :invoke :write [14 7] INFO [2022-03-13 17:09:50,494] jepsen worker 79 - jepsen.util 889 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,495] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,544] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,544] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:50,544] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:50,545] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:50,550] jepsen worker 31 - jepsen.util 1111 :info :write [10 18] :RequestTimeoutException INFO [2022-03-13 17:09:50,551] jepsen worker 31 - jepsen.util 1201 :invoke :write [10 28] INFO [2022-03-13 17:09:50,570] jepsen worker 46 - jepsen.util 1216 :ok :read [15 48] INFO [2022-03-13 17:09:50,570] jepsen worker 46 - jepsen.util 1216 :invoke :write [15 46] INFO [2022-03-13 17:09:50,679] jepsen worker 63 - jepsen.util 1143 :ok :read [21 :nil] INFO [2022-03-13 17:09:50,680] jepsen worker 63 - jepsen.util 1143 :invoke :write [21 5] INFO [2022-03-13 17:09:50,767] jepsen worker 3 - jepsen.util 993 :info :write [1 39] :RequestTimeoutException INFO [2022-03-13 17:09:50,768] jepsen worker 3 - jepsen.util 1083 :invoke :read [1 nil] INFO [2022-03-13 17:09:50,799] jepsen worker 3 - jepsen.util 1083 :ok :read [1 42] INFO [2022-03-13 17:09:50,799] jepsen worker 3 - jepsen.util 1083 :invoke :write [1 22] INFO [2022-03-13 17:09:50,897] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2111436950146628}, :ns_1@172.23.105.187 {:progress 0.2865497076023392}, :ns_1@172.23.107.227 {:progress 0.2111436950146628}, :ns_1@172.23.107.228 {:progress 0.2235294117647059}} INFO [2022-03-13 17:09:50,897] jepsen worker nemesis - couchbase.util Rebalance check count: 103 INFO [2022-03-13 17:09:50,897] jepsen worker nemesis - couchbase.util Rebalance stuck count: 58 INFO [2022-03-13 17:09:51,520] jepsen worker 65 - jepsen.util 1145 :info :write [21 6] :RequestTimeoutException INFO [2022-03-13 17:09:51,520] jepsen worker 65 - jepsen.util 1235 :invoke :write [21 41] INFO [2022-03-13 17:09:51,739] jepsen worker 78 - jepsen.util 438 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,739] jepsen worker 78 - jepsen.util 438 :invoke :write [26 3] INFO [2022-03-13 17:09:51,741] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,756] jepsen worker 4 - jepsen.util 1174 :info :write [1 27] :RequestTimeoutException INFO [2022-03-13 17:09:51,766] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:51,766] jepsen worker 4 - jepsen.util 1264 :invoke :read [1 nil] INFO [2022-03-13 17:09:51,768] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,769] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:51,770] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,771] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:09:51,778] jepsen worker 70 - jepsen.util 700 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,778] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,779] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,779] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:51,779] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:51,780] jepsen worker 71 - jepsen.util 431 :invoke :write [23 0] INFO [2022-03-13 17:09:51,794] jepsen worker 4 - jepsen.util 1264 :ok :read [1 42] INFO [2022-03-13 17:09:51,795] jepsen worker 4 - jepsen.util 1264 :invoke :read [1 nil] INFO [2022-03-13 17:09:51,826] jepsen worker 4 - jepsen.util 1264 :ok :read [1 42] INFO [2022-03-13 17:09:51,826] jepsen worker 4 - jepsen.util 1264 :invoke :read [1 nil] INFO [2022-03-13 17:09:51,832] jepsen worker 80 - jepsen.util 620 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,832] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:51,851] jepsen worker 4 - jepsen.util 1264 :ok :read [1 42] INFO [2022-03-13 17:09:51,852] jepsen worker 4 - jepsen.util 1264 :invoke :write [1 42] INFO [2022-03-13 17:09:51,853] jepsen worker 44 - jepsen.util 494 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,854] jepsen worker 44 - jepsen.util 494 :invoke :write [14 21] INFO [2022-03-13 17:09:51,868] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,868] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,869] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:51,870] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:51,903] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2111436950146628}, :ns_1@172.23.105.187 {:progress 0.2865497076023392}, :ns_1@172.23.107.227 {:progress 0.2111436950146628}, :ns_1@172.23.107.228 {:progress 0.2235294117647059}} INFO [2022-03-13 17:09:51,903] jepsen worker nemesis - couchbase.util Rebalance check count: 104 INFO [2022-03-13 17:09:51,903] jepsen worker nemesis - couchbase.util Rebalance stuck count: 59 INFO [2022-03-13 17:09:51,911] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,911] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,911] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:51,912] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:51,913] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,913] jepsen worker 79 - jepsen.util 889 :invoke :write [26 44] INFO [2022-03-13 17:09:51,913] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,913] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,914] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:51,914] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:51,915] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,916] jepsen worker 69 - jepsen.util 609 :invoke :write [23 12] INFO [2022-03-13 17:09:51,917] jepsen worker 22 - jepsen.util 562 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,917] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,919] jepsen worker 52 - jepsen.util 502 :invoke :write [17 17] INFO [2022-03-13 17:09:51,920] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:09:51,922] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:51,923] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:51,939] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,940] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:51,947] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:51,948] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:09:52,007] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,007] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,007] jepsen worker 71 - jepsen.util 431 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,077] jepsen worker 62 - jepsen.util 422 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,077] jepsen worker 14 - jepsen.util 194 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,078] jepsen worker 79 - jepsen.util 889 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,079] jepsen worker 21 - jepsen.util 651 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,079] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:52,081] jepsen worker 70 - jepsen.util 700 :invoke :write [23 41] INFO [2022-03-13 17:09:52,081] jepsen worker 60 - jepsen.util 600 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,081] jepsen worker 71 - jepsen.util 431 :invoke :write [23 49] INFO [2022-03-13 17:09:52,082] jepsen worker 21 - jepsen.util 651 :invoke :write [7 1] INFO [2022-03-13 17:09:52,082] jepsen worker 14 - jepsen.util 194 :invoke :write [4 19] INFO [2022-03-13 17:09:52,084] jepsen worker 62 - jepsen.util 422 :invoke :read [20 nil] INFO [2022-03-13 17:09:52,084] jepsen worker 60 - jepsen.util 600 :invoke :write [20 29] INFO [2022-03-13 17:09:52,089] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,089] jepsen worker 44 - jepsen.util 494 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,090] jepsen worker 6 - jepsen.util 546 :invoke :write [2 47] INFO [2022-03-13 17:09:52,090] jepsen worker 44 - jepsen.util 494 :invoke :write [14 47] INFO [2022-03-13 17:09:52,094] jepsen worker 69 - jepsen.util 609 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,095] jepsen worker 79 - jepsen.util 889 :invoke :write [26 1] INFO [2022-03-13 17:09:52,096] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:09:52,179] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,180] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:52,229] jepsen worker 79 - jepsen.util 889 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,230] jepsen worker 79 - jepsen.util 889 :invoke :write [26 44] INFO [2022-03-13 17:09:52,239] jepsen worker 7 - jepsen.util 457 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,240] jepsen worker 7 - jepsen.util 457 :invoke :write [2 32] INFO [2022-03-13 17:09:52,243] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,243] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,244] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:09:52,245] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:09:52,296] jepsen worker 71 - jepsen.util 431 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,296] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:52,312] jepsen worker 79 - jepsen.util 889 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,313] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:52,314] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,315] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:09:52,354] jepsen worker 13 - jepsen.util 553 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,354] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,355] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,355] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:09:52,355] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:52,356] jepsen worker 80 - jepsen.util 620 :invoke :write [26 35] INFO [2022-03-13 17:09:52,409] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,409] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,409] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:52,410] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:09:52,415] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,416] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:09:52,427] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,428] jepsen worker 79 - jepsen.util 889 :invoke :write [26 7] INFO [2022-03-13 17:09:52,429] jepsen worker 70 - jepsen.util 700 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,430] jepsen worker 70 - jepsen.util 700 :invoke :write [23 17] INFO [2022-03-13 17:09:52,524] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,524] jepsen worker 78 - jepsen.util 438 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,525] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:52,527] jepsen worker 8 - jepsen.util 638 :invoke :write [2 36] INFO [2022-03-13 17:09:52,560] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,561] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:09:52,563] jepsen worker 6 - jepsen.util 546 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,563] jepsen worker 6 - jepsen.util 546 :invoke :write [2 28] INFO [2022-03-13 17:09:52,587] jepsen worker 79 - jepsen.util 889 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,588] jepsen worker 79 - jepsen.util 889 :invoke :write [26 32] INFO [2022-03-13 17:09:52,616] jepsen worker 25 - jepsen.util 385 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,616] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:09:52,617] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:52,617] jepsen worker 22 - jepsen.util 562 :invoke :write [7 21] INFO [2022-03-13 17:09:52,619] jepsen worker 24 - jepsen.util 474 :invoke :write [8 3] INFO [2022-03-13 17:09:52,620] jepsen worker 25 - jepsen.util 385 :invoke :write [8 25] INFO [2022-03-13 17:09:52,638] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,638] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:52,664] jepsen worker 44 - jepsen.util 494 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,664] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,665] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:52,665] jepsen worker 44 - jepsen.util 494 :invoke :write [14 1] INFO [2022-03-13 17:09:52,670] jepsen worker 68 - jepsen.util 1058 :info :write [22 1] :RequestTimeoutException INFO [2022-03-13 17:09:52,671] jepsen worker 68 - jepsen.util 1148 :invoke :write [22 11] INFO [2022-03-13 17:09:52,844] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,844] jepsen worker 79 - jepsen.util 889 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,844] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,844] jepsen worker 79 - jepsen.util 889 :invoke :write [26 19] INFO [2022-03-13 17:09:52,845] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:09:52,845] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:52,848] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,849] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:52,850] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,851] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:52,852] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,853] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:52,854] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,855] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:09:52,875] jepsen worker 52 - jepsen.util 502 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,875] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,876] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:52,876] jepsen worker 13 - jepsen.util 553 :invoke :write [4 32] INFO [2022-03-13 17:09:52,901] jepsen worker 70 - jepsen.util 700 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,901] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:52,917] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.217008797653959}, :ns_1@172.23.105.187 {:progress 0.2923976608187134}, :ns_1@172.23.107.227 {:progress 0.217008797653959}, :ns_1@172.23.107.228 {:progress 0.2294117647058823}} INFO [2022-03-13 17:09:52,917] jepsen worker nemesis - couchbase.util Rebalance check count: 105 INFO [2022-03-13 17:09:52,917] jepsen worker nemesis - couchbase.util Rebalance stuck count: 59 INFO [2022-03-13 17:09:52,940] jepsen worker 20 - jepsen.util 1370 :info :write [6 23] :RequestTimeoutException INFO [2022-03-13 17:09:52,941] jepsen worker 20 - jepsen.util 1460 :invoke :write [6 42] INFO [2022-03-13 17:09:52,956] jepsen worker 27 - jepsen.util 1287 :info :write [9 31] :RequestTimeoutException INFO [2022-03-13 17:09:52,957] jepsen worker 27 - jepsen.util 1377 :invoke :write [9 37] INFO [2022-03-13 17:09:52,962] jepsen worker 79 - jepsen.util 889 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,963] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,963] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:52,964] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:52,965] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,966] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:52,971] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:52,971] jepsen worker 79 - jepsen.util 889 :invoke :write [26 0] INFO [2022-03-13 17:09:53,044] jepsen worker 7 - jepsen.util 457 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,045] jepsen worker 7 - jepsen.util 457 :invoke :write [2 38] INFO [2022-03-13 17:09:53,048] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,049] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,050] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:09:53,050] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:09:53,079] jepsen worker 55 - jepsen.util 1135 :ok :write [18 21] INFO [2022-03-13 17:09:53,080] jepsen worker 55 - jepsen.util 1135 :invoke :write [18 1] INFO [2022-03-13 17:09:53,104] jepsen worker 79 - jepsen.util 889 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,105] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,109] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,110] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,111] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,112] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,113] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,113] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:09:53,126] jepsen worker 6 - jepsen.util 546 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,127] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,127] jepsen worker 6 - jepsen.util 546 :invoke :write [2 23] INFO [2022-03-13 17:09:53,128] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,131] jepsen worker 42 - jepsen.util 402 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,131] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,132] jepsen worker 23 - jepsen.util 563 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,132] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:09:53,132] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:09:53,133] jepsen worker 43 - jepsen.util 403 :invoke :write [14 6] INFO [2022-03-13 17:09:53,135] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:09:53,146] jepsen worker 77 - jepsen.util 707 :invoke :write [25 13] INFO [2022-03-13 17:09:53,151] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,152] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,216] jepsen worker 87 - jepsen.util 1527 :info :write [29 28] :RequestTimeoutException INFO [2022-03-13 17:09:53,217] jepsen worker 87 - jepsen.util 1617 :invoke :write [29 6] INFO [2022-03-13 17:09:53,344] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,344] jepsen worker 80 - jepsen.util 620 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,345] jepsen worker 80 - jepsen.util 620 :invoke :write [26 6] INFO [2022-03-13 17:09:53,345] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,351] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,352] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,355] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,356] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,360] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,360] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,368] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,370] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,370] jepsen worker 44 - jepsen.util 494 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,370] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,370] jepsen worker 79 - jepsen.util 889 :invoke :write [26 49] INFO [2022-03-13 17:09:53,372] jepsen worker 44 - jepsen.util 494 :invoke :write [14 12] INFO [2022-03-13 17:09:53,373] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,373] jepsen worker 71 - jepsen.util 431 :invoke :write [23 25] INFO [2022-03-13 17:09:53,388] jepsen worker 66 - jepsen.util 1236 :ok :read [22 30] INFO [2022-03-13 17:09:53,388] jepsen worker 66 - jepsen.util 1236 :invoke :read [22 nil] INFO [2022-03-13 17:09:53,400] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,400] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,401] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,402] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] INFO [2022-03-13 17:09:53,402] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,403] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,404] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,405] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,406] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,406] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,407] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,408] jepsen worker 69 - jepsen.util 609 :invoke :write [23 18] INFO [2022-03-13 17:09:53,421] jepsen worker 8 - jepsen.util 638 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,422] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,422] jepsen worker 8 - jepsen.util 638 :invoke :write [2 17] INFO [2022-03-13 17:09:53,424] jepsen worker 52 - jepsen.util 502 :invoke :write [17 9] INFO [2022-03-13 17:09:53,459] jepsen worker 83 - jepsen.util 803 :ok :read [27 4] INFO [2022-03-13 17:09:53,460] jepsen worker 83 - jepsen.util 803 :invoke :write [27 40] INFO [2022-03-13 17:09:53,466] jepsen worker 84 - jepsen.util 1434 :ok :read [28 44] INFO [2022-03-13 17:09:53,467] jepsen worker 84 - jepsen.util 1434 :invoke :read [28 nil] INFO [2022-03-13 17:09:53,504] jepsen worker 79 - jepsen.util 889 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,504] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,516] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,517] jepsen worker 79 - jepsen.util 889 :invoke :write [26 10] INFO [2022-03-13 17:09:53,524] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,524] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:09:53,569] jepsen worker 6 - jepsen.util 546 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,569] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,569] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,570] jepsen worker 6 - jepsen.util 546 :invoke :write [2 15] INFO [2022-03-13 17:09:53,574] jepsen worker 69 - jepsen.util 609 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,574] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,624] jepsen worker 79 - jepsen.util 889 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,625] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,626] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,627] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,628] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,628] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:09:53,636] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,636] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,637] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,637] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:53,712] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,712] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,712] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:09:53,713] jepsen worker 13 - jepsen.util 553 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,713] jepsen worker 33 - jepsen.util 663 :invoke :write [11 37] INFO [2022-03-13 17:09:53,713] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:09:53,779] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,779] jepsen worker 79 - jepsen.util 889 :invoke :write [26 39] INFO [2022-03-13 17:09:53,788] jepsen worker 44 - jepsen.util 494 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,789] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:53,824] jepsen worker 33 - jepsen.util 663 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,824] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:53,838] jepsen worker 79 - jepsen.util 889 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,838] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,840] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,840] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,841] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,842] jepsen worker 79 - jepsen.util 889 :invoke :write [26 0] INFO [2022-03-13 17:09:53,877] jepsen worker 7 - jepsen.util 457 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,878] jepsen worker 7 - jepsen.util 457 :invoke :write [2 30] INFO [2022-03-13 17:09:53,923] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.217008797653959}, :ns_1@172.23.105.187 {:progress 0.2923976608187134}, :ns_1@172.23.107.227 {:progress 0.217008797653959}, :ns_1@172.23.107.228 {:progress 0.2294117647058823}} INFO [2022-03-13 17:09:53,923] jepsen worker nemesis - couchbase.util Rebalance check count: 106 INFO [2022-03-13 17:09:53,923] jepsen worker nemesis - couchbase.util Rebalance stuck count: 60 INFO [2022-03-13 17:09:53,925] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,925] jepsen worker 79 - jepsen.util 889 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,925] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,926] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:09:53,927] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,927] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,927] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:53,928] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,928] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,928] jepsen worker 71 - jepsen.util 431 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,928] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,929] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:53,930] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,930] jepsen worker 71 - jepsen.util 431 :invoke :write [23 3] INFO [2022-03-13 17:09:53,932] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,932] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,935] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,935] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:09:53,936] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,937] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:53,938] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,938] jepsen worker 69 - jepsen.util 609 :invoke :write [23 3] INFO [2022-03-13 17:09:53,988] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,989] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:53,990] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:53,991] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:54,024] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,024] jepsen worker 69 - jepsen.util 609 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,025] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:54,025] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:54,180] jepsen worker 74 - jepsen.util 1334 :info :write [24 24] :SyncWriteAmbiguous INFO [2022-03-13 17:09:54,181] jepsen worker 74 - jepsen.util 1424 :invoke :write [24 40] INFO [2022-03-13 17:09:54,212] jepsen worker 45 - jepsen.util 1305 :info :write [15 40] :SyncWriteAmbiguous INFO [2022-03-13 17:09:54,215] jepsen worker 45 - jepsen.util 1395 :invoke :write [15 21] INFO [2022-03-13 17:09:54,680] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,681] jepsen worker 21 - jepsen.util 651 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,681] jepsen worker 79 - jepsen.util 889 :invoke :write [26 3] INFO [2022-03-13 17:09:54,682] jepsen worker 21 - jepsen.util 651 :invoke :write [7 1] INFO [2022-03-13 17:09:54,732] jepsen worker 6 - jepsen.util 546 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,733] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:54,735] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:54,735] jepsen worker 44 - jepsen.util 494 :invoke :write [14 9] INFO [2022-03-13 17:09:54,736] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,736] jepsen worker 70 - jepsen.util 700 :invoke :write [23 10] INFO [2022-03-13 17:09:54,766] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,766] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,767] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:09:54,768] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:54,776] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,776] jepsen worker 69 - jepsen.util 609 :invoke :write [23 18] INFO [2022-03-13 17:09:54,804] jepsen worker 22 - jepsen.util 562 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,804] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:09:54,808] jepsen worker 79 - jepsen.util 889 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,809] jepsen worker 79 - jepsen.util 889 :invoke :write [26 24] INFO [2022-03-13 17:09:54,864] jepsen worker 71 - jepsen.util 431 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,864] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,865] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:54,866] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:54,872] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,873] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:09:54,915] jepsen worker 79 - jepsen.util 889 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,916] jepsen worker 79 - jepsen.util 889 :invoke :write [26 35] INFO [2022-03-13 17:09:54,929] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.217008797653959}, :ns_1@172.23.105.187 {:progress 0.2923976608187134}, :ns_1@172.23.107.227 {:progress 0.217008797653959}, :ns_1@172.23.107.228 {:progress 0.2294117647058823}} INFO [2022-03-13 17:09:54,929] jepsen worker nemesis - couchbase.util Rebalance check count: 107 INFO [2022-03-13 17:09:54,929] jepsen worker nemesis - couchbase.util Rebalance stuck count: 61 INFO [2022-03-13 17:09:54,946] jepsen worker 54 - jepsen.util 954 :info :write [18 32] :RequestTimeoutException INFO [2022-03-13 17:09:54,947] jepsen worker 54 - jepsen.util 1044 :invoke :write [18 12] INFO [2022-03-13 17:09:54,968] jepsen worker 25 - jepsen.util 385 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,968] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,969] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:09:54,970] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:54,971] jepsen worker 69 - jepsen.util 609 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:54,972] jepsen worker 69 - jepsen.util 609 :invoke :write [23 30] INFO [2022-03-13 17:09:55,024] jepsen worker 44 - jepsen.util 494 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,025] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:09:55,034] jepsen worker 79 - jepsen.util 889 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,034] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:09:55,051] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,052] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:55,088] jepsen worker 85 - jepsen.util 1255 :ok :read [28 44] INFO [2022-03-13 17:09:55,089] jepsen worker 85 - jepsen.util 1255 :invoke :write [28 6] INFO [2022-03-13 17:09:55,124] jepsen worker 67 - jepsen.util 1057 :ok :read [22 30] INFO [2022-03-13 17:09:55,124] jepsen worker 67 - jepsen.util 1057 :invoke :write [22 1] INFO [2022-03-13 17:09:55,146] jepsen worker 7 - jepsen.util 457 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,146] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:55,157] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,158] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:55,159] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,160] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:55,161] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,162] jepsen worker 69 - jepsen.util 609 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,162] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,162] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:55,163] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:55,163] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:55,164] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,164] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,165] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,165] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:55,167] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,167] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:55,168] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,168] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:55,169] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:09:55,170] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,170] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:09:55,171] jepsen worker 79 - jepsen.util 889 :invoke :write [26 38] INFO [2022-03-13 17:09:55,178] jepsen worker 56 - jepsen.util 956 :info :write [18 35] :RequestTimeoutException INFO [2022-03-13 17:09:55,178] jepsen worker 56 - jepsen.util 1046 :invoke :read [18 nil] INFO [2022-03-13 17:09:55,187] jepsen worker 70 - jepsen.util 700 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,187] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,187] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:09:55,187] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,188] jepsen worker 44 - jepsen.util 494 :invoke :write [14 34] INFO [2022-03-13 17:09:55,188] jepsen worker 70 - jepsen.util 700 :invoke :write [23 32] INFO [2022-03-13 17:09:55,189] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:55,189] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:55,207] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,207] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,208] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:55,208] jepsen worker 6 - jepsen.util 546 :invoke :write [2 22] INFO [2022-03-13 17:09:55,272] jepsen worker 79 - jepsen.util 889 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,273] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:55,274] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,275] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:55,276] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,277] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:55,278] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,278] jepsen worker 79 - jepsen.util 889 :invoke :write [26 39] INFO [2022-03-13 17:09:55,281] jepsen worker 52 - jepsen.util 502 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,282] jepsen worker 52 - jepsen.util 502 :invoke :write [17 23] INFO [2022-03-13 17:09:55,289] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,290] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:09:55,326] jepsen worker 28 - jepsen.util 1108 :info :write [9 22] :RequestTimeoutException INFO [2022-03-13 17:09:55,327] jepsen worker 28 - jepsen.util 1198 :invoke :read [9 nil] INFO [2022-03-13 17:09:55,363] jepsen worker 28 - jepsen.util 1198 :ok :read [9 37] INFO [2022-03-13 17:09:55,364] jepsen worker 28 - jepsen.util 1198 :invoke :read [9 nil] INFO [2022-03-13 17:09:55,374] jepsen worker 44 - jepsen.util 494 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,375] jepsen worker 44 - jepsen.util 494 :invoke :write [14 11] INFO [2022-03-13 17:09:55,375] jepsen worker 79 - jepsen.util 889 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,376] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:09:55,380] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,381] jepsen worker 69 - jepsen.util 609 :invoke :write [23 2] INFO [2022-03-13 17:09:55,389] jepsen worker 50 - jepsen.util 1130 :info :write [16 13] :RequestTimeoutException INFO [2022-03-13 17:09:55,391] jepsen worker 50 - jepsen.util 1220 :invoke :write [16 26] INFO [2022-03-13 17:09:55,396] jepsen worker 8 - jepsen.util 638 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,396] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:09:55,396] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,396] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:09:55,397] jepsen worker 25 - jepsen.util 385 :invoke :write [8 48] INFO [2022-03-13 17:09:55,398] jepsen worker 13 - jepsen.util 553 :invoke :write [4 4] INFO [2022-03-13 17:09:55,399] jepsen worker 8 - jepsen.util 638 :invoke :write [2 49] INFO [2022-03-13 17:09:55,400] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:09:55,442] jepsen worker 28 - jepsen.util 1198 :ok :read [9 37] INFO [2022-03-13 17:09:55,443] jepsen worker 28 - jepsen.util 1198 :invoke :read [9 nil] INFO [2022-03-13 17:09:55,445] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,446] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:09:55,471] jepsen worker 28 - jepsen.util 1198 :ok :read [9 37] INFO [2022-03-13 17:09:55,472] jepsen worker 28 - jepsen.util 1198 :invoke :read [9 nil] INFO [2022-03-13 17:09:55,485] jepsen worker 70 - jepsen.util 700 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,485] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,486] jepsen worker 70 - jepsen.util 700 :invoke :write [23 40] INFO [2022-03-13 17:09:55,486] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:55,501] jepsen worker 28 - jepsen.util 1198 :ok :read [9 37] INFO [2022-03-13 17:09:55,502] jepsen worker 28 - jepsen.util 1198 :invoke :write [9 34] INFO [2022-03-13 17:09:55,510] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,512] jepsen worker 79 - jepsen.util 889 :invoke :write [26 18] INFO [2022-03-13 17:09:55,534] jepsen worker 69 - jepsen.util 609 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,534] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] WARN [2022-03-13 17:09:55,582] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9891918,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10071472,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":9128806,"encode_duration_us":20655,"operation_id":"0x3092","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10002026},{"operation_name":"upsert","last_dispatch_duration_us":9643195,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10638244,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":1276349,"encode_duration_us":19051,"operation_id":"0x3091","timeout_ms":10000,"last_server_duration_us":565821,"total_duration_us":10002350},{"operation_name":"upsert","last_dispatch_duration_us":9497413,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10732440,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":238823,"encode_duration_us":26976,"operation_id":"0x3093","timeout_ms":10000,"last_server_duration_us":67485,"total_duration_us":10005539},{"operation_name":"upsert","last_dispatch_duration_us":23328972,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23328972,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9143767,"encode_duration_us":17606,"operation_id":"0x2c85","timeout_ms":10000,"last_server_duration_us":9143767,"total_duration_us":10010115}],"total_count":4}} WARN [2022-03-13 17:09:55,583] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18385792,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":18385792,"total_server_duration_us":49,"operation_id":"0x2e4b","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":18388005},{"operation_name":"get","last_dispatch_duration_us":13682582,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13682582,"total_server_duration_us":43,"operation_id":"0x2ff6","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":13682991},{"operation_name":"upsert","last_dispatch_duration_us":13300522,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13300522,"total_server_duration_us":73410,"encode_duration_us":18036,"operation_id":"0x3024","timeout_ms":10000,"last_server_duration_us":73410,"total_duration_us":10010262},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17234,"operation_id":"0x31f7","timeout_ms":10000,"total_duration_us":10007285},{"operation_name":"upsert","last_dispatch_duration_us":51540,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":374426,"total_server_duration_us":242262,"encode_duration_us":16643,"operation_id":"0x30a2","timeout_ms":10000,"last_server_duration_us":37550,"total_duration_us":10007071},{"operation_name":"upsert","last_dispatch_duration_us":9497413,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10732440,"total_server_duration_us":238823,"encode_duration_us":26976,"operation_id":"0x3093","timeout_ms":10000,"last_server_duration_us":67485,"total_duration_us":10005539},{"operation_name":"upsert","last_dispatch_duration_us":31953,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1073096,"total_server_duration_us":616331,"encode_duration_us":17579,"operation_id":"0x30c8","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10004908},{"operation_name":"upsert","last_dispatch_duration_us":9661754,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9661754,"total_server_duration_us":97329,"encode_duration_us":30819,"operation_id":"0x3155","timeout_ms":10000,"last_server_duration_us":97329,"total_duration_us":10004380},{"operation_name":"upsert","last_dispatch_duration_us":9643195,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10638244,"total_server_duration_us":1276349,"encode_duration_us":19051,"operation_id":"0x3091","timeout_ms":10000,"last_server_duration_us":565821,"total_duration_us":10002350},{"operation_name":"upsert","last_dispatch_duration_us":9891918,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10071472,"total_server_duration_us":9128806,"encode_duration_us":20655,"operation_id":"0x3092","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10002026}],"total_count":16}} INFO [2022-03-13 17:09:55,614] jepsen worker 21 - jepsen.util 651 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,615] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,615] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:55,615] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:55,663] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,664] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:09:55,665] jepsen worker 79 - jepsen.util 889 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,666] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:09:55,707] jepsen worker 6 - jepsen.util 546 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,707] jepsen worker 6 - jepsen.util 546 :invoke :write [2 26] INFO [2022-03-13 17:09:55,762] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:55,762] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:09:55,778] jepsen worker 17 - jepsen.util 1007 :info :write [5 45] :RequestTimeoutException INFO [2022-03-13 17:09:55,779] jepsen worker 17 - jepsen.util 1097 :invoke :write [5 23] INFO [2022-03-13 17:09:55,886] jepsen worker 57 - jepsen.util 1227 :info :write [19 19] :RequestTimeoutException INFO [2022-03-13 17:09:55,887] jepsen worker 57 - jepsen.util 1317 :invoke :write [19 19] INFO [2022-03-13 17:09:55,918] jepsen worker 2 - jepsen.util 1262 :info :write [0 38] :RequestTimeoutException INFO [2022-03-13 17:09:55,918] jepsen worker 2 - jepsen.util 1352 :invoke :write [0 42] INFO [2022-03-13 17:09:55,935] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2228739002932552}, :ns_1@172.23.105.187 {:progress 0.2982456140350878}, :ns_1@172.23.107.227 {:progress 0.2228739002932552}, :ns_1@172.23.107.228 {:progress 0.2352941176470589}} INFO [2022-03-13 17:09:55,935] jepsen worker nemesis - couchbase.util Rebalance check count: 108 INFO [2022-03-13 17:09:55,935] jepsen worker nemesis - couchbase.util Rebalance stuck count: 61 INFO [2022-03-13 17:09:55,957] jepsen worker 15 - jepsen.util 1185 :info :write [5 23] :RequestTimeoutException INFO [2022-03-13 17:09:55,958] jepsen worker 15 - jepsen.util 1275 :invoke :read [5 nil] INFO [2022-03-13 17:09:55,978] jepsen worker 0 - jepsen.util 1170 :info :write [0 9] :RequestTimeoutException INFO [2022-03-13 17:09:55,978] jepsen worker 0 - jepsen.util 1260 :invoke :write [0 25] INFO [2022-03-13 17:09:56,096] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,096] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:09:56,107] jepsen worker 24 - jepsen.util 474 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,107] jepsen worker 24 - jepsen.util 474 :invoke :write [8 38] INFO [2022-03-13 17:09:56,116] jepsen worker 59 - jepsen.util 1049 :info :write [19 13] :RequestTimeoutException INFO [2022-03-13 17:09:56,117] jepsen worker 59 - jepsen.util 1139 :invoke :read [19 nil] INFO [2022-03-13 17:09:56,243] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,243] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:09:56,273] jepsen worker 49 - jepsen.util 1039 :ok :read [16 :nil] INFO [2022-03-13 17:09:56,273] jepsen worker 49 - jepsen.util 1039 :invoke :write [16 1] INFO [2022-03-13 17:09:56,282] jepsen worker 60 - jepsen.util 600 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,282] jepsen worker 35 - jepsen.util 665 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,283] jepsen worker 35 - jepsen.util 665 :invoke :write [11 35] INFO [2022-03-13 17:09:56,283] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:09:56,307] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,308] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:09:56,384] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,384] jepsen worker 44 - jepsen.util 494 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,385] jepsen worker 44 - jepsen.util 494 :invoke :write [14 5] INFO [2022-03-13 17:09:56,385] jepsen worker 71 - jepsen.util 431 :invoke :write [23 23] INFO [2022-03-13 17:09:56,416] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,416] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:56,438] jepsen worker 1 - jepsen.util 991 :info :write [0 5] :RequestTimeoutException INFO [2022-03-13 17:09:56,438] jepsen worker 1 - jepsen.util 1081 :invoke :read [0 nil] INFO [2022-03-13 17:09:56,459] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,460] jepsen worker 79 - jepsen.util 889 :invoke :write [26 38] INFO [2022-03-13 17:09:56,482] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:09:56,482] jepsen worker 52 - jepsen.util 502 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,482] jepsen worker 52 - jepsen.util 502 :invoke :write [17 12] INFO [2022-03-13 17:09:56,483] jepsen worker 22 - jepsen.util 562 :invoke :write [7 32] INFO [2022-03-13 17:09:56,540] jepsen worker 70 - jepsen.util 700 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,541] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:56,574] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,574] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,575] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:56,575] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:56,577] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,577] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,577] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:56,578] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:09:56,578] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,579] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:56,580] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,581] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:09:56,601] jepsen worker 8 - jepsen.util 638 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,602] jepsen worker 8 - jepsen.util 638 :invoke :write [2 8] INFO [2022-03-13 17:09:56,621] jepsen worker 79 - jepsen.util 889 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,622] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:56,624] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,625] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:09:56,749] jepsen worker 39 - jepsen.util 1209 :ok :read [13 39] INFO [2022-03-13 17:09:56,750] jepsen worker 39 - jepsen.util 1209 :invoke :read [13 nil] INFO [2022-03-13 17:09:56,755] jepsen worker 86 - jepsen.util 1076 :ok :read [28 44] INFO [2022-03-13 17:09:56,775] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,790] jepsen worker 59 - jepsen.util 1139 :ok :read [19 6] INFO [2022-03-13 17:09:56,798] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,813] jepsen worker 69 - jepsen.util 609 :invoke :write [23 5] INFO [2022-03-13 17:09:56,813] jepsen worker 59 - jepsen.util 1139 :invoke :read [19 nil] INFO [2022-03-13 17:09:56,813] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:56,801] jepsen worker 6 - jepsen.util 546 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,810] jepsen worker 37 - jepsen.util 1207 :info :write [12 28] :SyncWriteAmbiguous INFO [2022-03-13 17:09:56,813] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:09:56,813] jepsen worker 37 - jepsen.util 1297 :invoke :write [12 13] INFO [2022-03-13 17:09:56,816] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,816] jepsen worker 79 - jepsen.util 889 :invoke :write [26 28] INFO [2022-03-13 17:09:56,813] jepsen worker 86 - jepsen.util 1076 :invoke :read [28 nil] INFO [2022-03-13 17:09:56,855] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,856] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:56,884] jepsen worker 58 - jepsen.util 1048 :info :write [19 46] :RequestTimeoutException INFO [2022-03-13 17:09:56,884] jepsen worker 58 - jepsen.util 1138 :invoke :read [19 nil] INFO [2022-03-13 17:09:56,886] jepsen worker 82 - jepsen.util 1162 :info :write [27 40] :RequestTimeoutException INFO [2022-03-13 17:09:56,887] jepsen worker 82 - jepsen.util 1252 :invoke :read [27 nil] INFO [2022-03-13 17:09:56,897] jepsen worker 39 - jepsen.util 1209 :ok :read [13 39] INFO [2022-03-13 17:09:56,897] jepsen worker 39 - jepsen.util 1209 :invoke :read [13 nil] INFO [2022-03-13 17:09:56,912] jepsen worker 25 - jepsen.util 385 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,912] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:09:56,912] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,912] jepsen worker 25 - jepsen.util 385 :invoke :write [8 6] INFO [2022-03-13 17:09:56,913] jepsen worker 21 - jepsen.util 651 :invoke :write [7 25] INFO [2022-03-13 17:09:56,913] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:56,916] jepsen worker 59 - jepsen.util 1139 :ok :read [19 6] INFO [2022-03-13 17:09:56,917] jepsen worker 59 - jepsen.util 1139 :invoke :read [19 nil] INFO [2022-03-13 17:09:56,930] jepsen worker 86 - jepsen.util 1076 :ok :read [28 44] INFO [2022-03-13 17:09:56,931] jepsen worker 86 - jepsen.util 1076 :invoke :read [28 nil] INFO [2022-03-13 17:09:56,942] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2228739002932552}, :ns_1@172.23.105.187 {:progress 0.2982456140350878}, :ns_1@172.23.107.227 {:progress 0.2228739002932552}, :ns_1@172.23.107.228 {:progress 0.2352941176470589}} INFO [2022-03-13 17:09:56,942] jepsen worker nemesis - couchbase.util Rebalance check count: 109 INFO [2022-03-13 17:09:56,942] jepsen worker nemesis - couchbase.util Rebalance stuck count: 62 INFO [2022-03-13 17:09:56,956] jepsen worker 69 - jepsen.util 609 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,956] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,956] jepsen worker 81 - jepsen.util 1341 :info :write [27 35] :RequestTimeoutException INFO [2022-03-13 17:09:56,957] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:56,957] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:09:56,962] jepsen worker 79 - jepsen.util 889 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,968] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,971] jepsen worker 81 - jepsen.util 1431 :invoke :write [27 46] INFO [2022-03-13 17:09:56,971] jepsen worker 79 - jepsen.util 889 :invoke :write [26 42] INFO [2022-03-13 17:09:56,973] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:56,974] jepsen worker 39 - jepsen.util 1209 :ok :read [13 39] INFO [2022-03-13 17:09:56,975] jepsen worker 39 - jepsen.util 1209 :invoke :write [13 2] INFO [2022-03-13 17:09:56,975] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,975] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:09:56,989] jepsen worker 44 - jepsen.util 494 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:56,991] jepsen worker 44 - jepsen.util 494 :invoke :write [14 38] INFO [2022-03-13 17:09:57,005] jepsen worker 59 - jepsen.util 1139 :ok :read [19 6] INFO [2022-03-13 17:09:57,007] jepsen worker 59 - jepsen.util 1139 :invoke :write [19 17] INFO [2022-03-13 17:09:57,015] jepsen worker 86 - jepsen.util 1076 :ok :read [28 44] INFO [2022-03-13 17:09:57,015] jepsen worker 86 - jepsen.util 1076 :invoke :read [28 nil] INFO [2022-03-13 17:09:57,049] jepsen worker 13 - jepsen.util 553 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,050] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:09:57,077] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,077] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:09:57,080] jepsen worker 79 - jepsen.util 889 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,080] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,082] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,082] jepsen worker 79 - jepsen.util 889 :invoke :write [26 44] INFO [2022-03-13 17:09:57,083] jepsen worker 86 - jepsen.util 1076 :ok :read [28 44] WARN [2022-03-13 17:09:57,083] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9092819,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18427063,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9114072,"encode_duration_us":19341,"operation_id":"0x2fcb","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10003895},{"operation_name":"upsert","last_dispatch_duration_us":9211235,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9211235,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9023573,"encode_duration_us":17831,"operation_id":"0x3005","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10009619},{"operation_name":"upsert","last_dispatch_duration_us":9183993,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9893974,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9512169,"encode_duration_us":19619,"operation_id":"0x316c","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10006306}],"total_count":3}} INFO [2022-03-13 17:09:57,084] jepsen worker 86 - jepsen.util 1076 :invoke :read [28 nil] INFO [2022-03-13 17:09:57,124] jepsen worker 47 - jepsen.util 1127 :info :write [15 2] :RequestTimeoutException INFO [2022-03-13 17:09:57,125] jepsen worker 47 - jepsen.util 1217 :invoke :read [15 nil] INFO [2022-03-13 17:09:57,136] jepsen worker 71 - jepsen.util 431 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,136] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,136] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,136] jepsen worker 6 - jepsen.util 546 :invoke :write [2 15] INFO [2022-03-13 17:09:57,137] jepsen worker 71 - jepsen.util 431 :invoke :write [23 27] INFO [2022-03-13 17:09:57,137] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,150] jepsen worker 86 - jepsen.util 1076 :ok :read [28 44] INFO [2022-03-13 17:09:57,151] jepsen worker 86 - jepsen.util 1076 :invoke :write [28 18] INFO [2022-03-13 17:09:57,167] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,168] jepsen worker 69 - jepsen.util 609 :invoke :write [23 10] WARN [2022-03-13 17:09:57,184] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18005391,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18070333,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9049547,"encode_duration_us":18912,"operation_id":"0x2e1d","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10006077},{"operation_name":"upsert","last_dispatch_duration_us":10048458,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11389088,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9350313,"encode_duration_us":21670,"operation_id":"0x2e82","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10009487},{"operation_name":"upsert","last_dispatch_duration_us":17664893,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19961151,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9375492,"encode_duration_us":17678,"operation_id":"0x2e81","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10009850}],"total_count":3}} INFO [2022-03-13 17:09:57,186] jepsen worker 9 - jepsen.util 1089 :info :write [3 42] :RequestTimeoutException INFO [2022-03-13 17:09:57,187] jepsen worker 9 - jepsen.util 1179 :invoke :write [3 45] WARN [2022-03-13 17:09:57,190] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":7836934,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7836934,"total_server_duration_us":32,"operation_id":"0x308e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":15556152},{"operation_name":"get","last_dispatch_duration_us":2274491,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":11440915,"total_server_duration_us":175,"operation_id":"0x2fc3","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":11445695},{"operation_name":"upsert","last_dispatch_duration_us":8107764,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11715989,"total_server_duration_us":8083228,"encode_duration_us":38406,"operation_id":"0x302e","timeout_ms":10000,"last_server_duration_us":8042024,"total_duration_us":10008656},{"operation_name":"upsert","last_dispatch_duration_us":214938,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":214938,"total_server_duration_us":23390,"encode_duration_us":18981,"operation_id":"0x32b2","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10007602},{"operation_name":"upsert","last_dispatch_duration_us":30844,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8241090,"total_server_duration_us":444773,"encode_duration_us":19306,"operation_id":"0x3213","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10006059},{"operation_name":"upsert","last_dispatch_duration_us":116203,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7646877,"total_server_duration_us":340080,"encode_duration_us":19266,"operation_id":"0x319f","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10005057},{"operation_name":"upsert","last_dispatch_duration_us":9086911,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9164306,"total_server_duration_us":9048979,"encode_duration_us":18171,"operation_id":"0x316b","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":9185709},{"operation_name":"upsert","last_dispatch_duration_us":9056082,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9056082,"total_server_duration_us":9044797,"encode_duration_us":20309,"operation_id":"0x30c6","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9059721},{"operation_name":"get","last_dispatch_duration_us":7815817,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7815817,"total_server_duration_us":92,"operation_id":"0x32db","timeout_ms":30000,"last_server_duration_us":92,"total_duration_us":7818372},{"operation_name":"get","last_dispatch_duration_us":6688734,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":6688734,"total_server_duration_us":32,"operation_id":"0x333f","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6690084}],"total_count":74}} INFO [2022-03-13 17:09:57,213] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,214] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,215] jepsen worker 24 - jepsen.util 474 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,216] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:57,216] jepsen worker 79 - jepsen.util 889 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,217] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:09:57,270] jepsen worker 69 - jepsen.util 609 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,270] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,271] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:09:57,272] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,276] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,277] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] WARN [2022-03-13 17:09:57,284] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11206657,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16808635,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9350927,"encode_duration_us":17658,"operation_id":"0x2ea2","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10001378},{"operation_name":"upsert","last_dispatch_duration_us":18206065,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18206065,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9049045,"encode_duration_us":23992,"operation_id":"0x3010","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10009408}],"total_count":2}} WARN [2022-03-13 17:09:57,284] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16967775,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":16967775,"total_server_duration_us":43,"operation_id":"0x309d","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":16969133},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18470,"operation_id":"0x334b","timeout_ms":10000,"total_duration_us":10010636},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18728,"operation_id":"0x32f9","timeout_ms":10000,"total_duration_us":10008417},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":30494,"operation_id":"0x3214","timeout_ms":10000,"total_duration_us":10007916},{"operation_name":"upsert","last_dispatch_duration_us":15667657,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":15667657,"total_server_duration_us":21568,"encode_duration_us":20217,"operation_id":"0x30dc","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10007442},{"operation_name":"upsert","last_dispatch_duration_us":37237,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1013292,"total_server_duration_us":607444,"encode_duration_us":16860,"operation_id":"0x31b7","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10007206},{"operation_name":"upsert","last_dispatch_duration_us":8578571,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":12222638,"total_server_duration_us":352081,"encode_duration_us":32620,"operation_id":"0x3022","timeout_ms":10000,"last_server_duration_us":46287,"total_duration_us":10005910},{"operation_name":"upsert","last_dispatch_duration_us":30369,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1042616,"total_server_duration_us":629864,"encode_duration_us":18159,"operation_id":"0x303b","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10005425},{"operation_name":"get","last_dispatch_duration_us":9098491,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9098491,"total_server_duration_us":49,"operation_id":"0x32f7","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9099767},{"operation_name":"upsert","last_dispatch_duration_us":3562158,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":3562158,"total_server_duration_us":97947,"encode_duration_us":21478,"operation_id":"0x331b","timeout_ms":10000,"last_server_duration_us":97947,"total_duration_us":3573923}],"total_count":18}} WARN [2022-03-13 17:09:57,284] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":60864,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9187372,"total_server_duration_us":216204,"encode_duration_us":26547,"operation_id":"0x32e2","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10010846},{"operation_name":"upsert","last_dispatch_duration_us":35097,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9264025,"total_server_duration_us":188988,"encode_duration_us":18174,"operation_id":"0x32f2","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10007202},{"operation_name":"upsert","last_dispatch_duration_us":59682,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8967113,"total_server_duration_us":121034,"encode_duration_us":21563,"operation_id":"0x32ff","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10007103},{"operation_name":"upsert","last_dispatch_duration_us":9183993,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9893974,"total_server_duration_us":9512169,"encode_duration_us":19619,"operation_id":"0x316c","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10006306},{"operation_name":"upsert","last_dispatch_duration_us":79959,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9327060,"total_server_duration_us":173314,"encode_duration_us":25626,"operation_id":"0x32f5","timeout_ms":10000,"last_server_duration_us":26685,"total_duration_us":10004869},{"operation_name":"upsert","last_dispatch_duration_us":69685,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":517705,"total_server_duration_us":107636,"encode_duration_us":19854,"operation_id":"0x32ef","timeout_ms":10000,"last_server_duration_us":11677,"total_duration_us":10004614},{"operation_name":"upsert","last_dispatch_duration_us":43356,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":253581,"total_server_duration_us":142493,"encode_duration_us":17314,"operation_id":"0x32af","timeout_ms":10000,"last_server_duration_us":25712,"total_duration_us":10001423},{"operation_name":"upsert","last_dispatch_duration_us":9156635,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9348295,"total_server_duration_us":9222961,"encode_duration_us":19643,"operation_id":"0x3366","timeout_ms":10000,"last_server_duration_us":9101143,"total_duration_us":9408393},{"operation_name":"upsert","last_dispatch_duration_us":9062235,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9062235,"total_server_duration_us":9047983,"encode_duration_us":18533,"operation_id":"0x32ae","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9067943},{"operation_name":"upsert","last_dispatch_duration_us":4158554,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4158554,"total_server_duration_us":121327,"encode_duration_us":33881,"operation_id":"0x34fc","timeout_ms":10000,"last_server_duration_us":121327,"total_duration_us":4163511}],"total_count":25}} WARN [2022-03-13 17:09:57,285] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13356823,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13356823,"total_server_duration_us":27,"operation_id":"0x3211","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":13357251},{"operation_name":"upsert","last_dispatch_duration_us":9430618,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9430618,"total_server_duration_us":21487,"encode_duration_us":21816,"operation_id":"0x334a","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10010224},{"operation_name":"upsert","last_dispatch_duration_us":106954,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":3225973,"total_server_duration_us":256654,"encode_duration_us":22078,"operation_id":"0x3361","timeout_ms":10000,"last_server_duration_us":76870,"total_duration_us":10009275},{"operation_name":"upsert","last_dispatch_duration_us":33829,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":1289717,"total_server_duration_us":636739,"encode_duration_us":22118,"operation_id":"0x30c7","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10006076},{"operation_name":"upsert","last_dispatch_duration_us":11590756,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11590756,"total_server_duration_us":21895,"encode_duration_us":22101,"operation_id":"0x309c","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10003922},{"operation_name":"upsert","last_dispatch_duration_us":12466366,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":12466366,"total_server_duration_us":22141,"encode_duration_us":29627,"operation_id":"0x302f","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10002558},{"operation_name":"get","last_dispatch_duration_us":8224000,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8224000,"total_server_duration_us":6777,"operation_id":"0x3198","timeout_ms":30000,"last_server_duration_us":6777,"total_duration_us":8226184},{"operation_name":"upsert","last_dispatch_duration_us":6391978,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6391978,"total_server_duration_us":64473,"encode_duration_us":21798,"operation_id":"0x3458","timeout_ms":10000,"last_server_duration_us":64473,"total_duration_us":6396826},{"operation_name":"get","last_dispatch_duration_us":4499658,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":4499658,"total_server_duration_us":7129,"operation_id":"0x32b0","timeout_ms":30000,"last_server_duration_us":7129,"total_duration_us":4501662},{"operation_name":"upsert","last_dispatch_duration_us":604914,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1903602,"total_server_duration_us":345450,"encode_duration_us":18189,"operation_id":"0x3245","timeout_ms":10000,"last_server_duration_us":26507,"total_duration_us":4023403}],"total_count":44}} WARN [2022-03-13 17:09:57,290] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16507038,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16507038,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":7821138,"encode_duration_us":20454,"operation_id":"0x3011","timeout_ms":10000,"last_server_duration_us":7821138,"total_duration_us":10000834},{"operation_name":"upsert","last_dispatch_duration_us":8107764,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11715989,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":8083228,"encode_duration_us":38406,"operation_id":"0x302e","timeout_ms":10000,"last_server_duration_us":8042024,"total_duration_us":10008656}],"total_count":2}} INFO [2022-03-13 17:09:57,319] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,320] jepsen worker 79 - jepsen.util 889 :invoke :write [26 9] INFO [2022-03-13 17:09:57,384] jepsen worker 44 - jepsen.util 494 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,384] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:09:57,384] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":91786,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8541924,"total_server_duration_us":289822,"encode_duration_us":22560,"operation_id":"0x32b1","timeout_ms":10000,"last_server_duration_us":25188,"total_duration_us":10010165},{"operation_name":"upsert","last_dispatch_duration_us":9075067,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10013696,"total_server_duration_us":9590594,"encode_duration_us":19946,"operation_id":"0x3130","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10009504},{"operation_name":"upsert","last_dispatch_duration_us":56733,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8844453,"total_server_duration_us":264806,"encode_duration_us":18762,"operation_id":"0x32a9","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10009494},{"operation_name":"upsert","last_dispatch_duration_us":9195519,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9324131,"total_server_duration_us":9047861,"encode_duration_us":17859,"operation_id":"0x3036","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10008590},{"operation_name":"upsert","last_dispatch_duration_us":9299038,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10753417,"total_server_duration_us":537991,"encode_duration_us":17546,"operation_id":"0x308f","timeout_ms":10000,"last_server_duration_us":475707,"total_duration_us":10005335},{"operation_name":"upsert","last_dispatch_duration_us":9190695,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":11134916,"total_server_duration_us":147566,"encode_duration_us":17684,"operation_id":"0x3068","timeout_ms":10000,"last_server_duration_us":43289,"total_duration_us":10005049},{"operation_name":"upsert","last_dispatch_duration_us":8213672,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8807043,"total_server_duration_us":328580,"encode_duration_us":23836,"operation_id":"0x32ee","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10004578},{"operation_name":"upsert","last_dispatch_duration_us":10553107,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":11008075,"total_server_duration_us":9338018,"encode_duration_us":17200,"operation_id":"0x321c","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002950},{"operation_name":"upsert","last_dispatch_duration_us":7888665,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8343773,"total_server_duration_us":313827,"encode_duration_us":16526,"operation_id":"0x3354","timeout_ms":10000,"last_server_duration_us":33721,"total_duration_us":10002857},{"operation_name":"upsert","last_dispatch_duration_us":10443927,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10443927,"total_server_duration_us":9146967,"encode_duration_us":18231,"operation_id":"0x306a","timeout_ms":10000,"last_server_duration_us":9146967,"total_duration_us":10002674}],"total_count":41}} INFO [2022-03-13 17:09:57,385] jepsen worker 44 - jepsen.util 494 :invoke :write [14 10] INFO [2022-03-13 17:09:57,385] jepsen worker 70 - jepsen.util 700 :invoke :write [23 38] WARN [2022-03-13 17:09:57,386] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9051123,"last_local_id":"A312307A00000006/000000005DE126B4","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9797605,"last_local_socket":"172.23.104.79:33480","total_server_duration_us":9513637,"encode_duration_us":19559,"operation_id":"0x2e3c","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10000724},{"operation_name":"upsert","last_dispatch_duration_us":10443927,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10443927,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":9146967,"encode_duration_us":18231,"operation_id":"0x306a","timeout_ms":10000,"last_server_duration_us":9146967,"total_duration_us":10002674},{"operation_name":"upsert","last_dispatch_duration_us":10553107,"last_local_id":"A312307A00000006/000000005DE126B4","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":11008075,"last_local_socket":"172.23.104.79:33480","total_server_duration_us":9338018,"encode_duration_us":17200,"operation_id":"0x321c","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002950},{"operation_name":"upsert","last_dispatch_duration_us":9190695,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11134916,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":147566,"encode_duration_us":17684,"operation_id":"0x3068","timeout_ms":10000,"last_server_duration_us":43289,"total_duration_us":10005049},{"operation_name":"upsert","last_dispatch_duration_us":9195519,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9324131,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9047861,"encode_duration_us":17859,"operation_id":"0x3036","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10008590},{"operation_name":"upsert","last_dispatch_duration_us":9075067,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10013696,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9590594,"encode_duration_us":19946,"operation_id":"0x3130","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10009504},{"operation_name":"upsert","last_dispatch_duration_us":9299038,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10753417,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":537991,"encode_duration_us":17546,"operation_id":"0x308f","timeout_ms":10000,"last_server_duration_us":475707,"total_duration_us":10005335}],"total_count":7}} INFO [2022-03-13 17:09:57,392] jepsen worker 26 - jepsen.util 476 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,393] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:09:57,396] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,396] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,428] jepsen worker 79 - jepsen.util 889 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,428] jepsen worker 79 - jepsen.util 889 :invoke :write [26 5] INFO [2022-03-13 17:09:57,476] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,476] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,477] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,477] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,479] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,479] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,479] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,480] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,480] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,481] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,481] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,482] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:09:57,482] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,483] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,484] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,484] jepsen worker 69 - jepsen.util 609 :invoke :write [23 6] INFO [2022-03-13 17:09:57,484] jepsen worker 22 - jepsen.util 562 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,485] jepsen worker 22 - jepsen.util 562 :invoke :write [7 1] INFO [2022-03-13 17:09:57,548] jepsen worker 79 - jepsen.util 889 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,549] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:09:57,584] jepsen worker 6 - jepsen.util 546 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,585] jepsen worker 6 - jepsen.util 546 :invoke :write [2 21] INFO [2022-03-13 17:09:57,592] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,593] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:09:57,640] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,641] jepsen worker 79 - jepsen.util 889 :invoke :write [26 7] INFO [2022-03-13 17:09:57,647] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,647] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:09:57,664] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,665] jepsen worker 52 - jepsen.util 502 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,665] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,666] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:09:57,666] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:09:57,667] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:09:57,674] jepsen worker 69 - jepsen.util 609 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,675] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:09:57,676] jepsen worker 40 - jepsen.util 1120 :info :write [13 42] :RequestTimeoutException INFO [2022-03-13 17:09:57,677] jepsen worker 40 - jepsen.util 1210 :invoke :write [13 41] INFO [2022-03-13 17:09:57,714] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,715] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:09:57,745] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,746] jepsen worker 69 - jepsen.util 609 :invoke :write [23 1] INFO [2022-03-13 17:09:57,756] jepsen worker 38 - jepsen.util 1388 :info :write [12 20] :RequestTimeoutException INFO [2022-03-13 17:09:57,757] jepsen worker 38 - jepsen.util 1478 :invoke :write [12 38] INFO [2022-03-13 17:09:57,775] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,775] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:09:57,775] jepsen worker 79 - jepsen.util 889 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,776] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,784] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,784] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,786] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,787] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,788] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,788] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,791] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,791] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,792] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,793] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:09:57,829] jepsen worker 69 - jepsen.util 609 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,829] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,872] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,872] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,877] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,878] jepsen worker 71 - jepsen.util 431 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,878] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:09:57,878] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:57,915] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,915] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,916] jepsen worker 69 - jepsen.util 609 :invoke :write [23 3] INFO [2022-03-13 17:09:57,916] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,921] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,922] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,923] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,924] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,925] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,925] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:57,926] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,927] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:09:57,949] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2287390029325513}, :ns_1@172.23.105.187 {:progress 0.304093567251462}, :ns_1@172.23.107.227 {:progress 0.2287390029325513}, :ns_1@172.23.107.228 {:progress 0.2411764705882353}} INFO [2022-03-13 17:09:57,949] jepsen worker nemesis - couchbase.util Rebalance check count: 110 INFO [2022-03-13 17:09:57,950] jepsen worker nemesis - couchbase.util Rebalance stuck count: 62 INFO [2022-03-13 17:09:57,957] jepsen worker 25 - jepsen.util 385 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,958] jepsen worker 89 - jepsen.util 1349 :info :write [29 0] :RequestTimeoutException INFO [2022-03-13 17:09:57,958] jepsen worker 25 - jepsen.util 385 :invoke :write [8 42] INFO [2022-03-13 17:09:57,959] jepsen worker 89 - jepsen.util 1439 :invoke :read [29 nil] INFO [2022-03-13 17:09:57,988] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,989] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,996] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,997] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:57,998] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:57,999] jepsen worker 79 - jepsen.util 889 :invoke :write [26 14] INFO [2022-03-13 17:09:58,008] jepsen worker 69 - jepsen.util 609 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,008] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:09:58,010] jepsen worker 89 - jepsen.util 1439 :ok :read [29 6] INFO [2022-03-13 17:09:58,010] jepsen worker 89 - jepsen.util 1439 :invoke :read [29 nil] INFO [2022-03-13 17:09:58,039] jepsen worker 89 - jepsen.util 1439 :ok :read [29 6] INFO [2022-03-13 17:09:58,040] jepsen worker 89 - jepsen.util 1439 :invoke :read [29 nil] INFO [2022-03-13 17:09:58,063] jepsen worker 89 - jepsen.util 1439 :ok :read [29 6] INFO [2022-03-13 17:09:58,063] jepsen worker 89 - jepsen.util 1439 :invoke :write [29 15] INFO [2022-03-13 17:09:58,086] jepsen worker 64 - jepsen.util 964 :info :write [21 28] :RequestTimeoutException INFO [2022-03-13 17:09:58,088] jepsen worker 64 - jepsen.util 1054 :invoke :read [21 nil] INFO [2022-03-13 17:09:58,092] jepsen worker 44 - jepsen.util 494 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,092] jepsen worker 44 - jepsen.util 494 :invoke :write [14 8] INFO [2022-03-13 17:09:58,103] jepsen worker 64 - jepsen.util 1054 :ok :read [21 :nil] INFO [2022-03-13 17:09:58,104] jepsen worker 64 - jepsen.util 1054 :invoke :read [21 nil] INFO [2022-03-13 17:09:58,105] jepsen worker 64 - jepsen.util 1054 :ok :read [21 :nil] INFO [2022-03-13 17:09:58,106] jepsen worker 64 - jepsen.util 1054 :invoke :write [21 22] INFO [2022-03-13 17:09:58,149] jepsen worker 21 - jepsen.util 651 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,149] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,149] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:58,150] jepsen worker 21 - jepsen.util 651 :invoke :write [7 42] INFO [2022-03-13 17:09:58,150] jepsen worker 79 - jepsen.util 889 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,151] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,151] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:09:58,152] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,153] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,154] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,156] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,156] jepsen worker 79 - jepsen.util 889 :invoke :write [26 10] INFO [2022-03-13 17:09:58,159] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,159] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:09:58,239] jepsen worker 70 - jepsen.util 700 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,239] jepsen worker 70 - jepsen.util 700 :invoke :write [23 31] INFO [2022-03-13 17:09:58,256] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,256] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,256] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:09:58,257] jepsen worker 69 - jepsen.util 609 :invoke :write [23 42] INFO [2022-03-13 17:09:58,269] jepsen worker 79 - jepsen.util 889 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,270] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,304] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,304] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:09:58,330] jepsen worker 16 - jepsen.util 1186 :info :write [5 27] :RequestTimeoutException INFO [2022-03-13 17:09:58,332] jepsen worker 16 - jepsen.util 1276 :invoke :read [5 nil] INFO [2022-03-13 17:09:58,520] jepsen worker 22 - jepsen.util 562 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,520] jepsen worker 22 - jepsen.util 562 :invoke :write [7 24] INFO [2022-03-13 17:09:58,569] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,569] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,575] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,575] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,576] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,577] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,578] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,578] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,580] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,580] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:09:58,592] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,592] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:09:58,597] jepsen worker 6 - jepsen.util 546 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,598] jepsen worker 6 - jepsen.util 546 :invoke :write [2 32] INFO [2022-03-13 17:09:58,600] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,601] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:58,641] jepsen worker 80 - jepsen.util 620 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,642] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,671] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,672] jepsen worker 79 - jepsen.util 889 :invoke :write [26 27] INFO [2022-03-13 17:09:58,704] jepsen worker 69 - jepsen.util 609 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,705] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:58,708] jepsen worker 44 - jepsen.util 494 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,709] jepsen worker 44 - jepsen.util 494 :invoke :write [14 3] INFO [2022-03-13 17:09:58,786] jepsen worker 79 - jepsen.util 889 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,787] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,788] jepsen worker 36 - jepsen.util 1296 :info :write [12 36] :RequestTimeoutException INFO [2022-03-13 17:09:58,788] jepsen worker 36 - jepsen.util 1386 :invoke :read [12 nil] INFO [2022-03-13 17:09:58,792] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,793] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,794] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,796] jepsen worker 79 - jepsen.util 889 :invoke :write [26 27] INFO [2022-03-13 17:09:58,851] jepsen worker 5 - jepsen.util 1265 :info :write [1 9] :RequestTimeoutException INFO [2022-03-13 17:09:58,851] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,851] jepsen worker 5 - jepsen.util 1355 :invoke :read [1 nil] INFO [2022-03-13 17:09:58,852] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:09:58,852] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,852] jepsen worker 33 - jepsen.util 663 :invoke :write [11 49] INFO [2022-03-13 17:09:58,856] jepsen worker 25 - jepsen.util 385 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,856] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:58,856] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:09:58,857] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:09:58,858] jepsen worker 24 - jepsen.util 474 :invoke :write [8 22] INFO [2022-03-13 17:09:58,858] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:09:58,880] jepsen worker 5 - jepsen.util 1355 :ok :read [1 42] INFO [2022-03-13 17:09:58,881] jepsen worker 5 - jepsen.util 1355 :invoke :read [1 nil] INFO [2022-03-13 17:09:58,909] jepsen worker 5 - jepsen.util 1355 :ok :read [1 42] INFO [2022-03-13 17:09:58,910] jepsen worker 5 - jepsen.util 1355 :invoke :write [1 47] INFO [2022-03-13 17:09:58,921] jepsen worker 19 - jepsen.util 919 :info :write [6 42] :SyncWriteAmbiguous INFO [2022-03-13 17:09:58,921] jepsen worker 19 - jepsen.util 1009 :invoke :write [6 14] INFO [2022-03-13 17:09:58,957] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2287390029325513}, :ns_1@172.23.105.187 {:progress 0.304093567251462}, :ns_1@172.23.107.227 {:progress 0.2287390029325513}, :ns_1@172.23.107.228 {:progress 0.2411764705882353}} INFO [2022-03-13 17:09:58,957] jepsen worker nemesis - couchbase.util Rebalance check count: 111 INFO [2022-03-13 17:09:58,957] jepsen worker nemesis - couchbase.util Rebalance stuck count: 63 INFO [2022-03-13 17:09:59,017] jepsen worker 30 - jepsen.util 1020 :ok :read [10 :nil] INFO [2022-03-13 17:09:59,018] jepsen worker 30 - jepsen.util 1020 :invoke :write [10 44] INFO [2022-03-13 17:09:59,096] jepsen worker 73 - jepsen.util 1243 :ok :read [24 24] INFO [2022-03-13 17:09:59,097] jepsen worker 73 - jepsen.util 1243 :invoke :write [24 4] INFO [2022-03-13 17:09:59,168] jepsen worker 56 - jepsen.util 1046 :ok :read [18 21] INFO [2022-03-13 17:09:59,168] jepsen worker 56 - jepsen.util 1046 :invoke :write [18 45] INFO [2022-03-13 17:09:59,610] jepsen worker 79 - jepsen.util 889 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,610] jepsen worker 79 - jepsen.util 889 :invoke :write [26 4] INFO [2022-03-13 17:09:59,633] jepsen worker 11 - jepsen.util 1001 :info :write [3 1] :SyncWriteAmbiguous INFO [2022-03-13 17:09:59,634] jepsen worker 77 - jepsen.util 707 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,634] jepsen worker 53 - jepsen.util 503 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,635] jepsen worker 53 - jepsen.util 503 :invoke :write [17 45] INFO [2022-03-13 17:09:59,635] jepsen worker 77 - jepsen.util 707 :invoke :write [25 27] INFO [2022-03-13 17:09:59,636] jepsen worker 11 - jepsen.util 1091 :invoke :write [3 28] INFO [2022-03-13 17:09:59,684] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,685] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:09:59,695] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,695] jepsen worker 70 - jepsen.util 700 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,695] jepsen worker 70 - jepsen.util 700 :invoke :write [23 40] INFO [2022-03-13 17:09:59,695] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:09:59,698] jepsen worker 32 - jepsen.util 932 :info :write [10 34] :SyncWriteAmbiguous INFO [2022-03-13 17:09:59,699] jepsen worker 32 - jepsen.util 1022 :invoke :read [10 nil] INFO [2022-03-13 17:09:59,725] jepsen worker 79 - jepsen.util 889 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,725] jepsen worker 79 - jepsen.util 889 :invoke :write [26 15] INFO [2022-03-13 17:09:59,743] jepsen worker 21 - jepsen.util 651 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,744] jepsen worker 21 - jepsen.util 651 :invoke :read [7 nil] INFO [2022-03-13 17:09:59,797] jepsen worker 47 - jepsen.util 1217 :ok :read [15 48] INFO [2022-03-13 17:09:59,798] jepsen worker 47 - jepsen.util 1217 :invoke :read [15 nil] INFO [2022-03-13 17:09:59,834] jepsen worker 79 - jepsen.util 889 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,835] jepsen worker 79 - jepsen.util 889 :invoke :write [26 29] INFO [2022-03-13 17:09:59,845] jepsen worker 6 - jepsen.util 546 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,845] jepsen worker 6 - jepsen.util 546 :invoke :write [2 25] INFO [2022-03-13 17:09:59,847] jepsen worker 33 - jepsen.util 663 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,847] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:09:59,851] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,851] jepsen worker 69 - jepsen.util 609 :invoke :write [23 23] INFO [2022-03-13 17:09:59,894] jepsen worker 51 - jepsen.util 501 :info :write [17 36] :RequestTimeoutException INFO [2022-03-13 17:09:59,894] jepsen worker 76 - jepsen.util 526 :info :write [25 13] :RequestTimeoutException INFO [2022-03-13 17:09:59,895] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:09:59,896] jepsen worker 51 - jepsen.util 591 :invoke :write [17 5] INFO [2022-03-13 17:09:59,898] jepsen worker 47 - jepsen.util 1217 :ok :read [15 48] INFO [2022-03-13 17:09:59,898] jepsen worker 47 - jepsen.util 1217 :invoke :write [15 11] INFO [2022-03-13 17:09:59,902] jepsen worker 8 - jepsen.util 638 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,903] jepsen worker 8 - jepsen.util 638 :invoke :write [2 17] INFO [2022-03-13 17:09:59,903] jepsen worker 18 - jepsen.util 1008 :info :write [6 22] :RequestTimeoutException INFO [2022-03-13 17:09:59,903] jepsen worker 61 - jepsen.util 421 :info :write [20 37] :RequestTimeoutException INFO [2022-03-13 17:09:59,903] jepsen worker 75 - jepsen.util 345 :info :write [25 14] :RequestTimeoutException INFO [2022-03-13 17:09:59,904] jepsen worker 61 - jepsen.util 511 :invoke :read [20 nil] INFO [2022-03-13 17:09:59,905] jepsen worker 18 - jepsen.util 1098 :invoke :read [6 nil] INFO [2022-03-13 17:09:59,906] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:09:59,948] jepsen worker 79 - jepsen.util 889 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,949] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,950] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:09:59,951] jepsen worker 76 - jepsen.util 616 :invoke :write [25 21] INFO [2022-03-13 17:09:59,963] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,963] jepsen worker 79 - jepsen.util 889 :invoke :write [26 2] INFO [2022-03-13 17:09:59,974] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2346041055718475}, :ns_1@172.23.105.187 {:progress 0.3099415204678363}, :ns_1@172.23.107.227 {:progress 0.2346041055718475}, :ns_1@172.23.107.228 {:progress 0.2470588235294118}} INFO [2022-03-13 17:09:59,974] jepsen worker nemesis - couchbase.util Rebalance check count: 112 INFO [2022-03-13 17:09:59,974] jepsen worker nemesis - couchbase.util Rebalance stuck count: 63 INFO [2022-03-13 17:09:59,994] jepsen worker 34 - jepsen.util 664 :info :write [11 31] :RequestTimeoutException INFO [2022-03-13 17:09:59,995] jepsen worker 34 - jepsen.util 754 :invoke :write [11 11] INFO [2022-03-13 17:09:59,996] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,997] jepsen worker 33 - jepsen.util 663 :invoke :write [11 45] INFO [2022-03-13 17:09:59,999] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:09:59,999] jepsen worker 44 - jepsen.util 494 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,000] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:00,000] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:00,030] jepsen worker 72 - jepsen.util 1062 :info :write [24 46] :RequestTimeoutException INFO [2022-03-13 17:10:00,031] jepsen worker 72 - jepsen.util 1152 :invoke :read [24 nil] INFO [2022-03-13 17:10:00,049] jepsen worker 76 - jepsen.util 616 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,049] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:00,056] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,056] jepsen worker 22 - jepsen.util 562 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,056] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,056] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,056] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,056] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:10:00,056] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,057] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:00,058] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:00,059] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:00,059] jepsen worker 13 - jepsen.util 553 :invoke :write [4 14] INFO [2022-03-13 17:10:00,060] jepsen worker 7 - jepsen.util 457 :invoke :write [2 34] INFO [2022-03-13 17:10:00,061] jepsen worker 52 - jepsen.util 502 :invoke :write [17 13] INFO [2022-03-13 17:10:00,061] jepsen worker 25 - jepsen.util 385 :invoke :write [8 31] INFO [2022-03-13 17:10:00,091] jepsen worker 69 - jepsen.util 609 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,091] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,091] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:00,092] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:00,126] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,126] jepsen worker 79 - jepsen.util 889 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,127] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:10:00,131] jepsen worker 76 - jepsen.util 616 :invoke :write [25 13] INFO [2022-03-13 17:10:00,146] jepsen worker 70 - jepsen.util 700 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,146] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,147] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,147] jepsen worker 33 - jepsen.util 663 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,147] jepsen worker 69 - jepsen.util 609 :invoke :write [23 23] INFO [2022-03-13 17:10:00,148] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:10:00,148] jepsen worker 70 - jepsen.util 700 :invoke :write [23 1] INFO [2022-03-13 17:10:00,150] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:10:00,175] jepsen worker 24 - jepsen.util 474 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,176] jepsen worker 24 - jepsen.util 474 :invoke :write [8 38] INFO [2022-03-13 17:10:00,177] jepsen worker 21 - jepsen.util 651 :ok :read [7 :nil] INFO [2022-03-13 17:10:00,177] jepsen worker 21 - jepsen.util 651 :invoke :write [7 42] INFO [2022-03-13 17:10:00,184] jepsen worker 61 - jepsen.util 511 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,185] jepsen worker 61 - jepsen.util 511 :invoke :write [20 28] INFO [2022-03-13 17:10:00,223] jepsen worker 12 - jepsen.util 642 :info :write [4 11] :RequestTimeoutException INFO [2022-03-13 17:10:00,224] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:00,248] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,248] jepsen worker 79 - jepsen.util 889 :invoke :write [26 22] INFO [2022-03-13 17:10:00,261] jepsen worker 6 - jepsen.util 546 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,261] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:00,261] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,262] jepsen worker 6 - jepsen.util 546 :invoke :write [2 25] INFO [2022-03-13 17:10:00,263] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:00,263] jepsen worker 44 - jepsen.util 494 :invoke :write [14 16] INFO [2022-03-13 17:10:00,274] jepsen worker 10 - jepsen.util 1000 :info :write [3 38] :RequestTimeoutException INFO [2022-03-13 17:10:00,275] jepsen worker 10 - jepsen.util 1090 :invoke :write [3 11] INFO [2022-03-13 17:10:00,301] jepsen worker 69 - jepsen.util 609 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,302] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:00,308] jepsen worker 76 - jepsen.util 616 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,309] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:00,326] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:00,326] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,326] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,326] jepsen worker 22 - jepsen.util 562 :invoke :write [7 17] INFO [2022-03-13 17:10:00,327] jepsen worker 80 - jepsen.util 620 :invoke :write [26 0] INFO [2022-03-13 17:10:00,327] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:00,335] jepsen worker 72 - jepsen.util 1152 :ok :read [24 24] INFO [2022-03-13 17:10:00,335] jepsen worker 72 - jepsen.util 1152 :invoke :read [24 nil] INFO [2022-03-13 17:10:00,372] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,372] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:10:00,500] jepsen worker 72 - jepsen.util 1152 :ok :read [24 24] INFO [2022-03-13 17:10:00,501] jepsen worker 72 - jepsen.util 1152 :invoke :write [24 47] INFO [2022-03-13 17:10:00,576] jepsen worker 31 - jepsen.util 1201 :info :write [10 28] :RequestTimeoutException INFO [2022-03-13 17:10:00,577] jepsen worker 31 - jepsen.util 1291 :invoke :read [10 nil] INFO [2022-03-13 17:10:00,600] jepsen worker 46 - jepsen.util 1216 :info :write [15 46] :RequestTimeoutException INFO [2022-03-13 17:10:00,601] jepsen worker 46 - jepsen.util 1306 :invoke :write [15 40] INFO [2022-03-13 17:10:00,700] jepsen worker 63 - jepsen.util 1143 :info :write [21 5] :RequestTimeoutException INFO [2022-03-13 17:10:00,701] jepsen worker 63 - jepsen.util 1233 :invoke :write [21 2] INFO [2022-03-13 17:10:00,779] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,779] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:10:00,810] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,810] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:00,821] jepsen worker 3 - jepsen.util 1083 :info :write [1 22] :RequestTimeoutException INFO [2022-03-13 17:10:00,822] jepsen worker 3 - jepsen.util 1173 :invoke :read [1 nil] INFO [2022-03-13 17:10:00,930] jepsen worker 79 - jepsen.util 889 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,931] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:00,965] jepsen worker 43 - jepsen.util 403 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,965] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:00,966] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:00,966] jepsen worker 33 - jepsen.util 663 :invoke :write [11 9] INFO [2022-03-13 17:10:00,981] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2346041055718475}, :ns_1@172.23.105.187 {:progress 0.3099415204678363}, :ns_1@172.23.107.227 {:progress 0.2346041055718475}, :ns_1@172.23.107.228 {:progress 0.2470588235294118}} INFO [2022-03-13 17:10:00,981] jepsen worker nemesis - couchbase.util Rebalance check count: 113 INFO [2022-03-13 17:10:00,981] jepsen worker nemesis - couchbase.util Rebalance stuck count: 64 INFO [2022-03-13 17:10:01,144] jepsen worker 25 - jepsen.util 385 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,144] jepsen worker 25 - jepsen.util 385 :invoke :write [8 27] INFO [2022-03-13 17:10:01,168] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,168] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:01,168] jepsen worker 60 - jepsen.util 600 :invoke :write [20 9] INFO [2022-03-13 17:10:01,169] jepsen worker 23 - jepsen.util 563 :invoke :write [7 11] INFO [2022-03-13 17:10:01,192] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,192] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,192] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:01,193] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:01,196] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,197] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:01,344] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,344] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,345] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:01,345] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] INFO [2022-03-13 17:10:01,346] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,347] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:01,348] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,349] jepsen worker 79 - jepsen.util 889 :invoke :write [26 8] INFO [2022-03-13 17:10:01,399] jepsen worker 44 - jepsen.util 494 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,400] jepsen worker 44 - jepsen.util 494 :invoke :write [14 9] INFO [2022-03-13 17:10:01,547] jepsen worker 65 - jepsen.util 1235 :info :write [21 41] :RequestTimeoutException INFO [2022-03-13 17:10:01,548] jepsen worker 65 - jepsen.util 1325 :invoke :read [21 nil] INFO [2022-03-13 17:10:01,548] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,548] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:01,553] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,554] jepsen worker 76 - jepsen.util 616 :invoke :write [25 24] INFO [2022-03-13 17:10:01,589] jepsen worker 7 - jepsen.util 457 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,590] jepsen worker 7 - jepsen.util 457 :invoke :write [2 11] INFO [2022-03-13 17:10:01,880] jepsen worker 4 - jepsen.util 1264 :info :write [1 42] :RequestTimeoutException INFO [2022-03-13 17:10:01,880] jepsen worker 4 - jepsen.util 1354 :invoke :read [1 nil] INFO [2022-03-13 17:10:01,941] jepsen worker 4 - jepsen.util 1354 :ok :read [1 42] INFO [2022-03-13 17:10:01,942] jepsen worker 4 - jepsen.util 1354 :invoke :write [1 16] INFO [2022-03-13 17:10:01,966] jepsen worker 33 - jepsen.util 663 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,966] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,966] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,968] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:01,969] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:10:01,969] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:01,970] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,971] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:01,971] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,972] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,972] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:10:01,973] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:10:01,975] jepsen worker 79 - jepsen.util 889 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:01,976] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:01,989] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2346041055718475}, :ns_1@172.23.105.187 {:progress 0.3099415204678363}, :ns_1@172.23.107.227 {:progress 0.2346041055718475}, :ns_1@172.23.107.228 {:progress 0.2470588235294118}} INFO [2022-03-13 17:10:01,989] jepsen worker nemesis - couchbase.util Rebalance check count: 114 INFO [2022-03-13 17:10:01,989] jepsen worker nemesis - couchbase.util Rebalance stuck count: 65 INFO [2022-03-13 17:10:02,030] jepsen worker 76 - jepsen.util 616 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,030] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:02,032] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,032] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:02,036] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,036] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,037] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:02,037] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:02,134] jepsen worker 14 - jepsen.util 194 :info :write [4 19] :RequestTimeoutException INFO [2022-03-13 17:10:02,135] jepsen worker 14 - jepsen.util 284 :invoke :write [4 36] INFO [2022-03-13 17:10:02,176] jepsen worker 77 - jepsen.util 707 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,177] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:02,450] jepsen worker 48 - jepsen.util 1128 :ok :read [16 :nil] INFO [2022-03-13 17:10:02,451] jepsen worker 48 - jepsen.util 1128 :invoke :read [16 nil] INFO [2022-03-13 17:10:02,472] jepsen worker 66 - jepsen.util 1236 :ok :read [22 30] INFO [2022-03-13 17:10:02,473] jepsen worker 66 - jepsen.util 1236 :invoke :read [22 nil] INFO [2022-03-13 17:10:02,483] jepsen worker 88 - jepsen.util 1078 :ok :read [29 6] INFO [2022-03-13 17:10:02,484] jepsen worker 88 - jepsen.util 1078 :invoke :read [29 nil] INFO [2022-03-13 17:10:02,505] jepsen worker 84 - jepsen.util 1434 :ok :read [28 44] INFO [2022-03-13 17:10:02,506] jepsen worker 84 - jepsen.util 1434 :invoke :read [28 nil] INFO [2022-03-13 17:10:02,521] jepsen worker 88 - jepsen.util 1078 :ok :read [29 6] INFO [2022-03-13 17:10:02,523] jepsen worker 88 - jepsen.util 1078 :invoke :read [29 nil] INFO [2022-03-13 17:10:02,547] jepsen worker 88 - jepsen.util 1078 :ok :read [29 6] INFO [2022-03-13 17:10:02,548] jepsen worker 88 - jepsen.util 1078 :invoke :read [29 nil] INFO [2022-03-13 17:10:02,555] jepsen worker 48 - jepsen.util 1128 :ok :read [16 :nil] INFO [2022-03-13 17:10:02,555] jepsen worker 48 - jepsen.util 1128 :invoke :write [16 31] INFO [2022-03-13 17:10:02,575] jepsen worker 88 - jepsen.util 1078 :ok :read [29 6] INFO [2022-03-13 17:10:02,575] jepsen worker 88 - jepsen.util 1078 :invoke :write [29 23] INFO [2022-03-13 17:10:02,582] jepsen worker 66 - jepsen.util 1236 :ok :read [22 30] INFO [2022-03-13 17:10:02,583] jepsen worker 66 - jepsen.util 1236 :invoke :read [22 nil] INFO [2022-03-13 17:10:02,615] jepsen worker 84 - jepsen.util 1434 :ok :read [28 44] INFO [2022-03-13 17:10:02,615] jepsen worker 84 - jepsen.util 1434 :invoke :read [28 nil] INFO [2022-03-13 17:10:02,693] jepsen worker 68 - jepsen.util 1148 :info :write [22 11] :RequestTimeoutException INFO [2022-03-13 17:10:02,694] jepsen worker 68 - jepsen.util 1238 :invoke :read [22 nil] INFO [2022-03-13 17:10:02,712] jepsen worker 66 - jepsen.util 1236 :ok :read [22 30] INFO [2022-03-13 17:10:02,713] jepsen worker 66 - jepsen.util 1236 :invoke :write [22 12] INFO [2022-03-13 17:10:02,777] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,777] jepsen worker 76 - jepsen.util 616 :invoke :write [25 28] INFO [2022-03-13 17:10:02,782] jepsen worker 84 - jepsen.util 1434 :ok :read [28 44] INFO [2022-03-13 17:10:02,783] jepsen worker 84 - jepsen.util 1434 :invoke :read [28 nil] INFO [2022-03-13 17:10:02,784] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,784] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:02,877] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,877] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:10:02,880] jepsen worker 51 - jepsen.util 591 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,880] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,881] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:02,881] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:02,883] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:02,883] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:02,891] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,892] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:02,916] jepsen worker 84 - jepsen.util 1434 :ok :read [28 44] INFO [2022-03-13 17:10:02,917] jepsen worker 84 - jepsen.util 1434 :invoke :write [28 2] INFO [2022-03-13 17:10:02,945] jepsen worker 76 - jepsen.util 616 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,946] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:02,966] jepsen worker 20 - jepsen.util 1460 :info :write [6 42] :RequestTimeoutException INFO [2022-03-13 17:10:02,966] jepsen worker 20 - jepsen.util 1550 :invoke :write [6 9] INFO [2022-03-13 17:10:02,977] jepsen worker 27 - jepsen.util 1377 :info :write [9 37] :RequestTimeoutException INFO [2022-03-13 17:10:02,977] jepsen worker 27 - jepsen.util 1467 :invoke :read [9 nil] INFO [2022-03-13 17:10:02,977] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,977] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,977] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,978] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:02,979] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:02,979] jepsen worker 75 - jepsen.util 435 :invoke :write [25 21] INFO [2022-03-13 17:10:02,984] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,984] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,984] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:02,985] jepsen worker 69 - jepsen.util 609 :invoke :write [23 24] INFO [2022-03-13 17:10:02,988] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:02,989] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:10:02,997] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2346041055718475}, :ns_1@172.23.105.187 {:progress 0.3099415204678363}, :ns_1@172.23.107.227 {:progress 0.2346041055718475}, :ns_1@172.23.107.228 {:progress 0.2470588235294118}} INFO [2022-03-13 17:10:02,997] jepsen worker nemesis - couchbase.util Rebalance check count: 115 INFO [2022-03-13 17:10:02,997] jepsen worker nemesis - couchbase.util Rebalance stuck count: 66 INFO [2022-03-13 17:10:02,999] jepsen worker 60 - jepsen.util 600 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,000] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:10:03,015] jepsen worker 27 - jepsen.util 1467 :ok :read [9 31] INFO [2022-03-13 17:10:03,016] jepsen worker 27 - jepsen.util 1467 :invoke :write [9 26] INFO [2022-03-13 17:10:03,048] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,048] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,049] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:03,049] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] INFO [2022-03-13 17:10:03,050] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,050] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:03,051] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,052] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:03,053] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,053] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:03,054] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,054] jepsen worker 79 - jepsen.util 889 :invoke :write [26 2] INFO [2022-03-13 17:10:03,085] jepsen worker 75 - jepsen.util 435 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,086] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:03,104] jepsen worker 55 - jepsen.util 1135 :info :write [18 1] :RequestTimeoutException INFO [2022-03-13 17:10:03,105] jepsen worker 55 - jepsen.util 1225 :invoke :write [18 5] INFO [2022-03-13 17:10:03,124] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:03,124] jepsen worker 23 - jepsen.util 563 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,125] jepsen worker 42 - jepsen.util 402 :invoke :write [14 8] INFO [2022-03-13 17:10:03,125] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:03,127] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,128] jepsen worker 77 - jepsen.util 707 :invoke :write [25 41] INFO [2022-03-13 17:10:03,129] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,129] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,129] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:03,130] jepsen worker 51 - jepsen.util 591 :invoke :write [17 6] INFO [2022-03-13 17:10:03,131] jepsen worker 43 - jepsen.util 403 :invoke :write [14 7] INFO [2022-03-13 17:10:03,131] jepsen worker 12 - jepsen.util 732 :invoke :write [4 17] INFO [2022-03-13 17:10:03,212] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,213] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:03,214] jepsen worker 69 - jepsen.util 609 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,215] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:10:03,240] jepsen worker 87 - jepsen.util 1617 :info :write [29 6] :RequestTimeoutException INFO [2022-03-13 17:10:03,241] jepsen worker 87 - jepsen.util 1707 :invoke :write [29 22] INFO [2022-03-13 17:10:03,289] jepsen worker 29 - jepsen.util 1199 :ok :read [9 31] INFO [2022-03-13 17:10:03,290] jepsen worker 29 - jepsen.util 1199 :invoke :write [9 43] INFO [2022-03-13 17:10:03,294] jepsen worker 45 - jepsen.util 1395 :info :write [15 21] :SyncWriteAmbiguous INFO [2022-03-13 17:10:03,296] jepsen worker 45 - jepsen.util 1485 :invoke :read [15 nil] INFO [2022-03-13 17:10:03,374] jepsen worker 65 - jepsen.util 1325 :ok :read [21 :nil] INFO [2022-03-13 17:10:03,374] jepsen worker 65 - jepsen.util 1325 :invoke :write [21 44] INFO [2022-03-13 17:10:03,425] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,425] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,426] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:10:03,428] jepsen worker 79 - jepsen.util 889 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,428] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,431] jepsen worker 76 - jepsen.util 616 :invoke :write [25 19] INFO [2022-03-13 17:10:03,431] jepsen worker 79 - jepsen.util 889 :invoke :write [26 0] INFO [2022-03-13 17:10:03,432] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:03,439] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:03,439] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,440] jepsen worker 23 - jepsen.util 563 :invoke :write [7 26] INFO [2022-03-13 17:10:03,441] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:10:03,480] jepsen worker 83 - jepsen.util 803 :info :write [27 40] :RequestTimeoutException INFO [2022-03-13 17:10:03,481] jepsen worker 83 - jepsen.util 893 :invoke :read [27 nil] INFO [2022-03-13 17:10:03,698] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,698] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,698] jepsen worker 76 - jepsen.util 616 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,698] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:03,699] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:03,699] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:03,845] jepsen worker 53 - jepsen.util 503 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,846] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:03,893] jepsen worker 79 - jepsen.util 889 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,893] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,893] jepsen worker 79 - jepsen.util 889 :invoke :write [26 6] INFO [2022-03-13 17:10:03,894] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:03,895] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,895] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,895] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:03,896] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:10:03,896] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:03,896] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:10:04,004] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2404692082111437}, :ns_1@172.23.105.187 {:progress 0.3157894736842105}, :ns_1@172.23.107.227 {:progress 0.2404692082111437}, :ns_1@172.23.107.228 {:progress 0.2529411764705882}} INFO [2022-03-13 17:10:04,004] jepsen worker nemesis - couchbase.util Rebalance check count: 116 INFO [2022-03-13 17:10:04,004] jepsen worker nemesis - couchbase.util Rebalance stuck count: 66 INFO [2022-03-13 17:10:04,050] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,050] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:04,050] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,051] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:04,061] jepsen worker 77 - jepsen.util 707 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,061] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:04,120] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,120] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:10:04,186] jepsen worker 1 - jepsen.util 1081 :ok :read [0 41] INFO [2022-03-13 17:10:04,187] jepsen worker 1 - jepsen.util 1081 :invoke :write [0 48] INFO [2022-03-13 17:10:04,207] jepsen worker 74 - jepsen.util 1424 :info :write [24 40] :RequestTimeoutException INFO [2022-03-13 17:10:04,208] jepsen worker 74 - jepsen.util 1514 :invoke :write [24 20] INFO [2022-03-13 17:10:04,261] jepsen worker 79 - jepsen.util 889 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,261] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,262] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:10:04,262] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:04,268] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,269] jepsen worker 76 - jepsen.util 616 :invoke :write [25 11] INFO [2022-03-13 17:10:04,280] jepsen worker 43 - jepsen.util 403 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,280] jepsen worker 14 - jepsen.util 284 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,281] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:04,282] jepsen worker 43 - jepsen.util 403 :invoke :write [14 9] INFO [2022-03-13 17:10:04,467] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,467] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,468] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:04,468] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] INFO [2022-03-13 17:10:04,661] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:10:04,662] jepsen worker 35 - jepsen.util 665 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,662] jepsen worker 51 - jepsen.util 591 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,662] jepsen worker 26 - jepsen.util 476 :invoke :write [8 5] INFO [2022-03-13 17:10:04,663] jepsen worker 35 - jepsen.util 665 :invoke :write [11 26] INFO [2022-03-13 17:10:04,664] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:04,895] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,895] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:04,920] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,920] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,921] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:10:04,921] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:04,936] jepsen worker 12 - jepsen.util 732 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,937] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,937] jepsen worker 42 - jepsen.util 402 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,937] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,937] jepsen worker 62 - jepsen.util 422 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,939] jepsen worker 53 - jepsen.util 503 :invoke :write [17 30] INFO [2022-03-13 17:10:04,940] jepsen worker 62 - jepsen.util 422 :invoke :write [20 28] INFO [2022-03-13 17:10:04,941] jepsen worker 12 - jepsen.util 732 :invoke :write [4 49] INFO [2022-03-13 17:10:04,942] jepsen worker 14 - jepsen.util 284 :invoke :write [4 13] INFO [2022-03-13 17:10:04,946] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:04,974] jepsen worker 54 - jepsen.util 1044 :info :write [18 12] :RequestTimeoutException INFO [2022-03-13 17:10:04,975] jepsen worker 54 - jepsen.util 1134 :invoke :write [18 11] INFO [2022-03-13 17:10:04,989] jepsen worker 6 - jepsen.util 546 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:04,989] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:05,010] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2404692082111437}, :ns_1@172.23.105.187 {:progress 0.3157894736842105}, :ns_1@172.23.107.227 {:progress 0.2404692082111437}, :ns_1@172.23.107.228 {:progress 0.2529411764705882}} INFO [2022-03-13 17:10:05,010] jepsen worker nemesis - couchbase.util Rebalance check count: 117 INFO [2022-03-13 17:10:05,010] jepsen worker nemesis - couchbase.util Rebalance stuck count: 67 INFO [2022-03-13 17:10:05,100] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,100] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,101] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:05,101] jepsen worker 33 - jepsen.util 663 :invoke :write [11 10] INFO [2022-03-13 17:10:05,110] jepsen worker 76 - jepsen.util 616 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,111] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,111] jepsen worker 76 - jepsen.util 616 :invoke :write [25 27] INFO [2022-03-13 17:10:05,115] jepsen worker 85 - jepsen.util 1255 :info :write [28 6] :RequestTimeoutException INFO [2022-03-13 17:10:05,116] jepsen worker 85 - jepsen.util 1345 :invoke :read [28 nil] INFO [2022-03-13 17:10:05,122] jepsen worker 79 - jepsen.util 889 :invoke :write [26 29] INFO [2022-03-13 17:10:05,146] jepsen worker 67 - jepsen.util 1057 :info :write [22 1] :RequestTimeoutException INFO [2022-03-13 17:10:05,147] jepsen worker 67 - jepsen.util 1147 :invoke :write [22 47] INFO [2022-03-13 17:10:05,312] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,312] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,313] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:05,313] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:10:05,346] jepsen worker 76 - jepsen.util 616 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,347] jepsen worker 76 - jepsen.util 616 :invoke :write [25 1] INFO [2022-03-13 17:10:05,430] jepsen worker 50 - jepsen.util 1220 :info :write [16 26] :RequestTimeoutException INFO [2022-03-13 17:10:05,431] jepsen worker 50 - jepsen.util 1310 :invoke :read [16 nil] INFO [2022-03-13 17:10:05,526] jepsen worker 28 - jepsen.util 1198 :info :write [9 34] :RequestTimeoutException INFO [2022-03-13 17:10:05,527] jepsen worker 28 - jepsen.util 1288 :invoke :write [9 12] INFO [2022-03-13 17:10:05,540] jepsen worker 33 - jepsen.util 663 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,541] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,541] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:10:05,544] jepsen worker 79 - jepsen.util 889 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,547] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:05,550] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:05,551] jepsen worker 26 - jepsen.util 476 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,551] jepsen worker 23 - jepsen.util 563 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,552] jepsen worker 23 - jepsen.util 563 :invoke :write [7 25] INFO [2022-03-13 17:10:05,553] jepsen worker 26 - jepsen.util 476 :invoke :write [8 44] INFO [2022-03-13 17:10:05,584] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,584] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,584] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:10:05,585] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] WARN [2022-03-13 17:10:05,600] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10369581,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10369581,"total_server_duration_us":27,"operation_id":"0x32fe","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":10371422},{"operation_name":"upsert","last_dispatch_duration_us":38319,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1791324,"total_server_duration_us":237952,"encode_duration_us":30025,"operation_id":"0x361a","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10008900},{"operation_name":"upsert","last_dispatch_duration_us":165276,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9812858,"total_server_duration_us":252507,"encode_duration_us":20387,"operation_id":"0x34a7","timeout_ms":10000,"last_server_duration_us":50886,"total_duration_us":10008635},{"operation_name":"upsert","last_dispatch_duration_us":36516,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":837045,"total_server_duration_us":518679,"encode_duration_us":17414,"operation_id":"0x33f2","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10006249},{"operation_name":"upsert","last_dispatch_duration_us":29526,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":652551,"total_server_duration_us":373893,"encode_duration_us":17062,"operation_id":"0x3560","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10005571},{"operation_name":"upsert","last_dispatch_duration_us":41653,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":775048,"total_server_duration_us":453972,"encode_duration_us":16940,"operation_id":"0x34b6","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10003897},{"operation_name":"upsert","last_dispatch_duration_us":52510,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9810162,"total_server_duration_us":265555,"encode_duration_us":17924,"operation_id":"0x34ae","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10001993},{"operation_name":"get","last_dispatch_duration_us":9744185,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9744185,"total_server_duration_us":15,"operation_id":"0x3355","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":9745538},{"operation_name":"upsert","last_dispatch_duration_us":9375691,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9375691,"total_server_duration_us":9052231,"encode_duration_us":18615,"operation_id":"0x347f","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9379356},{"operation_name":"upsert","last_dispatch_duration_us":9336500,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9336500,"total_server_duration_us":9052231,"encode_duration_us":17707,"operation_id":"0x347e","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9340197}],"total_count":20}} WARN [2022-03-13 17:10:05,600] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16892320,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17266746,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9279628,"encode_duration_us":16643,"operation_id":"0x30a2","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10007071}],"total_count":1}} INFO [2022-03-13 17:10:05,788] jepsen worker 80 - jepsen.util 620 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,789] jepsen worker 80 - jepsen.util 620 :invoke :write [26 14] INFO [2022-03-13 17:10:05,797] jepsen worker 17 - jepsen.util 1097 :info :write [5 23] :RequestTimeoutException INFO [2022-03-13 17:10:05,798] jepsen worker 17 - jepsen.util 1187 :invoke :read [5 nil] INFO [2022-03-13 17:10:05,806] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,807] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:05,809] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,810] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:10:05,910] jepsen worker 57 - jepsen.util 1317 :info :write [19 19] :RequestTimeoutException INFO [2022-03-13 17:10:05,911] jepsen worker 57 - jepsen.util 1407 :invoke :read [19 nil] INFO [2022-03-13 17:10:05,943] jepsen worker 2 - jepsen.util 1352 :info :write [0 42] :RequestTimeoutException INFO [2022-03-13 17:10:05,944] jepsen worker 2 - jepsen.util 1442 :invoke :write [0 28] INFO [2022-03-13 17:10:05,989] jepsen worker 76 - jepsen.util 616 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,989] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,991] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:05,992] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:05,994] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,994] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,994] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:05,995] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,995] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,996] jepsen worker 76 - jepsen.util 616 :invoke :write [25 38] INFO [2022-03-13 17:10:05,996] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:05,997] jepsen worker 35 - jepsen.util 665 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:05,997] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:05,998] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:05,998] jepsen worker 35 - jepsen.util 665 :invoke :write [11 39] INFO [2022-03-13 17:10:05,998] jepsen worker 42 - jepsen.util 402 :invoke :write [14 3] INFO [2022-03-13 17:10:06,003] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,003] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,003] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,006] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:06,007] jepsen worker 0 - jepsen.util 1260 :info :write [0 25] :RequestTimeoutException INFO [2022-03-13 17:10:06,010] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:06,012] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:06,013] jepsen worker 0 - jepsen.util 1350 :invoke :read [0 nil] INFO [2022-03-13 17:10:06,019] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2463343108504399}, :ns_1@172.23.105.187 {:progress 0.3216374269005848}, :ns_1@172.23.107.227 {:progress 0.2463343108504399}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:06,019] jepsen worker nemesis - couchbase.util Rebalance check count: 118 INFO [2022-03-13 17:10:06,019] jepsen worker nemesis - couchbase.util Rebalance stuck count: 67 INFO [2022-03-13 17:10:06,043] jepsen worker 17 - jepsen.util 1187 :ok :read [5 43] INFO [2022-03-13 17:10:06,044] jepsen worker 17 - jepsen.util 1187 :invoke :read [5 nil] INFO [2022-03-13 17:10:06,066] jepsen worker 36 - jepsen.util 1386 :ok :read [12 20] INFO [2022-03-13 17:10:06,066] jepsen worker 36 - jepsen.util 1386 :invoke :read [12 nil] INFO [2022-03-13 17:10:06,082] jepsen worker 17 - jepsen.util 1187 :ok :read [5 43] INFO [2022-03-13 17:10:06,082] jepsen worker 17 - jepsen.util 1187 :invoke :write [5 31] INFO [2022-03-13 17:10:06,096] jepsen worker 36 - jepsen.util 1386 :ok :read [12 20] INFO [2022-03-13 17:10:06,097] jepsen worker 36 - jepsen.util 1386 :invoke :read [12 nil] INFO [2022-03-13 17:10:06,122] jepsen worker 36 - jepsen.util 1386 :ok :read [12 20] INFO [2022-03-13 17:10:06,123] jepsen worker 36 - jepsen.util 1386 :invoke :read [12 nil] INFO [2022-03-13 17:10:06,148] jepsen worker 36 - jepsen.util 1386 :ok :read [12 20] INFO [2022-03-13 17:10:06,148] jepsen worker 36 - jepsen.util 1386 :invoke :read [12 nil] INFO [2022-03-13 17:10:06,186] jepsen worker 36 - jepsen.util 1386 :ok :read [12 20] INFO [2022-03-13 17:10:06,187] jepsen worker 36 - jepsen.util 1386 :invoke :read [12 nil] INFO [2022-03-13 17:10:06,194] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,195] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:06,211] jepsen worker 36 - jepsen.util 1386 :ok :read [12 20] INFO [2022-03-13 17:10:06,211] jepsen worker 36 - jepsen.util 1386 :invoke :write [12 20] INFO [2022-03-13 17:10:06,293] jepsen worker 49 - jepsen.util 1039 :info :write [16 1] :RequestTimeoutException INFO [2022-03-13 17:10:06,294] jepsen worker 49 - jepsen.util 1129 :invoke :write [16 47] INFO [2022-03-13 17:10:06,396] jepsen worker 43 - jepsen.util 403 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,396] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,397] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:06,398] jepsen worker 43 - jepsen.util 403 :invoke :write [14 35] INFO [2022-03-13 17:10:06,424] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,424] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,424] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,425] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:06,426] jepsen worker 69 - jepsen.util 609 :invoke :write [23 4] INFO [2022-03-13 17:10:06,426] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:10:06,427] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,427] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:10:06,508] jepsen worker 86 - jepsen.util 1076 :info :write [28 18] :SyncWriteAmbiguous INFO [2022-03-13 17:10:06,508] jepsen worker 39 - jepsen.util 1209 :info :write [13 2] :SyncWriteAmbiguous INFO [2022-03-13 17:10:06,509] jepsen worker 86 - jepsen.util 1166 :invoke :read [28 nil] INFO [2022-03-13 17:10:06,509] jepsen worker 39 - jepsen.util 1299 :invoke :write [13 25] INFO [2022-03-13 17:10:06,600] jepsen worker 53 - jepsen.util 503 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,601] jepsen worker 53 - jepsen.util 503 :invoke :write [17 45] INFO [2022-03-13 17:10:06,603] jepsen worker 12 - jepsen.util 732 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,604] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:06,613] jepsen worker 76 - jepsen.util 616 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,613] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,613] jepsen worker 16 - jepsen.util 1276 :ok :read [5 43] INFO [2022-03-13 17:10:06,614] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:06,615] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,616] jepsen worker 76 - jepsen.util 616 :invoke :write [25 21] INFO [2022-03-13 17:10:06,617] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:06,618] jepsen worker 16 - jepsen.util 1276 :invoke :write [5 31] INFO [2022-03-13 17:10:06,618] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,619] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:06,620] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,621] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:06,622] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,623] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:10:06,816] jepsen worker 14 - jepsen.util 284 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,816] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,816] jepsen worker 23 - jepsen.util 563 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,816] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,816] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,819] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:06,819] jepsen worker 76 - jepsen.util 616 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,820] jepsen worker 60 - jepsen.util 600 :invoke :write [20 39] INFO [2022-03-13 17:10:06,821] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:06,822] jepsen worker 76 - jepsen.util 616 :invoke :write [25 25] INFO [2022-03-13 17:10:06,822] jepsen worker 51 - jepsen.util 591 :invoke :write [17 11] INFO [2022-03-13 17:10:06,826] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,827] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:10:06,830] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:06,840] jepsen worker 37 - jepsen.util 1297 :info :write [12 13] :RequestTimeoutException INFO [2022-03-13 17:10:06,841] jepsen worker 37 - jepsen.util 1387 :invoke :write [12 42] INFO [2022-03-13 17:10:06,890] jepsen worker 34 - jepsen.util 754 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,891] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:06,903] jepsen worker 71 - jepsen.util 431 :invoke :write [23 26] INFO [2022-03-13 17:10:06,891] jepsen worker 34 - jepsen.util 754 :invoke :write [11 1] INFO [2022-03-13 17:10:07,006] jepsen worker 81 - jepsen.util 1431 :info :write [27 46] :RequestTimeoutException INFO [2022-03-13 17:10:07,006] jepsen worker 81 - jepsen.util 1521 :invoke :write [27 31] INFO [2022-03-13 17:10:07,020] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,020] jepsen worker 8 - jepsen.util 638 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,022] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:07,022] jepsen worker 8 - jepsen.util 638 :invoke :write [2 11] INFO [2022-03-13 17:10:07,023] jepsen worker 69 - jepsen.util 609 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,024] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:10:07,027] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2463343108504399}, :ns_1@172.23.105.187 {:progress 0.3216374269005848}, :ns_1@172.23.107.227 {:progress 0.2463343108504399}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:07,027] jepsen worker nemesis - couchbase.util Rebalance check count: 119 INFO [2022-03-13 17:10:07,027] jepsen worker nemesis - couchbase.util Rebalance stuck count: 68 INFO [2022-03-13 17:10:07,029] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,030] jepsen worker 79 - jepsen.util 889 :invoke :write [26 1] INFO [2022-03-13 17:10:07,040] jepsen worker 59 - jepsen.util 1139 :info :write [19 17] :RequestTimeoutException INFO [2022-03-13 17:10:07,041] jepsen worker 59 - jepsen.util 1229 :invoke :read [19 nil] WARN [2022-03-13 17:10:07,103] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17736743,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17990325,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9178798,"encode_duration_us":17314,"operation_id":"0x32af","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10001423},{"operation_name":"upsert","last_dispatch_duration_us":17442529,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17960234,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9137574,"encode_duration_us":19854,"operation_id":"0x32ef","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10004614},{"operation_name":"upsert","last_dispatch_duration_us":16961570,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":16961570,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9045859,"encode_duration_us":21606,"operation_id":"0x33b7","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10008057}],"total_count":3}} INFO [2022-03-13 17:10:07,108] jepsen worker 76 - jepsen.util 616 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,108] jepsen worker 76 - jepsen.util 616 :invoke :write [25 24] WARN [2022-03-13 17:10:07,200] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11018923,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11018923,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":190246,"encode_duration_us":26115,"operation_id":"0x345b","timeout_ms":10000,"last_server_duration_us":190246,"total_duration_us":10001579},{"operation_name":"upsert","last_dispatch_duration_us":12226733,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12226733,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":434330,"encode_duration_us":26248,"operation_id":"0x345c","timeout_ms":10000,"last_server_duration_us":434330,"total_duration_us":10001605},{"operation_name":"upsert","last_dispatch_duration_us":12773541,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12773541,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9145900,"encode_duration_us":18613,"operation_id":"0x353a","timeout_ms":10000,"last_server_duration_us":9145900,"total_duration_us":10002464},{"operation_name":"upsert","last_dispatch_duration_us":13040123,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":13040123,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":44722,"encode_duration_us":30974,"operation_id":"0x345f","timeout_ms":10000,"last_server_duration_us":44722,"total_duration_us":10006754},{"operation_name":"upsert","last_dispatch_duration_us":4906122,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":4906122,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":28127,"encode_duration_us":30382,"operation_id":"0x346c","timeout_ms":10000,"last_server_duration_us":28127,"total_duration_us":10003173},{"operation_name":"upsert","last_dispatch_duration_us":12868376,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12868376,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":625911,"encode_duration_us":32436,"operation_id":"0x345e","timeout_ms":10000,"last_server_duration_us":625911,"total_duration_us":10006833},{"operation_name":"upsert","last_dispatch_duration_us":2062475,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":2062475,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":116115,"encode_duration_us":44309,"operation_id":"0x34ff","timeout_ms":10000,"last_server_duration_us":116115,"total_duration_us":10006634}],"total_count":7}} INFO [2022-03-13 17:10:07,201] jepsen worker 64 - jepsen.util 1054 :info :write [21 22] :SyncWriteAmbiguous INFO [2022-03-13 17:10:07,203] jepsen worker 64 - jepsen.util 1144 :invoke :read [21 nil] WARN [2022-03-13 17:10:07,204] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16223494,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":16223494,"total_server_duration_us":69,"operation_id":"0x3358","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":16225370},{"operation_name":"upsert","last_dispatch_duration_us":11762846,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11762846,"total_server_duration_us":25015,"encode_duration_us":17451,"operation_id":"0x34c3","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10008575},{"operation_name":"upsert","last_dispatch_duration_us":197821,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9333286,"total_server_duration_us":42648,"encode_duration_us":17662,"operation_id":"0x35bd","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10008171},{"operation_name":"upsert","last_dispatch_duration_us":16961570,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":16961570,"total_server_duration_us":9045859,"encode_duration_us":21606,"operation_id":"0x33b7","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10008057},{"operation_name":"upsert","last_dispatch_duration_us":9081546,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":17245169,"total_server_duration_us":51488,"encode_duration_us":18634,"operation_id":"0x33e8","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10007225},{"operation_name":"upsert","last_dispatch_duration_us":36760,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8586448,"total_server_duration_us":150690,"encode_duration_us":17185,"operation_id":"0x364f","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10000902},{"operation_name":"upsert","last_dispatch_duration_us":7783271,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":7783271,"total_server_duration_us":39635,"encode_duration_us":23245,"operation_id":"0x355e","timeout_ms":10000,"last_server_duration_us":39635,"total_duration_us":7808114},{"operation_name":"get","last_dispatch_duration_us":13421,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":13421,"total_server_duration_us":280,"operation_id":"0x365f","timeout_ms":30000,"last_server_duration_us":280,"total_duration_us":7733836},{"operation_name":"get","last_dispatch_duration_us":7260884,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":7260884,"total_server_duration_us":43,"operation_id":"0x3758","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":7261849},{"operation_name":"upsert","last_dispatch_duration_us":6466117,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":6466117,"total_server_duration_us":713969,"encode_duration_us":17561,"operation_id":"0x355f","timeout_ms":10000,"last_server_duration_us":713969,"total_duration_us":6469653}],"total_count":34}} WARN [2022-03-13 17:10:07,204] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13358669,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13358669,"total_server_duration_us":32,"operation_id":"0x32dd","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":13359049},{"operation_name":"get","last_dispatch_duration_us":11918488,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":11918488,"total_server_duration_us":46175,"operation_id":"0x3362","timeout_ms":30000,"last_server_duration_us":46175,"total_duration_us":11925032},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":21419,"operation_id":"0x3392","timeout_ms":10000,"total_duration_us":10009915},{"operation_name":"upsert","last_dispatch_duration_us":166660,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7165181,"total_server_duration_us":379501,"encode_duration_us":16870,"operation_id":"0x354b","timeout_ms":10000,"last_server_duration_us":84574,"total_duration_us":10007230},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17178,"operation_id":"0x35f9","timeout_ms":10000,"total_duration_us":10004283},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":22934,"operation_id":"0x35f8","timeout_ms":10000,"total_duration_us":10003388},{"operation_name":"upsert","last_dispatch_duration_us":10974867,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10974867,"total_server_duration_us":31108,"encode_duration_us":17039,"operation_id":"0x33bc","timeout_ms":10000,"last_server_duration_us":31108,"total_duration_us":10003190},{"operation_name":"get","last_dispatch_duration_us":3970424,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":3970424,"total_server_duration_us":43,"operation_id":"0x3603","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3971843},{"operation_name":"upsert","last_dispatch_duration_us":405569,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":997543,"total_server_duration_us":387171,"encode_duration_us":24162,"operation_id":"0x37bb","timeout_ms":10000,"last_server_duration_us":199588,"total_duration_us":1024111},{"operation_name":"upsert","last_dispatch_duration_us":971765,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":971765,"total_server_duration_us":368872,"encode_duration_us":21386,"operation_id":"0x37f9","timeout_ms":10000,"last_server_duration_us":368872,"total_duration_us":977761}],"total_count":23}} WARN [2022-03-13 17:10:07,204] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17902,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":14710075,"total_server_duration_us":108,"operation_id":"0x3398","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":14714604},{"operation_name":"get","last_dispatch_duration_us":13009912,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13009912,"total_server_duration_us":23,"operation_id":"0x3419","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":13010205},{"operation_name":"upsert","last_dispatch_duration_us":33891,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1213916,"total_server_duration_us":633246,"encode_duration_us":17292,"operation_id":"0x34e8","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10010150},{"operation_name":"upsert","last_dispatch_duration_us":8790252,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10121223,"total_server_duration_us":491269,"encode_duration_us":19067,"operation_id":"0x346e","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10009660},{"operation_name":"upsert","last_dispatch_duration_us":8647792,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9908323,"total_server_duration_us":399897,"encode_duration_us":18164,"operation_id":"0x347d","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10007127},{"operation_name":"upsert","last_dispatch_duration_us":136082,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7524117,"total_server_duration_us":171678,"encode_duration_us":17232,"operation_id":"0x3650","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10005358},{"operation_name":"upsert","last_dispatch_duration_us":29819,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1142804,"total_server_duration_us":668532,"encode_duration_us":22593,"operation_id":"0x3697","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10003693},{"operation_name":"upsert","last_dispatch_duration_us":154701,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9873423,"total_server_duration_us":184678,"encode_duration_us":17760,"operation_id":"0x3574","timeout_ms":10000,"last_server_duration_us":32645,"total_duration_us":10002015},{"operation_name":"get","last_dispatch_duration_us":9067622,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9067622,"total_server_duration_us":19,"operation_id":"0x3567","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":9068758},{"operation_name":"upsert","last_dispatch_duration_us":9055418,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9055418,"total_server_duration_us":9040551,"encode_duration_us":16793,"operation_id":"0x35be","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9059612}],"total_count":40}} WARN [2022-03-13 17:10:07,204] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16063317,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16063317,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":7107043,"encode_duration_us":30494,"operation_id":"0x3214","timeout_ms":10000,"last_server_duration_us":7107043,"total_duration_us":10007916},{"operation_name":"upsert","last_dispatch_duration_us":18047288,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27220777,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9061302,"encode_duration_us":20063,"operation_id":"0x301a","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10009803},{"operation_name":"upsert","last_dispatch_duration_us":18076280,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18076280,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9036305,"encode_duration_us":18728,"operation_id":"0x32f9","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10008417}],"total_count":3}} INFO [2022-03-13 17:10:07,218] jepsen worker 9 - jepsen.util 1179 :info :write [3 45] :RequestTimeoutException INFO [2022-03-13 17:10:07,218] jepsen worker 9 - jepsen.util 1269 :invoke :read [3 nil] INFO [2022-03-13 17:10:07,237] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,238] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,251] jepsen worker 70 - jepsen.util 700 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,252] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:07,295] jepsen worker 79 - jepsen.util 889 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,295] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] WARN [2022-03-13 17:10:07,300] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":1552133,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1552133,"total_server_duration_us":30539,"operation_id":"0x34f7","timeout_ms":30000,"last_server_duration_us":30539,"total_duration_us":12852291},{"operation_name":"upsert","last_dispatch_duration_us":158461,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":587816,"total_server_duration_us":265467,"encode_duration_us":49279,"operation_id":"0x345d","timeout_ms":10000,"last_server_duration_us":55061,"total_duration_us":10009718},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":18943,"operation_id":"0x35f1","timeout_ms":10000,"total_duration_us":10007548},{"operation_name":"upsert","last_dispatch_duration_us":12868376,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":12868376,"total_server_duration_us":625911,"encode_duration_us":32436,"operation_id":"0x345e","timeout_ms":10000,"last_server_duration_us":625911,"total_duration_us":10006833},{"operation_name":"upsert","last_dispatch_duration_us":13040123,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":13040123,"total_server_duration_us":44722,"encode_duration_us":30974,"operation_id":"0x345f","timeout_ms":10000,"last_server_duration_us":44722,"total_duration_us":10006754},{"operation_name":"upsert","last_dispatch_duration_us":2062475,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2062475,"total_server_duration_us":116115,"encode_duration_us":44309,"operation_id":"0x34ff","timeout_ms":10000,"last_server_duration_us":116115,"total_duration_us":10006634},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":18103,"operation_id":"0x3554","timeout_ms":10000,"total_duration_us":10005606},{"operation_name":"upsert","last_dispatch_duration_us":10859429,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10859429,"total_server_duration_us":21895,"encode_duration_us":19336,"operation_id":"0x3651","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10004890},{"operation_name":"upsert","last_dispatch_duration_us":4906122,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4906122,"total_server_duration_us":28127,"encode_duration_us":30382,"operation_id":"0x346c","timeout_ms":10000,"last_server_duration_us":28127,"total_duration_us":10003173},{"operation_name":"upsert","last_dispatch_duration_us":12773541,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12773541,"total_server_duration_us":9145900,"encode_duration_us":18613,"operation_id":"0x353a","timeout_ms":10000,"last_server_duration_us":9145900,"total_duration_us":10002464}],"total_count":30}} INFO [2022-03-13 17:10:07,302] jepsen worker 32 - jepsen.util 1022 :ok :read [10 :nil] WARN [2022-03-13 17:10:07,302] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":14875893,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":14875893,"total_server_duration_us":5733002,"encode_duration_us":28929,"operation_id":"0x3397","timeout_ms":10000,"last_server_duration_us":5733002,"total_duration_us":10008152},{"operation_name":"upsert","last_dispatch_duration_us":70588,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10020421,"total_server_duration_us":47648,"encode_duration_us":19359,"operation_id":"0x3653","timeout_ms":10000,"last_server_duration_us":22720,"total_duration_us":10007701},{"operation_name":"upsert","last_dispatch_duration_us":28979,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7893470,"total_server_duration_us":286675,"encode_duration_us":17539,"operation_id":"0x3363","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10007283},{"operation_name":"upsert","last_dispatch_duration_us":29169,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9355439,"total_server_duration_us":233477,"encode_duration_us":27752,"operation_id":"0x36a6","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10006462},{"operation_name":"upsert","last_dispatch_duration_us":8187496,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12986840,"total_server_duration_us":479601,"encode_duration_us":17982,"operation_id":"0x34a0","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10005852},{"operation_name":"upsert","last_dispatch_duration_us":8361987,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9015823,"total_server_duration_us":375939,"encode_duration_us":20497,"operation_id":"0x362e","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10003005},{"operation_name":"upsert","last_dispatch_duration_us":17237239,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17454625,"total_server_duration_us":9186130,"encode_duration_us":18016,"operation_id":"0x33c2","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10002439},{"operation_name":"upsert","last_dispatch_duration_us":32970,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":492885,"total_server_duration_us":204404,"encode_duration_us":17448,"operation_id":"0x354f","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10001587},{"operation_name":"upsert","last_dispatch_duration_us":6340324,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6851560,"total_server_duration_us":139313,"encode_duration_us":28299,"operation_id":"0x37aa","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":6866373},{"operation_name":"get","last_dispatch_duration_us":6079368,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6079368,"total_server_duration_us":38,"operation_id":"0x37f5","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":6079579}],"total_count":23}} INFO [2022-03-13 17:10:07,303] jepsen worker 32 - jepsen.util 1022 :invoke :write [10 35] WARN [2022-03-13 17:10:07,304] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16832878,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17047816,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9058634,"encode_duration_us":18981,"operation_id":"0x32b2","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10007602}],"total_count":1}} WARN [2022-03-13 17:10:07,403] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17237239,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17454625,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9186130,"encode_duration_us":18016,"operation_id":"0x33c2","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10002439},{"operation_name":"upsert","last_dispatch_duration_us":14875893,"last_local_id":"A312307A00000006/000000005DE126B4","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":14875893,"last_local_socket":"172.23.104.79:33480","total_server_duration_us":5733002,"encode_duration_us":28929,"operation_id":"0x3397","timeout_ms":10000,"last_server_duration_us":5733002,"total_duration_us":10008152}],"total_count":2}} INFO [2022-03-13 17:10:07,446] jepsen worker 22 - jepsen.util 562 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,447] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:07,447] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,448] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:07,504] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,504] jepsen worker 76 - jepsen.util 616 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,505] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:07,505] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,511] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,511] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,514] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,515] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:07,648] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,648] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,650] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,650] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,651] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,651] jepsen worker 79 - jepsen.util 889 :invoke :write [26 45] INFO [2022-03-13 17:10:07,652] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,653] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,653] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,653] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,653] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:07,655] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,655] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:10:07,656] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,657] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,658] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,658] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,658] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,658] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,659] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:07,659] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,660] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,660] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,660] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,660] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:07,661] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:07,661] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,661] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,662] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,662] jepsen worker 76 - jepsen.util 616 :invoke :write [25 49] INFO [2022-03-13 17:10:07,662] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:10:07,663] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:07,664] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,664] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:10:07,672] jepsen worker 44 - jepsen.util 494 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,673] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:07,710] jepsen worker 40 - jepsen.util 1210 :info :write [13 41] :RequestTimeoutException INFO [2022-03-13 17:10:07,711] jepsen worker 40 - jepsen.util 1300 :invoke :write [13 1] INFO [2022-03-13 17:10:07,784] jepsen worker 38 - jepsen.util 1478 :info :write [12 38] :RequestTimeoutException INFO [2022-03-13 17:10:07,785] jepsen worker 38 - jepsen.util 1568 :invoke :read [12 nil] INFO [2022-03-13 17:10:07,855] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,856] jepsen worker 6 - jepsen.util 546 :invoke :write [2 20] INFO [2022-03-13 17:10:07,872] jepsen worker 79 - jepsen.util 889 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:07,872] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,034] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2521994134897361}, :ns_1@172.23.105.187 {:progress 0.3333333333333334}, :ns_1@172.23.107.227 {:progress 0.2521994134897361}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:08,034] jepsen worker nemesis - couchbase.util Rebalance check count: 120 INFO [2022-03-13 17:10:08,034] jepsen worker nemesis - couchbase.util Rebalance stuck count: 68 INFO [2022-03-13 17:10:08,056] jepsen worker 25 - jepsen.util 385 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,057] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:08,057] jepsen worker 76 - jepsen.util 616 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,058] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,059] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,060] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,060] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,060] jepsen worker 76 - jepsen.util 616 :invoke :write [25 17] INFO [2022-03-13 17:10:08,061] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,062] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,063] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,064] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,065] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,066] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,067] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,068] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:10:08,084] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,085] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:08,096] jepsen worker 89 - jepsen.util 1439 :info :write [29 15] :RequestTimeoutException INFO [2022-03-13 17:10:08,096] jepsen worker 89 - jepsen.util 1529 :invoke :read [29 nil] INFO [2022-03-13 17:10:08,121] jepsen worker 89 - jepsen.util 1529 :ok :read [29 6] INFO [2022-03-13 17:10:08,121] jepsen worker 89 - jepsen.util 1529 :invoke :write [29 6] INFO [2022-03-13 17:10:08,129] jepsen worker 34 - jepsen.util 754 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,130] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:08,142] jepsen worker 76 - jepsen.util 616 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,142] jepsen worker 76 - jepsen.util 616 :invoke :write [25 15] INFO [2022-03-13 17:10:08,218] jepsen worker 73 - jepsen.util 1243 :info :write [24 4] :SyncWriteAmbiguous INFO [2022-03-13 17:10:08,218] jepsen worker 73 - jepsen.util 1333 :invoke :read [24 nil] INFO [2022-03-13 17:10:08,255] jepsen worker 73 - jepsen.util 1333 :ok :read [24 24] INFO [2022-03-13 17:10:08,255] jepsen worker 73 - jepsen.util 1333 :invoke :read [24 nil] INFO [2022-03-13 17:10:08,268] jepsen worker 13 - jepsen.util 553 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,268] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,269] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,269] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:08,284] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,285] jepsen worker 79 - jepsen.util 889 :invoke :write [26 35] INFO [2022-03-13 17:10:08,285] jepsen worker 73 - jepsen.util 1333 :ok :read [24 24] INFO [2022-03-13 17:10:08,285] jepsen worker 73 - jepsen.util 1333 :invoke :write [24 13] INFO [2022-03-13 17:10:08,464] jepsen worker 76 - jepsen.util 616 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,464] jepsen worker 76 - jepsen.util 616 :invoke :write [25 38] INFO [2022-03-13 17:10:08,469] jepsen worker 26 - jepsen.util 476 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,470] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:08,471] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,472] jepsen worker 75 - jepsen.util 435 :invoke :write [25 37] INFO [2022-03-13 17:10:08,473] jepsen worker 43 - jepsen.util 403 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,474] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:08,476] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,476] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,476] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:08,477] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:08,477] jepsen worker 77 - jepsen.util 707 :invoke :write [25 28] INFO [2022-03-13 17:10:08,478] jepsen worker 23 - jepsen.util 563 :invoke :write [7 5] INFO [2022-03-13 17:10:08,518] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,518] jepsen worker 71 - jepsen.util 431 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,519] jepsen worker 71 - jepsen.util 431 :invoke :write [23 26] INFO [2022-03-13 17:10:08,519] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:08,521] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:08,521] jepsen worker 44 - jepsen.util 494 :invoke :write [14 12] INFO [2022-03-13 17:10:08,542] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,542] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,543] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:08,544] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:08,668] jepsen worker 79 - jepsen.util 889 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,668] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:08,749] jepsen worker 76 - jepsen.util 616 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,750] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,783] jepsen worker 11 - jepsen.util 1091 :info :write [3 28] :SyncWriteAmbiguous INFO [2022-03-13 17:10:08,783] jepsen worker 11 - jepsen.util 1181 :invoke :write [3 29] INFO [2022-03-13 17:10:08,876] jepsen worker 75 - jepsen.util 435 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,877] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,877] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,877] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,877] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:10:08,878] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:10:08,878] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,879] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,880] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,880] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,881] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,882] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,883] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,884] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,884] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:08,884] jepsen worker 8 - jepsen.util 638 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,885] jepsen worker 8 - jepsen.util 638 :invoke :write [2 28] INFO [2022-03-13 17:10:08,885] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:08,930] jepsen worker 5 - jepsen.util 1355 :info :write [1 47] :RequestTimeoutException INFO [2022-03-13 17:10:08,931] jepsen worker 5 - jepsen.util 1445 :invoke :write [1 14] INFO [2022-03-13 17:10:08,944] jepsen worker 19 - jepsen.util 1009 :info :write [6 14] :RequestTimeoutException INFO [2022-03-13 17:10:08,944] jepsen worker 19 - jepsen.util 1099 :invoke :read [6 nil] INFO [2022-03-13 17:10:08,977] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,977] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,978] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,978] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,979] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,980] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:10:08,992] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,992] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,993] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:08,994] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:08,994] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,995] jepsen worker 76 - jepsen.util 616 :invoke :write [25 32] INFO [2022-03-13 17:10:08,995] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:08,996] jepsen worker 79 - jepsen.util 889 :invoke :write [26 8] INFO [2022-03-13 17:10:09,040] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2521994134897361}, :ns_1@172.23.105.187 {:progress 0.3333333333333334}, :ns_1@172.23.107.227 {:progress 0.2521994134897361}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:09,040] jepsen worker nemesis - couchbase.util Rebalance check count: 121 INFO [2022-03-13 17:10:09,040] jepsen worker nemesis - couchbase.util Rebalance stuck count: 69 INFO [2022-03-13 17:10:09,046] jepsen worker 30 - jepsen.util 1020 :info :write [10 44] :RequestTimeoutException INFO [2022-03-13 17:10:09,047] jepsen worker 30 - jepsen.util 1110 :invoke :write [10 18] INFO [2022-03-13 17:10:09,071] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,072] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:09,078] jepsen worker 76 - jepsen.util 616 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,079] jepsen worker 76 - jepsen.util 616 :invoke :write [25 39] INFO [2022-03-13 17:10:09,098] jepsen worker 6 - jepsen.util 546 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,098] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,098] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,099] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:09,099] jepsen worker 34 - jepsen.util 754 :invoke :write [11 44] INFO [2022-03-13 17:10:09,099] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:09,184] jepsen worker 79 - jepsen.util 889 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,184] jepsen worker 79 - jepsen.util 889 :invoke :write [26 34] INFO [2022-03-13 17:10:09,191] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,191] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:09,196] jepsen worker 56 - jepsen.util 1046 :info :write [18 45] :RequestTimeoutException INFO [2022-03-13 17:10:09,196] jepsen worker 56 - jepsen.util 1136 :invoke :write [18 30] INFO [2022-03-13 17:10:09,197] jepsen worker 7 - jepsen.util 457 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,197] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:09,234] jepsen worker 76 - jepsen.util 616 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,235] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:09,240] jepsen worker 50 - jepsen.util 1310 :ok :read [16 :nil] INFO [2022-03-13 17:10:09,240] jepsen worker 50 - jepsen.util 1310 :invoke :read [16 nil] INFO [2022-03-13 17:10:09,255] jepsen worker 58 - jepsen.util 1138 :ok :read [19 6] INFO [2022-03-13 17:10:09,256] jepsen worker 58 - jepsen.util 1138 :invoke :write [19 3] INFO [2022-03-13 17:10:09,272] jepsen worker 50 - jepsen.util 1310 :ok :read [16 :nil] INFO [2022-03-13 17:10:09,273] jepsen worker 50 - jepsen.util 1310 :invoke :read [16 nil] INFO [2022-03-13 17:10:09,316] jepsen worker 50 - jepsen.util 1310 :ok :read [16 :nil] INFO [2022-03-13 17:10:09,316] jepsen worker 50 - jepsen.util 1310 :invoke :write [16 48] INFO [2022-03-13 17:10:09,371] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,371] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,371] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,372] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:09,372] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:09,373] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:09,376] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,376] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,376] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,376] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:09,377] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:09,377] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:10:09,377] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,378] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,378] jepsen worker 69 - jepsen.util 609 :invoke :write [23 9] INFO [2022-03-13 17:10:09,379] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:09,380] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,380] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:10:09,417] jepsen worker 71 - jepsen.util 431 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,418] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:09,549] jepsen worker 52 - jepsen.util 502 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,550] jepsen worker 79 - jepsen.util 889 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,550] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,550] jepsen worker 25 - jepsen.util 385 :ok :read [8 :nil] INFO [2022-03-13 17:10:09,551] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,551] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,551] jepsen worker 25 - jepsen.util 385 :invoke :write [8 44] INFO [2022-03-13 17:10:09,553] jepsen worker 78 - jepsen.util 438 :invoke :write [26 30] INFO [2022-03-13 17:10:09,553] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,554] jepsen worker 79 - jepsen.util 889 :invoke :write [26 42] INFO [2022-03-13 17:10:09,555] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:09,556] jepsen worker 76 - jepsen.util 616 :invoke :write [25 27] INFO [2022-03-13 17:10:09,556] jepsen worker 13 - jepsen.util 553 :invoke :write [4 46] INFO [2022-03-13 17:10:09,566] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:09,591] jepsen worker 69 - jepsen.util 609 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,592] jepsen worker 69 - jepsen.util 609 :invoke :write [23 6] INFO [2022-03-13 17:10:09,704] jepsen worker 9 - jepsen.util 1269 :ok :read [3 25] INFO [2022-03-13 17:10:09,704] jepsen worker 9 - jepsen.util 1269 :invoke :write [3 0] INFO [2022-03-13 17:10:09,785] jepsen worker 44 - jepsen.util 494 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,785] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,785] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,785] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:09,786] jepsen worker 6 - jepsen.util 546 :invoke :write [2 26] INFO [2022-03-13 17:10:09,786] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:09,868] jepsen worker 80 - jepsen.util 620 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,869] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:09,872] jepsen worker 76 - jepsen.util 616 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,872] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:09,877] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,877] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,878] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:09,879] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:10:09,920] jepsen worker 47 - jepsen.util 1217 :info :write [15 11] :RequestTimeoutException INFO [2022-03-13 17:10:09,921] jepsen worker 47 - jepsen.util 1307 :invoke :write [15 6] INFO [2022-03-13 17:10:09,947] jepsen worker 34 - jepsen.util 754 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,947] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,947] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:09,948] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:09,948] jepsen worker 34 - jepsen.util 754 :invoke :write [11 16] INFO [2022-03-13 17:10:09,949] jepsen worker 44 - jepsen.util 494 :invoke :write [14 20] INFO [2022-03-13 17:10:09,950] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:09,950] jepsen worker 70 - jepsen.util 700 :invoke :write [23 31] INFO [2022-03-13 17:10:10,019] jepsen worker 24 - jepsen.util 474 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,023] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:10,025] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,025] jepsen worker 69 - jepsen.util 609 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,026] jepsen worker 69 - jepsen.util 609 :invoke :write [23 2] INFO [2022-03-13 17:10:10,030] jepsen worker 79 - jepsen.util 889 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,030] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:10:10,034] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,035] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:10,036] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,048] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,049] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,049] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2521994134897361}, :ns_1@172.23.105.187 {:progress 0.3391812865497076}, :ns_1@172.23.107.227 {:progress 0.2521994134897361}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:10,049] jepsen worker nemesis - couchbase.util Rebalance check count: 122 INFO [2022-03-13 17:10:10,049] jepsen worker nemesis - couchbase.util Rebalance stuck count: 69 INFO [2022-03-13 17:10:10,050] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,051] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,051] jepsen worker 79 - jepsen.util 889 :invoke :write [26 4] INFO [2022-03-13 17:10:10,052] jepsen worker 76 - jepsen.util 616 :invoke :write [25 23] INFO [2022-03-13 17:10:10,154] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,154] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:10,179] jepsen worker 6 - jepsen.util 546 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,179] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,179] jepsen worker 6 - jepsen.util 546 :invoke :write [2 33] INFO [2022-03-13 17:10:10,182] jepsen worker 71 - jepsen.util 431 :invoke :write [23 7] INFO [2022-03-13 17:10:10,210] jepsen worker 21 - jepsen.util 651 :info :write [7 42] :RequestTimeoutException INFO [2022-03-13 17:10:10,210] jepsen worker 61 - jepsen.util 511 :info :write [20 28] :RequestTimeoutException INFO [2022-03-13 17:10:10,211] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:10,211] jepsen worker 21 - jepsen.util 741 :invoke :write [7 32] INFO [2022-03-13 17:10:10,266] jepsen worker 35 - jepsen.util 665 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,266] jepsen worker 35 - jepsen.util 665 :invoke :write [11 34] INFO [2022-03-13 17:10:10,271] jepsen worker 76 - jepsen.util 616 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,272] jepsen worker 76 - jepsen.util 616 :invoke :write [25 27] INFO [2022-03-13 17:10:10,277] jepsen worker 60 - jepsen.util 600 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,277] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:10,300] jepsen worker 10 - jepsen.util 1090 :info :write [3 11] :RequestTimeoutException INFO [2022-03-13 17:10:10,301] jepsen worker 10 - jepsen.util 1180 :invoke :write [3 25] INFO [2022-03-13 17:10:10,366] jepsen worker 79 - jepsen.util 889 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,366] jepsen worker 69 - jepsen.util 609 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,367] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:10,367] jepsen worker 79 - jepsen.util 889 :invoke :write [26 27] INFO [2022-03-13 17:10:10,463] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,463] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,464] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,464] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:10,480] jepsen worker 76 - jepsen.util 616 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,480] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,484] jepsen worker 34 - jepsen.util 754 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,485] jepsen worker 34 - jepsen.util 754 :invoke :write [11 36] INFO [2022-03-13 17:10:10,518] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,519] jepsen worker 21 - jepsen.util 741 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,519] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,519] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,520] jepsen worker 21 - jepsen.util 741 :invoke :write [7 20] INFO [2022-03-13 17:10:10,520] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:10:10,523] jepsen worker 33 - jepsen.util 663 :invoke :write [11 49] INFO [2022-03-13 17:10:10,523] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:10,530] jepsen worker 72 - jepsen.util 1152 :info :write [24 47] :RequestTimeoutException INFO [2022-03-13 17:10:10,531] jepsen worker 72 - jepsen.util 1242 :invoke :write [24 43] INFO [2022-03-13 17:10:10,588] jepsen worker 27 - jepsen.util 1467 :ok :write [9 26] INFO [2022-03-13 17:10:10,589] jepsen worker 27 - jepsen.util 1467 :invoke :read [9 nil] INFO [2022-03-13 17:10:10,626] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,626] jepsen worker 46 - jepsen.util 1306 :info :write [15 40] :RequestTimeoutException INFO [2022-03-13 17:10:10,627] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:10,627] jepsen worker 46 - jepsen.util 1396 :invoke :read [15 nil] INFO [2022-03-13 17:10:10,628] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,629] jepsen worker 52 - jepsen.util 502 :invoke :write [17 12] INFO [2022-03-13 17:10:10,638] jepsen worker 79 - jepsen.util 889 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,639] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,639] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:10,639] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,640] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,641] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,641] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:10:10,648] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,649] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,650] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,651] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,652] jepsen worker 76 - jepsen.util 616 :invoke :write [25 48] INFO [2022-03-13 17:10:10,684] jepsen worker 44 - jepsen.util 494 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,685] jepsen worker 44 - jepsen.util 494 :invoke :write [14 7] INFO [2022-03-13 17:10:10,726] jepsen worker 63 - jepsen.util 1233 :info :write [21 2] :RequestTimeoutException INFO [2022-03-13 17:10:10,727] jepsen worker 63 - jepsen.util 1323 :invoke :read [21 nil] INFO [2022-03-13 17:10:10,777] jepsen worker 13 - jepsen.util 553 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,778] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:10,793] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,793] jepsen worker 79 - jepsen.util 889 :invoke :write [26 4] INFO [2022-03-13 17:10:10,916] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,916] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,923] jepsen worker 76 - jepsen.util 616 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,924] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,960] jepsen worker 70 - jepsen.util 700 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,961] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:10,981] jepsen worker 79 - jepsen.util 889 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,981] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:10,982] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:10,982] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:10:11,056] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2521994134897361}, :ns_1@172.23.105.187 {:progress 0.3391812865497076}, :ns_1@172.23.107.227 {:progress 0.2521994134897361}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:11,056] jepsen worker nemesis - couchbase.util Rebalance check count: 123 INFO [2022-03-13 17:10:11,056] jepsen worker nemesis - couchbase.util Rebalance stuck count: 70 INFO [2022-03-13 17:10:11,068] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,068] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:11,078] jepsen worker 78 - jepsen.util 438 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,079] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:11,079] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:10:11,079] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,079] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:11,080] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:11,080] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,080] jepsen worker 7 - jepsen.util 457 :invoke :write [2 15] INFO [2022-03-13 17:10:11,081] jepsen worker 22 - jepsen.util 562 :invoke :write [7 37] INFO [2022-03-13 17:10:11,082] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:11,088] jepsen worker 33 - jepsen.util 663 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,088] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:10:11,133] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,133] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,134] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:10:11,134] jepsen worker 76 - jepsen.util 616 :invoke :write [25 18] INFO [2022-03-13 17:10:11,137] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,137] jepsen worker 21 - jepsen.util 741 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,138] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:10:11,138] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,138] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:11,139] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:11,378] jepsen worker 6 - jepsen.util 546 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,379] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:11,379] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,379] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:11,380] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,380] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:11,381] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:10:11,381] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:11,408] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,408] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,408] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:11,409] jepsen worker 21 - jepsen.util 741 :invoke :write [7 6] INFO [2022-03-13 17:10:11,411] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:10:11,411] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:10:11,455] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,455] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:10:11,520] jepsen worker 8 - jepsen.util 638 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,521] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:11,522] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,523] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:11,566] jepsen worker 21 - jepsen.util 741 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,567] jepsen worker 21 - jepsen.util 741 :invoke :write [7 39] INFO [2022-03-13 17:10:11,594] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,595] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:11,623] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,623] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:11,623] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,624] jepsen worker 75 - jepsen.util 435 :invoke :write [25 36] INFO [2022-03-13 17:10:11,700] jepsen worker 48 - jepsen.util 1128 :info :write [16 31] :SyncWriteAmbiguous INFO [2022-03-13 17:10:11,701] jepsen worker 48 - jepsen.util 1218 :invoke :write [16 45] INFO [2022-03-13 17:10:11,712] jepsen worker 76 - jepsen.util 616 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,712] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:11,734] jepsen worker 42 - jepsen.util 402 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,734] jepsen worker 51 - jepsen.util 591 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,734] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:11,735] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:11,735] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,735] jepsen worker 43 - jepsen.util 403 :invoke :write [14 40] INFO [2022-03-13 17:10:11,736] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:11,736] jepsen worker 12 - jepsen.util 732 :invoke :write [4 46] INFO [2022-03-13 17:10:11,761] jepsen worker 21 - jepsen.util 741 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,761] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,762] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,762] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:10:11,763] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:11,765] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,765] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,765] jepsen worker 76 - jepsen.util 616 :invoke :write [25 18] INFO [2022-03-13 17:10:11,766] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:11,767] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,768] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:10:11,770] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:11,864] jepsen worker 71 - jepsen.util 431 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,865] jepsen worker 71 - jepsen.util 431 :invoke :write [23 16] INFO [2022-03-13 17:10:11,924] jepsen worker 52 - jepsen.util 502 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,924] jepsen worker 52 - jepsen.util 502 :invoke :write [17 9] INFO [2022-03-13 17:10:11,931] jepsen worker 76 - jepsen.util 616 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:11,943] jepsen worker 76 - jepsen.util 616 :invoke :write [25 20] INFO [2022-03-13 17:10:11,970] jepsen worker 4 - jepsen.util 1354 :info :write [1 16] :RequestTimeoutException INFO [2022-03-13 17:10:11,970] jepsen worker 4 - jepsen.util 1444 :invoke :read [1 nil] INFO [2022-03-13 17:10:12,001] jepsen worker 4 - jepsen.util 1444 :ok :read [1 42] INFO [2022-03-13 17:10:12,001] jepsen worker 4 - jepsen.util 1444 :invoke :write [1 16] INFO [2022-03-13 17:10:12,020] jepsen worker 75 - jepsen.util 435 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,020] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:12,020] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,021] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:10:12,021] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:12,022] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:12,062] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2580645161290323}, :ns_1@172.23.105.187 {:progress 0.3450292397660819}, :ns_1@172.23.107.227 {:progress 0.2580645161290323}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:12,062] jepsen worker nemesis - couchbase.util Rebalance check count: 124 INFO [2022-03-13 17:10:12,062] jepsen worker nemesis - couchbase.util Rebalance stuck count: 70 INFO [2022-03-13 17:10:12,087] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,088] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:12,139] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,139] jepsen worker 76 - jepsen.util 616 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,140] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:12,140] jepsen worker 76 - jepsen.util 616 :invoke :write [25 18] INFO [2022-03-13 17:10:12,141] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,142] jepsen worker 79 - jepsen.util 889 :invoke :write [26 49] INFO [2022-03-13 17:10:12,181] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,181] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:12,181] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,182] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:12,182] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:12,183] jepsen worker 33 - jepsen.util 663 :invoke :write [11 45] INFO [2022-03-13 17:10:12,187] jepsen worker 34 - jepsen.util 754 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,188] jepsen worker 34 - jepsen.util 754 :invoke :write [11 48] INFO [2022-03-13 17:10:12,228] jepsen worker 25 - jepsen.util 385 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,228] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,229] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,229] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:10:12,229] jepsen worker 25 - jepsen.util 385 :invoke :write [8 40] INFO [2022-03-13 17:10:12,229] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,230] jepsen worker 78 - jepsen.util 438 :invoke :write [26 12] INFO [2022-03-13 17:10:12,231] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:12,231] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:12,232] jepsen worker 80 - jepsen.util 620 :invoke :write [26 5] INFO [2022-03-13 17:10:12,286] jepsen worker 76 - jepsen.util 616 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,287] jepsen worker 76 - jepsen.util 616 :invoke :write [25 26] INFO [2022-03-13 17:10:12,297] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,297] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,297] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:12,298] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:12,299] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:10:12,299] jepsen worker 21 - jepsen.util 741 :invoke :write [7 44] INFO [2022-03-13 17:10:12,338] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,338] jepsen worker 44 - jepsen.util 494 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,338] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,343] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:12,343] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:10:12,344] jepsen worker 6 - jepsen.util 546 :invoke :write [2 31] INFO [2022-03-13 17:10:12,412] jepsen worker 79 - jepsen.util 889 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,413] jepsen worker 79 - jepsen.util 889 :invoke :write [26 1] INFO [2022-03-13 17:10:12,444] jepsen worker 55 - jepsen.util 1225 :info :write [18 5] :SyncWriteAmbiguous INFO [2022-03-13 17:10:12,445] jepsen worker 55 - jepsen.util 1315 :invoke :read [18 nil] INFO [2022-03-13 17:10:12,464] jepsen worker 7 - jepsen.util 457 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,465] jepsen worker 7 - jepsen.util 457 :invoke :write [2 24] INFO [2022-03-13 17:10:12,470] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,470] jepsen worker 33 - jepsen.util 663 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,471] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:10:12,474] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:12,517] jepsen worker 76 - jepsen.util 616 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,518] jepsen worker 76 - jepsen.util 616 :invoke :write [25 27] INFO [2022-03-13 17:10:12,569] jepsen worker 21 - jepsen.util 741 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,570] jepsen worker 21 - jepsen.util 741 :invoke :write [7 20] INFO [2022-03-13 17:10:12,610] jepsen worker 88 - jepsen.util 1078 :info :write [29 23] :RequestTimeoutException INFO [2022-03-13 17:10:12,611] jepsen worker 88 - jepsen.util 1168 :invoke :write [29 27] INFO [2022-03-13 17:10:12,666] jepsen worker 79 - jepsen.util 889 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,666] jepsen worker 71 - jepsen.util 431 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,667] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:12,667] jepsen worker 71 - jepsen.util 431 :invoke :write [23 39] INFO [2022-03-13 17:10:12,671] jepsen worker 62 - jepsen.util 422 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,671] jepsen worker 23 - jepsen.util 563 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,672] jepsen worker 62 - jepsen.util 422 :invoke :write [20 46] INFO [2022-03-13 17:10:12,673] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:10:12,695] jepsen worker 22 - jepsen.util 562 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,696] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,696] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:12,697] jepsen worker 8 - jepsen.util 638 :invoke :write [2 42] INFO [2022-03-13 17:10:12,740] jepsen worker 66 - jepsen.util 1236 :info :write [22 12] :RequestTimeoutException INFO [2022-03-13 17:10:12,741] jepsen worker 66 - jepsen.util 1326 :invoke :read [22 nil] INFO [2022-03-13 17:10:12,804] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,804] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,805] jepsen worker 69 - jepsen.util 609 :invoke :write [23 24] INFO [2022-03-13 17:10:12,805] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:12,808] jepsen worker 34 - jepsen.util 754 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,808] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:12,809] jepsen worker 34 - jepsen.util 754 :invoke :write [11 14] INFO [2022-03-13 17:10:12,809] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:12,822] jepsen worker 46 - jepsen.util 1396 :ok :read [15 48] INFO [2022-03-13 17:10:12,823] jepsen worker 46 - jepsen.util 1396 :invoke :write [15 21] INFO [2022-03-13 17:10:12,824] jepsen worker 76 - jepsen.util 616 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,824] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,825] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:12,825] jepsen worker 76 - jepsen.util 616 :invoke :write [25 32] INFO [2022-03-13 17:10:12,828] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,828] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:12,832] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,833] jepsen worker 79 - jepsen.util 889 :invoke :write [26 1] INFO [2022-03-13 17:10:12,936] jepsen worker 52 - jepsen.util 502 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,936] jepsen worker 24 - jepsen.util 474 :ok :read [8 :nil] INFO [2022-03-13 17:10:12,936] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,937] jepsen worker 52 - jepsen.util 502 :invoke :write [17 23] INFO [2022-03-13 17:10:12,938] jepsen worker 24 - jepsen.util 474 :invoke :write [8 14] INFO [2022-03-13 17:10:12,938] jepsen worker 13 - jepsen.util 553 :invoke :write [4 34] INFO [2022-03-13 17:10:12,942] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:12,942] jepsen worker 84 - jepsen.util 1434 :info :write [28 2] :RequestTimeoutException INFO [2022-03-13 17:10:12,943] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:10:12,944] jepsen worker 84 - jepsen.util 1524 :invoke :write [28 9] INFO [2022-03-13 17:10:12,996] jepsen worker 20 - jepsen.util 1550 :info :write [6 9] :RequestTimeoutException INFO [2022-03-13 17:10:12,997] jepsen worker 20 - jepsen.util 1640 :invoke :write [6 14] INFO [2022-03-13 17:10:13,071] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2580645161290323}, :ns_1@172.23.105.187 {:progress 0.3450292397660819}, :ns_1@172.23.107.227 {:progress 0.2580645161290323}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:13,071] jepsen worker nemesis - couchbase.util Rebalance check count: 125 INFO [2022-03-13 17:10:13,071] jepsen worker nemesis - couchbase.util Rebalance stuck count: 71 INFO [2022-03-13 17:10:13,126] jepsen worker 53 - jepsen.util 503 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,126] jepsen worker 77 - jepsen.util 707 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,126] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,127] jepsen worker 26 - jepsen.util 476 :ok :read [8 :nil] INFO [2022-03-13 17:10:13,127] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,127] jepsen worker 77 - jepsen.util 707 :invoke :write [25 42] INFO [2022-03-13 17:10:13,127] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,128] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:13,128] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:13,129] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:13,130] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,130] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:13,132] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:13,134] jepsen worker 76 - jepsen.util 616 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,134] jepsen worker 76 - jepsen.util 616 :invoke :write [25 31] INFO [2022-03-13 17:10:13,142] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:13,144] jepsen worker 27 - jepsen.util 1467 :ok :read [9 26] INFO [2022-03-13 17:10:13,145] jepsen worker 27 - jepsen.util 1467 :invoke :write [9 48] INFO [2022-03-13 17:10:13,177] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,177] jepsen worker 21 - jepsen.util 741 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,178] jepsen worker 21 - jepsen.util 741 :invoke :write [7 7] INFO [2022-03-13 17:10:13,178] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:10:13,203] jepsen worker 6 - jepsen.util 546 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,203] jepsen worker 6 - jepsen.util 546 :invoke :write [2 42] INFO [2022-03-13 17:10:13,211] jepsen worker 69 - jepsen.util 609 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,212] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:13,220] jepsen worker 79 - jepsen.util 889 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,220] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:13,263] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,264] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:10:13,270] jepsen worker 87 - jepsen.util 1707 :info :write [29 22] :RequestTimeoutException INFO [2022-03-13 17:10:13,270] jepsen worker 87 - jepsen.util 1797 :invoke :write [29 37] INFO [2022-03-13 17:10:13,304] jepsen worker 25 - jepsen.util 385 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,304] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:13,317] jepsen worker 29 - jepsen.util 1199 :info :write [9 43] :RequestTimeoutException INFO [2022-03-13 17:10:13,318] jepsen worker 29 - jepsen.util 1289 :invoke :write [9 8] INFO [2022-03-13 17:10:13,342] jepsen worker 76 - jepsen.util 616 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,342] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,343] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,346] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:13,352] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,352] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,353] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:13,355] jepsen worker 79 - jepsen.util 889 :invoke :write [26 29] INFO [2022-03-13 17:10:13,356] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:10:13,356] jepsen worker 21 - jepsen.util 741 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,356] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,357] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:13,358] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,360] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,360] jepsen worker 15 - jepsen.util 1275 :ok :read [5 43] INFO [2022-03-13 17:10:13,362] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,362] jepsen worker 15 - jepsen.util 1275 :invoke :write [5 32] INFO [2022-03-13 17:10:13,364] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,365] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,368] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,369] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,370] jepsen worker 82 - jepsen.util 1252 :ok :read [27 4] INFO [2022-03-13 17:10:13,373] jepsen worker 82 - jepsen.util 1252 :invoke :write [27 42] INFO [2022-03-13 17:10:13,378] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,383] jepsen worker 68 - jepsen.util 1238 :ok :read [22 30] INFO [2022-03-13 17:10:13,383] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,384] jepsen worker 68 - jepsen.util 1238 :invoke :write [22 20] INFO [2022-03-13 17:10:13,386] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,390] jepsen worker 83 - jepsen.util 893 :ok :read [27 4] INFO [2022-03-13 17:10:13,392] jepsen worker 86 - jepsen.util 1166 :ok :read [28 44] INFO [2022-03-13 17:10:13,392] jepsen worker 83 - jepsen.util 893 :invoke :read [27 nil] INFO [2022-03-13 17:10:13,393] jepsen worker 86 - jepsen.util 1166 :invoke :write [28 25] INFO [2022-03-13 17:10:13,398] jepsen worker 65 - jepsen.util 1325 :info :write [21 44] :RequestTimeoutException INFO [2022-03-13 17:10:13,399] jepsen worker 65 - jepsen.util 1415 :invoke :write [21 14] INFO [2022-03-13 17:10:13,402] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:13,403] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:10:13,446] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,447] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:13,475] jepsen worker 79 - jepsen.util 889 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,475] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,477] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:10:13,478] jepsen worker 76 - jepsen.util 616 :invoke :write [25 19] INFO [2022-03-13 17:10:13,508] jepsen worker 83 - jepsen.util 893 :ok :read [27 4] INFO [2022-03-13 17:10:13,509] jepsen worker 83 - jepsen.util 893 :invoke :write [27 4] INFO [2022-03-13 17:10:13,523] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:13,524] jepsen worker 78 - jepsen.util 438 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,524] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,524] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,524] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:10:13,525] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:13,526] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:13,526] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,577] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,578] jepsen worker 79 - jepsen.util 889 :invoke :write [26 5] INFO [2022-03-13 17:10:13,624] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:13,625] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:10:13,644] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,644] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:13,644] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,644] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,644] jepsen worker 33 - jepsen.util 663 :invoke :write [11 19] INFO [2022-03-13 17:10:13,645] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,645] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:13,646] jepsen worker 69 - jepsen.util 609 :invoke :write [23 7] INFO [2022-03-13 17:10:13,648] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:13,648] jepsen worker 71 - jepsen.util 431 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,649] jepsen worker 71 - jepsen.util 431 :invoke :write [23 36] INFO [2022-03-13 17:10:13,649] jepsen worker 44 - jepsen.util 494 :invoke :write [14 30] INFO [2022-03-13 17:10:13,651] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,652] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,653] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,654] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:10:13,662] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:13,663] jepsen worker 21 - jepsen.util 741 :invoke :write [7 7] INFO [2022-03-13 17:10:13,689] jepsen worker 76 - jepsen.util 616 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,690] jepsen worker 76 - jepsen.util 616 :invoke :write [25 31] INFO [2022-03-13 17:10:13,734] jepsen worker 79 - jepsen.util 889 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,735] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:13,756] jepsen worker 33 - jepsen.util 663 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,757] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:13,762] jepsen worker 80 - jepsen.util 620 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,762] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:13,763] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:13,763] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:13,804] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,804] jepsen worker 76 - jepsen.util 616 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,805] jepsen worker 76 - jepsen.util 616 :invoke :write [25 44] INFO [2022-03-13 17:10:13,805] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:13,868] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,868] jepsen worker 79 - jepsen.util 889 :invoke :write [26 23] INFO [2022-03-13 17:10:13,871] jepsen worker 69 - jepsen.util 609 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,872] jepsen worker 69 - jepsen.util 609 :invoke :write [23 42] INFO [2022-03-13 17:10:13,880] jepsen worker 34 - jepsen.util 754 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,880] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:13,929] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,930] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:13,983] jepsen worker 8 - jepsen.util 638 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,983] jepsen worker 8 - jepsen.util 638 :invoke :write [2 46] INFO [2022-03-13 17:10:13,984] jepsen worker 76 - jepsen.util 616 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:13,984] jepsen worker 76 - jepsen.util 616 :invoke :write [25 15] INFO [2022-03-13 17:10:14,020] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,020] jepsen worker 21 - jepsen.util 741 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,021] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:14,021] jepsen worker 21 - jepsen.util 741 :invoke :write [7 34] INFO [2022-03-13 17:10:14,072] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,072] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,072] jepsen worker 69 - jepsen.util 609 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,072] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:14,073] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:10:14,073] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:14,078] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2639296187683284}, :ns_1@172.23.105.187 {:progress 0.3508771929824561}, :ns_1@172.23.107.227 {:progress 0.2639296187683284}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:14,078] jepsen worker nemesis - couchbase.util Rebalance check count: 126 INFO [2022-03-13 17:10:14,078] jepsen worker nemesis - couchbase.util Rebalance stuck count: 71 INFO [2022-03-13 17:10:14,208] jepsen worker 1 - jepsen.util 1081 :info :write [0 48] :RequestTimeoutException INFO [2022-03-13 17:10:14,209] jepsen worker 1 - jepsen.util 1171 :invoke :read [0 nil] INFO [2022-03-13 17:10:14,236] jepsen worker 74 - jepsen.util 1514 :info :write [24 20] :RequestTimeoutException INFO [2022-03-13 17:10:14,236] jepsen worker 74 - jepsen.util 1604 :invoke :read [24 nil] INFO [2022-03-13 17:10:14,398] jepsen worker 6 - jepsen.util 546 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,398] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,399] jepsen worker 70 - jepsen.util 700 :invoke :write [23 48] INFO [2022-03-13 17:10:14,402] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:14,403] jepsen worker 79 - jepsen.util 889 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,404] jepsen worker 79 - jepsen.util 889 :invoke :write [26 35] INFO [2022-03-13 17:10:14,430] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,430] jepsen worker 21 - jepsen.util 741 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,430] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,431] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:14,432] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:14,432] jepsen worker 21 - jepsen.util 741 :invoke :write [7 31] INFO [2022-03-13 17:10:14,472] jepsen worker 76 - jepsen.util 616 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,473] jepsen worker 76 - jepsen.util 616 :invoke :write [25 11] INFO [2022-03-13 17:10:14,487] jepsen worker 13 - jepsen.util 553 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,487] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,487] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,487] jepsen worker 25 - jepsen.util 385 :invoke :write [8 17] INFO [2022-03-13 17:10:14,488] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:14,488] jepsen worker 78 - jepsen.util 438 :invoke :write [26 2] INFO [2022-03-13 17:10:14,526] jepsen worker 79 - jepsen.util 889 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,527] jepsen worker 79 - jepsen.util 889 :invoke :write [26 5] INFO [2022-03-13 17:10:14,545] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,545] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,545] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,547] jepsen worker 75 - jepsen.util 435 :invoke :write [25 20] INFO [2022-03-13 17:10:14,548] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:14,549] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:14,600] jepsen worker 76 - jepsen.util 616 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,601] jepsen worker 76 - jepsen.util 616 :invoke :write [25 24] INFO [2022-03-13 17:10:14,606] jepsen worker 71 - jepsen.util 431 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,606] jepsen worker 71 - jepsen.util 431 :invoke :write [23 38] INFO [2022-03-13 17:10:14,618] jepsen worker 21 - jepsen.util 741 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,618] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,618] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,619] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:14,620] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:14,620] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:10:14,667] jepsen worker 52 - jepsen.util 502 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,667] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,668] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:14,668] jepsen worker 80 - jepsen.util 620 :invoke :write [26 20] INFO [2022-03-13 17:10:14,669] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:14,669] jepsen worker 52 - jepsen.util 502 :invoke :write [17 22] INFO [2022-03-13 17:10:14,681] jepsen worker 79 - jepsen.util 889 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,682] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:14,689] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,689] jepsen worker 75 - jepsen.util 435 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,689] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:14,690] jepsen worker 69 - jepsen.util 609 :invoke :write [23 36] INFO [2022-03-13 17:10:14,690] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:14,691] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:10:14,754] jepsen worker 44 - jepsen.util 494 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,754] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:14,755] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,755] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,756] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:14,756] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:14,761] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,761] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:14,762] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:10:14,762] jepsen worker 21 - jepsen.util 741 :invoke :write [7 11] INFO [2022-03-13 17:10:14,806] jepsen worker 8 - jepsen.util 638 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,807] jepsen worker 8 - jepsen.util 638 :invoke :write [2 34] INFO [2022-03-13 17:10:14,828] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,828] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:14,844] jepsen worker 76 - jepsen.util 616 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,844] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:14,911] jepsen worker 70 - jepsen.util 700 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,911] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:14,919] jepsen worker 69 - jepsen.util 609 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,919] jepsen worker 69 - jepsen.util 609 :invoke :write [23 5] INFO [2022-03-13 17:10:14,936] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,936] jepsen worker 76 - jepsen.util 616 :invoke :write [25 12] INFO [2022-03-13 17:10:14,941] jepsen worker 35 - jepsen.util 665 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:14,942] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:14,942] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:10:14,943] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:15,000] jepsen worker 54 - jepsen.util 1134 :info :write [18 11] :RequestTimeoutException INFO [2022-03-13 17:10:15,001] jepsen worker 54 - jepsen.util 1224 :invoke :read [18 nil] INFO [2022-03-13 17:10:15,086] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2639296187683284}, :ns_1@172.23.105.187 {:progress 0.3508771929824561}, :ns_1@172.23.107.227 {:progress 0.2639296187683284}, :ns_1@172.23.107.228 {:progress 0.2588235294117647}} INFO [2022-03-13 17:10:15,086] jepsen worker nemesis - couchbase.util Rebalance check count: 127 INFO [2022-03-13 17:10:15,086] jepsen worker nemesis - couchbase.util Rebalance stuck count: 72 INFO [2022-03-13 17:10:15,089] jepsen worker 57 - jepsen.util 1407 :ok :read [19 17] INFO [2022-03-13 17:10:15,090] jepsen worker 57 - jepsen.util 1407 :invoke :read [19 nil] INFO [2022-03-13 17:10:15,134] jepsen worker 57 - jepsen.util 1407 :ok :read [19 17] INFO [2022-03-13 17:10:15,135] jepsen worker 57 - jepsen.util 1407 :invoke :read [19 nil] INFO [2022-03-13 17:10:15,161] jepsen worker 17 - jepsen.util 1187 :info :write [5 31] :SyncWriteAmbiguous INFO [2022-03-13 17:10:15,161] jepsen worker 17 - jepsen.util 1277 :invoke :write [5 31] INFO [2022-03-13 17:10:15,170] jepsen worker 67 - jepsen.util 1147 :info :write [22 47] :RequestTimeoutException INFO [2022-03-13 17:10:15,170] jepsen worker 67 - jepsen.util 1237 :invoke :read [22 nil] INFO [2022-03-13 17:10:15,179] jepsen worker 57 - jepsen.util 1407 :ok :read [19 17] INFO [2022-03-13 17:10:15,179] jepsen worker 57 - jepsen.util 1407 :invoke :write [19 47] INFO [2022-03-13 17:10:15,196] jepsen worker 3 - jepsen.util 1173 :ok :read [1 42] INFO [2022-03-13 17:10:15,197] jepsen worker 3 - jepsen.util 1173 :invoke :write [1 0] INFO [2022-03-13 17:10:15,234] jepsen worker 38 - jepsen.util 1568 :ok :read [12 20] INFO [2022-03-13 17:10:15,235] jepsen worker 38 - jepsen.util 1568 :invoke :write [12 26] INFO [2022-03-13 17:10:15,277] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,278] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:15,278] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,278] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:15,278] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,278] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:15,279] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,279] jepsen worker 34 - jepsen.util 754 :invoke :write [11 5] INFO [2022-03-13 17:10:15,280] jepsen worker 6 - jepsen.util 546 :invoke :write [2 19] INFO [2022-03-13 17:10:15,291] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:15,291] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:15,294] jepsen worker 70 - jepsen.util 700 :invoke :write [23 31] INFO [2022-03-13 17:10:15,444] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,444] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,445] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,445] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:15,445] jepsen worker 61 - jepsen.util 601 :invoke :write [20 46] INFO [2022-03-13 17:10:15,446] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,447] jepsen worker 53 - jepsen.util 503 :invoke :write [17 32] INFO [2022-03-13 17:10:15,447] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,451] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:10:15,451] jepsen worker 26 - jepsen.util 476 :invoke :write [8 43] INFO [2022-03-13 17:10:15,451] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:15,452] jepsen worker 42 - jepsen.util 402 :invoke :write [14 45] INFO [2022-03-13 17:10:15,460] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,460] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,463] jepsen worker 35 - jepsen.util 665 :invoke :read [11 nil] INFO [2022-03-13 17:10:15,463] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:15,556] jepsen worker 28 - jepsen.util 1288 :info :write [9 12] :RequestTimeoutException INFO [2022-03-13 17:10:15,557] jepsen worker 28 - jepsen.util 1378 :invoke :write [9 12] INFO [2022-03-13 17:10:15,566] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,566] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,567] jepsen worker 51 - jepsen.util 591 :invoke :write [17 35] INFO [2022-03-13 17:10:15,567] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:10:15,587] jepsen worker 39 - jepsen.util 1299 :info :write [13 25] :SyncWriteAmbiguous INFO [2022-03-13 17:10:15,588] jepsen worker 39 - jepsen.util 1389 :invoke :write [13 45] WARN [2022-03-13 17:10:15,615] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14357021,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":14357021,"total_server_duration_us":27,"operation_id":"0x37f6","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14358165},{"operation_name":"upsert","last_dispatch_duration_us":8898887,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9743485,"total_server_duration_us":357507,"encode_duration_us":18765,"operation_id":"0x37d8","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10009924},{"operation_name":"upsert","last_dispatch_duration_us":17197440,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":17662316,"total_server_duration_us":8698251,"encode_duration_us":24700,"operation_id":"0x36ab","timeout_ms":10000,"last_server_duration_us":8505059,"total_duration_us":10007783},{"operation_name":"upsert","last_dispatch_duration_us":7990476,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7990476,"total_server_duration_us":21649,"encode_duration_us":18943,"operation_id":"0x3905","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10005555},{"operation_name":"upsert","last_dispatch_duration_us":9116990,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10257947,"total_server_duration_us":9501518,"encode_duration_us":17572,"operation_id":"0x37a2","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10003712},{"operation_name":"upsert","last_dispatch_duration_us":9497329,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9497329,"total_server_duration_us":9473718,"encode_duration_us":28211,"operation_id":"0x36a7","timeout_ms":10000,"last_server_duration_us":9473718,"total_duration_us":9503355},{"operation_name":"upsert","last_dispatch_duration_us":9331835,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9331835,"total_server_duration_us":9300061,"encode_duration_us":19561,"operation_id":"0x36b5","timeout_ms":10000,"last_server_duration_us":9300061,"total_duration_us":9336096},{"operation_name":"get","last_dispatch_duration_us":9159502,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9159502,"total_server_duration_us":43,"operation_id":"0x3940","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9160626},{"operation_name":"get","last_dispatch_duration_us":8263202,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8263202,"total_server_duration_us":62,"operation_id":"0x3728","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":8264183},{"operation_name":"upsert","last_dispatch_duration_us":7551093,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7551093,"total_server_duration_us":7537907,"encode_duration_us":17474,"operation_id":"0x387f","timeout_ms":10000,"last_server_duration_us":7537907,"total_duration_us":7554638}],"total_count":16}} INFO [2022-03-13 17:10:15,692] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,693] jepsen worker 35 - jepsen.util 665 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,694] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,694] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:10:15,695] jepsen worker 14 - jepsen.util 284 :invoke :write [4 24] INFO [2022-03-13 17:10:15,695] jepsen worker 35 - jepsen.util 665 :invoke :write [11 24] WARN [2022-03-13 17:10:15,715] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9116990,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10257947,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9501518,"encode_duration_us":17572,"operation_id":"0x37a2","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10003712},{"operation_name":"upsert","last_dispatch_duration_us":17302817,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":18077866,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9506203,"encode_duration_us":16940,"operation_id":"0x34b6","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10003897},{"operation_name":"upsert","last_dispatch_duration_us":17197440,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17662316,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":8698251,"encode_duration_us":24700,"operation_id":"0x36ab","timeout_ms":10000,"last_server_duration_us":8505059,"total_duration_us":10007783},{"operation_name":"upsert","last_dispatch_duration_us":9057349,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9894395,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9548617,"encode_duration_us":17414,"operation_id":"0x33f2","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10006249},{"operation_name":"upsert","last_dispatch_duration_us":17084202,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":17736753,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9415505,"encode_duration_us":17062,"operation_id":"0x3560","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10005571}],"total_count":5}} INFO [2022-03-13 17:10:15,756] jepsen worker 21 - jepsen.util 741 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,756] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:15,801] jepsen worker 77 - jepsen.util 707 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,801] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,802] jepsen worker 77 - jepsen.util 707 :invoke :write [25 41] INFO [2022-03-13 17:10:15,803] jepsen worker 79 - jepsen.util 889 :invoke :write [26 9] INFO [2022-03-13 17:10:15,818] jepsen worker 69 - jepsen.util 609 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,819] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:10:15,860] jepsen worker 76 - jepsen.util 616 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,862] jepsen worker 76 - jepsen.util 616 :invoke :write [25 12] INFO [2022-03-13 17:10:15,921] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,921] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:15,932] jepsen worker 61 - jepsen.util 601 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:15,933] jepsen worker 61 - jepsen.util 601 :invoke :write [20 2] INFO [2022-03-13 17:10:15,968] jepsen worker 2 - jepsen.util 1442 :info :write [0 28] :RequestTimeoutException INFO [2022-03-13 17:10:15,968] jepsen worker 2 - jepsen.util 1532 :invoke :write [0 12] INFO [2022-03-13 17:10:16,036] jepsen worker 43 - jepsen.util 403 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,037] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:16,044] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:16,044] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,045] jepsen worker 21 - jepsen.util 741 :invoke :write [7 4] INFO [2022-03-13 17:10:16,045] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:10:16,048] jepsen worker 79 - jepsen.util 889 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,048] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:16,092] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2697947214076246}, :ns_1@172.23.105.187 {:progress 0.3625730994152047}, :ns_1@172.23.107.227 {:progress 0.2697947214076246}, :ns_1@172.23.107.228 {:progress 0.2647058823529411}} INFO [2022-03-13 17:10:16,092] jepsen worker nemesis - couchbase.util Rebalance check count: 128 INFO [2022-03-13 17:10:16,092] jepsen worker nemesis - couchbase.util Rebalance stuck count: 72 INFO [2022-03-13 17:10:16,112] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,112] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,113] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:10:16,114] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:10:16,125] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,125] jepsen worker 76 - jepsen.util 616 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,126] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:10:16,126] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:16,133] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,134] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:16,143] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,144] jepsen worker 76 - jepsen.util 616 :invoke :write [25 43] INFO [2022-03-13 17:10:16,146] jepsen worker 53 - jepsen.util 503 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,147] jepsen worker 53 - jepsen.util 503 :invoke :write [17 43] INFO [2022-03-13 17:10:16,170] jepsen worker 21 - jepsen.util 741 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,171] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:16,238] jepsen worker 36 - jepsen.util 1386 :info :write [12 20] :RequestTimeoutException INFO [2022-03-13 17:10:16,239] jepsen worker 36 - jepsen.util 1476 :invoke :write [12 5] INFO [2022-03-13 17:10:16,281] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,281] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:16,324] jepsen worker 49 - jepsen.util 1129 :info :write [16 47] :RequestTimeoutException INFO [2022-03-13 17:10:16,325] jepsen worker 49 - jepsen.util 1219 :invoke :read [16 nil] INFO [2022-03-13 17:10:16,338] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,339] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:16,352] jepsen worker 12 - jepsen.util 732 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,353] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:16,382] jepsen worker 85 - jepsen.util 1345 :ok :read [28 44] INFO [2022-03-13 17:10:16,383] jepsen worker 85 - jepsen.util 1345 :invoke :write [28 42] INFO [2022-03-13 17:10:16,408] jepsen worker 76 - jepsen.util 616 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,408] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,408] jepsen worker 76 - jepsen.util 616 :invoke :write [25 21] INFO [2022-03-13 17:10:16,409] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:10:16,490] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:16,490] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:16,491] jepsen worker 21 - jepsen.util 741 :invoke :write [7 23] INFO [2022-03-13 17:10:16,491] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:10:16,650] jepsen worker 16 - jepsen.util 1276 :info :write [5 31] :RequestTimeoutException INFO [2022-03-13 17:10:16,651] jepsen worker 16 - jepsen.util 1366 :invoke :read [5 nil] INFO [2022-03-13 17:10:16,669] jepsen worker 19 - jepsen.util 1099 :ok :read [6 16] INFO [2022-03-13 17:10:16,670] jepsen worker 19 - jepsen.util 1099 :invoke :write [6 3] INFO [2022-03-13 17:10:16,677] jepsen worker 55 - jepsen.util 1315 :ok :read [18 21] INFO [2022-03-13 17:10:16,678] jepsen worker 55 - jepsen.util 1315 :invoke :write [18 29] INFO [2022-03-13 17:10:16,870] jepsen worker 37 - jepsen.util 1387 :info :write [12 42] :RequestTimeoutException INFO [2022-03-13 17:10:16,871] jepsen worker 37 - jepsen.util 1477 :invoke :read [12 nil] INFO [2022-03-13 17:10:17,028] jepsen worker 81 - jepsen.util 1521 :info :write [27 31] :RequestTimeoutException INFO [2022-03-13 17:10:17,028] jepsen worker 81 - jepsen.util 1611 :invoke :write [27 3] INFO [2022-03-13 17:10:17,098] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2697947214076246}, :ns_1@172.23.105.187 {:progress 0.3625730994152047}, :ns_1@172.23.107.227 {:progress 0.2697947214076246}, :ns_1@172.23.107.228 {:progress 0.2647058823529411}} INFO [2022-03-13 17:10:17,098] jepsen worker nemesis - couchbase.util Rebalance check count: 129 INFO [2022-03-13 17:10:17,098] jepsen worker nemesis - couchbase.util Rebalance stuck count: 73 WARN [2022-03-13 17:10:17,117] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17533341,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26500455,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9192391,"encode_duration_us":21563,"operation_id":"0x32ff","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10007103}],"total_count":1}} INFO [2022-03-13 17:10:17,165] jepsen worker 31 - jepsen.util 1291 :ok :read [10 :nil] INFO [2022-03-13 17:10:17,166] jepsen worker 31 - jepsen.util 1291 :invoke :read [10 nil] INFO [2022-03-13 17:10:17,178] jepsen worker 64 - jepsen.util 1144 :ok :read [21 :nil] INFO [2022-03-13 17:10:17,179] jepsen worker 64 - jepsen.util 1144 :invoke :read [21 nil] INFO [2022-03-13 17:10:17,179] jepsen worker 18 - jepsen.util 1098 :ok :read [6 16] INFO [2022-03-13 17:10:17,180] jepsen worker 18 - jepsen.util 1098 :invoke :read [6 nil] INFO [2022-03-13 17:10:17,192] jepsen worker 45 - jepsen.util 1485 :ok :read [15 48] INFO [2022-03-13 17:10:17,192] jepsen worker 45 - jepsen.util 1485 :invoke :write [15 26] WARN [2022-03-13 17:10:17,212] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9069065,"last_local_id":"A312307A00000004/0000000095638474","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9709986,"last_local_socket":"172.23.104.79:33498","total_server_duration_us":9459146,"encode_duration_us":23272,"operation_id":"0x36f8","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10003168},{"operation_name":"upsert","last_dispatch_duration_us":17435907,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18023724,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9319822,"encode_duration_us":49279,"operation_id":"0x345d","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10009718}],"total_count":2}} WARN [2022-03-13 17:10:17,216] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18090884,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18090884,"total_server_duration_us":49,"operation_id":"0x3472","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":23260715},{"operation_name":"get","last_dispatch_duration_us":17383306,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17383306,"total_server_duration_us":32,"operation_id":"0x3652","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":17384713},{"operation_name":"get","last_dispatch_duration_us":16460225,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":16460225,"total_server_duration_us":84,"operation_id":"0x369b","timeout_ms":30000,"last_server_duration_us":84,"total_duration_us":16462636},{"operation_name":"get","last_dispatch_duration_us":10665502,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10665502,"total_server_duration_us":32,"operation_id":"0x3864","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":10667388},{"operation_name":"upsert","last_dispatch_duration_us":31064,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":4564127,"total_server_duration_us":201551,"encode_duration_us":19242,"operation_id":"0x3922","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10009739},{"operation_name":"upsert","last_dispatch_duration_us":9160328,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10151719,"total_server_duration_us":9312929,"encode_duration_us":18490,"operation_id":"0x3767","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10009257},{"operation_name":"upsert","last_dispatch_duration_us":8910588,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10080531,"total_server_duration_us":372656,"encode_duration_us":17602,"operation_id":"0x3769","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10009090},{"operation_name":"upsert","last_dispatch_duration_us":48344,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":950938,"total_server_duration_us":593542,"encode_duration_us":23460,"operation_id":"0x3717","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10008311},{"operation_name":"upsert","last_dispatch_duration_us":30471,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9478998,"total_server_duration_us":234393,"encode_duration_us":17984,"operation_id":"0x37ec","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10007036},{"operation_name":"get","last_dispatch_duration_us":9885966,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9885966,"total_server_duration_us":27,"operation_id":"0x389d","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9888386}],"total_count":26}} WARN [2022-03-13 17:10:17,217] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9160328,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10151719,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9312929,"encode_duration_us":18490,"operation_id":"0x3767","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10009257},{"operation_name":"upsert","last_dispatch_duration_us":26353852,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26353852,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9028877,"encode_duration_us":18470,"operation_id":"0x334b","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10010636}],"total_count":2}} WARN [2022-03-13 17:10:17,217] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11249768,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11249768,"total_server_duration_us":175,"operation_id":"0x3907","timeout_ms":30000,"last_server_duration_us":175,"total_duration_us":11250804},{"operation_name":"upsert","last_dispatch_duration_us":31033,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":718130,"total_server_duration_us":428199,"encode_duration_us":17062,"operation_id":"0x3956","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10009332},{"operation_name":"upsert","last_dispatch_duration_us":8961114,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":15321284,"total_server_duration_us":66099,"encode_duration_us":21511,"operation_id":"0x36b7","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10008240},{"operation_name":"upsert","last_dispatch_duration_us":32634,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9762317,"total_server_duration_us":210635,"encode_duration_us":19507,"operation_id":"0x3892","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10007470},{"operation_name":"upsert","last_dispatch_duration_us":10428826,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10428826,"total_server_duration_us":20282,"encode_duration_us":18431,"operation_id":"0x3941","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10004133},{"operation_name":"upsert","last_dispatch_duration_us":41120,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9735620,"total_server_duration_us":223938,"encode_duration_us":18853,"operation_id":"0x3893","timeout_ms":10000,"last_server_duration_us":28218,"total_duration_us":10003647},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":16850,"operation_id":"0x3998","timeout_ms":10000,"total_duration_us":10003428},{"operation_name":"upsert","last_dispatch_duration_us":44281,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":223917,"total_server_duration_us":106642,"encode_duration_us":17956,"operation_id":"0x38cc","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10003064},{"operation_name":"upsert","last_dispatch_duration_us":9314727,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9314727,"total_server_duration_us":9054355,"encode_duration_us":17956,"operation_id":"0x3887","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9319767},{"operation_name":"upsert","last_dispatch_duration_us":4873059,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4873059,"total_server_duration_us":271657,"encode_duration_us":32253,"operation_id":"0x3993","timeout_ms":10000,"last_server_duration_us":271657,"total_duration_us":4878750}],"total_count":27}} WARN [2022-03-13 17:10:17,218] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":32335,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1060545,"total_server_duration_us":601485,"encode_duration_us":23547,"operation_id":"0x385c","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10010961},{"operation_name":"upsert","last_dispatch_duration_us":31912,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":993768,"total_server_duration_us":599579,"encode_duration_us":17612,"operation_id":"0x382e","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10009974},{"operation_name":"upsert","last_dispatch_duration_us":39512,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1047391,"total_server_duration_us":630217,"encode_duration_us":19107,"operation_id":"0x3890","timeout_ms":10000,"last_server_duration_us":27764,"total_duration_us":10009140},{"operation_name":"upsert","last_dispatch_duration_us":1931573,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1931573,"total_server_duration_us":174309,"encode_duration_us":23314,"operation_id":"0x37bd","timeout_ms":10000,"last_server_duration_us":174309,"total_duration_us":10008501},{"operation_name":"upsert","last_dispatch_duration_us":167180,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":748241,"total_server_duration_us":113983,"encode_duration_us":18244,"operation_id":"0x3865","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10008339},{"operation_name":"upsert","last_dispatch_duration_us":169083,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":622688,"total_server_duration_us":92423,"encode_duration_us":17632,"operation_id":"0x3874","timeout_ms":10000,"last_server_duration_us":26064,"total_duration_us":10007334},{"operation_name":"upsert","last_dispatch_duration_us":2186451,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2186451,"total_server_duration_us":136828,"encode_duration_us":18030,"operation_id":"0x37be","timeout_ms":10000,"last_server_duration_us":136828,"total_duration_us":10006538},{"operation_name":"upsert","last_dispatch_duration_us":30432,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":3711053,"total_server_duration_us":438441,"encode_duration_us":18610,"operation_id":"0x37c4","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10006270},{"operation_name":"upsert","last_dispatch_duration_us":8105144,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13889849,"total_server_duration_us":210756,"encode_duration_us":28128,"operation_id":"0x36f4","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10004458},{"operation_name":"upsert","last_dispatch_duration_us":31014,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1094621,"total_server_duration_us":629250,"encode_duration_us":18450,"operation_id":"0x3765","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10001399}],"total_count":57}} WARN [2022-03-13 17:10:17,313] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17256865,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":17256865,"total_server_duration_us":27,"operation_id":"0x379f","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":17258926},{"operation_name":"get","last_dispatch_duration_us":16587480,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16587480,"total_server_duration_us":15,"operation_id":"0x37df","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":16587843},{"operation_name":"get","last_dispatch_duration_us":13878819,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":13878819,"total_server_duration_us":27,"operation_id":"0x3891","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":13880946},{"operation_name":"get","last_dispatch_duration_us":12353128,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12353128,"total_server_duration_us":117,"operation_id":"0x369a","timeout_ms":30000,"last_server_duration_us":117,"total_duration_us":12354281},{"operation_name":"upsert","last_dispatch_duration_us":571264,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":4136425,"total_server_duration_us":261851,"encode_duration_us":18193,"operation_id":"0x3957","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10010632},{"operation_name":"upsert","last_dispatch_duration_us":18209881,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":18209881,"total_server_duration_us":9054355,"encode_duration_us":17884,"operation_id":"0x3766","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10003913},{"operation_name":"upsert","last_dispatch_duration_us":9069065,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9709986,"total_server_duration_us":9459146,"encode_duration_us":23272,"operation_id":"0x36f8","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10003168},{"operation_name":"get","last_dispatch_duration_us":9973929,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9973929,"total_server_duration_us":621,"operation_id":"0x399e","timeout_ms":30000,"last_server_duration_us":621,"total_duration_us":9974220},{"operation_name":"upsert","last_dispatch_duration_us":6253414,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":7682578,"total_server_duration_us":278419,"encode_duration_us":21249,"operation_id":"0x3901","timeout_ms":10000,"last_server_duration_us":77150,"total_duration_us":7709478},{"operation_name":"upsert","last_dispatch_duration_us":6501410,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6501410,"total_server_duration_us":112804,"encode_duration_us":19486,"operation_id":"0x3982","timeout_ms":10000,"last_server_duration_us":112804,"total_duration_us":6505260}],"total_count":26}} WARN [2022-03-13 17:10:17,316] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13388801,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":13388801,"total_server_duration_us":22720,"encode_duration_us":18347,"operation_id":"0x387a","timeout_ms":10000,"last_server_duration_us":22720,"total_duration_us":10010480},{"operation_name":"upsert","last_dispatch_duration_us":9101618,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9101618,"total_server_duration_us":24325,"encode_duration_us":19396,"operation_id":"0x38cd","timeout_ms":10000,"last_server_duration_us":24325,"total_duration_us":10007784},{"operation_name":"upsert","last_dispatch_duration_us":15615541,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":15615541,"total_server_duration_us":9030999,"encode_duration_us":19893,"operation_id":"0x37ed","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10004789},{"operation_name":"upsert","last_dispatch_duration_us":9066024,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9066024,"total_server_duration_us":9055417,"encode_duration_us":24113,"operation_id":"0x371b","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":9069488},{"operation_name":"upsert","last_dispatch_duration_us":9052477,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9052477,"total_server_duration_us":9039489,"encode_duration_us":17635,"operation_id":"0x3951","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":9058954},{"operation_name":"upsert","last_dispatch_duration_us":9052134,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9052134,"total_server_duration_us":9037366,"encode_duration_us":19717,"operation_id":"0x396b","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":9056050},{"operation_name":"get","last_dispatch_duration_us":7706562,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7706562,"total_server_duration_us":181700,"operation_id":"0x3a33","timeout_ms":30000,"last_server_duration_us":181700,"total_duration_us":7708743},{"operation_name":"get","last_dispatch_duration_us":7602304,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7602304,"total_server_duration_us":38,"operation_id":"0x378e","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":7602579},{"operation_name":"upsert","last_dispatch_duration_us":6430099,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7064756,"total_server_duration_us":311638,"encode_duration_us":20539,"operation_id":"0x37b8","timeout_ms":10000,"last_server_duration_us":129938,"total_duration_us":7081822},{"operation_name":"upsert","last_dispatch_duration_us":6247438,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6247438,"total_server_duration_us":153677,"encode_duration_us":18885,"operation_id":"0x3804","timeout_ms":10000,"last_server_duration_us":153677,"total_duration_us":6252451}],"total_count":49}} WARN [2022-03-13 17:10:17,318] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":2186451,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":2186451,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":136828,"encode_duration_us":18030,"operation_id":"0x37be","timeout_ms":10000,"last_server_duration_us":136828,"total_duration_us":10006538},{"operation_name":"upsert","last_dispatch_duration_us":1931573,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":1931573,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":174309,"encode_duration_us":23314,"operation_id":"0x37bd","timeout_ms":10000,"last_server_duration_us":174309,"total_duration_us":10008501}],"total_count":2}} INFO [2022-03-13 17:10:17,326] jepsen worker 32 - jepsen.util 1022 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:10:17,326] jepsen worker 32 - jepsen.util 1112 :invoke :read [10 nil] INFO [2022-03-13 17:10:17,368] jepsen worker 73 - jepsen.util 1333 :info :write [24 13] :SyncWriteAmbiguous INFO [2022-03-13 17:10:17,369] jepsen worker 73 - jepsen.util 1423 :invoke :read [24 nil] WARN [2022-03-13 17:10:17,416] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15615541,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15615541,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9030999,"encode_duration_us":19893,"operation_id":"0x37ed","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10004789}],"total_count":1}} INFO [2022-03-13 17:10:17,429] jepsen worker 63 - jepsen.util 1323 :ok :read [21 :nil] INFO [2022-03-13 17:10:17,429] jepsen worker 63 - jepsen.util 1323 :invoke :write [21 48] INFO [2022-03-13 17:10:17,449] jepsen worker 32 - jepsen.util 1112 :ok :read [10 :nil] INFO [2022-03-13 17:10:17,450] jepsen worker 32 - jepsen.util 1112 :invoke :write [10 1] INFO [2022-03-13 17:10:17,461] jepsen worker 74 - jepsen.util 1604 :ok :read [24 24] INFO [2022-03-13 17:10:17,462] jepsen worker 74 - jepsen.util 1604 :invoke :write [24 10] INFO [2022-03-13 17:10:17,676] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:17,677] jepsen worker 44 - jepsen.util 494 :invoke :write [14 36] INFO [2022-03-13 17:10:17,736] jepsen worker 40 - jepsen.util 1300 :info :write [13 1] :RequestTimeoutException INFO [2022-03-13 17:10:17,737] jepsen worker 40 - jepsen.util 1390 :invoke :read [13 nil] INFO [2022-03-13 17:10:17,999] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,000] jepsen worker 79 - jepsen.util 889 :invoke :write [26 22] INFO [2022-03-13 17:10:18,060] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:18,060] jepsen worker 77 - jepsen.util 707 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,061] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:18,061] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:18,062] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,063] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:18,080] jepsen worker 76 - jepsen.util 616 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,081] jepsen worker 76 - jepsen.util 616 :invoke :write [25 32] INFO [2022-03-13 17:10:18,105] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2697947214076246}, :ns_1@172.23.105.187 {:progress 0.3625730994152047}, :ns_1@172.23.107.227 {:progress 0.2697947214076246}, :ns_1@172.23.107.228 {:progress 0.2647058823529411}} INFO [2022-03-13 17:10:18,105] jepsen worker nemesis - couchbase.util Rebalance check count: 130 INFO [2022-03-13 17:10:18,105] jepsen worker nemesis - couchbase.util Rebalance stuck count: 74 INFO [2022-03-13 17:10:18,115] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,116] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:18,147] jepsen worker 89 - jepsen.util 1529 :info :write [29 6] :RequestTimeoutException INFO [2022-03-13 17:10:18,148] jepsen worker 89 - jepsen.util 1619 :invoke :write [29 36] INFO [2022-03-13 17:10:18,428] jepsen worker 79 - jepsen.util 889 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,428] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:18,435] jepsen worker 61 - jepsen.util 601 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,435] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:18,468] jepsen worker 76 - jepsen.util 616 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,468] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,469] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:18,470] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:10:18,470] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,471] jepsen worker 76 - jepsen.util 616 :invoke :write [25 16] INFO [2022-03-13 17:10:18,498] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,499] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:10:18,515] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,516] jepsen worker 69 - jepsen.util 609 :invoke :write [23 12] INFO [2022-03-13 17:10:18,557] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,558] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:10:18,589] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,589] jepsen worker 21 - jepsen.util 741 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,590] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:10:18,590] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:18,621] jepsen worker 69 - jepsen.util 609 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,622] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:18,646] jepsen worker 53 - jepsen.util 503 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,647] jepsen worker 53 - jepsen.util 503 :invoke :write [17 41] INFO [2022-03-13 17:10:18,664] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,665] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:18,720] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,720] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:18,721] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:18,721] jepsen worker 21 - jepsen.util 741 :invoke :write [7 33] INFO [2022-03-13 17:10:18,760] jepsen worker 6 - jepsen.util 546 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,760] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:18,769] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,769] jepsen worker 76 - jepsen.util 616 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,770] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:18,771] jepsen worker 79 - jepsen.util 889 :invoke :write [26 24] INFO [2022-03-13 17:10:18,771] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,772] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:18,773] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,774] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:18,775] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,775] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:18,776] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,777] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:10:18,791] jepsen worker 78 - jepsen.util 438 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,791] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:18,792] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:18,792] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:18,803] jepsen worker 11 - jepsen.util 1181 :info :write [3 29] :RequestTimeoutException INFO [2022-03-13 17:10:18,804] jepsen worker 11 - jepsen.util 1271 :invoke :read [3 nil] INFO [2022-03-13 17:10:18,832] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,832] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,832] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,833] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:10:18,833] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:18,834] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:10:18,856] jepsen worker 79 - jepsen.util 889 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:18,857] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:18,881] jepsen worker 9 - jepsen.util 1269 :info :write [3 0] :SyncWriteAmbiguous INFO [2022-03-13 17:10:18,882] jepsen worker 9 - jepsen.util 1359 :invoke :write [3 15] INFO [2022-03-13 17:10:18,956] jepsen worker 5 - jepsen.util 1445 :info :write [1 14] :RequestTimeoutException INFO [2022-03-13 17:10:18,957] jepsen worker 5 - jepsen.util 1535 :invoke :read [1 nil] INFO [2022-03-13 17:10:18,983] jepsen worker 5 - jepsen.util 1535 :ok :read [1 42] INFO [2022-03-13 17:10:18,984] jepsen worker 5 - jepsen.util 1535 :invoke :read [1 nil] INFO [2022-03-13 17:10:19,005] jepsen worker 21 - jepsen.util 741 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,005] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,006] jepsen worker 21 - jepsen.util 741 :invoke :write [7 13] INFO [2022-03-13 17:10:19,006] jepsen worker 69 - jepsen.util 609 :invoke :write [23 33] INFO [2022-03-13 17:10:19,008] jepsen worker 5 - jepsen.util 1535 :ok :read [1 42] INFO [2022-03-13 17:10:19,009] jepsen worker 5 - jepsen.util 1535 :invoke :read [1 nil] INFO [2022-03-13 17:10:19,037] jepsen worker 5 - jepsen.util 1535 :ok :read [1 42] INFO [2022-03-13 17:10:19,038] jepsen worker 5 - jepsen.util 1535 :invoke :write [1 21] INFO [2022-03-13 17:10:19,064] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,064] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,064] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:19,065] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:19,066] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,066] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,066] jepsen worker 79 - jepsen.util 889 :invoke :write [26 14] INFO [2022-03-13 17:10:19,067] jepsen worker 76 - jepsen.util 616 :invoke :write [25 11] INFO [2022-03-13 17:10:19,068] jepsen worker 70 - jepsen.util 700 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,069] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:19,070] jepsen worker 30 - jepsen.util 1110 :info :write [10 18] :RequestTimeoutException INFO [2022-03-13 17:10:19,071] jepsen worker 30 - jepsen.util 1200 :invoke :read [10 nil] INFO [2022-03-13 17:10:19,111] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2697947214076246}, :ns_1@172.23.105.187 {:progress 0.3625730994152047}, :ns_1@172.23.107.227 {:progress 0.2697947214076246}, :ns_1@172.23.107.228 {:progress 0.2647058823529411}} INFO [2022-03-13 17:10:19,111] jepsen worker nemesis - couchbase.util Rebalance check count: 131 INFO [2022-03-13 17:10:19,111] jepsen worker nemesis - couchbase.util Rebalance stuck count: 75 INFO [2022-03-13 17:10:19,127] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,127] jepsen worker 8 - jepsen.util 638 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,127] jepsen worker 13 - jepsen.util 553 :invoke :write [4 37] INFO [2022-03-13 17:10:19,128] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:19,128] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,129] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:10:19,218] jepsen worker 56 - jepsen.util 1136 :info :write [18 30] :RequestTimeoutException INFO [2022-03-13 17:10:19,219] jepsen worker 56 - jepsen.util 1226 :invoke :write [18 8] INFO [2022-03-13 17:10:19,283] jepsen worker 58 - jepsen.util 1138 :info :write [19 3] :RequestTimeoutException INFO [2022-03-13 17:10:19,284] jepsen worker 58 - jepsen.util 1228 :invoke :read [19 nil] INFO [2022-03-13 17:10:19,344] jepsen worker 50 - jepsen.util 1310 :info :write [16 48] :RequestTimeoutException INFO [2022-03-13 17:10:19,345] jepsen worker 50 - jepsen.util 1400 :invoke :read [16 nil] INFO [2022-03-13 17:10:19,950] jepsen worker 47 - jepsen.util 1307 :info :write [15 6] :RequestTimeoutException INFO [2022-03-13 17:10:19,950] jepsen worker 47 - jepsen.util 1397 :invoke :write [15 37] INFO [2022-03-13 17:10:19,970] jepsen worker 79 - jepsen.util 889 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,971] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:19,976] jepsen worker 34 - jepsen.util 754 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:19,977] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:10:20,020] jepsen worker 80 - jepsen.util 620 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,020] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:20,074] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,074] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:20,098] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,099] jepsen worker 60 - jepsen.util 600 :invoke :write [20 17] INFO [2022-03-13 17:10:20,111] jepsen worker 71 - jepsen.util 431 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,114] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:20,117] jepsen worker 76 - jepsen.util 616 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,117] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:20,117] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2697947214076246}, :ns_1@172.23.105.187 {:progress 0.3625730994152047}, :ns_1@172.23.107.227 {:progress 0.2697947214076246}, :ns_1@172.23.107.228 {:progress 0.2647058823529411}} INFO [2022-03-13 17:10:20,117] jepsen worker nemesis - couchbase.util Rebalance check count: 132 INFO [2022-03-13 17:10:20,117] jepsen worker nemesis - couchbase.util Rebalance stuck count: 76 INFO [2022-03-13 17:10:20,118] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,119] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,119] jepsen worker 79 - jepsen.util 889 :invoke :write [26 38] INFO [2022-03-13 17:10:20,119] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] INFO [2022-03-13 17:10:20,160] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,160] jepsen worker 52 - jepsen.util 502 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,160] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:20,160] jepsen worker 22 - jepsen.util 562 :invoke :write [7 33] INFO [2022-03-13 17:10:20,161] jepsen worker 52 - jepsen.util 502 :invoke :write [17 12] INFO [2022-03-13 17:10:20,161] jepsen worker 78 - jepsen.util 438 :invoke :write [26 31] INFO [2022-03-13 17:10:20,184] jepsen worker 21 - jepsen.util 741 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,185] jepsen worker 21 - jepsen.util 741 :invoke :write [7 39] INFO [2022-03-13 17:10:20,249] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,250] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:20,276] jepsen worker 69 - jepsen.util 609 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,277] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:10:20,320] jepsen worker 79 - jepsen.util 889 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,321] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,321] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:20,321] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:20,323] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,323] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:20,324] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,325] jepsen worker 76 - jepsen.util 616 :invoke :write [25 45] INFO [2022-03-13 17:10:20,326] jepsen worker 10 - jepsen.util 1180 :info :write [3 25] :RequestTimeoutException INFO [2022-03-13 17:10:20,326] jepsen worker 10 - jepsen.util 1270 :invoke :read [3 nil] INFO [2022-03-13 17:10:20,332] jepsen worker 44 - jepsen.util 494 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,332] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,333] jepsen worker 44 - jepsen.util 494 :invoke :write [14 34] INFO [2022-03-13 17:10:20,333] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,333] jepsen worker 70 - jepsen.util 700 :invoke :write [23 7] INFO [2022-03-13 17:10:20,336] jepsen worker 6 - jepsen.util 546 :invoke :write [2 10] INFO [2022-03-13 17:10:20,339] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,340] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:20,342] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,343] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:20,378] jepsen worker 25 - jepsen.util 385 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,379] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:20,452] jepsen worker 21 - jepsen.util 741 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,452] jepsen worker 21 - jepsen.util 741 :invoke :write [7 9] INFO [2022-03-13 17:10:20,459] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,459] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:10:20,479] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,479] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,480] jepsen worker 71 - jepsen.util 431 :invoke :write [23 47] INFO [2022-03-13 17:10:20,485] jepsen worker 34 - jepsen.util 754 :invoke :write [11 1] INFO [2022-03-13 17:10:20,513] jepsen worker 76 - jepsen.util 616 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,513] jepsen worker 76 - jepsen.util 616 :invoke :write [25 1] INFO [2022-03-13 17:10:20,520] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,520] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,520] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,521] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:20,522] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:10:20,529] jepsen worker 7 - jepsen.util 457 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,529] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,530] jepsen worker 7 - jepsen.util 457 :invoke :write [2 11] INFO [2022-03-13 17:10:20,530] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:20,534] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:20,561] jepsen worker 72 - jepsen.util 1242 :info :write [24 43] :RequestTimeoutException INFO [2022-03-13 17:10:20,561] jepsen worker 72 - jepsen.util 1332 :invoke :read [24 nil] INFO [2022-03-13 17:10:20,596] jepsen worker 44 - jepsen.util 494 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,597] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:20,616] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,617] jepsen worker 79 - jepsen.util 889 :invoke :write [26 14] INFO [2022-03-13 17:10:20,621] jepsen worker 21 - jepsen.util 741 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,621] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,621] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,621] jepsen worker 21 - jepsen.util 741 :invoke :write [7 2] INFO [2022-03-13 17:10:20,622] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:20,622] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:20,842] jepsen worker 0 - jepsen.util 1350 :ok :read [0 41] INFO [2022-03-13 17:10:20,842] jepsen worker 0 - jepsen.util 1350 :invoke :write [0 39] INFO [2022-03-13 17:10:20,847] jepsen worker 59 - jepsen.util 1229 :ok :read [19 17] INFO [2022-03-13 17:10:20,848] jepsen worker 59 - jepsen.util 1229 :invoke :read [19 nil] INFO [2022-03-13 17:10:20,976] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,976] jepsen worker 13 - jepsen.util 553 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:20,977] jepsen worker 13 - jepsen.util 553 :invoke :write [4 28] INFO [2022-03-13 17:10:20,978] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:21,028] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,028] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,028] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,028] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:10:21,029] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,029] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:10:21,050] jepsen worker 76 - jepsen.util 616 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,051] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,096] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,096] jepsen worker 21 - jepsen.util 741 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,097] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:21,097] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,106] jepsen worker 6 - jepsen.util 546 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,107] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:21,123] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2727272727272727}, :ns_1@172.23.105.187 {:progress 0.3625730994152047}, :ns_1@172.23.107.227 {:progress 0.2727272727272727}, :ns_1@172.23.107.228 {:progress 0.2705882352941177}} INFO [2022-03-13 17:10:21,123] jepsen worker nemesis - couchbase.util Rebalance check count: 133 INFO [2022-03-13 17:10:21,123] jepsen worker nemesis - couchbase.util Rebalance stuck count: 76 INFO [2022-03-13 17:10:21,145] jepsen worker 24 - jepsen.util 474 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,145] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:21,146] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,146] jepsen worker 79 - jepsen.util 889 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,147] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,147] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:10:21,151] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,151] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,153] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,153] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,154] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,155] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,156] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,156] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,157] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,158] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,158] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,159] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,160] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,160] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,161] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,162] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,163] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,163] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,164] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,165] jepsen worker 76 - jepsen.util 616 :invoke :write [25 6] INFO [2022-03-13 17:10:21,199] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,200] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:10:21,223] jepsen worker 70 - jepsen.util 700 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,223] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:21,259] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,259] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:21,275] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:21,275] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,275] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,276] jepsen worker 21 - jepsen.util 741 :invoke :write [7 8] INFO [2022-03-13 17:10:21,276] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,277] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] INFO [2022-03-13 17:10:21,284] jepsen worker 22 - jepsen.util 562 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,285] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,285] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:21,286] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:21,333] jepsen worker 76 - jepsen.util 616 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,333] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,334] jepsen worker 79 - jepsen.util 889 :invoke :write [26 6] INFO [2022-03-13 17:10:21,334] jepsen worker 76 - jepsen.util 616 :invoke :write [25 21] INFO [2022-03-13 17:10:21,345] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,346] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:10:21,346] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,347] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:21,401] jepsen worker 71 - jepsen.util 431 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,402] jepsen worker 71 - jepsen.util 431 :invoke :write [23 3] INFO [2022-03-13 17:10:21,476] jepsen worker 52 - jepsen.util 502 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,477] jepsen worker 52 - jepsen.util 502 :invoke :write [17 47] INFO [2022-03-13 17:10:21,477] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,478] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:10:21,484] jepsen worker 76 - jepsen.util 616 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,485] jepsen worker 76 - jepsen.util 616 :invoke :write [25 35] INFO [2022-03-13 17:10:21,522] jepsen worker 34 - jepsen.util 754 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,522] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:21,522] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,522] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,523] jepsen worker 44 - jepsen.util 494 :invoke :write [14 9] INFO [2022-03-13 17:10:21,524] jepsen worker 34 - jepsen.util 754 :invoke :write [11 14] INFO [2022-03-13 17:10:21,525] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:21,525] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:21,617] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,617] jepsen worker 21 - jepsen.util 741 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,617] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:10:21,618] jepsen worker 21 - jepsen.util 741 :invoke :write [7 4] INFO [2022-03-13 17:10:21,622] jepsen worker 78 - jepsen.util 438 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,622] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,622] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,623] jepsen worker 78 - jepsen.util 438 :invoke :write [26 49] INFO [2022-03-13 17:10:21,624] jepsen worker 80 - jepsen.util 620 :invoke :write [26 38] INFO [2022-03-13 17:10:21,625] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:21,642] jepsen worker 79 - jepsen.util 889 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,643] jepsen worker 79 - jepsen.util 889 :invoke :write [26 48] INFO [2022-03-13 17:10:21,722] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,722] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,723] jepsen worker 48 - jepsen.util 1218 :info :write [16 45] :RequestTimeoutException INFO [2022-03-13 17:10:21,724] jepsen worker 48 - jepsen.util 1308 :invoke :read [16 nil] INFO [2022-03-13 17:10:21,749] jepsen worker 26 - jepsen.util 476 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,749] jepsen worker 26 - jepsen.util 476 :invoke :write [8 48] INFO [2022-03-13 17:10:21,774] jepsen worker 76 - jepsen.util 616 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,775] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,782] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,782] jepsen worker 71 - jepsen.util 431 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,782] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,782] jepsen worker 71 - jepsen.util 431 :invoke :write [23 6] INFO [2022-03-13 17:10:21,783] jepsen worker 6 - jepsen.util 546 :invoke :write [2 26] INFO [2022-03-13 17:10:21,783] jepsen worker 70 - jepsen.util 700 :invoke :write [23 44] INFO [2022-03-13 17:10:21,839] jepsen worker 79 - jepsen.util 889 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,839] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,840] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:10:21,841] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:21,843] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,844] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:21,904] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,905] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] INFO [2022-03-13 17:10:21,920] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,920] jepsen worker 13 - jepsen.util 553 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,920] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:21,920] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:21,921] jepsen worker 22 - jepsen.util 562 :invoke :write [7 1] INFO [2022-03-13 17:10:21,921] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:21,958] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:21,959] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:22,026] jepsen worker 4 - jepsen.util 1444 :info :write [1 16] :RequestTimeoutException INFO [2022-03-13 17:10:22,027] jepsen worker 4 - jepsen.util 1534 :invoke :read [1 nil] INFO [2022-03-13 17:10:22,032] jepsen worker 21 - jepsen.util 741 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,032] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,032] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,044] jepsen worker 21 - jepsen.util 741 :invoke :write [7 43] INFO [2022-03-13 17:10:22,044] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:10:22,032] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,044] jepsen worker 69 - jepsen.util 609 :invoke :write [23 42] INFO [2022-03-13 17:10:22,036] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,044] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:22,044] jepsen worker 79 - jepsen.util 889 :invoke :write [26 15] INFO [2022-03-13 17:10:22,047] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,050] jepsen worker 33 - jepsen.util 663 :invoke :write [11 37] INFO [2022-03-13 17:10:22,072] jepsen worker 4 - jepsen.util 1534 :ok :read [1 42] INFO [2022-03-13 17:10:22,073] jepsen worker 4 - jepsen.util 1534 :invoke :write [1 37] INFO [2022-03-13 17:10:22,093] jepsen worker 46 - jepsen.util 1396 :info :write [15 21] :SyncWriteAmbiguous INFO [2022-03-13 17:10:22,094] jepsen worker 46 - jepsen.util 1486 :invoke :read [15 nil] INFO [2022-03-13 17:10:22,115] jepsen worker 51 - jepsen.util 591 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,115] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,116] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:22,116] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,116] jepsen worker 51 - jepsen.util 591 :invoke :write [17 5] INFO [2022-03-13 17:10:22,117] jepsen worker 12 - jepsen.util 732 :invoke :write [4 42] INFO [2022-03-13 17:10:22,117] jepsen worker 77 - jepsen.util 707 :invoke :write [25 47] INFO [2022-03-13 17:10:22,122] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:22,128] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2756598240469208}, :ns_1@172.23.105.187 {:progress 0.3742690058479532}, :ns_1@172.23.107.227 {:progress 0.2756598240469208}, :ns_1@172.23.107.228 {:progress 0.2764705882352941}} INFO [2022-03-13 17:10:22,128] jepsen worker nemesis - couchbase.util Rebalance check count: 134 INFO [2022-03-13 17:10:22,128] jepsen worker nemesis - couchbase.util Rebalance stuck count: 76 INFO [2022-03-13 17:10:22,134] jepsen worker 44 - jepsen.util 494 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,134] jepsen worker 44 - jepsen.util 494 :invoke :write [14 35] INFO [2022-03-13 17:10:22,497] jepsen worker 82 - jepsen.util 1252 :info :write [27 42] :SyncWriteAmbiguous INFO [2022-03-13 17:10:22,497] jepsen worker 82 - jepsen.util 1342 :invoke :read [27 nil] INFO [2022-03-13 17:10:22,558] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:22,559] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:10:22,640] jepsen worker 88 - jepsen.util 1168 :info :write [29 27] :RequestTimeoutException INFO [2022-03-13 17:10:22,641] jepsen worker 88 - jepsen.util 1258 :invoke :read [29 nil] INFO [2022-03-13 17:10:22,663] jepsen worker 88 - jepsen.util 1258 :ok :read [29 6] INFO [2022-03-13 17:10:22,663] jepsen worker 88 - jepsen.util 1258 :invoke :read [29 nil] INFO [2022-03-13 17:10:22,684] jepsen worker 88 - jepsen.util 1258 :ok :read [29 6] INFO [2022-03-13 17:10:22,685] jepsen worker 88 - jepsen.util 1258 :invoke :read [29 nil] INFO [2022-03-13 17:10:22,694] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,695] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:10:22,703] jepsen worker 52 - jepsen.util 502 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,703] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,703] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,703] jepsen worker 62 - jepsen.util 422 :info :write [20 46] :RequestTimeoutException INFO [2022-03-13 17:10:22,704] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,704] jepsen worker 24 - jepsen.util 474 :invoke :write [8 46] INFO [2022-03-13 17:10:22,705] jepsen worker 76 - jepsen.util 616 :invoke :write [25 47] INFO [2022-03-13 17:10:22,706] jepsen worker 52 - jepsen.util 502 :invoke :write [17 35] INFO [2022-03-13 17:10:22,706] jepsen worker 8 - jepsen.util 638 :invoke :write [2 24] INFO [2022-03-13 17:10:22,713] jepsen worker 62 - jepsen.util 512 :invoke :write [20 25] INFO [2022-03-13 17:10:22,717] jepsen worker 88 - jepsen.util 1258 :ok :read [29 6] INFO [2022-03-13 17:10:22,718] jepsen worker 88 - jepsen.util 1258 :invoke :read [29 nil] INFO [2022-03-13 17:10:22,732] jepsen worker 21 - jepsen.util 741 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,733] jepsen worker 21 - jepsen.util 741 :invoke :write [7 36] INFO [2022-03-13 17:10:22,744] jepsen worker 88 - jepsen.util 1258 :ok :read [29 6] INFO [2022-03-13 17:10:22,744] jepsen worker 88 - jepsen.util 1258 :invoke :write [29 33] INFO [2022-03-13 17:10:22,767] jepsen worker 69 - jepsen.util 609 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,767] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:22,800] jepsen worker 33 - jepsen.util 663 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,800] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:22,844] jepsen worker 79 - jepsen.util 889 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,844] jepsen worker 79 - jepsen.util 889 :invoke :write [26 23] INFO [2022-03-13 17:10:22,857] jepsen worker 34 - jepsen.util 754 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,858] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:22,871] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,872] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:22,888] jepsen worker 7 - jepsen.util 457 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,888] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:22,935] jepsen worker 76 - jepsen.util 616 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,936] jepsen worker 76 - jepsen.util 616 :invoke :write [25 40] INFO [2022-03-13 17:10:22,951] jepsen worker 6 - jepsen.util 546 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,951] jepsen worker 6 - jepsen.util 546 :invoke :write [2 37] INFO [2022-03-13 17:10:22,959] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,959] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,959] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,959] jepsen worker 21 - jepsen.util 741 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,960] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:22,961] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:22,961] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:22,961] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:22,962] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,962] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:22,970] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,970] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:22,971] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,971] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,971] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:10:22,971] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:10:22,972] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:22,972] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:22,973] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:22,973] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,974] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:22,974] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:22,975] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:22,975] jepsen worker 21 - jepsen.util 741 :invoke :write [7 3] INFO [2022-03-13 17:10:22,975] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,976] jepsen worker 69 - jepsen.util 609 :invoke :write [23 10] INFO [2022-03-13 17:10:22,980] jepsen worker 84 - jepsen.util 1524 :info :write [28 9] :RequestTimeoutException INFO [2022-03-13 17:10:22,981] jepsen worker 84 - jepsen.util 1614 :invoke :write [28 26] INFO [2022-03-13 17:10:22,994] jepsen worker 80 - jepsen.util 620 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:22,998] jepsen worker 80 - jepsen.util 620 :invoke :write [26 5] INFO [2022-03-13 17:10:23,024] jepsen worker 20 - jepsen.util 1640 :info :write [6 14] :RequestTimeoutException INFO [2022-03-13 17:10:23,024] jepsen worker 20 - jepsen.util 1730 :invoke :read [6 nil] INFO [2022-03-13 17:10:23,053] jepsen worker 79 - jepsen.util 889 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,054] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:23,068] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,069] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:10:23,084] jepsen worker 71 - jepsen.util 431 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,084] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:23,130] jepsen worker 69 - jepsen.util 609 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,131] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:23,135] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2756598240469208}, :ns_1@172.23.105.187 {:progress 0.3742690058479532}, :ns_1@172.23.107.227 {:progress 0.2756598240469208}, :ns_1@172.23.107.228 {:progress 0.2764705882352941}} INFO [2022-03-13 17:10:23,135] jepsen worker nemesis - couchbase.util Rebalance check count: 135 INFO [2022-03-13 17:10:23,135] jepsen worker nemesis - couchbase.util Rebalance stuck count: 77 INFO [2022-03-13 17:10:23,135] jepsen worker 76 - jepsen.util 616 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,135] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,136] jepsen worker 76 - jepsen.util 616 :invoke :write [25 42] INFO [2022-03-13 17:10:23,136] jepsen worker 79 - jepsen.util 889 :invoke :write [26 34] INFO [2022-03-13 17:10:23,150] jepsen worker 78 - jepsen.util 438 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,150] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:23,170] jepsen worker 27 - jepsen.util 1467 :info :write [9 48] :RequestTimeoutException INFO [2022-03-13 17:10:23,171] jepsen worker 27 - jepsen.util 1557 :invoke :read [9 nil] INFO [2022-03-13 17:10:23,191] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,192] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:10:23,233] jepsen worker 76 - jepsen.util 616 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,233] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,275] jepsen worker 70 - jepsen.util 700 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,275] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:23,288] jepsen worker 21 - jepsen.util 741 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,288] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:23,296] jepsen worker 87 - jepsen.util 1797 :info :write [29 37] :RequestTimeoutException INFO [2022-03-13 17:10:23,296] jepsen worker 87 - jepsen.util 1887 :invoke :write [29 10] INFO [2022-03-13 17:10:23,330] jepsen worker 79 - jepsen.util 889 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,330] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,331] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:23,331] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,346] jepsen worker 29 - jepsen.util 1289 :info :write [9 8] :RequestTimeoutException INFO [2022-03-13 17:10:23,347] jepsen worker 29 - jepsen.util 1379 :invoke :read [9 nil] INFO [2022-03-13 17:10:23,388] jepsen worker 22 - jepsen.util 562 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,389] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:23,391] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,391] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,391] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:23,392] jepsen worker 42 - jepsen.util 402 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,392] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:23,392] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,392] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,393] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:23,393] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,394] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:23,396] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,396] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,396] jepsen worker 15 - jepsen.util 1275 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:10:23,396] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] INFO [2022-03-13 17:10:23,397] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:10:23,398] jepsen worker 15 - jepsen.util 1365 :invoke :write [5 8] INFO [2022-03-13 17:10:23,416] jepsen worker 68 - jepsen.util 1238 :info :write [22 20] :RequestTimeoutException INFO [2022-03-13 17:10:23,416] jepsen worker 86 - jepsen.util 1166 :info :write [28 25] :RequestTimeoutException INFO [2022-03-13 17:10:23,416] jepsen worker 68 - jepsen.util 1328 :invoke :write [22 46] INFO [2022-03-13 17:10:23,417] jepsen worker 86 - jepsen.util 1256 :invoke :read [28 nil] INFO [2022-03-13 17:10:23,427] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:23,427] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,427] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,427] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,429] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,430] jepsen worker 65 - jepsen.util 1415 :info :write [21 14] :RequestTimeoutException INFO [2022-03-13 17:10:23,430] jepsen worker 21 - jepsen.util 741 :invoke :write [7 22] INFO [2022-03-13 17:10:23,431] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,431] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:23,432] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,433] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,433] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,433] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:10:23,434] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:10:23,434] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:23,434] jepsen worker 65 - jepsen.util 1505 :invoke :read [21 nil] INFO [2022-03-13 17:10:23,435] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,436] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:10:23,523] jepsen worker 44 - jepsen.util 494 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,523] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,523] jepsen worker 44 - jepsen.util 494 :invoke :write [14 14] INFO [2022-03-13 17:10:23,524] jepsen worker 34 - jepsen.util 754 :invoke :write [11 12] INFO [2022-03-13 17:10:23,535] jepsen worker 83 - jepsen.util 893 :info :write [27 4] :RequestTimeoutException INFO [2022-03-13 17:10:23,536] jepsen worker 83 - jepsen.util 983 :invoke :read [27 nil] INFO [2022-03-13 17:10:23,555] jepsen worker 21 - jepsen.util 741 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,555] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:23,569] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,569] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,603] jepsen worker 52 - jepsen.util 502 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,603] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,603] jepsen worker 52 - jepsen.util 502 :invoke :write [17 7] INFO [2022-03-13 17:10:23,605] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:23,665] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,665] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,666] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,667] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:23,673] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,673] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,674] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:23,674] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,748] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,748] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,748] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:23,749] jepsen worker 79 - jepsen.util 889 :invoke :write [26 17] INFO [2022-03-13 17:10:23,749] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:23,749] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:10:23,870] jepsen worker 65 - jepsen.util 1505 :ok :read [21 :nil] INFO [2022-03-13 17:10:23,871] jepsen worker 65 - jepsen.util 1505 :invoke :read [21 nil] INFO [2022-03-13 17:10:23,873] jepsen worker 46 - jepsen.util 1486 :ok :read [15 48] INFO [2022-03-13 17:10:23,873] jepsen worker 46 - jepsen.util 1486 :invoke :write [15 42] INFO [2022-03-13 17:10:23,883] jepsen worker 27 - jepsen.util 1557 :ok :read [9 26] INFO [2022-03-13 17:10:23,883] jepsen worker 27 - jepsen.util 1557 :invoke :read [9 nil] INFO [2022-03-13 17:10:23,910] jepsen worker 65 - jepsen.util 1505 :ok :read [21 :nil] INFO [2022-03-13 17:10:23,910] jepsen worker 65 - jepsen.util 1505 :invoke :write [21 29] INFO [2022-03-13 17:10:23,924] jepsen worker 27 - jepsen.util 1557 :ok :read [9 26] INFO [2022-03-13 17:10:23,924] jepsen worker 27 - jepsen.util 1557 :invoke :write [9 25] INFO [2022-03-13 17:10:24,119] jepsen worker 11 - jepsen.util 1271 :ok :read [3 25] INFO [2022-03-13 17:10:24,120] jepsen worker 11 - jepsen.util 1271 :invoke :read [3 nil] INFO [2022-03-13 17:10:24,141] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.281524926686217}, :ns_1@172.23.105.187 {:progress 0.3742690058479532}, :ns_1@172.23.107.227 {:progress 0.281524926686217}, :ns_1@172.23.107.228 {:progress 0.2823529411764706}} INFO [2022-03-13 17:10:24,141] jepsen worker nemesis - couchbase.util Rebalance check count: 136 INFO [2022-03-13 17:10:24,141] jepsen worker nemesis - couchbase.util Rebalance stuck count: 77 INFO [2022-03-13 17:10:24,251] jepsen worker 11 - jepsen.util 1271 :ok :read [3 25] INFO [2022-03-13 17:10:24,251] jepsen worker 11 - jepsen.util 1271 :invoke :read [3 nil] INFO [2022-03-13 17:10:24,257] jepsen worker 57 - jepsen.util 1407 :info :write [19 47] :SyncWriteAmbiguous INFO [2022-03-13 17:10:24,257] jepsen worker 50 - jepsen.util 1400 :ok :read [16 :nil] INFO [2022-03-13 17:10:24,258] jepsen worker 50 - jepsen.util 1400 :invoke :write [16 29] INFO [2022-03-13 17:10:24,259] jepsen worker 57 - jepsen.util 1497 :invoke :read [19 nil] INFO [2022-03-13 17:10:24,415] jepsen worker 11 - jepsen.util 1271 :ok :read [3 25] INFO [2022-03-13 17:10:24,416] jepsen worker 11 - jepsen.util 1271 :invoke :read [3 nil] INFO [2022-03-13 17:10:24,428] jepsen worker 6 - jepsen.util 546 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,428] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,428] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,429] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:10:24,430] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:24,431] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:24,432] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,433] jepsen worker 76 - jepsen.util 616 :invoke :write [25 6] INFO [2022-03-13 17:10:24,479] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:24,480] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,480] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,480] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:24,481] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:10:24,482] jepsen worker 21 - jepsen.util 741 :invoke :write [7 9] INFO [2022-03-13 17:10:24,518] jepsen worker 24 - jepsen.util 474 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,519] jepsen worker 24 - jepsen.util 474 :invoke :write [8 14] INFO [2022-03-13 17:10:24,526] jepsen worker 79 - jepsen.util 889 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,526] jepsen worker 79 - jepsen.util 889 :invoke :write [26 42] INFO [2022-03-13 17:10:24,537] jepsen worker 11 - jepsen.util 1271 :ok :read [3 25] INFO [2022-03-13 17:10:24,538] jepsen worker 11 - jepsen.util 1271 :invoke :read [3 nil] INFO [2022-03-13 17:10:24,554] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,554] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:24,561] jepsen worker 76 - jepsen.util 616 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,562] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:24,606] jepsen worker 79 - jepsen.util 889 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,606] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,606] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:24,607] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:24,608] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,608] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,609] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:24,609] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:24,610] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,610] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:24,610] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,611] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:24,612] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,613] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:24,614] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,614] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:24,615] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,616] jepsen worker 76 - jepsen.util 616 :invoke :write [25 8] INFO [2022-03-13 17:10:24,671] jepsen worker 11 - jepsen.util 1271 :ok :read [3 25] INFO [2022-03-13 17:10:24,672] jepsen worker 11 - jepsen.util 1271 :invoke :read [3 nil] INFO [2022-03-13 17:10:24,673] jepsen worker 67 - jepsen.util 1237 :ok :read [22 30] INFO [2022-03-13 17:10:24,674] jepsen worker 67 - jepsen.util 1237 :invoke :write [22 16] INFO [2022-03-13 17:10:24,686] jepsen worker 60 - jepsen.util 600 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,686] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:24,686] jepsen worker 58 - jepsen.util 1228 :ok :read [19 17] INFO [2022-03-13 17:10:24,686] jepsen worker 60 - jepsen.util 600 :invoke :write [20 47] INFO [2022-03-13 17:10:24,687] jepsen worker 58 - jepsen.util 1228 :invoke :write [19 44] INFO [2022-03-13 17:10:24,687] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:24,694] jepsen worker 86 - jepsen.util 1256 :ok :read [28 44] INFO [2022-03-13 17:10:24,694] jepsen worker 16 - jepsen.util 1366 :ok :read [5 43] INFO [2022-03-13 17:10:24,695] jepsen worker 86 - jepsen.util 1256 :invoke :read [28 nil] INFO [2022-03-13 17:10:24,695] jepsen worker 16 - jepsen.util 1366 :invoke :read [5 nil] INFO [2022-03-13 17:10:24,705] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,705] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,706] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:10:24,706] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:10:24,730] jepsen worker 86 - jepsen.util 1256 :ok :read [28 44] INFO [2022-03-13 17:10:24,731] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,731] jepsen worker 86 - jepsen.util 1256 :invoke :write [28 42] INFO [2022-03-13 17:10:24,732] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:24,742] jepsen worker 16 - jepsen.util 1366 :ok :read [5 43] INFO [2022-03-13 17:10:24,743] jepsen worker 16 - jepsen.util 1366 :invoke :write [5 47] INFO [2022-03-13 17:10:24,775] jepsen worker 21 - jepsen.util 741 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,776] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:24,789] jepsen worker 11 - jepsen.util 1271 :ok :read [3 25] INFO [2022-03-13 17:10:24,790] jepsen worker 11 - jepsen.util 1271 :invoke :write [3 3] INFO [2022-03-13 17:10:24,816] jepsen worker 14 - jepsen.util 284 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,816] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,817] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:24,818] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:24,839] jepsen worker 76 - jepsen.util 616 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,839] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,839] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:24,840] jepsen worker 79 - jepsen.util 889 :invoke :write [26 46] INFO [2022-03-13 17:10:24,841] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,841] jepsen worker 76 - jepsen.util 616 :invoke :write [25 22] INFO [2022-03-13 17:10:24,843] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,843] jepsen worker 33 - jepsen.util 663 :invoke :write [11 0] INFO [2022-03-13 17:10:24,886] jepsen worker 12 - jepsen.util 732 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,887] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:24,933] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,934] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:24,944] jepsen worker 76 - jepsen.util 616 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:24,944] jepsen worker 76 - jepsen.util 616 :invoke :write [25 20] INFO [2022-03-13 17:10:25,002] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,002] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:25,003] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:25,003] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:25,007] jepsen worker 79 - jepsen.util 889 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,007] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:25,008] jepsen worker 53 - jepsen.util 503 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,009] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:25,064] jepsen worker 33 - jepsen.util 663 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,064] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,065] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,065] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:25,065] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:10:25,066] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:25,066] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:25,066] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:25,069] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:25,069] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,069] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,070] jepsen worker 76 - jepsen.util 616 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,070] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,070] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:25,071] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:25,071] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,071] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:10:25,072] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] INFO [2022-03-13 17:10:25,072] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,072] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:25,073] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:25,073] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:25,074] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:10:25,074] jepsen worker 21 - jepsen.util 741 :invoke :write [7 19] INFO [2022-03-13 17:10:25,081] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,082] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:25,085] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,085] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:25,087] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,088] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:10:25,099] jepsen worker 51 - jepsen.util 591 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,100] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:25,148] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.281524926686217}, :ns_1@172.23.105.187 {:progress 0.3742690058479532}, :ns_1@172.23.107.227 {:progress 0.281524926686217}, :ns_1@172.23.107.228 {:progress 0.2823529411764706}} INFO [2022-03-13 17:10:25,148] jepsen worker nemesis - couchbase.util Rebalance check count: 137 INFO [2022-03-13 17:10:25,148] jepsen worker nemesis - couchbase.util Rebalance stuck count: 78 INFO [2022-03-13 17:10:25,179] jepsen worker 26 - jepsen.util 476 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,180] jepsen worker 26 - jepsen.util 476 :invoke :write [8 36] INFO [2022-03-13 17:10:25,196] jepsen worker 17 - jepsen.util 1277 :info :write [5 31] :RequestTimeoutException INFO [2022-03-13 17:10:25,197] jepsen worker 17 - jepsen.util 1367 :invoke :write [5 21] INFO [2022-03-13 17:10:25,211] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,211] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:25,223] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,224] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:10:25,231] jepsen worker 3 - jepsen.util 1173 :info :write [1 0] :RequestTimeoutException INFO [2022-03-13 17:10:25,232] jepsen worker 3 - jepsen.util 1263 :invoke :write [1 32] INFO [2022-03-13 17:10:25,260] jepsen worker 38 - jepsen.util 1568 :info :write [12 26] :RequestTimeoutException INFO [2022-03-13 17:10:25,262] jepsen worker 38 - jepsen.util 1658 :invoke :read [12 nil] INFO [2022-03-13 17:10:25,287] jepsen worker 77 - jepsen.util 707 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,288] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:25,288] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:25,289] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:25,289] jepsen worker 77 - jepsen.util 707 :invoke :write [25 19] INFO [2022-03-13 17:10:25,289] jepsen worker 43 - jepsen.util 403 :invoke :write [14 10] INFO [2022-03-13 17:10:25,297] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,299] jepsen worker 76 - jepsen.util 616 :invoke :write [25 16] INFO [2022-03-13 17:10:25,302] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,303] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,303] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:25,303] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:25,304] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,305] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:25,307] jepsen worker 38 - jepsen.util 1658 :ok :read [12 20] INFO [2022-03-13 17:10:25,307] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,308] jepsen worker 79 - jepsen.util 889 :invoke :write [26 6] INFO [2022-03-13 17:10:25,309] jepsen worker 38 - jepsen.util 1658 :invoke :read [12 nil] INFO [2022-03-13 17:10:25,361] jepsen worker 38 - jepsen.util 1658 :ok :read [12 20] INFO [2022-03-13 17:10:25,362] jepsen worker 38 - jepsen.util 1658 :invoke :write [12 27] INFO [2022-03-13 17:10:25,393] jepsen worker 62 - jepsen.util 512 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,393] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,394] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,395] jepsen worker 61 - jepsen.util 601 :invoke :write [20 12] INFO [2022-03-13 17:10:25,396] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:25,396] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:25,419] jepsen worker 21 - jepsen.util 741 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,420] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:25,456] jepsen worker 79 - jepsen.util 889 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,457] jepsen worker 79 - jepsen.util 889 :invoke :write [26 3] INFO [2022-03-13 17:10:25,484] jepsen worker 66 - jepsen.util 1326 :ok :read [22 30] INFO [2022-03-13 17:10:25,485] jepsen worker 66 - jepsen.util 1326 :invoke :read [22 nil] INFO [2022-03-13 17:10:25,490] jepsen worker 1 - jepsen.util 1171 :ok :read [0 41] INFO [2022-03-13 17:10:25,491] jepsen worker 1 - jepsen.util 1171 :invoke :write [0 22] INFO [2022-03-13 17:10:25,522] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:25,523] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:10:25,581] jepsen worker 28 - jepsen.util 1378 :info :write [9 12] :RequestTimeoutException INFO [2022-03-13 17:10:25,581] jepsen worker 28 - jepsen.util 1468 :invoke :read [9 nil] INFO [2022-03-13 17:10:25,608] jepsen worker 28 - jepsen.util 1468 :ok :read [9 26] INFO [2022-03-13 17:10:25,609] jepsen worker 28 - jepsen.util 1468 :invoke :write [9 3] INFO [2022-03-13 17:10:25,617] jepsen worker 39 - jepsen.util 1389 :info :write [13 45] :RequestTimeoutException INFO [2022-03-13 17:10:25,618] jepsen worker 39 - jepsen.util 1479 :invoke :write [13 36] WARN [2022-03-13 17:10:25,635] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10630155,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13188532,"total_server_duration_us":41442,"encode_duration_us":18471,"operation_id":"0x3abb","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10010772},{"operation_name":"upsert","last_dispatch_duration_us":9067468,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":17411157,"total_server_duration_us":42416,"encode_duration_us":18073,"operation_id":"0x3994","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10010509},{"operation_name":"upsert","last_dispatch_duration_us":11612847,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":11612847,"total_server_duration_us":9024634,"encode_duration_us":19927,"operation_id":"0x3b6c","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10008907},{"operation_name":"upsert","last_dispatch_duration_us":9040200,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":17531178,"total_server_duration_us":9075237,"encode_duration_us":23337,"operation_id":"0x3985","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10008717},{"operation_name":"upsert","last_dispatch_duration_us":10395681,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10395681,"total_server_duration_us":20441,"encode_duration_us":22175,"operation_id":"0x3bac","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10007365},{"operation_name":"upsert","last_dispatch_duration_us":9128470,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9128470,"total_server_duration_us":26330,"encode_duration_us":17494,"operation_id":"0x3c4a","timeout_ms":10000,"last_server_duration_us":26330,"total_duration_us":10007313},{"operation_name":"upsert","last_dispatch_duration_us":10624978,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10624978,"total_server_duration_us":22223,"encode_duration_us":18262,"operation_id":"0x3b90","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10006309},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","encode_duration_us":27148,"operation_id":"0x3c49","timeout_ms":10000,"total_duration_us":10005842},{"operation_name":"upsert","last_dispatch_duration_us":13021792,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13021792,"total_server_duration_us":9062854,"encode_duration_us":17520,"operation_id":"0x3a3c","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10004766},{"operation_name":"upsert","last_dispatch_duration_us":9048379,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9048379,"total_server_duration_us":9036305,"encode_duration_us":24383,"operation_id":"0x3c48","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":9052034}],"total_count":26}} WARN [2022-03-13 17:10:25,636] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13021792,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13021792,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9062854,"encode_duration_us":17520,"operation_id":"0x3a3c","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10004766},{"operation_name":"upsert","last_dispatch_duration_us":9040200,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17531178,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9075237,"encode_duration_us":23337,"operation_id":"0x3985","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10008717},{"operation_name":"upsert","last_dispatch_duration_us":11612847,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":11612847,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9024634,"encode_duration_us":19927,"operation_id":"0x3b6c","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10008907}],"total_count":3}} INFO [2022-03-13 17:10:25,724] jepsen worker 35 - jepsen.util 665 :info :write [11 24] :RequestTimeoutException INFO [2022-03-13 17:10:25,725] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:25,999] jepsen worker 2 - jepsen.util 1532 :info :write [0 12] :RequestTimeoutException INFO [2022-03-13 17:10:26,000] jepsen worker 2 - jepsen.util 1622 :invoke :write [0 33] INFO [2022-03-13 17:10:26,060] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,061] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:26,060] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,060] jepsen worker 60 - jepsen.util 600 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,061] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:26,062] jepsen worker 60 - jepsen.util 600 :invoke :write [20 33] INFO [2022-03-13 17:10:26,063] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:26,063] jepsen worker 12 - jepsen.util 732 :invoke :write [4 49] INFO [2022-03-13 17:10:26,063] jepsen worker 76 - jepsen.util 616 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,064] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:26,152] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,152] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,152] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:26,153] jepsen worker 69 - jepsen.util 609 :invoke :write [23 6] INFO [2022-03-13 17:10:26,153] jepsen worker 21 - jepsen.util 741 :invoke :write [7 26] INFO [2022-03-13 17:10:26,154] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.281524926686217}, :ns_1@172.23.105.187 {:progress 0.3742690058479532}, :ns_1@172.23.107.227 {:progress 0.281524926686217}, :ns_1@172.23.107.228 {:progress 0.2823529411764706}} INFO [2022-03-13 17:10:26,154] jepsen worker nemesis - couchbase.util Rebalance check count: 138 INFO [2022-03-13 17:10:26,154] jepsen worker nemesis - couchbase.util Rebalance stuck count: 79 INFO [2022-03-13 17:10:26,154] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:26,159] jepsen worker 79 - jepsen.util 889 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,160] jepsen worker 79 - jepsen.util 889 :invoke :write [26 29] INFO [2022-03-13 17:10:26,162] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,163] jepsen worker 76 - jepsen.util 616 :invoke :write [25 30] INFO [2022-03-13 17:10:26,166] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:26,166] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,167] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:26,168] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:26,192] jepsen worker 44 - jepsen.util 494 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,193] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:26,240] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,241] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:26,243] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,243] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:26,246] jepsen worker 79 - jepsen.util 889 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,246] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:26,254] jepsen worker 8 - jepsen.util 638 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,254] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,255] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,255] jepsen worker 8 - jepsen.util 638 :invoke :write [2 32] INFO [2022-03-13 17:10:26,259] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:26,259] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:26,263] jepsen worker 77 - jepsen.util 707 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,263] jepsen worker 77 - jepsen.util 707 :invoke :write [25 8] INFO [2022-03-13 17:10:26,269] jepsen worker 36 - jepsen.util 1476 :info :write [12 5] :RequestTimeoutException INFO [2022-03-13 17:10:26,269] jepsen worker 36 - jepsen.util 1566 :invoke :write [12 11] INFO [2022-03-13 17:10:26,311] jepsen worker 31 - jepsen.util 1291 :ok :read [10 :nil] INFO [2022-03-13 17:10:26,311] jepsen worker 31 - jepsen.util 1291 :invoke :write [10 31] INFO [2022-03-13 17:10:26,319] jepsen worker 54 - jepsen.util 1224 :ok :read [18 21] INFO [2022-03-13 17:10:26,320] jepsen worker 54 - jepsen.util 1224 :invoke :read [18 nil] INFO [2022-03-13 17:10:26,320] jepsen worker 69 - jepsen.util 609 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,321] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:10:26,348] jepsen worker 76 - jepsen.util 616 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,348] jepsen worker 64 - jepsen.util 1144 :ok :read [21 :nil] INFO [2022-03-13 17:10:26,349] jepsen worker 64 - jepsen.util 1144 :invoke :write [21 1] INFO [2022-03-13 17:10:26,349] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:26,359] jepsen worker 34 - jepsen.util 754 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,359] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,359] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,360] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:26,361] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:26,361] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:26,364] jepsen worker 18 - jepsen.util 1098 :ok :read [6 16] INFO [2022-03-13 17:10:26,365] jepsen worker 18 - jepsen.util 1098 :invoke :write [6 1] INFO [2022-03-13 17:10:26,388] jepsen worker 26 - jepsen.util 476 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,388] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,388] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,389] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:26,389] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:26,390] jepsen worker 26 - jepsen.util 476 :invoke :write [8 43] INFO [2022-03-13 17:10:26,409] jepsen worker 85 - jepsen.util 1345 :info :write [28 42] :RequestTimeoutException INFO [2022-03-13 17:10:26,409] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:26,416] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,416] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,417] jepsen worker 80 - jepsen.util 620 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,417] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:26,417] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,417] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,417] jepsen worker 76 - jepsen.util 616 :invoke :write [25 8] INFO [2022-03-13 17:10:26,418] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:26,420] jepsen worker 30 - jepsen.util 1200 :ok :read [10 :nil] INFO [2022-03-13 17:10:26,421] jepsen worker 22 - jepsen.util 562 :invoke :write [7 16] INFO [2022-03-13 17:10:26,423] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:26,424] jepsen worker 13 - jepsen.util 553 :invoke :write [4 33] INFO [2022-03-13 17:10:26,424] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:10:26,429] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,429] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,429] jepsen worker 21 - jepsen.util 741 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,430] jepsen worker 30 - jepsen.util 1200 :invoke :write [10 1] INFO [2022-03-13 17:10:26,430] jepsen worker 21 - jepsen.util 741 :invoke :write [7 35] INFO [2022-03-13 17:10:26,431] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:26,432] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:10:26,439] jepsen worker 73 - jepsen.util 1423 :ok :read [24 24] INFO [2022-03-13 17:10:26,446] jepsen worker 72 - jepsen.util 1332 :ok :read [24 24] INFO [2022-03-13 17:10:26,446] jepsen worker 73 - jepsen.util 1423 :invoke :read [24 nil] INFO [2022-03-13 17:10:26,447] jepsen worker 72 - jepsen.util 1332 :invoke :write [24 30] INFO [2022-03-13 17:10:26,469] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,469] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,469] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,469] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,471] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:26,471] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:26,472] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:26,472] jepsen worker 6 - jepsen.util 546 :invoke :write [2 28] INFO [2022-03-13 17:10:26,473] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:26,475] jepsen worker 44 - jepsen.util 494 :invoke :write [14 46] INFO [2022-03-13 17:10:26,483] jepsen worker 54 - jepsen.util 1224 :ok :read [18 21] INFO [2022-03-13 17:10:26,484] jepsen worker 54 - jepsen.util 1224 :invoke :read [18 nil] INFO [2022-03-13 17:10:26,519] jepsen worker 43 - jepsen.util 403 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,520] jepsen worker 43 - jepsen.util 403 :invoke :write [14 41] INFO [2022-03-13 17:10:26,522] jepsen worker 63 - jepsen.util 1323 :info :write [21 48] :SyncWriteAmbiguous INFO [2022-03-13 17:10:26,524] jepsen worker 52 - jepsen.util 502 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,524] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,524] jepsen worker 63 - jepsen.util 1413 :invoke :read [21 nil] INFO [2022-03-13 17:10:26,524] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,525] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,525] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,525] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:26,526] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:26,527] jepsen worker 71 - jepsen.util 431 :invoke :write [23 26] INFO [2022-03-13 17:10:26,527] jepsen worker 34 - jepsen.util 754 :invoke :write [11 4] INFO [2022-03-13 17:10:26,531] jepsen worker 23 - jepsen.util 563 :invoke :write [7 1] INFO [2022-03-13 17:10:26,532] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:26,532] jepsen worker 70 - jepsen.util 700 :invoke :write [23 35] INFO [2022-03-13 17:10:26,544] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,545] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:26,570] jepsen worker 73 - jepsen.util 1423 :ok :read [24 24] INFO [2022-03-13 17:10:26,571] jepsen worker 73 - jepsen.util 1423 :invoke :read [24 nil] INFO [2022-03-13 17:10:26,605] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,605] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,606] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:26,606] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:10:26,617] jepsen worker 61 - jepsen.util 601 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,618] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,618] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,618] jepsen worker 61 - jepsen.util 601 :invoke :write [20 38] INFO [2022-03-13 17:10:26,618] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,618] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:26,618] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:26,619] jepsen worker 14 - jepsen.util 284 :invoke :write [4 1] INFO [2022-03-13 17:10:26,621] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:26,626] jepsen worker 53 - jepsen.util 503 :invoke :write [17 31] INFO [2022-03-13 17:10:26,629] jepsen worker 24 - jepsen.util 474 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,629] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,630] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:26,631] jepsen worker 7 - jepsen.util 457 :invoke :write [2 39] INFO [2022-03-13 17:10:26,634] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,635] jepsen worker 78 - jepsen.util 438 :invoke :write [26 6] INFO [2022-03-13 17:10:26,649] jepsen worker 74 - jepsen.util 1604 :info :write [24 10] :SyncWriteAmbiguous INFO [2022-03-13 17:10:26,651] jepsen worker 74 - jepsen.util 1694 :invoke :write [24 16] INFO [2022-03-13 17:10:26,674] jepsen worker 76 - jepsen.util 616 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,675] jepsen worker 76 - jepsen.util 616 :invoke :write [25 27] INFO [2022-03-13 17:10:26,677] jepsen worker 54 - jepsen.util 1224 :ok :read [18 21] INFO [2022-03-13 17:10:26,677] jepsen worker 54 - jepsen.util 1224 :invoke :read [18 nil] INFO [2022-03-13 17:10:26,696] jepsen worker 19 - jepsen.util 1099 :info :write [6 3] :RequestTimeoutException INFO [2022-03-13 17:10:26,697] jepsen worker 19 - jepsen.util 1189 :invoke :write [6 48] INFO [2022-03-13 17:10:26,704] jepsen worker 6 - jepsen.util 546 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,704] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:26,707] jepsen worker 55 - jepsen.util 1315 :info :write [18 29] :RequestTimeoutException INFO [2022-03-13 17:10:26,707] jepsen worker 55 - jepsen.util 1405 :invoke :read [18 nil] INFO [2022-03-13 17:10:26,709] jepsen worker 12 - jepsen.util 732 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:26,710] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:26,741] jepsen worker 10 - jepsen.util 1270 :ok :read [3 25] INFO [2022-03-13 17:10:26,741] jepsen worker 10 - jepsen.util 1270 :invoke :read [3 nil] INFO [2022-03-13 17:10:26,757] jepsen worker 29 - jepsen.util 1379 :ok :read [9 26] INFO [2022-03-13 17:10:26,758] jepsen worker 29 - jepsen.util 1379 :invoke :write [9 40] INFO [2022-03-13 17:10:26,806] jepsen worker 10 - jepsen.util 1270 :ok :read [3 25] INFO [2022-03-13 17:10:26,807] jepsen worker 10 - jepsen.util 1270 :invoke :write [3 13] INFO [2022-03-13 17:10:27,077] jepsen worker 81 - jepsen.util 1611 :info :write [27 3] :RequestTimeoutException INFO [2022-03-13 17:10:27,078] jepsen worker 81 - jepsen.util 1701 :invoke :read [27 nil] WARN [2022-03-13 17:10:27,135] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":20848505,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21072423,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9150377,"encode_duration_us":17956,"operation_id":"0x38cc","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10003064},{"operation_name":"upsert","last_dispatch_duration_us":9231668,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9949799,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9600785,"encode_duration_us":17062,"operation_id":"0x3956","timeout_ms":10000,"last_server_duration_us":9172586,"total_duration_us":10009332},{"operation_name":"upsert","last_dispatch_duration_us":12463603,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12463603,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9080926,"encode_duration_us":17538,"operation_id":"0x3a45","timeout_ms":10000,"last_server_duration_us":9080926,"total_duration_us":10003464}],"total_count":3}} INFO [2022-03-13 17:10:27,159] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.281524926686217}, :ns_1@172.23.105.187 {:progress 0.3801169590643275}, :ns_1@172.23.107.227 {:progress 0.281524926686217}, :ns_1@172.23.107.228 {:progress 0.288235294117647}} INFO [2022-03-13 17:10:27,159] jepsen worker nemesis - couchbase.util Rebalance check count: 139 INFO [2022-03-13 17:10:27,159] jepsen worker nemesis - couchbase.util Rebalance stuck count: 79 INFO [2022-03-13 17:10:27,214] jepsen worker 45 - jepsen.util 1485 :info :write [15 26] :RequestTimeoutException INFO [2022-03-13 17:10:27,214] jepsen worker 45 - jepsen.util 1575 :invoke :write [15 23] WARN [2022-03-13 17:10:27,227] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":6705872,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11792305,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":257412,"encode_duration_us":29830,"operation_id":"0x3b71","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10000642},{"operation_name":"upsert","last_dispatch_duration_us":18209881,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18209881,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9054355,"encode_duration_us":17884,"operation_id":"0x3766","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10003913},{"operation_name":"upsert","last_dispatch_duration_us":14091867,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14822660,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9282956,"encode_duration_us":17445,"operation_id":"0x3a4a","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009912},{"operation_name":"upsert","last_dispatch_duration_us":1198943,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":1198943,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":74510,"encode_duration_us":18255,"operation_id":"0x3c7c","timeout_ms":10000,"last_server_duration_us":74510,"total_duration_us":10010090},{"operation_name":"upsert","last_dispatch_duration_us":13276717,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13276717,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9026756,"encode_duration_us":17467,"operation_id":"0x3b88","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008667}],"total_count":5}} WARN [2022-03-13 17:10:27,234] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9050611,"last_local_id":"A312307A00000005/00000000F4F37B9B","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9612514,"last_local_socket":"172.23.104.79:33502","total_server_duration_us":9354487,"encode_duration_us":18132,"operation_id":"0x3b2f","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10006427},{"operation_name":"upsert","last_dispatch_duration_us":9045386,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13609514,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9232550,"encode_duration_us":19242,"operation_id":"0x3922","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10009739}],"total_count":2}} WARN [2022-03-13 17:10:27,235] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8917365,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9110556,"total_server_duration_us":73477,"encode_duration_us":23820,"operation_id":"0x3ba5","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10010033},{"operation_name":"upsert","last_dispatch_duration_us":8912387,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9169645,"total_server_duration_us":77347,"encode_duration_us":22028,"operation_id":"0x3baa","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10009251},{"operation_name":"upsert","last_dispatch_duration_us":8388592,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":8888053,"total_server_duration_us":341980,"encode_duration_us":17645,"operation_id":"0x3b98","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10006957},{"operation_name":"upsert","last_dispatch_duration_us":8358424,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":9195956,"total_server_duration_us":503924,"encode_duration_us":17704,"operation_id":"0x3a34","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10006691},{"operation_name":"upsert","last_dispatch_duration_us":9050611,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":9612514,"total_server_duration_us":9354487,"encode_duration_us":18132,"operation_id":"0x3b2f","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10006427},{"operation_name":"upsert","last_dispatch_duration_us":96915,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":96915,"total_server_duration_us":21001,"encode_duration_us":21892,"operation_id":"0x3bc0","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10003688},{"operation_name":"upsert","last_dispatch_duration_us":144717,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":291314,"total_server_duration_us":51795,"encode_duration_us":18581,"operation_id":"0x3bab","timeout_ms":10000,"last_server_duration_us":32065,"total_duration_us":10003336},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":32043,"operation_id":"0x3c47","timeout_ms":10000,"total_duration_us":10001682},{"operation_name":"upsert","last_dispatch_duration_us":9150652,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9150652,"total_server_duration_us":9123510,"encode_duration_us":19956,"operation_id":"0x3a7b","timeout_ms":10000,"last_server_duration_us":9123510,"total_duration_us":9154631},{"operation_name":"upsert","last_dispatch_duration_us":9097583,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9097583,"total_server_duration_us":9045859,"encode_duration_us":19797,"operation_id":"0x3ba7","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":9102290}],"total_count":31}} WARN [2022-03-13 17:10:27,235] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":8625912,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8625912,"total_server_duration_us":38,"operation_id":"0x394a","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":14814259},{"operation_name":"get","last_dispatch_duration_us":8622956,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8622956,"total_server_duration_us":15,"operation_id":"0x399a","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":13792102},{"operation_name":"upsert","last_dispatch_duration_us":8135976,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":16118760,"total_server_duration_us":245188,"encode_duration_us":19689,"operation_id":"0x3a89","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10007917},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":28231,"operation_id":"0x3b87","timeout_ms":10000,"total_duration_us":10007081},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":24362,"operation_id":"0x3c8b","timeout_ms":10000,"total_duration_us":10005972},{"operation_name":"upsert","last_dispatch_duration_us":9159672,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9159672,"total_server_duration_us":9055417,"encode_duration_us":18108,"operation_id":"0x3cd6","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":9166454},{"operation_name":"upsert","last_dispatch_duration_us":9063945,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9063945,"total_server_duration_us":9032060,"encode_duration_us":22527,"operation_id":"0x3cd1","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":9067709},{"operation_name":"upsert","last_dispatch_duration_us":9055859,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9055859,"total_server_duration_us":9043735,"encode_duration_us":21121,"operation_id":"0x39f5","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":9059939},{"operation_name":"upsert","last_dispatch_duration_us":1061083,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1977324,"total_server_duration_us":103758,"encode_duration_us":33356,"operation_id":"0x3b86","timeout_ms":10000,"last_server_duration_us":36933,"total_duration_us":8173114},{"operation_name":"upsert","last_dispatch_duration_us":1475179,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2873247,"total_server_duration_us":102570,"encode_duration_us":22325,"operation_id":"0x3b50","timeout_ms":10000,"last_server_duration_us":43070,"total_duration_us":8041172}],"total_count":52}} WARN [2022-03-13 17:10:27,236] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12719612,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12719612,"total_server_duration_us":69,"operation_id":"0x3b73","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":12722200},{"operation_name":"get","last_dispatch_duration_us":11262755,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11262755,"total_server_duration_us":32,"operation_id":"0x3bf6","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11265834},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":19607,"operation_id":"0x3c74","timeout_ms":10000,"total_duration_us":10008512},{"operation_name":"upsert","last_dispatch_duration_us":9195696,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9195696,"total_server_duration_us":21649,"encode_duration_us":17855,"operation_id":"0x3c9d","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10007476},{"operation_name":"upsert","last_dispatch_duration_us":8721130,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9340815,"total_server_duration_us":398761,"encode_duration_us":19519,"operation_id":"0x39ef","timeout_ms":10000,"last_server_duration_us":25275,"total_duration_us":10005907},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":17229,"operation_id":"0x3cca","timeout_ms":10000,"total_duration_us":10005481},{"operation_name":"upsert","last_dispatch_duration_us":12463603,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12463603,"total_server_duration_us":9080926,"encode_duration_us":17538,"operation_id":"0x3a45","timeout_ms":10000,"last_server_duration_us":9080926,"total_duration_us":10003464},{"operation_name":"upsert","last_dispatch_duration_us":9240286,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9240286,"total_server_duration_us":9150168,"encode_duration_us":22431,"operation_id":"0x3b81","timeout_ms":10000,"last_server_duration_us":9150168,"total_duration_us":9246279},{"operation_name":"upsert","last_dispatch_duration_us":4223824,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":6502217,"total_server_duration_us":113377,"encode_duration_us":28025,"operation_id":"0x3c73","timeout_ms":10000,"last_server_duration_us":76033,"total_duration_us":6519457},{"operation_name":"upsert","last_dispatch_duration_us":6031145,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":6031145,"total_server_duration_us":84283,"encode_duration_us":19105,"operation_id":"0x3d12","timeout_ms":10000,"last_server_duration_us":84283,"total_duration_us":6034916}],"total_count":32}} WARN [2022-03-13 17:10:27,328] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11303373,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11303373,"total_server_duration_us":27,"operation_id":"0x3c43","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":11304686},{"operation_name":"upsert","last_dispatch_duration_us":1198943,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1198943,"total_server_duration_us":74510,"encode_duration_us":18255,"operation_id":"0x3c7c","timeout_ms":10000,"last_server_duration_us":74510,"total_duration_us":10010090},{"operation_name":"upsert","last_dispatch_duration_us":14091867,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":14822660,"total_server_duration_us":9282956,"encode_duration_us":17445,"operation_id":"0x3a4a","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009912},{"operation_name":"upsert","last_dispatch_duration_us":13276717,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":13276717,"total_server_duration_us":9026756,"encode_duration_us":17467,"operation_id":"0x3b88","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008667},{"operation_name":"upsert","last_dispatch_duration_us":10067048,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":14567122,"total_server_duration_us":285449,"encode_duration_us":18479,"operation_id":"0x3a49","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10008337},{"operation_name":"upsert","last_dispatch_duration_us":42043,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":632296,"total_server_duration_us":398761,"encode_duration_us":18359,"operation_id":"0x3c98","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10005387},{"operation_name":"upsert","last_dispatch_duration_us":13639994,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13639994,"total_server_duration_us":31681,"encode_duration_us":17946,"operation_id":"0x3b1a","timeout_ms":10000,"last_server_duration_us":31681,"total_duration_us":10003543},{"operation_name":"upsert","last_dispatch_duration_us":138561,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9376615,"total_server_duration_us":65722,"encode_duration_us":17722,"operation_id":"0x3cce","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10003155},{"operation_name":"upsert","last_dispatch_duration_us":9144938,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":17567259,"total_server_duration_us":42573,"encode_duration_us":17252,"operation_id":"0x3a29","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10001976},{"operation_name":"upsert","last_dispatch_duration_us":6705872,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":11792305,"total_server_duration_us":257412,"encode_duration_us":29830,"operation_id":"0x3b71","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10000642}],"total_count":33}} WARN [2022-03-13 17:10:27,333] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":35369,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1275457,"total_server_duration_us":281371,"encode_duration_us":17859,"operation_id":"0x3cc1","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10007384},{"operation_name":"upsert","last_dispatch_duration_us":16905101,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":16905101,"total_server_duration_us":9045859,"encode_duration_us":17198,"operation_id":"0x39da","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10006716},{"operation_name":"upsert","last_dispatch_duration_us":30420,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1281538,"total_server_duration_us":253885,"encode_duration_us":21970,"operation_id":"0x3cc2","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10005752},{"operation_name":"upsert","last_dispatch_duration_us":172099,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10065952,"total_server_duration_us":121077,"encode_duration_us":17240,"operation_id":"0x39a3","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10005245},{"operation_name":"upsert","last_dispatch_duration_us":30052,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7454704,"total_server_duration_us":238641,"encode_duration_us":18408,"operation_id":"0x3a9e","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10004979},{"operation_name":"upsert","last_dispatch_duration_us":40164,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":4525053,"total_server_duration_us":276669,"encode_duration_us":24383,"operation_id":"0x3b9a","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10002757},{"operation_name":"get","last_dispatch_duration_us":9478511,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9478511,"total_server_duration_us":43,"operation_id":"0x3c46","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9480246},{"operation_name":"get","last_dispatch_duration_us":8006593,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8006593,"total_server_duration_us":27,"operation_id":"0x3cc0","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":8008103},{"operation_name":"upsert","last_dispatch_duration_us":2415879,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":5444500,"total_server_duration_us":209687,"encode_duration_us":25371,"operation_id":"0x3c27","timeout_ms":10000,"last_server_duration_us":52777,"total_duration_us":5478265},{"operation_name":"get","last_dispatch_duration_us":5380742,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5380742,"total_server_duration_us":27,"operation_id":"0x3d53","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":5382309}],"total_count":38}} WARN [2022-03-13 17:10:27,335] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17167484,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17790173,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9138282,"encode_duration_us":17632,"operation_id":"0x3874","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007334},{"operation_name":"upsert","last_dispatch_duration_us":17207446,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17955688,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9157718,"encode_duration_us":18244,"operation_id":"0x3865","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10008339}],"total_count":2}} WARN [2022-03-13 17:10:27,434] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16905101,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16905101,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9045859,"encode_duration_us":17198,"operation_id":"0x39da","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10006716}],"total_count":1}} INFO [2022-03-13 17:10:27,470] jepsen worker 21 - jepsen.util 741 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,470] jepsen worker 32 - jepsen.util 1112 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:10:27,470] jepsen worker 21 - jepsen.util 741 :invoke :write [7 34] INFO [2022-03-13 17:10:27,471] jepsen worker 32 - jepsen.util 1202 :invoke :write [10 41] INFO [2022-03-13 17:10:27,472] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,472] jepsen worker 79 - jepsen.util 889 :invoke :write [26 46] INFO [2022-03-13 17:10:27,522] jepsen worker 26 - jepsen.util 476 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,523] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:27,524] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:27,524] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,525] jepsen worker 62 - jepsen.util 512 :invoke :write [20 24] INFO [2022-03-13 17:10:27,525] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:27,586] jepsen worker 8 - jepsen.util 638 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,586] jepsen worker 44 - jepsen.util 494 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,586] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,586] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,586] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,587] jepsen worker 8 - jepsen.util 638 :invoke :write [2 13] INFO [2022-03-13 17:10:27,588] jepsen worker 80 - jepsen.util 620 :invoke :write [26 1] INFO [2022-03-13 17:10:27,589] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:27,590] jepsen worker 25 - jepsen.util 385 :invoke :write [8 3] INFO [2022-03-13 17:10:27,591] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:27,621] jepsen worker 60 - jepsen.util 600 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,622] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:27,626] jepsen worker 76 - jepsen.util 616 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,627] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:27,684] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,684] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,685] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:27,686] jepsen worker 75 - jepsen.util 435 :invoke :write [25 49] INFO [2022-03-13 17:10:27,724] jepsen worker 14 - jepsen.util 284 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,724] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:27,746] jepsen worker 79 - jepsen.util 889 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,746] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,747] jepsen worker 79 - jepsen.util 889 :invoke :write [26 25] INFO [2022-03-13 17:10:27,750] jepsen worker 76 - jepsen.util 616 :invoke :write [25 20] INFO [2022-03-13 17:10:27,779] jepsen worker 77 - jepsen.util 707 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,779] jepsen worker 77 - jepsen.util 707 :invoke :write [25 34] INFO [2022-03-13 17:10:27,783] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,784] jepsen worker 51 - jepsen.util 591 :invoke :write [17 26] INFO [2022-03-13 17:10:27,823] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,824] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:27,828] jepsen worker 22 - jepsen.util 562 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,828] jepsen worker 34 - jepsen.util 754 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,828] jepsen worker 22 - jepsen.util 562 :invoke :write [7 21] INFO [2022-03-13 17:10:27,829] jepsen worker 34 - jepsen.util 754 :invoke :write [11 31] INFO [2022-03-13 17:10:27,865] jepsen worker 79 - jepsen.util 889 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,866] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:27,936] jepsen worker 61 - jepsen.util 601 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,937] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:27,941] jepsen worker 21 - jepsen.util 741 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,942] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,942] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:27,942] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:27,976] jepsen worker 76 - jepsen.util 616 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,976] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,977] jepsen worker 76 - jepsen.util 616 :invoke :write [25 18] INFO [2022-03-13 17:10:27,977] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:10:27,980] jepsen worker 13 - jepsen.util 553 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,981] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,981] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,980] jepsen worker 71 - jepsen.util 431 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:27,982] jepsen worker 71 - jepsen.util 431 :invoke :write [23 38] INFO [2022-03-13 17:10:27,982] jepsen worker 52 - jepsen.util 502 :invoke :write [17 31] INFO [2022-03-13 17:10:27,983] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:27,984] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:28,027] jepsen worker 43 - jepsen.util 403 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,027] jepsen worker 43 - jepsen.util 403 :invoke :write [14 14] INFO [2022-03-13 17:10:28,050] jepsen worker 75 - jepsen.util 435 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,050] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,050] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,050] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:28,051] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:28,051] jepsen worker 21 - jepsen.util 741 :invoke :write [7 22] INFO [2022-03-13 17:10:28,052] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:10:28,052] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:10:28,067] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,068] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:10:28,073] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,074] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:28,131] jepsen worker 7 - jepsen.util 457 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,131] jepsen worker 7 - jepsen.util 457 :invoke :write [2 36] INFO [2022-03-13 17:10:28,138] jepsen worker 63 - jepsen.util 1413 :ok :read [21 :nil] INFO [2022-03-13 17:10:28,138] jepsen worker 63 - jepsen.util 1413 :invoke :read [21 nil] INFO [2022-03-13 17:10:28,143] jepsen worker 63 - jepsen.util 1413 :ok :read [21 :nil] INFO [2022-03-13 17:10:28,144] jepsen worker 63 - jepsen.util 1413 :invoke :write [21 16] INFO [2022-03-13 17:10:28,165] jepsen worker 21 - jepsen.util 741 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,166] jepsen worker 21 - jepsen.util 741 :invoke :write [7 5] INFO [2022-03-13 17:10:28,166] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.281524926686217}, :ns_1@172.23.105.187 {:progress 0.3801169590643275}, :ns_1@172.23.107.227 {:progress 0.281524926686217}, :ns_1@172.23.107.228 {:progress 0.288235294117647}} INFO [2022-03-13 17:10:28,166] jepsen worker nemesis - couchbase.util Rebalance check count: 140 INFO [2022-03-13 17:10:28,166] jepsen worker nemesis - couchbase.util Rebalance stuck count: 80 INFO [2022-03-13 17:10:28,168] jepsen worker 53 - jepsen.util 503 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,168] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:28,168] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,170] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:28,171] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:28,172] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:28,176] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,176] jepsen worker 76 - jepsen.util 616 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,177] jepsen worker 89 - jepsen.util 1619 :info :write [29 36] :RequestTimeoutException INFO [2022-03-13 17:10:28,180] jepsen worker 89 - jepsen.util 1709 :invoke :read [29 nil] INFO [2022-03-13 17:10:28,181] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:28,181] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:28,188] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,188] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,188] jepsen worker 79 - jepsen.util 889 :invoke :write [26 22] INFO [2022-03-13 17:10:28,189] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:28,190] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,191] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:28,192] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,192] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:28,193] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,194] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:28,195] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,195] jepsen worker 76 - jepsen.util 616 :invoke :write [25 47] INFO [2022-03-13 17:10:28,213] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:28,213] jepsen worker 70 - jepsen.util 700 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,213] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,216] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:28,217] jepsen worker 6 - jepsen.util 546 :invoke :write [2 18] INFO [2022-03-13 17:10:28,217] jepsen worker 44 - jepsen.util 494 :invoke :write [14 17] INFO [2022-03-13 17:10:28,251] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,251] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:10:28,280] jepsen worker 23 - jepsen.util 563 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,280] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,280] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,281] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:28,282] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:28,282] jepsen worker 60 - jepsen.util 600 :invoke :write [20 42] INFO [2022-03-13 17:10:28,288] jepsen worker 79 - jepsen.util 889 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,289] jepsen worker 79 - jepsen.util 889 :invoke :write [26 29] INFO [2022-03-13 17:10:28,334] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,335] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:10:28,374] jepsen worker 76 - jepsen.util 616 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,375] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:28,413] jepsen worker 62 - jepsen.util 512 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,413] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,413] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,413] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:28,413] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,414] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,414] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:28,415] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:28,416] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:28,416] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,417] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:28,417] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,417] jepsen worker 42 - jepsen.util 402 :invoke :write [14 8] INFO [2022-03-13 17:10:28,417] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,418] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,418] jepsen worker 62 - jepsen.util 512 :invoke :write [20 34] INFO [2022-03-13 17:10:28,419] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:28,419] jepsen worker 26 - jepsen.util 476 :invoke :write [8 38] INFO [2022-03-13 17:10:28,420] jepsen worker 61 - jepsen.util 601 :invoke :write [20 37] INFO [2022-03-13 17:10:28,423] jepsen worker 53 - jepsen.util 503 :invoke :write [17 3] INFO [2022-03-13 17:10:28,423] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,424] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:28,425] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,426] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:28,430] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,431] jepsen worker 33 - jepsen.util 663 :invoke :write [11 13] INFO [2022-03-13 17:10:28,436] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,436] jepsen worker 78 - jepsen.util 438 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,436] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,437] jepsen worker 35 - jepsen.util 755 :invoke :write [11 24] INFO [2022-03-13 17:10:28,438] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:28,439] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:28,454] jepsen worker 34 - jepsen.util 754 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,455] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:28,478] jepsen worker 79 - jepsen.util 889 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,478] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,479] jepsen worker 76 - jepsen.util 616 :invoke :write [25 25] INFO [2022-03-13 17:10:28,480] jepsen worker 79 - jepsen.util 889 :invoke :write [26 0] INFO [2022-03-13 17:10:28,518] jepsen worker 21 - jepsen.util 741 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,518] jepsen worker 77 - jepsen.util 707 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,519] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,519] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:28,521] jepsen worker 14 - jepsen.util 284 :invoke :write [4 8] INFO [2022-03-13 17:10:28,522] jepsen worker 77 - jepsen.util 707 :invoke :write [25 8] INFO [2022-03-13 17:10:28,580] jepsen worker 76 - jepsen.util 616 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,581] jepsen worker 76 - jepsen.util 616 :invoke :write [25 20] INFO [2022-03-13 17:10:28,608] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:28,609] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:28,657] jepsen worker 20 - jepsen.util 1730 :ok :read [6 16] INFO [2022-03-13 17:10:28,657] jepsen worker 20 - jepsen.util 1730 :invoke :read [6 nil] INFO [2022-03-13 17:10:28,720] jepsen worker 20 - jepsen.util 1730 :ok :read [6 16] INFO [2022-03-13 17:10:28,721] jepsen worker 20 - jepsen.util 1730 :invoke :write [6 29] INFO [2022-03-13 17:10:28,908] jepsen worker 9 - jepsen.util 1359 :info :write [3 15] :RequestTimeoutException INFO [2022-03-13 17:10:28,909] jepsen worker 9 - jepsen.util 1449 :invoke :write [3 3] INFO [2022-03-13 17:10:29,060] jepsen worker 5 - jepsen.util 1535 :info :write [1 21] :RequestTimeoutException INFO [2022-03-13 17:10:29,060] jepsen worker 5 - jepsen.util 1625 :invoke :read [1 nil] INFO [2022-03-13 17:10:29,172] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.281524926686217}, :ns_1@172.23.105.187 {:progress 0.3801169590643275}, :ns_1@172.23.107.227 {:progress 0.281524926686217}, :ns_1@172.23.107.228 {:progress 0.288235294117647}} INFO [2022-03-13 17:10:29,172] jepsen worker nemesis - couchbase.util Rebalance check count: 141 INFO [2022-03-13 17:10:29,172] jepsen worker nemesis - couchbase.util Rebalance stuck count: 81 INFO [2022-03-13 17:10:29,246] jepsen worker 56 - jepsen.util 1226 :info :write [18 8] :RequestTimeoutException INFO [2022-03-13 17:10:29,247] jepsen worker 56 - jepsen.util 1316 :invoke :read [18 nil] INFO [2022-03-13 17:10:29,488] jepsen worker 80 - jepsen.util 620 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,488] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:29,489] jepsen worker 79 - jepsen.util 889 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,490] jepsen worker 79 - jepsen.util 889 :invoke :write [26 27] INFO [2022-03-13 17:10:29,491] jepsen worker 71 - jepsen.util 431 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,492] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,492] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:29,492] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:29,567] jepsen worker 53 - jepsen.util 503 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,567] jepsen worker 51 - jepsen.util 591 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,568] jepsen worker 53 - jepsen.util 503 :invoke :write [17 28] INFO [2022-03-13 17:10:29,568] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,568] jepsen worker 51 - jepsen.util 591 :invoke :write [17 11] INFO [2022-03-13 17:10:29,569] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:29,613] jepsen worker 76 - jepsen.util 616 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,614] jepsen worker 76 - jepsen.util 616 :invoke :write [25 11] INFO [2022-03-13 17:10:29,662] jepsen worker 6 - jepsen.util 546 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,662] jepsen worker 8 - jepsen.util 638 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,663] jepsen worker 8 - jepsen.util 638 :invoke :write [2 40] INFO [2022-03-13 17:10:29,663] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:29,684] jepsen worker 33 - jepsen.util 663 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,684] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:29,684] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,685] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,685] jepsen worker 21 - jepsen.util 741 :invoke :write [7 2] INFO [2022-03-13 17:10:29,686] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:10:29,690] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:29,691] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:29,692] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,692] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,692] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:29,693] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:10:29,693] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,694] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:10:29,719] jepsen worker 79 - jepsen.util 889 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,720] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:29,773] jepsen worker 43 - jepsen.util 403 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,773] jepsen worker 62 - jepsen.util 512 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,774] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,774] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:29,774] jepsen worker 43 - jepsen.util 403 :invoke :write [14 10] INFO [2022-03-13 17:10:29,775] jepsen worker 23 - jepsen.util 563 :invoke :write [7 26] INFO [2022-03-13 17:10:29,775] jepsen worker 12 - jepsen.util 732 :invoke :write [4 27] INFO [2022-03-13 17:10:29,776] jepsen worker 62 - jepsen.util 512 :invoke :write [20 6] INFO [2022-03-13 17:10:29,791] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,791] jepsen worker 25 - jepsen.util 385 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,791] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,793] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:29,794] jepsen worker 24 - jepsen.util 474 :invoke :write [8 35] INFO [2022-03-13 17:10:29,795] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,796] jepsen worker 76 - jepsen.util 616 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,797] jepsen worker 25 - jepsen.util 385 :invoke :write [8 48] INFO [2022-03-13 17:10:29,799] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:29,799] jepsen worker 79 - jepsen.util 889 :invoke :write [26 3] INFO [2022-03-13 17:10:29,804] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,805] jepsen worker 76 - jepsen.util 616 :invoke :write [25 45] INFO [2022-03-13 17:10:29,808] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,810] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:29,878] jepsen worker 44 - jepsen.util 494 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,878] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,878] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,878] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,879] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,880] jepsen worker 44 - jepsen.util 494 :invoke :write [14 46] INFO [2022-03-13 17:10:29,880] jepsen worker 34 - jepsen.util 754 :invoke :write [11 12] INFO [2022-03-13 17:10:29,881] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:29,882] jepsen worker 71 - jepsen.util 431 :invoke :write [23 35] INFO [2022-03-13 17:10:29,885] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,885] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:29,887] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,887] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:29,889] jepsen worker 70 - jepsen.util 700 :invoke :write [23 36] INFO [2022-03-13 17:10:29,894] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,895] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:29,903] jepsen worker 79 - jepsen.util 889 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,904] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:29,907] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,907] jepsen worker 6 - jepsen.util 546 :invoke :write [2 31] INFO [2022-03-13 17:10:29,925] jepsen worker 59 - jepsen.util 1229 :ok :read [19 17] INFO [2022-03-13 17:10:29,925] jepsen worker 59 - jepsen.util 1229 :invoke :write [19 26] INFO [2022-03-13 17:10:29,940] jepsen worker 26 - jepsen.util 476 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,941] jepsen worker 26 - jepsen.util 476 :invoke :write [8 44] INFO [2022-03-13 17:10:29,943] jepsen worker 60 - jepsen.util 600 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,944] jepsen worker 60 - jepsen.util 600 :invoke :write [20 16] INFO [2022-03-13 17:10:29,947] jepsen worker 21 - jepsen.util 741 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,947] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:29,964] jepsen worker 57 - jepsen.util 1497 :ok :read [19 17] INFO [2022-03-13 17:10:29,965] jepsen worker 57 - jepsen.util 1497 :invoke :write [19 38] INFO [2022-03-13 17:10:29,978] jepsen worker 47 - jepsen.util 1397 :info :write [15 37] :RequestTimeoutException INFO [2022-03-13 17:10:29,979] jepsen worker 47 - jepsen.util 1487 :invoke :write [15 19] INFO [2022-03-13 17:10:29,986] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,986] jepsen worker 76 - jepsen.util 616 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:29,987] jepsen worker 76 - jepsen.util 616 :invoke :write [25 44] INFO [2022-03-13 17:10:29,988] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:30,000] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,000] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:10:30,051] jepsen worker 52 - jepsen.util 502 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,051] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,051] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:30,052] jepsen worker 78 - jepsen.util 438 :invoke :write [26 23] INFO [2022-03-13 17:10:30,065] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,066] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:30,109] jepsen worker 76 - jepsen.util 616 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,110] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:30,123] jepsen worker 14 - jepsen.util 284 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,123] jepsen worker 77 - jepsen.util 707 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,124] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:30,124] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:30,169] jepsen worker 44 - jepsen.util 494 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,170] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:30,172] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,172] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,172] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:30,172] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,173] jepsen worker 21 - jepsen.util 741 :invoke :write [7 31] INFO [2022-03-13 17:10:30,173] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:30,173] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:30,174] jepsen worker 69 - jepsen.util 609 :invoke :write [23 3] INFO [2022-03-13 17:10:30,174] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,175] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,175] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:10:30,176] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:30,177] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,177] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:10:30,183] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.281524926686217}, :ns_1@172.23.105.187 {:progress 0.3801169590643275}, :ns_1@172.23.107.227 {:progress 0.281524926686217}, :ns_1@172.23.107.228 {:progress 0.288235294117647}} INFO [2022-03-13 17:10:30,183] jepsen worker nemesis - couchbase.util Rebalance check count: 142 INFO [2022-03-13 17:10:30,183] jepsen worker nemesis - couchbase.util Rebalance stuck count: 82 INFO [2022-03-13 17:10:30,209] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,209] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,210] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:10:30,210] jepsen worker 76 - jepsen.util 616 :invoke :write [25 47] INFO [2022-03-13 17:10:30,248] jepsen worker 51 - jepsen.util 591 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,249] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:30,321] jepsen worker 69 - jepsen.util 609 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,321] jepsen worker 35 - jepsen.util 755 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,321] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,322] jepsen worker 61 - jepsen.util 601 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,322] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:30,323] jepsen worker 69 - jepsen.util 609 :invoke :write [23 46] INFO [2022-03-13 17:10:30,323] jepsen worker 35 - jepsen.util 755 :invoke :write [11 21] INFO [2022-03-13 17:10:30,325] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:30,334] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,336] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:30,393] jepsen worker 70 - jepsen.util 700 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,394] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:30,451] jepsen worker 8 - jepsen.util 638 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,451] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:30,479] jepsen worker 76 - jepsen.util 616 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,480] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:30,499] jepsen worker 21 - jepsen.util 741 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,500] jepsen worker 21 - jepsen.util 741 :invoke :write [7 27] INFO [2022-03-13 17:10:30,673] jepsen worker 5 - jepsen.util 1625 :ok :read [1 42] INFO [2022-03-13 17:10:30,674] jepsen worker 5 - jepsen.util 1625 :invoke :write [1 3] INFO [2022-03-13 17:10:30,688] jepsen worker 34 - jepsen.util 754 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,688] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:30,693] jepsen worker 43 - jepsen.util 403 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,693] jepsen worker 42 - jepsen.util 402 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,693] jepsen worker 43 - jepsen.util 403 :invoke :write [14 8] INFO [2022-03-13 17:10:30,694] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:30,743] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,743] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,744] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:30,745] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,746] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:10:30,749] jepsen worker 76 - jepsen.util 616 :invoke :write [25 23] INFO [2022-03-13 17:10:30,787] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,788] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:10:30,870] jepsen worker 0 - jepsen.util 1350 :info :write [0 39] :RequestTimeoutException INFO [2022-03-13 17:10:30,871] jepsen worker 0 - jepsen.util 1440 :invoke :write [0 5] INFO [2022-03-13 17:10:30,886] jepsen worker 25 - jepsen.util 385 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,887] jepsen worker 25 - jepsen.util 385 :invoke :write [8 30] INFO [2022-03-13 17:10:30,923] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,923] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:30,939] jepsen worker 76 - jepsen.util 616 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:30,940] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:31,001] jepsen worker 23 - jepsen.util 563 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,002] jepsen worker 53 - jepsen.util 503 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,002] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:31,003] jepsen worker 23 - jepsen.util 563 :invoke :write [7 40] INFO [2022-03-13 17:10:31,049] jepsen worker 69 - jepsen.util 609 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,050] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:31,054] jepsen worker 71 - jepsen.util 431 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,055] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:31,078] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,079] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:31,080] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,081] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,081] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:31,081] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:10:31,082] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,082] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:31,083] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,084] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:31,085] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,086] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] INFO [2022-03-13 17:10:31,091] jepsen worker 87 - jepsen.util 1887 :ok :write [29 10] INFO [2022-03-13 17:10:31,092] jepsen worker 87 - jepsen.util 1887 :invoke :write [29 41] INFO [2022-03-13 17:10:31,116] jepsen worker 89 - jepsen.util 1709 :ok :read [29 10] INFO [2022-03-13 17:10:31,116] jepsen worker 89 - jepsen.util 1709 :invoke :read [29 nil] INFO [2022-03-13 17:10:31,142] jepsen worker 22 - jepsen.util 562 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,142] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,143] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:31,143] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:31,166] jepsen worker 21 - jepsen.util 741 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,166] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:31,192] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2844574780058651}, :ns_1@172.23.105.187 {:progress 0.3859649122807017}, :ns_1@172.23.107.227 {:progress 0.2844574780058651}, :ns_1@172.23.107.228 {:progress 0.2941176470588235}} INFO [2022-03-13 17:10:31,192] jepsen worker nemesis - couchbase.util Rebalance check count: 143 INFO [2022-03-13 17:10:31,192] jepsen worker nemesis - couchbase.util Rebalance stuck count: 82 INFO [2022-03-13 17:10:31,195] jepsen worker 62 - jepsen.util 512 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,195] jepsen worker 12 - jepsen.util 732 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,195] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:31,196] jepsen worker 12 - jepsen.util 732 :invoke :write [4 40] INFO [2022-03-13 17:10:31,198] jepsen worker 89 - jepsen.util 1709 :ok :read [29 10] INFO [2022-03-13 17:10:31,198] jepsen worker 89 - jepsen.util 1709 :invoke :read [29 nil] INFO [2022-03-13 17:10:31,201] jepsen worker 56 - jepsen.util 1316 :ok :read [18 21] INFO [2022-03-13 17:10:31,202] jepsen worker 56 - jepsen.util 1316 :invoke :write [18 10] INFO [2022-03-13 17:10:31,210] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,210] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:31,236] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:31,236] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,237] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,237] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,236] jepsen worker 6 - jepsen.util 546 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,237] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:31,238] jepsen worker 44 - jepsen.util 494 :invoke :write [14 7] INFO [2022-03-13 17:10:31,239] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,239] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:31,240] jepsen worker 6 - jepsen.util 546 :invoke :write [2 36] INFO [2022-03-13 17:10:31,240] jepsen worker 70 - jepsen.util 700 :invoke :write [23 16] INFO [2022-03-13 17:10:31,241] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:31,252] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,252] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,252] jepsen worker 34 - jepsen.util 754 :invoke :write [11 37] INFO [2022-03-13 17:10:31,253] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:31,255] jepsen worker 89 - jepsen.util 1709 :ok :read [29 10] INFO [2022-03-13 17:10:31,255] jepsen worker 89 - jepsen.util 1709 :invoke :read [29 nil] INFO [2022-03-13 17:10:31,260] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,260] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,260] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,261] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:31,261] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:10:31,262] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:10:31,263] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:31,264] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:10:31,265] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:31,266] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:31,267] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:31,268] jepsen worker 21 - jepsen.util 741 :invoke :write [7 9] INFO [2022-03-13 17:10:31,269] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,269] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:31,296] jepsen worker 89 - jepsen.util 1709 :ok :read [29 10] INFO [2022-03-13 17:10:31,297] jepsen worker 89 - jepsen.util 1709 :invoke :read [29 nil] INFO [2022-03-13 17:10:31,298] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,298] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,299] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:31,300] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:31,305] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,305] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,306] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:31,307] jepsen worker 76 - jepsen.util 616 :invoke :write [25 26] INFO [2022-03-13 17:10:31,308] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,309] jepsen worker 79 - jepsen.util 889 :invoke :write [26 22] INFO [2022-03-13 17:10:31,318] jepsen worker 78 - jepsen.util 438 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:31,319] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:31,324] jepsen worker 89 - jepsen.util 1709 :ok :read [29 10] INFO [2022-03-13 17:10:31,325] jepsen worker 89 - jepsen.util 1709 :invoke :read [29 nil] INFO [2022-03-13 17:10:31,354] jepsen worker 89 - jepsen.util 1709 :ok :read [29 10] INFO [2022-03-13 17:10:31,355] jepsen worker 89 - jepsen.util 1709 :invoke :write [29 20] INFO [2022-03-13 17:10:31,696] jepsen worker 49 - jepsen.util 1219 :ok :read [16 :nil] INFO [2022-03-13 17:10:31,697] jepsen worker 49 - jepsen.util 1219 :invoke :write [16 39] INFO [2022-03-13 17:10:31,766] jepsen worker 48 - jepsen.util 1308 :ok :read [16 :nil] INFO [2022-03-13 17:10:31,767] jepsen worker 48 - jepsen.util 1308 :invoke :write [16 6] INFO [2022-03-13 17:10:31,779] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:31,780] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:10:31,792] jepsen worker 40 - jepsen.util 1390 :ok :read [13 39] INFO [2022-03-13 17:10:31,792] jepsen worker 40 - jepsen.util 1390 :invoke :read [13 nil] INFO [2022-03-13 17:10:31,892] jepsen worker 81 - jepsen.util 1701 :ok :read [27 4] INFO [2022-03-13 17:10:31,893] jepsen worker 81 - jepsen.util 1701 :invoke :read [27 nil] INFO [2022-03-13 17:10:31,927] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:31,928] jepsen worker 41 - jepsen.util 1211 :invoke :read [13 nil] INFO [2022-03-13 17:10:31,964] jepsen worker 40 - jepsen.util 1390 :ok :read [13 39] INFO [2022-03-13 17:10:31,964] jepsen worker 40 - jepsen.util 1390 :invoke :write [13 40] INFO [2022-03-13 17:10:32,005] jepsen worker 81 - jepsen.util 1701 :ok :read [27 4] INFO [2022-03-13 17:10:32,006] jepsen worker 81 - jepsen.util 1701 :invoke :read [27 nil] INFO [2022-03-13 17:10:32,041] jepsen worker 41 - jepsen.util 1211 :ok :read [13 39] INFO [2022-03-13 17:10:32,042] jepsen worker 41 - jepsen.util 1211 :invoke :write [13 26] INFO [2022-03-13 17:10:32,096] jepsen worker 4 - jepsen.util 1534 :info :write [1 37] :RequestTimeoutException INFO [2022-03-13 17:10:32,097] jepsen worker 4 - jepsen.util 1624 :invoke :read [1 nil] INFO [2022-03-13 17:10:32,147] jepsen worker 81 - jepsen.util 1701 :ok :read [27 4] INFO [2022-03-13 17:10:32,148] jepsen worker 81 - jepsen.util 1701 :invoke :write [27 4] INFO [2022-03-13 17:10:32,200] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2844574780058651}, :ns_1@172.23.105.187 {:progress 0.3859649122807017}, :ns_1@172.23.107.227 {:progress 0.2844574780058651}, :ns_1@172.23.107.228 {:progress 0.2941176470588235}} INFO [2022-03-13 17:10:32,200] jepsen worker nemesis - couchbase.util Rebalance check count: 144 INFO [2022-03-13 17:10:32,200] jepsen worker nemesis - couchbase.util Rebalance stuck count: 83 INFO [2022-03-13 17:10:32,212] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,212] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,212] jepsen worker 60 - jepsen.util 600 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,212] jepsen worker 35 - jepsen.util 755 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,212] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,214] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:32,215] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:32,216] jepsen worker 35 - jepsen.util 755 :invoke :write [11 11] INFO [2022-03-13 17:10:32,216] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:32,217] jepsen worker 60 - jepsen.util 600 :invoke :write [20 42] INFO [2022-03-13 17:10:32,219] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,220] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:10:32,253] jepsen worker 76 - jepsen.util 616 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,253] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:32,255] jepsen worker 44 - jepsen.util 494 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,255] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:32,324] jepsen worker 21 - jepsen.util 741 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,325] jepsen worker 26 - jepsen.util 476 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,325] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,325] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,325] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:32,326] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:32,326] jepsen worker 14 - jepsen.util 284 :invoke :write [4 35] INFO [2022-03-13 17:10:32,327] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,327] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,327] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:32,328] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,328] jepsen worker 62 - jepsen.util 512 :invoke :write [20 7] INFO [2022-03-13 17:10:32,328] jepsen worker 61 - jepsen.util 601 :invoke :write [20 21] INFO [2022-03-13 17:10:32,329] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:10:32,329] jepsen worker 53 - jepsen.util 503 :invoke :write [17 21] INFO [2022-03-13 17:10:32,335] jepsen worker 26 - jepsen.util 476 :invoke :write [8 4] INFO [2022-03-13 17:10:32,367] jepsen worker 79 - jepsen.util 889 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,368] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,369] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,369] jepsen worker 76 - jepsen.util 616 :invoke :write [25 26] INFO [2022-03-13 17:10:32,371] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,371] jepsen worker 79 - jepsen.util 889 :invoke :write [26 10] INFO [2022-03-13 17:10:32,412] jepsen worker 43 - jepsen.util 403 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,413] jepsen worker 43 - jepsen.util 403 :invoke :write [14 15] INFO [2022-03-13 17:10:32,424] jepsen worker 7 - jepsen.util 457 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,425] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:32,469] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,470] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:10:32,477] jepsen worker 76 - jepsen.util 616 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,477] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:32,500] jepsen worker 70 - jepsen.util 700 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,501] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:10:32,542] jepsen worker 23 - jepsen.util 563 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,543] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:32,567] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,567] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] INFO [2022-03-13 17:10:32,584] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,584] jepsen worker 79 - jepsen.util 889 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,585] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,585] jepsen worker 76 - jepsen.util 616 :invoke :write [25 48] INFO [2022-03-13 17:10:32,586] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,587] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,588] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,589] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,590] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,591] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,592] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,592] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,594] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,594] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,595] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,596] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:10:32,619] jepsen worker 62 - jepsen.util 512 :fail :write [20 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,620] jepsen worker 62 - jepsen.util 512 :invoke :write [20 35] INFO [2022-03-13 17:10:32,660] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,660] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,660] jepsen worker 25 - jepsen.util 385 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,660] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,660] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:32,660] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,661] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:32,662] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:32,662] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:32,663] jepsen worker 80 - jepsen.util 620 :invoke :write [26 3] INFO [2022-03-13 17:10:32,663] jepsen worker 78 - jepsen.util 438 :invoke :write [26 35] INFO [2022-03-13 17:10:32,670] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,670] jepsen worker 52 - jepsen.util 502 :invoke :write [17 1] INFO [2022-03-13 17:10:32,671] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:10:32,670] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:32,681] jepsen worker 21 - jepsen.util 741 :invoke :write [7 33] INFO [2022-03-13 17:10:32,693] jepsen worker 76 - jepsen.util 616 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,695] jepsen worker 76 - jepsen.util 616 :invoke :write [25 4] INFO [2022-03-13 17:10:32,727] jepsen worker 6 - jepsen.util 546 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,727] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:32,740] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,740] jepsen worker 12 - jepsen.util 732 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,740] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,741] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:10:32,742] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:32,742] jepsen worker 51 - jepsen.util 591 :invoke :write [17 27] INFO [2022-03-13 17:10:32,767] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,767] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,770] jepsen worker 88 - jepsen.util 1258 :info :write [29 33] :RequestTimeoutException INFO [2022-03-13 17:10:32,771] jepsen worker 88 - jepsen.util 1348 :invoke :write [29 38] INFO [2022-03-13 17:10:32,779] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,780] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:10:32,799] jepsen worker 61 - jepsen.util 601 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,800] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:32,873] jepsen worker 76 - jepsen.util 616 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,873] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,874] jepsen worker 76 - jepsen.util 616 :invoke :write [25 43] INFO [2022-03-13 17:10:32,874] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:32,876] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,876] jepsen worker 79 - jepsen.util 889 :invoke :write [26 24] INFO [2022-03-13 17:10:32,916] jepsen worker 60 - jepsen.util 600 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,917] jepsen worker 60 - jepsen.util 600 :invoke :write [20 29] INFO [2022-03-13 17:10:32,939] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,940] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:32,944] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:32,944] jepsen worker 42 - jepsen.util 402 :invoke :write [14 47] INFO [2022-03-13 17:10:33,005] jepsen worker 76 - jepsen.util 616 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,006] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:33,006] jepsen worker 84 - jepsen.util 1614 :info :write [28 26] :RequestTimeoutException INFO [2022-03-13 17:10:33,007] jepsen worker 84 - jepsen.util 1704 :invoke :write [28 16] INFO [2022-03-13 17:10:33,013] jepsen worker 24 - jepsen.util 474 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,013] jepsen worker 34 - jepsen.util 754 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,013] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:33,014] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,014] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:33,014] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,014] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:33,015] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:33,015] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:33,015] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:33,069] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,070] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:33,107] jepsen worker 79 - jepsen.util 889 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,107] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,107] jepsen worker 79 - jepsen.util 889 :invoke :write [26 48] INFO [2022-03-13 17:10:33,108] jepsen worker 76 - jepsen.util 616 :invoke :write [25 31] INFO [2022-03-13 17:10:33,134] jepsen worker 43 - jepsen.util 403 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,134] jepsen worker 26 - jepsen.util 476 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,135] jepsen worker 43 - jepsen.util 403 :invoke :write [14 34] INFO [2022-03-13 17:10:33,136] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:33,138] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,138] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:33,139] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:33,139] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:33,150] jepsen worker 35 - jepsen.util 755 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,150] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,150] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:33,150] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,150] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,151] jepsen worker 7 - jepsen.util 457 :invoke :write [2 14] INFO [2022-03-13 17:10:33,151] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:33,152] jepsen worker 25 - jepsen.util 385 :invoke :write [8 21] INFO [2022-03-13 17:10:33,153] jepsen worker 13 - jepsen.util 553 :invoke :write [4 45] INFO [2022-03-13 17:10:33,153] jepsen worker 22 - jepsen.util 562 :invoke :write [7 31] INFO [2022-03-13 17:10:33,156] jepsen worker 21 - jepsen.util 741 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,157] jepsen worker 21 - jepsen.util 741 :invoke :write [7 37] INFO [2022-03-13 17:10:33,206] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2844574780058651}, :ns_1@172.23.105.187 {:progress 0.3859649122807017}, :ns_1@172.23.107.227 {:progress 0.2844574780058651}, :ns_1@172.23.107.228 {:progress 0.2941176470588235}} INFO [2022-03-13 17:10:33,206] jepsen worker nemesis - couchbase.util Rebalance check count: 145 INFO [2022-03-13 17:10:33,206] jepsen worker nemesis - couchbase.util Rebalance stuck count: 84 INFO [2022-03-13 17:10:33,263] jepsen worker 79 - jepsen.util 889 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,263] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:33,269] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,269] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,269] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:33,270] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,270] jepsen worker 70 - jepsen.util 700 :invoke :write [23 31] INFO [2022-03-13 17:10:33,270] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,271] jepsen worker 34 - jepsen.util 754 :invoke :write [11 28] INFO [2022-03-13 17:10:33,272] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:33,272] jepsen worker 44 - jepsen.util 494 :invoke :write [14 12] INFO [2022-03-13 17:10:33,274] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,275] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:33,276] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,277] jepsen worker 71 - jepsen.util 431 :invoke :write [23 39] INFO [2022-03-13 17:10:33,280] jepsen worker 6 - jepsen.util 546 :invoke :write [2 42] INFO [2022-03-13 17:10:33,287] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,287] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,287] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,287] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:33,288] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:33,288] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:10:33,340] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,340] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:33,391] jepsen worker 37 - jepsen.util 1477 :ok :read [12 20] INFO [2022-03-13 17:10:33,391] jepsen worker 37 - jepsen.util 1477 :invoke :read [12 nil] INFO [2022-03-13 17:10:33,400] jepsen worker 76 - jepsen.util 616 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,400] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,401] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:33,401] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:33,413] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,413] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,414] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:33,415] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:33,416] jepsen worker 21 - jepsen.util 741 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,416] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,416] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:33,416] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,417] jepsen worker 69 - jepsen.util 609 :invoke :write [23 6] INFO [2022-03-13 17:10:33,417] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:33,418] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,418] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:33,419] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,420] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:10:33,423] jepsen worker 14 - jepsen.util 284 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,424] jepsen worker 14 - jepsen.util 284 :invoke :write [4 37] INFO [2022-03-13 17:10:33,424] jepsen worker 15 - jepsen.util 1365 :info :write [5 8] :RequestTimeoutException INFO [2022-03-13 17:10:33,424] jepsen worker 15 - jepsen.util 1455 :invoke :write [5 29] INFO [2022-03-13 17:10:33,426] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,427] jepsen worker 76 - jepsen.util 616 :invoke :write [25 3] INFO [2022-03-13 17:10:33,431] jepsen worker 37 - jepsen.util 1477 :ok :read [12 20] INFO [2022-03-13 17:10:33,432] jepsen worker 37 - jepsen.util 1477 :invoke :write [12 38] INFO [2022-03-13 17:10:33,438] jepsen worker 78 - jepsen.util 438 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,439] jepsen worker 78 - jepsen.util 438 :invoke :write [26 13] INFO [2022-03-13 17:10:33,446] jepsen worker 68 - jepsen.util 1328 :info :write [22 46] :RequestTimeoutException INFO [2022-03-13 17:10:33,447] jepsen worker 68 - jepsen.util 1418 :invoke :read [22 nil] INFO [2022-03-13 17:10:33,571] jepsen worker 76 - jepsen.util 616 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,571] jepsen worker 76 - jepsen.util 616 :invoke :write [25 41] INFO [2022-03-13 17:10:33,596] jepsen worker 51 - jepsen.util 591 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,597] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:33,600] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,600] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:33,600] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,601] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:10:33,601] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:33,613] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:10:33,677] jepsen worker 53 - jepsen.util 503 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,678] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:33,680] jepsen worker 80 - jepsen.util 620 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,680] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,680] jepsen worker 80 - jepsen.util 620 :invoke :write [26 9] INFO [2022-03-13 17:10:33,681] jepsen worker 79 - jepsen.util 889 :invoke :write [26 26] INFO [2022-03-13 17:10:33,750] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,750] jepsen worker 60 - jepsen.util 600 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,750] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:33,751] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:33,751] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:10:33,752] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:33,758] jepsen worker 69 - jepsen.util 609 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,759] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:10:33,767] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:33,767] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:33,788] jepsen worker 62 - jepsen.util 512 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,788] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,789] jepsen worker 61 - jepsen.util 601 :invoke :write [20 8] INFO [2022-03-13 17:10:33,789] jepsen worker 62 - jepsen.util 512 :invoke :write [20 9] INFO [2022-03-13 17:10:33,797] jepsen worker 58 - jepsen.util 1228 :info :write [19 44] :SyncWriteAmbiguous INFO [2022-03-13 17:10:33,798] jepsen worker 58 - jepsen.util 1318 :invoke :read [19 nil] INFO [2022-03-13 17:10:33,803] jepsen worker 70 - jepsen.util 700 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,803] jepsen worker 70 - jepsen.util 700 :invoke :write [23 31] INFO [2022-03-13 17:10:33,900] jepsen worker 46 - jepsen.util 1486 :info :write [15 42] :RequestTimeoutException INFO [2022-03-13 17:10:33,901] jepsen worker 46 - jepsen.util 1576 :invoke :write [15 16] INFO [2022-03-13 17:10:33,905] jepsen worker 16 - jepsen.util 1366 :info :write [5 47] :SyncWriteAmbiguous INFO [2022-03-13 17:10:33,905] jepsen worker 16 - jepsen.util 1456 :invoke :write [5 32] INFO [2022-03-13 17:10:33,930] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,931] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:10:33,931] jepsen worker 65 - jepsen.util 1505 :info :write [21 29] :RequestTimeoutException INFO [2022-03-13 17:10:33,932] jepsen worker 65 - jepsen.util 1595 :invoke :write [21 32] INFO [2022-03-13 17:10:33,950] jepsen worker 27 - jepsen.util 1557 :info :write [9 25] :RequestTimeoutException INFO [2022-03-13 17:10:33,951] jepsen worker 27 - jepsen.util 1647 :invoke :read [9 nil] INFO [2022-03-13 17:10:33,965] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:33,965] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,965] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:33,966] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:33,975] jepsen worker 42 - jepsen.util 402 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,975] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,975] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,975] jepsen worker 43 - jepsen.util 403 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,976] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:33,976] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:33,976] jepsen worker 42 - jepsen.util 402 :invoke :write [14 9] INFO [2022-03-13 17:10:33,977] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:33,978] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,978] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,978] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,978] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:33,979] jepsen worker 51 - jepsen.util 591 :invoke :write [17 14] INFO [2022-03-13 17:10:33,979] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:33,999] jepsen worker 34 - jepsen.util 754 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,999] jepsen worker 52 - jepsen.util 502 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:33,999] jepsen worker 34 - jepsen.util 754 :invoke :write [11 24] INFO [2022-03-13 17:10:34,000] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:34,019] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,020] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:34,032] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,032] jepsen worker 76 - jepsen.util 616 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,033] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,033] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:34,033] jepsen worker 76 - jepsen.util 616 :invoke :write [25 30] INFO [2022-03-13 17:10:34,033] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,033] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,034] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,037] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:34,038] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:34,038] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,039] jepsen worker 60 - jepsen.util 600 :invoke :write [20 15] INFO [2022-03-13 17:10:34,039] jepsen worker 12 - jepsen.util 732 :invoke :write [4 20] INFO [2022-03-13 17:10:34,042] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:34,064] jepsen worker 68 - jepsen.util 1418 :ok :read [22 30] INFO [2022-03-13 17:10:34,064] jepsen worker 68 - jepsen.util 1418 :invoke :read [22 nil] INFO [2022-03-13 17:10:34,071] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,071] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:34,071] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,071] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:10:34,071] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,071] jepsen worker 21 - jepsen.util 741 :invoke :write [7 11] INFO [2022-03-13 17:10:34,072] jepsen worker 33 - jepsen.util 663 :invoke :write [11 44] INFO [2022-03-13 17:10:34,072] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:34,074] jepsen worker 14 - jepsen.util 284 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,074] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:34,077] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,078] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:34,087] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,088] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,090] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,092] jepsen worker 69 - jepsen.util 609 :invoke :write [23 18] INFO [2022-03-13 17:10:34,093] jepsen worker 68 - jepsen.util 1418 :ok :read [22 30] INFO [2022-03-13 17:10:34,095] jepsen worker 68 - jepsen.util 1418 :invoke :write [22 32] INFO [2022-03-13 17:10:34,108] jepsen worker 62 - jepsen.util 512 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,108] jepsen worker 51 - jepsen.util 591 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,108] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:34,108] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,108] jepsen worker 62 - jepsen.util 512 :invoke :write [20 0] INFO [2022-03-13 17:10:34,108] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:34,109] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:34,110] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:34,111] jepsen worker 51 - jepsen.util 591 :invoke :write [17 23] INFO [2022-03-13 17:10:34,113] jepsen worker 77 - jepsen.util 707 :invoke :write [25 41] INFO [2022-03-13 17:10:34,122] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,123] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,139] jepsen worker 44 - jepsen.util 494 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,140] jepsen worker 44 - jepsen.util 494 :invoke :write [14 2] INFO [2022-03-13 17:10:34,140] jepsen worker 7 - jepsen.util 457 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,140] jepsen worker 7 - jepsen.util 457 :invoke :write [2 3] INFO [2022-03-13 17:10:34,196] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,197] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,212] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2903225806451613}, :ns_1@172.23.105.187 {:progress 0.3918128654970761}, :ns_1@172.23.107.227 {:progress 0.2903225806451613}, :ns_1@172.23.107.228 {:progress 0.3}} INFO [2022-03-13 17:10:34,212] jepsen worker nemesis - couchbase.util Rebalance check count: 146 INFO [2022-03-13 17:10:34,212] jepsen worker nemesis - couchbase.util Rebalance stuck count: 84 INFO [2022-03-13 17:10:34,280] jepsen worker 50 - jepsen.util 1400 :info :write [16 29] :RequestTimeoutException INFO [2022-03-13 17:10:34,280] jepsen worker 50 - jepsen.util 1490 :invoke :read [16 nil] INFO [2022-03-13 17:10:34,330] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,331] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,348] jepsen worker 50 - jepsen.util 1490 :ok :read [16 :nil] INFO [2022-03-13 17:10:34,349] jepsen worker 50 - jepsen.util 1490 :invoke :write [16 28] INFO [2022-03-13 17:10:34,365] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,365] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,394] jepsen worker 58 - jepsen.util 1318 :ok :read [19 17] INFO [2022-03-13 17:10:34,395] jepsen worker 58 - jepsen.util 1318 :invoke :write [19 34] INFO [2022-03-13 17:10:34,413] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,413] jepsen worker 85 - jepsen.util 1435 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,489] jepsen worker 85 - jepsen.util 1435 :ok :read [28 44] INFO [2022-03-13 17:10:34,490] jepsen worker 85 - jepsen.util 1435 :invoke :write [28 15] INFO [2022-03-13 17:10:34,616] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,616] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:34,689] jepsen worker 12 - jepsen.util 732 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,689] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,689] jepsen worker 61 - jepsen.util 601 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,690] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:34,691] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:34,691] jepsen worker 61 - jepsen.util 601 :invoke :write [20 4] INFO [2022-03-13 17:10:34,706] jepsen worker 67 - jepsen.util 1237 :info :write [22 16] :RequestTimeoutException INFO [2022-03-13 17:10:34,707] jepsen worker 67 - jepsen.util 1327 :invoke :read [22 nil] INFO [2022-03-13 17:10:34,726] jepsen worker 33 - jepsen.util 663 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,727] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:34,756] jepsen worker 86 - jepsen.util 1256 :info :write [28 42] :RequestTimeoutException INFO [2022-03-13 17:10:34,757] jepsen worker 86 - jepsen.util 1346 :invoke :read [28 nil] INFO [2022-03-13 17:10:34,786] jepsen worker 67 - jepsen.util 1327 :ok :read [22 30] INFO [2022-03-13 17:10:34,787] jepsen worker 67 - jepsen.util 1327 :invoke :read [22 nil] INFO [2022-03-13 17:10:34,794] jepsen worker 6 - jepsen.util 546 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,795] jepsen worker 25 - jepsen.util 385 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,795] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:34,796] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:34,820] jepsen worker 11 - jepsen.util 1271 :info :write [3 3] :RequestTimeoutException INFO [2022-03-13 17:10:34,821] jepsen worker 11 - jepsen.util 1361 :invoke :write [3 5] INFO [2022-03-13 17:10:34,854] jepsen worker 67 - jepsen.util 1327 :ok :read [22 30] INFO [2022-03-13 17:10:34,854] jepsen worker 21 - jepsen.util 741 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,855] jepsen worker 67 - jepsen.util 1327 :invoke :read [22 nil] INFO [2022-03-13 17:10:34,856] jepsen worker 21 - jepsen.util 741 :invoke :write [7 40] INFO [2022-03-13 17:10:34,915] jepsen worker 60 - jepsen.util 600 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,915] jepsen worker 42 - jepsen.util 402 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,915] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:34,915] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:34,916] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,916] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,917] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:34,918] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:34,918] jepsen worker 43 - jepsen.util 403 :invoke :write [14 21] INFO [2022-03-13 17:10:34,919] jepsen worker 13 - jepsen.util 553 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,919] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:34,919] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] INFO [2022-03-13 17:10:34,920] jepsen worker 13 - jepsen.util 553 :invoke :write [4 27] INFO [2022-03-13 17:10:34,920] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:34,931] jepsen worker 69 - jepsen.util 609 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,932] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,932] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,933] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:10:34,934] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:34,934] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:10:34,945] jepsen worker 67 - jepsen.util 1327 :ok :read [22 30] INFO [2022-03-13 17:10:34,946] jepsen worker 67 - jepsen.util 1327 :invoke :write [22 17] INFO [2022-03-13 17:10:34,971] jepsen worker 71 - jepsen.util 431 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:34,973] jepsen worker 71 - jepsen.util 431 :invoke :write [23 12] INFO [2022-03-13 17:10:35,013] jepsen worker 77 - jepsen.util 707 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,014] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:35,038] jepsen worker 21 - jepsen.util 741 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,038] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,039] jepsen worker 33 - jepsen.util 663 :invoke :write [11 11] INFO [2022-03-13 17:10:35,040] jepsen worker 21 - jepsen.util 741 :invoke :write [7 43] INFO [2022-03-13 17:10:35,088] jepsen worker 62 - jepsen.util 512 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,089] jepsen worker 62 - jepsen.util 512 :invoke :write [20 44] INFO [2022-03-13 17:10:35,091] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,091] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:35,166] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,167] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:35,183] jepsen worker 70 - jepsen.util 700 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,183] jepsen worker 22 - jepsen.util 562 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,184] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,184] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,184] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,184] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:35,185] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:35,186] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,186] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:35,186] jepsen worker 24 - jepsen.util 474 :invoke :write [8 20] INFO [2022-03-13 17:10:35,219] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2903225806451613}, :ns_1@172.23.105.187 {:progress 0.3918128654970761}, :ns_1@172.23.107.227 {:progress 0.2903225806451613}, :ns_1@172.23.107.228 {:progress 0.3}} INFO [2022-03-13 17:10:35,219] jepsen worker nemesis - couchbase.util Rebalance check count: 147 INFO [2022-03-13 17:10:35,219] jepsen worker nemesis - couchbase.util Rebalance stuck count: 85 INFO [2022-03-13 17:10:35,224] jepsen worker 17 - jepsen.util 1367 :info :write [5 21] :RequestTimeoutException INFO [2022-03-13 17:10:35,225] jepsen worker 17 - jepsen.util 1457 :invoke :read [5 nil] INFO [2022-03-13 17:10:35,245] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,245] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,257] jepsen worker 3 - jepsen.util 1263 :info :write [1 32] :RequestTimeoutException INFO [2022-03-13 17:10:35,257] jepsen worker 3 - jepsen.util 1353 :invoke :write [1 10] INFO [2022-03-13 17:10:35,259] jepsen worker 61 - jepsen.util 601 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,259] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,259] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,260] jepsen worker 61 - jepsen.util 601 :invoke :write [20 2] INFO [2022-03-13 17:10:35,261] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,261] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,262] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:35,262] jepsen worker 51 - jepsen.util 591 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,263] jepsen worker 14 - jepsen.util 284 :invoke :write [4 27] INFO [2022-03-13 17:10:35,264] jepsen worker 60 - jepsen.util 600 :invoke :write [20 33] INFO [2022-03-13 17:10:35,264] jepsen worker 12 - jepsen.util 732 :invoke :write [4 12] INFO [2022-03-13 17:10:35,265] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:35,266] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:35,270] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:35,291] jepsen worker 2 - jepsen.util 1622 :info :write [0 33] :SyncWriteAmbiguous INFO [2022-03-13 17:10:35,293] jepsen worker 2 - jepsen.util 1712 :invoke :read [0 nil] INFO [2022-03-13 17:10:35,332] jepsen worker 2 - jepsen.util 1712 :ok :read [0 41] INFO [2022-03-13 17:10:35,333] jepsen worker 2 - jepsen.util 1712 :invoke :write [0 41] INFO [2022-03-13 17:10:35,338] jepsen worker 21 - jepsen.util 741 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,338] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:35,362] jepsen worker 79 - jepsen.util 889 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,363] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:10:35,386] jepsen worker 38 - jepsen.util 1658 :info :write [12 27] :RequestTimeoutException INFO [2022-03-13 17:10:35,387] jepsen worker 38 - jepsen.util 1748 :invoke :write [12 47] INFO [2022-03-13 17:10:35,395] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,395] jepsen worker 7 - jepsen.util 457 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,396] jepsen worker 7 - jepsen.util 457 :invoke :write [2 36] INFO [2022-03-13 17:10:35,397] jepsen worker 35 - jepsen.util 755 :invoke :write [11 5] INFO [2022-03-13 17:10:35,433] jepsen worker 31 - jepsen.util 1291 :info :write [10 31] :SyncWriteAmbiguous INFO [2022-03-13 17:10:35,433] jepsen worker 31 - jepsen.util 1381 :invoke :read [10 nil] INFO [2022-03-13 17:10:35,438] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,438] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,438] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:35,439] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:10:35,440] jepsen worker 21 - jepsen.util 741 :invoke :write [7 16] INFO [2022-03-13 17:10:35,440] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,442] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,443] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,445] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,446] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,448] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,449] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,518] jepsen worker 1 - jepsen.util 1171 :info :write [0 22] :RequestTimeoutException INFO [2022-03-13 17:10:35,518] jepsen worker 1 - jepsen.util 1261 :invoke :read [0 nil] INFO [2022-03-13 17:10:35,539] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,540] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,540] jepsen worker 43 - jepsen.util 403 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,540] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,540] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:35,541] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,541] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,542] jepsen worker 51 - jepsen.util 591 :invoke :write [17 45] INFO [2022-03-13 17:10:35,543] jepsen worker 26 - jepsen.util 476 :invoke :write [8 29] INFO [2022-03-13 17:10:35,544] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:35,544] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:35,545] jepsen worker 77 - jepsen.util 707 :invoke :write [25 36] INFO [2022-03-13 17:10:35,554] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,555] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,556] jepsen worker 25 - jepsen.util 385 :invoke :write [8 23] INFO [2022-03-13 17:10:35,556] jepsen worker 6 - jepsen.util 546 :invoke :write [2 18] INFO [2022-03-13 17:10:35,564] jepsen worker 18 - jepsen.util 1098 :info :write [6 1] :SyncWriteAmbiguous INFO [2022-03-13 17:10:35,566] jepsen worker 18 - jepsen.util 1188 :invoke :write [6 49] INFO [2022-03-13 17:10:35,631] jepsen worker 28 - jepsen.util 1468 :info :write [9 3] :RequestTimeoutException INFO [2022-03-13 17:10:35,632] jepsen worker 28 - jepsen.util 1558 :invoke :write [9 19] INFO [2022-03-13 17:10:35,648] jepsen worker 39 - jepsen.util 1479 :info :write [13 36] :RequestTimeoutException INFO [2022-03-13 17:10:35,648] jepsen worker 39 - jepsen.util 1569 :invoke :read [13 nil] WARN [2022-03-13 17:10:35,696] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16502417,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":16502417,"total_server_duration_us":32,"operation_id":"0x3cc9","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":16503481},{"operation_name":"upsert","last_dispatch_duration_us":52815,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2786092,"total_server_duration_us":301916,"encode_duration_us":17228,"operation_id":"0x3edd","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10009163},{"operation_name":"upsert","last_dispatch_duration_us":1782690,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2484809,"total_server_duration_us":228231,"encode_duration_us":20847,"operation_id":"0x3f2b","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10008753},{"operation_name":"upsert","last_dispatch_duration_us":115341,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2834452,"total_server_duration_us":289519,"encode_duration_us":19032,"operation_id":"0x3edf","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10006194},{"operation_name":"upsert","last_dispatch_duration_us":9039664,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9423772,"total_server_duration_us":9270829,"encode_duration_us":17427,"operation_id":"0x3ee3","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10003629},{"operation_name":"upsert","last_dispatch_duration_us":86057,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2786006,"total_server_duration_us":296930,"encode_duration_us":17248,"operation_id":"0x3ede","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10002575},{"operation_name":"upsert","last_dispatch_duration_us":41518,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9162160,"total_server_duration_us":243709,"encode_duration_us":21415,"operation_id":"0x3c6b","timeout_ms":10000,"last_server_duration_us":27222,"total_duration_us":10001401},{"operation_name":"upsert","last_dispatch_duration_us":1650258,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1650258,"total_server_duration_us":72453,"encode_duration_us":25923,"operation_id":"0x4256","timeout_ms":10000,"last_server_duration_us":72453,"total_duration_us":1654639},{"operation_name":"get","last_dispatch_duration_us":17504,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":17504,"total_server_duration_us":3686,"operation_id":"0x3fca","timeout_ms":30000,"last_server_duration_us":3686,"total_duration_us":1612511},{"operation_name":"upsert","last_dispatch_duration_us":1018238,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1018238,"total_server_duration_us":87359,"encode_duration_us":25111,"operation_id":"0x4172","timeout_ms":10000,"last_server_duration_us":87359,"total_duration_us":1032742}],"total_count":18}} WARN [2022-03-13 17:10:35,696] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9039664,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9423772,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9270829,"encode_duration_us":17427,"operation_id":"0x3ee3","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10003629},{"operation_name":"upsert","last_dispatch_duration_us":18135421,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":18135421,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9072420,"encode_duration_us":27148,"operation_id":"0x3c49","timeout_ms":10000,"last_server_duration_us":9072420,"total_duration_us":10005842}],"total_count":2}} INFO [2022-03-13 17:10:35,721] jepsen worker 64 - jepsen.util 1144 :info :write [21 1] :SyncWriteAmbiguous INFO [2022-03-13 17:10:35,722] jepsen worker 64 - jepsen.util 1234 :invoke :write [21 9] INFO [2022-03-13 17:10:35,724] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,725] jepsen worker 70 - jepsen.util 700 :invoke :write [23 7] INFO [2022-03-13 17:10:35,775] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,779] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:35,787] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,788] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:35,790] jepsen worker 73 - jepsen.util 1423 :ok :read [24 24] INFO [2022-03-13 17:10:35,790] jepsen worker 73 - jepsen.util 1423 :invoke :write [24 21] INFO [2022-03-13 17:10:35,834] jepsen worker 78 - jepsen.util 438 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,835] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:35,876] jepsen worker 54 - jepsen.util 1224 :ok :read [18 21] INFO [2022-03-13 17:10:35,877] jepsen worker 54 - jepsen.util 1224 :invoke :read [18 nil] INFO [2022-03-13 17:10:35,879] jepsen worker 55 - jepsen.util 1405 :ok :read [18 21] INFO [2022-03-13 17:10:35,880] jepsen worker 55 - jepsen.util 1405 :invoke :write [18 9] INFO [2022-03-13 17:10:35,901] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:35,901] jepsen worker 12 - jepsen.util 732 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,902] jepsen worker 43 - jepsen.util 403 :invoke :write [14 21] INFO [2022-03-13 17:10:35,902] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:35,902] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:35,903] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:35,919] jepsen worker 34 - jepsen.util 754 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,919] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:35,919] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,919] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,920] jepsen worker 52 - jepsen.util 502 :invoke :write [17 13] INFO [2022-03-13 17:10:35,920] jepsen worker 22 - jepsen.util 562 :invoke :write [7 21] INFO [2022-03-13 17:10:35,921] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:35,923] jepsen worker 8 - jepsen.util 638 :invoke :write [2 43] INFO [2022-03-13 17:10:35,927] jepsen worker 33 - jepsen.util 663 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,928] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:10:35,996] jepsen worker 14 - jepsen.util 284 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:35,996] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:36,008] jepsen worker 76 - jepsen.util 616 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,009] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:36,051] jepsen worker 21 - jepsen.util 741 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,051] jepsen worker 21 - jepsen.util 741 :invoke :write [7 22] INFO [2022-03-13 17:10:36,133] jepsen worker 24 - jepsen.util 474 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,133] jepsen worker 6 - jepsen.util 546 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,133] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:36,134] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:36,206] jepsen worker 77 - jepsen.util 707 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,206] jepsen worker 61 - jepsen.util 601 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,207] jepsen worker 77 - jepsen.util 707 :invoke :write [25 27] INFO [2022-03-13 17:10:36,207] jepsen worker 61 - jepsen.util 601 :invoke :write [20 11] INFO [2022-03-13 17:10:36,220] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,220] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,221] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:36,221] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:36,225] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2903225806451613}, :ns_1@172.23.105.187 {:progress 0.3918128654970761}, :ns_1@172.23.107.227 {:progress 0.2903225806451613}, :ns_1@172.23.107.228 {:progress 0.3}} INFO [2022-03-13 17:10:36,226] jepsen worker nemesis - couchbase.util Rebalance check count: 148 INFO [2022-03-13 17:10:36,226] jepsen worker nemesis - couchbase.util Rebalance stuck count: 86 INFO [2022-03-13 17:10:36,232] jepsen worker 82 - jepsen.util 1342 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,232] jepsen worker 83 - jepsen.util 983 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,232] jepsen worker 82 - jepsen.util 1342 :invoke :read [27 nil] INFO [2022-03-13 17:10:36,233] jepsen worker 83 - jepsen.util 983 :invoke :write [27 17] INFO [2022-03-13 17:10:36,257] jepsen worker 66 - jepsen.util 1326 :ok :read [22 30] INFO [2022-03-13 17:10:36,258] jepsen worker 66 - jepsen.util 1326 :invoke :write [22 27] INFO [2022-03-13 17:10:36,296] jepsen worker 36 - jepsen.util 1566 :info :write [12 11] :RequestTimeoutException INFO [2022-03-13 17:10:36,297] jepsen worker 36 - jepsen.util 1656 :invoke :write [12 5] INFO [2022-03-13 17:10:36,382] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,382] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:10:36,385] jepsen worker 7 - jepsen.util 457 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,385] jepsen worker 71 - jepsen.util 431 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,385] jepsen worker 7 - jepsen.util 457 :invoke :write [2 6] INFO [2022-03-13 17:10:36,386] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:36,443] jepsen worker 51 - jepsen.util 591 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,443] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,444] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:36,444] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:36,473] jepsen worker 72 - jepsen.util 1332 :info :write [24 30] :RequestTimeoutException INFO [2022-03-13 17:10:36,474] jepsen worker 30 - jepsen.util 1200 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:10:36,474] jepsen worker 72 - jepsen.util 1422 :invoke :read [24 nil] INFO [2022-03-13 17:10:36,475] jepsen worker 30 - jepsen.util 1290 :invoke :write [10 19] INFO [2022-03-13 17:10:36,496] jepsen worker 21 - jepsen.util 741 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,496] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,496] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,497] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:36,498] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:10:36,498] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:36,511] jepsen worker 72 - jepsen.util 1422 :ok :read [24 24] INFO [2022-03-13 17:10:36,511] jepsen worker 72 - jepsen.util 1422 :invoke :read [24 nil] INFO [2022-03-13 17:10:36,526] jepsen worker 35 - jepsen.util 755 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,526] jepsen worker 35 - jepsen.util 755 :invoke :write [11 29] INFO [2022-03-13 17:10:36,573] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,573] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,573] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:36,574] jepsen worker 69 - jepsen.util 609 :invoke :write [23 24] INFO [2022-03-13 17:10:36,574] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:36,575] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:10:36,605] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,605] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,605] jepsen worker 26 - jepsen.util 476 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,605] jepsen worker 60 - jepsen.util 600 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,605] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:36,605] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:36,606] jepsen worker 53 - jepsen.util 503 :invoke :write [17 36] INFO [2022-03-13 17:10:36,607] jepsen worker 60 - jepsen.util 600 :invoke :write [20 23] INFO [2022-03-13 17:10:36,607] jepsen worker 12 - jepsen.util 732 :invoke :write [4 31] INFO [2022-03-13 17:10:36,607] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:10:36,613] jepsen worker 70 - jepsen.util 700 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,613] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,614] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:36,614] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:36,650] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,651] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:36,651] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:10:36,652] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:36,678] jepsen worker 74 - jepsen.util 1694 :info :write [24 16] :RequestTimeoutException INFO [2022-03-13 17:10:36,678] jepsen worker 74 - jepsen.util 1784 :invoke :read [24 nil] INFO [2022-03-13 17:10:36,715] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,715] jepsen worker 79 - jepsen.util 889 :invoke :write [26 18] INFO [2022-03-13 17:10:36,717] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,717] jepsen worker 76 - jepsen.util 616 :invoke :write [25 31] INFO [2022-03-13 17:10:36,720] jepsen worker 19 - jepsen.util 1189 :info :write [6 48] :RequestTimeoutException INFO [2022-03-13 17:10:36,721] jepsen worker 19 - jepsen.util 1279 :invoke :write [6 17] INFO [2022-03-13 17:10:36,738] jepsen worker 69 - jepsen.util 609 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,738] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:36,745] jepsen worker 13 - jepsen.util 553 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,745] jepsen worker 13 - jepsen.util 553 :invoke :write [4 19] INFO [2022-03-13 17:10:36,768] jepsen worker 62 - jepsen.util 512 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,768] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,768] jepsen worker 43 - jepsen.util 403 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,769] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:36,769] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:36,770] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:36,780] jepsen worker 29 - jepsen.util 1379 :info :write [9 40] :RequestTimeoutException INFO [2022-03-13 17:10:36,781] jepsen worker 29 - jepsen.util 1469 :invoke :read [9 nil] INFO [2022-03-13 17:10:36,797] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,797] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,798] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,798] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,797] jepsen worker 44 - jepsen.util 494 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,798] jepsen worker 44 - jepsen.util 494 :invoke :write [14 23] INFO [2022-03-13 17:10:36,800] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:36,800] jepsen worker 34 - jepsen.util 754 :invoke :write [11 11] INFO [2022-03-13 17:10:36,801] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:36,801] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:36,827] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,827] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,827] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:36,827] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,828] jepsen worker 70 - jepsen.util 700 :invoke :write [23 37] INFO [2022-03-13 17:10:36,828] jepsen worker 71 - jepsen.util 431 :invoke :write [23 9] INFO [2022-03-13 17:10:36,830] jepsen worker 10 - jepsen.util 1270 :info :write [3 13] :RequestTimeoutException INFO [2022-03-13 17:10:36,831] jepsen worker 10 - jepsen.util 1360 :invoke :read [3 nil] INFO [2022-03-13 17:10:36,855] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:36,855] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,856] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:10:36,856] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:36,925] jepsen worker 61 - jepsen.util 601 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,925] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,925] jepsen worker 77 - jepsen.util 707 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,925] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:36,926] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:36,927] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:36,927] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,928] jepsen worker 61 - jepsen.util 601 :invoke :write [20 17] INFO [2022-03-13 17:10:36,928] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:36,929] jepsen worker 77 - jepsen.util 707 :invoke :write [25 40] INFO [2022-03-13 17:10:36,941] jepsen worker 52 - jepsen.util 502 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,942] jepsen worker 52 - jepsen.util 502 :invoke :write [17 2] INFO [2022-03-13 17:10:36,953] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,953] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:36,953] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:36,955] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:36,955] jepsen worker 69 - jepsen.util 609 :invoke :write [23 10] INFO [2022-03-13 17:10:36,955] jepsen worker 21 - jepsen.util 741 :invoke :write [7 16] INFO [2022-03-13 17:10:37,005] jepsen worker 34 - jepsen.util 754 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,005] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:37,021] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,021] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,021] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:37,022] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:10:37,072] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,072] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:37,072] jepsen worker 53 - jepsen.util 503 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,073] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,073] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:37,073] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,073] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,073] jepsen worker 53 - jepsen.util 503 :invoke :write [17 16] INFO [2022-03-13 17:10:37,074] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:37,075] jepsen worker 26 - jepsen.util 476 :invoke :write [8 37] INFO [2022-03-13 17:10:37,075] jepsen worker 62 - jepsen.util 512 :invoke :write [20 44] INFO [2022-03-13 17:10:37,075] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:37,092] jepsen worker 69 - jepsen.util 609 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,093] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] WARN [2022-03-13 17:10:37,193] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18956773,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18956773,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9062854,"encode_duration_us":19607,"operation_id":"0x3c74","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10008512},{"operation_name":"upsert","last_dispatch_duration_us":12796402,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12796402,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9038428,"encode_duration_us":16644,"operation_id":"0x3d79","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10010463}],"total_count":2}} WARN [2022-03-13 17:10:37,193] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9042419,"last_local_id":"A312307A00000004/0000000095638474","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9674716,"last_local_socket":"172.23.104.79:33498","total_server_duration_us":9426577,"encode_duration_us":18359,"operation_id":"0x3c98","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10005387}],"total_count":1}} INFO [2022-03-13 17:10:37,231] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2903225806451613}, :ns_1@172.23.105.187 {:progress 0.3918128654970761}, :ns_1@172.23.107.227 {:progress 0.2903225806451613}, :ns_1@172.23.107.228 {:progress 0.3}} INFO [2022-03-13 17:10:37,231] jepsen worker nemesis - couchbase.util Rebalance check count: 149 INFO [2022-03-13 17:10:37,231] jepsen worker nemesis - couchbase.util Rebalance stuck count: 87 INFO [2022-03-13 17:10:37,236] jepsen worker 45 - jepsen.util 1575 :info :write [15 23] :RequestTimeoutException INFO [2022-03-13 17:10:37,237] jepsen worker 45 - jepsen.util 1665 :invoke :write [15 7] WARN [2022-03-13 17:10:37,329] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13733035,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":13733035,"total_server_duration_us":49,"operation_id":"0x3e41","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":13733351},{"operation_name":"get","last_dispatch_duration_us":12694808,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12694808,"total_server_duration_us":19,"operation_id":"0x3eb5","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":12695037},{"operation_name":"get","last_dispatch_duration_us":10755668,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10755668,"total_server_duration_us":23,"operation_id":"0x3f64","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":10757178},{"operation_name":"upsert","last_dispatch_duration_us":12796402,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12796402,"total_server_duration_us":9038428,"encode_duration_us":16644,"operation_id":"0x3d79","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10010463},{"operation_name":"upsert","last_dispatch_duration_us":10782410,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10782410,"total_server_duration_us":20045,"encode_duration_us":17708,"operation_id":"0x3f65","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10008155},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":17801,"operation_id":"0x3ff0","timeout_ms":10000,"total_duration_us":10008031},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":22040,"operation_id":"0x3f81","timeout_ms":10000,"total_duration_us":10006627},{"operation_name":"upsert","last_dispatch_duration_us":9107096,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13881967,"total_server_duration_us":55936,"encode_duration_us":21162,"operation_id":"0x3d42","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10004314},{"operation_name":"upsert","last_dispatch_duration_us":6819817,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":7158308,"total_server_duration_us":7023426,"encode_duration_us":19229,"operation_id":"0x3e91","timeout_ms":10000,"last_server_duration_us":6807925,"total_duration_us":7774585},{"operation_name":"get","last_dispatch_duration_us":2917891,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":2917891,"total_server_duration_us":38,"operation_id":"0x4054","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":2919038}],"total_count":63}} WARN [2022-03-13 17:10:37,392] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17924208,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18215523,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9101901,"encode_duration_us":18581,"operation_id":"0x3bab","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10003336},{"operation_name":"upsert","last_dispatch_duration_us":9045890,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9794859,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9342162,"encode_duration_us":18464,"operation_id":"0x3d52","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10007952},{"operation_name":"upsert","last_dispatch_duration_us":17963136,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18060052,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9076418,"encode_duration_us":21892,"operation_id":"0x3bc0","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10003688}],"total_count":3}} WARN [2022-03-13 17:10:37,392] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9180322,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9180322,"total_server_duration_us":43,"operation_id":"0x3c9a","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":15370845},{"operation_name":"get","last_dispatch_duration_us":8866724,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8866724,"total_server_duration_us":32,"operation_id":"0x3cf3","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":14036739},{"operation_name":"get","last_dispatch_duration_us":8959417,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8959417,"total_server_duration_us":62,"operation_id":"0x3e19","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":10041430},{"operation_name":"upsert","last_dispatch_duration_us":34133,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1051528,"total_server_duration_us":615293,"encode_duration_us":17365,"operation_id":"0x3f9d","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10008379},{"operation_name":"upsert","last_dispatch_duration_us":9045890,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9794859,"total_server_duration_us":9342162,"encode_duration_us":18464,"operation_id":"0x3d52","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10007952},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17653,"operation_id":"0x3e7f","timeout_ms":10000,"total_duration_us":10007264},{"operation_name":"upsert","last_dispatch_duration_us":38244,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9552254,"total_server_duration_us":219671,"encode_duration_us":21500,"operation_id":"0x3ea4","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10007072},{"operation_name":"upsert","last_dispatch_duration_us":29842,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1117176,"total_server_duration_us":673146,"encode_duration_us":18837,"operation_id":"0x3f4d","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10004669},{"operation_name":"get","last_dispatch_duration_us":9202953,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9202953,"total_server_duration_us":49,"operation_id":"0x3e4e","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9206090},{"operation_name":"get","last_dispatch_duration_us":138391,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":138391,"total_server_duration_us":43,"operation_id":"0x4000","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":4793786}],"total_count":34}} WARN [2022-03-13 17:10:37,394] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":41391,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9681392,"total_server_duration_us":229404,"encode_duration_us":17335,"operation_id":"0x3dc9","timeout_ms":10000,"last_server_duration_us":27222,"total_duration_us":10009901},{"operation_name":"upsert","last_dispatch_duration_us":33309,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1014044,"total_server_duration_us":650505,"encode_duration_us":16946,"operation_id":"0x3e66","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10008289},{"operation_name":"upsert","last_dispatch_duration_us":31755,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":1279278,"total_server_duration_us":545189,"encode_duration_us":17366,"operation_id":"0x3ff8","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10004563},{"operation_name":"upsert","last_dispatch_duration_us":32520,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":970840,"total_server_duration_us":607200,"encode_duration_us":16912,"operation_id":"0x3d48","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10004385},{"operation_name":"upsert","last_dispatch_duration_us":75387,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8699458,"total_server_duration_us":303903,"encode_duration_us":17016,"operation_id":"0x3cd2","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10002170},{"operation_name":"upsert","last_dispatch_duration_us":30325,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":1262452,"total_server_duration_us":612294,"encode_duration_us":20045,"operation_id":"0x3fff","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10002127},{"operation_name":"upsert","last_dispatch_duration_us":41182,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":1197166,"total_server_duration_us":544543,"encode_duration_us":19851,"operation_id":"0x3ff6","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10002110},{"operation_name":"get","last_dispatch_duration_us":9056561,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9056561,"total_server_duration_us":43,"operation_id":"0x3dc8","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9058155},{"operation_name":"get","last_dispatch_duration_us":5688662,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5688662,"total_server_duration_us":62,"operation_id":"0x3ee2","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":5689671},{"operation_name":"upsert","last_dispatch_duration_us":1646556,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1646556,"total_server_duration_us":86182,"encode_duration_us":19827,"operation_id":"0x404f","timeout_ms":10000,"last_server_duration_us":86182,"total_duration_us":4272480}],"total_count":69}} WARN [2022-03-13 17:10:37,429] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9679545,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10513516,"total_server_duration_us":233277,"encode_duration_us":20319,"operation_id":"0x3f4b","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10006024},{"operation_name":"upsert","last_dispatch_duration_us":9169293,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9273596,"total_server_duration_us":45402,"encode_duration_us":20313,"operation_id":"0x3fb6","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10005887},{"operation_name":"upsert","last_dispatch_duration_us":9634851,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12232163,"total_server_duration_us":240429,"encode_duration_us":19836,"operation_id":"0x3e9e","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10005280},{"operation_name":"upsert","last_dispatch_duration_us":131918,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":131918,"total_server_duration_us":20441,"encode_duration_us":37587,"operation_id":"0x3fb7","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10005030},{"operation_name":"upsert","last_dispatch_duration_us":9207798,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9322627,"total_server_duration_us":9079765,"encode_duration_us":21236,"operation_id":"0x3fa6","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":9348607},{"operation_name":"upsert","last_dispatch_duration_us":9262664,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9262664,"total_server_duration_us":9245337,"encode_duration_us":19194,"operation_id":"0x3f89","timeout_ms":10000,"last_server_duration_us":9245337,"total_duration_us":9266553},{"operation_name":"get","last_dispatch_duration_us":9198224,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9198224,"total_server_duration_us":38690,"operation_id":"0x3fd8","timeout_ms":30000,"last_server_duration_us":38690,"total_duration_us":9203222},{"operation_name":"get","last_dispatch_duration_us":9176009,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9176009,"total_server_duration_us":9802,"operation_id":"0x3ff1","timeout_ms":30000,"last_server_duration_us":9802,"total_duration_us":9178652},{"operation_name":"upsert","last_dispatch_duration_us":9171386,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9171386,"total_server_duration_us":9050106,"encode_duration_us":20182,"operation_id":"0x3fa7","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9177171},{"operation_name":"get","last_dispatch_duration_us":9153646,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9153646,"total_server_duration_us":10272,"operation_id":"0x3ff4","timeout_ms":30000,"last_server_duration_us":10272,"total_duration_us":9157422}],"total_count":61}} WARN [2022-03-13 17:10:37,488] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":45355,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8680955,"total_server_duration_us":235796,"encode_duration_us":21032,"operation_id":"0x3f1c","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10010041},{"operation_name":"upsert","last_dispatch_duration_us":34302,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":943120,"total_server_duration_us":596328,"encode_duration_us":18870,"operation_id":"0x3d00","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10008465},{"operation_name":"upsert","last_dispatch_duration_us":88418,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8908616,"total_server_duration_us":254118,"encode_duration_us":18144,"operation_id":"0x3f22","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10005929},{"operation_name":"upsert","last_dispatch_duration_us":31905,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":1090016,"total_server_duration_us":630439,"encode_duration_us":17012,"operation_id":"0x3e34","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10005636},{"operation_name":"upsert","last_dispatch_duration_us":34071,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":725889,"total_server_duration_us":458090,"encode_duration_us":20031,"operation_id":"0x3f58","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10003400},{"operation_name":"upsert","last_dispatch_duration_us":31763,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":2162981,"total_server_duration_us":380020,"encode_duration_us":18127,"operation_id":"0x4001","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10003206},{"operation_name":"upsert","last_dispatch_duration_us":9136685,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9136685,"total_server_duration_us":9055417,"encode_duration_us":17803,"operation_id":"0x3f23","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":9141955},{"operation_name":"upsert","last_dispatch_duration_us":9066329,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9066329,"total_server_duration_us":9029938,"encode_duration_us":35273,"operation_id":"0x3f1e","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":9073029},{"operation_name":"get","last_dispatch_duration_us":7612776,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7612776,"total_server_duration_us":32,"operation_id":"0x3fab","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":7613802},{"operation_name":"get","last_dispatch_duration_us":5614440,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":5614440,"total_server_duration_us":38,"operation_id":"0x3e84","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":5616646}],"total_count":52}} WARN [2022-03-13 17:10:37,494] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":24666999,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24666999,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9043735,"encode_duration_us":20411,"operation_id":"0x390b","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10001111}],"total_count":1}} INFO [2022-03-13 17:10:37,500] jepsen worker 32 - jepsen.util 1202 :info :write [10 41] :RequestTimeoutException INFO [2022-03-13 17:10:37,501] jepsen worker 32 - jepsen.util 1292 :invoke :read [10 nil] WARN [2022-03-13 17:10:37,589] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9052322,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13577376,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9302364,"encode_duration_us":24383,"operation_id":"0x3b9a","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10002757},{"operation_name":"upsert","last_dispatch_duration_us":9060085,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16514789,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9286624,"encode_duration_us":18408,"operation_id":"0x3a9e","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10004979}],"total_count":2}} INFO [2022-03-13 17:10:37,740] jepsen worker 8 - jepsen.util 638 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,740] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:37,786] jepsen worker 79 - jepsen.util 889 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,787] jepsen worker 79 - jepsen.util 889 :invoke :write [26 29] INFO [2022-03-13 17:10:37,816] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,816] jepsen worker 21 - jepsen.util 741 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,817] jepsen worker 60 - jepsen.util 600 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,817] jepsen worker 21 - jepsen.util 741 :invoke :write [7 36] INFO [2022-03-13 17:10:37,818] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:37,818] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:37,839] jepsen worker 44 - jepsen.util 494 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,839] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,840] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:37,841] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:37,874] jepsen worker 61 - jepsen.util 601 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,874] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,874] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:37,875] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:37,875] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:37,876] jepsen worker 42 - jepsen.util 402 :invoke :write [14 39] INFO [2022-03-13 17:10:37,928] jepsen worker 29 - jepsen.util 1469 :ok :read [9 26] INFO [2022-03-13 17:10:37,929] jepsen worker 29 - jepsen.util 1469 :invoke :read [9 nil] INFO [2022-03-13 17:10:37,934] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,934] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,934] jepsen worker 33 - jepsen.util 663 :invoke :write [11 13] INFO [2022-03-13 17:10:37,935] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:37,937] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,937] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:37,952] jepsen worker 29 - jepsen.util 1469 :ok :read [9 26] INFO [2022-03-13 17:10:37,953] jepsen worker 29 - jepsen.util 1469 :invoke :write [9 26] INFO [2022-03-13 17:10:37,959] jepsen worker 22 - jepsen.util 562 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,960] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:37,993] jepsen worker 21 - jepsen.util 741 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,993] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:37,993] jepsen worker 12 - jepsen.util 732 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,993] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,994] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,994] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:37,994] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:37,995] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:37,996] jepsen worker 21 - jepsen.util 741 :invoke :write [7 44] INFO [2022-03-13 17:10:37,997] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:10:37,998] jepsen worker 51 - jepsen.util 591 :invoke :write [17 23] INFO [2022-03-13 17:10:37,999] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:10:38,071] jepsen worker 70 - jepsen.util 700 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,072] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:38,084] jepsen worker 26 - jepsen.util 476 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,085] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:38,126] jepsen worker 33 - jepsen.util 663 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,126] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,127] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:38,127] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:10:38,170] jepsen worker 63 - jepsen.util 1413 :info :write [21 16] :RequestTimeoutException INFO [2022-03-13 17:10:38,171] jepsen worker 63 - jepsen.util 1503 :invoke :read [21 nil] INFO [2022-03-13 17:10:38,173] jepsen worker 25 - jepsen.util 385 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,173] jepsen worker 25 - jepsen.util 385 :invoke :write [8 41] INFO [2022-03-13 17:10:38,225] jepsen worker 21 - jepsen.util 741 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,226] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:38,235] jepsen worker 71 - jepsen.util 431 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,235] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,235] jepsen worker 71 - jepsen.util 431 :invoke :write [23 0] INFO [2022-03-13 17:10:38,236] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:38,237] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:38,237] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,237] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,239] jepsen worker 6 - jepsen.util 546 :invoke :write [2 42] INFO [2022-03-13 17:10:38,239] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2932551319648093}, :ns_1@172.23.105.187 {:progress 0.3976608187134503}, :ns_1@172.23.107.227 {:progress 0.2932551319648093}, :ns_1@172.23.107.228 {:progress 0.3058823529411765}} INFO [2022-03-13 17:10:38,239] jepsen worker nemesis - couchbase.util Rebalance check count: 150 INFO [2022-03-13 17:10:38,239] jepsen worker nemesis - couchbase.util Rebalance stuck count: 87 INFO [2022-03-13 17:10:38,239] jepsen worker 77 - jepsen.util 707 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,240] jepsen worker 53 - jepsen.util 503 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,240] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:38,240] jepsen worker 44 - jepsen.util 494 :invoke :write [14 1] INFO [2022-03-13 17:10:38,241] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:38,242] jepsen worker 70 - jepsen.util 700 :invoke :write [23 26] INFO [2022-03-13 17:10:38,242] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,242] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:38,242] jepsen worker 77 - jepsen.util 707 :invoke :write [25 35] INFO [2022-03-13 17:10:38,243] jepsen worker 43 - jepsen.util 403 :invoke :write [14 49] INFO [2022-03-13 17:10:38,244] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:38,246] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,247] jepsen worker 34 - jepsen.util 754 :invoke :write [11 47] INFO [2022-03-13 17:10:38,247] jepsen worker 53 - jepsen.util 503 :invoke :write [17 15] INFO [2022-03-13 17:10:38,306] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,307] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,307] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] INFO [2022-03-13 17:10:38,308] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:10:38,319] jepsen worker 35 - jepsen.util 755 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,320] jepsen worker 35 - jepsen.util 755 :invoke :write [11 38] INFO [2022-03-13 17:10:38,345] jepsen worker 17 - jepsen.util 1457 :ok :read [5 43] INFO [2022-03-13 17:10:38,345] jepsen worker 17 - jepsen.util 1457 :invoke :write [5 25] INFO [2022-03-13 17:10:38,377] jepsen worker 86 - jepsen.util 1346 :ok :read [28 44] INFO [2022-03-13 17:10:38,378] jepsen worker 86 - jepsen.util 1346 :invoke :read [28 nil] INFO [2022-03-13 17:10:38,382] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:38,382] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,383] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:10:38,384] jepsen worker 21 - jepsen.util 741 :invoke :write [7 10] INFO [2022-03-13 17:10:38,403] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,403] jepsen worker 62 - jepsen.util 512 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,407] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:38,407] jepsen worker 12 - jepsen.util 732 :invoke :write [4 45] INFO [2022-03-13 17:10:38,407] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,408] jepsen worker 61 - jepsen.util 601 :invoke :write [20 22] INFO [2022-03-13 17:10:38,408] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,409] jepsen worker 14 - jepsen.util 284 :invoke :write [4 26] INFO [2022-03-13 17:10:38,746] jepsen worker 20 - jepsen.util 1730 :info :write [6 29] :RequestTimeoutException INFO [2022-03-13 17:10:38,747] jepsen worker 20 - jepsen.util 1820 :invoke :read [6 nil] INFO [2022-03-13 17:10:38,786] jepsen worker 13 - jepsen.util 553 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,786] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:38,800] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,800] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:10:38,820] jepsen worker 71 - jepsen.util 431 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,820] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:38,869] jepsen worker 76 - jepsen.util 616 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,869] jepsen worker 76 - jepsen.util 616 :invoke :write [25 9] INFO [2022-03-13 17:10:38,890] jepsen worker 51 - jepsen.util 591 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,890] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,891] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:38,891] jepsen worker 51 - jepsen.util 591 :invoke :write [17 33] INFO [2022-03-13 17:10:38,892] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:38,893] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:10:38,895] jepsen worker 52 - jepsen.util 502 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,896] jepsen worker 52 - jepsen.util 502 :invoke :write [17 22] INFO [2022-03-13 17:10:38,917] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,918] jepsen worker 6 - jepsen.util 546 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,918] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:38,918] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:38,931] jepsen worker 9 - jepsen.util 1449 :info :write [3 3] :RequestTimeoutException INFO [2022-03-13 17:10:38,932] jepsen worker 9 - jepsen.util 1539 :invoke :read [3 nil] INFO [2022-03-13 17:10:38,974] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:38,974] jepsen worker 25 - jepsen.util 385 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,974] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,975] jepsen worker 22 - jepsen.util 562 :invoke :write [7 25] INFO [2022-03-13 17:10:38,976] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:38,976] jepsen worker 42 - jepsen.util 402 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,976] jepsen worker 8 - jepsen.util 638 :invoke :write [2 22] INFO [2022-03-13 17:10:38,976] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:38,977] jepsen worker 26 - jepsen.util 476 :invoke :write [8 21] INFO [2022-03-13 17:10:38,978] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:39,028] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,029] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:10:39,056] jepsen worker 70 - jepsen.util 700 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,057] jepsen worker 70 - jepsen.util 700 :invoke :write [23 41] INFO [2022-03-13 17:10:39,086] jepsen worker 79 - jepsen.util 889 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,087] jepsen worker 79 - jepsen.util 889 :invoke :write [26 5] INFO [2022-03-13 17:10:39,098] jepsen worker 77 - jepsen.util 707 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,099] jepsen worker 77 - jepsen.util 707 :invoke :write [25 17] INFO [2022-03-13 17:10:39,111] jepsen worker 21 - jepsen.util 741 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,112] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:39,150] jepsen worker 53 - jepsen.util 503 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,150] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,151] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:39,151] jepsen worker 62 - jepsen.util 512 :invoke :write [20 44] INFO [2022-03-13 17:10:39,202] jepsen worker 76 - jepsen.util 616 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,203] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:39,216] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,216] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,217] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:39,217] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:10:39,245] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2932551319648093}, :ns_1@172.23.105.187 {:progress 0.3976608187134503}, :ns_1@172.23.107.227 {:progress 0.2932551319648093}, :ns_1@172.23.107.228 {:progress 0.3058823529411765}} INFO [2022-03-13 17:10:39,245] jepsen worker nemesis - couchbase.util Rebalance check count: 151 INFO [2022-03-13 17:10:39,245] jepsen worker nemesis - couchbase.util Rebalance stuck count: 88 INFO [2022-03-13 17:10:39,278] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,278] jepsen worker 35 - jepsen.util 755 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,278] jepsen worker 44 - jepsen.util 494 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,279] jepsen worker 44 - jepsen.util 494 :invoke :write [14 19] INFO [2022-03-13 17:10:39,280] jepsen worker 78 - jepsen.util 438 :invoke :write [26 49] INFO [2022-03-13 17:10:39,280] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:39,294] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,294] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:39,294] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,294] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:39,295] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:10:39,296] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:39,313] jepsen worker 79 - jepsen.util 889 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,313] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,314] jepsen worker 76 - jepsen.util 616 :invoke :write [25 32] INFO [2022-03-13 17:10:39,314] jepsen worker 79 - jepsen.util 889 :invoke :write [26 1] INFO [2022-03-13 17:10:39,330] jepsen worker 61 - jepsen.util 601 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,330] jepsen worker 43 - jepsen.util 403 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,331] jepsen worker 43 - jepsen.util 403 :invoke :write [14 35] INFO [2022-03-13 17:10:39,331] jepsen worker 61 - jepsen.util 601 :invoke :write [20 6] INFO [2022-03-13 17:10:39,395] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,395] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,395] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:39,395] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:10:39,396] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:39,397] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:39,408] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,408] jepsen worker 7 - jepsen.util 457 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,409] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:39,409] jepsen worker 7 - jepsen.util 457 :invoke :write [2 5] INFO [2022-03-13 17:10:39,468] jepsen worker 79 - jepsen.util 889 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,469] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,469] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,469] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:39,469] jepsen worker 79 - jepsen.util 889 :invoke :write [26 42] INFO [2022-03-13 17:10:39,470] jepsen worker 21 - jepsen.util 741 :invoke :write [7 46] INFO [2022-03-13 17:10:39,470] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:10:39,470] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:10:39,480] jepsen worker 34 - jepsen.util 754 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,480] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,480] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,481] jepsen worker 34 - jepsen.util 754 :invoke :write [11 20] INFO [2022-03-13 17:10:39,481] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:39,482] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:39,511] jepsen worker 12 - jepsen.util 732 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,511] jepsen worker 14 - jepsen.util 284 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,512] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:39,512] jepsen worker 12 - jepsen.util 732 :invoke :write [4 19] INFO [2022-03-13 17:10:39,512] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:39,513] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:39,514] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,514] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:39,601] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,602] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:10:39,607] jepsen worker 76 - jepsen.util 616 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,608] jepsen worker 76 - jepsen.util 616 :invoke :write [25 33] INFO [2022-03-13 17:10:39,629] jepsen worker 52 - jepsen.util 502 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,629] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,629] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:39,630] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:39,653] jepsen worker 51 - jepsen.util 591 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,653] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:39,698] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:39,699] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:39,950] jepsen worker 59 - jepsen.util 1229 :info :write [19 26] :RequestTimeoutException INFO [2022-03-13 17:10:39,950] jepsen worker 59 - jepsen.util 1319 :invoke :write [19 39] INFO [2022-03-13 17:10:39,990] jepsen worker 57 - jepsen.util 1497 :info :write [19 38] :RequestTimeoutException INFO [2022-03-13 17:10:39,990] jepsen worker 57 - jepsen.util 1587 :invoke :write [19 22] INFO [2022-03-13 17:10:40,006] jepsen worker 47 - jepsen.util 1487 :info :write [15 19] :RequestTimeoutException INFO [2022-03-13 17:10:40,007] jepsen worker 47 - jepsen.util 1577 :invoke :write [15 5] INFO [2022-03-13 17:10:40,014] jepsen worker 26 - jepsen.util 476 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,014] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,015] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:40,015] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:40,019] jepsen worker 79 - jepsen.util 889 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,020] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:10:40,028] jepsen worker 70 - jepsen.util 700 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,028] jepsen worker 70 - jepsen.util 700 :invoke :write [23 7] INFO [2022-03-13 17:10:40,059] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,059] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:40,143] jepsen worker 76 - jepsen.util 616 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,144] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:40,152] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,153] jepsen worker 23 - jepsen.util 563 :invoke :write [7 47] INFO [2022-03-13 17:10:40,161] jepsen worker 22 - jepsen.util 562 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,162] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:40,171] jepsen worker 21 - jepsen.util 741 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,172] jepsen worker 21 - jepsen.util 741 :invoke :write [7 16] INFO [2022-03-13 17:10:40,183] jepsen worker 87 - jepsen.util 1887 :info :write [29 41] :SyncWriteAmbiguous INFO [2022-03-13 17:10:40,184] jepsen worker 87 - jepsen.util 1977 :invoke :write [29 39] INFO [2022-03-13 17:10:40,222] jepsen worker 62 - jepsen.util 512 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,222] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:40,239] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,239] jepsen worker 44 - jepsen.util 494 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,239] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,240] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:40,241] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:40,241] jepsen worker 71 - jepsen.util 431 :invoke :write [23 2] INFO [2022-03-13 17:10:40,247] jepsen worker 4 - jepsen.util 1624 :ok :read [1 42] INFO [2022-03-13 17:10:40,248] jepsen worker 4 - jepsen.util 1624 :invoke :read [1 nil] INFO [2022-03-13 17:10:40,252] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2961876832844574}, :ns_1@172.23.105.187 {:progress 0.4093567251461988}, :ns_1@172.23.107.227 {:progress 0.2961876832844574}, :ns_1@172.23.107.228 {:progress 0.3117647058823529}} INFO [2022-03-13 17:10:40,252] jepsen worker nemesis - couchbase.util Rebalance check count: 152 INFO [2022-03-13 17:10:40,252] jepsen worker nemesis - couchbase.util Rebalance stuck count: 88 INFO [2022-03-13 17:10:40,259] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,260] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,261] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:10:40,261] jepsen worker 79 - jepsen.util 889 :invoke :write [26 34] INFO [2022-03-13 17:10:40,304] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,305] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,305] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,306] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:40,307] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:40,307] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:10:40,325] jepsen worker 77 - jepsen.util 707 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,325] jepsen worker 77 - jepsen.util 707 :invoke :write [25 23] INFO [2022-03-13 17:10:40,347] jepsen worker 8 - jepsen.util 638 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,347] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,348] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:40,348] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:40,365] jepsen worker 56 - jepsen.util 1316 :info :write [18 10] :SyncWriteAmbiguous INFO [2022-03-13 17:10:40,366] jepsen worker 56 - jepsen.util 1406 :invoke :write [18 12] INFO [2022-03-13 17:10:40,390] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,391] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:40,391] jepsen worker 34 - jepsen.util 754 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,391] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:40,393] jepsen worker 61 - jepsen.util 601 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,393] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,393] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,393] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,393] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:40,394] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,394] jepsen worker 26 - jepsen.util 476 :invoke :write [8 37] INFO [2022-03-13 17:10:40,395] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:40,395] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:40,396] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:40,396] jepsen worker 14 - jepsen.util 284 :invoke :write [4 30] INFO [2022-03-13 17:10:40,397] jepsen worker 27 - jepsen.util 1647 :ok :read [9 26] INFO [2022-03-13 17:10:40,397] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,397] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:40,397] jepsen worker 27 - jepsen.util 1647 :invoke :read [9 nil] INFO [2022-03-13 17:10:40,398] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,398] jepsen worker 53 - jepsen.util 503 :invoke :write [17 12] INFO [2022-03-13 17:10:40,399] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:40,399] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:10:40,400] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:40,400] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:10:40,406] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,406] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,406] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:10:40,407] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:40,410] jepsen worker 61 - jepsen.util 601 :invoke :write [20 12] INFO [2022-03-13 17:10:40,410] jepsen worker 21 - jepsen.util 741 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,411] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:40,422] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,423] jepsen worker 76 - jepsen.util 616 :invoke :write [25 24] INFO [2022-03-13 17:10:40,432] jepsen worker 63 - jepsen.util 1503 :ok :read [21 :nil] INFO [2022-03-13 17:10:40,433] jepsen worker 63 - jepsen.util 1503 :invoke :read [21 nil] INFO [2022-03-13 17:10:40,456] jepsen worker 74 - jepsen.util 1784 :ok :read [24 24] INFO [2022-03-13 17:10:40,456] jepsen worker 74 - jepsen.util 1784 :invoke :read [24 nil] INFO [2022-03-13 17:10:40,461] jepsen worker 27 - jepsen.util 1647 :ok :read [9 26] INFO [2022-03-13 17:10:40,461] jepsen worker 27 - jepsen.util 1647 :invoke :read [9 nil] INFO [2022-03-13 17:10:40,489] jepsen worker 63 - jepsen.util 1503 :ok :read [21 :nil] INFO [2022-03-13 17:10:40,490] jepsen worker 63 - jepsen.util 1503 :invoke :read [21 nil] INFO [2022-03-13 17:10:40,491] jepsen worker 79 - jepsen.util 889 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,492] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:10:40,499] jepsen worker 74 - jepsen.util 1784 :ok :read [24 24] INFO [2022-03-13 17:10:40,500] jepsen worker 74 - jepsen.util 1784 :invoke :write [24 5] INFO [2022-03-13 17:10:40,503] jepsen worker 27 - jepsen.util 1647 :ok :read [9 26] INFO [2022-03-13 17:10:40,504] jepsen worker 27 - jepsen.util 1647 :invoke :read [9 nil] INFO [2022-03-13 17:10:40,514] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,514] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,515] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:40,516] jepsen worker 33 - jepsen.util 663 :invoke :write [11 19] INFO [2022-03-13 17:10:40,517] jepsen worker 21 - jepsen.util 741 :invoke :write [7 19] INFO [2022-03-13 17:10:40,517] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:10:40,528] jepsen worker 78 - jepsen.util 438 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,528] jepsen worker 78 - jepsen.util 438 :invoke :write [26 24] INFO [2022-03-13 17:10:40,535] jepsen worker 63 - jepsen.util 1503 :ok :read [21 :nil] INFO [2022-03-13 17:10:40,536] jepsen worker 63 - jepsen.util 1503 :invoke :read [21 nil] INFO [2022-03-13 17:10:40,541] jepsen worker 43 - jepsen.util 403 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,542] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,542] jepsen worker 43 - jepsen.util 403 :invoke :write [14 6] INFO [2022-03-13 17:10:40,542] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:40,557] jepsen worker 27 - jepsen.util 1647 :ok :read [9 26] INFO [2022-03-13 17:10:40,557] jepsen worker 27 - jepsen.util 1647 :invoke :read [9 nil] INFO [2022-03-13 17:10:40,596] jepsen worker 63 - jepsen.util 1503 :ok :read [21 :nil] INFO [2022-03-13 17:10:40,596] jepsen worker 63 - jepsen.util 1503 :invoke :read [21 nil] INFO [2022-03-13 17:10:40,610] jepsen worker 27 - jepsen.util 1647 :ok :read [9 26] INFO [2022-03-13 17:10:40,610] jepsen worker 27 - jepsen.util 1647 :invoke :write [9 36] INFO [2022-03-13 17:10:40,630] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:40,630] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,630] jepsen worker 70 - jepsen.util 700 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,632] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:40,632] jepsen worker 76 - jepsen.util 616 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,633] jepsen worker 70 - jepsen.util 700 :invoke :write [23 9] INFO [2022-03-13 17:10:40,633] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:40,634] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:40,636] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,636] jepsen worker 69 - jepsen.util 609 :invoke :write [23 4] INFO [2022-03-13 17:10:40,650] jepsen worker 63 - jepsen.util 1503 :ok :read [21 :nil] INFO [2022-03-13 17:10:40,651] jepsen worker 63 - jepsen.util 1503 :invoke :read [21 nil] INFO [2022-03-13 17:10:40,694] jepsen worker 5 - jepsen.util 1625 :info :write [1 3] :RequestTimeoutException INFO [2022-03-13 17:10:40,694] jepsen worker 5 - jepsen.util 1715 :invoke :write [1 26] INFO [2022-03-13 17:10:40,704] jepsen worker 26 - jepsen.util 476 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,704] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:40,726] jepsen worker 63 - jepsen.util 1503 :ok :read [21 :nil] INFO [2022-03-13 17:10:40,727] jepsen worker 63 - jepsen.util 1503 :invoke :write [21 40] INFO [2022-03-13 17:10:40,752] jepsen worker 33 - jepsen.util 663 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,752] jepsen worker 33 - jepsen.util 663 :invoke :write [11 0] INFO [2022-03-13 17:10:40,759] jepsen worker 7 - jepsen.util 457 :fail :write [2 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,759] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:40,763] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,763] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,764] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:40,765] jepsen worker 76 - jepsen.util 616 :invoke :write [25 24] INFO [2022-03-13 17:10:40,765] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,766] jepsen worker 79 - jepsen.util 889 :invoke :write [26 1] INFO [2022-03-13 17:10:40,780] jepsen worker 12 - jepsen.util 732 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,780] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,781] jepsen worker 12 - jepsen.util 732 :invoke :write [4 23] INFO [2022-03-13 17:10:40,782] jepsen worker 51 - jepsen.util 591 :invoke :write [17 11] INFO [2022-03-13 17:10:40,794] jepsen worker 21 - jepsen.util 741 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,795] jepsen worker 21 - jepsen.util 741 :invoke :write [7 44] INFO [2022-03-13 17:10:40,838] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,839] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:10:40,880] jepsen worker 79 - jepsen.util 889 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,881] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:40,894] jepsen worker 71 - jepsen.util 431 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,895] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,895] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,895] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:40,895] jepsen worker 71 - jepsen.util 431 :invoke :write [23 22] INFO [2022-03-13 17:10:40,896] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:40,897] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:40,898] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:40,900] jepsen worker 0 - jepsen.util 1440 :info :write [0 5] :RequestTimeoutException INFO [2022-03-13 17:10:40,901] jepsen worker 0 - jepsen.util 1530 :invoke :read [0 nil] INFO [2022-03-13 17:10:40,931] jepsen worker 53 - jepsen.util 503 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,931] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:40,943] jepsen worker 69 - jepsen.util 609 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:40,944] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:10:41,008] jepsen worker 23 - jepsen.util 563 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,009] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:41,010] jepsen worker 76 - jepsen.util 616 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,010] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,011] jepsen worker 79 - jepsen.util 889 :invoke :write [26 31] INFO [2022-03-13 17:10:41,011] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:41,019] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,020] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:41,020] jepsen worker 80 - jepsen.util 620 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,021] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,021] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:41,021] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,021] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,021] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,021] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:41,022] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:41,023] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:41,023] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:41,024] jepsen worker 8 - jepsen.util 638 :invoke :write [2 25] INFO [2022-03-13 17:10:41,024] jepsen worker 76 - jepsen.util 616 :invoke :write [25 13] INFO [2022-03-13 17:10:41,071] jepsen worker 70 - jepsen.util 700 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,071] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,072] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:41,072] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,072] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:41,073] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:41,073] jepsen worker 34 - jepsen.util 754 :invoke :write [11 48] INFO [2022-03-13 17:10:41,075] jepsen worker 6 - jepsen.util 546 :invoke :write [2 23] INFO [2022-03-13 17:10:41,091] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,091] jepsen worker 33 - jepsen.util 663 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,093] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:41,093] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:10:41,136] jepsen worker 79 - jepsen.util 889 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,136] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:41,143] jepsen worker 77 - jepsen.util 707 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,144] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:41,147] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,148] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:41,150] jepsen worker 78 - jepsen.util 438 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,151] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,151] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,151] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,151] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,151] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,152] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:41,153] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:41,153] jepsen worker 52 - jepsen.util 502 :invoke :write [17 1] INFO [2022-03-13 17:10:41,154] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:41,155] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:41,155] jepsen worker 35 - jepsen.util 755 :invoke :write [11 10] INFO [2022-03-13 17:10:41,207] jepsen worker 21 - jepsen.util 741 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,207] jepsen worker 21 - jepsen.util 741 :invoke :write [7 45] INFO [2022-03-13 17:10:41,227] jepsen worker 14 - jepsen.util 284 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,228] jepsen worker 14 - jepsen.util 284 :invoke :write [4 42] INFO [2022-03-13 17:10:41,262] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2961876832844574}, :ns_1@172.23.105.187 {:progress 0.4093567251461988}, :ns_1@172.23.107.227 {:progress 0.2961876832844574}, :ns_1@172.23.107.228 {:progress 0.3117647058823529}} INFO [2022-03-13 17:10:41,262] jepsen worker nemesis - couchbase.util Rebalance check count: 153 INFO [2022-03-13 17:10:41,262] jepsen worker nemesis - couchbase.util Rebalance stuck count: 89 INFO [2022-03-13 17:10:41,272] jepsen worker 76 - jepsen.util 616 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,272] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,272] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:41,273] jepsen worker 76 - jepsen.util 616 :invoke :write [25 19] INFO [2022-03-13 17:10:41,363] jepsen worker 81 - jepsen.util 1701 :info :write [27 4] :SyncWriteAmbiguous INFO [2022-03-13 17:10:41,364] jepsen worker 81 - jepsen.util 1791 :invoke :write [27 48] INFO [2022-03-13 17:10:41,377] jepsen worker 89 - jepsen.util 1709 :info :write [29 20] :RequestTimeoutException INFO [2022-03-13 17:10:41,378] jepsen worker 89 - jepsen.util 1799 :invoke :write [29 38] INFO [2022-03-13 17:10:41,716] jepsen worker 49 - jepsen.util 1219 :info :write [16 39] :RequestTimeoutException INFO [2022-03-13 17:10:41,717] jepsen worker 49 - jepsen.util 1309 :invoke :read [16 nil] INFO [2022-03-13 17:10:41,781] jepsen worker 82 - jepsen.util 1342 :ok :read [27 4] INFO [2022-03-13 17:10:41,781] jepsen worker 82 - jepsen.util 1342 :invoke :write [27 32] INFO [2022-03-13 17:10:41,786] jepsen worker 48 - jepsen.util 1308 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:10:41,791] jepsen worker 48 - jepsen.util 1398 :invoke :write [16 6] INFO [2022-03-13 17:10:41,801] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,802] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:41,815] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,815] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:41,815] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,815] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:41,816] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,816] jepsen worker 13 - jepsen.util 553 :invoke :write [4 35] INFO [2022-03-13 17:10:41,817] jepsen worker 70 - jepsen.util 700 :invoke :write [23 45] INFO [2022-03-13 17:10:41,818] jepsen worker 44 - jepsen.util 494 :invoke :write [14 41] INFO [2022-03-13 17:10:41,818] jepsen worker 71 - jepsen.util 431 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,818] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:41,818] jepsen worker 22 - jepsen.util 562 :invoke :write [7 11] INFO [2022-03-13 17:10:41,823] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:41,849] jepsen worker 76 - jepsen.util 616 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,850] jepsen worker 76 - jepsen.util 616 :invoke :write [25 45] INFO [2022-03-13 17:10:41,854] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,855] jepsen worker 79 - jepsen.util 889 :invoke :write [26 25] INFO [2022-03-13 17:10:41,865] jepsen worker 43 - jepsen.util 403 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,866] jepsen worker 43 - jepsen.util 403 :invoke :write [14 47] INFO [2022-03-13 17:10:41,885] jepsen worker 31 - jepsen.util 1381 :ok :read [10 :nil] INFO [2022-03-13 17:10:41,885] jepsen worker 31 - jepsen.util 1381 :invoke :read [10 nil] INFO [2022-03-13 17:10:41,902] jepsen worker 31 - jepsen.util 1381 :ok :read [10 :nil] INFO [2022-03-13 17:10:41,904] jepsen worker 31 - jepsen.util 1381 :invoke :read [10 nil] INFO [2022-03-13 17:10:41,913] jepsen worker 10 - jepsen.util 1360 :ok :read [3 25] INFO [2022-03-13 17:10:41,914] jepsen worker 10 - jepsen.util 1360 :invoke :write [3 7] INFO [2022-03-13 17:10:41,924] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,924] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:41,929] jepsen worker 31 - jepsen.util 1381 :ok :read [10 :nil] INFO [2022-03-13 17:10:41,930] jepsen worker 31 - jepsen.util 1381 :invoke :read [10 nil] INFO [2022-03-13 17:10:41,931] jepsen worker 31 - jepsen.util 1381 :ok :read [10 :nil] INFO [2022-03-13 17:10:41,931] jepsen worker 31 - jepsen.util 1381 :invoke :read [10 nil] INFO [2022-03-13 17:10:41,956] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,956] jepsen worker 8 - jepsen.util 638 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,956] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,956] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,956] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,956] jepsen worker 7 - jepsen.util 457 :invoke :write [2 7] INFO [2022-03-13 17:10:41,957] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:41,958] jepsen worker 78 - jepsen.util 438 :invoke :write [26 8] INFO [2022-03-13 17:10:41,958] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:41,959] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:41,971] jepsen worker 31 - jepsen.util 1381 :ok :read [10 :nil] INFO [2022-03-13 17:10:41,972] jepsen worker 31 - jepsen.util 1381 :invoke :write [10 28] INFO [2022-03-13 17:10:41,985] jepsen worker 76 - jepsen.util 616 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:41,985] jepsen worker 76 - jepsen.util 616 :invoke :write [25 16] INFO [2022-03-13 17:10:41,996] jepsen worker 40 - jepsen.util 1390 :info :write [13 40] :RequestTimeoutException INFO [2022-03-13 17:10:41,997] jepsen worker 40 - jepsen.util 1480 :invoke :read [13 nil] INFO [2022-03-13 17:10:42,016] jepsen worker 61 - jepsen.util 601 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,016] jepsen worker 61 - jepsen.util 601 :invoke :write [20 14] INFO [2022-03-13 17:10:42,023] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,023] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:10:42,031] jepsen worker 34 - jepsen.util 754 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,033] jepsen worker 34 - jepsen.util 754 :invoke :write [11 27] INFO [2022-03-13 17:10:42,066] jepsen worker 41 - jepsen.util 1211 :info :write [13 26] :RequestTimeoutException INFO [2022-03-13 17:10:42,067] jepsen worker 41 - jepsen.util 1301 :invoke :read [13 nil] INFO [2022-03-13 17:10:42,096] jepsen worker 12 - jepsen.util 732 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,097] jepsen worker 12 - jepsen.util 732 :invoke :write [4 1] INFO [2022-03-13 17:10:42,136] jepsen worker 79 - jepsen.util 889 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,137] jepsen worker 79 - jepsen.util 889 :invoke :write [26 45] INFO [2022-03-13 17:10:42,159] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,160] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:42,171] jepsen worker 52 - jepsen.util 502 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,172] jepsen worker 52 - jepsen.util 502 :invoke :write [17 39] INFO [2022-03-13 17:10:42,237] jepsen worker 76 - jepsen.util 616 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,238] jepsen worker 76 - jepsen.util 616 :invoke :write [25 35] INFO [2022-03-13 17:10:42,264] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,264] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,282] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:42,264] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,282] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:42,265] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,282] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:42,268] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2991202346041055}, :ns_1@172.23.105.187 {:progress 0.4093567251461988}, :ns_1@172.23.107.227 {:progress 0.2991202346041055}, :ns_1@172.23.107.228 {:progress 0.3176470588235294}} INFO [2022-03-13 17:10:42,282] jepsen worker 62 - jepsen.util 512 :invoke :write [20 49] INFO [2022-03-13 17:10:42,282] jepsen worker nemesis - couchbase.util Rebalance check count: 154 INFO [2022-03-13 17:10:42,283] jepsen worker nemesis - couchbase.util Rebalance stuck count: 89 INFO [2022-03-13 17:10:42,292] jepsen worker 6 - jepsen.util 546 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,292] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,292] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:42,293] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:42,323] jepsen worker 79 - jepsen.util 889 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,324] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:42,327] jepsen worker 21 - jepsen.util 741 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,327] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,327] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:42,328] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:10:42,392] jepsen worker 20 - jepsen.util 1820 :ok :read [6 16] INFO [2022-03-13 17:10:42,393] jepsen worker 20 - jepsen.util 1820 :invoke :read [6 nil] INFO [2022-03-13 17:10:42,395] jepsen worker 51 - jepsen.util 591 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,395] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:42,395] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,395] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,396] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:10:42,397] jepsen worker 51 - jepsen.util 591 :invoke :write [17 27] INFO [2022-03-13 17:10:42,399] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:10:42,406] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:42,463] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,463] jepsen worker 76 - jepsen.util 616 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,464] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:42,464] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:42,465] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,465] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,465] jepsen worker 76 - jepsen.util 616 :invoke :write [25 7] INFO [2022-03-13 17:10:42,466] jepsen worker 79 - jepsen.util 889 :invoke :write [26 14] INFO [2022-03-13 17:10:42,473] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,473] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:42,474] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,474] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:10:42,474] jepsen worker 21 - jepsen.util 741 :invoke :write [7 2] INFO [2022-03-13 17:10:42,484] jepsen worker 35 - jepsen.util 755 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,484] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,485] jepsen worker 80 - jepsen.util 620 :invoke :write [26 24] INFO [2022-03-13 17:10:42,485] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:42,487] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:42,518] jepsen worker 86 - jepsen.util 1346 :ok :read [28 44] INFO [2022-03-13 17:10:42,519] jepsen worker 86 - jepsen.util 1346 :invoke :read [28 nil] INFO [2022-03-13 17:10:42,542] jepsen worker 70 - jepsen.util 700 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,543] jepsen worker 14 - jepsen.util 284 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,543] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:42,543] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:42,587] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,588] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,589] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:10:42,589] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:10:42,606] jepsen worker 76 - jepsen.util 616 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,607] jepsen worker 76 - jepsen.util 616 :invoke :write [25 5] INFO [2022-03-13 17:10:42,618] jepsen worker 43 - jepsen.util 403 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,618] jepsen worker 43 - jepsen.util 403 :invoke :write [14 44] INFO [2022-03-13 17:10:42,684] jepsen worker 13 - jepsen.util 553 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,685] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:42,700] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,701] jepsen worker 69 - jepsen.util 609 :invoke :write [23 5] INFO [2022-03-13 17:10:42,711] jepsen worker 79 - jepsen.util 889 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,712] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:42,761] jepsen worker 61 - jepsen.util 601 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,761] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:42,761] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,761] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,761] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:42,762] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:42,763] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:42,763] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:42,780] jepsen worker 44 - jepsen.util 494 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:42,780] jepsen worker 44 - jepsen.util 494 :invoke :write [14 20] INFO [2022-03-13 17:10:42,800] jepsen worker 88 - jepsen.util 1348 :info :write [29 38] :RequestTimeoutException INFO [2022-03-13 17:10:42,801] jepsen worker 88 - jepsen.util 1438 :invoke :read [29 nil] INFO [2022-03-13 17:10:42,826] jepsen worker 88 - jepsen.util 1438 :ok :read [29 10] INFO [2022-03-13 17:10:42,827] jepsen worker 88 - jepsen.util 1438 :invoke :read [29 nil] INFO [2022-03-13 17:10:42,850] jepsen worker 88 - jepsen.util 1438 :ok :read [29 10] INFO [2022-03-13 17:10:42,851] jepsen worker 88 - jepsen.util 1438 :invoke :write [29 36] INFO [2022-03-13 17:10:42,924] jepsen worker 15 - jepsen.util 1455 :info :write [5 29] :SyncWriteAmbiguous INFO [2022-03-13 17:10:42,925] jepsen worker 15 - jepsen.util 1545 :invoke :write [5 6] INFO [2022-03-13 17:10:43,004] jepsen worker 21 - jepsen.util 741 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,005] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:43,012] jepsen worker 12 - jepsen.util 732 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,012] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,012] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:10:43,013] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:43,014] jepsen worker 76 - jepsen.util 616 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,014] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,015] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:43,016] jepsen worker 79 - jepsen.util 889 :invoke :write [26 38] INFO [2022-03-13 17:10:43,018] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,019] jepsen worker 76 - jepsen.util 616 :invoke :write [25 44] INFO [2022-03-13 17:10:43,034] jepsen worker 84 - jepsen.util 1704 :info :write [28 16] :RequestTimeoutException INFO [2022-03-13 17:10:43,036] jepsen worker 84 - jepsen.util 1794 :invoke :read [28 nil] INFO [2022-03-13 17:10:43,037] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,037] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:10:43,073] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,074] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:43,103] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:43,103] jepsen worker 69 - jepsen.util 609 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,104] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:43,104] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:43,148] jepsen worker 1 - jepsen.util 1261 :ok :read [0 41] INFO [2022-03-13 17:10:43,149] jepsen worker 1 - jepsen.util 1261 :invoke :write [0 44] INFO [2022-03-13 17:10:43,156] jepsen worker 22 - jepsen.util 562 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,156] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,156] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,156] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,157] jepsen worker 24 - jepsen.util 474 :invoke :write [8 43] INFO [2022-03-13 17:10:43,157] jepsen worker 8 - jepsen.util 638 :invoke :write [2 8] INFO [2022-03-13 17:10:43,160] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:43,166] jepsen worker 76 - jepsen.util 616 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,166] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:43,169] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,169] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,169] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:43,170] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:43,170] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:10:43,170] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,170] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:10:43,180] jepsen worker 62 - jepsen.util 512 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,181] jepsen worker 33 - jepsen.util 663 :invoke :write [11 44] INFO [2022-03-13 17:10:43,182] jepsen worker 62 - jepsen.util 512 :invoke :write [20 45] INFO [2022-03-13 17:10:43,182] jepsen worker 21 - jepsen.util 741 :invoke :write [7 26] INFO [2022-03-13 17:10:43,183] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,183] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:43,184] jepsen worker 42 - jepsen.util 402 :invoke :write [14 28] INFO [2022-03-13 17:10:43,184] jepsen worker 26 - jepsen.util 476 :invoke :write [8 27] INFO [2022-03-13 17:10:43,190] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,191] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,191] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,191] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:43,192] jepsen worker 53 - jepsen.util 503 :invoke :write [17 21] INFO [2022-03-13 17:10:43,202] jepsen worker 61 - jepsen.util 601 :invoke :write [20 46] INFO [2022-03-13 17:10:43,219] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,219] jepsen worker 34 - jepsen.util 754 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,219] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,219] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,223] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:43,224] jepsen worker 6 - jepsen.util 546 :invoke :write [2 23] INFO [2022-03-13 17:10:43,226] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:43,227] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:43,276] jepsen worker 51 - jepsen.util 591 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,277] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,277] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:43,278] jepsen worker 14 - jepsen.util 284 :invoke :write [4 28] INFO [2022-03-13 17:10:43,288] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.2991202346041055}, :ns_1@172.23.105.187 {:progress 0.4093567251461988}, :ns_1@172.23.107.227 {:progress 0.2991202346041055}, :ns_1@172.23.107.228 {:progress 0.3176470588235294}} INFO [2022-03-13 17:10:43,288] jepsen worker nemesis - couchbase.util Rebalance check count: 155 INFO [2022-03-13 17:10:43,288] jepsen worker nemesis - couchbase.util Rebalance stuck count: 90 INFO [2022-03-13 17:10:43,289] jepsen worker 79 - jepsen.util 889 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,289] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,289] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:43,290] jepsen worker 76 - jepsen.util 616 :invoke :write [25 43] INFO [2022-03-13 17:10:43,291] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,292] jepsen worker 79 - jepsen.util 889 :invoke :write [26 25] INFO [2022-03-13 17:10:43,303] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,304] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:43,365] jepsen worker 78 - jepsen.util 438 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,367] jepsen worker 78 - jepsen.util 438 :invoke :write [26 13] INFO [2022-03-13 17:10:43,397] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,398] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:43,400] jepsen worker 33 - jepsen.util 663 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,400] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:43,403] jepsen worker 76 - jepsen.util 616 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,404] jepsen worker 76 - jepsen.util 616 :invoke :write [25 45] INFO [2022-03-13 17:10:43,413] jepsen worker 50 - jepsen.util 1490 :info :write [16 28] :SyncWriteAmbiguous INFO [2022-03-13 17:10:43,418] jepsen worker 50 - jepsen.util 1580 :invoke :read [16 nil] INFO [2022-03-13 17:10:43,443] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,443] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,443] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,443] jepsen worker 44 - jepsen.util 494 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,444] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:43,445] jepsen worker 44 - jepsen.util 494 :invoke :write [14 23] INFO [2022-03-13 17:10:43,446] jepsen worker 70 - jepsen.util 700 :invoke :write [23 2] INFO [2022-03-13 17:10:43,446] jepsen worker 71 - jepsen.util 431 :invoke :write [23 49] INFO [2022-03-13 17:10:43,461] jepsen worker 37 - jepsen.util 1477 :info :write [12 38] :RequestTimeoutException INFO [2022-03-13 17:10:43,462] jepsen worker 37 - jepsen.util 1567 :invoke :read [12 nil] INFO [2022-03-13 17:10:43,488] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,488] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:43,492] jepsen worker 62 - jepsen.util 512 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,492] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:43,505] jepsen worker 79 - jepsen.util 889 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,506] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:43,507] jepsen worker 39 - jepsen.util 1569 :ok :read [13 39] INFO [2022-03-13 17:10:43,508] jepsen worker 39 - jepsen.util 1569 :invoke :read [13 nil] INFO [2022-03-13 17:10:43,536] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,537] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,536] jepsen worker 7 - jepsen.util 457 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,537] jepsen worker 7 - jepsen.util 457 :invoke :write [2 40] INFO [2022-03-13 17:10:43,537] jepsen worker 35 - jepsen.util 755 :invoke :write [11 24] INFO [2022-03-13 17:10:43,541] jepsen worker 13 - jepsen.util 553 :invoke :write [4 33] INFO [2022-03-13 17:10:43,575] jepsen worker 76 - jepsen.util 616 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,575] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,576] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:43,576] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] INFO [2022-03-13 17:10:43,577] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,578] jepsen worker 21 - jepsen.util 741 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,578] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,578] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,578] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,588] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,590] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:43,592] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:43,594] jepsen worker 60 - jepsen.util 600 :invoke :write [20 14] INFO [2022-03-13 17:10:43,595] jepsen worker 69 - jepsen.util 609 :invoke :write [23 0] INFO [2022-03-13 17:10:43,596] jepsen worker 79 - jepsen.util 889 :invoke :write [26 3] INFO [2022-03-13 17:10:43,602] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:10:43,603] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:43,603] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,603] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:43,603] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:10:43,604] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:43,605] jepsen worker 21 - jepsen.util 741 :invoke :write [7 19] INFO [2022-03-13 17:10:43,617] jepsen worker 6 - jepsen.util 546 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,617] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,618] jepsen worker 6 - jepsen.util 546 :invoke :write [2 26] INFO [2022-03-13 17:10:43,618] jepsen worker 34 - jepsen.util 754 :invoke :write [11 32] INFO [2022-03-13 17:10:43,628] jepsen worker 26 - jepsen.util 476 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,629] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:43,893] jepsen worker 42 - jepsen.util 402 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,893] jepsen worker 43 - jepsen.util 403 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,893] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,894] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:43,894] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:43,894] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,895] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:43,895] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:10:43,927] jepsen worker 46 - jepsen.util 1576 :info :write [15 16] :RequestTimeoutException INFO [2022-03-13 17:10:43,928] jepsen worker 46 - jepsen.util 1666 :invoke :read [15 nil] INFO [2022-03-13 17:10:43,930] jepsen worker 16 - jepsen.util 1456 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:10:43,931] jepsen worker 16 - jepsen.util 1546 :invoke :read [5 nil] INFO [2022-03-13 17:10:43,945] jepsen worker 69 - jepsen.util 609 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,946] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:43,953] jepsen worker 65 - jepsen.util 1595 :info :write [21 32] :RequestTimeoutException INFO [2022-03-13 17:10:43,954] jepsen worker 65 - jepsen.util 1685 :invoke :write [21 40] INFO [2022-03-13 17:10:43,958] jepsen worker 80 - jepsen.util 620 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,959] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:43,959] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,960] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:43,960] jepsen worker 25 - jepsen.util 385 :invoke :write [8 27] INFO [2022-03-13 17:10:43,962] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:43,979] jepsen worker 79 - jepsen.util 889 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:43,980] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:10:44,002] jepsen worker 71 - jepsen.util 431 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,003] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:44,052] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,052] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:44,063] jepsen worker 21 - jepsen.util 741 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,064] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:44,126] jepsen worker 68 - jepsen.util 1418 :info :write [22 32] :RequestTimeoutException INFO [2022-03-13 17:10:44,127] jepsen worker 68 - jepsen.util 1508 :invoke :read [22 nil] INFO [2022-03-13 17:10:44,168] jepsen worker 14 - jepsen.util 284 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,168] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,168] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:44,169] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:44,169] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:44,170] jepsen worker 14 - jepsen.util 284 :invoke :write [4 44] INFO [2022-03-13 17:10:44,174] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,174] jepsen worker 76 - jepsen.util 616 :invoke :write [25 43] INFO [2022-03-13 17:10:44,269] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:44,269] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,270] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:44,270] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:10:44,290] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,291] jepsen worker 79 - jepsen.util 889 :invoke :write [26 4] INFO [2022-03-13 17:10:44,294] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3020527859237536}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3020527859237536}, :ns_1@172.23.107.228 {:progress 0.3235294117647058}} INFO [2022-03-13 17:10:44,295] jepsen worker nemesis - couchbase.util Rebalance check count: 156 INFO [2022-03-13 17:10:44,295] jepsen worker nemesis - couchbase.util Rebalance stuck count: 90 INFO [2022-03-13 17:10:44,303] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,303] jepsen worker 60 - jepsen.util 600 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,303] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,304] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:44,303] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:44,305] jepsen worker 12 - jepsen.util 732 :invoke :write [4 44] INFO [2022-03-13 17:10:44,306] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:44,306] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:44,307] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:44,308] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:10:44,361] jepsen worker 8 - jepsen.util 638 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,362] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:44,420] jepsen worker 58 - jepsen.util 1318 :info :write [19 34] :RequestTimeoutException INFO [2022-03-13 17:10:44,421] jepsen worker 58 - jepsen.util 1408 :invoke :read [19 nil] INFO [2022-03-13 17:10:44,516] jepsen worker 85 - jepsen.util 1435 :info :write [28 15] :RequestTimeoutException INFO [2022-03-13 17:10:44,517] jepsen worker 85 - jepsen.util 1525 :invoke :write [28 14] INFO [2022-03-13 17:10:44,559] jepsen worker 40 - jepsen.util 1480 :ok :read [13 39] INFO [2022-03-13 17:10:44,559] jepsen worker 40 - jepsen.util 1480 :invoke :write [13 4] INFO [2022-03-13 17:10:44,750] jepsen worker 72 - jepsen.util 1422 :ok :read [24 24] INFO [2022-03-13 17:10:44,750] jepsen worker 72 - jepsen.util 1422 :invoke :write [24 36] INFO [2022-03-13 17:10:44,850] jepsen worker 11 - jepsen.util 1361 :info :write [3 5] :RequestTimeoutException INFO [2022-03-13 17:10:44,851] jepsen worker 11 - jepsen.util 1451 :invoke :write [3 14] INFO [2022-03-13 17:10:44,865] jepsen worker 53 - jepsen.util 503 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,866] jepsen worker 53 - jepsen.util 503 :invoke :write [17 3] INFO [2022-03-13 17:10:44,875] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,875] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:44,875] jepsen worker 76 - jepsen.util 616 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,876] jepsen worker 21 - jepsen.util 741 :invoke :write [7 11] INFO [2022-03-13 17:10:44,878] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:44,879] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:44,881] jepsen worker 70 - jepsen.util 700 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,881] jepsen worker 70 - jepsen.util 700 :invoke :write [23 45] INFO [2022-03-13 17:10:44,980] jepsen worker 67 - jepsen.util 1327 :info :write [22 17] :RequestTimeoutException INFO [2022-03-13 17:10:44,980] jepsen worker 54 - jepsen.util 1224 :ok :read [18 21] INFO [2022-03-13 17:10:44,981] jepsen worker 54 - jepsen.util 1224 :invoke :read [18 nil] INFO [2022-03-13 17:10:44,982] jepsen worker 67 - jepsen.util 1417 :invoke :read [22 nil] INFO [2022-03-13 17:10:44,985] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,985] jepsen worker 79 - jepsen.util 889 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,986] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:10:44,987] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:44,991] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,992] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:44,993] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,994] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:44,996] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:44,998] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,000] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,000] jepsen worker 76 - jepsen.util 616 :invoke :write [25 9] INFO [2022-03-13 17:10:45,007] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,007] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,007] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:45,007] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,007] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:45,008] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:45,009] jepsen worker 43 - jepsen.util 403 :invoke :write [14 7] INFO [2022-03-13 17:10:45,010] jepsen worker 23 - jepsen.util 563 :invoke :write [7 45] INFO [2022-03-13 17:10:45,011] jepsen worker 60 - jepsen.util 600 :invoke :write [20 10] INFO [2022-03-13 17:10:45,022] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,066] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,066] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,066] jepsen worker 61 - jepsen.util 601 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,066] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:45,067] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:45,068] jepsen worker 42 - jepsen.util 402 :invoke :write [14 32] INFO [2022-03-13 17:10:45,068] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:45,071] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,071] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:45,072] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:45,073] jepsen worker 54 - jepsen.util 1224 :ok :read [18 21] INFO [2022-03-13 17:10:45,073] jepsen worker 54 - jepsen.util 1224 :invoke :write [18 33] INFO [2022-03-13 17:10:45,085] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,086] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:45,116] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,116] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:10:45,119] jepsen worker 76 - jepsen.util 616 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,119] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,120] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,120] jepsen worker 13 - jepsen.util 553 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,120] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,121] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:45,121] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,121] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,122] jepsen worker 76 - jepsen.util 616 :invoke :write [25 22] INFO [2022-03-13 17:10:45,122] jepsen worker 79 - jepsen.util 889 :invoke :write [26 15] INFO [2022-03-13 17:10:45,177] jepsen worker 6 - jepsen.util 546 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,178] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:45,239] jepsen worker 21 - jepsen.util 741 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,239] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,240] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,240] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:45,241] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:45,242] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,256] jepsen worker 79 - jepsen.util 889 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,257] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:45,294] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,294] jepsen worker 78 - jepsen.util 438 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,294] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:45,295] jepsen worker 78 - jepsen.util 438 :invoke :write [26 41] INFO [2022-03-13 17:10:45,296] jepsen worker 80 - jepsen.util 620 :invoke :write [26 44] INFO [2022-03-13 17:10:45,296] jepsen worker 22 - jepsen.util 562 :invoke :write [7 45] INFO [2022-03-13 17:10:45,301] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3020527859237536}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3020527859237536}, :ns_1@172.23.107.228 {:progress 0.3235294117647058}} INFO [2022-03-13 17:10:45,301] jepsen worker nemesis - couchbase.util Rebalance check count: 157 INFO [2022-03-13 17:10:45,301] jepsen worker nemesis - couchbase.util Rebalance stuck count: 91 INFO [2022-03-13 17:10:45,306] jepsen worker 3 - jepsen.util 1353 :info :write [1 10] :RequestTimeoutException INFO [2022-03-13 17:10:45,306] jepsen worker 3 - jepsen.util 1443 :invoke :read [1 nil] INFO [2022-03-13 17:10:45,322] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,322] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,322] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,322] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:45,323] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:45,324] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:45,325] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:10:45,327] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,327] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,328] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:45,328] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:45,329] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:10:45,329] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,329] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:45,330] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:10:45,331] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:45,335] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:45,335] jepsen worker 3 - jepsen.util 1443 :ok :read [1 42] INFO [2022-03-13 17:10:45,336] jepsen worker 21 - jepsen.util 741 :invoke :write [7 0] INFO [2022-03-13 17:10:45,337] jepsen worker 3 - jepsen.util 1443 :invoke :write [1 27] INFO [2022-03-13 17:10:45,357] jepsen worker 2 - jepsen.util 1712 :info :write [0 41] :RequestTimeoutException INFO [2022-03-13 17:10:45,357] jepsen worker 2 - jepsen.util 1802 :invoke :read [0 nil] INFO [2022-03-13 17:10:45,361] jepsen worker 76 - jepsen.util 616 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,361] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,362] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:45,362] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,364] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,364] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,365] jepsen worker 79 - jepsen.util 889 :invoke :write [26 27] INFO [2022-03-13 17:10:45,365] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,367] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,367] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:10:45,380] jepsen worker 32 - jepsen.util 1292 :ok :read [10 :nil] INFO [2022-03-13 17:10:45,381] jepsen worker 32 - jepsen.util 1292 :invoke :read [10 nil] INFO [2022-03-13 17:10:45,389] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,390] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:45,390] jepsen worker 34 - jepsen.util 754 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,391] jepsen worker 34 - jepsen.util 754 :invoke :write [11 25] INFO [2022-03-13 17:10:45,410] jepsen worker 38 - jepsen.util 1748 :info :write [12 47] :RequestTimeoutException INFO [2022-03-13 17:10:45,411] jepsen worker 38 - jepsen.util 1838 :invoke :write [12 7] INFO [2022-03-13 17:10:45,438] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,438] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,476] jepsen worker 79 - jepsen.util 889 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,477] jepsen worker 79 - jepsen.util 889 :invoke :write [26 32] INFO [2022-03-13 17:10:45,522] jepsen worker 32 - jepsen.util 1292 :ok :read [10 :nil] INFO [2022-03-13 17:10:45,523] jepsen worker 32 - jepsen.util 1292 :invoke :read [10 nil] INFO [2022-03-13 17:10:45,527] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,528] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:45,569] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,569] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,594] jepsen worker 18 - jepsen.util 1188 :info :write [6 49] :RequestTimeoutException INFO [2022-03-13 17:10:45,594] jepsen worker 18 - jepsen.util 1278 :invoke :write [6 4] INFO [2022-03-13 17:10:45,595] jepsen worker 32 - jepsen.util 1292 :ok :read [10 :nil] INFO [2022-03-13 17:10:45,596] jepsen worker 32 - jepsen.util 1292 :invoke :write [10 43] INFO [2022-03-13 17:10:45,626] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,626] jepsen worker 79 - jepsen.util 889 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,627] jepsen worker 76 - jepsen.util 616 :invoke :write [25 11] INFO [2022-03-13 17:10:45,628] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:45,630] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,630] jepsen worker 79 - jepsen.util 889 :invoke :write [26 22] INFO [2022-03-13 17:10:45,632] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,633] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:10:45,638] jepsen worker 44 - jepsen.util 494 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,639] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:45,639] jepsen worker 25 - jepsen.util 385 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,639] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:45,656] jepsen worker 28 - jepsen.util 1558 :info :write [9 19] :RequestTimeoutException INFO [2022-03-13 17:10:45,657] jepsen worker 28 - jepsen.util 1648 :invoke :read [9 nil] WARN [2022-03-13 17:10:45,711] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":32854,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":723686,"total_server_duration_us":431984,"encode_duration_us":18067,"operation_id":"0x422e","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10009551},{"operation_name":"upsert","last_dispatch_duration_us":8857340,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9255494,"total_server_duration_us":86137,"encode_duration_us":20707,"operation_id":"0x4008","timeout_ms":10000,"last_server_duration_us":25362,"total_duration_us":10008080},{"operation_name":"upsert","last_dispatch_duration_us":57622,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":413159,"total_server_duration_us":187097,"encode_duration_us":18734,"operation_id":"0x4050","timeout_ms":10000,"last_server_duration_us":36933,"total_duration_us":10006898},{"operation_name":"upsert","last_dispatch_duration_us":82867,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9069391,"total_server_duration_us":244561,"encode_duration_us":27160,"operation_id":"0x42d3","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10006126},{"operation_name":"upsert","last_dispatch_duration_us":8112506,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8797797,"total_server_duration_us":303207,"encode_duration_us":19423,"operation_id":"0x4296","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10005909},{"operation_name":"upsert","last_dispatch_duration_us":34398,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":687179,"total_server_duration_us":415165,"encode_duration_us":18600,"operation_id":"0x42ee","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10003773},{"operation_name":"upsert","last_dispatch_duration_us":9067505,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11443856,"total_server_duration_us":9294590,"encode_duration_us":17938,"operation_id":"0x3f80","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10003182},{"operation_name":"upsert","last_dispatch_duration_us":9093959,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":13041545,"total_server_duration_us":9288404,"encode_duration_us":26707,"operation_id":"0x4143","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10001534},{"operation_name":"get","last_dispatch_duration_us":2540907,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2540907,"total_server_duration_us":175499,"operation_id":"0x455c","timeout_ms":30000,"last_server_duration_us":175499,"total_duration_us":2543145},{"operation_name":"upsert","last_dispatch_duration_us":2128034,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2128034,"total_server_duration_us":69615,"encode_duration_us":17766,"operation_id":"0x437d","timeout_ms":10000,"last_server_duration_us":69615,"total_duration_us":2132964}],"total_count":22}} INFO [2022-03-13 17:10:45,735] jepsen worker 52 - jepsen.util 502 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,735] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,735] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,736] jepsen worker 52 - jepsen.util 502 :invoke :write [17 20] INFO [2022-03-13 17:10:45,737] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:45,737] jepsen worker 13 - jepsen.util 553 :invoke :write [4 28] INFO [2022-03-13 17:10:45,740] jepsen worker 21 - jepsen.util 741 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,741] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,741] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,741] jepsen worker 21 - jepsen.util 741 :invoke :write [7 29] INFO [2022-03-13 17:10:45,742] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:10:45,742] jepsen worker 33 - jepsen.util 663 :invoke :write [11 19] INFO [2022-03-13 17:10:45,750] jepsen worker 64 - jepsen.util 1234 :info :write [21 9] :RequestTimeoutException INFO [2022-03-13 17:10:45,751] jepsen worker 64 - jepsen.util 1324 :invoke :write [21 30] INFO [2022-03-13 17:10:45,758] jepsen worker 76 - jepsen.util 616 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,758] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] WARN [2022-03-13 17:10:45,811] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9093959,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13041545,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9288404,"encode_duration_us":26707,"operation_id":"0x4143","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10001534},{"operation_name":"upsert","last_dispatch_duration_us":9055581,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11540390,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9265597,"encode_duration_us":20847,"operation_id":"0x3f2b","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10008753},{"operation_name":"upsert","last_dispatch_duration_us":9067505,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11443856,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9294590,"encode_duration_us":17938,"operation_id":"0x3f80","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10003182}],"total_count":3}} INFO [2022-03-13 17:10:45,814] jepsen worker 73 - jepsen.util 1423 :info :write [24 21] :RequestTimeoutException INFO [2022-03-13 17:10:45,815] jepsen worker 73 - jepsen.util 1513 :invoke :write [24 18] INFO [2022-03-13 17:10:45,816] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,816] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,816] jepsen worker 70 - jepsen.util 700 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,817] jepsen worker 6 - jepsen.util 546 :invoke :write [2 15] INFO [2022-03-13 17:10:45,818] jepsen worker 70 - jepsen.util 700 :invoke :write [23 46] INFO [2022-03-13 17:10:45,818] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:45,887] jepsen worker 35 - jepsen.util 755 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,888] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:45,890] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,890] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:45,891] jepsen worker 79 - jepsen.util 889 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,891] jepsen worker 79 - jepsen.util 889 :invoke :write [26 26] INFO [2022-03-13 17:10:45,903] jepsen worker 55 - jepsen.util 1405 :info :write [18 9] :RequestTimeoutException INFO [2022-03-13 17:10:45,904] jepsen worker 55 - jepsen.util 1495 :invoke :read [18 nil] INFO [2022-03-13 17:10:45,967] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,967] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,978] jepsen worker 34 - jepsen.util 754 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,978] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,979] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:45,979] jepsen worker 71 - jepsen.util 431 :invoke :write [23 31] INFO [2022-03-13 17:10:45,983] jepsen worker 34 - jepsen.util 754 :invoke :write [11 44] INFO [2022-03-13 17:10:45,983] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:45,987] jepsen worker 14 - jepsen.util 284 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,988] jepsen worker 12 - jepsen.util 732 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,990] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:10:45,991] jepsen worker 12 - jepsen.util 732 :invoke :write [4 11] INFO [2022-03-13 17:10:45,991] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,991] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:45,992] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:45,992] jepsen worker 51 - jepsen.util 591 :invoke :write [17 47] INFO [2022-03-13 17:10:46,008] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,009] jepsen worker 76 - jepsen.util 616 :invoke :write [25 7] INFO [2022-03-13 17:10:46,064] jepsen worker 24 - jepsen.util 474 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,066] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:46,069] jepsen worker 21 - jepsen.util 741 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,069] jepsen worker 21 - jepsen.util 741 :invoke :write [7 17] INFO [2022-03-13 17:10:46,070] jepsen worker 79 - jepsen.util 889 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,072] jepsen worker 79 - jepsen.util 889 :invoke :write [26 5] INFO [2022-03-13 17:10:46,140] jepsen worker 76 - jepsen.util 616 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,140] jepsen worker 70 - jepsen.util 700 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,141] jepsen worker 70 - jepsen.util 700 :invoke :write [23 45] INFO [2022-03-13 17:10:46,141] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:46,169] jepsen worker 33 - jepsen.util 663 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,169] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,169] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,170] jepsen worker 69 - jepsen.util 609 :invoke :write [23 21] INFO [2022-03-13 17:10:46,171] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:10:46,171] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:46,232] jepsen worker 79 - jepsen.util 889 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,232] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,233] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:46,233] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:10:46,238] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,238] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:46,240] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,241] jepsen worker 76 - jepsen.util 616 :invoke :write [25 49] INFO [2022-03-13 17:10:46,255] jepsen worker 21 - jepsen.util 741 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,255] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,256] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:46,256] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:46,268] jepsen worker 83 - jepsen.util 983 :info :write [27 17] :RequestTimeoutException INFO [2022-03-13 17:10:46,268] jepsen worker 83 - jepsen.util 1073 :invoke :read [27 nil] INFO [2022-03-13 17:10:46,278] jepsen worker 66 - jepsen.util 1326 :info :write [22 27] :RequestTimeoutException INFO [2022-03-13 17:10:46,279] jepsen worker 66 - jepsen.util 1416 :invoke :write [22 31] INFO [2022-03-13 17:10:46,282] jepsen worker 22 - jepsen.util 562 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,282] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:46,282] jepsen worker 6 - jepsen.util 546 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,283] jepsen worker 44 - jepsen.util 494 :invoke :write [14 16] INFO [2022-03-13 17:10:46,283] jepsen worker 6 - jepsen.util 546 :invoke :write [2 14] INFO [2022-03-13 17:10:46,284] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:46,308] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3020527859237536}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3020527859237536}, :ns_1@172.23.107.228 {:progress 0.3235294117647058}} INFO [2022-03-13 17:10:46,308] jepsen worker nemesis - couchbase.util Rebalance check count: 158 INFO [2022-03-13 17:10:46,308] jepsen worker nemesis - couchbase.util Rebalance stuck count: 92 INFO [2022-03-13 17:10:46,318] jepsen worker 36 - jepsen.util 1656 :info :write [12 5] :RequestTimeoutException INFO [2022-03-13 17:10:46,318] jepsen worker 36 - jepsen.util 1746 :invoke :read [12 nil] INFO [2022-03-13 17:10:46,347] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,348] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,348] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:10:46,348] jepsen worker 79 - jepsen.util 889 :invoke :write [26 18] INFO [2022-03-13 17:10:46,402] jepsen worker 80 - jepsen.util 620 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,402] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:46,430] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:46,430] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,430] jepsen worker 69 - jepsen.util 609 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,431] jepsen worker 33 - jepsen.util 663 :invoke :write [11 13] INFO [2022-03-13 17:10:46,431] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:46,432] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:46,439] jepsen worker 76 - jepsen.util 616 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,439] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:10:46,449] jepsen worker 71 - jepsen.util 431 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,449] jepsen worker 71 - jepsen.util 431 :invoke :write [23 27] INFO [2022-03-13 17:10:46,504] jepsen worker 30 - jepsen.util 1290 :info :write [10 19] :RequestTimeoutException INFO [2022-03-13 17:10:46,504] jepsen worker 30 - jepsen.util 1380 :invoke :read [10 nil] INFO [2022-03-13 17:10:46,506] jepsen worker 30 - jepsen.util 1380 :ok :read [10 :nil] INFO [2022-03-13 17:10:46,506] jepsen worker 30 - jepsen.util 1380 :invoke :read [10 nil] INFO [2022-03-13 17:10:46,508] jepsen worker 30 - jepsen.util 1380 :ok :read [10 :nil] INFO [2022-03-13 17:10:46,508] jepsen worker 30 - jepsen.util 1380 :invoke :write [10 34] INFO [2022-03-13 17:10:46,540] jepsen worker 79 - jepsen.util 889 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,541] jepsen worker 79 - jepsen.util 889 :invoke :write [26 32] INFO [2022-03-13 17:10:46,542] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,542] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:46,542] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,543] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:46,544] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:46,544] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:46,562] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,562] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,562] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,562] jepsen worker 78 - jepsen.util 438 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,563] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:46,563] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:46,564] jepsen worker 35 - jepsen.util 755 :invoke :write [11 14] INFO [2022-03-13 17:10:46,564] jepsen worker 8 - jepsen.util 638 :invoke :write [2 48] INFO [2022-03-13 17:10:46,660] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,661] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:46,746] jepsen worker 19 - jepsen.util 1279 :info :write [6 17] :RequestTimeoutException INFO [2022-03-13 17:10:46,746] jepsen worker 19 - jepsen.util 1369 :invoke :write [6 17] INFO [2022-03-13 17:10:46,925] jepsen worker 34 - jepsen.util 754 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,926] jepsen worker 34 - jepsen.util 754 :invoke :write [11 16] INFO [2022-03-13 17:10:46,928] jepsen worker 33 - jepsen.util 663 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,929] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,929] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:46,929] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:46,930] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:46,930] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:46,931] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,932] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:10:46,935] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,935] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:46,935] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,935] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:46,936] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:46,936] jepsen worker 21 - jepsen.util 741 :invoke :write [7 44] INFO [2022-03-13 17:10:46,937] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,938] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,939] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:46,940] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:46,940] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,941] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,941] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:10:46,947] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:46,949] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:46,950] jepsen worker 69 - jepsen.util 609 :invoke :write [23 23] INFO [2022-03-13 17:10:47,009] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,009] jepsen worker 79 - jepsen.util 889 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,009] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:47,010] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:47,017] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,018] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,018] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:47,019] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:47,020] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,020] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,021] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:47,021] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:47,022] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,022] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,023] jepsen worker 76 - jepsen.util 616 :invoke :write [25 23] INFO [2022-03-13 17:10:47,024] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:10:47,035] jepsen worker 52 - jepsen.util 502 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,036] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:47,049] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,049] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:47,105] jepsen worker 21 - jepsen.util 741 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,106] jepsen worker 21 - jepsen.util 741 :invoke :write [7 15] INFO [2022-03-13 17:10:47,118] jepsen worker 70 - jepsen.util 700 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,118] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] WARN [2022-03-13 17:10:47,146] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15176834,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15176834,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9040551,"encode_duration_us":17801,"operation_id":"0x3ff0","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10008031}],"total_count":1}} INFO [2022-03-13 17:10:47,170] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,171] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:10:47,174] jepsen worker 76 - jepsen.util 616 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,175] jepsen worker 76 - jepsen.util 616 :invoke :write [25 35] INFO [2022-03-13 17:10:47,220] jepsen worker 9 - jepsen.util 1539 :ok :read [3 25] INFO [2022-03-13 17:10:47,221] jepsen worker 9 - jepsen.util 1539 :invoke :read [3 nil] WARN [2022-03-13 17:10:47,254] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9064232,"last_local_id":"A312307A00000004/0000000095638474","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9613549,"last_local_socket":"172.23.104.79:33498","total_server_duration_us":9370742,"encode_duration_us":16940,"operation_id":"0x4136","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002751},{"operation_name":"upsert","last_dispatch_duration_us":18191126,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18323044,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9056746,"encode_duration_us":37587,"operation_id":"0x3fb7","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10005030}],"total_count":2}} INFO [2022-03-13 17:10:47,260] jepsen worker 45 - jepsen.util 1665 :info :write [15 7] :RequestTimeoutException INFO [2022-03-13 17:10:47,263] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,272] jepsen worker 69 - jepsen.util 609 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,273] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,314] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:47,509] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] WARN [2022-03-13 17:10:47,325] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":35704,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":832903,"total_server_duration_us":502585,"encode_duration_us":37847,"operation_id":"0x42ea","timeout_ms":10000,"last_server_duration_us":25275,"total_duration_us":10009850},{"operation_name":"upsert","last_dispatch_duration_us":78534,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5551332,"total_server_duration_us":313018,"encode_duration_us":18033,"operation_id":"0x4321","timeout_ms":10000,"last_server_duration_us":52420,"total_duration_us":10005953},{"operation_name":"upsert","last_dispatch_duration_us":8758763,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10034568,"total_server_duration_us":276459,"encode_duration_us":18312,"operation_id":"0x41a0","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10005485},{"operation_name":"upsert","last_dispatch_duration_us":77384,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1296530,"total_server_duration_us":262077,"encode_duration_us":17138,"operation_id":"0x4194","timeout_ms":10000,"last_server_duration_us":27402,"total_duration_us":10001695},{"operation_name":"upsert","last_dispatch_duration_us":9110198,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10504814,"total_server_duration_us":9305826,"encode_duration_us":29788,"operation_id":"0x419e","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10001390},{"operation_name":"upsert","last_dispatch_duration_us":8890287,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10335422,"total_server_duration_us":278125,"encode_duration_us":17339,"operation_id":"0x4192","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10001301},{"operation_name":"upsert","last_dispatch_duration_us":9062450,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9360956,"total_server_duration_us":9178961,"encode_duration_us":23308,"operation_id":"0x422d","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9475181},{"operation_name":"upsert","last_dispatch_duration_us":9189434,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9189434,"total_server_duration_us":9028877,"encode_duration_us":17537,"operation_id":"0x41a2","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":9195868},{"operation_name":"upsert","last_dispatch_duration_us":9085040,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9130406,"total_server_duration_us":9084001,"encode_duration_us":17385,"operation_id":"0x415a","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":9144173},{"operation_name":"get","last_dispatch_duration_us":6421208,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6421208,"total_server_duration_us":38,"operation_id":"0x4269","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":6423210}],"total_count":31}} INFO [2022-03-13 17:10:47,509] jepsen worker nemesis - couchbase.util Rebalance check count: 159 INFO [2022-03-13 17:10:47,328] jepsen worker 21 - jepsen.util 741 :fail :write [7 15] :ServerOutOfMemoryException WARN [2022-03-13 17:10:47,346] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":2155552,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":2155552,"total_server_duration_us":20124,"encode_duration_us":24843,"operation_id":"0x4342","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10009439},{"operation_name":"upsert","last_dispatch_duration_us":56251,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8112714,"total_server_duration_us":86074,"encode_duration_us":20270,"operation_id":"0x4266","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10005679},{"operation_name":"upsert","last_dispatch_duration_us":104408,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":104408,"total_server_duration_us":23644,"encode_duration_us":17650,"operation_id":"0x4173","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10003741},{"operation_name":"upsert","last_dispatch_duration_us":2199757,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":2199757,"total_server_duration_us":19809,"encode_duration_us":17096,"operation_id":"0x4343","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10002735},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","encode_duration_us":19209,"operation_id":"0x4344","timeout_ms":10000,"total_duration_us":10001036},{"operation_name":"upsert","last_dispatch_duration_us":9061294,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9061294,"total_server_duration_us":9049045,"encode_duration_us":20910,"operation_id":"0x4151","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":9068266},{"operation_name":"get","last_dispatch_duration_us":8129712,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":8129712,"total_server_duration_us":62,"operation_id":"0x41a1","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":8130851},{"operation_name":"get","last_dispatch_duration_us":6450428,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":6450428,"total_server_duration_us":43,"operation_id":"0x42f1","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":6450806},{"operation_name":"get","last_dispatch_duration_us":5532781,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5532781,"total_server_duration_us":1979857,"operation_id":"0x4341","timeout_ms":30000,"last_server_duration_us":1979857,"total_duration_us":5534672},{"operation_name":"get","last_dispatch_duration_us":5065436,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5065436,"total_server_duration_us":32,"operation_id":"0x4391","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":5066435}],"total_count":60}} INFO [2022-03-13 17:10:47,370] jepsen worker 7 - jepsen.util 457 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,370] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:47,370] jepsen worker 44 - jepsen.util 494 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,391] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,510] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] WARN [2022-03-13 17:10:47,416] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8177756,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8751990,"total_server_duration_us":295120,"encode_duration_us":17859,"operation_id":"0x42c6","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10010562},{"operation_name":"upsert","last_dispatch_duration_us":39061,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":992255,"total_server_duration_us":609375,"encode_duration_us":20264,"operation_id":"0x41f5","timeout_ms":10000,"last_server_duration_us":25362,"total_duration_us":10008032},{"operation_name":"upsert","last_dispatch_duration_us":49091,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":491063,"total_server_duration_us":303526,"encode_duration_us":19015,"operation_id":"0x4267","timeout_ms":10000,"last_server_duration_us":37859,"total_duration_us":10004854},{"operation_name":"upsert","last_dispatch_duration_us":9086377,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9940186,"total_server_duration_us":9385210,"encode_duration_us":19808,"operation_id":"0x40f8","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10004576},{"operation_name":"upsert","last_dispatch_duration_us":7799269,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8609225,"total_server_duration_us":505055,"encode_duration_us":17407,"operation_id":"0x40a6","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10003461},{"operation_name":"upsert","last_dispatch_duration_us":90665,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1005430,"total_server_duration_us":469447,"encode_duration_us":22666,"operation_id":"0x40f5","timeout_ms":10000,"last_server_duration_us":78273,"total_duration_us":10001089},{"operation_name":"get","last_dispatch_duration_us":8223004,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8223004,"total_server_duration_us":62,"operation_id":"0x4357","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":8224883},{"operation_name":"get","last_dispatch_duration_us":7611744,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7611744,"total_server_duration_us":38,"operation_id":"0x4304","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":7614650},{"operation_name":"upsert","last_dispatch_duration_us":876555,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1637923,"total_server_duration_us":128468,"encode_duration_us":18850,"operation_id":"0x4255","timeout_ms":10000,"last_server_duration_us":83413,"total_duration_us":7320682},{"operation_name":"get","last_dispatch_duration_us":360994,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":360994,"total_server_duration_us":15,"operation_id":"0x433b","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":5017392}],"total_count":78}} INFO [2022-03-13 17:10:47,510] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:47,419] jepsen worker 76 - jepsen.util 616 :fail :write [25 35] :ServerOutOfMemoryException WARN [2022-03-13 17:10:47,454] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17842,"operation_id":"0x4351","timeout_ms":10000,"total_duration_us":10010021},{"operation_name":"upsert","last_dispatch_duration_us":10901481,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10901481,"total_server_duration_us":24239,"encode_duration_us":12289,"operation_id":"0x4268","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10008476},{"operation_name":"upsert","last_dispatch_duration_us":6285742,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11943115,"total_server_duration_us":192067,"encode_duration_us":18633,"operation_id":"0x4207","timeout_ms":10000,"last_server_duration_us":25800,"total_duration_us":10007800},{"operation_name":"upsert","last_dispatch_duration_us":104974,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9638970,"total_server_duration_us":122389,"encode_duration_us":18442,"operation_id":"0x432f","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10003977},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":24100,"operation_id":"0x4319","timeout_ms":10000,"total_duration_us":10003115},{"operation_name":"upsert","last_dispatch_duration_us":89659,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9748261,"total_server_duration_us":161282,"encode_duration_us":20025,"operation_id":"0x432c","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10002787},{"operation_name":"upsert","last_dispatch_duration_us":9064232,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9613549,"total_server_duration_us":9370742,"encode_duration_us":16940,"operation_id":"0x4136","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002751},{"operation_name":"get","last_dispatch_duration_us":9086369,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9086369,"total_server_duration_us":32,"operation_id":"0x432e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9088571},{"operation_name":"get","last_dispatch_duration_us":166107,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":166107,"total_server_duration_us":56,"operation_id":"0x43b0","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":7877886},{"operation_name":"get","last_dispatch_duration_us":3101732,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":3101732,"total_server_duration_us":43,"operation_id":"0x42e0","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3104364}],"total_count":51}} INFO [2022-03-13 17:10:47,511] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:47,455] jepsen worker 13 - jepsen.util 553 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,455] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,455] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,511] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:47,511] jepsen worker 80 - jepsen.util 620 :invoke :write [26 46] INFO [2022-03-13 17:10:47,456] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,511] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] WARN [2022-03-13 17:10:47,507] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":68285,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":671915,"total_server_duration_us":275703,"encode_duration_us":21878,"operation_id":"0x4288","timeout_ms":10000,"last_server_duration_us":40589,"total_duration_us":10008949},{"operation_name":"upsert","last_dispatch_duration_us":102552,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":631024,"total_server_duration_us":286611,"encode_duration_us":17751,"operation_id":"0x429d","timeout_ms":10000,"last_server_duration_us":33328,"total_duration_us":10007862},{"operation_name":"upsert","last_dispatch_duration_us":7469360,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8260412,"total_server_duration_us":494035,"encode_duration_us":17703,"operation_id":"0x409f","timeout_ms":10000,"last_server_duration_us":68680,"total_duration_us":10007316},{"operation_name":"upsert","last_dispatch_duration_us":5603716,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":5603716,"total_server_duration_us":20441,"encode_duration_us":17307,"operation_id":"0x437f","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10006629},{"operation_name":"upsert","last_dispatch_duration_us":7807212,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8062514,"total_server_duration_us":129776,"encode_duration_us":17929,"operation_id":"0x42ec","timeout_ms":10000,"last_server_duration_us":24841,"total_duration_us":10005329},{"operation_name":"upsert","last_dispatch_duration_us":7451984,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8111417,"total_server_duration_us":422046,"encode_duration_us":21516,"operation_id":"0x40fa","timeout_ms":10000,"last_server_duration_us":35916,"total_duration_us":10004906},{"operation_name":"upsert","last_dispatch_duration_us":9046946,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9046946,"total_server_duration_us":9027816,"encode_duration_us":10423,"operation_id":"0x4293","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":9051492},{"operation_name":"get","last_dispatch_duration_us":7835549,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7835549,"total_server_duration_us":49,"operation_id":"0x4320","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":7843484},{"operation_name":"get","last_dispatch_duration_us":3626927,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":3626927,"total_server_duration_us":13756,"operation_id":"0x4420","timeout_ms":30000,"last_server_duration_us":13756,"total_duration_us":3629415},{"operation_name":"upsert","last_dispatch_duration_us":1388708,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2156287,"total_server_duration_us":142805,"encode_duration_us":20653,"operation_id":"0x45e3","timeout_ms":10000,"last_server_duration_us":56770,"total_duration_us":2172084}],"total_count":52}} INFO [2022-03-13 17:10:47,509] jepsen worker 45 - jepsen.util 1755 :invoke :read [15 nil] INFO [2022-03-13 17:10:47,509] jepsen worker 79 - jepsen.util 889 :invoke :write [26 28] INFO [2022-03-13 17:10:47,509] jepsen worker 69 - jepsen.util 609 :invoke :write [23 42] INFO [2022-03-13 17:10:47,510] jepsen worker nemesis - couchbase.util Rebalance stuck count: 92 INFO [2022-03-13 17:10:47,512] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,513] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,513] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:10:47,513] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] WARN [2022-03-13 17:10:47,510] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9110198,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10504814,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9305826,"encode_duration_us":29788,"operation_id":"0x419e","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10001390},{"operation_name":"upsert","last_dispatch_duration_us":18163166,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27332812,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9111530,"encode_duration_us":22028,"operation_id":"0x3baa","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10009251},{"operation_name":"upsert","last_dispatch_duration_us":17800542,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17800542,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9025695,"encode_duration_us":17653,"operation_id":"0x3e7f","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10007264}],"total_count":3}} INFO [2022-03-13 17:10:47,510] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:47,510] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:47,511] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] WARN [2022-03-13 17:10:47,511] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9048153,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10245319,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9572359,"encode_duration_us":19851,"operation_id":"0x3ff6","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10002110},{"operation_name":"upsert","last_dispatch_duration_us":9086377,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9940186,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9385210,"encode_duration_us":19808,"operation_id":"0x40f8","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10004576}],"total_count":2}} INFO [2022-03-13 17:10:47,511] jepsen worker 78 - jepsen.util 438 :invoke :write [26 16] INFO [2022-03-13 17:10:47,516] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:47,517] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,517] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:47,518] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,519] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:47,519] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:47,520] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:47,521] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,522] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:47,522] jepsen worker 76 - jepsen.util 616 :invoke :write [25 6] INFO [2022-03-13 17:10:47,523] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:47,525] jepsen worker 21 - jepsen.util 741 :invoke :write [7 39] INFO [2022-03-13 17:10:47,535] jepsen worker 6 - jepsen.util 546 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,536] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:47,604] jepsen worker 79 - jepsen.util 889 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,604] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] WARN [2022-03-13 17:10:47,612] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9057206,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11220187,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9426941,"encode_duration_us":18127,"operation_id":"0x4001","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10003206},{"operation_name":"upsert","last_dispatch_duration_us":9072154,"last_local_id":"A312307A00000006/000000005DE126B4","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9798043,"last_local_socket":"172.23.104.79:33480","total_server_duration_us":9496518,"encode_duration_us":20031,"operation_id":"0x3f58","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10003400}],"total_count":2}} INFO [2022-03-13 17:10:47,680] jepsen worker 76 - jepsen.util 616 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,681] jepsen worker 69 - jepsen.util 609 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,681] jepsen worker 76 - jepsen.util 616 :invoke :write [25 45] INFO [2022-03-13 17:10:47,682] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:10:47,695] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,695] jepsen worker 23 - jepsen.util 563 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,696] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,695] jepsen worker 53 - jepsen.util 503 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,695] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,698] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:47,699] jepsen worker 61 - jepsen.util 601 :invoke :write [20 13] INFO [2022-03-13 17:10:47,699] jepsen worker 26 - jepsen.util 476 :invoke :write [8 18] INFO [2022-03-13 17:10:47,700] jepsen worker 53 - jepsen.util 503 :invoke :write [17 7] INFO [2022-03-13 17:10:47,701] jepsen worker 8 - jepsen.util 638 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,702] jepsen worker 8 - jepsen.util 638 :invoke :write [2 16] INFO [2022-03-13 17:10:47,705] jepsen worker 62 - jepsen.util 512 :invoke :write [20 19] INFO [2022-03-13 17:10:47,779] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,779] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:10:47,785] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,786] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:10:47,858] jepsen worker 71 - jepsen.util 431 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,859] jepsen worker 71 - jepsen.util 431 :invoke :write [23 37] INFO [2022-03-13 17:10:47,892] jepsen worker 76 - jepsen.util 616 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,893] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:47,896] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,897] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:10:47,980] jepsen worker 29 - jepsen.util 1469 :info :write [9 26] :RequestTimeoutException INFO [2022-03-13 17:10:47,981] jepsen worker 29 - jepsen.util 1559 :invoke :read [9 nil] INFO [2022-03-13 17:10:47,992] jepsen worker 21 - jepsen.util 741 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,992] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,992] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,993] jepsen worker 76 - jepsen.util 616 :invoke :write [25 47] INFO [2022-03-13 17:10:47,994] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:47,995] jepsen worker 21 - jepsen.util 741 :invoke :write [7 17] INFO [2022-03-13 17:10:47,997] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:47,999] jepsen worker 79 - jepsen.util 889 :invoke :write [26 26] INFO [2022-03-13 17:10:48,064] jepsen worker 35 - jepsen.util 755 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,064] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,064] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,065] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:48,065] jepsen worker 35 - jepsen.util 755 :invoke :write [11 3] INFO [2022-03-13 17:10:48,066] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:48,067] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,066] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,067] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,067] jepsen worker 13 - jepsen.util 553 :invoke :write [4 4] INFO [2022-03-13 17:10:48,070] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:48,070] jepsen worker 52 - jepsen.util 502 :invoke :write [17 22] INFO [2022-03-13 17:10:48,071] jepsen worker 7 - jepsen.util 457 :invoke :write [2 10] INFO [2022-03-13 17:10:48,071] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:48,076] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,076] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:48,124] jepsen worker 76 - jepsen.util 616 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,125] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,142] jepsen worker 34 - jepsen.util 754 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,142] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,143] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:48,143] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,143] jepsen worker 34 - jepsen.util 754 :invoke :write [11 24] INFO [2022-03-13 17:10:48,145] jepsen worker 70 - jepsen.util 700 :invoke :write [23 12] INFO [2022-03-13 17:10:48,147] jepsen worker 44 - jepsen.util 494 :invoke :write [14 27] INFO [2022-03-13 17:10:48,147] jepsen worker 6 - jepsen.util 546 :invoke :write [2 34] INFO [2022-03-13 17:10:48,150] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,151] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:10:48,192] jepsen worker 79 - jepsen.util 889 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,192] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,193] jepsen worker 79 - jepsen.util 889 :invoke :write [26 47] INFO [2022-03-13 17:10:48,194] jepsen worker 76 - jepsen.util 616 :invoke :write [25 14] INFO [2022-03-13 17:10:48,209] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,210] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:10:48,265] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,265] jepsen worker 21 - jepsen.util 741 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,265] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:48,266] jepsen worker 21 - jepsen.util 741 :invoke :write [7 25] INFO [2022-03-13 17:10:48,320] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,320] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:48,321] jepsen worker 76 - jepsen.util 616 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,322] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,374] jepsen worker 17 - jepsen.util 1457 :info :write [5 25] :RequestTimeoutException INFO [2022-03-13 17:10:48,374] jepsen worker 17 - jepsen.util 1547 :invoke :write [5 17] INFO [2022-03-13 17:10:48,397] jepsen worker 78 - jepsen.util 438 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,397] jepsen worker 71 - jepsen.util 431 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,397] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:48,398] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:48,432] jepsen worker 79 - jepsen.util 889 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,432] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,432] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,432] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,433] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,433] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:48,434] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:48,434] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,437] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,437] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,438] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:48,438] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,439] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,439] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,440] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:48,440] jepsen worker 76 - jepsen.util 616 :invoke :write [25 39] INFO [2022-03-13 17:10:48,440] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,441] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:48,442] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,443] jepsen worker 79 - jepsen.util 889 :invoke :write [26 32] INFO [2022-03-13 17:10:48,506] jepsen worker 80 - jepsen.util 620 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,507] jepsen worker 80 - jepsen.util 620 :invoke :write [26 1] INFO [2022-03-13 17:10:48,517] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:48,517] jepsen worker nemesis - couchbase.util Rebalance check count: 160 INFO [2022-03-13 17:10:48,517] jepsen worker nemesis - couchbase.util Rebalance stuck count: 93 INFO [2022-03-13 17:10:48,527] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,527] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,527] jepsen worker 21 - jepsen.util 741 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,527] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,528] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:48,528] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:48,529] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:48,529] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,531] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,531] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,532] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:48,532] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,533] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:48,533] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,534] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:48,534] jepsen worker 21 - jepsen.util 741 :invoke :write [7 33] INFO [2022-03-13 17:10:48,535] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,535] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,536] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,536] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,536] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:10:48,537] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:10:48,537] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,537] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,538] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,539] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:10:48,543] jepsen worker 34 - jepsen.util 754 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,545] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:48,548] jepsen worker 76 - jepsen.util 616 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,549] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:48,637] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,637] jepsen worker 79 - jepsen.util 889 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:48,638] jepsen worker 76 - jepsen.util 616 :invoke :write [25 6] INFO [2022-03-13 17:10:48,638] jepsen worker 79 - jepsen.util 889 :invoke :write [26 48] INFO [2022-03-13 17:10:49,172] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,173] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:49,176] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,176] jepsen worker 8 - jepsen.util 638 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,176] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,176] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:49,176] jepsen worker 25 - jepsen.util 385 :invoke :write [8 36] INFO [2022-03-13 17:10:49,177] jepsen worker 8 - jepsen.util 638 :invoke :write [2 44] INFO [2022-03-13 17:10:49,178] jepsen worker 22 - jepsen.util 562 :invoke :write [7 44] INFO [2022-03-13 17:10:49,181] jepsen worker 24 - jepsen.util 474 :invoke :write [8 19] INFO [2022-03-13 17:10:49,208] jepsen worker 76 - jepsen.util 616 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,208] jepsen worker 76 - jepsen.util 616 :invoke :write [25 16] INFO [2022-03-13 17:10:49,241] jepsen worker 6 - jepsen.util 546 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,242] jepsen worker 6 - jepsen.util 546 :invoke :write [2 17] INFO [2022-03-13 17:10:49,256] jepsen worker 21 - jepsen.util 741 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,256] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:49,298] jepsen worker 79 - jepsen.util 889 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,299] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:49,307] jepsen worker 4 - jepsen.util 1624 :ok :read [1 42] INFO [2022-03-13 17:10:49,307] jepsen worker 4 - jepsen.util 1624 :invoke :read [1 nil] INFO [2022-03-13 17:10:49,312] jepsen worker 36 - jepsen.util 1746 :ok :read [12 20] INFO [2022-03-13 17:10:49,313] jepsen worker 36 - jepsen.util 1746 :invoke :read [12 nil] INFO [2022-03-13 17:10:49,332] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,333] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:49,333] jepsen worker 4 - jepsen.util 1624 :ok :read [1 42] INFO [2022-03-13 17:10:49,334] jepsen worker 4 - jepsen.util 1624 :invoke :write [1 3] INFO [2022-03-13 17:10:49,341] jepsen worker 36 - jepsen.util 1746 :ok :read [12 20] INFO [2022-03-13 17:10:49,341] jepsen worker 36 - jepsen.util 1746 :invoke :read [12 nil] INFO [2022-03-13 17:10:49,368] jepsen worker 35 - jepsen.util 755 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,368] jepsen worker 36 - jepsen.util 1746 :ok :read [12 20] INFO [2022-03-13 17:10:49,369] jepsen worker 35 - jepsen.util 755 :invoke :write [11 10] INFO [2022-03-13 17:10:49,369] jepsen worker 36 - jepsen.util 1746 :invoke :read [12 nil] INFO [2022-03-13 17:10:49,374] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,374] jepsen worker 76 - jepsen.util 616 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,375] jepsen worker 76 - jepsen.util 616 :invoke :write [25 1] INFO [2022-03-13 17:10:49,375] jepsen worker 79 - jepsen.util 889 :invoke :write [26 25] INFO [2022-03-13 17:10:49,389] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,390] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:49,393] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,393] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:49,393] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,393] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:49,394] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:49,395] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:49,396] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,396] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,397] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:49,397] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:10:49,408] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:49,408] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,408] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,409] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:10:49,410] jepsen worker 21 - jepsen.util 741 :invoke :write [7 32] INFO [2022-03-13 17:10:49,415] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:10:49,433] jepsen worker 36 - jepsen.util 1746 :ok :read [12 20] INFO [2022-03-13 17:10:49,434] jepsen worker 36 - jepsen.util 1746 :invoke :write [12 38] INFO [2022-03-13 17:10:49,443] jepsen worker 43 - jepsen.util 403 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,444] jepsen worker 43 - jepsen.util 403 :invoke :write [14 31] INFO [2022-03-13 17:10:49,452] jepsen worker 70 - jepsen.util 700 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,452] jepsen worker 70 - jepsen.util 700 :invoke :write [23 22] INFO [2022-03-13 17:10:49,495] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,496] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:10:49,523] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:49,523] jepsen worker nemesis - couchbase.util Rebalance check count: 161 INFO [2022-03-13 17:10:49,523] jepsen worker nemesis - couchbase.util Rebalance stuck count: 94 INFO [2022-03-13 17:10:49,525] jepsen worker 79 - jepsen.util 889 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,525] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:49,567] jepsen worker 21 - jepsen.util 741 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,567] jepsen worker 21 - jepsen.util 741 :invoke :write [7 26] INFO [2022-03-13 17:10:49,605] jepsen worker 13 - jepsen.util 553 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,605] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:49,631] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,631] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:49,643] jepsen worker 76 - jepsen.util 616 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,643] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,644] jepsen worker 79 - jepsen.util 889 :invoke :write [26 13] INFO [2022-03-13 17:10:49,647] jepsen worker 76 - jepsen.util 616 :invoke :write [25 5] INFO [2022-03-13 17:10:49,670] jepsen worker 44 - jepsen.util 494 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,670] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,670] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,670] jepsen worker 44 - jepsen.util 494 :invoke :write [14 36] INFO [2022-03-13 17:10:49,671] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:49,671] jepsen worker 34 - jepsen.util 754 :invoke :write [11 25] INFO [2022-03-13 17:10:49,674] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,674] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:49,742] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,742] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:49,768] jepsen worker 21 - jepsen.util 741 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,768] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,768] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,768] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,769] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:49,770] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:10:49,775] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:49,775] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:49,848] jepsen worker 79 - jepsen.util 889 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,848] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:49,858] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:49,859] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,859] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,859] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:49,860] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:10:49,860] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:10:49,918] jepsen worker 6 - jepsen.util 546 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,918] jepsen worker 76 - jepsen.util 616 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,918] jepsen worker 7 - jepsen.util 457 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,919] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:49,920] jepsen worker 7 - jepsen.util 457 :invoke :write [2 37] INFO [2022-03-13 17:10:49,920] jepsen worker 6 - jepsen.util 546 :invoke :write [2 39] INFO [2022-03-13 17:10:49,924] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,925] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:49,927] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,927] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:49,928] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:49,929] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:49,978] jepsen worker 59 - jepsen.util 1319 :info :write [19 39] :RequestTimeoutException INFO [2022-03-13 17:10:49,979] jepsen worker 59 - jepsen.util 1409 :invoke :read [19 nil] INFO [2022-03-13 17:10:49,984] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,984] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:49,985] jepsen worker 76 - jepsen.util 616 :invoke :write [25 44] INFO [2022-03-13 17:10:49,985] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:10:50,000] jepsen worker 80 - jepsen.util 620 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,000] jepsen worker 80 - jepsen.util 620 :invoke :write [26 32] INFO [2022-03-13 17:10:50,020] jepsen worker 57 - jepsen.util 1587 :info :write [19 22] :RequestTimeoutException INFO [2022-03-13 17:10:50,021] jepsen worker 57 - jepsen.util 1677 :invoke :write [19 16] INFO [2022-03-13 17:10:50,034] jepsen worker 47 - jepsen.util 1577 :info :write [15 5] :RequestTimeoutException INFO [2022-03-13 17:10:50,034] jepsen worker 47 - jepsen.util 1667 :invoke :read [15 nil] INFO [2022-03-13 17:10:50,108] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,108] jepsen worker 70 - jepsen.util 700 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,109] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:50,110] jepsen worker 70 - jepsen.util 700 :invoke :write [23 40] INFO [2022-03-13 17:10:50,112] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,112] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,168] jepsen worker 76 - jepsen.util 616 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,168] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,206] jepsen worker 87 - jepsen.util 1977 :info :write [29 39] :RequestTimeoutException INFO [2022-03-13 17:10:50,207] jepsen worker 87 - jepsen.util 2067 :invoke :read [29 nil] INFO [2022-03-13 17:10:50,252] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:50,252] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,252] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,252] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,253] jepsen worker 69 - jepsen.util 609 :invoke :write [23 47] INFO [2022-03-13 17:10:50,253] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,254] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:50,254] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:10:50,254] jepsen worker 8 - jepsen.util 638 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,255] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:50,256] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,256] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,259] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,259] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,260] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:50,260] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,260] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:10:50,261] jepsen worker 76 - jepsen.util 616 :invoke :write [25 41] INFO [2022-03-13 17:10:50,261] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:10:50,262] jepsen worker 21 - jepsen.util 741 :invoke :write [7 25] INFO [2022-03-13 17:10:50,285] jepsen worker 44 - jepsen.util 494 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,286] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:50,295] jepsen worker 42 - jepsen.util 402 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,295] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,296] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:50,297] jepsen worker 14 - jepsen.util 284 :invoke :write [4 20] INFO [2022-03-13 17:10:50,396] jepsen worker 56 - jepsen.util 1406 :info :write [18 12] :RequestTimeoutException INFO [2022-03-13 17:10:50,397] jepsen worker 56 - jepsen.util 1496 :invoke :read [18 nil] INFO [2022-03-13 17:10:50,422] jepsen worker 25 - jepsen.util 385 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,423] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:50,527] jepsen worker 74 - jepsen.util 1784 :info :write [24 5] :RequestTimeoutException INFO [2022-03-13 17:10:50,527] jepsen worker 74 - jepsen.util 1874 :invoke :write [24 23] INFO [2022-03-13 17:10:50,531] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:50,531] jepsen worker nemesis - couchbase.util Rebalance check count: 162 INFO [2022-03-13 17:10:50,531] jepsen worker nemesis - couchbase.util Rebalance stuck count: 95 INFO [2022-03-13 17:10:50,552] jepsen worker 34 - jepsen.util 754 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,552] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:50,555] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,556] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:10:50,573] jepsen worker 69 - jepsen.util 609 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,573] jepsen worker 69 - jepsen.util 609 :invoke :write [23 28] INFO [2022-03-13 17:10:50,584] jepsen worker 76 - jepsen.util 616 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,586] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,614] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,614] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,615] jepsen worker 79 - jepsen.util 889 :invoke :write [26 9] INFO [2022-03-13 17:10:50,616] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,636] jepsen worker 27 - jepsen.util 1647 :info :write [9 36] :RequestTimeoutException INFO [2022-03-13 17:10:50,637] jepsen worker 27 - jepsen.util 1737 :invoke :read [9 nil] INFO [2022-03-13 17:10:50,712] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,713] jepsen worker 76 - jepsen.util 616 :invoke :write [25 35] INFO [2022-03-13 17:10:50,715] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,715] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:50,731] jepsen worker 5 - jepsen.util 1715 :info :write [1 26] :RequestTimeoutException INFO [2022-03-13 17:10:50,731] jepsen worker 5 - jepsen.util 1805 :invoke :read [1 nil] INFO [2022-03-13 17:10:50,756] jepsen worker 63 - jepsen.util 1503 :info :write [21 40] :RequestTimeoutException INFO [2022-03-13 17:10:50,757] jepsen worker 63 - jepsen.util 1593 :invoke :read [21 nil] INFO [2022-03-13 17:10:50,848] jepsen worker 79 - jepsen.util 889 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,848] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:50,859] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,859] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,895] jepsen worker 22 - jepsen.util 562 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,895] jepsen worker 6 - jepsen.util 546 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,895] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,896] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:50,896] jepsen worker 71 - jepsen.util 431 :invoke :write [23 46] INFO [2022-03-13 17:10:50,897] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:50,990] jepsen worker 10 - jepsen.util 1360 :info :write [3 7] :SyncWriteAmbiguous INFO [2022-03-13 17:10:50,992] jepsen worker 10 - jepsen.util 1450 :invoke :write [3 7] INFO [2022-03-13 17:10:50,992] jepsen worker 76 - jepsen.util 616 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,992] jepsen worker 21 - jepsen.util 741 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,993] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:50,993] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:50,996] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,996] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:50,996] jepsen worker 79 - jepsen.util 889 :invoke :write [26 44] INFO [2022-03-13 17:10:50,997] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:10:51,043] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,043] jepsen worker 7 - jepsen.util 457 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,044] jepsen worker 13 - jepsen.util 553 :invoke :write [4 21] INFO [2022-03-13 17:10:51,044] jepsen worker 7 - jepsen.util 457 :invoke :write [2 11] INFO [2022-03-13 17:10:51,083] jepsen worker 69 - jepsen.util 609 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,084] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,084] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:51,083] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,084] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:51,085] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:10:51,085] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:51,085] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:51,106] jepsen worker 55 - jepsen.util 1495 :ok :read [18 21] INFO [2022-03-13 17:10:51,106] jepsen worker 55 - jepsen.util 1495 :invoke :read [18 nil] INFO [2022-03-13 17:10:51,136] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,136] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:51,136] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,137] jepsen worker 33 - jepsen.util 663 :invoke :write [11 13] INFO [2022-03-13 17:10:51,137] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:51,138] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:51,138] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:51,138] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,138] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,139] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:51,140] jepsen worker 34 - jepsen.util 754 :invoke :write [11 17] INFO [2022-03-13 17:10:51,140] jepsen worker 6 - jepsen.util 546 :invoke :write [2 38] INFO [2022-03-13 17:10:51,141] jepsen worker 70 - jepsen.util 700 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,141] jepsen worker 70 - jepsen.util 700 :invoke :write [23 25] INFO [2022-03-13 17:10:51,152] jepsen worker 79 - jepsen.util 889 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,153] jepsen worker 79 - jepsen.util 889 :invoke :write [26 32] INFO [2022-03-13 17:10:51,221] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,221] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,221] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:51,222] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:51,223] jepsen worker 69 - jepsen.util 609 :invoke :write [23 43] INFO [2022-03-13 17:10:51,223] jepsen worker 75 - jepsen.util 435 :invoke :write [25 40] INFO [2022-03-13 17:10:51,284] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,284] jepsen worker 76 - jepsen.util 616 :invoke :write [25 46] INFO [2022-03-13 17:10:51,293] jepsen worker 52 - jepsen.util 502 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,294] jepsen worker 52 - jepsen.util 502 :invoke :write [17 29] INFO [2022-03-13 17:10:51,311] jepsen worker 79 - jepsen.util 889 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,311] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:51,389] jepsen worker 81 - jepsen.util 1791 :info :write [27 48] :RequestTimeoutException INFO [2022-03-13 17:10:51,390] jepsen worker 81 - jepsen.util 1881 :invoke :read [27 nil] INFO [2022-03-13 17:10:51,403] jepsen worker 89 - jepsen.util 1799 :info :write [29 38] :RequestTimeoutException INFO [2022-03-13 17:10:51,404] jepsen worker 89 - jepsen.util 1889 :invoke :read [29 nil] INFO [2022-03-13 17:10:51,447] jepsen worker 33 - jepsen.util 663 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,448] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:51,448] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:51,448] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:51,458] jepsen worker 76 - jepsen.util 616 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,458] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,459] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:51,459] jepsen worker 71 - jepsen.util 431 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,459] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:51,459] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:51,460] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:10:51,460] jepsen worker 44 - jepsen.util 494 :invoke :write [14 31] INFO [2022-03-13 17:10:51,466] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,467] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] INFO [2022-03-13 17:10:51,514] jepsen worker 20 - jepsen.util 1820 :ok :read [6 16] INFO [2022-03-13 17:10:51,515] jepsen worker 20 - jepsen.util 1820 :invoke :read [6 nil] INFO [2022-03-13 17:10:51,526] jepsen worker 29 - jepsen.util 1559 :ok :read [9 26] INFO [2022-03-13 17:10:51,527] jepsen worker 29 - jepsen.util 1559 :invoke :read [9 nil] INFO [2022-03-13 17:10:51,527] jepsen worker 56 - jepsen.util 1496 :ok :read [18 21] INFO [2022-03-13 17:10:51,528] jepsen worker 56 - jepsen.util 1496 :invoke :read [18 nil] INFO [2022-03-13 17:10:51,537] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:51,538] jepsen worker nemesis - couchbase.util Rebalance check count: 163 INFO [2022-03-13 17:10:51,538] jepsen worker nemesis - couchbase.util Rebalance stuck count: 96 INFO [2022-03-13 17:10:51,547] jepsen worker 20 - jepsen.util 1820 :ok :read [6 16] INFO [2022-03-13 17:10:51,548] jepsen worker 20 - jepsen.util 1820 :invoke :read [6 nil] INFO [2022-03-13 17:10:51,556] jepsen worker 29 - jepsen.util 1559 :ok :read [9 26] INFO [2022-03-13 17:10:51,557] jepsen worker 29 - jepsen.util 1559 :invoke :write [9 0] INFO [2022-03-13 17:10:51,558] jepsen worker 56 - jepsen.util 1496 :ok :read [18 21] INFO [2022-03-13 17:10:51,559] jepsen worker 56 - jepsen.util 1496 :invoke :read [18 nil] INFO [2022-03-13 17:10:51,572] jepsen worker 20 - jepsen.util 1820 :ok :read [6 16] INFO [2022-03-13 17:10:51,572] jepsen worker 20 - jepsen.util 1820 :invoke :write [6 0] INFO [2022-03-13 17:10:51,580] jepsen worker 56 - jepsen.util 1496 :ok :read [18 21] INFO [2022-03-13 17:10:51,580] jepsen worker 56 - jepsen.util 1496 :invoke :write [18 1] INFO [2022-03-13 17:10:51,646] jepsen worker 69 - jepsen.util 609 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:51,646] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:51,807] jepsen worker 82 - jepsen.util 1342 :info :write [27 32] :RequestTimeoutException INFO [2022-03-13 17:10:51,808] jepsen worker 82 - jepsen.util 1432 :invoke :write [27 6] INFO [2022-03-13 17:10:51,816] jepsen worker 48 - jepsen.util 1398 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:10:51,816] jepsen worker 48 - jepsen.util 1488 :invoke :write [16 45] INFO [2022-03-13 17:10:51,998] jepsen worker 31 - jepsen.util 1381 :info :write [10 28] :RequestTimeoutException INFO [2022-03-13 17:10:51,998] jepsen worker 87 - jepsen.util 2067 :ok :read [29 10] INFO [2022-03-13 17:10:51,999] jepsen worker 87 - jepsen.util 2067 :invoke :write [29 3] INFO [2022-03-13 17:10:51,999] jepsen worker 31 - jepsen.util 1471 :invoke :read [10 nil] INFO [2022-03-13 17:10:52,040] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,040] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:52,045] jepsen worker 80 - jepsen.util 620 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,045] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,045] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:52,046] jepsen worker 25 - jepsen.util 385 :invoke :write [8 13] INFO [2022-03-13 17:10:52,098] jepsen worker 34 - jepsen.util 754 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,099] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:52,107] jepsen worker 60 - jepsen.util 600 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,107] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,108] jepsen worker 60 - jepsen.util 600 :invoke :write [20 21] INFO [2022-03-13 17:10:52,109] jepsen worker 77 - jepsen.util 707 :invoke :write [25 24] INFO [2022-03-13 17:10:52,144] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,144] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,145] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,146] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:52,146] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,146] jepsen worker 75 - jepsen.util 435 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,146] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,146] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,147] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,147] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,147] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:10:52,147] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,148] jepsen worker 79 - jepsen.util 889 :invoke :write [26 24] INFO [2022-03-13 17:10:52,148] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:10:52,153] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,154] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,154] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,155] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,155] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,155] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,156] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,156] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,156] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,157] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:10:52,157] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,157] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,157] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,158] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,159] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,159] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,159] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,160] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,160] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,161] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,161] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,162] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,162] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,163] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,163] jepsen worker 21 - jepsen.util 741 :invoke :write [7 4] INFO [2022-03-13 17:10:52,163] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,164] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,165] jepsen worker 76 - jepsen.util 616 :invoke :write [25 8] INFO [2022-03-13 17:10:52,261] jepsen worker 79 - jepsen.util 889 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,262] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,271] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:52,271] jepsen worker 79 - jepsen.util 889 :invoke :write [26 35] INFO [2022-03-13 17:10:52,277] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,277] jepsen worker 24 - jepsen.util 474 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,277] jepsen worker 70 - jepsen.util 700 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,278] jepsen worker 70 - jepsen.util 700 :invoke :write [23 7] INFO [2022-03-13 17:10:52,279] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,279] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:52,330] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,331] jepsen worker 76 - jepsen.util 616 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,332] jepsen worker 76 - jepsen.util 616 :invoke :write [25 39] INFO [2022-03-13 17:10:52,333] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:52,369] jepsen worker 26 - jepsen.util 476 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,370] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:52,398] jepsen worker 35 - jepsen.util 755 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,398] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:52,514] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,514] jepsen worker 6 - jepsen.util 546 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,515] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:52,515] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:52,519] jepsen worker 79 - jepsen.util 889 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,519] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:52,522] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,523] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:10:52,544] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:52,544] jepsen worker nemesis - couchbase.util Rebalance check count: 164 INFO [2022-03-13 17:10:52,544] jepsen worker nemesis - couchbase.util Rebalance stuck count: 97 INFO [2022-03-13 17:10:52,584] jepsen worker 50 - jepsen.util 1580 :ok :read [16 :nil] INFO [2022-03-13 17:10:52,585] jepsen worker 50 - jepsen.util 1580 :invoke :write [16 32] INFO [2022-03-13 17:10:52,643] jepsen worker 76 - jepsen.util 616 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,643] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,644] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,644] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:10:52,650] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,651] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,651] jepsen worker 21 - jepsen.util 741 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,651] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,652] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,652] jepsen worker 21 - jepsen.util 741 :invoke :write [7 16] INFO [2022-03-13 17:10:52,652] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,652] jepsen worker 69 - jepsen.util 609 :invoke :write [23 1] INFO [2022-03-13 17:10:52,653] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:52,653] jepsen worker 76 - jepsen.util 616 :invoke :write [25 40] INFO [2022-03-13 17:10:52,653] jepsen worker 39 - jepsen.util 1569 :ok :read [13 39] INFO [2022-03-13 17:10:52,654] jepsen worker 39 - jepsen.util 1569 :invoke :write [13 20] INFO [2022-03-13 17:10:52,660] jepsen worker 13 - jepsen.util 553 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,661] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:52,712] jepsen worker 49 - jepsen.util 1309 :ok :read [16 :nil] INFO [2022-03-13 17:10:52,713] jepsen worker 49 - jepsen.util 1309 :invoke :write [16 28] INFO [2022-03-13 17:10:52,732] jepsen worker 44 - jepsen.util 494 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,734] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:52,735] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,736] jepsen worker 34 - jepsen.util 754 :invoke :write [11 16] INFO [2022-03-13 17:10:52,744] jepsen worker 84 - jepsen.util 1794 :ok :read [28 44] INFO [2022-03-13 17:10:52,746] jepsen worker 84 - jepsen.util 1794 :invoke :write [28 2] INFO [2022-03-13 17:10:52,754] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,754] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,755] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,756] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:52,762] jepsen worker 68 - jepsen.util 1508 :ok :read [22 30] INFO [2022-03-13 17:10:52,763] jepsen worker 68 - jepsen.util 1508 :invoke :write [22 23] INFO [2022-03-13 17:10:52,768] jepsen worker 2 - jepsen.util 1802 :ok :read [0 41] INFO [2022-03-13 17:10:52,769] jepsen worker 2 - jepsen.util 1802 :invoke :read [0 nil] INFO [2022-03-13 17:10:52,822] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,822] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:52,867] jepsen worker 2 - jepsen.util 1802 :ok :read [0 41] INFO [2022-03-13 17:10:52,868] jepsen worker 2 - jepsen.util 1802 :invoke :read [0 nil] INFO [2022-03-13 17:10:52,868] jepsen worker 7 - jepsen.util 457 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,868] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,868] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,869] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:52,869] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,869] jepsen worker 7 - jepsen.util 457 :invoke :write [2 3] INFO [2022-03-13 17:10:52,869] jepsen worker 8 - jepsen.util 638 :invoke :write [2 49] INFO [2022-03-13 17:10:52,870] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:52,871] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:52,873] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,880] jepsen worker 88 - jepsen.util 1438 :info :write [29 36] :RequestTimeoutException INFO [2022-03-13 17:10:52,880] jepsen worker 88 - jepsen.util 1528 :invoke :read [29 nil] INFO [2022-03-13 17:10:52,905] jepsen worker 88 - jepsen.util 1528 :ok :read [29 10] INFO [2022-03-13 17:10:52,905] jepsen worker 88 - jepsen.util 1528 :invoke :read [29 nil] INFO [2022-03-13 17:10:52,930] jepsen worker 88 - jepsen.util 1528 :ok :read [29 10] INFO [2022-03-13 17:10:52,930] jepsen worker 88 - jepsen.util 1528 :invoke :read [29 nil] INFO [2022-03-13 17:10:52,954] jepsen worker 15 - jepsen.util 1545 :info :write [5 6] :RequestTimeoutException INFO [2022-03-13 17:10:52,955] jepsen worker 88 - jepsen.util 1528 :ok :read [29 10] INFO [2022-03-13 17:10:52,957] jepsen worker 88 - jepsen.util 1528 :invoke :write [29 30] INFO [2022-03-13 17:10:52,957] jepsen worker 15 - jepsen.util 1635 :invoke :write [5 12] INFO [2022-03-13 17:10:52,985] jepsen worker 2 - jepsen.util 1802 :ok :read [0 41] INFO [2022-03-13 17:10:52,985] jepsen worker 21 - jepsen.util 741 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,987] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:52,987] jepsen worker 2 - jepsen.util 1802 :invoke :write [0 5] INFO [2022-03-13 17:10:52,992] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,992] jepsen worker 76 - jepsen.util 616 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,994] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:52,994] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:52,995] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,996] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:52,996] jepsen worker 76 - jepsen.util 616 :invoke :write [25 12] INFO [2022-03-13 17:10:52,997] jepsen worker 79 - jepsen.util 889 :invoke :write [26 49] INFO [2022-03-13 17:10:53,025] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,025] jepsen worker 69 - jepsen.util 609 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,025] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,026] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:53,027] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:10:53,027] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:10:53,028] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:53,028] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:53,031] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,032] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:53,031] jepsen worker 70 - jepsen.util 700 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,032] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,032] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:53,033] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:53,034] jepsen worker 6 - jepsen.util 546 :invoke :write [2 8] INFO [2022-03-13 17:10:53,034] jepsen worker 71 - jepsen.util 431 :invoke :write [23 29] INFO [2022-03-13 17:10:53,052] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,052] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:53,053] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:53,053] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:53,120] jepsen worker 79 - jepsen.util 889 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,121] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:10:53,127] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,127] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,127] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:53,128] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:53,129] jepsen worker 33 - jepsen.util 663 :invoke :write [11 1] INFO [2022-03-13 17:10:53,129] jepsen worker 21 - jepsen.util 741 :invoke :write [7 5] INFO [2022-03-13 17:10:53,170] jepsen worker 1 - jepsen.util 1261 :info :write [0 44] :RequestTimeoutException INFO [2022-03-13 17:10:53,171] jepsen worker 1 - jepsen.util 1351 :invoke :read [0 nil] INFO [2022-03-13 17:10:53,287] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,287] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,288] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:10:53,288] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:10:53,291] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:53,291] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,291] jepsen worker 25 - jepsen.util 385 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,292] jepsen worker 35 - jepsen.util 755 :invoke :write [11 47] INFO [2022-03-13 17:10:53,294] jepsen worker 25 - jepsen.util 385 :invoke :write [8 42] INFO [2022-03-13 17:10:53,298] jepsen worker 34 - jepsen.util 754 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,298] jepsen worker 44 - jepsen.util 494 :invoke :write [14 32] INFO [2022-03-13 17:10:53,299] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,299] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:53,299] jepsen worker 70 - jepsen.util 700 :invoke :write [23 17] INFO [2022-03-13 17:10:53,308] jepsen worker 76 - jepsen.util 616 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,309] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:10:53,326] jepsen worker 21 - jepsen.util 741 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,327] jepsen worker 21 - jepsen.util 741 :invoke :write [7 21] INFO [2022-03-13 17:10:53,473] jepsen worker 33 - jepsen.util 663 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,473] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:53,492] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,493] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:53,499] jepsen worker 52 - jepsen.util 502 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,499] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,499] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,500] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,500] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:53,500] jepsen worker 52 - jepsen.util 502 :invoke :write [17 22] INFO [2022-03-13 17:10:53,501] jepsen worker 80 - jepsen.util 620 :invoke :write [26 41] INFO [2022-03-13 17:10:53,502] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:53,504] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:53,504] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:53,549] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:53,549] jepsen worker nemesis - couchbase.util Rebalance check count: 165 INFO [2022-03-13 17:10:53,549] jepsen worker nemesis - couchbase.util Rebalance stuck count: 98 INFO [2022-03-13 17:10:53,599] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,599] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:10:53,626] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,626] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,627] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:53,628] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:53,628] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,629] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:53,629] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,629] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:53,630] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,630] jepsen worker 76 - jepsen.util 616 :invoke :write [25 14] INFO [2022-03-13 17:10:53,632] jepsen worker 40 - jepsen.util 1480 :info :write [13 4] :SyncWriteAmbiguous INFO [2022-03-13 17:10:53,632] jepsen worker 40 - jepsen.util 1570 :invoke :read [13 nil] INFO [2022-03-13 17:10:53,663] jepsen worker 0 - jepsen.util 1530 :ok :read [0 41] INFO [2022-03-13 17:10:53,664] jepsen worker 0 - jepsen.util 1530 :invoke :write [0 16] INFO [2022-03-13 17:10:53,707] jepsen worker 41 - jepsen.util 1301 :ok :read [13 39] INFO [2022-03-13 17:10:53,708] jepsen worker 41 - jepsen.util 1301 :invoke :write [13 38] INFO [2022-03-13 17:10:53,748] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,748] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:53,758] jepsen worker 71 - jepsen.util 431 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,758] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:53,763] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,763] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:53,779] jepsen worker 76 - jepsen.util 616 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,779] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,780] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:53,780] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:53,787] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,788] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,788] jepsen worker 76 - jepsen.util 616 :invoke :write [25 22] INFO [2022-03-13 17:10:53,788] jepsen worker 79 - jepsen.util 889 :invoke :write [26 39] INFO [2022-03-13 17:10:53,912] jepsen worker 21 - jepsen.util 741 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,912] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,912] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:10:53,913] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:53,956] jepsen worker 76 - jepsen.util 616 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,956] jepsen worker 76 - jepsen.util 616 :invoke :write [25 45] INFO [2022-03-13 17:10:53,967] jepsen worker 8 - jepsen.util 638 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,968] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:53,971] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,972] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:53,972] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:10:53,979] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:10:53,980] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:53,981] jepsen worker 21 - jepsen.util 741 :invoke :write [7 46] INFO [2022-03-13 17:10:53,985] jepsen worker 65 - jepsen.util 1685 :info :write [21 40] :RequestTimeoutException INFO [2022-03-13 17:10:53,986] jepsen worker 65 - jepsen.util 1775 :invoke :write [21 1] INFO [2022-03-13 17:10:54,017] jepsen worker 79 - jepsen.util 889 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,018] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:54,076] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,077] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:54,085] jepsen worker 6 - jepsen.util 546 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,086] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,086] jepsen worker 34 - jepsen.util 754 :invoke :write [11 14] INFO [2022-03-13 17:10:54,087] jepsen worker 6 - jepsen.util 546 :invoke :write [2 44] INFO [2022-03-13 17:10:54,108] jepsen worker 45 - jepsen.util 1755 :ok :read [15 48] INFO [2022-03-13 17:10:54,109] jepsen worker 45 - jepsen.util 1755 :invoke :write [15 34] INFO [2022-03-13 17:10:54,126] jepsen worker 46 - jepsen.util 1666 :ok :read [15 48] INFO [2022-03-13 17:10:54,127] jepsen worker 46 - jepsen.util 1666 :invoke :read [15 nil] INFO [2022-03-13 17:10:54,158] jepsen worker 55 - jepsen.util 1495 :ok :read [18 21] INFO [2022-03-13 17:10:54,158] jepsen worker 55 - jepsen.util 1495 :invoke :read [18 nil] INFO [2022-03-13 17:10:54,221] jepsen worker 55 - jepsen.util 1495 :ok :read [18 21] INFO [2022-03-13 17:10:54,221] jepsen worker 55 - jepsen.util 1495 :invoke :read [18 nil] INFO [2022-03-13 17:10:54,259] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,260] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:10:54,265] jepsen worker 55 - jepsen.util 1495 :ok :read [18 21] INFO [2022-03-13 17:10:54,265] jepsen worker 55 - jepsen.util 1495 :invoke :write [18 20] INFO [2022-03-13 17:10:54,290] jepsen worker 7 - jepsen.util 457 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,290] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,291] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:54,291] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:54,293] jepsen worker 12 - jepsen.util 732 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,293] jepsen worker 61 - jepsen.util 601 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,293] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:54,294] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:10:54,428] jepsen worker 44 - jepsen.util 494 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,429] jepsen worker 44 - jepsen.util 494 :invoke :write [14 23] INFO [2022-03-13 17:10:54,448] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,448] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:10:54,471] jepsen worker 51 - jepsen.util 591 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,471] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:54,471] jepsen worker 53 - jepsen.util 503 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,472] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:10:54,472] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:54,472] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:54,509] jepsen worker 43 - jepsen.util 403 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,509] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:54,550] jepsen worker 85 - jepsen.util 1525 :info :write [28 14] :RequestTimeoutException INFO [2022-03-13 17:10:54,551] jepsen worker 85 - jepsen.util 1615 :invoke :write [28 37] INFO [2022-03-13 17:10:54,555] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:54,555] jepsen worker nemesis - couchbase.util Rebalance check count: 166 INFO [2022-03-13 17:10:54,555] jepsen worker nemesis - couchbase.util Rebalance stuck count: 99 INFO [2022-03-13 17:10:54,615] jepsen worker 62 - jepsen.util 512 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,615] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:54,616] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:54,616] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:54,628] jepsen worker 21 - jepsen.util 741 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,628] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,629] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:54,629] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:54,681] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,682] jepsen worker 69 - jepsen.util 609 :invoke :write [23 29] INFO [2022-03-13 17:10:54,718] jepsen worker 31 - jepsen.util 1471 :ok :read [10 :nil] INFO [2022-03-13 17:10:54,718] jepsen worker 31 - jepsen.util 1471 :invoke :write [10 17] INFO [2022-03-13 17:10:54,770] jepsen worker 72 - jepsen.util 1422 :info :write [24 36] :RequestTimeoutException INFO [2022-03-13 17:10:54,770] jepsen worker 72 - jepsen.util 1512 :invoke :write [24 36] INFO [2022-03-13 17:10:54,781] jepsen worker 47 - jepsen.util 1667 :ok :read [15 48] INFO [2022-03-13 17:10:54,782] jepsen worker 47 - jepsen.util 1667 :invoke :write [15 33] INFO [2022-03-13 17:10:54,804] jepsen worker 70 - jepsen.util 700 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,804] jepsen worker 35 - jepsen.util 755 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,804] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,806] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:54,806] jepsen worker 35 - jepsen.util 755 :invoke :write [11 30] INFO [2022-03-13 17:10:54,808] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:54,815] jepsen worker 60 - jepsen.util 600 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,817] jepsen worker 60 - jepsen.util 600 :invoke :write [20 5] INFO [2022-03-13 17:10:54,858] jepsen worker 14 - jepsen.util 284 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,858] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,858] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:54,858] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,859] jepsen worker 14 - jepsen.util 284 :invoke :write [4 5] INFO [2022-03-13 17:10:54,859] jepsen worker 61 - jepsen.util 601 :invoke :write [20 29] INFO [2022-03-13 17:10:54,860] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,861] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,861] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:54,861] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:54,861] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:54,862] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:10:54,862] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,863] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,863] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,863] jepsen worker 62 - jepsen.util 512 :invoke :write [20 33] INFO [2022-03-13 17:10:54,863] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:10:54,864] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:54,877] jepsen worker 11 - jepsen.util 1451 :info :write [3 14] :RequestTimeoutException INFO [2022-03-13 17:10:54,878] jepsen worker 11 - jepsen.util 1541 :invoke :write [3 46] INFO [2022-03-13 17:10:54,992] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,992] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:54,992] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:54,993] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:54,994] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:54,995] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:55,000] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:10:55,000] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,000] jepsen worker 42 - jepsen.util 402 :invoke :write [14 11] INFO [2022-03-13 17:10:55,002] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,003] jepsen worker 26 - jepsen.util 476 :invoke :write [8 2] INFO [2022-03-13 17:10:55,006] jepsen worker 53 - jepsen.util 503 :invoke :write [17 4] INFO [2022-03-13 17:10:55,022] jepsen worker 63 - jepsen.util 1593 :ok :read [21 :nil] INFO [2022-03-13 17:10:55,024] jepsen worker 63 - jepsen.util 1593 :invoke :write [21 35] INFO [2022-03-13 17:10:55,026] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,026] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,026] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:55,027] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:55,028] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:55,030] jepsen worker 69 - jepsen.util 609 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,031] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] INFO [2022-03-13 17:10:55,032] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,032] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,032] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:10:55,033] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:10:55,036] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:55,042] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:55,042] jepsen worker 21 - jepsen.util 741 :invoke :write [7 6] INFO [2022-03-13 17:10:55,058] jepsen worker 77 - jepsen.util 707 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,058] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,059] jepsen worker 12 - jepsen.util 732 :invoke :write [4 14] INFO [2022-03-13 17:10:55,059] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,060] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:55,060] jepsen worker 51 - jepsen.util 591 :invoke :write [17 15] INFO [2022-03-13 17:10:55,094] jepsen worker 54 - jepsen.util 1224 :info :write [18 33] :RequestTimeoutException INFO [2022-03-13 17:10:55,095] jepsen worker 54 - jepsen.util 1314 :invoke :write [18 32] INFO [2022-03-13 17:10:55,160] jepsen worker 62 - jepsen.util 512 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,160] jepsen worker 25 - jepsen.util 385 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,160] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,161] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:55,161] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,161] jepsen worker 25 - jepsen.util 385 :invoke :write [8 37] INFO [2022-03-13 17:10:55,162] jepsen worker 22 - jepsen.util 562 :invoke :write [7 19] INFO [2022-03-13 17:10:55,163] jepsen worker 13 - jepsen.util 553 :invoke :write [4 37] INFO [2022-03-13 17:10:55,164] jepsen worker 62 - jepsen.util 512 :invoke :write [20 46] INFO [2022-03-13 17:10:55,168] jepsen worker 24 - jepsen.util 474 :invoke :write [8 1] INFO [2022-03-13 17:10:55,206] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,206] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:55,206] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:55,207] jepsen worker 43 - jepsen.util 403 :invoke :write [14 41] INFO [2022-03-13 17:10:55,352] jepsen worker 34 - jepsen.util 754 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,353] jepsen worker 34 - jepsen.util 754 :invoke :write [11 20] INFO [2022-03-13 17:10:55,364] jepsen worker 3 - jepsen.util 1443 :info :write [1 27] :RequestTimeoutException INFO [2022-03-13 17:10:55,365] jepsen worker 3 - jepsen.util 1533 :invoke :write [1 1] INFO [2022-03-13 17:10:55,366] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,367] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:55,387] jepsen worker 61 - jepsen.util 601 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,388] jepsen worker 61 - jepsen.util 601 :invoke :write [20 12] INFO [2022-03-13 17:10:55,437] jepsen worker 38 - jepsen.util 1838 :info :write [12 7] :RequestTimeoutException INFO [2022-03-13 17:10:55,437] jepsen worker 38 - jepsen.util 1928 :invoke :read [12 nil] INFO [2022-03-13 17:10:55,561] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:55,561] jepsen worker nemesis - couchbase.util Rebalance check count: 167 INFO [2022-03-13 17:10:55,561] jepsen worker nemesis - couchbase.util Rebalance stuck count: 100 INFO [2022-03-13 17:10:55,591] jepsen worker 80 - jepsen.util 620 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,592] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,592] jepsen worker 80 - jepsen.util 620 :invoke :write [26 24] INFO [2022-03-13 17:10:55,592] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:55,595] jepsen worker 60 - jepsen.util 600 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,595] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,595] jepsen worker 60 - jepsen.util 600 :invoke :write [20 40] INFO [2022-03-13 17:10:55,596] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:55,621] jepsen worker 18 - jepsen.util 1278 :info :write [6 4] :RequestTimeoutException INFO [2022-03-13 17:10:55,621] jepsen worker 18 - jepsen.util 1368 :invoke :write [6 11] INFO [2022-03-13 17:10:55,623] jepsen worker 32 - jepsen.util 1292 :info :write [10 43] :RequestTimeoutException INFO [2022-03-13 17:10:55,624] jepsen worker 32 - jepsen.util 1382 :invoke :read [10 nil] INFO [2022-03-13 17:10:55,772] jepsen worker 6 - jepsen.util 546 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,772] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:10:55,779] jepsen worker 64 - jepsen.util 1324 :info :write [21 30] :RequestTimeoutException INFO [2022-03-13 17:10:55,780] jepsen worker 64 - jepsen.util 1414 :invoke :write [21 6] INFO [2022-03-13 17:10:55,783] jepsen worker 14 - jepsen.util 284 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,784] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] WARN [2022-03-13 17:10:55,826] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17881816,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20716269,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9332193,"encode_duration_us":19032,"operation_id":"0x3edf","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10006194},{"operation_name":"upsert","last_dispatch_duration_us":17945408,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18358567,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9230832,"encode_duration_us":18734,"operation_id":"0x4050","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10006898},{"operation_name":"upsert","last_dispatch_duration_us":9044921,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9768607,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9465105,"encode_duration_us":18067,"operation_id":"0x422e","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10009551}],"total_count":3}} WARN [2022-03-13 17:10:55,826] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":37692,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":209129,"total_server_duration_us":97109,"encode_duration_us":23150,"operation_id":"0x464d","timeout_ms":10000,"last_server_duration_us":18799,"total_duration_us":10009934},{"operation_name":"upsert","last_dispatch_duration_us":7955984,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8024683,"total_server_duration_us":62464,"encode_duration_us":17752,"operation_id":"0x4322","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10009692},{"operation_name":"upsert","last_dispatch_duration_us":29863,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":345835,"total_server_duration_us":221758,"encode_duration_us":17831,"operation_id":"0x43de","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10008727},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":17810,"operation_id":"0x46bc","timeout_ms":10000,"total_duration_us":10007509},{"operation_name":"upsert","last_dispatch_duration_us":8171508,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":8221907,"total_server_duration_us":55054,"encode_duration_us":29839,"operation_id":"0x44e7","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10005768},{"operation_name":"upsert","last_dispatch_duration_us":9050239,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9050239,"total_server_duration_us":9028877,"encode_duration_us":16626,"operation_id":"0x464e","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":9053620},{"operation_name":"get","last_dispatch_duration_us":8270846,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8270846,"total_server_duration_us":325370,"operation_id":"0x4431","timeout_ms":30000,"last_server_duration_us":325370,"total_duration_us":8272740},{"operation_name":"upsert","last_dispatch_duration_us":649909,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":649909,"total_server_duration_us":71366,"encode_duration_us":17908,"operation_id":"0x48cc","timeout_ms":10000,"last_server_duration_us":71366,"total_duration_us":658575},{"operation_name":"upsert","last_dispatch_duration_us":634181,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":634181,"total_server_duration_us":64084,"encode_duration_us":18908,"operation_id":"0x47f2","timeout_ms":10000,"last_server_duration_us":64084,"total_duration_us":640427},{"operation_name":"upsert","last_dispatch_duration_us":552372,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":552372,"total_server_duration_us":533240,"encode_duration_us":17335,"operation_id":"0x48c4","timeout_ms":10000,"last_server_duration_us":533240,"total_duration_us":561836}],"total_count":11}} INFO [2022-03-13 17:10:55,843] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,843] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:55,844] jepsen worker 73 - jepsen.util 1513 :info :write [24 18] :RequestTimeoutException INFO [2022-03-13 17:10:55,844] jepsen worker 73 - jepsen.util 1603 :invoke :read [24 nil] INFO [2022-03-13 17:10:55,851] jepsen worker 51 - jepsen.util 591 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,852] jepsen worker 51 - jepsen.util 591 :invoke :write [17 17] INFO [2022-03-13 17:10:55,888] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,888] jepsen worker 76 - jepsen.util 616 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,889] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:55,889] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:10:55,900] jepsen worker 42 - jepsen.util 402 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,901] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:10:55,926] jepsen worker 9 - jepsen.util 1539 :ok :read [3 25] INFO [2022-03-13 17:10:55,927] jepsen worker 9 - jepsen.util 1539 :invoke :read [3 nil] INFO [2022-03-13 17:10:55,974] jepsen worker 52 - jepsen.util 502 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,975] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,975] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,975] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,975] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:55,976] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:55,976] jepsen worker 8 - jepsen.util 638 :invoke :write [2 35] INFO [2022-03-13 17:10:55,977] jepsen worker 7 - jepsen.util 457 :invoke :read [2 nil] INFO [2022-03-13 17:10:55,981] jepsen worker 21 - jepsen.util 741 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,981] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,981] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,981] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,982] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:55,983] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:10:55,984] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,984] jepsen worker 21 - jepsen.util 741 :invoke :write [7 19] INFO [2022-03-13 17:10:55,985] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:10:55,986] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:55,987] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,988] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:10:55,999] jepsen worker 44 - jepsen.util 494 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:10:55,999] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,000] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,000] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:56,001] jepsen worker 71 - jepsen.util 431 :invoke :write [23 9] INFO [2022-03-13 17:10:56,010] jepsen worker 44 - jepsen.util 494 :invoke :write [14 24] INFO [2022-03-13 17:10:56,021] jepsen worker 73 - jepsen.util 1603 :ok :read [24 24] INFO [2022-03-13 17:10:56,022] jepsen worker 73 - jepsen.util 1603 :invoke :write [24 26] INFO [2022-03-13 17:10:56,046] jepsen worker 9 - jepsen.util 1539 :ok :read [3 25] INFO [2022-03-13 17:10:56,046] jepsen worker 9 - jepsen.util 1539 :invoke :read [3 nil] INFO [2022-03-13 17:10:56,154] jepsen worker 9 - jepsen.util 1539 :ok :read [3 25] INFO [2022-03-13 17:10:56,154] jepsen worker 9 - jepsen.util 1539 :invoke :write [3 49] INFO [2022-03-13 17:10:56,196] jepsen worker 12 - jepsen.util 732 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,196] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:56,197] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:56,198] jepsen worker 23 - jepsen.util 563 :invoke :write [7 22] INFO [2022-03-13 17:10:56,199] jepsen worker 35 - jepsen.util 755 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,200] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:10:56,310] jepsen worker 66 - jepsen.util 1416 :info :write [22 31] :RequestTimeoutException INFO [2022-03-13 17:10:56,311] jepsen worker 66 - jepsen.util 1506 :invoke :read [22 nil] INFO [2022-03-13 17:10:56,380] jepsen worker 34 - jepsen.util 754 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,381] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:56,383] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,383] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,383] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,384] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:56,385] jepsen worker 6 - jepsen.util 546 :invoke :write [2 46] INFO [2022-03-13 17:10:56,386] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:10:56,391] jepsen worker 26 - jepsen.util 476 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,391] jepsen worker 26 - jepsen.util 476 :invoke :write [8 30] INFO [2022-03-13 17:10:56,451] jepsen worker 43 - jepsen.util 403 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,451] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,451] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:56,452] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:56,530] jepsen worker 30 - jepsen.util 1380 :info :write [10 34] :RequestTimeoutException INFO [2022-03-13 17:10:56,530] jepsen worker 30 - jepsen.util 1470 :invoke :read [10 nil] INFO [2022-03-13 17:10:56,566] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:56,566] jepsen worker nemesis - couchbase.util Rebalance check count: 168 INFO [2022-03-13 17:10:56,567] jepsen worker nemesis - couchbase.util Rebalance stuck count: 101 INFO [2022-03-13 17:10:56,584] jepsen worker 13 - jepsen.util 553 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,585] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:56,590] jepsen worker 21 - jepsen.util 741 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,591] jepsen worker 21 - jepsen.util 741 :invoke :write [7 36] INFO [2022-03-13 17:10:56,605] jepsen worker 53 - jepsen.util 503 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,606] jepsen worker 53 - jepsen.util 503 :invoke :write [17 37] INFO [2022-03-13 17:10:56,683] jepsen worker 60 - jepsen.util 600 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,683] jepsen worker 60 - jepsen.util 600 :invoke :write [20 10] INFO [2022-03-13 17:10:56,774] jepsen worker 19 - jepsen.util 1369 :info :write [6 17] :RequestTimeoutException INFO [2022-03-13 17:10:56,774] jepsen worker 19 - jepsen.util 1459 :invoke :write [6 24] INFO [2022-03-13 17:10:56,799] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,799] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,799] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:56,800] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:10:56,800] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,801] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:56,802] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,803] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:56,803] jepsen worker 71 - jepsen.util 431 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,804] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:10:56,816] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,816] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:10:56,817] jepsen worker 62 - jepsen.util 512 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:56,818] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:10:57,004] jepsen worker 51 - jepsen.util 591 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:10:57,005] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:10:57,005] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:57,005] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] WARN [2022-03-13 17:10:57,222] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17756124,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":17860532,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9081186,"encode_duration_us":17650,"operation_id":"0x4173","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10003741},{"operation_name":"upsert","last_dispatch_duration_us":9101826,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17214540,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9132995,"encode_duration_us":20270,"operation_id":"0x4266","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10005679},{"operation_name":"upsert","last_dispatch_duration_us":15128172,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17283725,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9058552,"encode_duration_us":24843,"operation_id":"0x4342","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10009439}],"total_count":3}} WARN [2022-03-13 17:10:57,229] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18323735,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18323735,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9058604,"encode_duration_us":24100,"operation_id":"0x4319","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10003115},{"operation_name":"upsert","last_dispatch_duration_us":9197200,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9197200,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9038428,"encode_duration_us":17842,"operation_id":"0x4351","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10010021}],"total_count":2}} WARN [2022-03-13 17:10:57,330] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10994523,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10994523,"total_server_duration_us":27,"operation_id":"0x453d","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":10994797},{"operation_name":"upsert","last_dispatch_duration_us":8276797,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9153550,"total_server_duration_us":245452,"encode_duration_us":18362,"operation_id":"0x44e8","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10010358},{"operation_name":"upsert","last_dispatch_duration_us":8283837,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9261665,"total_server_duration_us":302852,"encode_duration_us":18132,"operation_id":"0x44df","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10007478},{"operation_name":"upsert","last_dispatch_duration_us":9039685,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9841837,"total_server_duration_us":9248687,"encode_duration_us":20082,"operation_id":"0x44c3","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10005528},{"operation_name":"upsert","last_dispatch_duration_us":7956034,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":8599142,"total_server_duration_us":384030,"encode_duration_us":17526,"operation_id":"0x4491","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10004425},{"operation_name":"get","last_dispatch_duration_us":9689364,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9689364,"total_server_duration_us":9002,"operation_id":"0x45b9","timeout_ms":30000,"last_server_duration_us":9002,"total_duration_us":9691889},{"operation_name":"get","last_dispatch_duration_us":8614452,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8614452,"total_server_duration_us":9802,"operation_id":"0x462c","timeout_ms":30000,"last_server_duration_us":9802,"total_duration_us":8617030},{"operation_name":"get","last_dispatch_duration_us":7391075,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7391075,"total_server_duration_us":18953,"operation_id":"0x4694","timeout_ms":30000,"last_server_duration_us":18953,"total_duration_us":7394025},{"operation_name":"get","last_dispatch_duration_us":6577324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6577324,"total_server_duration_us":2997207,"operation_id":"0x4765","timeout_ms":30000,"last_server_duration_us":2997207,"total_duration_us":6581307},{"operation_name":"get","last_dispatch_duration_us":5185671,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5185671,"total_server_duration_us":27,"operation_id":"0x46df","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":5187497}],"total_count":31}} WARN [2022-03-13 17:10:57,331] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18251,"operation_id":"0x4486","timeout_ms":10000,"total_duration_us":10010406},{"operation_name":"upsert","last_dispatch_duration_us":33088,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":736757,"total_server_duration_us":466682,"encode_duration_us":19250,"operation_id":"0x45b3","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10009074},{"operation_name":"upsert","last_dispatch_duration_us":9059573,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9447260,"total_server_duration_us":9251268,"encode_duration_us":22039,"operation_id":"0x46d1","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10005877},{"operation_name":"upsert","last_dispatch_duration_us":39375,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8337658,"total_server_duration_us":343075,"encode_duration_us":17536,"operation_id":"0x43a9","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10005194},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":20484,"operation_id":"0x451d","timeout_ms":10000,"total_duration_us":10004511},{"operation_name":"upsert","last_dispatch_duration_us":8097650,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8959563,"total_server_duration_us":384021,"encode_duration_us":17018,"operation_id":"0x465b","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10001918},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18915,"operation_id":"0x45c2","timeout_ms":10000,"total_duration_us":10001376},{"operation_name":"upsert","last_dispatch_duration_us":34870,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8742367,"total_server_duration_us":330317,"encode_duration_us":20152,"operation_id":"0x4714","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10000450},{"operation_name":"get","last_dispatch_duration_us":1239956,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1239956,"total_server_duration_us":38,"operation_id":"0x47b7","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":5891868},{"operation_name":"upsert","last_dispatch_duration_us":1186484,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1186484,"total_server_duration_us":83268,"encode_duration_us":23944,"operation_id":"0x45fb","timeout_ms":10000,"last_server_duration_us":83268,"total_duration_us":3808847}],"total_count":71}} WARN [2022-03-13 17:10:57,361] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17982066,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19278597,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9319619,"encode_duration_us":17138,"operation_id":"0x4194","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10001695},{"operation_name":"upsert","last_dispatch_duration_us":9057501,"last_local_id":"A312307A00000005/00000000F4F37B9B","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9890405,"last_local_socket":"172.23.104.79:33502","total_server_duration_us":9547382,"encode_duration_us":37847,"operation_id":"0x42ea","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10009850},{"operation_name":"upsert","last_dispatch_duration_us":9039685,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9841837,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9248687,"encode_duration_us":20082,"operation_id":"0x44c3","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10005528}],"total_count":3}} WARN [2022-03-13 17:10:57,422] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12742122,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12742122,"total_server_duration_us":7231,"operation_id":"0x44fc","timeout_ms":30000,"last_server_duration_us":7231,"total_duration_us":12744158},{"operation_name":"get","last_dispatch_duration_us":11623093,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11623093,"total_server_duration_us":7909,"operation_id":"0x4561","timeout_ms":30000,"last_server_duration_us":7909,"total_duration_us":11625067},{"operation_name":"get","last_dispatch_duration_us":10178518,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10178518,"total_server_duration_us":2885273,"operation_id":"0x4621","timeout_ms":30000,"last_server_duration_us":2885273,"total_duration_us":10180843},{"operation_name":"upsert","last_dispatch_duration_us":13652629,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":13652629,"total_server_duration_us":21731,"encode_duration_us":16694,"operation_id":"0x4485","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10010016},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":18216,"operation_id":"0x4544","timeout_ms":10000,"total_duration_us":10007408},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":18251,"operation_id":"0x4662","timeout_ms":10000,"total_duration_us":10007368},{"operation_name":"upsert","last_dispatch_duration_us":8663346,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9081317,"total_server_duration_us":267138,"encode_duration_us":20431,"operation_id":"0x455b","timeout_ms":10000,"last_server_duration_us":23222,"total_duration_us":10005215},{"operation_name":"upsert","last_dispatch_duration_us":9053418,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9053418,"total_server_duration_us":9040551,"encode_duration_us":17022,"operation_id":"0x4554","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9057352},{"operation_name":"get","last_dispatch_duration_us":9042223,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9042223,"total_server_duration_us":43,"operation_id":"0x4495","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9043263},{"operation_name":"upsert","last_dispatch_duration_us":8448794,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":8448794,"total_server_duration_us":119973,"encode_duration_us":24585,"operation_id":"0x46e8","timeout_ms":10000,"last_server_duration_us":119973,"total_duration_us":8453174}],"total_count":34}} WARN [2022-03-13 17:10:57,429] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11058344,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11058344,"total_server_duration_us":9060729,"encode_duration_us":18573,"operation_id":"0x449f","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10010773},{"operation_name":"upsert","last_dispatch_duration_us":32845,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":1101602,"total_server_duration_us":607490,"encode_duration_us":17837,"operation_id":"0x469e","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10006787},{"operation_name":"upsert","last_dispatch_duration_us":10767107,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10767107,"total_server_duration_us":26507,"encode_duration_us":19654,"operation_id":"0x4546","timeout_ms":10000,"last_server_duration_us":26507,"total_duration_us":10004395},{"operation_name":"get","last_dispatch_duration_us":9164677,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9164677,"total_server_duration_us":100,"operation_id":"0x45dd","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":9164978},{"operation_name":"get","last_dispatch_duration_us":9122309,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9122309,"total_server_duration_us":9570,"operation_id":"0x45ed","timeout_ms":30000,"last_server_duration_us":9570,"total_duration_us":9124924},{"operation_name":"get","last_dispatch_duration_us":9099897,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9099897,"total_server_duration_us":32,"operation_id":"0x4570","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9102929},{"operation_name":"get","last_dispatch_duration_us":3519406,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":3519406,"total_server_duration_us":15,"operation_id":"0x479c","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":3522297},{"operation_name":"upsert","last_dispatch_duration_us":1658718,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1658718,"total_server_duration_us":171154,"encode_duration_us":19898,"operation_id":"0x49c6","timeout_ms":10000,"last_server_duration_us":171154,"total_duration_us":1664377},{"operation_name":"upsert","last_dispatch_duration_us":1545771,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1545771,"total_server_duration_us":22306,"encode_duration_us":16924,"operation_id":"0x49d1","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":1552491},{"operation_name":"upsert","last_dispatch_duration_us":1486142,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1486142,"total_server_duration_us":60002,"encode_duration_us":29288,"operation_id":"0x47ad","timeout_ms":10000,"last_server_duration_us":60002,"total_duration_us":1493050}],"total_count":55}} WARN [2022-03-13 17:10:57,429] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":77686,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9524839,"total_server_duration_us":169638,"encode_duration_us":18050,"operation_id":"0x46da","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10010326},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":16718,"operation_id":"0x4730","timeout_ms":10000,"total_duration_us":10009659},{"operation_name":"upsert","last_dispatch_duration_us":2670716,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9435673,"total_server_duration_us":205589,"encode_duration_us":18380,"operation_id":"0x43fa","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10009459},{"operation_name":"upsert","last_dispatch_duration_us":88009,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":88009,"total_server_duration_us":33721,"encode_duration_us":20362,"operation_id":"0x4626","timeout_ms":10000,"last_server_duration_us":33721,"total_duration_us":10009359},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17654,"operation_id":"0x46bd","timeout_ms":10000,"total_duration_us":10008867},{"operation_name":"upsert","last_dispatch_duration_us":29058,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":5692368,"total_server_duration_us":336876,"encode_duration_us":19893,"operation_id":"0x45b4","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10007839},{"operation_name":"upsert","last_dispatch_duration_us":30114,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":1062564,"total_server_duration_us":597511,"encode_duration_us":20090,"operation_id":"0x4693","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10006234},{"operation_name":"upsert","last_dispatch_duration_us":8780540,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8909960,"total_server_duration_us":50281,"encode_duration_us":20891,"operation_id":"0x4489","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10005259},{"operation_name":"upsert","last_dispatch_duration_us":32543,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2054743,"total_server_duration_us":574779,"encode_duration_us":19843,"operation_id":"0x451e","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10004576},{"operation_name":"upsert","last_dispatch_duration_us":98392,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9830791,"total_server_duration_us":182520,"encode_duration_us":18715,"operation_id":"0x467b","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10001767}],"total_count":33}} WARN [2022-03-13 17:10:57,461] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16544148,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17549578,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9514244,"encode_duration_us":22666,"operation_id":"0x40f5","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10001089},{"operation_name":"upsert","last_dispatch_duration_us":16154616,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16645679,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9338770,"encode_duration_us":19015,"operation_id":"0x4267","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10004854},{"operation_name":"upsert","last_dispatch_duration_us":9059573,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9447260,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9251268,"encode_duration_us":22039,"operation_id":"0x46d1","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10005877}],"total_count":3}} INFO [2022-03-13 17:10:57,573] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:57,573] jepsen worker nemesis - couchbase.util Rebalance check count: 169 INFO [2022-03-13 17:10:57,573] jepsen worker nemesis - couchbase.util Rebalance stuck count: 102 WARN [2022-03-13 17:10:57,630] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16745197,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17376221,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9343090,"encode_duration_us":17751,"operation_id":"0x429d","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":10007862},{"operation_name":"upsert","last_dispatch_duration_us":11058344,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11058344,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9060729,"encode_duration_us":18573,"operation_id":"0x449f","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10010773},{"operation_name":"upsert","last_dispatch_duration_us":16607561,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17279477,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9324748,"encode_duration_us":21878,"operation_id":"0x4288","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10008949}],"total_count":3}} INFO [2022-03-13 17:10:58,049] jepsen worker 37 - jepsen.util 1567 :ok :read [12 20] INFO [2022-03-13 17:10:58,050] jepsen worker 37 - jepsen.util 1567 :invoke :read [12 nil] INFO [2022-03-13 17:10:58,084] jepsen worker 89 - jepsen.util 1889 :ok :read [29 10] INFO [2022-03-13 17:10:58,085] jepsen worker 89 - jepsen.util 1889 :invoke :write [29 29] INFO [2022-03-13 17:10:58,090] jepsen worker 37 - jepsen.util 1567 :ok :read [12 20] INFO [2022-03-13 17:10:58,091] jepsen worker 37 - jepsen.util 1567 :invoke :read [12 nil] INFO [2022-03-13 17:10:58,112] jepsen worker 37 - jepsen.util 1567 :ok :read [12 20] INFO [2022-03-13 17:10:58,113] jepsen worker 37 - jepsen.util 1567 :invoke :write [12 28] INFO [2022-03-13 17:10:58,400] jepsen worker 17 - jepsen.util 1547 :info :write [5 17] :RequestTimeoutException INFO [2022-03-13 17:10:58,401] jepsen worker 17 - jepsen.util 1637 :invoke :read [5 nil] INFO [2022-03-13 17:10:58,532] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,532] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,533] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:10:58,533] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:58,550] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,550] jepsen worker 61 - jepsen.util 601 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,551] jepsen worker 61 - jepsen.util 601 :invoke :write [20 21] INFO [2022-03-13 17:10:58,551] jepsen worker 14 - jepsen.util 284 :invoke :write [4 37] INFO [2022-03-13 17:10:58,576] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,577] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:10:58,578] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3049853372434017}, :ns_1@172.23.105.187 {:progress 0.4210526315789473}, :ns_1@172.23.107.227 {:progress 0.3049853372434017}, :ns_1@172.23.107.228 {:progress 0.3294117647058824}} INFO [2022-03-13 17:10:58,579] jepsen worker nemesis - couchbase.util Rebalance check count: 170 INFO [2022-03-13 17:10:58,579] jepsen worker nemesis - couchbase.util Rebalance stuck count: 103 INFO [2022-03-13 17:10:58,593] jepsen worker 23 - jepsen.util 563 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,594] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:58,594] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,594] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:10:58,595] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:58,595] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:10:58,622] jepsen worker 22 - jepsen.util 562 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,623] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:58,657] jepsen worker 5 - jepsen.util 1805 :ok :read [1 42] INFO [2022-03-13 17:10:58,657] jepsen worker 5 - jepsen.util 1805 :invoke :write [1 16] INFO [2022-03-13 17:10:58,667] jepsen worker 38 - jepsen.util 1928 :ok :read [12 20] INFO [2022-03-13 17:10:58,668] jepsen worker 38 - jepsen.util 1928 :invoke :write [12 26] INFO [2022-03-13 17:10:58,688] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,689] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:10:58,710] jepsen worker 21 - jepsen.util 741 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,711] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:58,737] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,737] jepsen worker 44 - jepsen.util 494 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,738] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:58,738] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:58,779] jepsen worker 60 - jepsen.util 600 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,780] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,780] jepsen worker 60 - jepsen.util 600 :invoke :write [20 35] INFO [2022-03-13 17:10:58,781] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:58,784] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,785] jepsen worker 51 - jepsen.util 591 :invoke :write [17 28] INFO [2022-03-13 17:10:58,788] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:58,788] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,788] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:10:58,790] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:58,790] jepsen worker 23 - jepsen.util 563 :invoke :write [7 43] INFO [2022-03-13 17:10:58,791] jepsen worker 77 - jepsen.util 707 :invoke :write [25 17] INFO [2022-03-13 17:10:58,820] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,821] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:10:58,822] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,823] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:58,843] jepsen worker 80 - jepsen.util 620 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,843] jepsen worker 7 - jepsen.util 457 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,843] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,843] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,844] jepsen worker 80 - jepsen.util 620 :invoke :read [26 nil] INFO [2022-03-13 17:10:58,844] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:10:58,845] jepsen worker 7 - jepsen.util 457 :invoke :write [2 36] INFO [2022-03-13 17:10:58,845] jepsen worker 78 - jepsen.util 438 :invoke :read [26 nil] INFO [2022-03-13 17:10:58,871] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,871] jepsen worker 26 - jepsen.util 476 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,871] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:10:58,873] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:58,873] jepsen worker 43 - jepsen.util 403 :invoke :write [14 22] INFO [2022-03-13 17:10:58,874] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:10:58,926] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,926] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:58,928] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,929] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:58,929] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,929] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:58,930] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:10:58,931] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:58,979] jepsen worker 53 - jepsen.util 503 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,979] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:58,980] jepsen worker 62 - jepsen.util 512 :invoke :write [20 2] INFO [2022-03-13 17:10:58,980] jepsen worker 53 - jepsen.util 503 :invoke :write [17 41] INFO [2022-03-13 17:10:59,044] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,044] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,044] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,044] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:10:59,045] jepsen worker 75 - jepsen.util 435 :invoke :write [25 10] INFO [2022-03-13 17:10:59,045] jepsen worker 21 - jepsen.util 741 :invoke :write [7 26] INFO [2022-03-13 17:10:59,046] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:10:59,051] jepsen worker 8 - jepsen.util 638 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,051] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,051] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,052] jepsen worker 35 - jepsen.util 755 :invoke :write [11 38] INFO [2022-03-13 17:10:59,053] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:59,053] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:59,058] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:10:59,082] jepsen worker 60 - jepsen.util 600 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,083] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:10:59,134] jepsen worker 6 - jepsen.util 546 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,135] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,135] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,135] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,135] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:59,135] jepsen worker 6 - jepsen.util 546 :invoke :write [2 21] INFO [2022-03-13 17:10:59,136] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:59,137] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:10:59,137] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:59,138] jepsen worker 71 - jepsen.util 431 :invoke :write [23 43] INFO [2022-03-13 17:10:59,196] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,196] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:10:59,198] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,199] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:10:59,199] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:59,199] jepsen worker 44 - jepsen.util 494 :invoke :write [14 7] INFO [2022-03-13 17:10:59,216] jepsen worker 75 - jepsen.util 435 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,217] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:10:59,264] jepsen worker 25 - jepsen.util 385 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,265] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:10:59,356] jepsen worker 14 - jepsen.util 284 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,356] jepsen worker 14 - jepsen.util 284 :invoke :write [4 9] INFO [2022-03-13 17:10:59,358] jepsen worker 4 - jepsen.util 1624 :info :write [1 3] :RequestTimeoutException INFO [2022-03-13 17:10:59,358] jepsen worker 6 - jepsen.util 546 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,359] jepsen worker 6 - jepsen.util 546 :invoke :write [2 19] INFO [2022-03-13 17:10:59,359] jepsen worker 4 - jepsen.util 1714 :invoke :write [1 2] INFO [2022-03-13 17:10:59,379] jepsen worker 21 - jepsen.util 741 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,380] jepsen worker 21 - jepsen.util 741 :invoke :write [7 23] INFO [2022-03-13 17:10:59,406] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:10:59,406] jepsen worker 24 - jepsen.util 474 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,406] jepsen worker 80 - jepsen.util 620 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,406] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,406] jepsen worker 78 - jepsen.util 438 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,407] jepsen worker 80 - jepsen.util 620 :invoke :write [26 34] INFO [2022-03-13 17:10:59,407] jepsen worker 78 - jepsen.util 438 :invoke :write [26 27] INFO [2022-03-13 17:10:59,408] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:10:59,409] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:10:59,409] jepsen worker 52 - jepsen.util 502 :invoke :write [17 15] INFO [2022-03-13 17:10:59,415] jepsen worker 51 - jepsen.util 591 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,416] jepsen worker 51 - jepsen.util 591 :invoke :write [17 23] INFO [2022-03-13 17:10:59,457] jepsen worker 36 - jepsen.util 1746 :info :write [12 38] :RequestTimeoutException INFO [2022-03-13 17:10:59,458] jepsen worker 36 - jepsen.util 1836 :invoke :read [12 nil] INFO [2022-03-13 17:10:59,531] jepsen worker 71 - jepsen.util 431 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,531] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,532] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:10:59,533] jepsen worker 71 - jepsen.util 431 :invoke :write [23 18] INFO [2022-03-13 17:10:59,545] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,546] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:10:59,585] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:10:59,585] jepsen worker nemesis - couchbase.util Rebalance check count: 171 INFO [2022-03-13 17:10:59,585] jepsen worker nemesis - couchbase.util Rebalance stuck count: 103 INFO [2022-03-13 17:10:59,604] jepsen worker 7 - jepsen.util 457 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,604] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,604] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,604] jepsen worker 13 - jepsen.util 553 :invoke :read [4 nil] INFO [2022-03-13 17:10:59,605] jepsen worker 8 - jepsen.util 638 :invoke :read [2 nil] INFO [2022-03-13 17:10:59,605] jepsen worker 7 - jepsen.util 457 :invoke :write [2 25] INFO [2022-03-13 17:10:59,706] jepsen worker 61 - jepsen.util 601 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,706] jepsen worker 23 - jepsen.util 563 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,707] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:10:59,707] jepsen worker 61 - jepsen.util 601 :invoke :write [20 25] INFO [2022-03-13 17:10:59,707] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,708] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:10:59,727] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,728] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:10:59,779] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,779] jepsen worker 43 - jepsen.util 403 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,779] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,779] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,779] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:10:59,780] jepsen worker 42 - jepsen.util 402 :invoke :write [14 32] INFO [2022-03-13 17:10:59,781] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:10:59,781] jepsen worker 12 - jepsen.util 732 :invoke :write [4 39] INFO [2022-03-13 17:10:59,904] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,904] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:10:59,905] jepsen worker 53 - jepsen.util 503 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:10:59,906] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:00,050] jepsen worker 57 - jepsen.util 1677 :info :write [19 16] :RequestTimeoutException INFO [2022-03-13 17:11:00,051] jepsen worker 57 - jepsen.util 1767 :invoke :write [19 26] INFO [2022-03-13 17:11:00,074] jepsen worker 77 - jepsen.util 707 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,075] jepsen worker 77 - jepsen.util 707 :invoke :write [25 45] INFO [2022-03-13 17:11:00,092] jepsen worker 35 - jepsen.util 755 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,092] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,092] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:11:00,092] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,093] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:11:00,094] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:11:00,095] jepsen worker 25 - jepsen.util 385 :invoke :read [8 nil] INFO [2022-03-13 17:11:00,095] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:11:00,127] jepsen worker 62 - jepsen.util 512 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,128] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:00,260] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,260] jepsen worker 60 - jepsen.util 600 :invoke :write [20 43] INFO [2022-03-13 17:11:00,269] jepsen worker 44 - jepsen.util 494 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,269] jepsen worker 44 - jepsen.util 494 :invoke :write [14 8] INFO [2022-03-13 17:11:00,285] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,285] jepsen worker 21 - jepsen.util 741 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,285] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,285] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,286] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:00,287] jepsen worker 69 - jepsen.util 609 :invoke :write [23 7] INFO [2022-03-13 17:11:00,287] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,287] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:00,288] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:00,288] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:00,289] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:00,289] jepsen worker 21 - jepsen.util 741 :invoke :write [7 3] INFO [2022-03-13 17:11:00,289] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,289] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,290] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:11:00,291] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:11:00,326] jepsen worker 14 - jepsen.util 284 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,327] jepsen worker 14 - jepsen.util 284 :invoke :write [4 17] INFO [2022-03-13 17:11:00,350] jepsen worker 51 - jepsen.util 591 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,351] jepsen worker 51 - jepsen.util 591 :invoke :write [17 17] INFO [2022-03-13 17:11:00,352] jepsen worker 28 - jepsen.util 1648 :ok :read [9 26] INFO [2022-03-13 17:11:00,352] jepsen worker 28 - jepsen.util 1648 :invoke :read [9 nil] INFO [2022-03-13 17:11:00,360] jepsen worker 27 - jepsen.util 1737 :ok :read [9 26] INFO [2022-03-13 17:11:00,360] jepsen worker 27 - jepsen.util 1737 :invoke :write [9 21] INFO [2022-03-13 17:11:00,373] jepsen worker 46 - jepsen.util 1666 :ok :read [15 48] INFO [2022-03-13 17:11:00,373] jepsen worker 46 - jepsen.util 1666 :invoke :write [15 21] INFO [2022-03-13 17:11:00,387] jepsen worker 28 - jepsen.util 1648 :ok :read [9 26] INFO [2022-03-13 17:11:00,387] jepsen worker 28 - jepsen.util 1648 :invoke :read [9 nil] INFO [2022-03-13 17:11:00,398] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,398] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,399] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:00,400] jepsen worker 79 - jepsen.util 889 :invoke :write [26 9] INFO [2022-03-13 17:11:00,452] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,452] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:00,466] jepsen worker 80 - jepsen.util 620 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,467] jepsen worker 80 - jepsen.util 620 :invoke :write [26 27] INFO [2022-03-13 17:11:00,485] jepsen worker 28 - jepsen.util 1648 :ok :read [9 26] INFO [2022-03-13 17:11:00,485] jepsen worker 28 - jepsen.util 1648 :invoke :read [9 nil] INFO [2022-03-13 17:11:00,498] jepsen worker 61 - jepsen.util 601 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,499] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,499] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:00,500] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:00,500] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:11:00,500] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:11:00,501] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,502] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:11:00,509] jepsen worker 69 - jepsen.util 609 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,509] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:11:00,536] jepsen worker 28 - jepsen.util 1648 :ok :read [9 26] INFO [2022-03-13 17:11:00,536] jepsen worker 28 - jepsen.util 1648 :invoke :read [9 nil] INFO [2022-03-13 17:11:00,551] jepsen worker 74 - jepsen.util 1874 :info :write [24 23] :RequestTimeoutException INFO [2022-03-13 17:11:00,551] jepsen worker 74 - jepsen.util 1964 :invoke :write [24 38] INFO [2022-03-13 17:11:00,583] jepsen worker 28 - jepsen.util 1648 :ok :read [9 26] INFO [2022-03-13 17:11:00,583] jepsen worker 28 - jepsen.util 1648 :invoke :read [9 nil] INFO [2022-03-13 17:11:00,591] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:11:00,591] jepsen worker nemesis - couchbase.util Rebalance check count: 172 INFO [2022-03-13 17:11:00,591] jepsen worker nemesis - couchbase.util Rebalance stuck count: 104 INFO [2022-03-13 17:11:00,641] jepsen worker 28 - jepsen.util 1648 :ok :read [9 26] INFO [2022-03-13 17:11:00,642] jepsen worker 28 - jepsen.util 1648 :invoke :write [9 22] INFO [2022-03-13 17:11:00,665] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,666] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:00,669] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,669] jepsen worker 6 - jepsen.util 546 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,670] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:00,670] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:11:00,680] jepsen worker 56 - jepsen.util 1496 :info :write [18 1] :SyncWriteAmbiguous INFO [2022-03-13 17:11:00,683] jepsen worker 56 - jepsen.util 1586 :invoke :read [18 nil] INFO [2022-03-13 17:11:00,689] jepsen worker 42 - jepsen.util 402 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,689] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,689] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,689] jepsen worker 42 - jepsen.util 402 :invoke :write [14 3] INFO [2022-03-13 17:11:00,690] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:00,690] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:00,708] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,708] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:00,712] jepsen worker 21 - jepsen.util 741 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,713] jepsen worker 21 - jepsen.util 741 :invoke :write [7 13] INFO [2022-03-13 17:11:00,754] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,755] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,755] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:00,755] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:00,756] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,757] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:00,880] jepsen worker 12 - jepsen.util 732 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,881] jepsen worker 12 - jepsen.util 732 :invoke :write [4 42] INFO [2022-03-13 17:11:00,887] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,887] jepsen worker 21 - jepsen.util 741 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,887] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,888] jepsen worker 78 - jepsen.util 438 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,888] jepsen worker 13 - jepsen.util 553 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,888] jepsen worker 75 - jepsen.util 435 :invoke :write [25 20] INFO [2022-03-13 17:11:00,888] jepsen worker 8 - jepsen.util 638 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,890] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:00,891] jepsen worker 8 - jepsen.util 638 :invoke :write [2 19] INFO [2022-03-13 17:11:00,891] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:00,892] jepsen worker 78 - jepsen.util 438 :invoke :write [26 35] INFO [2022-03-13 17:11:00,892] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:00,895] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:00,895] jepsen worker 13 - jepsen.util 553 :invoke :write [4 18] INFO [2022-03-13 17:11:00,904] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:00,905] jepsen worker 21 - jepsen.util 741 :invoke :write [7 45] INFO [2022-03-13 17:11:01,026] jepsen worker 10 - jepsen.util 1450 :info :write [3 7] :RequestTimeoutException INFO [2022-03-13 17:11:01,027] jepsen worker 10 - jepsen.util 1540 :invoke :write [3 19] INFO [2022-03-13 17:11:01,076] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,076] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,076] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:11:01,077] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:01,084] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,085] jepsen worker 76 - jepsen.util 616 :invoke :write [25 0] INFO [2022-03-13 17:11:01,125] jepsen worker 71 - jepsen.util 431 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,125] jepsen worker 14 - jepsen.util 284 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,125] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,125] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,125] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,125] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,125] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,126] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:11:01,127] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:01,127] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:11:01,128] jepsen worker 61 - jepsen.util 601 :invoke :write [20 3] INFO [2022-03-13 17:11:01,131] jepsen worker 14 - jepsen.util 284 :invoke :write [4 34] INFO [2022-03-13 17:11:01,131] jepsen worker 53 - jepsen.util 503 :invoke :write [17 26] INFO [2022-03-13 17:11:01,131] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:01,307] jepsen worker 77 - jepsen.util 707 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,308] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:01,361] jepsen worker 16 - jepsen.util 1546 :ok :read [5 43] INFO [2022-03-13 17:11:01,361] jepsen worker 16 - jepsen.util 1546 :invoke :read [5 nil] INFO [2022-03-13 17:11:01,500] jepsen worker 75 - jepsen.util 435 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,501] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:01,513] jepsen worker 42 - jepsen.util 402 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,513] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,514] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:01,515] jepsen worker 26 - jepsen.util 476 :invoke :write [8 10] INFO [2022-03-13 17:11:01,515] jepsen worker 52 - jepsen.util 502 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,516] jepsen worker 52 - jepsen.util 502 :invoke :read [17 nil] INFO [2022-03-13 17:11:01,567] jepsen worker 67 - jepsen.util 1417 :ok :read [22 30] INFO [2022-03-13 17:11:01,567] jepsen worker 67 - jepsen.util 1417 :invoke :read [22 nil] INFO [2022-03-13 17:11:01,586] jepsen worker 29 - jepsen.util 1559 :info :write [9 0] :RequestTimeoutException INFO [2022-03-13 17:11:01,587] jepsen worker 29 - jepsen.util 1649 :invoke :write [9 48] INFO [2022-03-13 17:11:01,596] jepsen worker 20 - jepsen.util 1820 :info :write [6 0] :RequestTimeoutException INFO [2022-03-13 17:11:01,597] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:11:01,597] jepsen worker 20 - jepsen.util 1910 :invoke :write [6 37] INFO [2022-03-13 17:11:01,597] jepsen worker nemesis - couchbase.util Rebalance check count: 173 INFO [2022-03-13 17:11:01,597] jepsen worker nemesis - couchbase.util Rebalance stuck count: 105 INFO [2022-03-13 17:11:01,665] jepsen worker 58 - jepsen.util 1408 :ok :read [19 17] INFO [2022-03-13 17:11:01,666] jepsen worker 58 - jepsen.util 1408 :invoke :write [19 41] INFO [2022-03-13 17:11:01,686] jepsen worker 83 - jepsen.util 1073 :ok :read [27 4] INFO [2022-03-13 17:11:01,687] jepsen worker 83 - jepsen.util 1073 :invoke :write [27 23] INFO [2022-03-13 17:11:01,703] jepsen worker 81 - jepsen.util 1881 :ok :read [27 4] INFO [2022-03-13 17:11:01,704] jepsen worker 81 - jepsen.util 1881 :invoke :write [27 49] INFO [2022-03-13 17:11:01,706] jepsen worker 60 - jepsen.util 600 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,707] jepsen worker 60 - jepsen.util 600 :invoke :write [20 40] INFO [2022-03-13 17:11:01,837] jepsen worker 82 - jepsen.util 1432 :info :write [27 6] :RequestTimeoutException INFO [2022-03-13 17:11:01,838] jepsen worker 82 - jepsen.util 1522 :invoke :write [27 33] INFO [2022-03-13 17:11:01,840] jepsen worker 48 - jepsen.util 1488 :info :write [16 45] :RequestTimeoutException INFO [2022-03-13 17:11:01,841] jepsen worker 48 - jepsen.util 1578 :invoke :write [16 7] INFO [2022-03-13 17:11:01,889] jepsen worker 68 - jepsen.util 1508 :info :write [22 23] :SyncWriteAmbiguous INFO [2022-03-13 17:11:01,889] jepsen worker 68 - jepsen.util 1598 :invoke :write [22 40] INFO [2022-03-13 17:11:01,908] jepsen worker 51 - jepsen.util 591 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,909] jepsen worker 51 - jepsen.util 591 :invoke :write [17 10] INFO [2022-03-13 17:11:01,910] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,912] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:11:01,913] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:01,914] jepsen worker 43 - jepsen.util 403 :invoke :write [14 7] INFO [2022-03-13 17:11:01,920] jepsen worker 21 - jepsen.util 741 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,920] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,921] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:11:01,921] jepsen worker 21 - jepsen.util 741 :invoke :write [7 42] INFO [2022-03-13 17:11:01,924] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,924] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,924] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,925] jepsen worker 34 - jepsen.util 754 :invoke :write [11 7] INFO [2022-03-13 17:11:01,926] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:11:01,926] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,926] jepsen worker 71 - jepsen.util 431 :invoke :read [23 nil] INFO [2022-03-13 17:11:01,927] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:01,930] jepsen worker 44 - jepsen.util 494 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,931] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:11:01,932] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,933] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:11:01,933] jepsen worker 71 - jepsen.util 431 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,933] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,933] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:11:01,934] jepsen worker 44 - jepsen.util 494 :invoke :write [14 35] INFO [2022-03-13 17:11:01,934] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,935] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:01,935] jepsen worker 6 - jepsen.util 546 :invoke :write [2 38] INFO [2022-03-13 17:11:01,936] jepsen worker 71 - jepsen.util 431 :invoke :write [23 0] INFO [2022-03-13 17:11:01,945] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:01,948] jepsen worker 49 - jepsen.util 1309 :info :write [16 28] :SyncWriteAmbiguous INFO [2022-03-13 17:11:01,950] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:11:01,951] jepsen worker 49 - jepsen.util 1399 :invoke :read [16 nil] INFO [2022-03-13 17:11:02,022] jepsen worker 53 - jepsen.util 503 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,023] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:02,026] jepsen worker 87 - jepsen.util 2067 :info :write [29 3] :RequestTimeoutException INFO [2022-03-13 17:11:02,027] jepsen worker 87 - jepsen.util 2157 :invoke :read [29 nil] INFO [2022-03-13 17:11:02,055] jepsen worker 87 - jepsen.util 2157 :ok :read [29 10] INFO [2022-03-13 17:11:02,055] jepsen worker 87 - jepsen.util 2157 :invoke :read [29 nil] INFO [2022-03-13 17:11:02,078] jepsen worker 87 - jepsen.util 2157 :ok :read [29 10] INFO [2022-03-13 17:11:02,079] jepsen worker 87 - jepsen.util 2157 :invoke :write [29 6] INFO [2022-03-13 17:11:02,128] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,128] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,131] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:11:02,132] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:02,340] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:11:02,340] jepsen worker 25 - jepsen.util 385 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,340] jepsen worker 7 - jepsen.util 457 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,468] jepsen worker 7 - jepsen.util 457 :invoke :write [2 12] INFO [2022-03-13 17:11:02,340] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,468] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:11:02,340] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,342] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:11:02,471] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:11:02,343] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,357] jepsen worker 21 - jepsen.util 741 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,468] jepsen worker 25 - jepsen.util 385 :invoke :write [8 37] INFO [2022-03-13 17:11:02,471] jepsen worker 21 - jepsen.util 741 :invoke :write [7 49] INFO [2022-03-13 17:11:02,471] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] INFO [2022-03-13 17:11:02,606] jepsen worker 50 - jepsen.util 1580 :info :write [16 32] :RequestTimeoutException INFO [2022-03-13 17:11:02,606] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:11:02,606] jepsen worker nemesis - couchbase.util Rebalance check count: 174 INFO [2022-03-13 17:11:02,606] jepsen worker nemesis - couchbase.util Rebalance stuck count: 106 INFO [2022-03-13 17:11:02,607] jepsen worker 50 - jepsen.util 1670 :invoke :read [16 nil] INFO [2022-03-13 17:11:02,686] jepsen worker 39 - jepsen.util 1569 :info :write [13 20] :RequestTimeoutException INFO [2022-03-13 17:11:02,687] jepsen worker 39 - jepsen.util 1659 :invoke :read [13 nil] INFO [2022-03-13 17:11:02,705] jepsen worker 86 - jepsen.util 1346 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,706] jepsen worker 86 - jepsen.util 1346 :invoke :read [28 nil] INFO [2022-03-13 17:11:02,706] jepsen worker 59 - jepsen.util 1409 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,706] jepsen worker 59 - jepsen.util 1409 :invoke :read [19 nil] INFO [2022-03-13 17:11:02,755] jepsen worker 34 - jepsen.util 754 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,756] jepsen worker 34 - jepsen.util 754 :invoke :write [11 38] INFO [2022-03-13 17:11:02,764] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,764] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,765] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:02,765] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:02,776] jepsen worker 84 - jepsen.util 1794 :info :write [28 2] :RequestTimeoutException INFO [2022-03-13 17:11:02,777] jepsen worker 1 - jepsen.util 1351 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,779] jepsen worker 84 - jepsen.util 1884 :invoke :read [28 nil] INFO [2022-03-13 17:11:02,779] jepsen worker 1 - jepsen.util 1351 :invoke :read [0 nil] INFO [2022-03-13 17:11:02,782] jepsen worker 50 - jepsen.util 1670 :ok :read [16 :nil] INFO [2022-03-13 17:11:02,783] jepsen worker 50 - jepsen.util 1670 :invoke :write [16 41] INFO [2022-03-13 17:11:02,783] jepsen worker 1 - jepsen.util 1351 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,784] jepsen worker 1 - jepsen.util 1351 :invoke :read [0 nil] INFO [2022-03-13 17:11:02,785] jepsen worker 1 - jepsen.util 1351 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,786] jepsen worker 1 - jepsen.util 1351 :invoke :write [0 44] INFO [2022-03-13 17:11:02,813] jepsen worker 61 - jepsen.util 601 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,813] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:02,819] jepsen worker 40 - jepsen.util 1570 :ok :read [13 39] INFO [2022-03-13 17:11:02,820] jepsen worker 40 - jepsen.util 1570 :invoke :write [13 44] INFO [2022-03-13 17:11:02,952] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,953] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:02,956] jepsen worker 79 - jepsen.util 889 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,957] jepsen worker 79 - jepsen.util 889 :invoke :write [26 1] INFO [2022-03-13 17:11:02,957] jepsen worker 80 - jepsen.util 620 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:02,958] jepsen worker 80 - jepsen.util 620 :invoke :write [26 35] INFO [2022-03-13 17:11:02,986] jepsen worker 88 - jepsen.util 1528 :info :write [29 30] :RequestTimeoutException INFO [2022-03-13 17:11:02,987] jepsen worker 88 - jepsen.util 1618 :invoke :write [29 17] INFO [2022-03-13 17:11:02,994] jepsen worker 66 - jepsen.util 1506 :ok :read [22 30] INFO [2022-03-13 17:11:02,994] jepsen worker 66 - jepsen.util 1506 :invoke :write [22 27] INFO [2022-03-13 17:11:03,006] jepsen worker 15 - jepsen.util 1635 :info :write [5 12] :RequestTimeoutException INFO [2022-03-13 17:11:03,007] jepsen worker 15 - jepsen.util 1725 :invoke :write [5 12] INFO [2022-03-13 17:11:03,016] jepsen worker 2 - jepsen.util 1802 :info :write [0 5] :RequestTimeoutException INFO [2022-03-13 17:11:03,017] jepsen worker 2 - jepsen.util 1892 :invoke :write [0 30] INFO [2022-03-13 17:11:03,151] jepsen worker 39 - jepsen.util 1659 :ok :read [13 39] INFO [2022-03-13 17:11:03,151] jepsen worker 39 - jepsen.util 1659 :invoke :read [13 nil] INFO [2022-03-13 17:11:03,176] jepsen worker 45 - jepsen.util 1755 :info :write [15 34] :SyncWriteAmbiguous INFO [2022-03-13 17:11:03,176] jepsen worker 45 - jepsen.util 1845 :invoke :write [15 10] INFO [2022-03-13 17:11:03,286] jepsen worker 32 - jepsen.util 1382 :ok :read [10 :nil] INFO [2022-03-13 17:11:03,289] jepsen worker 32 - jepsen.util 1382 :invoke :write [10 37] INFO [2022-03-13 17:11:03,361] jepsen worker 21 - jepsen.util 741 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,361] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,361] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,361] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,362] jepsen worker 12 - jepsen.util 732 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,362] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,362] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:03,363] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:03,364] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,364] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:03,364] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,365] jepsen worker 69 - jepsen.util 609 :invoke :write [23 9] INFO [2022-03-13 17:11:03,367] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:03,368] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:03,368] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,368] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:03,369] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:03,369] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:03,369] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,370] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:03,371] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:03,371] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:03,371] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:03,372] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,372] jepsen worker 21 - jepsen.util 741 :invoke :write [7 29] INFO [2022-03-13 17:11:03,372] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,373] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:03,373] jepsen worker 33 - jepsen.util 663 :invoke :write [11 13] INFO [2022-03-13 17:11:03,375] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,375] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:11:03,420] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:03,420] jepsen worker 14 - jepsen.util 284 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,421] jepsen worker 42 - jepsen.util 402 :invoke :write [14 8] INFO [2022-03-13 17:11:03,421] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:03,464] jepsen worker 39 - jepsen.util 1659 :ok :read [13 39] INFO [2022-03-13 17:11:03,464] jepsen worker 39 - jepsen.util 1659 :invoke :write [13 15] INFO [2022-03-13 17:11:03,612] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:11:03,612] jepsen worker nemesis - couchbase.util Rebalance check count: 175 INFO [2022-03-13 17:11:03,613] jepsen worker nemesis - couchbase.util Rebalance stuck count: 107 INFO [2022-03-13 17:11:03,688] jepsen worker 0 - jepsen.util 1530 :info :write [0 16] :RequestTimeoutException INFO [2022-03-13 17:11:03,688] jepsen worker 0 - jepsen.util 1620 :invoke :read [0 nil] INFO [2022-03-13 17:11:03,728] jepsen worker 41 - jepsen.util 1301 :info :write [13 38] :RequestTimeoutException INFO [2022-03-13 17:11:03,728] jepsen worker 41 - jepsen.util 1391 :invoke :write [13 2] INFO [2022-03-13 17:11:03,882] jepsen worker 59 - jepsen.util 1409 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,883] jepsen worker 59 - jepsen.util 1409 :invoke :read [19 nil] INFO [2022-03-13 17:11:03,890] jepsen worker 86 - jepsen.util 1346 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,891] jepsen worker 86 - jepsen.util 1346 :invoke :read [28 nil] INFO [2022-03-13 17:11:03,979] jepsen worker 6 - jepsen.util 546 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,980] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:11:03,984] jepsen worker 69 - jepsen.util 609 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:03,985] jepsen worker 69 - jepsen.util 609 :invoke :write [23 37] INFO [2022-03-13 17:11:03,988] jepsen worker 56 - jepsen.util 1586 :ok :read [18 21] INFO [2022-03-13 17:11:03,989] jepsen worker 56 - jepsen.util 1586 :invoke :read [18 nil] INFO [2022-03-13 17:11:04,011] jepsen worker 65 - jepsen.util 1775 :info :write [21 1] :RequestTimeoutException INFO [2022-03-13 17:11:04,012] jepsen worker 65 - jepsen.util 1865 :invoke :write [21 45] INFO [2022-03-13 17:11:04,038] jepsen worker 56 - jepsen.util 1586 :ok :read [18 21] INFO [2022-03-13 17:11:04,039] jepsen worker 56 - jepsen.util 1586 :invoke :read [18 nil] INFO [2022-03-13 17:11:04,061] jepsen worker 86 - jepsen.util 1346 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,061] jepsen worker 59 - jepsen.util 1409 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,062] jepsen worker 86 - jepsen.util 1346 :invoke :write [28 25] INFO [2022-03-13 17:11:04,062] jepsen worker 59 - jepsen.util 1409 :invoke :write [19 5] INFO [2022-03-13 17:11:04,168] jepsen worker 0 - jepsen.util 1620 :ok :read [0 41] INFO [2022-03-13 17:11:04,169] jepsen worker 0 - jepsen.util 1620 :invoke :write [0 1] INFO [2022-03-13 17:11:04,250] jepsen worker 56 - jepsen.util 1586 :ok :read [18 21] INFO [2022-03-13 17:11:04,251] jepsen worker 56 - jepsen.util 1586 :invoke :write [18 20] INFO [2022-03-13 17:11:04,296] jepsen worker 55 - jepsen.util 1495 :info :write [18 20] :RequestTimeoutException INFO [2022-03-13 17:11:04,297] jepsen worker 55 - jepsen.util 1585 :invoke :write [18 12] INFO [2022-03-13 17:11:04,384] jepsen worker 8 - jepsen.util 638 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,385] jepsen worker 8 - jepsen.util 638 :invoke :write [2 12] INFO [2022-03-13 17:11:04,385] jepsen worker 60 - jepsen.util 600 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,386] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:04,574] jepsen worker 85 - jepsen.util 1615 :info :write [28 37] :RequestTimeoutException INFO [2022-03-13 17:11:04,575] jepsen worker 85 - jepsen.util 1705 :invoke :write [28 41] INFO [2022-03-13 17:11:04,592] jepsen worker 21 - jepsen.util 741 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,593] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:04,596] jepsen worker 76 - jepsen.util 616 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,597] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:11:04,619] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:11:04,620] jepsen worker nemesis - couchbase.util Rebalance check count: 176 INFO [2022-03-13 17:11:04,620] jepsen worker nemesis - couchbase.util Rebalance stuck count: 108 INFO [2022-03-13 17:11:04,744] jepsen worker 31 - jepsen.util 1471 :info :write [10 17] :RequestTimeoutException INFO [2022-03-13 17:11:04,744] jepsen worker 31 - jepsen.util 1561 :invoke :write [10 3] INFO [2022-03-13 17:11:04,792] jepsen worker 26 - jepsen.util 476 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,792] jepsen worker 51 - jepsen.util 591 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,792] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,792] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,793] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:04,793] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:11:04,793] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:04,794] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:04,794] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,794] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:04,795] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:04,795] jepsen worker 61 - jepsen.util 601 :invoke :write [20 27] INFO [2022-03-13 17:11:04,796] jepsen worker 72 - jepsen.util 1512 :info :write [24 36] :RequestTimeoutException INFO [2022-03-13 17:11:04,797] jepsen worker 72 - jepsen.util 1602 :invoke :write [24 45] INFO [2022-03-13 17:11:04,804] jepsen worker 47 - jepsen.util 1667 :info :write [15 33] :RequestTimeoutException INFO [2022-03-13 17:11:04,805] jepsen worker 47 - jepsen.util 1757 :invoke :write [15 7] INFO [2022-03-13 17:11:04,906] jepsen worker 11 - jepsen.util 1541 :info :write [3 46] :RequestTimeoutException INFO [2022-03-13 17:11:04,907] jepsen worker 11 - jepsen.util 1631 :invoke :write [3 41] INFO [2022-03-13 17:11:05,064] jepsen worker 63 - jepsen.util 1593 :info :write [21 35] :RequestTimeoutException INFO [2022-03-13 17:11:05,064] jepsen worker 63 - jepsen.util 1683 :invoke :read [21 nil] INFO [2022-03-13 17:11:05,117] jepsen worker 30 - jepsen.util 1470 :ok :read [10 :nil] INFO [2022-03-13 17:11:05,117] jepsen worker 30 - jepsen.util 1470 :invoke :write [10 5] INFO [2022-03-13 17:11:05,120] jepsen worker 54 - jepsen.util 1314 :info :write [18 32] :RequestTimeoutException INFO [2022-03-13 17:11:05,121] jepsen worker 54 - jepsen.util 1404 :invoke :read [18 nil] INFO [2022-03-13 17:11:05,201] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:05,201] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:11:05,218] jepsen worker 54 - jepsen.util 1404 :ok :read [18 21] INFO [2022-03-13 17:11:05,218] jepsen worker 54 - jepsen.util 1404 :invoke :write [18 41] INFO [2022-03-13 17:11:05,398] jepsen worker 3 - jepsen.util 1533 :info :write [1 1] :RequestTimeoutException INFO [2022-03-13 17:11:05,399] jepsen worker 3 - jepsen.util 1623 :invoke :read [1 nil] INFO [2022-03-13 17:11:05,408] jepsen worker 79 - jepsen.util 889 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:05,408] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:05,425] jepsen worker 33 - jepsen.util 663 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:05,426] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:11:05,625] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:11:05,625] jepsen worker nemesis - couchbase.util Rebalance check count: 177 INFO [2022-03-13 17:11:05,625] jepsen worker nemesis - couchbase.util Rebalance stuck count: 109 INFO [2022-03-13 17:11:05,644] jepsen worker 18 - jepsen.util 1368 :info :write [6 11] :RequestTimeoutException INFO [2022-03-13 17:11:05,645] jepsen worker 18 - jepsen.util 1458 :invoke :read [6 nil] WARN [2022-03-13 17:11:05,743] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14571942,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":14571942,"total_server_duration_us":100,"operation_id":"0x45e0","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":14572950},{"operation_name":"upsert","last_dispatch_duration_us":16438052,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":16438052,"total_server_duration_us":29415,"encode_duration_us":20873,"operation_id":"0x46fa","timeout_ms":10000,"last_server_duration_us":29415,"total_duration_us":10010237},{"operation_name":"upsert","last_dispatch_duration_us":14379976,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":14379976,"total_server_duration_us":20760,"encode_duration_us":17053,"operation_id":"0x47b6","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10007902},{"operation_name":"upsert","last_dispatch_duration_us":57091,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6372695,"total_server_duration_us":362986,"encode_duration_us":16887,"operation_id":"0x4881","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10005710},{"operation_name":"upsert","last_dispatch_duration_us":39012,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1894054,"total_server_duration_us":364167,"encode_duration_us":19819,"operation_id":"0x4a23","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10004473},{"operation_name":"get","last_dispatch_duration_us":9605039,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9605039,"total_server_duration_us":43,"operation_id":"0x4976","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9605318},{"operation_name":"get","last_dispatch_duration_us":8688864,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8688864,"total_server_duration_us":38,"operation_id":"0x4752","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":8690825},{"operation_name":"get","last_dispatch_duration_us":6663207,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":6663207,"total_server_duration_us":32,"operation_id":"0x48bf","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6664313},{"operation_name":"upsert","last_dispatch_duration_us":3574577,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3574577,"total_server_duration_us":115449,"encode_duration_us":18370,"operation_id":"0x4aae","timeout_ms":10000,"last_server_duration_us":115449,"total_duration_us":3579263},{"operation_name":"upsert","last_dispatch_duration_us":3486320,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3486320,"total_server_duration_us":607277,"encode_duration_us":18975,"operation_id":"0x4bd5","timeout_ms":10000,"last_server_duration_us":607277,"total_duration_us":3491665}],"total_count":19}} WARN [2022-03-13 17:11:05,744] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17216336,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":17903516,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9441921,"encode_duration_us":18600,"operation_id":"0x42ee","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10003773},{"operation_name":"upsert","last_dispatch_duration_us":16945745,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17291581,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9246392,"encode_duration_us":17831,"operation_id":"0x43de","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10008727},{"operation_name":"upsert","last_dispatch_duration_us":17867552,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18076681,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9144030,"encode_duration_us":23150,"operation_id":"0x464d","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10009934}],"total_count":3}} INFO [2022-03-13 17:11:05,806] jepsen worker 64 - jepsen.util 1414 :info :write [21 6] :RequestTimeoutException INFO [2022-03-13 17:11:05,807] jepsen worker 64 - jepsen.util 1504 :invoke :read [21 nil] INFO [2022-03-13 17:11:05,812] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:05,812] jepsen worker 69 - jepsen.util 609 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:05,813] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:05,813] jepsen worker 69 - jepsen.util 609 :invoke :write [23 23] INFO [2022-03-13 17:11:05,987] jepsen worker 64 - jepsen.util 1504 :ok :read [21 :nil] INFO [2022-03-13 17:11:05,988] jepsen worker 64 - jepsen.util 1504 :invoke :write [21 22] INFO [2022-03-13 17:11:06,036] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,037] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:11:06,061] jepsen worker 73 - jepsen.util 1603 :info :write [24 26] :RequestTimeoutException INFO [2022-03-13 17:11:06,061] jepsen worker 73 - jepsen.util 1693 :invoke :write [24 34] INFO [2022-03-13 17:11:06,180] jepsen worker 9 - jepsen.util 1539 :info :write [3 49] :RequestTimeoutException INFO [2022-03-13 17:11:06,181] jepsen worker 9 - jepsen.util 1629 :invoke :read [3 nil] INFO [2022-03-13 17:11:06,425] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,425] jepsen worker 42 - jepsen.util 402 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,425] jepsen worker 43 - jepsen.util 403 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,425] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,425] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,425] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:11:06,426] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:06,427] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:06,427] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:06,428] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:06,631] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4269005847953217}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3352941176470589}} INFO [2022-03-13 17:11:06,631] jepsen worker nemesis - couchbase.util Rebalance check count: 178 INFO [2022-03-13 17:11:06,631] jepsen worker nemesis - couchbase.util Rebalance stuck count: 110 INFO [2022-03-13 17:11:06,793] jepsen worker 19 - jepsen.util 1459 :info :write [6 24] :RequestTimeoutException INFO [2022-03-13 17:11:06,794] jepsen worker 19 - jepsen.util 1549 :invoke :write [6 18] INFO [2022-03-13 17:11:06,832] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,832] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,833] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,833] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:06,833] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,833] jepsen worker 76 - jepsen.util 616 :invoke :write [25 23] INFO [2022-03-13 17:11:06,833] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,834] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:11:06,835] jepsen worker 62 - jepsen.util 512 :invoke :write [20 21] INFO [2022-03-13 17:11:06,837] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,838] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,839] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:11:06,839] jepsen worker 12 - jepsen.util 732 :invoke :write [4 35] INFO [2022-03-13 17:11:06,840] jepsen worker 43 - jepsen.util 403 :invoke :read [14 nil] INFO [2022-03-13 17:11:06,841] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:06,841] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:06,842] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:06,842] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:07,036] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,037] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:07,040] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,040] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:11:07,040] jepsen worker 71 - jepsen.util 431 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,041] jepsen worker 71 - jepsen.util 431 :invoke :write [23 8] INFO [2022-03-13 17:11:07,041] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:07,042] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:07,049] jepsen worker 43 - jepsen.util 403 :ok :read [14 :nil] INFO [2022-03-13 17:11:07,049] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:11:07,049] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,050] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,050] jepsen worker 43 - jepsen.util 403 :invoke :write [14 48] INFO [2022-03-13 17:11:07,050] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:07,051] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:07,056] jepsen worker 51 - jepsen.util 591 :invoke :write [17 27] INFO [2022-03-13 17:11:07,111] jepsen worker 13 - jepsen.util 553 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,112] jepsen worker 13 - jepsen.util 553 :invoke :write [4 1] INFO [2022-03-13 17:11:07,182] jepsen worker 37 - jepsen.util 1567 :info :write [12 28] :SyncWriteAmbiguous INFO [2022-03-13 17:11:07,182] jepsen worker 37 - jepsen.util 1657 :invoke :read [12 nil] WARN [2022-03-13 17:11:07,238] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9092134,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9620172,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9341602,"encode_duration_us":18682,"operation_id":"0x4806","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10004899},{"operation_name":"upsert","last_dispatch_duration_us":15404041,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15404041,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9052231,"encode_duration_us":18251,"operation_id":"0x4662","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10007368},{"operation_name":"upsert","last_dispatch_duration_us":20892408,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20892408,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9047983,"encode_duration_us":18216,"operation_id":"0x4544","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10007408}],"total_count":3}} INFO [2022-03-13 17:11:07,241] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:07,241] jepsen worker 61 - jepsen.util 601 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,241] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,241] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,242] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:07,242] jepsen worker 42 - jepsen.util 402 :invoke :write [14 39] INFO [2022-03-13 17:11:07,243] jepsen worker 61 - jepsen.util 601 :invoke :write [20 37] INFO [2022-03-13 17:11:07,243] jepsen worker 26 - jepsen.util 476 :invoke :write [8 33] INFO [2022-03-13 17:11:07,243] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,244] jepsen worker 14 - jepsen.util 284 :invoke :write [4 27] WARN [2022-03-13 17:11:07,265] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17908762,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27657023,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9203956,"encode_duration_us":20025,"operation_id":"0x432c","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10002787},{"operation_name":"upsert","last_dispatch_duration_us":17942879,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17942879,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9039489,"encode_duration_us":17654,"operation_id":"0x46bd","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10008867},{"operation_name":"upsert","last_dispatch_duration_us":9053915,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14746284,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9379550,"encode_duration_us":19893,"operation_id":"0x45b4","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10007839},{"operation_name":"upsert","last_dispatch_duration_us":9211379,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9211379,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9036305,"encode_duration_us":16718,"operation_id":"0x4730","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10009659}],"total_count":4}} WARN [2022-03-13 17:11:07,374] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":69221,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":257275,"total_server_duration_us":105487,"encode_duration_us":20498,"operation_id":"0x49cc","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10010142},{"operation_name":"upsert","last_dispatch_duration_us":33028,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":974613,"total_server_duration_us":572890,"encode_duration_us":17489,"operation_id":"0x48f7","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10009152},{"operation_name":"upsert","last_dispatch_duration_us":8494697,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8494697,"total_server_duration_us":23813,"encode_duration_us":17091,"operation_id":"0x49ff","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10008074},{"operation_name":"upsert","last_dispatch_duration_us":30048,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":973999,"total_server_duration_us":597083,"encode_duration_us":21238,"operation_id":"0x4960","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10007951},{"operation_name":"upsert","last_dispatch_duration_us":8433675,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8939181,"total_server_duration_us":242898,"encode_duration_us":21513,"operation_id":"0x494d","timeout_ms":10000,"last_server_duration_us":26153,"total_duration_us":10007694},{"operation_name":"upsert","last_dispatch_duration_us":39170,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":372945,"total_server_duration_us":181815,"encode_duration_us":24811,"operation_id":"0x4965","timeout_ms":10000,"last_server_duration_us":27312,"total_duration_us":10003429},{"operation_name":"upsert","last_dispatch_duration_us":8614013,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":12014344,"total_server_duration_us":207755,"encode_duration_us":31632,"operation_id":"0x48a8","timeout_ms":10000,"last_server_duration_us":26064,"total_duration_us":10002045},{"operation_name":"upsert","last_dispatch_duration_us":9128199,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9179579,"total_server_duration_us":9061060,"encode_duration_us":22579,"operation_id":"0x4949","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":9191097},{"operation_name":"upsert","last_dispatch_duration_us":9102261,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9102261,"total_server_duration_us":9027816,"encode_duration_us":18613,"operation_id":"0x494f","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":9105980},{"operation_name":"upsert","last_dispatch_duration_us":9042053,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9042053,"total_server_duration_us":9025695,"encode_duration_us":16869,"operation_id":"0x49c7","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9047800}],"total_count":40}} WARN [2022-03-13 17:11:07,376] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17413024,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17413024,"total_server_duration_us":11614,"operation_id":"0x4622","timeout_ms":30000,"last_server_duration_us":11614,"total_duration_us":17415889},{"operation_name":"get","last_dispatch_duration_us":16567387,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16567387,"total_server_duration_us":202112,"operation_id":"0x4669","timeout_ms":30000,"last_server_duration_us":202112,"total_duration_us":16569728},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":20937,"operation_id":"0x485a","timeout_ms":10000,"total_duration_us":10006632},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":17897,"operation_id":"0x48f5","timeout_ms":10000,"total_duration_us":10004684},{"operation_name":"upsert","last_dispatch_duration_us":9101485,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11080133,"total_server_duration_us":9154495,"encode_duration_us":20604,"operation_id":"0x49c1","timeout_ms":10000,"last_server_duration_us":9080926,"total_duration_us":10002595},{"operation_name":"get","last_dispatch_duration_us":8585194,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8585194,"total_server_duration_us":43,"operation_id":"0x4a8b","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8585498},{"operation_name":"get","last_dispatch_duration_us":7908825,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":7908825,"total_server_duration_us":253746,"operation_id":"0x4897","timeout_ms":30000,"last_server_duration_us":253746,"total_duration_us":7910900},{"operation_name":"upsert","last_dispatch_duration_us":6189831,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":6189831,"total_server_duration_us":63955,"encode_duration_us":19503,"operation_id":"0x4bcf","timeout_ms":10000,"last_server_duration_us":63955,"total_duration_us":6195825},{"operation_name":"upsert","last_dispatch_duration_us":2601106,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2601106,"total_server_duration_us":27043,"encode_duration_us":27005,"operation_id":"0x4a27","timeout_ms":10000,"last_server_duration_us":27043,"total_duration_us":4210210},{"operation_name":"upsert","last_dispatch_duration_us":2467864,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2467864,"total_server_duration_us":26774,"encode_duration_us":30567,"operation_id":"0x4a26","timeout_ms":10000,"last_server_duration_us":26774,"total_duration_us":4072050}],"total_count":42}} INFO [2022-03-13 17:11:07,411] jepsen worker 89 - jepsen.util 1889 :info :write [29 29] :SyncWriteAmbiguous INFO [2022-03-13 17:11:07,416] jepsen worker 89 - jepsen.util 1979 :invoke :write [29 20] INFO [2022-03-13 17:11:07,437] jepsen worker 36 - jepsen.util 1836 :ok :read [12 20] INFO [2022-03-13 17:11:07,438] jepsen worker 36 - jepsen.util 1836 :invoke :read [12 nil] WARN [2022-03-13 17:11:07,438] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":20185526,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":20185526,"total_server_duration_us":8284,"operation_id":"0x458a","timeout_ms":30000,"last_server_duration_us":8284,"total_duration_us":20185800},{"operation_name":"get","last_dispatch_duration_us":14674619,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":14674619,"total_server_duration_us":9002,"operation_id":"0x46ca","timeout_ms":30000,"last_server_duration_us":9002,"total_duration_us":14677591},{"operation_name":"get","last_dispatch_duration_us":12726053,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12726053,"total_server_duration_us":145,"operation_id":"0x4855","timeout_ms":30000,"last_server_duration_us":145,"total_duration_us":12726299},{"operation_name":"upsert","last_dispatch_duration_us":10944602,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10944602,"total_server_duration_us":56157,"encode_duration_us":20150,"operation_id":"0x48f4","timeout_ms":10000,"last_server_duration_us":56157,"total_duration_us":10008132},{"operation_name":"upsert","last_dispatch_duration_us":8474275,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9047721,"total_server_duration_us":305815,"encode_duration_us":17349,"operation_id":"0x4814","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10005270},{"operation_name":"upsert","last_dispatch_duration_us":9092134,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9620172,"total_server_duration_us":9341602,"encode_duration_us":18682,"operation_id":"0x4806","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10004899},{"operation_name":"upsert","last_dispatch_duration_us":461803,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10303042,"total_server_duration_us":134588,"encode_duration_us":19349,"operation_id":"0x49ac","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10003825},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","encode_duration_us":29684,"operation_id":"0x4a2d","timeout_ms":10000,"total_duration_us":10002617},{"operation_name":"upsert","last_dispatch_duration_us":245735,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9762649,"total_server_duration_us":98475,"encode_duration_us":17705,"operation_id":"0x49ad","timeout_ms":10000,"last_server_duration_us":36423,"total_duration_us":10001313},{"operation_name":"get","last_dispatch_duration_us":9704232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9704232,"total_server_duration_us":38,"operation_id":"0x4894","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9706408}],"total_count":46}} INFO [2022-03-13 17:11:07,441] jepsen worker 3 - jepsen.util 1623 :ok :read [1 42] INFO [2022-03-13 17:11:07,441] jepsen worker 3 - jepsen.util 1623 :invoke :write [1 36] INFO [2022-03-13 17:11:07,444] jepsen worker 69 - jepsen.util 609 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,444] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] WARN [2022-03-13 17:11:07,446] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17225129,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17225129,"total_server_duration_us":9341,"operation_id":"0x463a","timeout_ms":30000,"last_server_duration_us":9341,"total_duration_us":17227392},{"operation_name":"get","last_dispatch_duration_us":15396780,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":15396780,"total_server_duration_us":11118,"operation_id":"0x46f8","timeout_ms":30000,"last_server_duration_us":11118,"total_duration_us":15398977},{"operation_name":"get","last_dispatch_duration_us":10291091,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10291091,"total_server_duration_us":19,"operation_id":"0x48be","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":10293268},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":36567,"operation_id":"0x4964","timeout_ms":10000,"total_duration_us":10010923},{"operation_name":"upsert","last_dispatch_duration_us":87900,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9874028,"total_server_duration_us":181949,"encode_duration_us":18811,"operation_id":"0x48de","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10010125},{"operation_name":"upsert","last_dispatch_duration_us":71477,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7091237,"total_server_duration_us":344614,"encode_duration_us":18304,"operation_id":"0x4a11","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10008640},{"operation_name":"upsert","last_dispatch_duration_us":204668,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":6475362,"total_server_duration_us":411737,"encode_duration_us":18377,"operation_id":"0x4a5c","timeout_ms":10000,"last_server_duration_us":77711,"total_duration_us":10007166},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":27722,"operation_id":"0x4947","timeout_ms":10000,"total_duration_us":10004183},{"operation_name":"upsert","last_dispatch_duration_us":90340,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9885335,"total_server_duration_us":185180,"encode_duration_us":19352,"operation_id":"0x48e5","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10003599},{"operation_name":"upsert","last_dispatch_duration_us":9116066,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9116066,"total_server_duration_us":31012,"encode_duration_us":19926,"operation_id":"0x4938","timeout_ms":10000,"last_server_duration_us":31012,"total_duration_us":10001017}],"total_count":30}} INFO [2022-03-13 17:11:07,449] jepsen worker 76 - jepsen.util 616 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,449] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,450] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,451] jepsen worker 79 - jepsen.util 889 :invoke :write [26 45] INFO [2022-03-13 17:11:07,451] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:07,452] jepsen worker 76 - jepsen.util 616 :invoke :write [25 40] INFO [2022-03-13 17:11:07,456] jepsen worker 37 - jepsen.util 1657 :ok :read [12 20] INFO [2022-03-13 17:11:07,456] jepsen worker 37 - jepsen.util 1657 :invoke :write [12 40] WARN [2022-03-13 17:11:07,466] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":112444,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8914880,"total_server_duration_us":192867,"encode_duration_us":17132,"operation_id":"0x49fb","timeout_ms":10000,"last_server_duration_us":24325,"total_duration_us":10007593},{"operation_name":"upsert","last_dispatch_duration_us":57779,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9297422,"total_server_duration_us":243920,"encode_duration_us":34928,"operation_id":"0x4a1e","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10004560},{"operation_name":"upsert","last_dispatch_duration_us":37031,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7227239,"total_server_duration_us":271144,"encode_duration_us":17288,"operation_id":"0x49e9","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10004490},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":20573,"operation_id":"0x4a53","timeout_ms":10000,"total_duration_us":10001506},{"operation_name":"upsert","last_dispatch_duration_us":147336,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8983656,"total_server_duration_us":187875,"encode_duration_us":20046,"operation_id":"0x4a03","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10001484},{"operation_name":"upsert","last_dispatch_duration_us":444028,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":691368,"total_server_duration_us":81002,"encode_duration_us":17059,"operation_id":"0x4aa5","timeout_ms":10000,"last_server_duration_us":40163,"total_duration_us":10001464},{"operation_name":"get","last_dispatch_duration_us":155435,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":155435,"total_server_duration_us":35211,"operation_id":"0x4bba","timeout_ms":30000,"last_server_duration_us":35211,"total_duration_us":3287196},{"operation_name":"upsert","last_dispatch_duration_us":3253975,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3253975,"total_server_duration_us":206560,"encode_duration_us":18861,"operation_id":"0x4bee","timeout_ms":10000,"last_server_duration_us":206560,"total_duration_us":3257819},{"operation_name":"upsert","last_dispatch_duration_us":3136813,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3136813,"total_server_duration_us":1536696,"encode_duration_us":19496,"operation_id":"0x4a2e","timeout_ms":10000,"last_server_duration_us":1536696,"total_duration_us":3141704},{"operation_name":"upsert","last_dispatch_duration_us":2980233,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2980233,"total_server_duration_us":390372,"encode_duration_us":18554,"operation_id":"0x4c7b","timeout_ms":10000,"last_server_duration_us":390372,"total_duration_us":2984498}],"total_count":49}} INFO [2022-03-13 17:11:07,470] jepsen worker 36 - jepsen.util 1836 :ok :read [12 20] INFO [2022-03-13 17:11:07,470] jepsen worker 36 - jepsen.util 1836 :invoke :read [12 nil] WARN [2022-03-13 17:11:07,487] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9101485,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11080133,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9154495,"encode_duration_us":20604,"operation_id":"0x49c1","timeout_ms":10000,"last_server_duration_us":9080926,"total_duration_us":10002595},{"operation_name":"upsert","last_dispatch_duration_us":21249647,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21249647,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9035244,"encode_duration_us":18251,"operation_id":"0x4486","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10010406},{"operation_name":"upsert","last_dispatch_duration_us":9060497,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9797254,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9514665,"encode_duration_us":19250,"operation_id":"0x45b3","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10009074}],"total_count":3}} INFO [2022-03-13 17:11:07,605] jepsen worker 36 - jepsen.util 1836 :ok :read [12 20] INFO [2022-03-13 17:11:07,606] jepsen worker 36 - jepsen.util 1836 :invoke :write [12 42] INFO [2022-03-13 17:11:07,640] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4327485380116959}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3411764705882353}} INFO [2022-03-13 17:11:07,640] jepsen worker nemesis - couchbase.util Rebalance check count: 179 INFO [2022-03-13 17:11:07,640] jepsen worker nemesis - couchbase.util Rebalance stuck count: 110 WARN [2022-03-13 17:11:07,647] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18150348,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26212863,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9168204,"encode_duration_us":17929,"operation_id":"0x42ec","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10005329}],"total_count":1}} INFO [2022-03-13 17:11:07,648] jepsen worker 44 - jepsen.util 494 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:07,649] jepsen worker 44 - jepsen.util 494 :invoke :read [14 nil] INFO [2022-03-13 17:11:08,645] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4327485380116959}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3411764705882353}} INFO [2022-03-13 17:11:08,646] jepsen worker nemesis - couchbase.util Rebalance check count: 180 INFO [2022-03-13 17:11:08,646] jepsen worker nemesis - couchbase.util Rebalance stuck count: 111 INFO [2022-03-13 17:11:08,680] jepsen worker 5 - jepsen.util 1805 :info :write [1 16] :RequestTimeoutException INFO [2022-03-13 17:11:08,681] jepsen worker 5 - jepsen.util 1895 :invoke :write [1 41] INFO [2022-03-13 17:11:08,700] jepsen worker 38 - jepsen.util 1928 :info :write [12 26] :RequestTimeoutException INFO [2022-03-13 17:11:08,700] jepsen worker 38 - jepsen.util 2018 :invoke :read [12 nil] INFO [2022-03-13 17:11:08,723] jepsen worker 38 - jepsen.util 2018 :ok :read [12 20] INFO [2022-03-13 17:11:08,723] jepsen worker 38 - jepsen.util 2018 :invoke :read [12 nil] INFO [2022-03-13 17:11:08,745] jepsen worker 38 - jepsen.util 2018 :ok :read [12 20] INFO [2022-03-13 17:11:08,746] jepsen worker 38 - jepsen.util 2018 :invoke :write [12 36] INFO [2022-03-13 17:11:09,386] jepsen worker 4 - jepsen.util 1714 :info :write [1 2] :RequestTimeoutException INFO [2022-03-13 17:11:09,387] jepsen worker 4 - jepsen.util 1804 :invoke :write [1 31] INFO [2022-03-13 17:11:09,464] jepsen worker 27 - jepsen.util 1737 :info :write [9 21] :SyncWriteAmbiguous INFO [2022-03-13 17:11:09,464] jepsen worker 27 - jepsen.util 1827 :invoke :read [9 nil] INFO [2022-03-13 17:11:09,506] jepsen worker 27 - jepsen.util 1827 :ok :read [9 26] INFO [2022-03-13 17:11:09,506] jepsen worker 27 - jepsen.util 1827 :invoke :read [9 nil] INFO [2022-03-13 17:11:09,534] jepsen worker 27 - jepsen.util 1827 :ok :read [9 26] INFO [2022-03-13 17:11:09,535] jepsen worker 27 - jepsen.util 1827 :invoke :read [9 nil] INFO [2022-03-13 17:11:09,635] jepsen worker 27 - jepsen.util 1827 :ok :read [9 26] INFO [2022-03-13 17:11:09,636] jepsen worker 27 - jepsen.util 1827 :invoke :write [9 37] INFO [2022-03-13 17:11:09,651] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4327485380116959}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3411764705882353}} INFO [2022-03-13 17:11:09,651] jepsen worker nemesis - couchbase.util Rebalance check count: 181 INFO [2022-03-13 17:11:09,651] jepsen worker nemesis - couchbase.util Rebalance stuck count: 112 INFO [2022-03-13 17:11:09,812] jepsen worker 18 - jepsen.util 1458 :ok :read [6 16] INFO [2022-03-13 17:11:09,813] jepsen worker 18 - jepsen.util 1458 :invoke :read [6 nil] INFO [2022-03-13 17:11:09,983] jepsen worker 18 - jepsen.util 1458 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:09,984] jepsen worker 18 - jepsen.util 1458 :invoke :write [6 43] INFO [2022-03-13 17:11:10,078] jepsen worker 57 - jepsen.util 1767 :info :write [19 26] :RequestTimeoutException INFO [2022-03-13 17:11:10,078] jepsen worker 57 - jepsen.util 1857 :invoke :read [19 nil] INFO [2022-03-13 17:11:10,397] jepsen worker 46 - jepsen.util 1666 :info :write [15 21] :RequestTimeoutException INFO [2022-03-13 17:11:10,398] jepsen worker 46 - jepsen.util 1756 :invoke :read [15 nil] INFO [2022-03-13 17:11:10,448] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,448] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,448] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:11:10,448] jepsen worker 78 - jepsen.util 438 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,449] jepsen worker 24 - jepsen.util 474 :invoke :read [8 nil] INFO [2022-03-13 17:11:10,450] jepsen worker 22 - jepsen.util 562 :invoke :read [7 nil] INFO [2022-03-13 17:11:10,451] jepsen worker 35 - jepsen.util 755 :invoke :read [11 nil] INFO [2022-03-13 17:11:10,452] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,452] jepsen worker 62 - jepsen.util 512 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,452] jepsen worker 78 - jepsen.util 438 :invoke :write [26 23] INFO [2022-03-13 17:11:10,453] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:10,457] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:10,536] jepsen worker 76 - jepsen.util 616 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,537] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,537] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:10,538] jepsen worker 76 - jepsen.util 616 :invoke :write [25 46] INFO [2022-03-13 17:11:10,538] jepsen worker 21 - jepsen.util 741 :invoke :write [7 42] INFO [2022-03-13 17:11:10,539] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:10,577] jepsen worker 12 - jepsen.util 732 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,577] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,577] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:11:10,578] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:10,578] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:11:10,579] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:10,580] jepsen worker 74 - jepsen.util 1964 :info :write [24 38] :RequestTimeoutException INFO [2022-03-13 17:11:10,581] jepsen worker 74 - jepsen.util 2054 :invoke :write [24 11] INFO [2022-03-13 17:11:10,644] jepsen worker 46 - jepsen.util 1756 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,645] jepsen worker 46 - jepsen.util 1756 :invoke :read [15 nil] INFO [2022-03-13 17:11:10,657] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,658] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4327485380116959}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3411764705882353}} INFO [2022-03-13 17:11:10,658] jepsen worker nemesis - couchbase.util Rebalance check count: 182 INFO [2022-03-13 17:11:10,658] jepsen worker nemesis - couchbase.util Rebalance stuck count: 113 INFO [2022-03-13 17:11:10,658] jepsen worker 70 - jepsen.util 700 :invoke :write [23 18] INFO [2022-03-13 17:11:10,667] jepsen worker 79 - jepsen.util 889 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,668] jepsen worker 28 - jepsen.util 1648 :info :write [9 22] :RequestTimeoutException INFO [2022-03-13 17:11:10,668] jepsen worker 79 - jepsen.util 889 :invoke :write [26 19] INFO [2022-03-13 17:11:10,669] jepsen worker 28 - jepsen.util 1738 :invoke :write [9 12] INFO [2022-03-13 17:11:10,709] jepsen worker 42 - jepsen.util 402 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,710] jepsen worker 42 - jepsen.util 402 :invoke :write [14 20] INFO [2022-03-13 17:11:10,715] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,715] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,715] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:10,716] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:11:10,787] jepsen worker 7 - jepsen.util 457 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,790] jepsen worker 7 - jepsen.util 457 :invoke :write [2 25] INFO [2022-03-13 17:11:10,793] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,793] jepsen worker 21 - jepsen.util 741 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,794] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:10,795] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:11:10,799] jepsen worker 16 - jepsen.util 1546 :ok :read [5 43] INFO [2022-03-13 17:11:10,800] jepsen worker 16 - jepsen.util 1546 :invoke :write [5 43] INFO [2022-03-13 17:11:10,877] jepsen worker 43 - jepsen.util 403 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,877] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,877] jepsen worker 43 - jepsen.util 403 :invoke :write [14 30] INFO [2022-03-13 17:11:10,878] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:10,879] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:10,879] jepsen worker 21 - jepsen.util 741 :invoke :write [7 39] INFO [2022-03-13 17:11:10,880] jepsen worker 76 - jepsen.util 616 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,881] jepsen worker 76 - jepsen.util 616 :invoke :write [25 31] INFO [2022-03-13 17:11:10,922] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:10,923] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:11:11,056] jepsen worker 10 - jepsen.util 1540 :info :write [3 19] :RequestTimeoutException INFO [2022-03-13 17:11:11,057] jepsen worker 10 - jepsen.util 1630 :invoke :write [3 31] INFO [2022-03-13 17:11:11,107] jepsen worker 68 - jepsen.util 1598 :info :write [22 40] :SyncWriteAmbiguous INFO [2022-03-13 17:11:11,108] jepsen worker 68 - jepsen.util 1688 :invoke :write [22 24] INFO [2022-03-13 17:11:11,284] jepsen worker 34 - jepsen.util 754 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:11,284] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:11,285] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:11,285] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:11:11,606] jepsen worker 29 - jepsen.util 1649 :info :write [9 48] :RequestTimeoutException INFO [2022-03-13 17:11:11,607] jepsen worker 29 - jepsen.util 1739 :invoke :write [9 29] INFO [2022-03-13 17:11:11,620] jepsen worker 20 - jepsen.util 1910 :info :write [6 37] :RequestTimeoutException INFO [2022-03-13 17:11:11,621] jepsen worker 20 - jepsen.util 2000 :invoke :write [6 9] INFO [2022-03-13 17:11:11,666] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3079178885630498}, :ns_1@172.23.105.187 {:progress 0.4327485380116959}, :ns_1@172.23.107.227 {:progress 0.3079178885630498}, :ns_1@172.23.107.228 {:progress 0.3411764705882353}} INFO [2022-03-13 17:11:11,666] jepsen worker nemesis - couchbase.util Rebalance check count: 183 INFO [2022-03-13 17:11:11,666] jepsen worker nemesis - couchbase.util Rebalance stuck count: 114 INFO [2022-03-13 17:11:11,716] jepsen worker 83 - jepsen.util 1073 :info :write [27 23] :RequestTimeoutException INFO [2022-03-13 17:11:11,717] jepsen worker 83 - jepsen.util 1163 :invoke :read [27 nil] INFO [2022-03-13 17:11:11,727] jepsen worker 58 - jepsen.util 1408 :info :write [19 41] :RequestTimeoutException INFO [2022-03-13 17:11:11,728] jepsen worker 58 - jepsen.util 1498 :invoke :write [19 41] INFO [2022-03-13 17:11:11,746] jepsen worker 81 - jepsen.util 1881 :info :write [27 49] :RequestTimeoutException INFO [2022-03-13 17:11:11,747] jepsen worker 81 - jepsen.util 1971 :invoke :read [27 nil] INFO [2022-03-13 17:11:11,858] jepsen worker 82 - jepsen.util 1522 :info :write [27 33] :RequestTimeoutException INFO [2022-03-13 17:11:11,858] jepsen worker 82 - jepsen.util 1612 :invoke :write [27 46] INFO [2022-03-13 17:11:11,870] jepsen worker 48 - jepsen.util 1578 :info :write [16 7] :RequestTimeoutException INFO [2022-03-13 17:11:11,870] jepsen worker 48 - jepsen.util 1668 :invoke :read [16 nil] INFO [2022-03-13 17:11:12,106] jepsen worker 87 - jepsen.util 2157 :info :write [29 6] :RequestTimeoutException INFO [2022-03-13 17:11:12,107] jepsen worker 87 - jepsen.util 2247 :invoke :write [29 47] INFO [2022-03-13 17:11:12,510] jepsen worker 25 - jepsen.util 385 :info :write [8 37] :RequestTimeoutException INFO [2022-03-13 17:11:12,511] jepsen worker 25 - jepsen.util 475 :invoke :read [8 nil] INFO [2022-03-13 17:11:12,656] jepsen worker 50 - jepsen.util 1670 :info :write [16 41] :SyncWriteAmbiguous INFO [2022-03-13 17:11:12,672] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3137829912023461}, :ns_1@172.23.105.187 {:progress 0.4385964912280702}, :ns_1@172.23.107.227 {:progress 0.3137829912023461}, :ns_1@172.23.107.228 {:progress 0.3470588235294118}} INFO [2022-03-13 17:11:13,372] jepsen worker 50 - jepsen.util 1760 :invoke :read [16 nil] INFO [2022-03-13 17:11:12,811] jepsen worker 1 - jepsen.util 1351 :info :write [0 44] :RequestTimeoutException INFO [2022-03-13 17:11:13,372] jepsen worker nemesis - couchbase.util Rebalance check count: 184 INFO [2022-03-13 17:11:13,372] jepsen worker 1 - jepsen.util 1441 :invoke :read [0 nil] INFO [2022-03-13 17:11:13,372] jepsen worker nemesis - couchbase.util Rebalance stuck count: 114 INFO [2022-03-13 17:11:12,848] jepsen worker 40 - jepsen.util 1570 :info :write [13 44] :RequestTimeoutException INFO [2022-03-13 17:11:12,990] jepsen worker 80 - jepsen.util 620 :info :write [26 35] :RequestTimeoutException INFO [2022-03-13 17:11:13,373] jepsen worker 40 - jepsen.util 1660 :invoke :read [13 nil] INFO [2022-03-13 17:11:13,373] jepsen worker 80 - jepsen.util 710 :invoke :write [26 18] INFO [2022-03-13 17:11:13,014] jepsen worker 88 - jepsen.util 1618 :info :write [29 17] :RequestTimeoutException INFO [2022-03-13 17:11:13,027] jepsen worker 66 - jepsen.util 1506 :info :write [22 27] :RequestTimeoutException INFO [2022-03-13 17:11:13,373] jepsen worker 88 - jepsen.util 1708 :invoke :write [29 40] INFO [2022-03-13 17:11:13,373] jepsen worker 66 - jepsen.util 1596 :invoke :read [22 nil] INFO [2022-03-13 17:11:13,036] jepsen worker 15 - jepsen.util 1725 :info :write [5 12] :RequestTimeoutException INFO [2022-03-13 17:11:13,046] jepsen worker 2 - jepsen.util 1892 :info :write [0 30] :RequestTimeoutException INFO [2022-03-13 17:11:13,091] jepsen worker 79 - jepsen.util 889 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,378] jepsen worker 15 - jepsen.util 1815 :invoke :read [5 nil] INFO [2022-03-13 17:11:13,092] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,378] jepsen worker 79 - jepsen.util 889 :invoke :write [26 35] INFO [2022-03-13 17:11:13,092] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,378] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:13,378] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:11:13,096] jepsen worker 26 - jepsen.util 476 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,162] jepsen worker 44 - jepsen.util 494 :ok :read [14 :nil] INFO [2022-03-13 17:11:13,379] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:13,162] jepsen worker 71 - jepsen.util 431 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,379] jepsen worker 44 - jepsen.util 494 :invoke :write [14 47] INFO [2022-03-13 17:11:13,200] jepsen worker 45 - jepsen.util 1845 :info :write [15 10] :RequestTimeoutException INFO [2022-03-13 17:11:13,379] jepsen worker 71 - jepsen.util 431 :invoke :write [23 22] INFO [2022-03-13 17:11:13,304] jepsen worker 76 - jepsen.util 616 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,379] jepsen worker 76 - jepsen.util 616 :invoke :write [25 4] INFO [2022-03-13 17:11:13,310] jepsen worker 51 - jepsen.util 591 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,311] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,312] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,312] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,317] jepsen worker 32 - jepsen.util 1382 :info :write [10 37] :RequestTimeoutException INFO [2022-03-13 17:11:13,319] jepsen worker 21 - jepsen.util 741 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,386] jepsen worker 32 - jepsen.util 1472 :invoke :write [10 27] INFO [2022-03-13 17:11:13,386] jepsen worker 51 - jepsen.util 591 :invoke :write [17 9] INFO [2022-03-13 17:11:13,386] jepsen worker 21 - jepsen.util 741 :invoke :write [7 25] INFO [2022-03-13 17:11:13,386] jepsen worker 77 - jepsen.util 707 :invoke :write [25 26] INFO [2022-03-13 17:11:13,386] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:13,378] jepsen worker 2 - jepsen.util 1982 :invoke :write [0 3] INFO [2022-03-13 17:11:13,394] jepsen worker 61 - jepsen.util 601 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,397] jepsen worker 61 - jepsen.util 601 :invoke :write [20 4] INFO [2022-03-13 17:11:13,379] jepsen worker 45 - jepsen.util 1935 :invoke :read [15 nil] INFO [2022-03-13 17:11:13,386] jepsen worker 60 - jepsen.util 600 :invoke :write [20 44] INFO [2022-03-13 17:11:13,409] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,409] jepsen worker 70 - jepsen.util 700 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,410] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:13,410] jepsen worker 6 - jepsen.util 546 :invoke :read [2 nil] INFO [2022-03-13 17:11:13,413] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,413] jepsen worker 34 - jepsen.util 754 :invoke :write [11 45] INFO [2022-03-13 17:11:13,480] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,481] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:11:13,483] jepsen worker 45 - jepsen.util 1935 :ok :read [15 48] INFO [2022-03-13 17:11:13,484] jepsen worker 45 - jepsen.util 1935 :invoke :write [15 17] INFO [2022-03-13 17:11:13,488] jepsen worker 39 - jepsen.util 1659 :info :write [13 15] :RequestTimeoutException INFO [2022-03-13 17:11:13,488] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,488] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:11:13,490] jepsen worker 39 - jepsen.util 1749 :invoke :read [13 nil] INFO [2022-03-13 17:11:13,512] jepsen worker 22 - jepsen.util 562 :ok :read [7 :nil] INFO [2022-03-13 17:11:13,512] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,512] jepsen worker 24 - jepsen.util 474 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,513] jepsen worker 35 - jepsen.util 755 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,513] jepsen worker 6 - jepsen.util 546 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,514] jepsen worker 22 - jepsen.util 562 :invoke :write [7 4] INFO [2022-03-13 17:11:13,515] jepsen worker 35 - jepsen.util 755 :invoke :write [11 6] INFO [2022-03-13 17:11:13,515] jepsen worker 6 - jepsen.util 546 :invoke :write [2 16] INFO [2022-03-13 17:11:13,516] jepsen worker 70 - jepsen.util 700 :invoke :write [23 2] INFO [2022-03-13 17:11:13,517] jepsen worker 24 - jepsen.util 474 :invoke :write [8 33] INFO [2022-03-13 17:11:13,736] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,736] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:13,736] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:13,736] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:13,750] jepsen worker 41 - jepsen.util 1391 :info :write [13 2] :RequestTimeoutException INFO [2022-03-13 17:11:13,751] jepsen worker 41 - jepsen.util 1481 :invoke :write [13 15] INFO [2022-03-13 17:11:14,054] jepsen worker 65 - jepsen.util 1865 :info :write [21 45] :RequestTimeoutException INFO [2022-03-13 17:11:14,054] jepsen worker 65 - jepsen.util 1955 :invoke :read [21 nil] INFO [2022-03-13 17:11:14,087] jepsen worker 86 - jepsen.util 1346 :info :write [28 25] :RequestTimeoutException INFO [2022-03-13 17:11:14,088] jepsen worker 59 - jepsen.util 1409 :info :write [19 5] :RequestTimeoutException INFO [2022-03-13 17:11:14,088] jepsen worker 59 - jepsen.util 1499 :invoke :read [19 nil] INFO [2022-03-13 17:11:14,088] jepsen worker 86 - jepsen.util 1436 :invoke :write [28 19] INFO [2022-03-13 17:11:14,117] jepsen worker 65 - jepsen.util 1955 :ok :read [21 :nil] INFO [2022-03-13 17:11:14,118] jepsen worker 65 - jepsen.util 1955 :invoke :read [21 nil] INFO [2022-03-13 17:11:14,132] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,133] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:14,183] jepsen worker 65 - jepsen.util 1955 :ok :read [21 :nil] INFO [2022-03-13 17:11:14,184] jepsen worker 65 - jepsen.util 1955 :invoke :write [21 34] INFO [2022-03-13 17:11:14,197] jepsen worker 0 - jepsen.util 1620 :info :write [0 1] :RequestTimeoutException INFO [2022-03-13 17:11:14,198] jepsen worker 0 - jepsen.util 1710 :invoke :write [0 29] INFO [2022-03-13 17:11:14,274] jepsen worker 56 - jepsen.util 1586 :info :write [18 20] :RequestTimeoutException INFO [2022-03-13 17:11:14,274] jepsen worker 56 - jepsen.util 1676 :invoke :read [18 nil] INFO [2022-03-13 17:11:14,326] jepsen worker 55 - jepsen.util 1585 :info :write [18 12] :RequestTimeoutException INFO [2022-03-13 17:11:14,326] jepsen worker 55 - jepsen.util 1675 :invoke :read [18 nil] INFO [2022-03-13 17:11:14,359] jepsen worker 79 - jepsen.util 889 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,360] jepsen worker 79 - jepsen.util 889 :invoke :write [26 8] INFO [2022-03-13 17:11:14,378] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3137829912023461}, :ns_1@172.23.105.187 {:progress 0.4385964912280702}, :ns_1@172.23.107.227 {:progress 0.3137829912023461}, :ns_1@172.23.107.228 {:progress 0.3470588235294118}} INFO [2022-03-13 17:11:14,378] jepsen worker nemesis - couchbase.util Rebalance check count: 185 INFO [2022-03-13 17:11:14,378] jepsen worker nemesis - couchbase.util Rebalance stuck count: 115 INFO [2022-03-13 17:11:14,410] jepsen worker 8 - jepsen.util 638 :info :write [2 12] :RequestTimeoutException INFO [2022-03-13 17:11:14,411] jepsen worker 8 - jepsen.util 728 :invoke :write [2 30] INFO [2022-03-13 17:11:14,606] jepsen worker 85 - jepsen.util 1705 :info :write [28 41] :RequestTimeoutException INFO [2022-03-13 17:11:14,607] jepsen worker 85 - jepsen.util 1795 :invoke :write [28 13] INFO [2022-03-13 17:11:14,736] jepsen worker 76 - jepsen.util 616 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,737] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:14,743] jepsen worker 80 - jepsen.util 710 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,743] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:14,751] jepsen worker 42 - jepsen.util 402 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,751] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,753] jepsen worker 42 - jepsen.util 402 :invoke :write [14 22] INFO [2022-03-13 17:11:14,753] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:14,771] jepsen worker 31 - jepsen.util 1561 :info :write [10 3] :RequestTimeoutException INFO [2022-03-13 17:11:14,772] jepsen worker 79 - jepsen.util 889 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,772] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,772] jepsen worker 31 - jepsen.util 1651 :invoke :read [10 nil] INFO [2022-03-13 17:11:14,773] jepsen worker 76 - jepsen.util 616 :invoke :write [25 47] INFO [2022-03-13 17:11:14,774] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:11:14,830] jepsen worker 47 - jepsen.util 1757 :info :write [15 7] :RequestTimeoutException INFO [2022-03-13 17:11:14,830] jepsen worker 47 - jepsen.util 1847 :invoke :read [15 nil] INFO [2022-03-13 17:11:14,838] jepsen worker 72 - jepsen.util 1602 :info :write [24 45] :RequestTimeoutException INFO [2022-03-13 17:11:14,840] jepsen worker 72 - jepsen.util 1692 :invoke :read [24 nil] INFO [2022-03-13 17:11:14,911] jepsen worker 47 - jepsen.util 1847 :ok :read [15 48] INFO [2022-03-13 17:11:14,912] jepsen worker 47 - jepsen.util 1847 :invoke :write [15 33] INFO [2022-03-13 17:11:14,933] jepsen worker 11 - jepsen.util 1631 :info :write [3 41] :RequestTimeoutException INFO [2022-03-13 17:11:14,934] jepsen worker 11 - jepsen.util 1721 :invoke :write [3 13] INFO [2022-03-13 17:11:14,940] jepsen worker 60 - jepsen.util 600 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,944] jepsen worker 60 - jepsen.util 600 :invoke :write [20 10] INFO [2022-03-13 17:11:14,951] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:14,952] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:11:15,144] jepsen worker 76 - jepsen.util 616 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,145] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:15,150] jepsen worker 30 - jepsen.util 1470 :info :write [10 5] :RequestTimeoutException INFO [2022-03-13 17:11:15,150] jepsen worker 30 - jepsen.util 1560 :invoke :write [10 22] INFO [2022-03-13 17:11:15,246] jepsen worker 54 - jepsen.util 1404 :info :write [18 41] :RequestTimeoutException INFO [2022-03-13 17:11:15,246] jepsen worker 54 - jepsen.util 1494 :invoke :read [18 nil] INFO [2022-03-13 17:11:15,356] jepsen worker 71 - jepsen.util 431 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,356] jepsen worker 52 - jepsen.util 502 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,356] jepsen worker 25 - jepsen.util 475 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,357] jepsen worker 25 - jepsen.util 475 :invoke :write [8 20] INFO [2022-03-13 17:11:15,357] jepsen worker 71 - jepsen.util 431 :invoke :write [23 21] INFO [2022-03-13 17:11:15,359] jepsen worker 52 - jepsen.util 502 :invoke :write [17 41] INFO [2022-03-13 17:11:15,375] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,375] jepsen worker 61 - jepsen.util 601 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,375] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,376] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:11:15,377] jepsen worker 53 - jepsen.util 503 :invoke :write [17 26] INFO [2022-03-13 17:11:15,377] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:15,384] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3137829912023461}, :ns_1@172.23.105.187 {:progress 0.4385964912280702}, :ns_1@172.23.107.227 {:progress 0.3137829912023461}, :ns_1@172.23.107.228 {:progress 0.3470588235294118}} INFO [2022-03-13 17:11:15,385] jepsen worker nemesis - couchbase.util Rebalance check count: 186 INFO [2022-03-13 17:11:15,385] jepsen worker nemesis - couchbase.util Rebalance stuck count: 116 INFO [2022-03-13 17:11:15,391] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,391] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:15,392] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:11:15,393] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] WARN [2022-03-13 17:11:15,797] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9051946,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9970416,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9329762,"encode_duration_us":19138,"operation_id":"0x4a73","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10006316}],"total_count":1}} WARN [2022-03-13 17:11:15,798] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8040753,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9120282,"total_server_duration_us":387121,"encode_duration_us":27343,"operation_id":"0x4a71","timeout_ms":10000,"last_server_duration_us":26330,"total_duration_us":10011008},{"operation_name":"upsert","last_dispatch_duration_us":30288,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6645549,"total_server_duration_us":458780,"encode_duration_us":17224,"operation_id":"0x4cd2","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10008532},{"operation_name":"upsert","last_dispatch_duration_us":8751161,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9539774,"total_server_duration_us":340586,"encode_duration_us":17809,"operation_id":"0x4c5d","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10006537},{"operation_name":"upsert","last_dispatch_duration_us":9051946,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9970416,"total_server_duration_us":9329762,"encode_duration_us":19138,"operation_id":"0x4a73","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10006316},{"operation_name":"upsert","last_dispatch_duration_us":45503,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7972242,"total_server_duration_us":411196,"encode_duration_us":17702,"operation_id":"0x4c69","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10004943},{"operation_name":"upsert","last_dispatch_duration_us":30079,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9319317,"total_server_duration_us":303955,"encode_duration_us":19831,"operation_id":"0x4c03","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10002501},{"operation_name":"upsert","last_dispatch_duration_us":9089753,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9739429,"total_server_duration_us":9288519,"encode_duration_us":22648,"operation_id":"0x4c5e","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9848903},{"operation_name":"upsert","last_dispatch_duration_us":9065427,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9245248,"total_server_duration_us":9146715,"encode_duration_us":19321,"operation_id":"0x4ad8","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9304812},{"operation_name":"upsert","last_dispatch_duration_us":9046864,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9046864,"total_server_duration_us":9029938,"encode_duration_us":16956,"operation_id":"0x4ad9","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":9050446},{"operation_name":"get","last_dispatch_duration_us":7957294,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":7957294,"total_server_duration_us":56,"operation_id":"0x4b3f","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":7958337}],"total_count":21}} INFO [2022-03-13 17:11:16,010] jepsen worker 64 - jepsen.util 1504 :info :write [21 22] :RequestTimeoutException INFO [2022-03-13 17:11:16,010] jepsen worker 64 - jepsen.util 1594 :invoke :write [21 49] INFO [2022-03-13 17:11:16,094] jepsen worker 73 - jepsen.util 1693 :info :write [24 34] :RequestTimeoutException INFO [2022-03-13 17:11:16,095] jepsen worker 73 - jepsen.util 1783 :invoke :read [24 nil] INFO [2022-03-13 17:11:16,390] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3137829912023461}, :ns_1@172.23.105.187 {:progress 0.4385964912280702}, :ns_1@172.23.107.227 {:progress 0.3137829912023461}, :ns_1@172.23.107.228 {:progress 0.3470588235294118}} INFO [2022-03-13 17:11:16,390] jepsen worker nemesis - couchbase.util Rebalance check count: 187 INFO [2022-03-13 17:11:16,390] jepsen worker nemesis - couchbase.util Rebalance stuck count: 117 INFO [2022-03-13 17:11:16,510] jepsen worker 17 - jepsen.util 1637 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,510] jepsen worker 49 - jepsen.util 1399 :ok :read [16 :nil] INFO [2022-03-13 17:11:16,511] jepsen worker 17 - jepsen.util 1637 :invoke :read [5 nil] INFO [2022-03-13 17:11:16,511] jepsen worker 49 - jepsen.util 1399 :invoke :write [16 47] INFO [2022-03-13 17:11:16,532] jepsen worker 67 - jepsen.util 1417 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,533] jepsen worker 67 - jepsen.util 1417 :invoke :read [22 nil] INFO [2022-03-13 17:11:16,795] jepsen worker 81 - jepsen.util 1971 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,796] jepsen worker 81 - jepsen.util 1971 :invoke :read [27 nil] INFO [2022-03-13 17:11:16,814] jepsen worker 59 - jepsen.util 1499 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,814] jepsen worker 1 - jepsen.util 1441 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,814] jepsen worker 59 - jepsen.util 1499 :invoke :read [19 nil] INFO [2022-03-13 17:11:16,815] jepsen worker 1 - jepsen.util 1441 :invoke :read [0 nil] INFO [2022-03-13 17:11:16,816] jepsen worker 19 - jepsen.util 1549 :info :write [6 18] :RequestTimeoutException INFO [2022-03-13 17:11:16,817] jepsen worker 19 - jepsen.util 1639 :invoke :read [6 nil] INFO [2022-03-13 17:11:16,818] jepsen worker 57 - jepsen.util 1857 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,819] jepsen worker 48 - jepsen.util 1668 :ok :read [16 :nil] INFO [2022-03-13 17:11:16,819] jepsen worker 40 - jepsen.util 1660 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,819] jepsen worker 57 - jepsen.util 1857 :invoke :read [19 nil] INFO [2022-03-13 17:11:16,820] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,820] jepsen worker 48 - jepsen.util 1668 :invoke :read [16 nil] INFO [2022-03-13 17:11:16,821] jepsen worker 75 - jepsen.util 435 :invoke :write [25 21] INFO [2022-03-13 17:11:16,822] jepsen worker 40 - jepsen.util 1660 :invoke :write [13 40] INFO [2022-03-13 17:11:16,877] jepsen worker 84 - jepsen.util 1884 :ok :read [28 44] INFO [2022-03-13 17:11:16,878] jepsen worker 84 - jepsen.util 1884 :invoke :write [28 8] INFO [2022-03-13 17:11:16,903] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,904] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:16,916] jepsen worker 77 - jepsen.util 707 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,917] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:16,931] jepsen worker 55 - jepsen.util 1675 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,932] jepsen worker 55 - jepsen.util 1675 :invoke :read [18 nil] INFO [2022-03-13 17:11:16,939] jepsen worker 85 - jepsen.util 1795 :fail :write [28 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,941] jepsen worker 34 - jepsen.util 754 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,941] jepsen worker 13 - jepsen.util 553 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,942] jepsen worker 17 - jepsen.util 1637 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,943] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:16,943] jepsen worker 85 - jepsen.util 1795 :invoke :write [28 41] INFO [2022-03-13 17:11:16,944] jepsen worker 17 - jepsen.util 1637 :invoke :read [5 nil] INFO [2022-03-13 17:11:16,945] jepsen worker 13 - jepsen.util 553 :invoke :write [4 31] INFO [2022-03-13 17:11:16,992] jepsen worker 55 - jepsen.util 1675 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:16,993] jepsen worker 55 - jepsen.util 1675 :invoke :read [18 nil] INFO [2022-03-13 17:11:17,024] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,024] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:17,032] jepsen worker 83 - jepsen.util 1163 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,033] jepsen worker 83 - jepsen.util 1163 :invoke :read [27 nil] INFO [2022-03-13 17:11:17,054] jepsen worker 81 - jepsen.util 1971 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,055] jepsen worker 81 - jepsen.util 1971 :invoke :read [27 nil] INFO [2022-03-13 17:11:17,055] jepsen worker 59 - jepsen.util 1499 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,055] jepsen worker 1 - jepsen.util 1441 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,056] jepsen worker 17 - jepsen.util 1637 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,056] jepsen worker 59 - jepsen.util 1499 :invoke :write [19 22] INFO [2022-03-13 17:11:17,057] jepsen worker 1 - jepsen.util 1441 :invoke :write [0 48] INFO [2022-03-13 17:11:17,057] jepsen worker 17 - jepsen.util 1637 :invoke :write [5 11] INFO [2022-03-13 17:11:17,071] jepsen worker 22 - jepsen.util 562 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,071] jepsen worker 22 - jepsen.util 562 :invoke :write [7 8] INFO [2022-03-13 17:11:17,085] jepsen worker 55 - jepsen.util 1675 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,086] jepsen worker 55 - jepsen.util 1675 :invoke :write [18 28] INFO [2022-03-13 17:11:17,118] jepsen worker 81 - jepsen.util 1971 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,119] jepsen worker 81 - jepsen.util 1971 :invoke :write [27 23] INFO [2022-03-13 17:11:17,158] jepsen worker 67 - jepsen.util 1417 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,158] jepsen worker 67 - jepsen.util 1417 :invoke :write [22 30] WARN [2022-03-13 17:11:17,195] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10229646,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12217776,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":10164047,"encode_duration_us":16796,"operation_id":"0x4c0e","timeout_ms":10000,"last_server_duration_us":10067221,"total_duration_us":10001810},{"operation_name":"upsert","last_dispatch_duration_us":14827009,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":14827009,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9059667,"encode_duration_us":29684,"operation_id":"0x4a2d","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":10002617},{"operation_name":"upsert","last_dispatch_duration_us":10179002,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10179002,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":10001751,"encode_duration_us":18117,"operation_id":"0x4caa","timeout_ms":10000,"last_server_duration_us":10001751,"total_duration_us":10007186}],"total_count":3}} INFO [2022-03-13 17:11:17,216] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,216] jepsen worker 14 - jepsen.util 284 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,217] jepsen worker 70 - jepsen.util 700 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,218] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,218] jepsen worker 76 - jepsen.util 616 :invoke :write [25 10] INFO [2022-03-13 17:11:17,219] jepsen worker 14 - jepsen.util 284 :invoke :write [4 11] INFO [2022-03-13 17:11:17,219] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:17,222] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:11:17,275] jepsen worker 21 - jepsen.util 741 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,276] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] WARN [2022-03-13 17:11:17,295] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9129485,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18113142,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9225241,"encode_duration_us":20046,"operation_id":"0x4a03","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10001484},{"operation_name":"upsert","last_dispatch_duration_us":17211498,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17211498,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9034183,"encode_duration_us":20573,"operation_id":"0x4a53","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10001506}],"total_count":2}} INFO [2022-03-13 17:11:17,299] jepsen worker 83 - jepsen.util 1163 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,300] jepsen worker 83 - jepsen.util 1163 :invoke :write [27 18] INFO [2022-03-13 17:11:17,396] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3137829912023461}, :ns_1@172.23.105.187 {:progress 0.4385964912280702}, :ns_1@172.23.107.227 {:progress 0.3137829912023461}, :ns_1@172.23.107.228 {:progress 0.3470588235294118}} INFO [2022-03-13 17:11:17,397] jepsen worker nemesis - couchbase.util Rebalance check count: 188 INFO [2022-03-13 17:11:17,397] jepsen worker nemesis - couchbase.util Rebalance stuck count: 118 WARN [2022-03-13 17:11:17,402] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17436086,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17809032,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9274443,"encode_duration_us":24811,"operation_id":"0x4965","timeout_ms":10000,"last_server_duration_us":9092628,"total_duration_us":10003429},{"operation_name":"upsert","last_dispatch_duration_us":17627105,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17884380,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9155593,"encode_duration_us":20498,"operation_id":"0x49cc","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10010142}],"total_count":2}} WARN [2022-03-13 17:11:17,402] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18107711,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18107711,"total_server_duration_us":5471473,"operation_id":"0x4ada","timeout_ms":30000,"last_server_duration_us":5471473,"total_duration_us":18108100},{"operation_name":"get","last_dispatch_duration_us":14557543,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14557543,"total_server_duration_us":23,"operation_id":"0x4c1c","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":14557748},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18286,"operation_id":"0x4bd2","timeout_ms":10000,"total_duration_us":10009856},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18440,"operation_id":"0x4cae","timeout_ms":10000,"total_duration_us":10009211},{"operation_name":"upsert","last_dispatch_duration_us":13590891,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":13590891,"total_server_duration_us":37963,"encode_duration_us":20138,"operation_id":"0x4c66","timeout_ms":10000,"last_server_duration_us":37963,"total_duration_us":10007870},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":20338,"operation_id":"0x4c6a","timeout_ms":10000,"total_duration_us":10007052},{"operation_name":"get","last_dispatch_duration_us":5047105,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":5047105,"total_server_duration_us":177090,"operation_id":"0x4e43","timeout_ms":30000,"last_server_duration_us":177090,"total_duration_us":5047399},{"operation_name":"upsert","last_dispatch_duration_us":4477828,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":4477828,"total_server_duration_us":3053852,"encode_duration_us":17996,"operation_id":"0x4d13","timeout_ms":10000,"last_server_duration_us":3053852,"total_duration_us":4481675},{"operation_name":"upsert","last_dispatch_duration_us":2925018,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":3841239,"total_server_duration_us":544327,"encode_duration_us":29874,"operation_id":"0x4e81","timeout_ms":10000,"last_server_duration_us":182506,"total_duration_us":3858020},{"operation_name":"upsert","last_dispatch_duration_us":3649065,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":3649065,"total_server_duration_us":111983,"encode_duration_us":20720,"operation_id":"0x4d4e","timeout_ms":10000,"last_server_duration_us":111983,"total_duration_us":3652939}],"total_count":23}} WARN [2022-03-13 17:11:17,408] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14963241,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":14963241,"total_server_duration_us":5833953,"operation_id":"0x4bf5","timeout_ms":30000,"last_server_duration_us":5833953,"total_duration_us":14963536},{"operation_name":"get","last_dispatch_duration_us":14079232,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":14079232,"total_server_duration_us":6083770,"operation_id":"0x4c55","timeout_ms":30000,"last_server_duration_us":6083770,"total_duration_us":14081613},{"operation_name":"get","last_dispatch_duration_us":2025301,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2025301,"total_server_duration_us":49,"operation_id":"0x4bed","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":13839483},{"operation_name":"upsert","last_dispatch_duration_us":9089585,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":10061063,"total_server_duration_us":9592792,"encode_duration_us":21252,"operation_id":"0x4b03","timeout_ms":10000,"last_server_duration_us":9072420,"total_duration_us":10010405},{"operation_name":"upsert","last_dispatch_duration_us":10131684,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":10131684,"total_server_duration_us":24154,"encode_duration_us":21431,"operation_id":"0x4c63","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10009627},{"operation_name":"upsert","last_dispatch_duration_us":2211045,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2211045,"total_server_duration_us":219932,"encode_duration_us":16718,"operation_id":"0x4cb0","timeout_ms":10000,"last_server_duration_us":219932,"total_duration_us":10008043},{"operation_name":"upsert","last_dispatch_duration_us":9579649,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":15312997,"total_server_duration_us":9907455,"encode_duration_us":20598,"operation_id":"0x4ba8","timeout_ms":10000,"last_server_duration_us":9564863,"total_duration_us":10007692},{"operation_name":"upsert","last_dispatch_duration_us":6177368,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":15098503,"total_server_duration_us":58615,"encode_duration_us":20954,"operation_id":"0x4c0f","timeout_ms":10000,"last_server_duration_us":31393,"total_duration_us":10007594},{"operation_name":"upsert","last_dispatch_duration_us":7159471,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":7159471,"total_server_duration_us":138798,"encode_duration_us":32442,"operation_id":"0x4c33","timeout_ms":10000,"last_server_duration_us":138798,"total_duration_us":10005798},{"operation_name":"upsert","last_dispatch_duration_us":119748,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1132300,"total_server_duration_us":662902,"encode_duration_us":17025,"operation_id":"0x4b02","timeout_ms":10000,"last_server_duration_us":91985,"total_duration_us":10005673}],"total_count":29}} INFO [2022-03-13 17:11:17,428] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,429] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:17,442] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,443] jepsen worker 23 - jepsen.util 563 :invoke :write [7 46] INFO [2022-03-13 17:11:17,447] jepsen worker 89 - jepsen.util 1979 :info :write [29 20] :RequestTimeoutException INFO [2022-03-13 17:11:17,448] jepsen worker 89 - jepsen.util 2069 :invoke :write [29 34] INFO [2022-03-13 17:11:17,471] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,472] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:11:17,480] jepsen worker 3 - jepsen.util 1623 :info :write [1 36] :RequestTimeoutException INFO [2022-03-13 17:11:17,481] jepsen worker 3 - jepsen.util 1713 :invoke :read [1 nil] INFO [2022-03-13 17:11:17,491] jepsen worker 37 - jepsen.util 1657 :info :write [12 40] :RequestTimeoutException INFO [2022-03-13 17:11:17,492] jepsen worker 37 - jepsen.util 1747 :invoke :write [12 3] WARN [2022-03-13 17:11:17,495] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":330508,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5762282,"total_server_duration_us":147625,"encode_duration_us":29438,"operation_id":"0x4ce0","timeout_ms":10000,"last_server_duration_us":25101,"total_duration_us":10010329},{"operation_name":"upsert","last_dispatch_duration_us":97166,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1576299,"total_server_duration_us":160666,"encode_duration_us":17525,"operation_id":"0x4c60","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10009755},{"operation_name":"upsert","last_dispatch_duration_us":170105,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1360657,"total_server_duration_us":106125,"encode_duration_us":23239,"operation_id":"0x4c65","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10009245},{"operation_name":"upsert","last_dispatch_duration_us":174856,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":3485998,"total_server_duration_us":108251,"encode_duration_us":17126,"operation_id":"0x4b74","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10008676},{"operation_name":"upsert","last_dispatch_duration_us":7989235,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8491639,"total_server_duration_us":276110,"encode_duration_us":16928,"operation_id":"0x4bb6","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10008000},{"operation_name":"upsert","last_dispatch_duration_us":10179002,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10179002,"total_server_duration_us":10001751,"encode_duration_us":18117,"operation_id":"0x4caa","timeout_ms":10000,"last_server_duration_us":10001751,"total_duration_us":10007186},{"operation_name":"upsert","last_dispatch_duration_us":159687,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":393274,"total_server_duration_us":44063,"encode_duration_us":17899,"operation_id":"0x4cab","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10007156},{"operation_name":"upsert","last_dispatch_duration_us":117361,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":879585,"total_server_duration_us":82876,"encode_duration_us":17854,"operation_id":"0x4c82","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10005142},{"operation_name":"upsert","last_dispatch_duration_us":337957,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8913772,"total_server_duration_us":289646,"encode_duration_us":19724,"operation_id":"0x4b8a","timeout_ms":10000,"last_server_duration_us":38899,"total_duration_us":10003625},{"operation_name":"upsert","last_dispatch_duration_us":137531,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":364143,"total_server_duration_us":44948,"encode_duration_us":25317,"operation_id":"0x4cac","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10002882}],"total_count":25}} WARN [2022-03-13 17:11:17,496] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":30304,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":1183763,"total_server_duration_us":608255,"encode_duration_us":16855,"operation_id":"0x4c64","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10009543},{"operation_name":"upsert","last_dispatch_duration_us":128103,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8450356,"total_server_duration_us":411958,"encode_duration_us":22919,"operation_id":"0x4d14","timeout_ms":10000,"last_server_duration_us":29975,"total_duration_us":10008707},{"operation_name":"upsert","last_dispatch_duration_us":97615,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8966522,"total_server_duration_us":366249,"encode_duration_us":17646,"operation_id":"0x4ce1","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10007840},{"operation_name":"upsert","last_dispatch_duration_us":144184,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8653919,"total_server_duration_us":313100,"encode_duration_us":34615,"operation_id":"0x4ca9","timeout_ms":10000,"last_server_duration_us":37756,"total_duration_us":10006234},{"operation_name":"upsert","last_dispatch_duration_us":208619,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8578690,"total_server_duration_us":331562,"encode_duration_us":18078,"operation_id":"0x4cad","timeout_ms":10000,"last_server_duration_us":42417,"total_duration_us":10004314},{"operation_name":"upsert","last_dispatch_duration_us":3465998,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9927603,"total_server_duration_us":430642,"encode_duration_us":19291,"operation_id":"0x4d57","timeout_ms":10000,"last_server_duration_us":252354,"total_duration_us":9952277},{"operation_name":"upsert","last_dispatch_duration_us":9136999,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9185268,"total_server_duration_us":9148721,"encode_duration_us":16690,"operation_id":"0x4c10","timeout_ms":10000,"last_server_duration_us":9118182,"total_duration_us":9198849},{"operation_name":"upsert","last_dispatch_duration_us":6123995,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6123995,"total_server_duration_us":55790,"encode_duration_us":17481,"operation_id":"0x4d56","timeout_ms":10000,"last_server_duration_us":55790,"total_duration_us":6133617},{"operation_name":"upsert","last_dispatch_duration_us":5828604,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":5828604,"total_server_duration_us":2157845,"encode_duration_us":16661,"operation_id":"0x4d55","timeout_ms":10000,"last_server_duration_us":2157845,"total_duration_us":5835427},{"operation_name":"upsert","last_dispatch_duration_us":3941782,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3941782,"total_server_duration_us":604587,"encode_duration_us":93693,"operation_id":"0x4e13","timeout_ms":10000,"last_server_duration_us":604587,"total_duration_us":3947095}],"total_count":19}} WARN [2022-03-13 17:11:17,508] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13162933,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13162933,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":43727,"encode_duration_us":18767,"operation_id":"0x4c9c","timeout_ms":10000,"last_server_duration_us":43727,"total_duration_us":10002267},{"operation_name":"upsert","last_dispatch_duration_us":9798599,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11048210,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9308444,"encode_duration_us":26514,"operation_id":"0x4ce4","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10005264},{"operation_name":"upsert","last_dispatch_duration_us":7159471,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":7159471,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":138798,"encode_duration_us":32442,"operation_id":"0x4c33","timeout_ms":10000,"last_server_duration_us":138798,"total_duration_us":10005798},{"operation_name":"upsert","last_dispatch_duration_us":2211045,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":2211045,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":219932,"encode_duration_us":16718,"operation_id":"0x4cb0","timeout_ms":10000,"last_server_duration_us":219932,"total_duration_us":10008043},{"operation_name":"upsert","last_dispatch_duration_us":9089585,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":10061063,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9592792,"encode_duration_us":21252,"operation_id":"0x4b03","timeout_ms":10000,"last_server_duration_us":9072420,"total_duration_us":10010405},{"operation_name":"upsert","last_dispatch_duration_us":9579649,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15312997,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9907455,"encode_duration_us":20598,"operation_id":"0x4ba8","timeout_ms":10000,"last_server_duration_us":9564863,"total_duration_us":10007692},{"operation_name":"upsert","last_dispatch_duration_us":20603702,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20603702,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9079863,"encode_duration_us":20937,"operation_id":"0x485a","timeout_ms":10000,"last_server_duration_us":9079863,"total_duration_us":10006632},{"operation_name":"upsert","last_dispatch_duration_us":10131684,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10131684,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":24154,"encode_duration_us":21431,"operation_id":"0x4c63","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10009627}],"total_count":8}} WARN [2022-03-13 17:11:17,596] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":113142,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8539179,"total_server_duration_us":122603,"encode_duration_us":17851,"operation_id":"0x4c67","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10010709},{"operation_name":"upsert","last_dispatch_duration_us":9072436,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":9717403,"total_server_duration_us":9459980,"encode_duration_us":17164,"operation_id":"0x4c25","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10009511},{"operation_name":"upsert","last_dispatch_duration_us":101920,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":101920,"total_server_duration_us":20921,"encode_duration_us":21246,"operation_id":"0x4cca","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10008978},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":52218,"operation_id":"0x4c05","timeout_ms":10000,"total_duration_us":10008260},{"operation_name":"upsert","last_dispatch_duration_us":100905,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9863550,"total_server_duration_us":150418,"encode_duration_us":20536,"operation_id":"0x4c04","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10008207},{"operation_name":"upsert","last_dispatch_duration_us":89528,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9866165,"total_server_duration_us":145452,"encode_duration_us":33973,"operation_id":"0x4c07","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10007940},{"operation_name":"upsert","last_dispatch_duration_us":31888,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":2184265,"total_server_duration_us":961338,"encode_duration_us":21953,"operation_id":"0x4ce6","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10004771},{"operation_name":"upsert","last_dispatch_duration_us":96448,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1840293,"total_server_duration_us":946208,"encode_duration_us":17176,"operation_id":"0x4d3f","timeout_ms":10000,"last_server_duration_us":47079,"total_duration_us":10004034},{"operation_name":"upsert","last_dispatch_duration_us":7935781,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":8915407,"total_server_duration_us":585292,"encode_duration_us":22417,"operation_id":"0x4b36","timeout_ms":10000,"last_server_duration_us":25800,"total_duration_us":10004023},{"operation_name":"upsert","last_dispatch_duration_us":203560,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":2259251,"total_server_duration_us":596440,"encode_duration_us":16971,"operation_id":"0x4bfc","timeout_ms":10000,"last_server_duration_us":27312,"total_duration_us":10001330}],"total_count":23}} WARN [2022-03-13 17:11:17,611] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18114865,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18114865,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9042674,"encode_duration_us":27722,"operation_id":"0x4947","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10004183},{"operation_name":"upsert","last_dispatch_duration_us":17871876,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17871876,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9076673,"encode_duration_us":36567,"operation_id":"0x4964","timeout_ms":10000,"last_server_duration_us":9076673,"total_duration_us":10010923},{"operation_name":"upsert","last_dispatch_duration_us":9072436,"last_local_id":"A312307A00000006/000000005DE126B4","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9717403,"last_local_socket":"172.23.104.79:33480","total_server_duration_us":9459980,"encode_duration_us":17164,"operation_id":"0x4c25","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10009511}],"total_count":3}} INFO [2022-03-13 17:11:17,631] jepsen worker 36 - jepsen.util 1836 :info :write [12 42] :RequestTimeoutException INFO [2022-03-13 17:11:17,631] jepsen worker 36 - jepsen.util 1926 :invoke :read [12 nil] INFO [2022-03-13 17:11:17,636] jepsen worker 35 - jepsen.util 755 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,637] jepsen worker 35 - jepsen.util 755 :invoke :write [11 11] INFO [2022-03-13 17:11:17,638] jepsen worker 6 - jepsen.util 546 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,639] jepsen worker 6 - jepsen.util 546 :invoke :write [2 33] INFO [2022-03-13 17:11:17,664] jepsen worker 36 - jepsen.util 1926 :ok :read [12 20] INFO [2022-03-13 17:11:17,665] jepsen worker 36 - jepsen.util 1926 :invoke :write [12 4] INFO [2022-03-13 17:11:17,896] jepsen worker 41 - jepsen.util 1481 :fail :write [13 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:17,897] jepsen worker 41 - jepsen.util 1481 :invoke :write [13 4] INFO [2022-03-13 17:11:18,066] jepsen worker 1 - jepsen.util 1441 :fail :write [0 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,067] jepsen worker 1 - jepsen.util 1441 :invoke :write [0 47] INFO [2022-03-13 17:11:18,069] jepsen worker 0 - jepsen.util 1710 :fail :write [0 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,070] jepsen worker 57 - jepsen.util 1857 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,070] jepsen worker 48 - jepsen.util 1668 :ok :read [16 :nil] INFO [2022-03-13 17:11:18,070] jepsen worker 0 - jepsen.util 1710 :invoke :read [0 nil] INFO [2022-03-13 17:11:18,071] jepsen worker 48 - jepsen.util 1668 :invoke :write [16 6] INFO [2022-03-13 17:11:18,072] jepsen worker 57 - jepsen.util 1857 :invoke :read [19 nil] INFO [2022-03-13 17:11:18,402] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3137829912023461}, :ns_1@172.23.105.187 {:progress 0.4385964912280702}, :ns_1@172.23.107.227 {:progress 0.3137829912023461}, :ns_1@172.23.107.228 {:progress 0.3470588235294118}} INFO [2022-03-13 17:11:18,402] jepsen worker nemesis - couchbase.util Rebalance check count: 189 INFO [2022-03-13 17:11:18,402] jepsen worker nemesis - couchbase.util Rebalance stuck count: 119 INFO [2022-03-13 17:11:18,515] jepsen worker 83 - jepsen.util 1163 :fail :write [27 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,516] jepsen worker 83 - jepsen.util 1163 :invoke :write [27 18] INFO [2022-03-13 17:11:18,602] jepsen worker 49 - jepsen.util 1399 :fail :write [16 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,602] jepsen worker 49 - jepsen.util 1399 :invoke :write [16 23] INFO [2022-03-13 17:11:18,679] jepsen worker 1 - jepsen.util 1441 :fail :write [0 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,680] jepsen worker 1 - jepsen.util 1441 :invoke :read [0 nil] INFO [2022-03-13 17:11:18,709] jepsen worker 5 - jepsen.util 1895 :info :write [1 41] :RequestTimeoutException INFO [2022-03-13 17:11:18,710] jepsen worker 5 - jepsen.util 1985 :invoke :write [1 10] INFO [2022-03-13 17:11:18,774] jepsen worker 38 - jepsen.util 2018 :info :write [12 36] :RequestTimeoutException INFO [2022-03-13 17:11:18,774] jepsen worker 38 - jepsen.util 2108 :invoke :read [12 nil] INFO [2022-03-13 17:11:18,807] jepsen worker 38 - jepsen.util 2108 :ok :read [12 20] INFO [2022-03-13 17:11:18,808] jepsen worker 38 - jepsen.util 2108 :invoke :read [12 nil] INFO [2022-03-13 17:11:18,820] jepsen worker 8 - jepsen.util 728 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,820] jepsen worker 75 - jepsen.util 435 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,820] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,820] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,820] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:18,822] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:18,823] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:18,823] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:11:18,824] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:18,825] jepsen worker 8 - jepsen.util 728 :invoke :write [2 17] INFO [2022-03-13 17:11:18,826] jepsen worker 76 - jepsen.util 616 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,827] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:18,829] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,830] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:18,830] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,831] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,831] jepsen worker 79 - jepsen.util 889 :invoke :write [26 18] INFO [2022-03-13 17:11:18,832] jepsen worker 76 - jepsen.util 616 :invoke :write [25 49] INFO [2022-03-13 17:11:18,835] jepsen worker 38 - jepsen.util 2108 :ok :read [12 20] INFO [2022-03-13 17:11:18,836] jepsen worker 38 - jepsen.util 2108 :invoke :read [12 nil] INFO [2022-03-13 17:11:18,850] jepsen worker 1 - jepsen.util 1441 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,851] jepsen worker 1 - jepsen.util 1441 :invoke :read [0 nil] INFO [2022-03-13 17:11:18,870] jepsen worker 38 - jepsen.util 2108 :ok :read [12 20] INFO [2022-03-13 17:11:18,871] jepsen worker 38 - jepsen.util 2108 :invoke :read [12 nil] INFO [2022-03-13 17:11:18,884] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,885] jepsen worker 12 - jepsen.util 732 :invoke :write [4 3] INFO [2022-03-13 17:11:18,905] jepsen worker 38 - jepsen.util 2108 :ok :read [12 20] INFO [2022-03-13 17:11:18,906] jepsen worker 38 - jepsen.util 2108 :invoke :read [12 nil] INFO [2022-03-13 17:11:18,928] jepsen worker 38 - jepsen.util 2108 :ok :read [12 20] INFO [2022-03-13 17:11:18,929] jepsen worker 38 - jepsen.util 2108 :invoke :write [12 24] INFO [2022-03-13 17:11:18,980] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:18,980] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:18,981] jepsen worker 21 - jepsen.util 741 :invoke :write [7 11] INFO [2022-03-13 17:11:18,982] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:19,037] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,038] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:19,094] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,095] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:19,138] jepsen worker 1 - jepsen.util 1441 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,139] jepsen worker 1 - jepsen.util 1441 :invoke :read [0 nil] INFO [2022-03-13 17:11:19,290] jepsen worker 4 - jepsen.util 1804 :info :write [1 31] :SyncWriteAmbiguous INFO [2022-03-13 17:11:19,291] jepsen worker 4 - jepsen.util 1894 :invoke :write [1 34] INFO [2022-03-13 17:11:19,299] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,299] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:19,307] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,308] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:19,410] jepsen worker 1 - jepsen.util 1441 :ok :read [0 41] INFO [2022-03-13 17:11:19,411] jepsen worker 1 - jepsen.util 1441 :invoke :read [0 nil] INFO [2022-03-13 17:11:19,421] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,421] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,421] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,422] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:19,422] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:19,423] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:19,438] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.4444444444444444}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:19,438] jepsen worker nemesis - couchbase.util Rebalance check count: 190 INFO [2022-03-13 17:11:19,438] jepsen worker nemesis - couchbase.util Rebalance stuck count: 119 INFO [2022-03-13 17:11:19,593] jepsen worker 16 - jepsen.util 1546 :fail :write [5 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,595] jepsen worker 16 - jepsen.util 1546 :invoke :write [5 16] INFO [2022-03-13 17:11:19,637] jepsen worker 1 - jepsen.util 1441 :ok :read [0 41] INFO [2022-03-13 17:11:19,637] jepsen worker 1 - jepsen.util 1441 :invoke :write [0 16] INFO [2022-03-13 17:11:19,657] jepsen worker 27 - jepsen.util 1827 :info :write [9 37] :RequestTimeoutException INFO [2022-03-13 17:11:19,657] jepsen worker 27 - jepsen.util 1917 :invoke :read [9 nil] INFO [2022-03-13 17:11:19,691] jepsen worker 27 - jepsen.util 1917 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:19,691] jepsen worker 27 - jepsen.util 1917 :invoke :write [9 31] INFO [2022-03-13 17:11:20,017] jepsen worker 18 - jepsen.util 1458 :info :write [6 43] :RequestTimeoutException INFO [2022-03-13 17:11:20,018] jepsen worker 18 - jepsen.util 1548 :invoke :read [6 nil] INFO [2022-03-13 17:11:20,353] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:20,354] jepsen worker 69 - jepsen.util 609 :invoke :write [23 7] INFO [2022-03-13 17:11:20,388] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:20,389] jepsen worker 26 - jepsen.util 476 :invoke :write [8 12] INFO [2022-03-13 17:11:20,444] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.4444444444444444}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:20,444] jepsen worker nemesis - couchbase.util Rebalance check count: 191 INFO [2022-03-13 17:11:20,444] jepsen worker nemesis - couchbase.util Rebalance stuck count: 120 INFO [2022-03-13 17:11:20,490] jepsen worker 78 - jepsen.util 438 :info :write [26 23] :RequestTimeoutException INFO [2022-03-13 17:11:20,491] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:20,610] jepsen worker 74 - jepsen.util 2054 :info :write [24 11] :RequestTimeoutException INFO [2022-03-13 17:11:20,610] jepsen worker 74 - jepsen.util 2144 :invoke :read [24 nil] INFO [2022-03-13 17:11:20,694] jepsen worker 28 - jepsen.util 1738 :info :write [9 12] :RequestTimeoutException INFO [2022-03-13 17:11:20,695] jepsen worker 28 - jepsen.util 1828 :invoke :read [9 nil] INFO [2022-03-13 17:11:20,821] jepsen worker 7 - jepsen.util 457 :info :write [2 25] :RequestTimeoutException INFO [2022-03-13 17:11:20,822] jepsen worker 7 - jepsen.util 547 :invoke :write [2 35] INFO [2022-03-13 17:11:20,833] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:20,841] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:20,907] jepsen worker 43 - jepsen.util 403 :info :write [14 30] :RequestTimeoutException INFO [2022-03-13 17:11:20,908] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:11:21,070] jepsen worker 79 - jepsen.util 889 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:21,071] jepsen worker 79 - jepsen.util 889 :invoke :write [26 33] INFO [2022-03-13 17:11:21,075] jepsen worker 10 - jepsen.util 1630 :info :write [3 31] :RequestTimeoutException INFO [2022-03-13 17:11:21,076] jepsen worker 10 - jepsen.util 1720 :invoke :write [3 37] INFO [2022-03-13 17:11:21,136] jepsen worker 68 - jepsen.util 1688 :info :write [22 24] :RequestTimeoutException INFO [2022-03-13 17:11:21,137] jepsen worker 68 - jepsen.util 1778 :invoke :write [22 49] INFO [2022-03-13 17:11:21,450] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.4444444444444444}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:21,450] jepsen worker nemesis - couchbase.util Rebalance check count: 192 INFO [2022-03-13 17:11:21,450] jepsen worker nemesis - couchbase.util Rebalance stuck count: 121 INFO [2022-03-13 17:11:21,517] jepsen worker 63 - jepsen.util 1683 :ok :read [21 :nil] INFO [2022-03-13 17:11:21,518] jepsen worker 63 - jepsen.util 1683 :invoke :read [21 nil] INFO [2022-03-13 17:11:21,540] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:21,541] jepsen worker 77 - jepsen.util 707 :invoke :write [25 30] INFO [2022-03-13 17:11:21,557] jepsen worker 9 - jepsen.util 1629 :ok :read [3 25] INFO [2022-03-13 17:11:21,558] jepsen worker 9 - jepsen.util 1629 :invoke :write [3 36] INFO [2022-03-13 17:11:21,631] jepsen worker 29 - jepsen.util 1739 :info :write [9 29] :RequestTimeoutException INFO [2022-03-13 17:11:21,631] jepsen worker 29 - jepsen.util 1829 :invoke :read [9 nil] INFO [2022-03-13 17:11:21,648] jepsen worker 20 - jepsen.util 2000 :info :write [6 9] :RequestTimeoutException INFO [2022-03-13 17:11:21,648] jepsen worker 20 - jepsen.util 2090 :invoke :read [6 nil] INFO [2022-03-13 17:11:21,754] jepsen worker 58 - jepsen.util 1498 :info :write [19 41] :RequestTimeoutException INFO [2022-03-13 17:11:21,755] jepsen worker 58 - jepsen.util 1588 :invoke :write [19 41] INFO [2022-03-13 17:11:21,761] jepsen worker 31 - jepsen.util 1651 :ok :read [10 :nil] INFO [2022-03-13 17:11:21,761] jepsen worker 31 - jepsen.util 1651 :invoke :write [10 17] INFO [2022-03-13 17:11:21,801] jepsen worker 46 - jepsen.util 1756 :ok :read [15 48] INFO [2022-03-13 17:11:21,801] jepsen worker 46 - jepsen.util 1756 :invoke :write [15 19] INFO [2022-03-13 17:11:21,852] jepsen worker 63 - jepsen.util 1683 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:21,853] jepsen worker 63 - jepsen.util 1683 :invoke :write [21 48] INFO [2022-03-13 17:11:21,886] jepsen worker 82 - jepsen.util 1612 :info :write [27 46] :RequestTimeoutException INFO [2022-03-13 17:11:21,887] jepsen worker 82 - jepsen.util 1702 :invoke :read [27 nil] INFO [2022-03-13 17:11:21,968] jepsen worker 18 - jepsen.util 1548 :ok :read [6 16] INFO [2022-03-13 17:11:21,969] jepsen worker 18 - jepsen.util 1548 :invoke :write [6 18] INFO [2022-03-13 17:11:22,003] jepsen worker 8 - jepsen.util 728 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,003] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:11:22,003] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,004] jepsen worker 43 - jepsen.util 493 :invoke :write [14 22] INFO [2022-03-13 17:11:22,004] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:11:22,005] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:22,013] jepsen worker 56 - jepsen.util 1676 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,014] jepsen worker 56 - jepsen.util 1676 :invoke :read [18 nil] INFO [2022-03-13 17:11:22,016] jepsen worker 72 - jepsen.util 1692 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,017] jepsen worker 72 - jepsen.util 1692 :invoke :write [24 3] INFO [2022-03-13 17:11:22,048] jepsen worker 19 - jepsen.util 1639 :ok :read [6 16] INFO [2022-03-13 17:11:22,048] jepsen worker 19 - jepsen.util 1639 :invoke :read [6 nil] INFO [2022-03-13 17:11:22,115] jepsen worker 28 - jepsen.util 1828 :ok :read [9 26] INFO [2022-03-13 17:11:22,117] jepsen worker 28 - jepsen.util 1828 :invoke :write [9 0] INFO [2022-03-13 17:11:22,128] jepsen worker 87 - jepsen.util 2247 :info :write [29 47] :RequestTimeoutException INFO [2022-03-13 17:11:22,131] jepsen worker 87 - jepsen.util 2337 :invoke :read [29 nil] INFO [2022-03-13 17:11:22,142] jepsen worker 20 - jepsen.util 2090 :ok :read [6 16] INFO [2022-03-13 17:11:22,143] jepsen worker 20 - jepsen.util 2090 :invoke :write [6 44] INFO [2022-03-13 17:11:22,160] jepsen worker 87 - jepsen.util 2337 :ok :read [29 10] INFO [2022-03-13 17:11:22,161] jepsen worker 87 - jepsen.util 2337 :invoke :write [29 20] INFO [2022-03-13 17:11:22,184] jepsen worker 51 - jepsen.util 591 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,185] jepsen worker 51 - jepsen.util 591 :invoke :write [17 5] INFO [2022-03-13 17:11:22,203] jepsen worker 56 - jepsen.util 1676 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,203] jepsen worker 56 - jepsen.util 1676 :invoke :write [18 32] INFO [2022-03-13 17:11:22,328] jepsen worker 19 - jepsen.util 1639 :ok :read [6 16] INFO [2022-03-13 17:11:22,328] jepsen worker 19 - jepsen.util 1639 :invoke :write [6 1] INFO [2022-03-13 17:11:22,384] jepsen worker 26 - jepsen.util 476 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,385] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:22,386] jepsen worker 79 - jepsen.util 889 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,386] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:22,392] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,393] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:11:22,424] jepsen worker 76 - jepsen.util 616 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,424] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,425] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:22,425] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:22,456] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.4444444444444444}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:22,456] jepsen worker nemesis - couchbase.util Rebalance check count: 193 INFO [2022-03-13 17:11:22,456] jepsen worker nemesis - couchbase.util Rebalance stuck count: 122 INFO [2022-03-13 17:11:22,536] jepsen worker 54 - jepsen.util 1494 :ok :read [18 21] INFO [2022-03-13 17:11:22,537] jepsen worker 54 - jepsen.util 1494 :invoke :read [18 nil] INFO [2022-03-13 17:11:22,590] jepsen worker 60 - jepsen.util 600 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,590] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,590] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,592] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:22,592] jepsen worker 79 - jepsen.util 889 :invoke :write [26 44] INFO [2022-03-13 17:11:22,593] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,594] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:22,595] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:11:22,596] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,597] jepsen worker 76 - jepsen.util 616 :invoke :write [25 40] INFO [2022-03-13 17:11:22,642] jepsen worker 42 - jepsen.util 402 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,642] jepsen worker 42 - jepsen.util 402 :invoke :write [14 17] INFO [2022-03-13 17:11:22,796] jepsen worker 79 - jepsen.util 889 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,796] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:22,882] jepsen worker 54 - jepsen.util 1494 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:22,883] jepsen worker 54 - jepsen.util 1494 :invoke :write [18 41] INFO [2022-03-13 17:11:23,416] jepsen worker 32 - jepsen.util 1472 :info :write [10 27] :RequestTimeoutException INFO [2022-03-13 17:11:23,416] jepsen worker 2 - jepsen.util 1982 :info :write [0 3] :RequestTimeoutException INFO [2022-03-13 17:11:23,417] jepsen worker 2 - jepsen.util 2072 :invoke :write [0 31] INFO [2022-03-13 17:11:23,418] jepsen worker 32 - jepsen.util 1562 :invoke :read [10 nil] INFO [2022-03-13 17:11:23,426] jepsen worker 44 - jepsen.util 494 :info :write [14 47] :RequestTimeoutException INFO [2022-03-13 17:11:23,427] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:23,430] jepsen worker 88 - jepsen.util 1708 :info :write [29 40] :RequestTimeoutException INFO [2022-03-13 17:11:23,431] jepsen worker 88 - jepsen.util 1798 :invoke :read [29 nil] INFO [2022-03-13 17:11:23,461] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.4444444444444444}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:23,461] jepsen worker nemesis - couchbase.util Rebalance check count: 194 INFO [2022-03-13 17:11:23,461] jepsen worker nemesis - couchbase.util Rebalance stuck count: 123 INFO [2022-03-13 17:11:23,511] jepsen worker 45 - jepsen.util 1935 :info :write [15 17] :RequestTimeoutException INFO [2022-03-13 17:11:23,512] jepsen worker 45 - jepsen.util 2025 :invoke :write [15 8] INFO [2022-03-13 17:11:23,550] jepsen worker 24 - jepsen.util 474 :info :write [8 33] :RequestTimeoutException INFO [2022-03-13 17:11:23,551] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:11:23,620] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,620] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:23,626] jepsen worker 77 - jepsen.util 707 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,627] jepsen worker 77 - jepsen.util 707 :invoke :write [25 37] INFO [2022-03-13 17:11:23,643] jepsen worker 76 - jepsen.util 616 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,644] jepsen worker 76 - jepsen.util 616 :invoke :write [25 7] INFO [2022-03-13 17:11:23,647] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,648] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:23,649] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,650] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:23,651] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,652] jepsen worker 79 - jepsen.util 889 :invoke :write [26 42] INFO [2022-03-13 17:11:23,702] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,702] jepsen worker 62 - jepsen.util 512 :invoke :write [20 39] INFO [2022-03-13 17:11:23,729] jepsen worker 23 - jepsen.util 563 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,729] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:11:23,773] jepsen worker 76 - jepsen.util 616 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,773] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:23,776] jepsen worker 69 - jepsen.util 609 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,776] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:23,824] jepsen worker 53 - jepsen.util 503 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,825] jepsen worker 53 - jepsen.util 503 :invoke :write [17 28] INFO [2022-03-13 17:11:23,841] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,841] jepsen worker 79 - jepsen.util 889 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,842] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:23,842] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:23,844] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,844] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,847] jepsen worker 76 - jepsen.util 616 :invoke :write [25 6] INFO [2022-03-13 17:11:23,848] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:23,849] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,850] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:23,851] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,852] jepsen worker 79 - jepsen.util 889 :invoke :write [26 25] INFO [2022-03-13 17:11:23,860] jepsen worker 21 - jepsen.util 741 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,861] jepsen worker 21 - jepsen.util 741 :invoke :write [7 16] INFO [2022-03-13 17:11:23,888] jepsen worker 12 - jepsen.util 732 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,889] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:23,939] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,940] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:11:23,969] jepsen worker 14 - jepsen.util 284 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:23,970] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:24,094] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,095] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,094] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,095] jepsen worker 76 - jepsen.util 616 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,095] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] INFO [2022-03-13 17:11:24,096] jepsen worker 76 - jepsen.util 616 :invoke :write [25 46] INFO [2022-03-13 17:11:24,097] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:24,098] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:11:24,116] jepsen worker 86 - jepsen.util 1436 :info :write [28 19] :RequestTimeoutException INFO [2022-03-13 17:11:24,116] jepsen worker 86 - jepsen.util 1526 :invoke :read [28 nil] INFO [2022-03-13 17:11:24,140] jepsen worker 21 - jepsen.util 741 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,141] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:24,203] jepsen worker 65 - jepsen.util 1955 :info :write [21 34] :RequestTimeoutException INFO [2022-03-13 17:11:24,204] jepsen worker 65 - jepsen.util 2045 :invoke :read [21 nil] INFO [2022-03-13 17:11:24,206] jepsen worker 50 - jepsen.util 1760 :ok :read [16 :nil] INFO [2022-03-13 17:11:24,207] jepsen worker 50 - jepsen.util 1760 :invoke :read [16 nil] INFO [2022-03-13 17:11:24,241] jepsen worker 15 - jepsen.util 1815 :ok :read [5 43] INFO [2022-03-13 17:11:24,242] jepsen worker 15 - jepsen.util 1815 :invoke :read [5 nil] INFO [2022-03-13 17:11:24,255] jepsen worker 50 - jepsen.util 1760 :ok :read [16 :nil] INFO [2022-03-13 17:11:24,256] jepsen worker 50 - jepsen.util 1760 :invoke :write [16 18] INFO [2022-03-13 17:11:24,269] jepsen worker 82 - jepsen.util 1702 :ok :read [27 4] INFO [2022-03-13 17:11:24,270] jepsen worker 82 - jepsen.util 1702 :invoke :read [27 nil] INFO [2022-03-13 17:11:24,281] jepsen worker 15 - jepsen.util 1815 :ok :read [5 43] INFO [2022-03-13 17:11:24,281] jepsen worker 15 - jepsen.util 1815 :invoke :read [5 nil] INFO [2022-03-13 17:11:24,296] jepsen worker 51 - jepsen.util 591 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,296] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,297] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,297] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:24,298] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:11:24,302] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:11:24,304] jepsen worker 79 - jepsen.util 889 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,305] jepsen worker 79 - jepsen.util 889 :invoke :write [26 2] INFO [2022-03-13 17:11:24,324] jepsen worker 82 - jepsen.util 1702 :ok :read [27 4] INFO [2022-03-13 17:11:24,325] jepsen worker 82 - jepsen.util 1702 :invoke :write [27 29] INFO [2022-03-13 17:11:24,405] jepsen worker 15 - jepsen.util 1815 :ok :read [5 43] INFO [2022-03-13 17:11:24,406] jepsen worker 15 - jepsen.util 1815 :invoke :write [5 30] INFO [2022-03-13 17:11:24,467] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.4444444444444444}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:24,467] jepsen worker nemesis - couchbase.util Rebalance check count: 195 INFO [2022-03-13 17:11:24,467] jepsen worker nemesis - couchbase.util Rebalance stuck count: 124 INFO [2022-03-13 17:11:24,470] jepsen worker 66 - jepsen.util 1596 :ok :read [22 30] INFO [2022-03-13 17:11:24,470] jepsen worker 66 - jepsen.util 1596 :invoke :read [22 nil] INFO [2022-03-13 17:11:24,481] jepsen worker 39 - jepsen.util 1749 :ok :read [13 39] INFO [2022-03-13 17:11:24,481] jepsen worker 39 - jepsen.util 1749 :invoke :read [13 nil] INFO [2022-03-13 17:11:24,512] jepsen worker 66 - jepsen.util 1596 :ok :read [22 30] INFO [2022-03-13 17:11:24,513] jepsen worker 66 - jepsen.util 1596 :invoke :write [22 14] INFO [2022-03-13 17:11:24,521] jepsen worker 39 - jepsen.util 1749 :ok :read [13 39] INFO [2022-03-13 17:11:24,522] jepsen worker 39 - jepsen.util 1749 :invoke :write [13 22] INFO [2022-03-13 17:11:24,733] jepsen worker 47 - jepsen.util 1847 :info :write [15 33] :SyncWriteAmbiguous INFO [2022-03-13 17:11:24,734] jepsen worker 47 - jepsen.util 1937 :invoke :read [15 nil] INFO [2022-03-13 17:11:24,763] jepsen worker 73 - jepsen.util 1783 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,764] jepsen worker 73 - jepsen.util 1783 :invoke :read [24 nil] INFO [2022-03-13 17:11:24,800] jepsen worker 29 - jepsen.util 1829 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,800] jepsen worker 74 - jepsen.util 2144 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,800] jepsen worker 32 - jepsen.util 1562 :ok :read [10 :nil] INFO [2022-03-13 17:11:24,801] jepsen worker 32 - jepsen.util 1562 :invoke :read [10 nil] INFO [2022-03-13 17:11:24,802] jepsen worker 74 - jepsen.util 2144 :invoke :read [24 nil] INFO [2022-03-13 17:11:24,802] jepsen worker 29 - jepsen.util 1829 :invoke :write [9 13] INFO [2022-03-13 17:11:24,805] jepsen worker 65 - jepsen.util 2045 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,805] jepsen worker 73 - jepsen.util 1783 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,805] jepsen worker 65 - jepsen.util 2045 :invoke :write [21 34] INFO [2022-03-13 17:11:24,806] jepsen worker 73 - jepsen.util 1783 :invoke :read [24 nil] INFO [2022-03-13 17:11:24,807] jepsen worker 32 - jepsen.util 1562 :ok :read [10 :nil] INFO [2022-03-13 17:11:24,807] jepsen worker 74 - jepsen.util 2144 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,807] jepsen worker 74 - jepsen.util 2144 :invoke :write [24 42] INFO [2022-03-13 17:11:24,808] jepsen worker 32 - jepsen.util 1562 :invoke :write [10 43] INFO [2022-03-13 17:11:24,812] jepsen worker 73 - jepsen.util 1783 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:24,813] jepsen worker 73 - jepsen.util 1783 :invoke :write [24 34] INFO [2022-03-13 17:11:24,970] jepsen worker 11 - jepsen.util 1721 :info :write [3 13] :RequestTimeoutException INFO [2022-03-13 17:11:24,971] jepsen worker 11 - jepsen.util 1811 :invoke :read [3 nil] INFO [2022-03-13 17:11:25,066] jepsen worker 11 - jepsen.util 1811 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:25,067] jepsen worker 11 - jepsen.util 1811 :invoke :write [3 34] INFO [2022-03-13 17:11:25,173] jepsen worker 30 - jepsen.util 1560 :info :write [10 22] :RequestTimeoutException INFO [2022-03-13 17:11:25,174] jepsen worker 30 - jepsen.util 1650 :invoke :write [10 33] INFO [2022-03-13 17:11:25,386] jepsen worker 71 - jepsen.util 431 :info :write [23 21] :RequestTimeoutException INFO [2022-03-13 17:11:25,386] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:25,390] jepsen worker 25 - jepsen.util 475 :info :write [8 20] :RequestTimeoutException INFO [2022-03-13 17:11:25,390] jepsen worker 52 - jepsen.util 502 :info :write [17 41] :RequestTimeoutException INFO [2022-03-13 17:11:25,391] jepsen worker 52 - jepsen.util 592 :invoke :write [17 2] INFO [2022-03-13 17:11:25,391] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:25,472] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.456140350877193}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:25,472] jepsen worker nemesis - couchbase.util Rebalance check count: 196 INFO [2022-03-13 17:11:25,473] jepsen worker nemesis - couchbase.util Rebalance stuck count: 124 INFO [2022-03-13 17:11:25,563] jepsen worker 3 - jepsen.util 1713 :ok :read [1 42] INFO [2022-03-13 17:11:25,564] jepsen worker 3 - jepsen.util 1713 :invoke :read [1 nil] INFO [2022-03-13 17:11:25,592] jepsen worker 3 - jepsen.util 1713 :ok :read [1 42] INFO [2022-03-13 17:11:25,592] jepsen worker 3 - jepsen.util 1713 :invoke :write [1 26] INFO [2022-03-13 17:11:25,673] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:25,674] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:25,716] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:25,716] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:25,716] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:25,717] jepsen worker 80 - jepsen.util 710 :invoke :write [26 17] INFO [2022-03-13 17:11:25,718] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:25,718] jepsen worker 70 - jepsen.util 700 :invoke :write [23 36] INFO [2022-03-13 17:11:25,750] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:25,750] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:11:25,758] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:25,759] jepsen worker 44 - jepsen.util 584 :invoke :write [14 29] INFO [2022-03-13 17:11:25,792] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:25,792] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] WARN [2022-03-13 17:11:25,811] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":35013,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1417802,"total_server_duration_us":593788,"encode_duration_us":23646,"operation_id":"0x4d61","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10009764},{"operation_name":"upsert","last_dispatch_duration_us":34852,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1410434,"total_server_duration_us":592457,"encode_duration_us":28468,"operation_id":"0x4d5c","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10009542},{"operation_name":"upsert","last_dispatch_duration_us":8799777,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9615332,"total_server_duration_us":506992,"encode_duration_us":18581,"operation_id":"0x4e94","timeout_ms":10000,"last_server_duration_us":22720,"total_duration_us":10007323},{"operation_name":"upsert","last_dispatch_duration_us":33025,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1311133,"total_server_duration_us":562007,"encode_duration_us":18027,"operation_id":"0x4d7f","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10006610},{"operation_name":"upsert","last_dispatch_duration_us":8645163,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9489517,"total_server_duration_us":523659,"encode_duration_us":21479,"operation_id":"0x4e3f","timeout_ms":10000,"last_server_duration_us":27132,"total_duration_us":10001100},{"operation_name":"upsert","last_dispatch_duration_us":9057780,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9573216,"total_server_duration_us":9301334,"encode_duration_us":16708,"operation_id":"0x4ef5","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9802927},{"operation_name":"get","last_dispatch_duration_us":8667086,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8667086,"total_server_duration_us":49,"operation_id":"0x4f29","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8667475},{"operation_name":"get","last_dispatch_duration_us":6738683,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6738683,"total_server_duration_us":4130298,"operation_id":"0x4dda","timeout_ms":30000,"last_server_duration_us":4130298,"total_duration_us":6738959},{"operation_name":"get","last_dispatch_duration_us":4947301,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":4947301,"total_server_duration_us":27,"operation_id":"0x4e4a","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":4947512},{"operation_name":"get","last_dispatch_duration_us":4188322,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":4188322,"total_server_duration_us":32,"operation_id":"0x502e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":4188644}],"total_count":26}} INFO [2022-03-13 17:11:26,036] jepsen worker 64 - jepsen.util 1594 :info :write [21 49] :RequestTimeoutException INFO [2022-03-13 17:11:26,037] jepsen worker 64 - jepsen.util 1684 :invoke :write [21 32] INFO [2022-03-13 17:11:26,381] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,381] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,383] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:26,384] jepsen worker 51 - jepsen.util 591 :invoke :write [17 2] INFO [2022-03-13 17:11:26,414] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,415] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:11:26,458] jepsen worker 53 - jepsen.util 503 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,459] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,459] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,459] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,459] jepsen worker 53 - jepsen.util 503 :invoke :write [17 34] INFO [2022-03-13 17:11:26,462] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:26,463] jepsen worker 26 - jepsen.util 476 :invoke :write [8 27] INFO [2022-03-13 17:11:26,463] jepsen worker 61 - jepsen.util 601 :invoke :write [20 8] INFO [2022-03-13 17:11:26,478] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.456140350877193}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:26,478] jepsen worker nemesis - couchbase.util Rebalance check count: 197 INFO [2022-03-13 17:11:26,478] jepsen worker nemesis - couchbase.util Rebalance stuck count: 125 INFO [2022-03-13 17:11:26,518] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,519] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:26,573] jepsen worker 79 - jepsen.util 889 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,574] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:26,584] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,584] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,585] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:11:26,586] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:26,660] jepsen worker 42 - jepsen.util 402 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,660] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:26,776] jepsen worker 76 - jepsen.util 616 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,777] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:26,821] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,821] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,822] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,822] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:11:26,823] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:11:26,823] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:26,850] jepsen worker 40 - jepsen.util 1660 :info :write [13 40] :RequestTimeoutException INFO [2022-03-13 17:11:26,851] jepsen worker 40 - jepsen.util 1750 :invoke :write [13 20] INFO [2022-03-13 17:11:26,900] jepsen worker 84 - jepsen.util 1884 :info :write [28 8] :RequestTimeoutException INFO [2022-03-13 17:11:26,901] jepsen worker 84 - jepsen.util 1974 :invoke :read [28 nil] INFO [2022-03-13 17:11:26,966] jepsen worker 85 - jepsen.util 1795 :info :write [28 41] :RequestTimeoutException INFO [2022-03-13 17:11:26,967] jepsen worker 85 - jepsen.util 1885 :invoke :read [28 nil] INFO [2022-03-13 17:11:26,979] jepsen worker 13 - jepsen.util 553 :info :write [4 31] :RequestTimeoutException INFO [2022-03-13 17:11:26,980] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,980] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:26,981] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:26,983] jepsen worker 44 - jepsen.util 584 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,984] jepsen worker 44 - jepsen.util 584 :invoke :write [14 48] INFO [2022-03-13 17:11:26,988] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:26,988] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:26,989] jepsen worker 21 - jepsen.util 741 :invoke :write [7 5] INFO [2022-03-13 17:11:26,989] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:11:27,032] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,033] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:27,081] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,081] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,081] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,081] jepsen worker 76 - jepsen.util 616 :invoke :write [25 40] INFO [2022-03-13 17:11:27,082] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:27,082] jepsen worker 79 - jepsen.util 889 :invoke :write [26 31] INFO [2022-03-13 17:11:27,086] jepsen worker 17 - jepsen.util 1637 :info :write [5 11] :RequestTimeoutException INFO [2022-03-13 17:11:27,086] jepsen worker 17 - jepsen.util 1727 :invoke :read [5 nil] INFO [2022-03-13 17:11:27,088] jepsen worker 59 - jepsen.util 1499 :info :write [19 22] :RequestTimeoutException INFO [2022-03-13 17:11:27,088] jepsen worker 59 - jepsen.util 1589 :invoke :read [19 nil] INFO [2022-03-13 17:11:27,100] jepsen worker 22 - jepsen.util 562 :info :write [7 8] :RequestTimeoutException INFO [2022-03-13 17:11:27,100] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:27,110] jepsen worker 55 - jepsen.util 1675 :info :write [18 28] :RequestTimeoutException INFO [2022-03-13 17:11:27,111] jepsen worker 55 - jepsen.util 1765 :invoke :read [18 nil] INFO [2022-03-13 17:11:27,146] jepsen worker 81 - jepsen.util 1971 :info :write [27 23] :RequestTimeoutException INFO [2022-03-13 17:11:27,146] jepsen worker 81 - jepsen.util 2061 :invoke :read [27 nil] INFO [2022-03-13 17:11:27,180] jepsen worker 67 - jepsen.util 1417 :info :write [22 30] :RequestTimeoutException INFO [2022-03-13 17:11:27,180] jepsen worker 67 - jepsen.util 1507 :invoke :read [22 nil] INFO [2022-03-13 17:11:27,188] jepsen worker 43 - jepsen.util 493 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,189] jepsen worker 43 - jepsen.util 493 :invoke :write [14 18] INFO [2022-03-13 17:11:27,196] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,197] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:11:27,205] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,206] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:27,209] jepsen worker 0 - jepsen.util 1710 :ok :read [0 41] WARN [2022-03-13 17:11:27,210] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9465700,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17957340,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9584768,"encode_duration_us":16928,"operation_id":"0x4bb6","timeout_ms":10000,"last_server_duration_us":9308658,"total_duration_us":10008000}],"total_count":1}} INFO [2022-03-13 17:11:27,210] jepsen worker 0 - jepsen.util 1710 :invoke :write [0 35] INFO [2022-03-13 17:11:27,220] jepsen worker 57 - jepsen.util 1857 :ok :read [19 17] INFO [2022-03-13 17:11:27,222] jepsen worker 57 - jepsen.util 1857 :invoke :read [19 nil] INFO [2022-03-13 17:11:27,227] jepsen worker 17 - jepsen.util 1727 :ok :read [5 43] INFO [2022-03-13 17:11:27,228] jepsen worker 17 - jepsen.util 1727 :invoke :write [5 5] INFO [2022-03-13 17:11:27,242] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,242] jepsen worker 77 - jepsen.util 707 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,243] jepsen worker 77 - jepsen.util 707 :invoke :write [25 24] INFO [2022-03-13 17:11:27,243] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:11:27,263] jepsen worker 57 - jepsen.util 1857 :ok :read [19 17] INFO [2022-03-13 17:11:27,267] jepsen worker 57 - jepsen.util 1857 :invoke :write [19 40] WARN [2022-03-13 17:11:27,317] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9043459,"last_local_id":"A312307A00000004/0000000095638474","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9998881,"last_local_socket":"172.23.104.79:33498","total_server_duration_us":9539772,"encode_duration_us":16726,"operation_id":"0x4da8","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10010653}],"total_count":1}} INFO [2022-03-13 17:11:27,392] jepsen worker 76 - jepsen.util 616 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,393] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] WARN [2022-03-13 17:11:27,410] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11134667,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11134667,"total_server_duration_us":17588,"operation_id":"0x4e07","timeout_ms":30000,"last_server_duration_us":17588,"total_duration_us":11139192},{"operation_name":"get","last_dispatch_duration_us":297630,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":297630,"total_server_duration_us":24411,"operation_id":"0x4e75","timeout_ms":30000,"last_server_duration_us":24411,"total_duration_us":11074852},{"operation_name":"get","last_dispatch_duration_us":190598,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":190598,"total_server_duration_us":24325,"operation_id":"0x4e92","timeout_ms":30000,"last_server_duration_us":24325,"total_duration_us":10969696},{"operation_name":"upsert","last_dispatch_duration_us":11145836,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11667426,"total_server_duration_us":165521,"encode_duration_us":22768,"operation_id":"0x4dd9","timeout_ms":10000,"last_server_duration_us":90931,"total_duration_us":10010168},{"operation_name":"upsert","last_dispatch_duration_us":1977770,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":10187236,"total_server_duration_us":529765,"encode_duration_us":21673,"operation_id":"0x4e1a","timeout_ms":10000,"last_server_duration_us":97174,"total_duration_us":10007948},{"operation_name":"upsert","last_dispatch_duration_us":10368484,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10368484,"total_server_duration_us":62412,"encode_duration_us":18103,"operation_id":"0x4e40","timeout_ms":10000,"last_server_duration_us":62412,"total_duration_us":10007740},{"operation_name":"upsert","last_dispatch_duration_us":30304,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":3690770,"total_server_duration_us":546209,"encode_duration_us":26986,"operation_id":"0x4d59","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10003348},{"operation_name":"upsert","last_dispatch_duration_us":34784,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":905850,"total_server_duration_us":571233,"encode_duration_us":17023,"operation_id":"0x4e4c","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10002541},{"operation_name":"upsert","last_dispatch_duration_us":1121943,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":8415825,"total_server_duration_us":266787,"encode_duration_us":51463,"operation_id":"0x4e89","timeout_ms":10000,"last_server_duration_us":172334,"total_duration_us":8745411},{"operation_name":"get","last_dispatch_duration_us":7737594,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7737594,"total_server_duration_us":62,"operation_id":"0x4ec3","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":7737948}],"total_count":29}} WARN [2022-03-13 17:11:27,416] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16451911,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":16451911,"total_server_duration_us":1138,"operation_id":"0x4ce2","timeout_ms":30000,"last_server_duration_us":1138,"total_duration_us":16452269},{"operation_name":"get","last_dispatch_duration_us":15354133,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":15354133,"total_server_duration_us":6143,"operation_id":"0x4d15","timeout_ms":30000,"last_server_duration_us":6143,"total_duration_us":15356672},{"operation_name":"upsert","last_dispatch_duration_us":181657,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":2226048,"total_server_duration_us":294369,"encode_duration_us":19217,"operation_id":"0x4f65","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10010177},{"operation_name":"upsert","last_dispatch_duration_us":538691,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9431425,"total_server_duration_us":243332,"encode_duration_us":19938,"operation_id":"0x4e7c","timeout_ms":10000,"last_server_duration_us":66825,"total_duration_us":10009008},{"operation_name":"upsert","last_dispatch_duration_us":118662,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":2213243,"total_server_duration_us":270840,"encode_duration_us":18153,"operation_id":"0x4f6a","timeout_ms":10000,"last_server_duration_us":32548,"total_duration_us":10007949},{"operation_name":"upsert","last_dispatch_duration_us":242668,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1927477,"total_server_duration_us":246294,"encode_duration_us":22658,"operation_id":"0x4f67","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10005355},{"operation_name":"upsert","last_dispatch_duration_us":237248,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1904631,"total_server_duration_us":282813,"encode_duration_us":18767,"operation_id":"0x4f5f","timeout_ms":10000,"last_server_duration_us":60380,"total_duration_us":10003185},{"operation_name":"upsert","last_dispatch_duration_us":10539123,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10539123,"total_server_duration_us":51003,"encode_duration_us":16680,"operation_id":"0x4e1d","timeout_ms":10000,"last_server_duration_us":51003,"total_duration_us":10001287},{"operation_name":"upsert","last_dispatch_duration_us":9108696,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":9526124,"total_server_duration_us":9386469,"encode_duration_us":18446,"operation_id":"0x4daf","timeout_ms":10000,"last_server_duration_us":9095821,"total_duration_us":9882394},{"operation_name":"get","last_dispatch_duration_us":574455,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":574455,"total_server_duration_us":284211,"operation_id":"0x4efb","timeout_ms":30000,"last_server_duration_us":284211,"total_duration_us":7275199}],"total_count":31}} WARN [2022-03-13 17:11:27,416] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":20275591,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20275591,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9071357,"encode_duration_us":18286,"operation_id":"0x4bd2","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10009856}],"total_count":1}} WARN [2022-03-13 17:11:27,424] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":4013760,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4013760,"total_server_duration_us":122688,"encode_duration_us":20251,"operation_id":"0x4efe","timeout_ms":10000,"last_server_duration_us":122688,"total_duration_us":10010320},{"operation_name":"upsert","last_dispatch_duration_us":3443340,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3443340,"total_server_duration_us":360471,"encode_duration_us":27614,"operation_id":"0x4de4","timeout_ms":10000,"last_server_duration_us":360471,"total_duration_us":10010069},{"operation_name":"upsert","last_dispatch_duration_us":81676,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7447802,"total_server_duration_us":453076,"encode_duration_us":17314,"operation_id":"0x4e00","timeout_ms":10000,"last_server_duration_us":56401,"total_duration_us":10010057},{"operation_name":"upsert","last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","encode_duration_us":19361,"operation_id":"0x4f68","timeout_ms":10000,"total_duration_us":10008603},{"operation_name":"upsert","last_dispatch_duration_us":5552863,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5552863,"total_server_duration_us":80113,"encode_duration_us":24638,"operation_id":"0x4ea5","timeout_ms":10000,"last_server_duration_us":80113,"total_duration_us":10008106},{"operation_name":"upsert","last_dispatch_duration_us":151038,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":1796963,"total_server_duration_us":765412,"encode_duration_us":17614,"operation_id":"0x4f69","timeout_ms":10000,"last_server_duration_us":38899,"total_duration_us":10006211},{"operation_name":"upsert","last_dispatch_duration_us":9065112,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":16879762,"total_server_duration_us":45296,"encode_duration_us":21413,"operation_id":"0x4da6","timeout_ms":10000,"last_server_duration_us":25800,"total_duration_us":10005641},{"operation_name":"upsert","last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","encode_duration_us":28518,"operation_id":"0x4f60","timeout_ms":10000,"total_duration_us":10005516},{"operation_name":"upsert","last_dispatch_duration_us":12125940,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":12125940,"total_server_duration_us":9022513,"encode_duration_us":51220,"operation_id":"0x4e85","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10005340},{"operation_name":"upsert","last_dispatch_duration_us":28904,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":3568781,"total_server_duration_us":645843,"encode_duration_us":17432,"operation_id":"0x4f59","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10004579}],"total_count":18}} INFO [2022-03-13 17:11:27,429] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,429] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,430] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:11:27,431] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:27,470] jepsen worker 89 - jepsen.util 2069 :info :write [29 34] :RequestTimeoutException INFO [2022-03-13 17:11:27,470] jepsen worker 89 - jepsen.util 2159 :invoke :read [29 nil] INFO [2022-03-13 17:11:27,485] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.456140350877193}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:27,485] jepsen worker nemesis - couchbase.util Rebalance check count: 198 INFO [2022-03-13 17:11:27,485] jepsen worker nemesis - couchbase.util Rebalance stuck count: 126 INFO [2022-03-13 17:11:27,490] jepsen worker 59 - jepsen.util 1589 :ok :read [19 17] INFO [2022-03-13 17:11:27,491] jepsen worker 59 - jepsen.util 1589 :invoke :write [19 47] INFO [2022-03-13 17:11:27,516] jepsen worker 37 - jepsen.util 1747 :info :write [12 3] :RequestTimeoutException INFO [2022-03-13 17:11:27,517] jepsen worker 37 - jepsen.util 1837 :invoke :read [12 nil] WARN [2022-03-13 17:11:27,518] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10844014,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10844014,"total_server_duration_us":6727,"operation_id":"0x4e76","timeout_ms":30000,"last_server_duration_us":6727,"total_duration_us":10846589},{"operation_name":"get","last_dispatch_duration_us":10832458,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10832458,"total_server_duration_us":38,"operation_id":"0x4e72","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":10833228},{"operation_name":"upsert","last_dispatch_duration_us":9043459,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9998881,"total_server_duration_us":9539772,"encode_duration_us":16726,"operation_id":"0x4da8","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10010653},{"operation_name":"upsert","last_dispatch_duration_us":12456488,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12456488,"total_server_duration_us":18799,"encode_duration_us":17308,"operation_id":"0x4e42","timeout_ms":10000,"last_server_duration_us":18799,"total_duration_us":10007691},{"operation_name":"upsert","last_dispatch_duration_us":47150,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":1610087,"total_server_duration_us":839393,"encode_duration_us":19229,"operation_id":"0x4efa","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10002971},{"operation_name":"upsert","last_dispatch_duration_us":177162,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":871242,"total_server_duration_us":337795,"encode_duration_us":20861,"operation_id":"0x4e0a","timeout_ms":10000,"last_server_duration_us":35011,"total_duration_us":10002691},{"operation_name":"upsert","last_dispatch_duration_us":61819,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":2095901,"total_server_duration_us":888268,"encode_duration_us":17862,"operation_id":"0x4ec1","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10001107},{"operation_name":"upsert","last_dispatch_duration_us":2175741,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":8332407,"total_server_duration_us":190362,"encode_duration_us":42712,"operation_id":"0x4f01","timeout_ms":10000,"last_server_duration_us":60885,"total_duration_us":8404478},{"operation_name":"upsert","last_dispatch_duration_us":2112652,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":7973579,"total_server_duration_us":183254,"encode_duration_us":25610,"operation_id":"0x4f00","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":8299515},{"operation_name":"get","last_dispatch_duration_us":8056913,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":8056913,"total_server_duration_us":9284,"operation_id":"0x4f78","timeout_ms":30000,"last_server_duration_us":9284,"total_duration_us":8060685}],"total_count":28}} WARN [2022-03-13 17:11:27,524] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":12125940,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":12125940,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9022513,"encode_duration_us":51220,"operation_id":"0x4e85","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10005340}],"total_count":1}} WARN [2022-03-13 17:11:27,525] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":149736,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9484495,"total_server_duration_us":495754,"encode_duration_us":17504,"operation_id":"0x4e4b","timeout_ms":10000,"last_server_duration_us":92740,"total_duration_us":10009505},{"operation_name":"upsert","last_dispatch_duration_us":120639,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9543007,"total_server_duration_us":342894,"encode_duration_us":19581,"operation_id":"0x4e1e","timeout_ms":10000,"last_server_duration_us":53974,"total_duration_us":10008433},{"operation_name":"upsert","last_dispatch_duration_us":4031037,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4031037,"total_server_duration_us":80683,"encode_duration_us":19546,"operation_id":"0x4efc","timeout_ms":10000,"last_server_duration_us":80683,"total_duration_us":10008233},{"operation_name":"upsert","last_dispatch_duration_us":6585953,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":12253882,"total_server_duration_us":203628,"encode_duration_us":39683,"operation_id":"0x4e83","timeout_ms":10000,"last_server_duration_us":99814,"total_duration_us":10007571},{"operation_name":"upsert","last_dispatch_duration_us":32924,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":513494,"total_server_duration_us":254067,"encode_duration_us":19020,"operation_id":"0x4ebe","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10007554},{"operation_name":"upsert","last_dispatch_duration_us":104861,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7555771,"total_server_duration_us":616734,"encode_duration_us":17801,"operation_id":"0x4f28","timeout_ms":10000,"last_server_duration_us":67353,"total_duration_us":10007429},{"operation_name":"upsert","last_dispatch_duration_us":141168,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":480003,"total_server_duration_us":160312,"encode_duration_us":17689,"operation_id":"0x4e7f","timeout_ms":10000,"last_server_duration_us":65645,"total_duration_us":10004004},{"operation_name":"upsert","last_dispatch_duration_us":7099885,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8531032,"total_server_duration_us":784096,"encode_duration_us":17696,"operation_id":"0x4dd1","timeout_ms":10000,"last_server_duration_us":112639,"total_duration_us":10002631},{"operation_name":"get","last_dispatch_duration_us":6983193,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":6983193,"total_server_duration_us":38,"operation_id":"0x4eea","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":6987420},{"operation_name":"get","last_dispatch_duration_us":6293553,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6293553,"total_server_duration_us":32,"operation_id":"0x4fe7","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6293846}],"total_count":16}} INFO [2022-03-13 17:11:27,604] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,616] jepsen worker 79 - jepsen.util 889 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,013] jepsen worker 8 - jepsen.util 728 :invoke :write [2 25] INFO [2022-03-13 17:11:27,616] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,616] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,014] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] WARN [2022-03-13 17:11:27,625] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9110927,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11295192,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":10057159,"encode_duration_us":21953,"operation_id":"0x4ce6","timeout_ms":10000,"last_server_duration_us":9095821,"total_duration_us":10004771},{"operation_name":"upsert","last_dispatch_duration_us":18223450,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18223450,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9030999,"encode_duration_us":52218,"operation_id":"0x4c05","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10008260}],"total_count":2}} INFO [2022-03-13 17:11:28,014] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:11:27,650] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,016] jepsen worker 23 - jepsen.util 563 :invoke :write [7 10] INFO [2022-03-13 17:11:27,660] jepsen worker 35 - jepsen.util 755 :info :write [11 11] :RequestTimeoutException INFO [2022-03-13 17:11:27,666] jepsen worker 6 - jepsen.util 546 :info :write [2 33] :RequestTimeoutException INFO [2022-03-13 17:11:27,681] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,017] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:28,017] jepsen worker 6 - jepsen.util 636 :invoke :write [2 25] INFO [2022-03-13 17:11:27,686] jepsen worker 36 - jepsen.util 1926 :info :write [12 4] :RequestTimeoutException INFO [2022-03-13 17:11:27,836] jepsen worker 62 - jepsen.util 512 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,018] jepsen worker 36 - jepsen.util 2016 :invoke :read [12 nil] INFO [2022-03-13 17:11:28,018] jepsen worker 62 - jepsen.util 512 :invoke :write [20 27] INFO [2022-03-13 17:11:27,836] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,018] jepsen worker 14 - jepsen.util 284 :invoke :write [4 10] INFO [2022-03-13 17:11:27,846] jepsen worker 21 - jepsen.util 741 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,875] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:27,919] jepsen worker 41 - jepsen.util 1481 :info :write [13 4] :RequestTimeoutException INFO [2022-03-13 17:11:28,013] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:28,026] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:11:28,026] jepsen worker 41 - jepsen.util 1571 :invoke :read [13 nil] INFO [2022-03-13 17:11:28,018] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:28,026] jepsen worker 21 - jepsen.util 741 :invoke :write [7 13] INFO [2022-03-13 17:11:28,048] jepsen worker 36 - jepsen.util 2016 :ok :read [12 20] INFO [2022-03-13 17:11:28,048] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,048] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,049] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:28,049] jepsen worker 79 - jepsen.util 889 :invoke :write [26 18] INFO [2022-03-13 17:11:28,049] jepsen worker 36 - jepsen.util 2016 :invoke :write [12 31] INFO [2022-03-13 17:11:28,057] jepsen worker 51 - jepsen.util 591 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,058] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,058] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:28,058] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:28,090] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,091] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,091] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:28,091] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:11:28,100] jepsen worker 48 - jepsen.util 1668 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:11:28,101] jepsen worker 48 - jepsen.util 1758 :invoke :read [16 nil] INFO [2022-03-13 17:11:28,148] jepsen worker 48 - jepsen.util 1758 :ok :read [16 :nil] INFO [2022-03-13 17:11:28,148] jepsen worker 48 - jepsen.util 1758 :invoke :read [16 nil] INFO [2022-03-13 17:11:28,185] jepsen worker 48 - jepsen.util 1758 :ok :read [16 :nil] INFO [2022-03-13 17:11:28,186] jepsen worker 48 - jepsen.util 1758 :invoke :write [16 42] INFO [2022-03-13 17:11:28,248] jepsen worker 53 - jepsen.util 503 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,248] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:28,260] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,260] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,261] jepsen worker 76 - jepsen.util 616 :invoke :write [25 19] INFO [2022-03-13 17:11:28,261] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:11:28,302] jepsen worker 5 - jepsen.util 1985 :info :write [1 10] :SyncWriteAmbiguous INFO [2022-03-13 17:11:28,303] jepsen worker 5 - jepsen.util 2075 :invoke :write [1 14] INFO [2022-03-13 17:11:28,336] jepsen worker 89 - jepsen.util 2159 :ok :read [29 10] INFO [2022-03-13 17:11:28,336] jepsen worker 89 - jepsen.util 2159 :invoke :read [29 nil] INFO [2022-03-13 17:11:28,363] jepsen worker 89 - jepsen.util 2159 :ok :read [29 10] INFO [2022-03-13 17:11:28,364] jepsen worker 89 - jepsen.util 2159 :invoke :write [29 10] INFO [2022-03-13 17:11:28,460] jepsen worker 79 - jepsen.util 889 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,460] jepsen worker 79 - jepsen.util 889 :invoke :write [26 18] INFO [2022-03-13 17:11:28,470] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,470] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,470] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:11:28,471] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:11:28,488] jepsen worker 44 - jepsen.util 584 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,489] jepsen worker 44 - jepsen.util 584 :invoke :write [14 48] INFO [2022-03-13 17:11:28,492] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3167155425219942}, :ns_1@172.23.105.187 {:progress 0.456140350877193}, :ns_1@172.23.107.227 {:progress 0.3167155425219942}, :ns_1@172.23.107.228 {:progress 0.3529411764705882}} INFO [2022-03-13 17:11:28,492] jepsen worker nemesis - couchbase.util Rebalance check count: 199 INFO [2022-03-13 17:11:28,492] jepsen worker nemesis - couchbase.util Rebalance stuck count: 127 INFO [2022-03-13 17:11:28,505] jepsen worker 76 - jepsen.util 616 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,505] jepsen worker 76 - jepsen.util 616 :invoke :write [25 15] INFO [2022-03-13 17:11:28,540] jepsen worker 83 - jepsen.util 1163 :info :write [27 18] :RequestTimeoutException INFO [2022-03-13 17:11:28,541] jepsen worker 83 - jepsen.util 1253 :invoke :write [27 26] INFO [2022-03-13 17:11:28,586] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,586] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:28,593] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,594] jepsen worker 78 - jepsen.util 528 :invoke :write [26 5] INFO [2022-03-13 17:11:28,600] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,601] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:28,626] jepsen worker 49 - jepsen.util 1399 :info :write [16 23] :RequestTimeoutException INFO [2022-03-13 17:11:28,626] jepsen worker 49 - jepsen.util 1489 :invoke :read [16 nil] INFO [2022-03-13 17:11:28,687] jepsen worker 61 - jepsen.util 601 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,687] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:28,691] jepsen worker 79 - jepsen.util 889 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,691] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:11:28,734] jepsen worker 76 - jepsen.util 616 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,735] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,735] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] INFO [2022-03-13 17:11:28,736] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:28,819] jepsen worker 81 - jepsen.util 2061 :ok :read [27 4] INFO [2022-03-13 17:11:28,819] jepsen worker 81 - jepsen.util 2061 :invoke :write [27 46] INFO [2022-03-13 17:11:28,868] jepsen worker 77 - jepsen.util 707 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:28,869] jepsen worker 77 - jepsen.util 707 :invoke :write [25 25] INFO [2022-03-13 17:11:28,873] jepsen worker 85 - jepsen.util 1885 :ok :read [28 44] INFO [2022-03-13 17:11:28,873] jepsen worker 85 - jepsen.util 1885 :invoke :write [28 23] INFO [2022-03-13 17:11:28,958] jepsen worker 38 - jepsen.util 2108 :info :write [12 24] :RequestTimeoutException INFO [2022-03-13 17:11:28,959] jepsen worker 38 - jepsen.util 2198 :invoke :read [12 nil] INFO [2022-03-13 17:11:28,986] jepsen worker 38 - jepsen.util 2198 :ok :read [12 20] INFO [2022-03-13 17:11:28,987] jepsen worker 38 - jepsen.util 2198 :invoke :write [12 31] INFO [2022-03-13 17:11:29,112] jepsen worker 21 - jepsen.util 741 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,112] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,112] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:29,113] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:11:29,131] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,131] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,132] jepsen worker 79 - jepsen.util 889 :invoke :write [26 44] INFO [2022-03-13 17:11:29,132] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:29,182] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,182] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,182] jepsen worker 76 - jepsen.util 616 :invoke :write [25 40] INFO [2022-03-13 17:11:29,183] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:11:29,201] jepsen worker 26 - jepsen.util 476 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,202] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:29,202] jepsen worker 42 - jepsen.util 402 :invoke :write [14 17] INFO [2022-03-13 17:11:29,203] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:29,320] jepsen worker 4 - jepsen.util 1894 :info :write [1 34] :RequestTimeoutException INFO [2022-03-13 17:11:29,320] jepsen worker 4 - jepsen.util 1984 :invoke :write [1 5] INFO [2022-03-13 17:11:29,412] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,413] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:29,464] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,466] jepsen worker 79 - jepsen.util 889 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,467] jepsen worker 12 - jepsen.util 732 :invoke :write [4 6] INFO [2022-03-13 17:11:29,467] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:29,498] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3225806451612904}, :ns_1@172.23.105.187 {:progress 0.456140350877193}, :ns_1@172.23.107.227 {:progress 0.3225806451612904}, :ns_1@172.23.107.228 {:progress 0.3647058823529412}} INFO [2022-03-13 17:11:29,498] jepsen worker nemesis - couchbase.util Rebalance check count: 200 INFO [2022-03-13 17:11:29,498] jepsen worker nemesis - couchbase.util Rebalance stuck count: 127 INFO [2022-03-13 17:11:29,500] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,501] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:29,608] jepsen worker 76 - jepsen.util 616 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,608] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:29,609] jepsen worker 43 - jepsen.util 493 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,609] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:11:29,620] jepsen worker 16 - jepsen.util 1546 :info :write [5 16] :RequestTimeoutException INFO [2022-03-13 17:11:29,621] jepsen worker 16 - jepsen.util 1636 :invoke :read [5 nil] INFO [2022-03-13 17:11:29,639] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,640] jepsen worker 51 - jepsen.util 591 :invoke :write [17 23] INFO [2022-03-13 17:11:29,643] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,643] jepsen worker 60 - jepsen.util 600 :invoke :write [20 36] INFO [2022-03-13 17:11:29,654] jepsen worker 16 - jepsen.util 1636 :ok :read [5 43] INFO [2022-03-13 17:11:29,655] jepsen worker 16 - jepsen.util 1636 :invoke :read [5 nil] INFO [2022-03-13 17:11:29,666] jepsen worker 1 - jepsen.util 1441 :info :write [0 16] :RequestTimeoutException INFO [2022-03-13 17:11:29,667] jepsen worker 1 - jepsen.util 1531 :invoke :read [0 nil] INFO [2022-03-13 17:11:29,680] jepsen worker 16 - jepsen.util 1636 :ok :read [5 43] INFO [2022-03-13 17:11:29,680] jepsen worker 16 - jepsen.util 1636 :invoke :write [5 9] INFO [2022-03-13 17:11:29,714] jepsen worker 27 - jepsen.util 1917 :info :write [9 31] :RequestTimeoutException INFO [2022-03-13 17:11:29,714] jepsen worker 27 - jepsen.util 2007 :invoke :read [9 nil] INFO [2022-03-13 17:11:29,824] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,824] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,825] jepsen worker 76 - jepsen.util 616 :invoke :write [25 21] INFO [2022-03-13 17:11:29,825] jepsen worker 79 - jepsen.util 889 :invoke :write [26 39] INFO [2022-03-13 17:11:29,833] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:29,833] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:29,834] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:29,834] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:30,503] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3225806451612904}, :ns_1@172.23.105.187 {:progress 0.456140350877193}, :ns_1@172.23.107.227 {:progress 0.3225806451612904}, :ns_1@172.23.107.228 {:progress 0.3647058823529412}} INFO [2022-03-13 17:11:30,503] jepsen worker nemesis - couchbase.util Rebalance check count: 201 INFO [2022-03-13 17:11:30,503] jepsen worker nemesis - couchbase.util Rebalance stuck count: 128 INFO [2022-03-13 17:11:30,820] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,820] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,820] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,820] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:30,821] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:30,821] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:30,823] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:30,824] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,824] jepsen worker 21 - jepsen.util 741 :invoke :write [7 46] INFO [2022-03-13 17:11:30,825] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:11:30,841] jepsen worker 8 - jepsen.util 728 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,841] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:11:30,845] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,845] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:30,850] jepsen worker 7 - jepsen.util 547 :info :write [2 35] :RequestTimeoutException INFO [2022-03-13 17:11:30,851] jepsen worker 7 - jepsen.util 637 :invoke :write [2 42] INFO [2022-03-13 17:11:30,861] jepsen worker 58 - jepsen.util 1588 :info :write [19 41] :SyncWriteAmbiguous INFO [2022-03-13 17:11:30,862] jepsen worker 58 - jepsen.util 1678 :invoke :read [19 nil] INFO [2022-03-13 17:11:30,889] jepsen worker 76 - jepsen.util 616 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,890] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:30,893] jepsen worker 77 - jepsen.util 707 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,894] jepsen worker 77 - jepsen.util 707 :invoke :write [25 19] INFO [2022-03-13 17:11:30,919] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,919] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,919] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,919] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,920] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:30,920] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:30,921] jepsen worker 69 - jepsen.util 609 :invoke :write [23 24] INFO [2022-03-13 17:11:30,922] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:11:30,982] jepsen worker 79 - jepsen.util 889 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:30,984] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:31,032] jepsen worker 6 - jepsen.util 636 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,032] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,032] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:11:31,033] jepsen worker 61 - jepsen.util 601 :invoke :write [20 33] INFO [2022-03-13 17:11:31,033] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,034] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:31,096] jepsen worker 12 - jepsen.util 732 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,097] jepsen worker 12 - jepsen.util 732 :invoke :write [4 0] INFO [2022-03-13 17:11:31,106] jepsen worker 10 - jepsen.util 1720 :info :write [3 37] :RequestTimeoutException INFO [2022-03-13 17:11:31,106] jepsen worker 10 - jepsen.util 1810 :invoke :read [3 nil] INFO [2022-03-13 17:11:31,137] jepsen worker 10 - jepsen.util 1810 :ok :read [3 25] INFO [2022-03-13 17:11:31,137] jepsen worker 10 - jepsen.util 1810 :invoke :write [3 28] INFO [2022-03-13 17:11:31,166] jepsen worker 68 - jepsen.util 1778 :info :write [22 49] :RequestTimeoutException INFO [2022-03-13 17:11:31,167] jepsen worker 68 - jepsen.util 1868 :invoke :write [22 10] INFO [2022-03-13 17:11:31,232] jepsen worker 21 - jepsen.util 741 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,232] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,232] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,233] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:11:31,233] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:31,233] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:11:31,248] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,248] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,248] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,249] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:31,249] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:11:31,250] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:31,251] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,251] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,255] jepsen worker 79 - jepsen.util 889 :invoke :write [26 47] INFO [2022-03-13 17:11:31,256] jepsen worker 76 - jepsen.util 616 :invoke :write [25 1] INFO [2022-03-13 17:11:31,277] jepsen worker 69 - jepsen.util 609 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,278] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:31,282] jepsen worker 72 - jepsen.util 1692 :info :write [24 3] :SyncWriteAmbiguous INFO [2022-03-13 17:11:31,283] jepsen worker 72 - jepsen.util 1782 :invoke :write [24 26] INFO [2022-03-13 17:11:31,323] jepsen worker 62 - jepsen.util 512 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,323] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,324] jepsen worker 62 - jepsen.util 512 :invoke :write [20 18] INFO [2022-03-13 17:11:31,324] jepsen worker 26 - jepsen.util 476 :invoke :write [8 13] INFO [2022-03-13 17:11:31,369] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,369] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,369] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:31,370] jepsen worker 24 - jepsen.util 564 :invoke :write [8 1] INFO [2022-03-13 17:11:31,370] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:11:31,371] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:31,444] jepsen worker 23 - jepsen.util 563 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,444] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,444] jepsen worker 52 - jepsen.util 592 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,445] jepsen worker 34 - jepsen.util 754 :invoke :write [11 47] INFO [2022-03-13 17:11:31,446] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:31,447] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:31,505] jepsen worker 63 - jepsen.util 1683 :info :write [21 48] :SyncWriteAmbiguous INFO [2022-03-13 17:11:31,506] jepsen worker 63 - jepsen.util 1773 :invoke :read [21 nil] INFO [2022-03-13 17:11:31,511] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3225806451612904}, :ns_1@172.23.105.187 {:progress 0.4736842105263158}, :ns_1@172.23.107.227 {:progress 0.3225806451612904}, :ns_1@172.23.107.228 {:progress 0.3647058823529412}} INFO [2022-03-13 17:11:31,511] jepsen worker nemesis - couchbase.util Rebalance check count: 202 INFO [2022-03-13 17:11:31,511] jepsen worker nemesis - couchbase.util Rebalance stuck count: 128 INFO [2022-03-13 17:11:31,586] jepsen worker 9 - jepsen.util 1629 :info :write [3 36] :RequestTimeoutException INFO [2022-03-13 17:11:31,587] jepsen worker 9 - jepsen.util 1719 :invoke :read [3 nil] INFO [2022-03-13 17:11:31,689] jepsen worker 47 - jepsen.util 1937 :ok :read [15 48] INFO [2022-03-13 17:11:31,691] jepsen worker 47 - jepsen.util 1937 :invoke :read [15 nil] INFO [2022-03-13 17:11:31,784] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,785] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:11:31,786] jepsen worker 31 - jepsen.util 1651 :info :write [10 17] :RequestTimeoutException INFO [2022-03-13 17:11:31,787] jepsen worker 31 - jepsen.util 1741 :invoke :write [10 37] INFO [2022-03-13 17:11:31,825] jepsen worker 42 - jepsen.util 402 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,826] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:11:31,826] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:31,827] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:11:31,828] jepsen worker 46 - jepsen.util 1756 :info :write [15 19] :RequestTimeoutException INFO [2022-03-13 17:11:31,829] jepsen worker 46 - jepsen.util 1846 :invoke :write [15 21] INFO [2022-03-13 17:11:31,884] jepsen worker 51 - jepsen.util 591 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,884] jepsen worker 51 - jepsen.util 591 :invoke :write [17 43] INFO [2022-03-13 17:11:31,899] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,900] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:31,925] jepsen worker 79 - jepsen.util 889 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,925] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:11:31,989] jepsen worker 14 - jepsen.util 284 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,990] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:31,993] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,994] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:31,994] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:11:31,994] jepsen worker 53 - jepsen.util 503 :invoke :write [17 9] INFO [2022-03-13 17:11:31,996] jepsen worker 18 - jepsen.util 1548 :info :write [6 18] :RequestTimeoutException INFO [2022-03-13 17:11:31,996] jepsen worker 18 - jepsen.util 1638 :invoke :read [6 nil] INFO [2022-03-13 17:11:32,026] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,026] jepsen worker 76 - jepsen.util 616 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,026] jepsen worker 80 - jepsen.util 710 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,027] jepsen worker 35 - jepsen.util 845 :invoke :write [11 9] INFO [2022-03-13 17:11:32,027] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:32,028] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:32,053] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,054] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:32,054] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,054] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:32,055] jepsen worker 21 - jepsen.util 741 :invoke :write [7 14] INFO [2022-03-13 17:11:32,055] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:32,060] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,061] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:11:32,148] jepsen worker 28 - jepsen.util 1828 :info :write [9 0] :RequestTimeoutException INFO [2022-03-13 17:11:32,149] jepsen worker 28 - jepsen.util 1918 :invoke :write [9 32] INFO [2022-03-13 17:11:32,168] jepsen worker 20 - jepsen.util 2090 :info :write [6 44] :RequestTimeoutException INFO [2022-03-13 17:11:32,169] jepsen worker 20 - jepsen.util 2180 :invoke :write [6 11] INFO [2022-03-13 17:11:32,186] jepsen worker 87 - jepsen.util 2337 :info :write [29 20] :RequestTimeoutException INFO [2022-03-13 17:11:32,187] jepsen worker 87 - jepsen.util 2427 :invoke :read [29 nil] INFO [2022-03-13 17:11:32,228] jepsen worker 56 - jepsen.util 1676 :info :write [18 32] :RequestTimeoutException INFO [2022-03-13 17:11:32,229] jepsen worker 56 - jepsen.util 1766 :invoke :read [18 nil] INFO [2022-03-13 17:11:32,234] jepsen worker 60 - jepsen.util 600 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,234] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:32,244] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,245] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,245] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:32,246] jepsen worker 76 - jepsen.util 616 :invoke :write [25 23] INFO [2022-03-13 17:11:32,299] jepsen worker 56 - jepsen.util 1766 :ok :read [18 21] INFO [2022-03-13 17:11:32,299] jepsen worker 56 - jepsen.util 1766 :invoke :write [18 25] INFO [2022-03-13 17:11:32,308] jepsen worker 24 - jepsen.util 564 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,309] jepsen worker 24 - jepsen.util 564 :invoke :write [8 48] INFO [2022-03-13 17:11:32,358] jepsen worker 19 - jepsen.util 1639 :info :write [6 1] :RequestTimeoutException INFO [2022-03-13 17:11:32,359] jepsen worker 19 - jepsen.util 1729 :invoke :write [6 21] INFO [2022-03-13 17:11:32,451] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,451] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,451] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:32,452] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:11:32,455] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,455] jepsen worker 7 - jepsen.util 637 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,458] jepsen worker 6 - jepsen.util 636 :invoke :write [2 40] INFO [2022-03-13 17:11:32,459] jepsen worker 7 - jepsen.util 637 :invoke :write [2 2] INFO [2022-03-13 17:11:32,498] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,498] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,498] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,498] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,499] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:32,499] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:32,500] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:32,501] jepsen worker 69 - jepsen.util 609 :invoke :write [23 44] INFO [2022-03-13 17:11:32,517] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3225806451612904}, :ns_1@172.23.105.187 {:progress 0.4736842105263158}, :ns_1@172.23.107.227 {:progress 0.3225806451612904}, :ns_1@172.23.107.228 {:progress 0.3647058823529412}} INFO [2022-03-13 17:11:32,517] jepsen worker nemesis - couchbase.util Rebalance check count: 203 INFO [2022-03-13 17:11:32,517] jepsen worker nemesis - couchbase.util Rebalance stuck count: 129 INFO [2022-03-13 17:11:32,651] jepsen worker 77 - jepsen.util 707 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,652] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:32,656] jepsen worker 22 - jepsen.util 652 :ok :read [7 :nil] INFO [2022-03-13 17:11:32,660] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:11:32,656] jepsen worker 70 - jepsen.util 700 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,657] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,657] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,665] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:32,657] jepsen worker 76 - jepsen.util 616 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,665] jepsen worker 13 - jepsen.util 643 :invoke :write [4 16] INFO [2022-03-13 17:11:32,665] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:32,665] jepsen worker 76 - jepsen.util 616 :invoke :write [25 36] INFO [2022-03-13 17:11:32,739] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,739] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,739] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:11:32,740] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:11:32,743] jepsen worker 21 - jepsen.util 741 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,744] jepsen worker 21 - jepsen.util 741 :invoke :write [7 44] INFO [2022-03-13 17:11:32,765] jepsen worker 61 - jepsen.util 601 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,766] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:32,847] jepsen worker 24 - jepsen.util 564 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,847] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,847] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,847] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,848] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:11:32,848] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:32,849] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:32,849] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:32,867] jepsen worker 76 - jepsen.util 616 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,868] jepsen worker 76 - jepsen.util 616 :invoke :write [25 33] INFO [2022-03-13 17:11:32,872] jepsen worker 12 - jepsen.util 732 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,873] jepsen worker 12 - jepsen.util 732 :invoke :write [4 36] INFO [2022-03-13 17:11:32,901] jepsen worker 35 - jepsen.util 845 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,901] jepsen worker 78 - jepsen.util 528 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,901] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:32,901] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:32,902] jepsen worker 78 - jepsen.util 528 :invoke :write [26 3] INFO [2022-03-13 17:11:32,903] jepsen worker 52 - jepsen.util 592 :invoke :write [17 46] INFO [2022-03-13 17:11:32,906] jepsen worker 54 - jepsen.util 1494 :info :write [18 41] :RequestTimeoutException INFO [2022-03-13 17:11:32,907] jepsen worker 54 - jepsen.util 1584 :invoke :read [18 nil] INFO [2022-03-13 17:11:33,064] jepsen worker 69 - jepsen.util 609 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,065] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:33,131] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,131] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:33,134] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,134] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,135] jepsen worker 26 - jepsen.util 476 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,135] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:33,135] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:33,136] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:33,181] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,182] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,181] jepsen worker 21 - jepsen.util 741 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,181] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,182] jepsen worker 71 - jepsen.util 521 :invoke :write [23 12] INFO [2022-03-13 17:11:33,183] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:11:33,183] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:33,184] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:11:33,297] jepsen worker 22 - jepsen.util 652 :ok :read [7 :nil] INFO [2022-03-13 17:11:33,297] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,298] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,298] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,297] jepsen worker 34 - jepsen.util 754 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,298] jepsen worker 80 - jepsen.util 710 :invoke :write [26 19] INFO [2022-03-13 17:11:33,299] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:11:33,300] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:33,300] jepsen worker 70 - jepsen.util 700 :invoke :write [23 35] INFO [2022-03-13 17:11:33,301] jepsen worker 35 - jepsen.util 845 :invoke :write [11 9] INFO [2022-03-13 17:11:33,301] jepsen worker 44 - jepsen.util 584 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,302] jepsen worker 44 - jepsen.util 584 :invoke :write [14 23] INFO [2022-03-13 17:11:33,361] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:33,361] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,361] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,363] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:33,363] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:33,364] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:33,421] jepsen worker 50 - jepsen.util 1760 :info :write [16 18] :SyncWriteAmbiguous INFO [2022-03-13 17:11:33,422] jepsen worker 50 - jepsen.util 1850 :invoke :write [16 1] INFO [2022-03-13 17:11:33,446] jepsen worker 2 - jepsen.util 2072 :info :write [0 31] :RequestTimeoutException INFO [2022-03-13 17:11:33,447] jepsen worker 2 - jepsen.util 2162 :invoke :read [0 nil] INFO [2022-03-13 17:11:33,496] jepsen worker 13 - jepsen.util 643 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,497] jepsen worker 13 - jepsen.util 643 :invoke :write [4 44] INFO [2022-03-13 17:11:33,497] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,498] jepsen worker 25 - jepsen.util 565 :invoke :write [8 10] INFO [2022-03-13 17:11:33,523] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3225806451612904}, :ns_1@172.23.105.187 {:progress 0.4736842105263158}, :ns_1@172.23.107.227 {:progress 0.3225806451612904}, :ns_1@172.23.107.228 {:progress 0.3647058823529412}} INFO [2022-03-13 17:11:33,524] jepsen worker nemesis - couchbase.util Rebalance check count: 204 INFO [2022-03-13 17:11:33,524] jepsen worker nemesis - couchbase.util Rebalance stuck count: 130 INFO [2022-03-13 17:11:33,534] jepsen worker 45 - jepsen.util 2025 :info :write [15 8] :RequestTimeoutException INFO [2022-03-13 17:11:33,534] jepsen worker 45 - jepsen.util 2115 :invoke :write [15 26] INFO [2022-03-13 17:11:33,706] jepsen worker 86 - jepsen.util 1526 :ok :read [28 44] INFO [2022-03-13 17:11:33,706] jepsen worker 86 - jepsen.util 1526 :invoke :read [28 nil] INFO [2022-03-13 17:11:33,736] jepsen worker 62 - jepsen.util 512 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,736] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,736] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:33,736] jepsen worker 62 - jepsen.util 512 :invoke :write [20 22] INFO [2022-03-13 17:11:33,736] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:11:33,737] jepsen worker 14 - jepsen.util 284 :invoke :write [4 34] INFO [2022-03-13 17:11:33,737] jepsen worker 42 - jepsen.util 402 :invoke :write [14 13] INFO [2022-03-13 17:11:33,743] jepsen worker 43 - jepsen.util 493 :invoke :write [14 44] INFO [2022-03-13 17:11:33,746] jepsen worker 78 - jepsen.util 528 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,747] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:33,763] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,764] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:11:33,809] jepsen worker 76 - jepsen.util 616 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,809] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,810] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:33,810] jepsen worker 79 - jepsen.util 889 :invoke :write [26 0] INFO [2022-03-13 17:11:33,813] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,813] jepsen worker 33 - jepsen.util 663 :invoke :write [11 44] INFO [2022-03-13 17:11:33,846] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:11:33,846] jepsen worker 51 - jepsen.util 591 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,847] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,847] jepsen worker 51 - jepsen.util 591 :invoke :write [17 47] INFO [2022-03-13 17:11:33,848] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:33,848] jepsen worker 60 - jepsen.util 600 :invoke :write [20 34] INFO [2022-03-13 17:11:33,850] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,851] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:33,863] jepsen worker 52 - jepsen.util 592 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,863] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,864] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:33,865] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:33,901] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,903] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:11:33,904] jepsen worker 25 - jepsen.util 565 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,904] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,905] jepsen worker 76 - jepsen.util 616 :invoke :write [25 16] INFO [2022-03-13 17:11:33,905] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:33,920] jepsen worker 84 - jepsen.util 1974 :ok :read [28 44] INFO [2022-03-13 17:11:33,920] jepsen worker 84 - jepsen.util 1974 :invoke :read [28 nil] INFO [2022-03-13 17:11:33,924] jepsen worker 35 - jepsen.util 845 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,925] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:33,959] jepsen worker 49 - jepsen.util 1489 :ok :read [16 :nil] INFO [2022-03-13 17:11:33,960] jepsen worker 49 - jepsen.util 1489 :invoke :write [16 5] INFO [2022-03-13 17:11:33,984] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:33,986] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:11:33,991] jepsen worker 84 - jepsen.util 1974 :ok :read [28 44] INFO [2022-03-13 17:11:33,991] jepsen worker 84 - jepsen.util 1974 :invoke :read [28 nil] INFO [2022-03-13 17:11:34,050] jepsen worker 71 - jepsen.util 521 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,050] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:34,051] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,050] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,051] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:34,052] jepsen worker 71 - jepsen.util 521 :invoke :write [23 47] INFO [2022-03-13 17:11:34,052] jepsen worker 84 - jepsen.util 1974 :ok :read [28 44] INFO [2022-03-13 17:11:34,052] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:34,053] jepsen worker 84 - jepsen.util 1974 :invoke :read [28 nil] INFO [2022-03-13 17:11:34,053] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:34,055] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,055] jepsen worker 79 - jepsen.util 889 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,055] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:11:34,056] jepsen worker 79 - jepsen.util 889 :invoke :write [26 43] INFO [2022-03-13 17:11:34,101] jepsen worker 84 - jepsen.util 1974 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,102] jepsen worker 84 - jepsen.util 1974 :invoke :write [28 32] INFO [2022-03-13 17:11:34,128] jepsen worker 12 - jepsen.util 732 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,129] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:34,188] jepsen worker 33 - jepsen.util 663 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,189] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,205] jepsen worker 76 - jepsen.util 616 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,205] jepsen worker 76 - jepsen.util 616 :invoke :write [25 15] INFO [2022-03-13 17:11:34,244] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,244] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,244] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:34,244] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,244] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:34,245] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:34,246] jepsen worker 24 - jepsen.util 564 :invoke :write [8 38] INFO [2022-03-13 17:11:34,246] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:34,287] jepsen worker 80 - jepsen.util 710 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,287] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,287] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,288] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:11:34,288] jepsen worker 80 - jepsen.util 710 :invoke :write [26 42] INFO [2022-03-13 17:11:34,295] jepsen worker 35 - jepsen.util 845 :invoke :write [11 45] INFO [2022-03-13 17:11:34,343] jepsen worker 82 - jepsen.util 1702 :info :write [27 29] :RequestTimeoutException INFO [2022-03-13 17:11:34,344] jepsen worker 82 - jepsen.util 1792 :invoke :read [27 nil] INFO [2022-03-13 17:11:34,396] jepsen worker 53 - jepsen.util 503 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,396] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,397] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:34,398] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:34,399] jepsen worker 71 - jepsen.util 521 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,400] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,400] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:34,400] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:34,400] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,400] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,401] jepsen worker 21 - jepsen.util 741 :invoke :write [7 36] INFO [2022-03-13 17:11:34,402] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:34,402] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,402] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:11:34,434] jepsen worker 15 - jepsen.util 1815 :info :write [5 30] :RequestTimeoutException INFO [2022-03-13 17:11:34,434] jepsen worker 15 - jepsen.util 1905 :invoke :read [5 nil] INFO [2022-03-13 17:11:34,493] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,494] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:11:34,508] jepsen worker 44 - jepsen.util 584 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,509] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:34,511] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:11:34,511] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,512] jepsen worker 77 - jepsen.util 707 :invoke :write [25 24] INFO [2022-03-13 17:11:34,513] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:34,530] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3225806451612904}, :ns_1@172.23.105.187 {:progress 0.47953216374269}, :ns_1@172.23.107.227 {:progress 0.3225806451612904}, :ns_1@172.23.107.228 {:progress 0.3705882352941177}} INFO [2022-03-13 17:11:34,530] jepsen worker nemesis - couchbase.util Rebalance check count: 205 INFO [2022-03-13 17:11:34,530] jepsen worker nemesis - couchbase.util Rebalance stuck count: 130 INFO [2022-03-13 17:11:34,538] jepsen worker 66 - jepsen.util 1596 :info :write [22 14] :RequestTimeoutException INFO [2022-03-13 17:11:34,538] jepsen worker 66 - jepsen.util 1686 :invoke :read [22 nil] INFO [2022-03-13 17:11:34,548] jepsen worker 39 - jepsen.util 1749 :info :write [13 22] :RequestTimeoutException INFO [2022-03-13 17:11:34,548] jepsen worker 39 - jepsen.util 1839 :invoke :read [13 nil] INFO [2022-03-13 17:11:34,561] jepsen worker 24 - jepsen.util 564 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,561] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,562] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,562] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,563] jepsen worker 71 - jepsen.util 521 :invoke :write [23 3] INFO [2022-03-13 17:11:34,564] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:11:34,564] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:11:34,565] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,580] jepsen worker 70 - jepsen.util 700 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,581] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:11:34,623] jepsen worker 7 - jepsen.util 637 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,623] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:11:34,624] jepsen worker 79 - jepsen.util 889 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,625] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:34,629] jepsen worker 11 - jepsen.util 1811 :info :write [3 34] :SyncWriteAmbiguous INFO [2022-03-13 17:11:34,630] jepsen worker 21 - jepsen.util 741 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,630] jepsen worker 11 - jepsen.util 1901 :invoke :write [3 13] INFO [2022-03-13 17:11:34,631] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:34,645] jepsen worker 37 - jepsen.util 1837 :ok :read [12 20] INFO [2022-03-13 17:11:34,646] jepsen worker 37 - jepsen.util 1837 :invoke :write [12 5] INFO [2022-03-13 17:11:34,704] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,705] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:34,774] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,774] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,774] jepsen worker 13 - jepsen.util 643 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,774] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,775] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,775] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:34,776] jepsen worker 78 - jepsen.util 528 :invoke :write [26 30] INFO [2022-03-13 17:11:34,777] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:34,788] jepsen worker 88 - jepsen.util 1798 :ok :read [29 10] INFO [2022-03-13 17:11:34,788] jepsen worker 88 - jepsen.util 1798 :invoke :read [29 nil] INFO [2022-03-13 17:11:34,791] jepsen worker 51 - jepsen.util 591 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,792] jepsen worker 51 - jepsen.util 591 :invoke :write [17 21] INFO [2022-03-13 17:11:34,803] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,803] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,804] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,804] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:34,812] jepsen worker 88 - jepsen.util 1798 :ok :read [29 10] INFO [2022-03-13 17:11:34,812] jepsen worker 88 - jepsen.util 1798 :invoke :read [29 nil] INFO [2022-03-13 17:11:34,830] jepsen worker 65 - jepsen.util 2045 :info :write [21 34] :RequestTimeoutException INFO [2022-03-13 17:11:34,830] jepsen worker 29 - jepsen.util 1829 :info :write [9 13] :RequestTimeoutException INFO [2022-03-13 17:11:34,830] jepsen worker 74 - jepsen.util 2144 :info :write [24 42] :RequestTimeoutException INFO [2022-03-13 17:11:34,831] jepsen worker 32 - jepsen.util 1562 :info :write [10 43] :RequestTimeoutException INFO [2022-03-13 17:11:34,832] jepsen worker 74 - jepsen.util 2234 :invoke :write [24 17] INFO [2022-03-13 17:11:34,833] jepsen worker 29 - jepsen.util 1919 :invoke :write [9 43] INFO [2022-03-13 17:11:34,834] jepsen worker 65 - jepsen.util 2135 :invoke :write [21 18] INFO [2022-03-13 17:11:34,836] jepsen worker 32 - jepsen.util 1652 :invoke :read [10 nil] INFO [2022-03-13 17:11:34,840] jepsen worker 73 - jepsen.util 1783 :info :write [24 34] :RequestTimeoutException INFO [2022-03-13 17:11:34,841] jepsen worker 73 - jepsen.util 1873 :invoke :write [24 49] INFO [2022-03-13 17:11:34,842] jepsen worker 63 - jepsen.util 1773 :ok :read [21 :nil] INFO [2022-03-13 17:11:34,843] jepsen worker 63 - jepsen.util 1773 :invoke :read [21 nil] INFO [2022-03-13 17:11:34,845] jepsen worker 88 - jepsen.util 1798 :ok :read [29 10] INFO [2022-03-13 17:11:34,845] jepsen worker 88 - jepsen.util 1798 :invoke :write [29 46] INFO [2022-03-13 17:11:34,855] jepsen worker 27 - jepsen.util 2007 :ok :read [9 26] INFO [2022-03-13 17:11:34,857] jepsen worker 27 - jepsen.util 2007 :invoke :read [9 nil] INFO [2022-03-13 17:11:34,924] jepsen worker 63 - jepsen.util 1773 :ok :read [21 :nil] INFO [2022-03-13 17:11:34,925] jepsen worker 63 - jepsen.util 1773 :invoke :write [21 27] INFO [2022-03-13 17:11:34,927] jepsen worker 35 - jepsen.util 845 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,927] jepsen worker 76 - jepsen.util 616 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,927] jepsen worker 22 - jepsen.util 652 :ok :read [7 :nil] INFO [2022-03-13 17:11:34,928] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,928] jepsen worker 76 - jepsen.util 616 :invoke :write [25 28] INFO [2022-03-13 17:11:34,929] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:11:34,936] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,937] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,939] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,939] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,941] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,942] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,944] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,944] jepsen worker 27 - jepsen.util 2007 :ok :read [9 26] INFO [2022-03-13 17:11:34,945] jepsen worker 27 - jepsen.util 2007 :invoke :read [9 nil] INFO [2022-03-13 17:11:34,946] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:34,948] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:34,948] jepsen worker 35 - jepsen.util 845 :invoke :write [11 14] INFO [2022-03-13 17:11:35,008] jepsen worker 71 - jepsen.util 521 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,008] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:11:35,010] jepsen worker 6 - jepsen.util 636 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,010] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,010] jepsen worker 6 - jepsen.util 636 :invoke :write [2 20] INFO [2022-03-13 17:11:35,011] jepsen worker 26 - jepsen.util 476 :invoke :write [8 19] INFO [2022-03-13 17:11:35,080] jepsen worker 27 - jepsen.util 2007 :ok :read [9 26] INFO [2022-03-13 17:11:35,080] jepsen worker 27 - jepsen.util 2007 :invoke :write [9 6] INFO [2022-03-13 17:11:35,124] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,124] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,125] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:11:35,126] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:11:35,144] jepsen worker 80 - jepsen.util 710 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,145] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:35,190] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,190] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:35,191] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:35,191] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:35,193] jepsen worker 30 - jepsen.util 1650 :info :write [10 33] :RequestTimeoutException INFO [2022-03-13 17:11:35,194] jepsen worker 30 - jepsen.util 1740 :invoke :write [10 41] INFO [2022-03-13 17:11:35,196] jepsen worker 60 - jepsen.util 600 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,196] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,197] jepsen worker 60 - jepsen.util 600 :invoke :write [20 44] INFO [2022-03-13 17:11:35,198] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:35,199] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:35,199] jepsen worker 23 - jepsen.util 563 :ok :read [7 :nil] INFO [2022-03-13 17:11:35,200] jepsen worker 12 - jepsen.util 732 :invoke :write [4 21] INFO [2022-03-13 17:11:35,200] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:11:35,224] jepsen worker 76 - jepsen.util 616 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,225] jepsen worker 76 - jepsen.util 616 :invoke :write [25 15] INFO [2022-03-13 17:11:35,264] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,264] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:35,508] jepsen worker 32 - jepsen.util 1652 :ok :read [10 :nil] INFO [2022-03-13 17:11:35,511] jepsen worker 32 - jepsen.util 1652 :invoke :read [10 nil] INFO [2022-03-13 17:11:35,513] jepsen worker 32 - jepsen.util 1652 :ok :read [10 :nil] INFO [2022-03-13 17:11:35,513] jepsen worker 32 - jepsen.util 1652 :invoke :read [10 nil] INFO [2022-03-13 17:11:35,515] jepsen worker 32 - jepsen.util 1652 :ok :read [10 :nil] INFO [2022-03-13 17:11:35,515] jepsen worker 18 - jepsen.util 1638 :ok :read [6 16] INFO [2022-03-13 17:11:35,515] jepsen worker 32 - jepsen.util 1652 :invoke :read [10 nil] INFO [2022-03-13 17:11:35,516] jepsen worker 18 - jepsen.util 1638 :invoke :read [6 nil] INFO [2022-03-13 17:11:35,517] jepsen worker 32 - jepsen.util 1652 :ok :read [10 :nil] INFO [2022-03-13 17:11:35,518] jepsen worker 32 - jepsen.util 1652 :invoke :write [10 8] INFO [2022-03-13 17:11:35,527] jepsen worker 54 - jepsen.util 1584 :ok :read [18 21] INFO [2022-03-13 17:11:35,528] jepsen worker 54 - jepsen.util 1584 :invoke :read [18 nil] INFO [2022-03-13 17:11:35,535] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3225806451612904}, :ns_1@172.23.105.187 {:progress 0.47953216374269}, :ns_1@172.23.107.227 {:progress 0.3225806451612904}, :ns_1@172.23.107.228 {:progress 0.3705882352941177}} INFO [2022-03-13 17:11:35,536] jepsen worker nemesis - couchbase.util Rebalance check count: 206 INFO [2022-03-13 17:11:35,536] jepsen worker nemesis - couchbase.util Rebalance stuck count: 131 INFO [2022-03-13 17:11:35,546] jepsen worker 18 - jepsen.util 1638 :ok :read [6 16] INFO [2022-03-13 17:11:35,547] jepsen worker 18 - jepsen.util 1638 :invoke :read [6 nil] INFO [2022-03-13 17:11:35,557] jepsen worker 54 - jepsen.util 1584 :ok :read [18 21] INFO [2022-03-13 17:11:35,558] jepsen worker 54 - jepsen.util 1584 :invoke :write [18 24] INFO [2022-03-13 17:11:35,607] jepsen worker 18 - jepsen.util 1638 :ok :read [6 16] INFO [2022-03-13 17:11:35,608] jepsen worker 18 - jepsen.util 1638 :invoke :write [6 11] INFO [2022-03-13 17:11:35,614] jepsen worker 3 - jepsen.util 1713 :info :write [1 26] :RequestTimeoutException INFO [2022-03-13 17:11:35,614] jepsen worker 3 - jepsen.util 1803 :invoke :read [1 nil] WARN [2022-03-13 17:11:35,824] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9230119,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10118855,"total_server_duration_us":9237538,"encode_duration_us":17265,"operation_id":"0x512e","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10010030},{"operation_name":"upsert","last_dispatch_duration_us":7469673,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9995604,"total_server_duration_us":276343,"encode_duration_us":18101,"operation_id":"0x5132","timeout_ms":10000,"last_server_duration_us":41660,"total_duration_us":10008296},{"operation_name":"upsert","last_dispatch_duration_us":10299928,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10299928,"total_server_duration_us":9060729,"encode_duration_us":19865,"operation_id":"0x4f58","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10007168},{"operation_name":"upsert","last_dispatch_duration_us":7498514,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10074940,"total_server_duration_us":301684,"encode_duration_us":18070,"operation_id":"0x512f","timeout_ms":10000,"last_server_duration_us":56401,"total_duration_us":10006337},{"operation_name":"upsert","last_dispatch_duration_us":86679,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9737510,"total_server_duration_us":195195,"encode_duration_us":22609,"operation_id":"0x4fa4","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10006151},{"operation_name":"upsert","last_dispatch_duration_us":7311776,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9737619,"total_server_duration_us":313689,"encode_duration_us":17241,"operation_id":"0x5130","timeout_ms":10000,"last_server_duration_us":44944,"total_duration_us":10005281},{"operation_name":"upsert","last_dispatch_duration_us":7341168,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9813172,"total_server_duration_us":331485,"encode_duration_us":17759,"operation_id":"0x5131","timeout_ms":10000,"last_server_duration_us":52301,"total_duration_us":10004510},{"operation_name":"upsert","last_dispatch_duration_us":9066461,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9355392,"total_server_duration_us":9229091,"encode_duration_us":16505,"operation_id":"0x4fc6","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9573362},{"operation_name":"get","last_dispatch_duration_us":9122283,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9122283,"total_server_duration_us":28493,"operation_id":"0x4fa2","timeout_ms":30000,"last_server_duration_us":28493,"total_duration_us":9124654},{"operation_name":"get","last_dispatch_duration_us":9117790,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9117790,"total_server_duration_us":28493,"operation_id":"0x4fa1","timeout_ms":30000,"last_server_duration_us":28493,"total_duration_us":9120271}],"total_count":30}} WARN [2022-03-13 17:11:35,824] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10299928,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10299928,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9060729,"encode_duration_us":19865,"operation_id":"0x4f58","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10007168},{"operation_name":"upsert","last_dispatch_duration_us":9230119,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10118855,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9237538,"encode_duration_us":17265,"operation_id":"0x512e","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10010030}],"total_count":2}} INFO [2022-03-13 17:11:35,870] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,870] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,870] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,871] jepsen worker 75 - jepsen.util 435 :invoke :write [25 49] INFO [2022-03-13 17:11:35,871] jepsen worker 69 - jepsen.util 609 :invoke :write [23 24] INFO [2022-03-13 17:11:35,871] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:11:35,876] jepsen worker 62 - jepsen.util 512 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,877] jepsen worker 62 - jepsen.util 512 :invoke :write [20 36] INFO [2022-03-13 17:11:35,881] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,881] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:35,930] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,931] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,931] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:35,932] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:11:35,933] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:35,934] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:11:35,963] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,963] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,964] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,963] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,964] jepsen worker 70 - jepsen.util 700 :invoke :write [23 10] INFO [2022-03-13 17:11:35,965] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:35,965] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:35,966] jepsen worker 52 - jepsen.util 592 :invoke :write [17 30] INFO [2022-03-13 17:11:35,978] jepsen worker 77 - jepsen.util 707 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:35,978] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,042] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,042] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:11:36,058] jepsen worker 64 - jepsen.util 1684 :info :write [21 32] :RequestTimeoutException INFO [2022-03-13 17:11:36,059] jepsen worker 64 - jepsen.util 1774 :invoke :write [21 10] INFO [2022-03-13 17:11:36,072] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,073] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:11:36,084] jepsen worker 35 - jepsen.util 845 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,084] jepsen worker 22 - jepsen.util 652 :ok :read [7 :nil] INFO [2022-03-13 17:11:36,085] jepsen worker 35 - jepsen.util 845 :invoke :write [11 23] INFO [2022-03-13 17:11:36,085] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:11:36,103] jepsen worker 14 - jepsen.util 284 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,104] jepsen worker 14 - jepsen.util 284 :invoke :write [4 24] INFO [2022-03-13 17:11:36,147] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,147] jepsen worker 78 - jepsen.util 528 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,148] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:36,148] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:36,149] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,149] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,149] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:11:36,150] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:36,166] jepsen worker 51 - jepsen.util 591 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,166] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:36,166] jepsen worker 51 - jepsen.util 591 :invoke :write [17 22] INFO [2022-03-13 17:11:36,167] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:36,216] jepsen worker 76 - jepsen.util 616 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,217] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,227] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,228] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:36,230] jepsen worker 69 - jepsen.util 609 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,231] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:11:36,238] jepsen worker 42 - jepsen.util 402 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,238] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,239] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,239] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,240] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:11:36,243] jepsen worker 53 - jepsen.util 503 :invoke :write [17 0] INFO [2022-03-13 17:11:36,243] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:11:36,244] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:36,248] jepsen worker 70 - jepsen.util 700 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,249] jepsen worker 70 - jepsen.util 700 :invoke :write [23 44] INFO [2022-03-13 17:11:36,302] jepsen worker 35 - jepsen.util 845 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,303] jepsen worker 35 - jepsen.util 845 :invoke :write [11 3] INFO [2022-03-13 17:11:36,351] jepsen worker 60 - jepsen.util 600 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,351] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:36,352] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,352] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,352] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,353] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,353] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:11:36,354] jepsen worker 79 - jepsen.util 889 :invoke :write [26 3] INFO [2022-03-13 17:11:36,354] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,354] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,355] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,356] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,357] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,357] jepsen worker 76 - jepsen.util 616 :invoke :write [25 49] INFO [2022-03-13 17:11:36,366] jepsen worker 75 - jepsen.util 435 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,366] jepsen worker 21 - jepsen.util 741 :ok :read [7 :nil] INFO [2022-03-13 17:11:36,367] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,368] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:36,441] jepsen worker 43 - jepsen.util 493 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,441] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:11:36,542] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3255131964809385}, :ns_1@172.23.105.187 {:progress 0.4853801169590644}, :ns_1@172.23.107.227 {:progress 0.3255131964809385}, :ns_1@172.23.107.228 {:progress 0.3764705882352941}} INFO [2022-03-13 17:11:36,542] jepsen worker nemesis - couchbase.util Rebalance check count: 207 INFO [2022-03-13 17:11:36,542] jepsen worker nemesis - couchbase.util Rebalance stuck count: 131 INFO [2022-03-13 17:11:36,551] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,552] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:11:36,554] jepsen worker 52 - jepsen.util 592 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,554] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,554] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,554] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,555] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:36,556] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:36,556] jepsen worker 78 - jepsen.util 528 :invoke :write [26 38] INFO [2022-03-13 17:11:36,557] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:36,616] jepsen worker 79 - jepsen.util 889 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,617] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:36,644] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,645] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:36,661] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,662] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:11:36,683] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,684] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:11:36,769] jepsen worker 76 - jepsen.util 616 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,770] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,870] jepsen worker 40 - jepsen.util 1750 :info :write [13 20] :RequestTimeoutException INFO [2022-03-13 17:11:36,871] jepsen worker 40 - jepsen.util 1840 :invoke :write [13 0] INFO [2022-03-13 17:11:36,876] jepsen worker 26 - jepsen.util 476 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,877] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:36,877] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,878] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:36,889] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,889] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:11:36,908] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,908] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,908] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,910] jepsen worker 79 - jepsen.util 889 :invoke :write [26 26] INFO [2022-03-13 17:11:36,910] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:36,911] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:36,913] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,913] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:36,915] jepsen worker 76 - jepsen.util 616 :invoke :write [25 49] INFO [2022-03-13 17:11:36,916] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:11:37,013] jepsen worker 35 - jepsen.util 845 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,013] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,014] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:37,014] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:37,016] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,016] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,017] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:37,017] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:37,025] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,025] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,025] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:11:37,025] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:37,027] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,029] jepsen worker 35 - jepsen.util 845 :invoke :write [11 24] INFO [2022-03-13 17:11:37,037] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,037] jepsen worker 12 - jepsen.util 732 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,038] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:37,039] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:37,040] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:37,040] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:37,063] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,063] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,063] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,064] jepsen worker 21 - jepsen.util 741 :invoke :write [7 11] INFO [2022-03-13 17:11:37,064] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:11:37,064] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:37,071] jepsen worker 79 - jepsen.util 889 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,071] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:37,148] jepsen worker 6 - jepsen.util 636 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,149] jepsen worker 6 - jepsen.util 636 :invoke :write [2 0] INFO [2022-03-13 17:11:37,180] jepsen worker 76 - jepsen.util 616 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,181] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:37,188] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,189] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] WARN [2022-03-13 17:11:37,226] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9086334,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9926683,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9540869,"encode_duration_us":19201,"operation_id":"0x4fdd","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10008729}],"total_count":1}} INFO [2022-03-13 17:11:37,241] jepsen worker 0 - jepsen.util 1710 :info :write [0 35] :RequestTimeoutException INFO [2022-03-13 17:11:37,241] jepsen worker 0 - jepsen.util 1800 :invoke :read [0 nil] INFO [2022-03-13 17:11:37,250] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,250] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,250] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,250] jepsen worker 17 - jepsen.util 1727 :info :write [5 5] :RequestTimeoutException INFO [2022-03-13 17:11:37,251] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:37,251] jepsen worker 76 - jepsen.util 616 :invoke :write [25 12] INFO [2022-03-13 17:11:37,252] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:11:37,252] jepsen worker 17 - jepsen.util 1817 :invoke :write [5 46] INFO [2022-03-13 17:11:37,300] jepsen worker 57 - jepsen.util 1857 :info :write [19 40] :RequestTimeoutException INFO [2022-03-13 17:11:37,300] jepsen worker 57 - jepsen.util 1947 :invoke :read [19 nil] WARN [2022-03-13 17:11:37,335] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9072172,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10682259,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9878882,"encode_duration_us":19229,"operation_id":"0x4efa","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10002971}],"total_count":1}} INFO [2022-03-13 17:11:37,368] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,369] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:11:37,370] jepsen worker 51 - jepsen.util 591 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,370] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,371] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:37,373] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:37,383] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,384] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:11:37,387] jepsen worker 87 - jepsen.util 2427 :ok :read [29 10] INFO [2022-03-13 17:11:37,387] jepsen worker 87 - jepsen.util 2427 :invoke :read [29 nil] INFO [2022-03-13 17:11:37,406] jepsen worker 3 - jepsen.util 1803 :ok :read [1 42] INFO [2022-03-13 17:11:37,408] jepsen worker 3 - jepsen.util 1803 :invoke :read [1 nil] INFO [2022-03-13 17:11:37,414] jepsen worker 87 - jepsen.util 2427 :ok :read [29 10] INFO [2022-03-13 17:11:37,414] jepsen worker 87 - jepsen.util 2427 :invoke :read [29 nil] WARN [2022-03-13 17:11:37,427] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9086334,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9926683,"total_server_duration_us":9540869,"encode_duration_us":19201,"operation_id":"0x4fdd","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10008729},{"operation_name":"upsert","last_dispatch_duration_us":49902,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1027590,"total_server_duration_us":270551,"encode_duration_us":17152,"operation_id":"0x5110","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10008369},{"operation_name":"upsert","last_dispatch_duration_us":327036,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":704326,"total_server_duration_us":131294,"encode_duration_us":18017,"operation_id":"0x506d","timeout_ms":10000,"last_server_duration_us":66431,"total_duration_us":10008205},{"operation_name":"upsert","last_dispatch_duration_us":428156,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":428156,"total_server_duration_us":47079,"encode_duration_us":23101,"operation_id":"0x507d","timeout_ms":10000,"last_server_duration_us":47079,"total_duration_us":10006216},{"operation_name":"upsert","last_dispatch_duration_us":255125,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9555745,"total_server_duration_us":137456,"encode_duration_us":24339,"operation_id":"0x5078","timeout_ms":10000,"last_server_duration_us":54697,"total_duration_us":10006124},{"operation_name":"upsert","last_dispatch_duration_us":8608728,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9312271,"total_server_duration_us":111576,"encode_duration_us":17996,"operation_id":"0x5079","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10006031},{"operation_name":"upsert","last_dispatch_duration_us":244927,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9392637,"total_server_duration_us":191221,"encode_duration_us":19229,"operation_id":"0x507c","timeout_ms":10000,"last_server_duration_us":55425,"total_duration_us":10005075},{"operation_name":"upsert","last_dispatch_duration_us":1276003,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":2264406,"total_server_duration_us":292920,"encode_duration_us":19981,"operation_id":"0x510f","timeout_ms":10000,"last_server_duration_us":23306,"total_duration_us":10004291},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":17065,"operation_id":"0x5166","timeout_ms":10000,"total_duration_us":10003190},{"operation_name":"upsert","last_dispatch_duration_us":9234727,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9234727,"total_server_duration_us":9055417,"encode_duration_us":22260,"operation_id":"0x5075","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":9240702}],"total_count":49}} WARN [2022-03-13 17:11:37,434] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":33350,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9509886,"total_server_duration_us":254386,"encode_duration_us":17233,"operation_id":"0x5009","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10010764},{"operation_name":"upsert","last_dispatch_duration_us":8333127,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9061102,"total_server_duration_us":169275,"encode_duration_us":18826,"operation_id":"0x5040","timeout_ms":10000,"last_server_duration_us":27402,"total_duration_us":10010057},{"operation_name":"upsert","last_dispatch_duration_us":243785,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10168766,"total_server_duration_us":237400,"encode_duration_us":18228,"operation_id":"0x5056","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10008898},{"operation_name":"upsert","last_dispatch_duration_us":84549,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9994520,"total_server_duration_us":246530,"encode_duration_us":18978,"operation_id":"0x5070","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10007583},{"operation_name":"upsert","last_dispatch_duration_us":33587,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1038270,"total_server_duration_us":622198,"encode_duration_us":18293,"operation_id":"0x507a","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10006247},{"operation_name":"upsert","last_dispatch_duration_us":31973,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":2734436,"total_server_duration_us":552773,"encode_duration_us":18452,"operation_id":"0x4f80","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10005286},{"operation_name":"upsert","last_dispatch_duration_us":147728,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10026981,"total_server_duration_us":211410,"encode_duration_us":17726,"operation_id":"0x4fc4","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10005113},{"operation_name":"upsert","last_dispatch_duration_us":80563,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9499093,"total_server_duration_us":249457,"encode_duration_us":18401,"operation_id":"0x50af","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10003896},{"operation_name":"upsert","last_dispatch_duration_us":9361412,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9615263,"total_server_duration_us":9114583,"encode_duration_us":20205,"operation_id":"0x506e","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":9629553},{"operation_name":"upsert","last_dispatch_duration_us":1068053,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":3093209,"total_server_duration_us":616420,"encode_duration_us":17660,"operation_id":"0x50fc","timeout_ms":10000,"last_server_duration_us":178288,"total_duration_us":3112970}],"total_count":63}} WARN [2022-03-13 17:11:37,434] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9279497,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11184129,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9324425,"encode_duration_us":18767,"operation_id":"0x4f5f","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10003185}],"total_count":1}} WARN [2022-03-13 17:11:37,437] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11337516,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":11337516,"total_server_duration_us":23,"operation_id":"0x50b2","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":11338481},{"operation_name":"upsert","last_dispatch_duration_us":15431610,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":15431610,"total_server_duration_us":61139,"encode_duration_us":20343,"operation_id":"0x4fe5","timeout_ms":10000,"last_server_duration_us":61139,"total_duration_us":10007478},{"operation_name":"upsert","last_dispatch_duration_us":9090871,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11545439,"total_server_duration_us":9537508,"encode_duration_us":16991,"operation_id":"0x4fb1","timeout_ms":10000,"last_server_duration_us":9079863,"total_duration_us":10006655},{"operation_name":"upsert","last_dispatch_duration_us":9057109,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10721156,"total_server_duration_us":9419022,"encode_duration_us":18433,"operation_id":"0x5002","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10006086},{"operation_name":"upsert","last_dispatch_duration_us":4092500,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4092500,"total_server_duration_us":57632,"encode_duration_us":21930,"operation_id":"0x503d","timeout_ms":10000,"last_server_duration_us":57632,"total_duration_us":10005812},{"operation_name":"upsert","last_dispatch_duration_us":8416834,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11440597,"total_server_duration_us":630296,"encode_duration_us":17366,"operation_id":"0x4fa0","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10004629},{"operation_name":"upsert","encode_duration_us":18908,"operation_id":"0x4f97","timeout_ms":10000,"total_duration_us":10003681},{"operation_name":"upsert","last_dispatch_duration_us":17177563,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":17177563,"total_server_duration_us":9039489,"encode_duration_us":17529,"operation_id":"0x4f74","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10003326},{"operation_name":"upsert","last_dispatch_duration_us":1791448,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7185256,"total_server_duration_us":131811,"encode_duration_us":17328,"operation_id":"0x519b","timeout_ms":10000,"last_server_duration_us":33426,"total_duration_us":10001008},{"operation_name":"upsert","last_dispatch_duration_us":9043249,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9483703,"total_server_duration_us":9307338,"encode_duration_us":18487,"operation_id":"0x5134","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":9540930}],"total_count":23}} INFO [2022-03-13 17:11:37,444] jepsen worker 3 - jepsen.util 1803 :ok :read [1 42] INFO [2022-03-13 17:11:37,445] jepsen worker 87 - jepsen.util 2427 :ok :read [29 10] INFO [2022-03-13 17:11:37,445] jepsen worker 3 - jepsen.util 1803 :invoke :write [1 32] INFO [2022-03-13 17:11:37,445] jepsen worker 87 - jepsen.util 2427 :invoke :read [29 nil] INFO [2022-03-13 17:11:37,452] jepsen worker 62 - jepsen.util 512 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,453] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:37,456] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,456] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,457] jepsen worker 69 - jepsen.util 609 :invoke :write [23 21] INFO [2022-03-13 17:11:37,457] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:37,478] jepsen worker 87 - jepsen.util 2427 :ok :read [29 10] INFO [2022-03-13 17:11:37,478] jepsen worker 87 - jepsen.util 2427 :invoke :write [29 44] INFO [2022-03-13 17:11:37,480] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,481] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:11:37,517] jepsen worker 59 - jepsen.util 1589 :info :write [19 47] :RequestTimeoutException INFO [2022-03-13 17:11:37,518] jepsen worker 59 - jepsen.util 1679 :invoke :write [19 49] INFO [2022-03-13 17:11:37,528] jepsen worker 76 - jepsen.util 616 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,528] jepsen worker 76 - jepsen.util 616 :invoke :write [25 30] WARN [2022-03-13 17:11:37,535] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":6882121,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7888438,"total_server_duration_us":303600,"encode_duration_us":21072,"operation_id":"0x5100","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10006147},{"operation_name":"upsert","last_dispatch_duration_us":7218542,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8799755,"total_server_duration_us":641944,"encode_duration_us":17795,"operation_id":"0x500b","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10004043},{"operation_name":"upsert","last_dispatch_duration_us":7251235,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8032688,"total_server_duration_us":335681,"encode_duration_us":17042,"operation_id":"0x50c0","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10003917},{"operation_name":"upsert","last_dispatch_duration_us":9064371,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9579171,"total_server_duration_us":9370314,"encode_duration_us":18727,"operation_id":"0x5152","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10002085},{"operation_name":"upsert","last_dispatch_duration_us":198232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":370632,"total_server_duration_us":102386,"encode_duration_us":17334,"operation_id":"0x5135","timeout_ms":10000,"last_server_duration_us":39846,"total_duration_us":10001387},{"operation_name":"upsert","last_dispatch_duration_us":6969828,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7918904,"total_server_duration_us":274267,"encode_duration_us":17395,"operation_id":"0x50ff","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10000487},{"operation_name":"upsert","last_dispatch_duration_us":9140136,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9140136,"total_server_duration_us":9070294,"encode_duration_us":19016,"operation_id":"0x50f9","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":9144597},{"operation_name":"get","last_dispatch_duration_us":6999644,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":6999644,"total_server_duration_us":274053,"operation_id":"0x519c","timeout_ms":30000,"last_server_duration_us":274053,"total_duration_us":7001817},{"operation_name":"upsert","last_dispatch_duration_us":1566254,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":5252401,"total_server_duration_us":273766,"encode_duration_us":20701,"operation_id":"0x5074","timeout_ms":10000,"last_server_duration_us":192512,"total_duration_us":5578540},{"operation_name":"get","last_dispatch_duration_us":5331772,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":5331772,"total_server_duration_us":43,"operation_id":"0x5225","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5332057}],"total_count":53}} WARN [2022-03-13 17:11:37,537] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17177563,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":17177563,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9039489,"encode_duration_us":17529,"operation_id":"0x4f74","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10003326},{"operation_name":"upsert","last_dispatch_duration_us":9507133,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9507133,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":36220,"encode_duration_us":28518,"operation_id":"0x4f60","timeout_ms":10000,"last_server_duration_us":36220,"total_duration_us":10005516},{"operation_name":"upsert","last_dispatch_duration_us":4092500,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":4092500,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":57632,"encode_duration_us":21930,"operation_id":"0x503d","timeout_ms":10000,"last_server_duration_us":57632,"total_duration_us":10005812},{"operation_name":"upsert","last_dispatch_duration_us":9057109,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10721156,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9419022,"encode_duration_us":18433,"operation_id":"0x5002","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10006086},{"operation_name":"upsert","last_dispatch_duration_us":9090871,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11545439,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9537508,"encode_duration_us":16991,"operation_id":"0x4fb1","timeout_ms":10000,"last_server_duration_us":9079863,"total_duration_us":10006655},{"operation_name":"upsert","last_dispatch_duration_us":11850445,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":15864205,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":338267,"encode_duration_us":20251,"operation_id":"0x4efe","timeout_ms":10000,"last_server_duration_us":215579,"total_duration_us":10010320},{"operation_name":"upsert","last_dispatch_duration_us":10302288,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":15855152,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":156983,"encode_duration_us":24638,"operation_id":"0x4ea5","timeout_ms":10000,"last_server_duration_us":76870,"total_duration_us":10008106},{"operation_name":"upsert","last_dispatch_duration_us":11748391,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11748391,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":791773,"encode_duration_us":19361,"operation_id":"0x4f68","timeout_ms":10000,"last_server_duration_us":791773,"total_duration_us":10008603},{"operation_name":"upsert","last_dispatch_duration_us":9154581,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10951545,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9859105,"encode_duration_us":17614,"operation_id":"0x4f69","timeout_ms":10000,"last_server_duration_us":9093693,"total_duration_us":10006211}],"total_count":9}} INFO [2022-03-13 17:11:37,549] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3255131964809385}, :ns_1@172.23.105.187 {:progress 0.4853801169590644}, :ns_1@172.23.107.227 {:progress 0.3255131964809385}, :ns_1@172.23.107.228 {:progress 0.3764705882352941}} INFO [2022-03-13 17:11:37,550] jepsen worker nemesis - couchbase.util Rebalance check count: 208 INFO [2022-03-13 17:11:37,550] jepsen worker nemesis - couchbase.util Rebalance stuck count: 132 INFO [2022-03-13 17:11:37,574] jepsen worker 21 - jepsen.util 741 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,575] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:37,581] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,581] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,581] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,581] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:37,581] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,581] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,582] jepsen worker 60 - jepsen.util 600 :invoke :write [20 36] INFO [2022-03-13 17:11:37,583] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:37,584] jepsen worker 77 - jepsen.util 707 :invoke :write [25 17] INFO [2022-03-13 17:11:37,584] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:37,585] jepsen worker 12 - jepsen.util 732 :invoke :write [4 34] INFO [2022-03-13 17:11:37,586] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:37,592] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,593] jepsen worker 79 - jepsen.util 889 :invoke :write [26 19] INFO [2022-03-13 17:11:37,595] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:37,595] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,595] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,596] jepsen worker 44 - jepsen.util 584 :invoke :write [14 47] INFO [2022-03-13 17:11:37,597] jepsen worker 23 - jepsen.util 563 :invoke :write [7 47] INFO [2022-03-13 17:11:37,602] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:37,607] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,607] jepsen worker 70 - jepsen.util 700 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,607] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,607] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,608] jepsen worker 52 - jepsen.util 592 :invoke :write [17 45] INFO [2022-03-13 17:11:37,609] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:37,609] jepsen worker 80 - jepsen.util 710 :invoke :write [26 19] INFO [2022-03-13 17:11:37,610] jepsen worker 13 - jepsen.util 643 :invoke :write [4 15] WARN [2022-03-13 17:11:37,631] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9109738,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9961876,"total_server_duration_us":9467247,"encode_duration_us":17651,"operation_id":"0x5041","timeout_ms":10000,"last_server_duration_us":9078800,"total_duration_us":10010332},{"operation_name":"upsert","last_dispatch_duration_us":87218,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":210932,"total_server_duration_us":120051,"encode_duration_us":17984,"operation_id":"0x50b3","timeout_ms":10000,"last_server_duration_us":40482,"total_duration_us":10010014},{"operation_name":"upsert","last_dispatch_duration_us":63307,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":2379836,"total_server_duration_us":846977,"encode_duration_us":18969,"operation_id":"0x5066","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10005705},{"operation_name":"upsert","last_dispatch_duration_us":10110906,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":13277111,"total_server_duration_us":344662,"encode_duration_us":21368,"operation_id":"0x4f98","timeout_ms":10000,"last_server_duration_us":54576,"total_duration_us":10005325},{"operation_name":"upsert","last_dispatch_duration_us":30298,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":896961,"total_server_duration_us":578203,"encode_duration_us":17621,"operation_id":"0x4f99","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10002236},{"operation_name":"get","last_dispatch_duration_us":9571390,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9571390,"total_server_duration_us":1428947,"operation_id":"0x50f2","timeout_ms":30000,"last_server_duration_us":1428947,"total_duration_us":9573907},{"operation_name":"get","last_dispatch_duration_us":8101162,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":8101162,"total_server_duration_us":43,"operation_id":"0x501b","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8101442},{"operation_name":"upsert","last_dispatch_duration_us":6909743,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6909743,"total_server_duration_us":208479,"encode_duration_us":19490,"operation_id":"0x515c","timeout_ms":10000,"last_server_duration_us":208479,"total_duration_us":6917417},{"operation_name":"upsert","last_dispatch_duration_us":6284932,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6284932,"total_server_duration_us":93346,"encode_duration_us":19298,"operation_id":"0x515b","timeout_ms":10000,"last_server_duration_us":93346,"total_duration_us":6288991},{"operation_name":"upsert","last_dispatch_duration_us":6029528,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6029528,"total_server_duration_us":118121,"encode_duration_us":18984,"operation_id":"0x5138","timeout_ms":10000,"last_server_duration_us":118121,"total_duration_us":6033414}],"total_count":36}} WARN [2022-03-13 17:11:37,631] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9046661,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9526664,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9193433,"encode_duration_us":17689,"operation_id":"0x4e7f","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10004004},{"operation_name":"upsert","last_dispatch_duration_us":10110906,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":13277111,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":344662,"encode_duration_us":21368,"operation_id":"0x4f98","timeout_ms":10000,"last_server_duration_us":54576,"total_duration_us":10005325},{"operation_name":"upsert","last_dispatch_duration_us":9109738,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9961876,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9467247,"encode_duration_us":17651,"operation_id":"0x5041","timeout_ms":10000,"last_server_duration_us":9078800,"total_duration_us":10010332},{"operation_name":"upsert","last_dispatch_duration_us":9162340,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":13193377,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":132393,"encode_duration_us":19546,"operation_id":"0x4efc","timeout_ms":10000,"last_server_duration_us":51710,"total_duration_us":10008233}],"total_count":4}} INFO [2022-03-13 17:11:37,671] jepsen worker 14 - jepsen.util 284 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,672] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:37,701] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,701] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,701] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,702] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:37,702] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:37,703] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:11:37,769] jepsen worker 76 - jepsen.util 616 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,770] jepsen worker 76 - jepsen.util 616 :invoke :write [25 15] INFO [2022-03-13 17:11:37,808] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,808] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,809] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,809] jepsen worker 71 - jepsen.util 521 :invoke :write [23 3] INFO [2022-03-13 17:11:37,810] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:37,810] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:37,849] jepsen worker 60 - jepsen.util 600 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,849] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,850] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:37,850] jepsen worker 60 - jepsen.util 600 :invoke :write [20 1] INFO [2022-03-13 17:11:37,888] jepsen worker 79 - jepsen.util 889 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,888] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:37,912] jepsen worker 69 - jepsen.util 609 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,912] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,913] jepsen worker 69 - jepsen.util 609 :invoke :write [23 5] INFO [2022-03-13 17:11:37,913] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:37,915] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,915] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,916] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:11:37,916] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:37,916] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,917] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:11:37,919] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,920] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:11:37,972] jepsen worker 76 - jepsen.util 616 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,972] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,973] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:37,973] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:37,992] jepsen worker 53 - jepsen.util 503 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,992] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:37,993] jepsen worker 8 - jepsen.util 728 :invoke :write [2 24] INFO [2022-03-13 17:11:37,994] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:38,016] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,016] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,016] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,017] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:38,017] jepsen worker 76 - jepsen.util 616 :invoke :write [25 4] INFO [2022-03-13 17:11:38,018] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:38,018] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,018] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:38,019] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,019] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:38,020] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,020] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:11:38,020] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,021] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:38,022] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,023] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:38,056] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,056] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,057] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,057] jepsen worker 24 - jepsen.util 564 :invoke :write [8 28] INFO [2022-03-13 17:11:38,056] jepsen worker 71 - jepsen.util 521 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,059] jepsen worker 21 - jepsen.util 741 :invoke :write [7 23] INFO [2022-03-13 17:11:38,060] jepsen worker 79 - jepsen.util 889 :invoke :write [26 29] INFO [2022-03-13 17:11:38,061] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:38,076] jepsen worker 36 - jepsen.util 2016 :info :write [12 31] :RequestTimeoutException INFO [2022-03-13 17:11:38,078] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:11:38,087] jepsen worker 85 - jepsen.util 1885 :info :write [28 23] :SyncWriteAmbiguous INFO [2022-03-13 17:11:38,088] jepsen worker 85 - jepsen.util 1975 :invoke :read [28 nil] INFO [2022-03-13 17:11:38,179] jepsen worker 12 - jepsen.util 732 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,179] jepsen worker 12 - jepsen.util 732 :invoke :write [4 34] INFO [2022-03-13 17:11:38,199] jepsen worker 58 - jepsen.util 1678 :ok :read [19 17] INFO [2022-03-13 17:11:38,200] jepsen worker 58 - jepsen.util 1678 :invoke :write [19 23] INFO [2022-03-13 17:11:38,210] jepsen worker 48 - jepsen.util 1758 :info :write [16 42] :RequestTimeoutException INFO [2022-03-13 17:11:38,211] jepsen worker 48 - jepsen.util 1848 :invoke :read [16 nil] INFO [2022-03-13 17:11:38,219] jepsen worker 82 - jepsen.util 1792 :ok :read [27 4] INFO [2022-03-13 17:11:38,219] jepsen worker 76 - jepsen.util 616 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,219] jepsen worker 82 - jepsen.util 1792 :invoke :read [27 nil] INFO [2022-03-13 17:11:38,219] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:38,263] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,264] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:11:38,278] jepsen worker 69 - jepsen.util 609 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,279] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:38,297] jepsen worker 1 - jepsen.util 1531 :ok :read [0 41] INFO [2022-03-13 17:11:38,298] jepsen worker 1 - jepsen.util 1531 :invoke :read [0 nil] INFO [2022-03-13 17:11:38,305] jepsen worker 82 - jepsen.util 1792 :ok :read [27 4] INFO [2022-03-13 17:11:38,305] jepsen worker 82 - jepsen.util 1792 :invoke :write [27 34] INFO [2022-03-13 17:11:38,320] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,321] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,320] jepsen worker 6 - jepsen.util 636 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,321] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,320] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:11:38,321] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,321] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:38,322] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,322] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,322] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:38,323] jepsen worker 61 - jepsen.util 601 :invoke :write [20 36] INFO [2022-03-13 17:11:38,324] jepsen worker 14 - jepsen.util 284 :invoke :write [4 26] INFO [2022-03-13 17:11:38,325] jepsen worker 53 - jepsen.util 503 :invoke :write [17 12] INFO [2022-03-13 17:11:38,326] jepsen worker 26 - jepsen.util 476 :invoke :write [8 18] INFO [2022-03-13 17:11:38,328] jepsen worker 5 - jepsen.util 2075 :info :write [1 14] :RequestTimeoutException INFO [2022-03-13 17:11:38,329] jepsen worker 6 - jepsen.util 636 :invoke :write [2 19] INFO [2022-03-13 17:11:38,329] jepsen worker 43 - jepsen.util 493 :invoke :write [14 31] INFO [2022-03-13 17:11:38,329] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:11:38,330] jepsen worker 5 - jepsen.util 2165 :invoke :write [1 6] INFO [2022-03-13 17:11:38,332] jepsen worker 42 - jepsen.util 402 :invoke :write [14 24] INFO [2022-03-13 17:11:38,340] jepsen worker 39 - jepsen.util 1839 :ok :read [13 39] INFO [2022-03-13 17:11:38,341] jepsen worker 39 - jepsen.util 1839 :invoke :write [13 11] INFO [2022-03-13 17:11:38,356] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,357] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:11:38,369] jepsen worker 79 - jepsen.util 889 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,369] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,371] jepsen worker 76 - jepsen.util 616 :invoke :write [25 40] INFO [2022-03-13 17:11:38,371] jepsen worker 79 - jepsen.util 889 :invoke :write [26 46] INFO [2022-03-13 17:11:38,390] jepsen worker 89 - jepsen.util 2159 :info :write [29 10] :RequestTimeoutException INFO [2022-03-13 17:11:38,392] jepsen worker 89 - jepsen.util 2249 :invoke :read [29 nil] INFO [2022-03-13 17:11:38,409] jepsen worker 1 - jepsen.util 1531 :ok :read [0 41] INFO [2022-03-13 17:11:38,409] jepsen worker 1 - jepsen.util 1531 :invoke :read [0 nil] INFO [2022-03-13 17:11:38,418] jepsen worker 89 - jepsen.util 2249 :ok :read [29 10] INFO [2022-03-13 17:11:38,419] jepsen worker 89 - jepsen.util 2249 :invoke :write [29 1] INFO [2022-03-13 17:11:38,419] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,419] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,419] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:38,420] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:38,425] jepsen worker 77 - jepsen.util 707 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,426] jepsen worker 77 - jepsen.util 707 :invoke :write [25 25] INFO [2022-03-13 17:11:38,471] jepsen worker 1 - jepsen.util 1531 :ok :read [0 41] INFO [2022-03-13 17:11:38,471] jepsen worker 1 - jepsen.util 1531 :invoke :write [0 3] INFO [2022-03-13 17:11:38,486] jepsen worker 24 - jepsen.util 564 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,487] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:38,496] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,497] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:11:38,555] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,556] jepsen worker 21 - jepsen.util 741 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,556] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:11:38,556] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3284457478005866}, :ns_1@172.23.105.187 {:progress 0.4912280701754386}, :ns_1@172.23.107.227 {:progress 0.3284457478005866}, :ns_1@172.23.107.228 {:progress 0.3941176470588236}} INFO [2022-03-13 17:11:38,556] jepsen worker nemesis - couchbase.util Rebalance check count: 209 INFO [2022-03-13 17:11:38,556] jepsen worker nemesis - couchbase.util Rebalance stuck count: 132 INFO [2022-03-13 17:11:38,557] jepsen worker 21 - jepsen.util 741 :invoke :write [7 36] INFO [2022-03-13 17:11:38,570] jepsen worker 83 - jepsen.util 1253 :info :write [27 26] :RequestTimeoutException INFO [2022-03-13 17:11:38,571] jepsen worker 83 - jepsen.util 1343 :invoke :read [27 nil] INFO [2022-03-13 17:11:38,587] jepsen worker 79 - jepsen.util 889 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,588] jepsen worker 79 - jepsen.util 889 :invoke :write [26 10] INFO [2022-03-13 17:11:38,634] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,635] jepsen worker 22 - jepsen.util 652 :invoke :write [7 8] INFO [2022-03-13 17:11:38,647] jepsen worker 8 - jepsen.util 728 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,647] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,647] jepsen worker 23 - jepsen.util 563 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,647] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,648] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:38,648] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:38,648] jepsen worker 8 - jepsen.util 728 :invoke :write [2 48] INFO [2022-03-13 17:11:38,649] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:11:38,691] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,691] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,692] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,692] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,692] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:11:38,692] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:38,693] jepsen worker 71 - jepsen.util 521 :invoke :write [23 34] INFO [2022-03-13 17:11:38,693] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:38,725] jepsen worker 76 - jepsen.util 616 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,726] jepsen worker 76 - jepsen.util 616 :invoke :write [25 33] INFO [2022-03-13 17:11:38,782] jepsen worker 53 - jepsen.util 503 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,783] jepsen worker 53 - jepsen.util 503 :invoke :write [17 24] INFO [2022-03-13 17:11:38,831] jepsen worker 21 - jepsen.util 741 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,831] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:38,838] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,839] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:38,846] jepsen worker 81 - jepsen.util 2061 :info :write [27 46] :RequestTimeoutException INFO [2022-03-13 17:11:38,846] jepsen worker 81 - jepsen.util 2151 :invoke :write [27 42] INFO [2022-03-13 17:11:38,881] jepsen worker 44 - jepsen.util 584 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,881] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,882] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:38,883] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:38,902] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,902] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,902] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,903] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:38,903] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:38,904] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:11:38,938] jepsen worker 79 - jepsen.util 889 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,939] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:11:38,960] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:38,961] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:39,016] jepsen worker 38 - jepsen.util 2198 :info :write [12 31] :RequestTimeoutException INFO [2022-03-13 17:11:39,017] jepsen worker 38 - jepsen.util 2288 :invoke :read [12 nil] INFO [2022-03-13 17:11:39,044] jepsen worker 61 - jepsen.util 601 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,045] jepsen worker 61 - jepsen.util 601 :invoke :write [20 24] INFO [2022-03-13 17:11:39,047] jepsen worker 38 - jepsen.util 2288 :ok :read [12 20] INFO [2022-03-13 17:11:39,048] jepsen worker 38 - jepsen.util 2288 :invoke :read [12 nil] INFO [2022-03-13 17:11:39,070] jepsen worker 38 - jepsen.util 2288 :ok :read [12 20] INFO [2022-03-13 17:11:39,071] jepsen worker 38 - jepsen.util 2288 :invoke :write [12 9] INFO [2022-03-13 17:11:39,121] jepsen worker 76 - jepsen.util 616 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,121] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,121] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:39,122] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:11:39,142] jepsen worker 60 - jepsen.util 600 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,143] jepsen worker 60 - jepsen.util 600 :invoke :write [20 31] INFO [2022-03-13 17:11:39,174] jepsen worker 71 - jepsen.util 521 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,174] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,174] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,174] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,175] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,175] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:11:39,176] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:39,176] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:39,259] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,259] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,259] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:39,260] jepsen worker 76 - jepsen.util 616 :invoke :write [25 49] INFO [2022-03-13 17:11:39,264] jepsen worker 43 - jepsen.util 493 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,265] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:11:39,306] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,306] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,306] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,306] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,306] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,307] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:11:39,308] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:39,309] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,310] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:39,311] jepsen worker 21 - jepsen.util 741 :invoke :write [7 45] INFO [2022-03-13 17:11:39,328] jepsen worker 12 - jepsen.util 732 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,330] jepsen worker 12 - jepsen.util 732 :invoke :write [4 43] INFO [2022-03-13 17:11:39,346] jepsen worker 4 - jepsen.util 1984 :info :write [1 5] :RequestTimeoutException INFO [2022-03-13 17:11:39,347] jepsen worker 4 - jepsen.util 2074 :invoke :read [1 nil] INFO [2022-03-13 17:11:39,361] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,361] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,362] jepsen worker 79 - jepsen.util 889 :invoke :write [26 7] INFO [2022-03-13 17:11:39,363] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:11:39,413] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,413] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,413] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,413] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,414] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:39,414] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,415] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:39,415] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:39,441] jepsen worker 76 - jepsen.util 616 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,442] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:39,520] jepsen worker 21 - jepsen.util 741 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,520] jepsen worker 21 - jepsen.util 741 :invoke :write [7 25] INFO [2022-03-13 17:11:39,562] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3284457478005866}, :ns_1@172.23.105.187 {:progress 0.4912280701754386}, :ns_1@172.23.107.227 {:progress 0.3284457478005866}, :ns_1@172.23.107.228 {:progress 0.3941176470588236}} INFO [2022-03-13 17:11:39,562] jepsen worker nemesis - couchbase.util Rebalance check count: 210 INFO [2022-03-13 17:11:39,562] jepsen worker nemesis - couchbase.util Rebalance stuck count: 133 INFO [2022-03-13 17:11:39,582] jepsen worker 79 - jepsen.util 889 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,583] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:39,592] jepsen worker 77 - jepsen.util 707 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,592] jepsen worker 6 - jepsen.util 636 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,593] jepsen worker 77 - jepsen.util 707 :invoke :write [25 34] INFO [2022-03-13 17:11:39,594] jepsen worker 6 - jepsen.util 636 :invoke :write [2 16] INFO [2022-03-13 17:11:39,596] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,596] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,597] jepsen worker 23 - jepsen.util 563 :invoke :write [7 14] INFO [2022-03-13 17:11:39,597] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:39,598] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:39,598] jepsen worker 44 - jepsen.util 584 :invoke :write [14 37] INFO [2022-03-13 17:11:39,609] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,609] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,609] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,609] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,609] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:11:39,609] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,610] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,610] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:11:39,611] jepsen worker 69 - jepsen.util 609 :invoke :write [23 4] INFO [2022-03-13 17:11:39,619] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:39,675] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,675] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,675] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,676] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,676] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:11:39,677] jepsen worker 79 - jepsen.util 889 :invoke :write [26 2] INFO [2022-03-13 17:11:39,678] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,679] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,701] jepsen worker 16 - jepsen.util 1636 :info :write [5 9] :RequestTimeoutException INFO [2022-03-13 17:11:39,701] jepsen worker 16 - jepsen.util 1726 :invoke :read [5 nil] INFO [2022-03-13 17:11:39,717] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,717] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,721] jepsen worker 42 - jepsen.util 402 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,722] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:39,729] jepsen worker 21 - jepsen.util 741 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,730] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:39,731] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,731] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,731] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:39,732] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:39,799] jepsen worker 60 - jepsen.util 600 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,800] jepsen worker 60 - jepsen.util 600 :invoke :write [20 23] INFO [2022-03-13 17:11:39,834] jepsen worker 79 - jepsen.util 889 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,834] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:11:39,850] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,851] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:39,899] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,899] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:39,933] jepsen worker 14 - jepsen.util 284 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,934] jepsen worker 14 - jepsen.util 284 :invoke :write [4 9] INFO [2022-03-13 17:11:39,936] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,936] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,937] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:11:39,937] jepsen worker 76 - jepsen.util 616 :invoke :write [25 30] INFO [2022-03-13 17:11:39,939] jepsen worker 69 - jepsen.util 609 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,940] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,940] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,940] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:11:39,940] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,941] jepsen worker 24 - jepsen.util 564 :invoke :write [8 48] INFO [2022-03-13 17:11:39,942] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:39,942] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,942] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:11:39,943] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:39,944] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:39,944] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:40,020] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,020] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,021] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,021] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,022] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:40,023] jepsen worker 21 - jepsen.util 741 :invoke :write [7 37] INFO [2022-03-13 17:11:40,070] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,071] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:40,124] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,125] jepsen worker 79 - jepsen.util 889 :invoke :write [26 3] INFO [2022-03-13 17:11:40,141] jepsen worker 12 - jepsen.util 732 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,141] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,142] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:40,143] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:40,192] jepsen worker 76 - jepsen.util 616 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,193] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,238] jepsen worker 24 - jepsen.util 564 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,239] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:40,262] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,262] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:11:40,339] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,339] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,340] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,339] jepsen worker 26 - jepsen.util 476 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,340] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,340] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,340] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,342] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:40,343] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:11:40,344] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:11:40,344] jepsen worker 7 - jepsen.util 637 :invoke :write [2 33] INFO [2022-03-13 17:11:40,345] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:40,377] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,377] jepsen worker 79 - jepsen.util 889 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,377] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,378] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:40,444] jepsen worker 77 - jepsen.util 707 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,444] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:11:40,445] jepsen worker 21 - jepsen.util 741 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,445] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,446] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,446] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,446] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,447] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:40,448] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,449] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:11:40,449] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:11:40,450] jepsen worker 21 - jepsen.util 741 :invoke :write [7 42] INFO [2022-03-13 17:11:40,492] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,492] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,492] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,493] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:40,494] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:40,494] jepsen worker 76 - jepsen.util 616 :invoke :write [25 8] INFO [2022-03-13 17:11:40,495] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,495] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,496] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:11:40,496] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:40,497] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,498] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:11:40,513] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,514] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:40,514] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,514] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,515] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:40,515] jepsen worker 75 - jepsen.util 435 :invoke :write [25 10] INFO [2022-03-13 17:11:40,568] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3343108504398827}, :ns_1@172.23.105.187 {:progress 0.4970760233918129}, :ns_1@172.23.107.227 {:progress 0.3343108504398827}, :ns_1@172.23.107.228 {:progress 0.4}} INFO [2022-03-13 17:11:40,568] jepsen worker nemesis - couchbase.util Rebalance check count: 211 INFO [2022-03-13 17:11:40,568] jepsen worker nemesis - couchbase.util Rebalance stuck count: 133 INFO [2022-03-13 17:11:40,577] jepsen worker 72 - jepsen.util 1782 :info :write [24 26] :SyncWriteAmbiguous INFO [2022-03-13 17:11:40,577] jepsen worker 72 - jepsen.util 1872 :invoke :write [24 20] INFO [2022-03-13 17:11:40,583] jepsen worker 76 - jepsen.util 616 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,584] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,589] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,589] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:11:40,656] jepsen worker 23 - jepsen.util 563 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,657] jepsen worker 8 - jepsen.util 728 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,657] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:11:40,658] jepsen worker 8 - jepsen.util 728 :invoke :write [2 3] INFO [2022-03-13 17:11:40,666] jepsen worker 47 - jepsen.util 1937 :ok :read [15 48] INFO [2022-03-13 17:11:40,667] jepsen worker 47 - jepsen.util 1937 :invoke :write [15 23] INFO [2022-03-13 17:11:40,697] jepsen worker 21 - jepsen.util 741 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,697] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,698] jepsen worker 21 - jepsen.util 741 :invoke :write [7 20] INFO [2022-03-13 17:11:40,699] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:11:40,758] jepsen worker 61 - jepsen.util 601 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,758] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:11:40,758] jepsen worker 61 - jepsen.util 601 :invoke :write [20 37] INFO [2022-03-13 17:11:40,759] jepsen worker 43 - jepsen.util 493 :invoke :write [14 21] INFO [2022-03-13 17:11:40,799] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,799] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,800] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,800] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,800] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:11:40,801] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,801] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,801] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:40,801] jepsen worker 69 - jepsen.util 609 :invoke :write [23 5] INFO [2022-03-13 17:11:40,802] jepsen worker 71 - jepsen.util 521 :invoke :write [23 27] INFO [2022-03-13 17:11:40,826] jepsen worker 44 - jepsen.util 584 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,827] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:40,860] jepsen worker 68 - jepsen.util 1868 :info :write [22 10] :SyncWriteAmbiguous INFO [2022-03-13 17:11:40,861] jepsen worker 68 - jepsen.util 1958 :invoke :write [22 43] INFO [2022-03-13 17:11:40,895] jepsen worker 75 - jepsen.util 435 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,895] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,895] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,896] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:11:40,896] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:40,899] jepsen worker 79 - jepsen.util 889 :invoke :write [26 0] INFO [2022-03-13 17:11:40,921] jepsen worker 78 - jepsen.util 528 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,921] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,922] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,922] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:40,923] jepsen worker 70 - jepsen.util 700 :invoke :write [23 40] INFO [2022-03-13 17:11:40,926] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:40,950] jepsen worker 6 - jepsen.util 636 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:40,950] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:40,951] jepsen worker 6 - jepsen.util 636 :invoke :write [2 39] INFO [2022-03-13 17:11:40,951] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:41,001] jepsen worker 48 - jepsen.util 1848 :ok :read [16 :nil] INFO [2022-03-13 17:11:41,002] jepsen worker 48 - jepsen.util 1848 :invoke :read [16 nil] INFO [2022-03-13 17:11:41,003] jepsen worker 48 - jepsen.util 1848 :ok :read [16 :nil] INFO [2022-03-13 17:11:41,004] jepsen worker 48 - jepsen.util 1848 :invoke :write [16 23] INFO [2022-03-13 17:11:41,016] jepsen worker 66 - jepsen.util 1686 :ok :read [22 30] INFO [2022-03-13 17:11:41,017] jepsen worker 66 - jepsen.util 1686 :invoke :read [22 nil] INFO [2022-03-13 17:11:41,017] jepsen worker 0 - jepsen.util 1800 :ok :read [0 41] INFO [2022-03-13 17:11:41,018] jepsen worker 0 - jepsen.util 1800 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,054] jepsen worker 66 - jepsen.util 1686 :ok :read [22 30] INFO [2022-03-13 17:11:41,055] jepsen worker 66 - jepsen.util 1686 :invoke :write [22 20] INFO [2022-03-13 17:11:41,056] jepsen worker 0 - jepsen.util 1800 :ok :read [0 41] INFO [2022-03-13 17:11:41,056] jepsen worker 0 - jepsen.util 1800 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,106] jepsen worker 0 - jepsen.util 1800 :ok :read [0 41] INFO [2022-03-13 17:11:41,107] jepsen worker 0 - jepsen.util 1800 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,156] jepsen worker 10 - jepsen.util 1810 :info :write [3 28] :RequestTimeoutException INFO [2022-03-13 17:11:41,157] jepsen worker 10 - jepsen.util 1900 :invoke :read [3 nil] INFO [2022-03-13 17:11:41,164] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,164] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,165] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:41,165] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:41,171] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,172] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:41,172] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,173] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,172] jepsen worker 60 - jepsen.util 600 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,173] jepsen worker 51 - jepsen.util 591 :invoke :write [17 17] INFO [2022-03-13 17:11:41,174] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:41,174] jepsen worker 12 - jepsen.util 732 :invoke :write [4 6] INFO [2022-03-13 17:11:41,179] jepsen worker 0 - jepsen.util 1800 :ok :read [0 41] INFO [2022-03-13 17:11:41,179] jepsen worker 0 - jepsen.util 1800 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,207] jepsen worker 2 - jepsen.util 2162 :ok :read [0 41] INFO [2022-03-13 17:11:41,207] jepsen worker 2 - jepsen.util 2162 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,222] jepsen worker 0 - jepsen.util 1800 :ok :read [0 41] INFO [2022-03-13 17:11:41,223] jepsen worker 0 - jepsen.util 1800 :invoke :write [0 41] INFO [2022-03-13 17:11:41,250] jepsen worker 53 - jepsen.util 503 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,251] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:41,263] jepsen worker 57 - jepsen.util 1947 :ok :read [19 17] INFO [2022-03-13 17:11:41,263] jepsen worker 57 - jepsen.util 1947 :invoke :read [19 nil] INFO [2022-03-13 17:11:41,274] jepsen worker 35 - jepsen.util 845 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,275] jepsen worker 35 - jepsen.util 845 :invoke :write [11 5] INFO [2022-03-13 17:11:41,290] jepsen worker 79 - jepsen.util 889 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,290] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,291] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,292] jepsen worker 79 - jepsen.util 889 :invoke :write [26 26] INFO [2022-03-13 17:11:41,292] jepsen worker 67 - jepsen.util 1507 :ok :read [22 30] INFO [2022-03-13 17:11:41,292] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:11:41,297] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,298] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:41,299] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,301] jepsen worker 2 - jepsen.util 2162 :ok :read [0 41] INFO [2022-03-13 17:11:41,301] jepsen worker 76 - jepsen.util 616 :invoke :write [25 33] INFO [2022-03-13 17:11:41,302] jepsen worker 2 - jepsen.util 2162 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,304] jepsen worker 67 - jepsen.util 1507 :invoke :write [22 14] INFO [2022-03-13 17:11:41,306] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:11:41,297] jepsen worker 21 - jepsen.util 741 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,310] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:41,320] jepsen worker 15 - jepsen.util 1905 :ok :read [5 43] INFO [2022-03-13 17:11:41,321] jepsen worker 57 - jepsen.util 1947 :ok :read [19 17] INFO [2022-03-13 17:11:41,321] jepsen worker 15 - jepsen.util 1905 :invoke :write [5 16] INFO [2022-03-13 17:11:41,323] jepsen worker 57 - jepsen.util 1947 :invoke :write [19 28] INFO [2022-03-13 17:11:41,334] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,335] jepsen worker 41 - jepsen.util 1571 :ok :read [13 39] INFO [2022-03-13 17:11:41,338] jepsen worker 41 - jepsen.util 1571 :invoke :write [13 19] INFO [2022-03-13 17:11:41,339] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:41,350] jepsen worker 2 - jepsen.util 2162 :ok :read [0 41] INFO [2022-03-13 17:11:41,351] jepsen worker 2 - jepsen.util 2162 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,366] jepsen worker 86 - jepsen.util 1526 :ok :read [28 44] INFO [2022-03-13 17:11:41,367] jepsen worker 86 - jepsen.util 1526 :invoke :write [28 27] INFO [2022-03-13 17:11:41,374] jepsen worker 69 - jepsen.util 609 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,376] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:41,390] jepsen worker 2 - jepsen.util 2162 :ok :read [0 41] INFO [2022-03-13 17:11:41,391] jepsen worker 2 - jepsen.util 2162 :invoke :read [0 nil] INFO [2022-03-13 17:11:41,413] jepsen worker 85 - jepsen.util 1975 :ok :read [28 44] INFO [2022-03-13 17:11:41,414] jepsen worker 85 - jepsen.util 1975 :invoke :read [28 nil] INFO [2022-03-13 17:11:41,447] jepsen worker 79 - jepsen.util 889 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,448] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:41,459] jepsen worker 2 - jepsen.util 2162 :ok :read [0 41] INFO [2022-03-13 17:11:41,460] jepsen worker 2 - jepsen.util 2162 :invoke :write [0 5] INFO [2022-03-13 17:11:41,490] jepsen worker 71 - jepsen.util 521 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,491] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:41,505] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,506] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:41,540] jepsen worker 76 - jepsen.util 616 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,541] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,541] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,541] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:41,542] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:41,542] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:11:41,542] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,543] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:41,543] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,544] jepsen worker 79 - jepsen.util 889 :invoke :write [26 39] INFO [2022-03-13 17:11:41,544] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,544] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,545] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,545] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,545] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:41,545] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:41,546] jepsen worker 14 - jepsen.util 284 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,546] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:11:41,547] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:41,547] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:41,548] jepsen worker 60 - jepsen.util 600 :invoke :write [20 31] INFO [2022-03-13 17:11:41,549] jepsen worker 26 - jepsen.util 476 :invoke :write [8 5] INFO [2022-03-13 17:11:41,555] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,556] jepsen worker 76 - jepsen.util 616 :invoke :write [25 41] INFO [2022-03-13 17:11:41,575] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3343108504398827}, :ns_1@172.23.105.187 {:progress 0.4970760233918129}, :ns_1@172.23.107.227 {:progress 0.3343108504398827}, :ns_1@172.23.107.228 {:progress 0.4}} INFO [2022-03-13 17:11:41,575] jepsen worker nemesis - couchbase.util Rebalance check count: 212 INFO [2022-03-13 17:11:41,576] jepsen worker nemesis - couchbase.util Rebalance stuck count: 134 INFO [2022-03-13 17:11:41,583] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,583] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,583] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,584] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:11:41,587] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:41,588] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:41,658] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,659] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,659] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,661] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:41,661] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,661] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,661] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:41,662] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,662] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,662] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:41,663] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:41,664] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:41,665] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:41,672] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:11:41,695] jepsen worker 85 - jepsen.util 1975 :ok :read [28 44] INFO [2022-03-13 17:11:41,701] jepsen worker 85 - jepsen.util 1975 :invoke :read [28 nil] INFO [2022-03-13 17:11:41,748] jepsen worker 51 - jepsen.util 591 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,748] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:41,768] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,768] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,769] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:11:41,769] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,769] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,769] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,769] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:41,770] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:41,770] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:11:41,771] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:41,774] jepsen worker 76 - jepsen.util 616 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,775] jepsen worker 76 - jepsen.util 616 :invoke :write [25 25] INFO [2022-03-13 17:11:41,816] jepsen worker 31 - jepsen.util 1741 :info :write [10 37] :RequestTimeoutException INFO [2022-03-13 17:11:41,817] jepsen worker 31 - jepsen.util 1831 :invoke :read [10 nil] INFO [2022-03-13 17:11:41,847] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,848] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,848] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:41,849] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:41,850] jepsen worker 21 - jepsen.util 741 :invoke :write [7 2] INFO [2022-03-13 17:11:41,851] jepsen worker 69 - jepsen.util 609 :invoke :write [23 47] INFO [2022-03-13 17:11:41,857] jepsen worker 46 - jepsen.util 1846 :info :write [15 21] :RequestTimeoutException INFO [2022-03-13 17:11:41,857] jepsen worker 46 - jepsen.util 1936 :invoke :read [15 nil] INFO [2022-03-13 17:11:41,905] jepsen worker 85 - jepsen.util 1975 :ok :read [28 44] INFO [2022-03-13 17:11:41,909] jepsen worker 85 - jepsen.util 1975 :invoke :write [28 45] INFO [2022-03-13 17:11:41,936] jepsen worker 46 - jepsen.util 1936 :ok :read [15 48] INFO [2022-03-13 17:11:41,937] jepsen worker 46 - jepsen.util 1936 :invoke :read [15 nil] INFO [2022-03-13 17:11:41,993] jepsen worker 46 - jepsen.util 1936 :ok :read [15 48] INFO [2022-03-13 17:11:41,994] jepsen worker 46 - jepsen.util 1936 :invoke :read [15 nil] INFO [2022-03-13 17:11:42,056] jepsen worker 46 - jepsen.util 1936 :ok :read [15 48] INFO [2022-03-13 17:11:42,056] jepsen worker 46 - jepsen.util 1936 :invoke :read [15 nil] INFO [2022-03-13 17:11:42,085] jepsen worker 46 - jepsen.util 1936 :ok :read [15 48] INFO [2022-03-13 17:11:42,085] jepsen worker 46 - jepsen.util 1936 :invoke :write [15 30] INFO [2022-03-13 17:11:42,178] jepsen worker 28 - jepsen.util 1918 :info :write [9 32] :RequestTimeoutException INFO [2022-03-13 17:11:42,179] jepsen worker 28 - jepsen.util 2008 :invoke :write [9 33] INFO [2022-03-13 17:11:42,190] jepsen worker 20 - jepsen.util 2180 :info :write [6 11] :RequestTimeoutException INFO [2022-03-13 17:11:42,191] jepsen worker 55 - jepsen.util 1765 :ok :read [18 21] INFO [2022-03-13 17:11:42,192] jepsen worker 20 - jepsen.util 2270 :invoke :write [6 23] INFO [2022-03-13 17:11:42,192] jepsen worker 55 - jepsen.util 1765 :invoke :write [18 49] INFO [2022-03-13 17:11:42,223] jepsen worker 9 - jepsen.util 1719 :ok :read [3 25] INFO [2022-03-13 17:11:42,224] jepsen worker 9 - jepsen.util 1719 :invoke :write [3 14] INFO [2022-03-13 17:11:42,326] jepsen worker 56 - jepsen.util 1766 :info :write [18 25] :RequestTimeoutException INFO [2022-03-13 17:11:42,326] jepsen worker 56 - jepsen.util 1856 :invoke :read [18 nil] INFO [2022-03-13 17:11:42,376] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,376] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,376] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:11:42,377] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:42,379] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:42,379] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:42,386] jepsen worker 19 - jepsen.util 1729 :info :write [6 21] :RequestTimeoutException INFO [2022-03-13 17:11:42,387] jepsen worker 19 - jepsen.util 1819 :invoke :read [6 nil] INFO [2022-03-13 17:11:42,467] jepsen worker 56 - jepsen.util 1856 :ok :read [18 21] INFO [2022-03-13 17:11:42,468] jepsen worker 56 - jepsen.util 1856 :invoke :read [18 nil] INFO [2022-03-13 17:11:42,476] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,478] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:11:42,592] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3372434017595308}, :ns_1@172.23.105.187 {:progress 0.5029239766081872}, :ns_1@172.23.107.227 {:progress 0.3372434017595308}, :ns_1@172.23.107.228 {:progress 0.4058823529411765}} INFO [2022-03-13 17:11:42,592] jepsen worker nemesis - couchbase.util Rebalance check count: 213 INFO [2022-03-13 17:11:42,592] jepsen worker nemesis - couchbase.util Rebalance stuck count: 134 INFO [2022-03-13 17:11:42,640] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,641] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:42,651] jepsen worker 56 - jepsen.util 1856 :ok :read [18 21] INFO [2022-03-13 17:11:42,651] jepsen worker 56 - jepsen.util 1856 :invoke :read [18 nil] INFO [2022-03-13 17:11:42,719] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:42,719] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,735] jepsen worker 42 - jepsen.util 402 :invoke :write [14 15] INFO [2022-03-13 17:11:42,735] jepsen worker 53 - jepsen.util 503 :invoke :write [17 7] INFO [2022-03-13 17:11:42,779] jepsen worker 56 - jepsen.util 1856 :ok :read [18 21] INFO [2022-03-13 17:11:42,779] jepsen worker 56 - jepsen.util 1856 :invoke :read [18 nil] INFO [2022-03-13 17:11:42,838] jepsen worker 60 - jepsen.util 600 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,838] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,839] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:11:42,856] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,859] jepsen worker 51 - jepsen.util 591 :invoke :write [17 23] INFO [2022-03-13 17:11:42,876] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:11:42,921] jepsen worker 56 - jepsen.util 1856 :ok :read [18 21] INFO [2022-03-13 17:11:42,922] jepsen worker 56 - jepsen.util 1856 :invoke :read [18 nil] INFO [2022-03-13 17:11:42,960] jepsen worker 79 - jepsen.util 889 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,961] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:11:42,968] jepsen worker 69 - jepsen.util 609 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:42,969] jepsen worker 69 - jepsen.util 609 :invoke :write [23 6] INFO [2022-03-13 17:11:43,034] jepsen worker 76 - jepsen.util 616 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,035] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:43,052] jepsen worker 56 - jepsen.util 1856 :ok :read [18 21] INFO [2022-03-13 17:11:43,053] jepsen worker 56 - jepsen.util 1856 :invoke :read [18 nil] INFO [2022-03-13 17:11:43,125] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,125] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:43,126] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,126] jepsen worker 6 - jepsen.util 636 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,126] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:43,126] jepsen worker 12 - jepsen.util 732 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,127] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:43,128] jepsen worker 6 - jepsen.util 636 :invoke :write [2 48] INFO [2022-03-13 17:11:43,128] jepsen worker 44 - jepsen.util 584 :invoke :write [14 33] INFO [2022-03-13 17:11:43,129] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:43,139] jepsen worker 21 - jepsen.util 741 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,140] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:43,200] jepsen worker 56 - jepsen.util 1856 :ok :read [18 21] INFO [2022-03-13 17:11:43,200] jepsen worker 56 - jepsen.util 1856 :invoke :write [18 21] INFO [2022-03-13 17:11:43,212] jepsen worker 26 - jepsen.util 476 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,212] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,213] jepsen worker 26 - jepsen.util 476 :invoke :write [8 49] INFO [2022-03-13 17:11:43,218] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:43,220] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,220] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,220] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,221] jepsen worker 79 - jepsen.util 889 :invoke :write [26 14] INFO [2022-03-13 17:11:43,221] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:43,222] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] INFO [2022-03-13 17:11:43,233] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,233] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,233] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,234] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:11:43,234] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:11:43,235] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:43,259] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,261] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:11:43,331] jepsen worker 79 - jepsen.util 889 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,331] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:43,341] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,342] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:43,377] jepsen worker 49 - jepsen.util 1489 :info :write [16 5] :SyncWriteAmbiguous INFO [2022-03-13 17:11:43,377] jepsen worker 49 - jepsen.util 1579 :invoke :read [16 nil] INFO [2022-03-13 17:11:43,380] jepsen worker 7 - jepsen.util 637 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,380] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:11:43,413] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,414] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:43,423] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,424] jepsen worker 23 - jepsen.util 563 :invoke :write [7 26] INFO [2022-03-13 17:11:43,425] jepsen worker 49 - jepsen.util 1579 :ok :read [16 :nil] INFO [2022-03-13 17:11:43,426] jepsen worker 49 - jepsen.util 1579 :invoke :read [16 nil] INFO [2022-03-13 17:11:43,427] jepsen worker 49 - jepsen.util 1579 :ok :read [16 :nil] INFO [2022-03-13 17:11:43,428] jepsen worker 49 - jepsen.util 1579 :invoke :read [16 nil] INFO [2022-03-13 17:11:43,429] jepsen worker 49 - jepsen.util 1579 :ok :read [16 :nil] INFO [2022-03-13 17:11:43,429] jepsen worker 49 - jepsen.util 1579 :invoke :write [16 29] INFO [2022-03-13 17:11:43,432] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,433] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:11:43,448] jepsen worker 50 - jepsen.util 1850 :info :write [16 1] :RequestTimeoutException INFO [2022-03-13 17:11:43,449] jepsen worker 50 - jepsen.util 1940 :invoke :read [16 nil] INFO [2022-03-13 17:11:43,463] jepsen worker 83 - jepsen.util 1343 :ok :read [27 4] INFO [2022-03-13 17:11:43,463] jepsen worker 83 - jepsen.util 1343 :invoke :read [27 nil] INFO [2022-03-13 17:11:43,501] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,501] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,502] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,502] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:43,503] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:43,503] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:11:43,512] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,512] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,513] jepsen worker 79 - jepsen.util 889 :invoke :write [26 47] INFO [2022-03-13 17:11:43,513] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:11:43,537] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,537] jepsen worker 69 - jepsen.util 609 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,537] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,538] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:43,538] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:43,539] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:43,544] jepsen worker 83 - jepsen.util 1343 :ok :read [27 4] INFO [2022-03-13 17:11:43,545] jepsen worker 83 - jepsen.util 1343 :invoke :write [27 49] INFO [2022-03-13 17:11:43,554] jepsen worker 45 - jepsen.util 2115 :info :write [15 26] :RequestTimeoutException INFO [2022-03-13 17:11:43,555] jepsen worker 45 - jepsen.util 2205 :invoke :write [15 33] INFO [2022-03-13 17:11:43,582] jepsen worker 61 - jepsen.util 601 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,583] jepsen worker 61 - jepsen.util 601 :invoke :write [20 26] INFO [2022-03-13 17:11:43,600] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3372434017595308}, :ns_1@172.23.105.187 {:progress 0.5029239766081872}, :ns_1@172.23.107.227 {:progress 0.3372434017595308}, :ns_1@172.23.107.228 {:progress 0.4058823529411765}} INFO [2022-03-13 17:11:43,600] jepsen worker nemesis - couchbase.util Rebalance check count: 214 INFO [2022-03-13 17:11:43,600] jepsen worker nemesis - couchbase.util Rebalance stuck count: 135 INFO [2022-03-13 17:11:43,616] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,617] jepsen worker 60 - jepsen.util 600 :invoke :write [20 11] INFO [2022-03-13 17:11:43,692] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,692] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,693] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] INFO [2022-03-13 17:11:43,693] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:11:43,703] jepsen worker 79 - jepsen.util 889 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,704] jepsen worker 79 - jepsen.util 889 :invoke :write [26 2] INFO [2022-03-13 17:11:43,734] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,736] jepsen worker 76 - jepsen.util 616 :invoke :write [25 17] INFO [2022-03-13 17:11:43,764] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,765] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:43,830] jepsen worker 79 - jepsen.util 889 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,831] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:43,844] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,844] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,844] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,845] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,845] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:43,846] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:43,846] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:11:43,846] jepsen worker 21 - jepsen.util 741 :invoke :write [7 15] INFO [2022-03-13 17:11:43,863] jepsen worker 76 - jepsen.util 616 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,863] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,864] jepsen worker 76 - jepsen.util 616 :invoke :write [25 9] INFO [2022-03-13 17:11:43,864] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:11:43,866] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,868] jepsen worker 79 - jepsen.util 889 :invoke :write [26 32] INFO [2022-03-13 17:11:43,920] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,920] jepsen worker 51 - jepsen.util 591 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,920] jepsen worker 43 - jepsen.util 493 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,921] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:43,921] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:43,922] jepsen worker 43 - jepsen.util 493 :invoke :write [14 44] INFO [2022-03-13 17:11:43,934] jepsen worker 11 - jepsen.util 1901 :info :write [3 13] :SyncWriteAmbiguous INFO [2022-03-13 17:11:43,934] jepsen worker 11 - jepsen.util 1991 :invoke :write [3 44] INFO [2022-03-13 17:11:43,939] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:43,940] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:11:43,942] jepsen worker 65 - jepsen.util 2135 :info :write [21 18] :SyncWriteAmbiguous INFO [2022-03-13 17:11:43,942] jepsen worker 65 - jepsen.util 2225 :invoke :write [21 18] INFO [2022-03-13 17:11:43,983] jepsen worker 4 - jepsen.util 2074 :ok :read [1 42] INFO [2022-03-13 17:11:43,984] jepsen worker 4 - jepsen.util 2074 :invoke :read [1 nil] INFO [2022-03-13 17:11:44,013] jepsen worker 4 - jepsen.util 2074 :ok :read [1 42] INFO [2022-03-13 17:11:44,013] jepsen worker 4 - jepsen.util 2074 :invoke :read [1 nil] INFO [2022-03-13 17:11:44,052] jepsen worker 4 - jepsen.util 2074 :ok :read [1 42] INFO [2022-03-13 17:11:44,053] jepsen worker 4 - jepsen.util 2074 :invoke :read [1 nil] INFO [2022-03-13 17:11:44,078] jepsen worker 4 - jepsen.util 2074 :ok :read [1 42] INFO [2022-03-13 17:11:44,078] jepsen worker 4 - jepsen.util 2074 :invoke :read [1 nil] INFO [2022-03-13 17:11:44,084] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,084] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,085] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,085] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:44,086] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:11:44,086] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:44,104] jepsen worker 4 - jepsen.util 2074 :ok :read [1 42] INFO [2022-03-13 17:11:44,104] jepsen worker 79 - jepsen.util 889 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,107] jepsen worker 4 - jepsen.util 2074 :invoke :write [1 17] INFO [2022-03-13 17:11:44,107] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:44,124] jepsen worker 84 - jepsen.util 1974 :info :write [28 32] :RequestTimeoutException INFO [2022-03-13 17:11:44,125] jepsen worker 84 - jepsen.util 2064 :invoke :read [28 nil] INFO [2022-03-13 17:11:44,144] jepsen worker 80 - jepsen.util 710 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,144] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:44,160] jepsen worker 29 - jepsen.util 1919 :info :write [9 43] :SyncWriteAmbiguous INFO [2022-03-13 17:11:44,161] jepsen worker 29 - jepsen.util 2009 :invoke :write [9 44] INFO [2022-03-13 17:11:44,191] jepsen worker 76 - jepsen.util 616 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,191] jepsen worker 76 - jepsen.util 616 :invoke :write [25 28] INFO [2022-03-13 17:11:44,268] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,271] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:11:44,271] jepsen worker 22 - jepsen.util 652 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,272] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:44,336] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,336] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,337] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:11:44,338] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:44,350] jepsen worker 53 - jepsen.util 503 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,351] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:44,352] jepsen worker 10 - jepsen.util 1900 :ok :read [3 25] INFO [2022-03-13 17:11:44,353] jepsen worker 10 - jepsen.util 1900 :invoke :write [3 9] INFO [2022-03-13 17:11:44,353] jepsen worker 19 - jepsen.util 1819 :ok :read [6 16] INFO [2022-03-13 17:11:44,354] jepsen worker 19 - jepsen.util 1819 :invoke :read [6 nil] INFO [2022-03-13 17:11:44,388] jepsen worker 21 - jepsen.util 741 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,388] jepsen worker 21 - jepsen.util 741 :invoke :write [7 1] INFO [2022-03-13 17:11:44,396] jepsen worker 19 - jepsen.util 1819 :ok :read [6 16] INFO [2022-03-13 17:11:44,396] jepsen worker 19 - jepsen.util 1819 :invoke :write [6 32] INFO [2022-03-13 17:11:44,427] jepsen worker 52 - jepsen.util 592 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,427] jepsen worker 52 - jepsen.util 592 :invoke :write [17 19] INFO [2022-03-13 17:11:44,504] jepsen worker 76 - jepsen.util 616 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,504] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,504] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:44,505] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:44,510] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,510] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,510] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,511] jepsen worker 44 - jepsen.util 584 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,511] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:44,511] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:44,512] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:11:44,512] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:44,558] jepsen worker 35 - jepsen.util 845 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,559] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,559] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:44,560] jepsen worker 22 - jepsen.util 652 :invoke :write [7 7] INFO [2022-03-13 17:11:44,606] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3372434017595308}, :ns_1@172.23.105.187 {:progress 0.5029239766081872}, :ns_1@172.23.107.227 {:progress 0.3372434017595308}, :ns_1@172.23.107.228 {:progress 0.4058823529411765}} INFO [2022-03-13 17:11:44,606] jepsen worker nemesis - couchbase.util Rebalance check count: 215 INFO [2022-03-13 17:11:44,607] jepsen worker nemesis - couchbase.util Rebalance stuck count: 136 INFO [2022-03-13 17:11:44,668] jepsen worker 37 - jepsen.util 1837 :info :write [12 5] :RequestTimeoutException INFO [2022-03-13 17:11:44,669] jepsen worker 37 - jepsen.util 1927 :invoke :write [12 4] INFO [2022-03-13 17:11:44,868] jepsen worker 73 - jepsen.util 1873 :info :write [24 49] :RequestTimeoutException INFO [2022-03-13 17:11:44,868] jepsen worker 73 - jepsen.util 1963 :invoke :read [24 nil] INFO [2022-03-13 17:11:44,871] jepsen worker 88 - jepsen.util 1798 :info :write [29 46] :RequestTimeoutException INFO [2022-03-13 17:11:44,872] jepsen worker 88 - jepsen.util 1888 :invoke :write [29 44] INFO [2022-03-13 17:11:44,874] jepsen worker 74 - jepsen.util 2234 :info :write [24 17] :RequestTimeoutException INFO [2022-03-13 17:11:44,876] jepsen worker 74 - jepsen.util 2324 :invoke :read [24 nil] INFO [2022-03-13 17:11:44,940] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,940] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:44,945] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,945] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:44,948] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,948] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,949] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,949] jepsen worker 79 - jepsen.util 889 :invoke :write [26 15] INFO [2022-03-13 17:11:44,950] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:44,950] jepsen worker 63 - jepsen.util 1773 :info :write [21 27] :RequestTimeoutException INFO [2022-03-13 17:11:44,950] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:44,951] jepsen worker 63 - jepsen.util 1863 :invoke :read [21 nil] INFO [2022-03-13 17:11:44,956] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,956] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:44,959] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,959] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:44,961] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:11:44,964] jepsen worker 76 - jepsen.util 616 :invoke :write [25 15] INFO [2022-03-13 17:11:45,023] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,024] jepsen worker 24 - jepsen.util 564 :invoke :write [8 18] INFO [2022-03-13 17:11:45,061] jepsen worker 79 - jepsen.util 889 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,063] jepsen worker 79 - jepsen.util 889 :invoke :write [26 38] INFO [2022-03-13 17:11:45,085] jepsen worker 13 - jepsen.util 643 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,085] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,086] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,087] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:45,087] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:45,087] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:45,101] jepsen worker 27 - jepsen.util 2007 :info :write [9 6] :RequestTimeoutException INFO [2022-03-13 17:11:45,102] jepsen worker 27 - jepsen.util 2097 :invoke :write [9 27] INFO [2022-03-13 17:11:45,124] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,124] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,124] jepsen worker 21 - jepsen.util 741 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,125] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:45,126] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:11:45,127] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:45,164] jepsen worker 76 - jepsen.util 616 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,165] jepsen worker 76 - jepsen.util 616 :invoke :write [25 28] INFO [2022-03-13 17:11:45,184] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,184] jepsen worker 22 - jepsen.util 652 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,185] jepsen worker 35 - jepsen.util 845 :invoke :write [11 34] INFO [2022-03-13 17:11:45,186] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:11:45,187] jepsen worker 74 - jepsen.util 2324 :ok :read [24 24] INFO [2022-03-13 17:11:45,188] jepsen worker 74 - jepsen.util 2324 :invoke :read [24 nil] INFO [2022-03-13 17:11:45,191] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,191] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,191] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:11:45,192] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:11:45,219] jepsen worker 63 - jepsen.util 1863 :ok :read [21 :nil] INFO [2022-03-13 17:11:45,219] jepsen worker 63 - jepsen.util 1863 :invoke :write [21 19] INFO [2022-03-13 17:11:45,219] jepsen worker 30 - jepsen.util 1740 :info :write [10 41] :RequestTimeoutException INFO [2022-03-13 17:11:45,220] jepsen worker 30 - jepsen.util 1830 :invoke :write [10 16] INFO [2022-03-13 17:11:45,274] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,274] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:45,282] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,282] jepsen worker 24 - jepsen.util 564 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,282] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:45,282] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,283] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:45,283] jepsen worker 21 - jepsen.util 741 :invoke :write [7 1] INFO [2022-03-13 17:11:45,315] jepsen worker 70 - jepsen.util 700 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,316] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:45,323] jepsen worker 74 - jepsen.util 2324 :ok :read [24 24] INFO [2022-03-13 17:11:45,324] jepsen worker 74 - jepsen.util 2324 :invoke :read [24 nil] INFO [2022-03-13 17:11:45,335] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,339] jepsen worker 80 - jepsen.util 710 :invoke :write [26 38] INFO [2022-03-13 17:11:45,351] jepsen worker 74 - jepsen.util 2324 :ok :read [24 24] INFO [2022-03-13 17:11:45,351] jepsen worker 74 - jepsen.util 2324 :invoke :write [24 4] INFO [2022-03-13 17:11:45,371] jepsen worker 79 - jepsen.util 889 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,372] jepsen worker 79 - jepsen.util 889 :invoke :write [26 24] INFO [2022-03-13 17:11:45,417] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,417] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:45,464] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,465] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:11:45,468] jepsen worker 76 - jepsen.util 616 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,468] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,469] jepsen worker 76 - jepsen.util 616 :invoke :write [25 5] INFO [2022-03-13 17:11:45,469] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:45,493] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,494] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:45,550] jepsen worker 32 - jepsen.util 1652 :info :write [10 8] :RequestTimeoutException INFO [2022-03-13 17:11:45,551] jepsen worker 32 - jepsen.util 1742 :invoke :read [10 nil] INFO [2022-03-13 17:11:45,556] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,556] jepsen worker 42 - jepsen.util 402 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,557] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:11:45,557] jepsen worker 14 - jepsen.util 284 :invoke :write [4 34] INFO [2022-03-13 17:11:45,567] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,567] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,567] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,568] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:11:45,568] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:11:45,569] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:45,581] jepsen worker 54 - jepsen.util 1584 :info :write [18 24] :RequestTimeoutException INFO [2022-03-13 17:11:45,582] jepsen worker 54 - jepsen.util 1674 :invoke :read [18 nil] INFO [2022-03-13 17:11:45,582] jepsen worker 79 - jepsen.util 889 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,583] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,583] jepsen worker 79 - jepsen.util 889 :invoke :write [26 31] INFO [2022-03-13 17:11:45,584] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:11:45,612] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3401759530791789}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.3401759530791789}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:45,613] jepsen worker nemesis - couchbase.util Rebalance check count: 216 INFO [2022-03-13 17:11:45,613] jepsen worker nemesis - couchbase.util Rebalance stuck count: 136 INFO [2022-03-13 17:11:45,621] jepsen worker 52 - jepsen.util 592 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,621] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,622] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,622] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,622] jepsen worker 13 - jepsen.util 643 :invoke :write [4 18] INFO [2022-03-13 17:11:45,622] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,623] jepsen worker 52 - jepsen.util 592 :invoke :write [17 28] INFO [2022-03-13 17:11:45,624] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:45,625] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:45,626] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:45,631] jepsen worker 18 - jepsen.util 1638 :info :write [6 11] :RequestTimeoutException INFO [2022-03-13 17:11:45,631] jepsen worker 18 - jepsen.util 1728 :invoke :write [6 14] INFO [2022-03-13 17:11:45,660] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,660] jepsen worker 21 - jepsen.util 741 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,661] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,661] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,662] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:11:45,662] jepsen worker 71 - jepsen.util 521 :invoke :write [23 42] INFO [2022-03-13 17:11:45,663] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:11:45,664] jepsen worker 21 - jepsen.util 741 :invoke :write [7 41] INFO [2022-03-13 17:11:45,699] jepsen worker 35 - jepsen.util 845 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,699] jepsen worker 35 - jepsen.util 845 :invoke :write [11 3] INFO [2022-03-13 17:11:45,766] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,767] jepsen worker 77 - jepsen.util 707 :invoke :write [25 36] INFO [2022-03-13 17:11:45,776] jepsen worker 76 - jepsen.util 616 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,776] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:45,782] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,782] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] WARN [2022-03-13 17:11:45,844] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9129081,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10334432,"total_server_duration_us":9359145,"encode_duration_us":24696,"operation_id":"0x51b5","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10007855},{"operation_name":"upsert","last_dispatch_duration_us":8693202,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9587524,"total_server_duration_us":374483,"encode_duration_us":16814,"operation_id":"0x51f7","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10007230},{"operation_name":"upsert","last_dispatch_duration_us":9087319,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9811231,"total_server_duration_us":9512369,"encode_duration_us":19213,"operation_id":"0x51fd","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10006944},{"operation_name":"upsert","last_dispatch_duration_us":48791,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1123702,"total_server_duration_us":496067,"encode_duration_us":24917,"operation_id":"0x5429","timeout_ms":10000,"last_server_duration_us":33230,"total_duration_us":10006508},{"operation_name":"upsert","last_dispatch_duration_us":8862260,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9987947,"total_server_duration_us":366418,"encode_duration_us":24722,"operation_id":"0x51b1","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10005088},{"operation_name":"upsert","last_dispatch_duration_us":113386,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1267210,"total_server_duration_us":497525,"encode_duration_us":18136,"operation_id":"0x542b","timeout_ms":10000,"last_server_duration_us":47876,"total_duration_us":10003829},{"operation_name":"upsert","last_dispatch_duration_us":9094278,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9535882,"total_server_duration_us":264501,"encode_duration_us":17023,"operation_id":"0x5267","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10003130},{"operation_name":"upsert","last_dispatch_duration_us":151586,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1318565,"total_server_duration_us":478254,"encode_duration_us":18822,"operation_id":"0x5439","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10003044},{"operation_name":"upsert","last_dispatch_duration_us":8761362,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9992228,"total_server_duration_us":403279,"encode_duration_us":19712,"operation_id":"0x51b2","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10001372},{"operation_name":"upsert","last_dispatch_duration_us":9090849,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9512660,"total_server_duration_us":9250638,"encode_duration_us":16801,"operation_id":"0x52ad","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9671731}],"total_count":22}} WARN [2022-03-13 17:11:45,845] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9087319,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9811231,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9512369,"encode_duration_us":19213,"operation_id":"0x51fd","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10006944},{"operation_name":"upsert","last_dispatch_duration_us":9129081,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10334432,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9359145,"encode_duration_us":24696,"operation_id":"0x51b5","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10007855}],"total_count":2}} INFO [2022-03-13 17:11:45,905] jepsen worker 80 - jepsen.util 710 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,905] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,905] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,906] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,906] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:45,906] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:45,907] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:45,907] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:45,909] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,909] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:45,941] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:45,942] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:46,022] jepsen worker 79 - jepsen.util 889 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,023] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:11:46,035] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,035] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:46,062] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,063] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:11:46,086] jepsen worker 64 - jepsen.util 1774 :info :write [21 10] :RequestTimeoutException INFO [2022-03-13 17:11:46,087] jepsen worker 64 - jepsen.util 1864 :invoke :read [21 nil] INFO [2022-03-13 17:11:46,130] jepsen worker 64 - jepsen.util 1864 :ok :read [21 :nil] INFO [2022-03-13 17:11:46,130] jepsen worker 64 - jepsen.util 1864 :invoke :read [21 nil] INFO [2022-03-13 17:11:46,132] jepsen worker 64 - jepsen.util 1864 :ok :read [21 :nil] INFO [2022-03-13 17:11:46,133] jepsen worker 64 - jepsen.util 1864 :invoke :read [21 nil] INFO [2022-03-13 17:11:46,134] jepsen worker 64 - jepsen.util 1864 :ok :read [21 :nil] INFO [2022-03-13 17:11:46,135] jepsen worker 64 - jepsen.util 1864 :invoke :read [21 nil] INFO [2022-03-13 17:11:46,137] jepsen worker 64 - jepsen.util 1864 :ok :read [21 :nil] INFO [2022-03-13 17:11:46,137] jepsen worker 64 - jepsen.util 1864 :invoke :read [21 nil] INFO [2022-03-13 17:11:46,138] jepsen worker 64 - jepsen.util 1864 :ok :read [21 :nil] INFO [2022-03-13 17:11:46,139] jepsen worker 64 - jepsen.util 1864 :invoke :read [21 nil] INFO [2022-03-13 17:11:46,141] jepsen worker 64 - jepsen.util 1864 :ok :read [21 :nil] INFO [2022-03-13 17:11:46,141] jepsen worker 64 - jepsen.util 1864 :invoke :write [21 31] INFO [2022-03-13 17:11:46,167] jepsen worker 13 - jepsen.util 643 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,168] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:46,174] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,174] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,175] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:46,175] jepsen worker 71 - jepsen.util 521 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,175] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,176] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:46,176] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:11:46,177] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:46,327] jepsen worker 35 - jepsen.util 845 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,327] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,327] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:11:46,328] jepsen worker 35 - jepsen.util 845 :invoke :write [11 39] INFO [2022-03-13 17:11:46,337] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,337] jepsen worker 21 - jepsen.util 741 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,337] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,338] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:46,338] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:46,338] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:11:46,345] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,345] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,345] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,346] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:46,347] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:46,347] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:11:46,355] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,355] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,357] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:46,358] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:11:46,358] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,359] jepsen worker 76 - jepsen.util 616 :invoke :write [25 23] INFO [2022-03-13 17:11:46,376] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,376] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,377] jepsen worker 69 - jepsen.util 609 :invoke :write [23 12] INFO [2022-03-13 17:11:46,378] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:11:46,430] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,430] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,430] jepsen worker 52 - jepsen.util 592 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,430] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,430] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,430] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,431] jepsen worker 70 - jepsen.util 700 :invoke :write [23 17] INFO [2022-03-13 17:11:46,432] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:46,433] jepsen worker 52 - jepsen.util 592 :invoke :write [17 12] INFO [2022-03-13 17:11:46,433] jepsen worker 34 - jepsen.util 754 :invoke :write [11 13] INFO [2022-03-13 17:11:46,434] jepsen worker 80 - jepsen.util 710 :invoke :write [26 40] INFO [2022-03-13 17:11:46,440] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,441] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:46,441] jepsen worker 78 - jepsen.util 528 :invoke :write [26 24] INFO [2022-03-13 17:11:46,447] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,448] jepsen worker 13 - jepsen.util 643 :invoke :write [4 30] INFO [2022-03-13 17:11:46,490] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,490] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,490] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,491] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:46,492] jepsen worker 71 - jepsen.util 521 :invoke :write [23 37] INFO [2022-03-13 17:11:46,493] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] INFO [2022-03-13 17:11:46,552] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,553] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:11:46,569] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,570] jepsen worker 79 - jepsen.util 889 :invoke :write [26 21] INFO [2022-03-13 17:11:46,619] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3401759530791789}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.3401759530791789}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:46,619] jepsen worker nemesis - couchbase.util Rebalance check count: 217 INFO [2022-03-13 17:11:46,619] jepsen worker nemesis - couchbase.util Rebalance stuck count: 137 INFO [2022-03-13 17:11:46,634] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,635] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:46,696] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,697] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:11:46,700] jepsen worker 35 - jepsen.util 845 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,700] jepsen worker 6 - jepsen.util 636 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,700] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,700] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,701] jepsen worker 35 - jepsen.util 845 :invoke :write [11 45] INFO [2022-03-13 17:11:46,701] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:46,702] jepsen worker 7 - jepsen.util 637 :invoke :write [2 26] INFO [2022-03-13 17:11:46,706] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:11:46,765] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,766] jepsen worker 24 - jepsen.util 564 :invoke :write [8 15] INFO [2022-03-13 17:11:46,820] jepsen worker 70 - jepsen.util 700 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,820] jepsen worker 70 - jepsen.util 700 :invoke :write [23 18] INFO [2022-03-13 17:11:46,827] jepsen worker 76 - jepsen.util 616 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,828] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:46,883] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,883] jepsen worker 69 - jepsen.util 609 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,883] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:46,884] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:46,896] jepsen worker 40 - jepsen.util 1840 :info :write [13 0] :RequestTimeoutException INFO [2022-03-13 17:11:46,896] jepsen worker 40 - jepsen.util 1930 :invoke :write [13 15] INFO [2022-03-13 17:11:46,952] jepsen worker 23 - jepsen.util 563 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,952] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:11:46,961] jepsen worker 79 - jepsen.util 889 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:46,962] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:47,025] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,025] jepsen worker 80 - jepsen.util 710 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,028] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:11:47,028] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,029] jepsen worker 80 - jepsen.util 710 :invoke :write [26 30] INFO [2022-03-13 17:11:47,034] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:47,089] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,089] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,089] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,089] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:11:47,090] jepsen worker 76 - jepsen.util 616 :invoke :write [25 4] INFO [2022-03-13 17:11:47,090] jepsen worker 79 - jepsen.util 889 :invoke :write [26 7] INFO [2022-03-13 17:11:47,150] jepsen worker 71 - jepsen.util 521 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,151] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,151] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:47,152] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:11:47,204] jepsen worker 52 - jepsen.util 592 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,204] jepsen worker 35 - jepsen.util 845 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,205] jepsen worker 35 - jepsen.util 845 :invoke :write [11 31] INFO [2022-03-13 17:11:47,206] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:47,217] jepsen worker 24 - jepsen.util 564 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,217] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,217] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,218] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,218] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,219] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:11:47,227] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:47,227] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:11:47,228] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:47,228] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] WARN [2022-03-13 17:11:47,241] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9072764,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":16053525,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9391086,"encode_duration_us":19425,"operation_id":"0x51fb","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10004720},{"operation_name":"upsert","last_dispatch_duration_us":17658411,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18086567,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9097185,"encode_duration_us":23101,"operation_id":"0x507d","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10006216},{"operation_name":"upsert","last_dispatch_duration_us":17796790,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18501116,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9184587,"encode_duration_us":18017,"operation_id":"0x506d","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10008205},{"operation_name":"upsert","last_dispatch_duration_us":10956191,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11983781,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9341908,"encode_duration_us":17152,"operation_id":"0x5110","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10008369}],"total_count":4}} INFO [2022-03-13 17:11:47,248] jepsen worker 79 - jepsen.util 889 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,249] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:47,277] jepsen worker 17 - jepsen.util 1817 :info :write [5 46] :RequestTimeoutException INFO [2022-03-13 17:11:47,278] jepsen worker 17 - jepsen.util 1907 :invoke :read [5 nil] INFO [2022-03-13 17:11:47,350] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,350] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,350] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,351] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,351] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:47,352] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:47,352] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:11:47,352] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] WARN [2022-03-13 17:11:47,355] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16569650,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16940282,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9135507,"encode_duration_us":17334,"operation_id":"0x5135","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10001387},{"operation_name":"upsert","last_dispatch_duration_us":9064371,"last_local_id":"A312307A00000004/0000000095638474","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9579171,"last_local_socket":"172.23.104.79:33498","total_server_duration_us":9370314,"encode_duration_us":18727,"operation_id":"0x5152","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10002085}],"total_count":2}} INFO [2022-03-13 17:11:47,405] jepsen worker 34 - jepsen.util 754 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,405] jepsen worker 34 - jepsen.util 754 :invoke :write [11 34] INFO [2022-03-13 17:11:47,412] jepsen worker 39 - jepsen.util 1839 :info :write [13 11] :SyncWriteAmbiguous INFO [2022-03-13 17:11:47,413] jepsen worker 39 - jepsen.util 1929 :invoke :write [13 27] WARN [2022-03-13 17:11:47,455] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":12613717,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":12613717,"total_server_duration_us":9039489,"encode_duration_us":18227,"operation_id":"0x5215","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10010112},{"operation_name":"upsert","last_dispatch_duration_us":119631,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":520023,"total_server_duration_us":167436,"encode_duration_us":18956,"operation_id":"0x53fc","timeout_ms":10000,"last_server_duration_us":32743,"total_duration_us":10005647},{"operation_name":"upsert","last_dispatch_duration_us":48369,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1022188,"total_server_duration_us":647898,"encode_duration_us":20016,"operation_id":"0x53f7","timeout_ms":10000,"last_server_duration_us":33328,"total_duration_us":10004932},{"operation_name":"upsert","last_dispatch_duration_us":8419899,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8809211,"total_server_duration_us":85669,"encode_duration_us":21716,"operation_id":"0x540d","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10002888},{"operation_name":"upsert","last_dispatch_duration_us":8419640,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11764501,"total_server_duration_us":192589,"encode_duration_us":18320,"operation_id":"0x52fc","timeout_ms":10000,"last_server_duration_us":29601,"total_duration_us":10002661},{"operation_name":"upsert","last_dispatch_duration_us":8438333,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8877964,"total_server_duration_us":103458,"encode_duration_us":17370,"operation_id":"0x5405","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10002076},{"operation_name":"upsert","last_dispatch_duration_us":9271026,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9271026,"total_server_duration_us":9055417,"encode_duration_us":26146,"operation_id":"0x53de","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":9275028},{"operation_name":"get","last_dispatch_duration_us":7742154,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7742154,"total_server_duration_us":62,"operation_id":"0x5357","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":7744120},{"operation_name":"upsert","last_dispatch_duration_us":5611084,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5611084,"total_server_duration_us":120649,"encode_duration_us":17258,"operation_id":"0x5565","timeout_ms":10000,"last_server_duration_us":120649,"total_duration_us":5618063},{"operation_name":"upsert","last_dispatch_duration_us":4221477,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4221477,"total_server_duration_us":96096,"encode_duration_us":18749,"operation_id":"0x54ac","timeout_ms":10000,"last_server_duration_us":96096,"total_duration_us":4225472}],"total_count":16}} WARN [2022-03-13 17:11:47,456] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":33344,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":741829,"total_server_duration_us":432147,"encode_duration_us":18487,"operation_id":"0x5234","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10010306},{"operation_name":"upsert","last_dispatch_duration_us":7108080,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8304135,"total_server_duration_us":427583,"encode_duration_us":18265,"operation_id":"0x52e8","timeout_ms":10000,"last_server_duration_us":31203,"total_duration_us":10010000},{"operation_name":"upsert","last_dispatch_duration_us":70042,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8979635,"total_server_duration_us":235632,"encode_duration_us":18649,"operation_id":"0x5230","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10007210},{"operation_name":"upsert","last_dispatch_duration_us":9107291,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9746282,"total_server_duration_us":9373778,"encode_duration_us":19120,"operation_id":"0x5301","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10006482},{"operation_name":"upsert","last_dispatch_duration_us":37046,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8567965,"total_server_duration_us":411860,"encode_duration_us":20177,"operation_id":"0x544e","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10005814},{"operation_name":"upsert","last_dispatch_duration_us":9072551,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":9840985,"total_server_duration_us":9485380,"encode_duration_us":23417,"operation_id":"0x51f2","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10002522},{"operation_name":"upsert","last_dispatch_duration_us":9163003,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9281779,"total_server_duration_us":9187981,"encode_duration_us":18221,"operation_id":"0x53f2","timeout_ms":10000,"last_server_duration_us":9117117,"total_duration_us":9303002},{"operation_name":"upsert","last_dispatch_duration_us":9093484,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9177238,"total_server_duration_us":9058853,"encode_duration_us":19345,"operation_id":"0x5232","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":9191350},{"operation_name":"get","last_dispatch_duration_us":7319686,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7319686,"total_server_duration_us":57139,"operation_id":"0x529e","timeout_ms":30000,"last_server_duration_us":57139,"total_duration_us":7322268},{"operation_name":"get","last_dispatch_duration_us":3849750,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":3849750,"total_server_duration_us":9513,"operation_id":"0x53a9","timeout_ms":30000,"last_server_duration_us":9513,"total_duration_us":3851583}],"total_count":41}} WARN [2022-03-13 17:11:47,456] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9072551,"last_local_id":"A312307A00000005/00000000F4F37B9B","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9840985,"last_local_socket":"172.23.104.79:33502","total_server_duration_us":9485380,"encode_duration_us":23417,"operation_id":"0x51f2","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10002522},{"operation_name":"upsert","last_dispatch_duration_us":9107291,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9746282,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9373778,"encode_duration_us":19120,"operation_id":"0x5301","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10006482}],"total_count":2}} INFO [2022-03-13 17:11:47,473] jepsen worker 3 - jepsen.util 1803 :info :write [1 32] :RequestTimeoutException INFO [2022-03-13 17:11:47,474] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:11:47,504] jepsen worker 87 - jepsen.util 2427 :info :write [29 44] :RequestTimeoutException INFO [2022-03-13 17:11:47,505] jepsen worker 87 - jepsen.util 2517 :invoke :read [29 nil] INFO [2022-03-13 17:11:47,527] jepsen worker 87 - jepsen.util 2517 :ok :read [29 10] INFO [2022-03-13 17:11:47,528] jepsen worker 87 - jepsen.util 2517 :invoke :read [29 nil] INFO [2022-03-13 17:11:47,544] jepsen worker 59 - jepsen.util 1679 :info :write [19 49] :RequestTimeoutException INFO [2022-03-13 17:11:47,545] jepsen worker 59 - jepsen.util 1769 :invoke :read [19 nil] WARN [2022-03-13 17:11:47,550] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":37576,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9023913,"total_server_duration_us":207362,"encode_duration_us":18796,"operation_id":"0x52fa","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10009886},{"operation_name":"upsert","last_dispatch_duration_us":10702281,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10702281,"total_server_duration_us":20282,"encode_duration_us":16862,"operation_id":"0x51c0","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10008874},{"operation_name":"upsert","last_dispatch_duration_us":33281,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5000716,"total_server_duration_us":85877,"encode_duration_us":17497,"operation_id":"0x5356","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10008077},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":17791,"operation_id":"0x53f5","timeout_ms":10000,"total_duration_us":10007542},{"operation_name":"upsert","last_dispatch_duration_us":10076019,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10076019,"total_server_duration_us":9028877,"encode_duration_us":17115,"operation_id":"0x54b7","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10007090},{"operation_name":"upsert","last_dispatch_duration_us":9072764,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":16053525,"total_server_duration_us":9391086,"encode_duration_us":19425,"operation_id":"0x51fb","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10004720},{"operation_name":"upsert","last_dispatch_duration_us":9073383,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9746035,"total_server_duration_us":348477,"encode_duration_us":19608,"operation_id":"0x53df","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10002070},{"operation_name":"upsert","last_dispatch_duration_us":9267384,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9267384,"total_server_duration_us":9052231,"encode_duration_us":19907,"operation_id":"0x52b7","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9271625},{"operation_name":"get","last_dispatch_duration_us":8957136,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8957136,"total_server_duration_us":62,"operation_id":"0x52e0","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":8959026},{"operation_name":"get","last_dispatch_duration_us":8601510,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8601510,"total_server_duration_us":8502,"operation_id":"0x5266","timeout_ms":30000,"last_server_duration_us":8502,"total_duration_us":8604084}],"total_count":55}} INFO [2022-03-13 17:11:47,552] jepsen worker 87 - jepsen.util 2517 :ok :read [29 10] INFO [2022-03-13 17:11:47,552] jepsen worker 87 - jepsen.util 2517 :invoke :read [29 nil] WARN [2022-03-13 17:11:47,555] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":12613717,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12613717,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9039489,"encode_duration_us":18227,"operation_id":"0x5215","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10010112}],"total_count":1}} WARN [2022-03-13 17:11:47,556] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15054267,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":15054267,"total_server_duration_us":38,"operation_id":"0x51aa","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":15056372},{"operation_name":"get","last_dispatch_duration_us":10615850,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10615850,"total_server_duration_us":23,"operation_id":"0x52d9","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":10619269},{"operation_name":"upsert","last_dispatch_duration_us":38917,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8483510,"total_server_duration_us":262231,"encode_duration_us":32625,"operation_id":"0x53f6","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10008284},{"operation_name":"upsert","last_dispatch_duration_us":52530,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9333257,"total_server_duration_us":241196,"encode_duration_us":19487,"operation_id":"0x539e","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10002178},{"operation_name":"upsert","last_dispatch_duration_us":80399,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9649129,"total_server_duration_us":232757,"encode_duration_us":17461,"operation_id":"0x536c","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10001192},{"operation_name":"upsert","last_dispatch_duration_us":9060449,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9320762,"total_server_duration_us":9179249,"encode_duration_us":18390,"operation_id":"0x5394","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9395770},{"operation_name":"get","last_dispatch_duration_us":4871509,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":4871509,"total_server_duration_us":43,"operation_id":"0x5552","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":4873781},{"operation_name":"get","last_dispatch_duration_us":878315,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":878315,"total_server_duration_us":49,"operation_id":"0x5395","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":4005472},{"operation_name":"upsert","last_dispatch_duration_us":3370756,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3370756,"total_server_duration_us":125429,"encode_duration_us":19640,"operation_id":"0x56f3","timeout_ms":10000,"last_server_duration_us":125429,"total_duration_us":3551896},{"operation_name":"get","last_dispatch_duration_us":3370890,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3370890,"total_server_duration_us":62,"operation_id":"0x56f6","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":3481233}],"total_count":53}} INFO [2022-03-13 17:11:47,577] jepsen worker 87 - jepsen.util 2517 :ok :read [29 10] INFO [2022-03-13 17:11:47,578] jepsen worker 87 - jepsen.util 2517 :invoke :write [29 5] INFO [2022-03-13 17:11:47,627] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3401759530791789}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.3401759530791789}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:47,627] jepsen worker nemesis - couchbase.util Rebalance check count: 218 INFO [2022-03-13 17:11:47,627] jepsen worker nemesis - couchbase.util Rebalance stuck count: 138 WARN [2022-03-13 17:11:47,650] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14084766,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":14084766,"total_server_duration_us":43,"operation_id":"0x51ac","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":14087143},{"operation_name":"get","last_dispatch_duration_us":13253538,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":13253538,"total_server_duration_us":27,"operation_id":"0x51e5","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":13258480},{"operation_name":"upsert","last_dispatch_duration_us":8357854,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9081064,"total_server_duration_us":408029,"encode_duration_us":18141,"operation_id":"0x52eb","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10008663},{"operation_name":"upsert","last_dispatch_duration_us":93905,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":1151347,"total_server_duration_us":697412,"encode_duration_us":18812,"operation_id":"0x523d","timeout_ms":10000,"last_server_duration_us":31585,"total_duration_us":10006626},{"operation_name":"upsert","last_dispatch_duration_us":8369950,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9110694,"total_server_duration_us":338366,"encode_duration_us":20412,"operation_id":"0x5303","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10005541},{"operation_name":"upsert","last_dispatch_duration_us":35084,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5753129,"total_server_duration_us":284549,"encode_duration_us":21075,"operation_id":"0x549c","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10003329},{"operation_name":"upsert","last_dispatch_duration_us":119772,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":782768,"total_server_duration_us":515481,"encode_duration_us":16835,"operation_id":"0x52ac","timeout_ms":10000,"last_server_duration_us":97638,"total_duration_us":10000882},{"operation_name":"upsert","last_dispatch_duration_us":9083048,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9083048,"total_server_duration_us":9056479,"encode_duration_us":18553,"operation_id":"0x53f3","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":9086401},{"operation_name":"get","last_dispatch_duration_us":7633418,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7633418,"total_server_duration_us":13091,"operation_id":"0x5379","timeout_ms":30000,"last_server_duration_us":13091,"total_duration_us":7637055},{"operation_name":"upsert","last_dispatch_duration_us":7442932,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7442932,"total_server_duration_us":66168,"encode_duration_us":23218,"operation_id":"0x54f6","timeout_ms":10000,"last_server_duration_us":66168,"total_duration_us":7451218}],"total_count":39}} INFO [2022-03-13 17:11:47,673] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,674] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:47,676] jepsen worker 60 - jepsen.util 600 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,676] jepsen worker 26 - jepsen.util 476 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,676] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,676] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:47,676] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:47,676] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,677] jepsen worker 26 - jepsen.util 476 :invoke :write [8 28] INFO [2022-03-13 17:11:47,678] jepsen worker 12 - jepsen.util 732 :invoke :write [4 20] INFO [2022-03-13 17:11:47,679] jepsen worker 44 - jepsen.util 584 :invoke :write [14 30] INFO [2022-03-13 17:11:47,679] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:47,683] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,683] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] WARN [2022-03-13 17:11:47,750] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17543774,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17754706,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9163786,"encode_duration_us":17984,"operation_id":"0x50b3","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10010014}],"total_count":1}} INFO [2022-03-13 17:11:47,752] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,762] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:47,771] jepsen worker 76 - jepsen.util 616 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,771] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,771] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,771] jepsen worker 76 - jepsen.util 616 :invoke :write [25 12] INFO [2022-03-13 17:11:47,772] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:47,772] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:47,775] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,775] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,776] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:47,776] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:11:47,777] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,777] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:47,778] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,779] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:47,786] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,786] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,787] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,786] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,787] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,787] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:47,788] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:11:47,789] jepsen worker 79 - jepsen.util 889 :invoke :write [26 20] INFO [2022-03-13 17:11:47,789] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:47,790] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:47,842] jepsen worker 61 - jepsen.util 601 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,843] jepsen worker 61 - jepsen.util 601 :invoke :write [20 26] INFO [2022-03-13 17:11:47,902] jepsen worker 77 - jepsen.util 707 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,902] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:47,909] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,910] jepsen worker 71 - jepsen.util 521 :invoke :write [23 10] INFO [2022-03-13 17:11:47,920] jepsen worker 76 - jepsen.util 616 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,921] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:47,983] jepsen worker 78 - jepsen.util 528 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,983] jepsen worker 35 - jepsen.util 845 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,983] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,984] jepsen worker 78 - jepsen.util 528 :invoke :write [26 49] INFO [2022-03-13 17:11:47,985] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:47,985] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:47,986] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,987] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,987] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:47,988] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:11:47,996] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,997] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:47,998] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:47,999] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:48,000] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,001] jepsen worker 35 - jepsen.util 845 :invoke :write [11 8] INFO [2022-03-13 17:11:48,005] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,005] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,005] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,005] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,006] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:48,006] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:48,007] jepsen worker 21 - jepsen.util 741 :invoke :write [7 34] INFO [2022-03-13 17:11:48,007] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:11:48,020] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,021] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:48,039] jepsen worker 43 - jepsen.util 493 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,039] jepsen worker 43 - jepsen.util 493 :invoke :write [14 44] INFO [2022-03-13 17:11:48,078] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,078] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:11:48,105] jepsen worker 16 - jepsen.util 1726 :ok :read [5 43] INFO [2022-03-13 17:11:48,105] jepsen worker 16 - jepsen.util 1726 :invoke :write [5 46] INFO [2022-03-13 17:11:48,227] jepsen worker 58 - jepsen.util 1678 :info :write [19 23] :RequestTimeoutException INFO [2022-03-13 17:11:48,227] jepsen worker 58 - jepsen.util 1768 :invoke :write [19 20] INFO [2022-03-13 17:11:48,296] jepsen worker 70 - jepsen.util 700 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,296] jepsen worker 70 - jepsen.util 700 :invoke :write [23 43] INFO [2022-03-13 17:11:48,300] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,300] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,300] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,300] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:48,301] jepsen worker 51 - jepsen.util 591 :invoke :write [17 24] INFO [2022-03-13 17:11:48,301] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:48,309] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,309] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,309] jepsen worker 79 - jepsen.util 889 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,310] jepsen worker 76 - jepsen.util 616 :invoke :write [25 36] INFO [2022-03-13 17:11:48,310] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:48,311] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:48,312] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,312] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:48,315] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,315] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,316] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:48,317] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:48,317] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,318] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:48,318] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,318] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:11:48,319] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,320] jepsen worker 79 - jepsen.util 889 :invoke :write [26 12] INFO [2022-03-13 17:11:48,343] jepsen worker 82 - jepsen.util 1792 :info :write [27 34] :RequestTimeoutException INFO [2022-03-13 17:11:48,344] jepsen worker 82 - jepsen.util 1882 :invoke :write [27 37] INFO [2022-03-13 17:11:48,368] jepsen worker 71 - jepsen.util 521 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,368] jepsen worker 5 - jepsen.util 2165 :info :write [1 6] :RequestTimeoutException INFO [2022-03-13 17:11:48,368] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,369] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:11:48,369] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,369] jepsen worker 5 - jepsen.util 2255 :invoke :write [1 24] INFO [2022-03-13 17:11:48,370] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:48,370] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:48,421] jepsen worker 8 - jepsen.util 728 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,421] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,421] jepsen worker 80 - jepsen.util 710 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,421] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,422] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:11:48,422] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:48,423] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:48,423] jepsen worker 53 - jepsen.util 503 :invoke :write [17 41] INFO [2022-03-13 17:11:48,432] jepsen worker 76 - jepsen.util 616 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,432] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:48,449] jepsen worker 21 - jepsen.util 741 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,449] jepsen worker 21 - jepsen.util 741 :invoke :write [7 15] INFO [2022-03-13 17:11:48,450] jepsen worker 89 - jepsen.util 2249 :info :write [29 1] :RequestTimeoutException INFO [2022-03-13 17:11:48,451] jepsen worker 89 - jepsen.util 2339 :invoke :write [29 16] INFO [2022-03-13 17:11:48,455] jepsen worker 44 - jepsen.util 584 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,456] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:48,498] jepsen worker 1 - jepsen.util 1531 :info :write [0 3] :RequestTimeoutException INFO [2022-03-13 17:11:48,498] jepsen worker 1 - jepsen.util 1621 :invoke :read [0 nil] INFO [2022-03-13 17:11:48,505] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,506] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:48,533] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,534] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:11:48,541] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,542] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:48,611] jepsen worker 34 - jepsen.util 754 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,612] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:48,628] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,628] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,628] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,630] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:48,630] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:48,631] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:48,633] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.343108504398827}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.343108504398827}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:48,633] jepsen worker nemesis - couchbase.util Rebalance check count: 219 INFO [2022-03-13 17:11:48,633] jepsen worker nemesis - couchbase.util Rebalance stuck count: 138 INFO [2022-03-13 17:11:48,648] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,648] jepsen worker 12 - jepsen.util 732 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,648] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,648] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,648] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,649] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:48,649] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:11:48,651] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:48,651] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:11:48,652] jepsen worker 77 - jepsen.util 707 :invoke :write [25 48] INFO [2022-03-13 17:11:48,656] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,656] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,656] jepsen worker 79 - jepsen.util 889 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,657] jepsen worker 76 - jepsen.util 616 :invoke :write [25 33] INFO [2022-03-13 17:11:48,658] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:11:48,658] jepsen worker 79 - jepsen.util 889 :invoke :write [26 35] INFO [2022-03-13 17:11:48,667] jepsen worker 35 - jepsen.util 845 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,667] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,668] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:48,668] jepsen worker 35 - jepsen.util 845 :invoke :write [11 7] INFO [2022-03-13 17:11:48,689] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,691] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:11:48,716] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,717] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:48,726] jepsen worker 14 - jepsen.util 284 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,726] jepsen worker 14 - jepsen.util 284 :invoke :write [4 28] INFO [2022-03-13 17:11:48,769] jepsen worker 51 - jepsen.util 591 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,769] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:48,769] jepsen worker 51 - jepsen.util 591 :invoke :write [17 7] INFO [2022-03-13 17:11:48,770] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:48,771] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,772] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,772] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:48,773] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:48,776] jepsen worker 21 - jepsen.util 741 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,776] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:48,802] jepsen worker 76 - jepsen.util 616 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,803] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:48,860] jepsen worker 13 - jepsen.util 643 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:48,860] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:48,866] jepsen worker 81 - jepsen.util 2151 :info :write [27 42] :RequestTimeoutException INFO [2022-03-13 17:11:48,866] jepsen worker 81 - jepsen.util 2241 :invoke :read [27 nil] INFO [2022-03-13 17:11:49,050] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,050] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:49,056] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,056] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,056] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,056] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,056] jepsen worker 75 - jepsen.util 435 :invoke :write [25 36] INFO [2022-03-13 17:11:49,057] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,057] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,057] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:49,058] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:11:49,058] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:49,059] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:49,059] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,060] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:49,061] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,061] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:49,062] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:11:49,064] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,065] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,065] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,066] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:49,067] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:49,067] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:49,067] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,068] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,068] jepsen worker 21 - jepsen.util 741 :invoke :write [7 39] INFO [2022-03-13 17:11:49,068] jepsen worker 69 - jepsen.util 609 :invoke :write [23 10] INFO [2022-03-13 17:11:49,068] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,069] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:11:49,081] jepsen worker 26 - jepsen.util 476 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,082] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:49,096] jepsen worker 38 - jepsen.util 2288 :info :write [12 9] :RequestTimeoutException INFO [2022-03-13 17:11:49,096] jepsen worker 38 - jepsen.util 2378 :invoke :read [12 nil] INFO [2022-03-13 17:11:49,128] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,128] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:49,153] jepsen worker 35 - jepsen.util 845 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,154] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:49,160] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,161] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,161] jepsen worker 79 - jepsen.util 889 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,161] jepsen worker 76 - jepsen.util 616 :invoke :write [25 47] INFO [2022-03-13 17:11:49,162] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:49,162] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:49,163] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,163] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,163] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:49,164] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:49,164] jepsen worker 75 - jepsen.util 435 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,165] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,165] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:49,165] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,165] jepsen worker 79 - jepsen.util 889 :invoke :write [26 47] INFO [2022-03-13 17:11:49,166] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:11:49,237] jepsen worker 61 - jepsen.util 601 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,237] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:49,262] jepsen worker 78 - jepsen.util 528 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,263] jepsen worker 78 - jepsen.util 528 :invoke :write [26 45] INFO [2022-03-13 17:11:49,267] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,267] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:49,283] jepsen worker 76 - jepsen.util 616 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,283] jepsen worker 76 - jepsen.util 616 :invoke :write [25 29] INFO [2022-03-13 17:11:49,305] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,305] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:49,305] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,306] jepsen worker 77 - jepsen.util 707 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,306] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:49,307] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:49,307] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:49,307] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:49,334] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,334] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,335] jepsen worker 35 - jepsen.util 845 :invoke :write [11 30] INFO [2022-03-13 17:11:49,335] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:11:49,415] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,416] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] INFO [2022-03-13 17:11:49,419] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,420] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:11:49,456] jepsen worker 84 - jepsen.util 2064 :ok :read [28 44] INFO [2022-03-13 17:11:49,457] jepsen worker 84 - jepsen.util 2064 :invoke :read [28 nil] INFO [2022-03-13 17:11:49,461] jepsen worker 59 - jepsen.util 1769 :ok :read [19 23] INFO [2022-03-13 17:11:49,463] jepsen worker 59 - jepsen.util 1769 :invoke :write [19 48] INFO [2022-03-13 17:11:49,470] jepsen worker 81 - jepsen.util 2241 :ok :read [27 4] INFO [2022-03-13 17:11:49,470] jepsen worker 81 - jepsen.util 2241 :invoke :read [27 nil] INFO [2022-03-13 17:11:49,484] jepsen worker 84 - jepsen.util 2064 :ok :read [28 44] INFO [2022-03-13 17:11:49,485] jepsen worker 84 - jepsen.util 2064 :invoke :read [28 nil] INFO [2022-03-13 17:11:49,493] jepsen worker 81 - jepsen.util 2241 :ok :read [27 4] INFO [2022-03-13 17:11:49,494] jepsen worker 81 - jepsen.util 2241 :invoke :read [27 nil] INFO [2022-03-13 17:11:49,540] jepsen worker 21 - jepsen.util 741 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,540] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,540] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,540] jepsen worker 70 - jepsen.util 700 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,540] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,540] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,542] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:49,543] jepsen worker 80 - jepsen.util 710 :invoke :write [26 40] INFO [2022-03-13 17:11:49,547] jepsen worker 21 - jepsen.util 741 :invoke :write [7 9] INFO [2022-03-13 17:11:49,547] jepsen worker 84 - jepsen.util 2064 :ok :read [28 44] INFO [2022-03-13 17:11:49,547] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:49,548] jepsen worker 81 - jepsen.util 2241 :ok :read [27 4] INFO [2022-03-13 17:11:49,548] jepsen worker 84 - jepsen.util 2064 :invoke :read [28 nil] INFO [2022-03-13 17:11:49,549] jepsen worker 81 - jepsen.util 2241 :invoke :read [27 nil] INFO [2022-03-13 17:11:49,549] jepsen worker 34 - jepsen.util 754 :invoke :write [11 41] INFO [2022-03-13 17:11:49,549] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:49,565] jepsen worker 43 - jepsen.util 493 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,566] jepsen worker 43 - jepsen.util 493 :invoke :write [14 43] INFO [2022-03-13 17:11:49,574] jepsen worker 79 - jepsen.util 889 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,575] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:49,577] jepsen worker 84 - jepsen.util 2064 :ok :read [28 44] INFO [2022-03-13 17:11:49,577] jepsen worker 84 - jepsen.util 2064 :invoke :read [28 nil] INFO [2022-03-13 17:11:49,587] jepsen worker 81 - jepsen.util 2241 :ok :read [27 4] INFO [2022-03-13 17:11:49,588] jepsen worker 81 - jepsen.util 2241 :invoke :write [27 37] INFO [2022-03-13 17:11:49,608] jepsen worker 84 - jepsen.util 2064 :ok :read [28 44] INFO [2022-03-13 17:11:49,609] jepsen worker 84 - jepsen.util 2064 :invoke :read [28 nil] INFO [2022-03-13 17:11:49,639] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.343108504398827}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.343108504398827}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:49,639] jepsen worker nemesis - couchbase.util Rebalance check count: 220 INFO [2022-03-13 17:11:49,639] jepsen worker nemesis - couchbase.util Rebalance stuck count: 139 INFO [2022-03-13 17:11:49,643] jepsen worker 38 - jepsen.util 2378 :ok :read [12 20] INFO [2022-03-13 17:11:49,643] jepsen worker 38 - jepsen.util 2378 :invoke :write [12 30] INFO [2022-03-13 17:11:49,664] jepsen worker 51 - jepsen.util 591 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,664] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,664] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,664] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:49,664] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,665] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,665] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:49,666] jepsen worker 60 - jepsen.util 600 :invoke :write [20 40] INFO [2022-03-13 17:11:49,666] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:49,667] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:49,667] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:49,668] jepsen worker 23 - jepsen.util 563 :invoke :write [7 46] INFO [2022-03-13 17:11:49,670] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,670] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:49,671] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,671] jepsen worker 12 - jepsen.util 732 :invoke :write [4 37] INFO [2022-03-13 17:11:49,671] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,674] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:49,675] jepsen worker 51 - jepsen.util 591 :invoke :write [17 45] INFO [2022-03-13 17:11:49,682] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:49,684] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:49,685] jepsen worker 44 - jepsen.util 584 :invoke :write [14 42] INFO [2022-03-13 17:11:49,696] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,697] jepsen worker 77 - jepsen.util 707 :invoke :write [25 41] INFO [2022-03-13 17:11:49,708] jepsen worker 69 - jepsen.util 609 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,709] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:11:49,732] jepsen worker 76 - jepsen.util 616 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,733] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:49,753] jepsen worker 78 - jepsen.util 528 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,753] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,754] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,754] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,754] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:49,755] jepsen worker 70 - jepsen.util 700 :invoke :write [23 45] INFO [2022-03-13 17:11:49,756] jepsen worker 52 - jepsen.util 592 :invoke :write [17 48] INFO [2022-03-13 17:11:49,756] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:49,855] jepsen worker 47 - jepsen.util 1937 :info :write [15 23] :SyncWriteAmbiguous INFO [2022-03-13 17:11:49,855] jepsen worker 47 - jepsen.util 2027 :invoke :write [15 27] INFO [2022-03-13 17:11:49,904] jepsen worker 53 - jepsen.util 503 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,904] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:49,911] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,912] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,912] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,913] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:49,914] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:11:49,914] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:11:49,961] jepsen worker 35 - jepsen.util 845 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,961] jepsen worker 35 - jepsen.util 845 :invoke :write [11 0] INFO [2022-03-13 17:11:49,979] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,979] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,979] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,980] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:49,980] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:49,981] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:49,987] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,987] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,987] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:49,987] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:11:49,988] jepsen worker 76 - jepsen.util 616 :invoke :write [25 44] INFO [2022-03-13 17:11:49,989] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:11:50,003] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,003] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:11:50,013] jepsen worker 60 - jepsen.util 600 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,014] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:50,119] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,119] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,120] jepsen worker 42 - jepsen.util 402 :invoke :write [14 48] INFO [2022-03-13 17:11:50,121] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:11:50,123] jepsen worker 66 - jepsen.util 1686 :info :write [22 20] :SyncWriteAmbiguous INFO [2022-03-13 17:11:50,123] jepsen worker 66 - jepsen.util 1776 :invoke :read [22 nil] INFO [2022-03-13 17:11:50,125] jepsen worker 21 - jepsen.util 741 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,125] jepsen worker 21 - jepsen.util 741 :invoke :write [7 41] INFO [2022-03-13 17:11:50,140] jepsen worker 34 - jepsen.util 754 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,140] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,141] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:11:50,141] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:11:50,234] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,235] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,236] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:50,236] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:11:50,257] jepsen worker 23 - jepsen.util 563 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,258] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:50,266] jepsen worker 76 - jepsen.util 616 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,266] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:50,324] jepsen worker 0 - jepsen.util 1800 :info :write [0 41] :SyncWriteAmbiguous INFO [2022-03-13 17:11:50,324] jepsen worker 0 - jepsen.util 1890 :invoke :write [0 16] INFO [2022-03-13 17:11:50,348] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,348] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:11:50,356] jepsen worker 50 - jepsen.util 1940 :ok :read [16 :nil] INFO [2022-03-13 17:11:50,357] jepsen worker 50 - jepsen.util 1940 :invoke :write [16 46] INFO [2022-03-13 17:11:50,358] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,358] jepsen worker 14 - jepsen.util 284 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,358] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,358] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,358] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,359] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:50,360] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:11:50,361] jepsen worker 62 - jepsen.util 512 :invoke :write [20 45] INFO [2022-03-13 17:11:50,361] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:50,362] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:50,362] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,363] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:50,371] jepsen worker 35 - jepsen.util 845 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,371] jepsen worker 80 - jepsen.util 710 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,371] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,371] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,372] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:50,373] jepsen worker 35 - jepsen.util 845 :invoke :write [11 25] INFO [2022-03-13 17:11:50,374] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:50,374] jepsen worker 80 - jepsen.util 710 :invoke :write [26 32] INFO [2022-03-13 17:11:50,390] jepsen worker 17 - jepsen.util 1907 :ok :read [5 43] INFO [2022-03-13 17:11:50,391] jepsen worker 17 - jepsen.util 1907 :invoke :write [5 16] INFO [2022-03-13 17:11:50,463] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,463] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,463] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,464] jepsen worker 79 - jepsen.util 889 :invoke :write [26 17] INFO [2022-03-13 17:11:50,465] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:11:50,470] jepsen worker 76 - jepsen.util 616 :invoke :write [25 31] INFO [2022-03-13 17:11:50,478] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,478] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:11:50,525] jepsen worker 12 - jepsen.util 732 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,526] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:50,562] jepsen worker 70 - jepsen.util 700 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,562] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,563] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:50,563] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,563] jepsen worker 22 - jepsen.util 652 :invoke :write [7 10] INFO [2022-03-13 17:11:50,564] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:50,600] jepsen worker 72 - jepsen.util 1872 :info :write [24 20] :RequestTimeoutException INFO [2022-03-13 17:11:50,601] jepsen worker 72 - jepsen.util 1962 :invoke :read [24 nil] INFO [2022-03-13 17:11:50,635] jepsen worker 79 - jepsen.util 889 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,635] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:50,641] jepsen worker 7 - jepsen.util 637 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,641] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,641] jepsen worker 8 - jepsen.util 728 :invoke :write [2 38] INFO [2022-03-13 17:11:50,642] jepsen worker 7 - jepsen.util 637 :invoke :write [2 34] INFO [2022-03-13 17:11:50,645] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.343108504398827}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.343108504398827}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:50,645] jepsen worker nemesis - couchbase.util Rebalance check count: 221 INFO [2022-03-13 17:11:50,645] jepsen worker nemesis - couchbase.util Rebalance stuck count: 140 INFO [2022-03-13 17:11:50,657] jepsen worker 21 - jepsen.util 741 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,657] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,658] jepsen worker 21 - jepsen.util 741 :invoke :write [7 1] INFO [2022-03-13 17:11:50,658] jepsen worker 69 - jepsen.util 609 :invoke :write [23 16] INFO [2022-03-13 17:11:50,688] jepsen worker 52 - jepsen.util 592 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,689] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:50,761] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,762] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:11:50,819] jepsen worker 35 - jepsen.util 845 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,819] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:50,823] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,824] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:11:50,827] jepsen worker 51 - jepsen.util 591 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,831] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:50,884] jepsen worker 68 - jepsen.util 1958 :info :write [22 43] :RequestTimeoutException INFO [2022-03-13 17:11:50,884] jepsen worker 68 - jepsen.util 2048 :invoke :write [22 28] INFO [2022-03-13 17:11:50,897] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,897] jepsen worker 76 - jepsen.util 616 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,898] jepsen worker 79 - jepsen.util 889 :invoke :write [26 26] INFO [2022-03-13 17:11:50,899] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:50,901] jepsen worker 86 - jepsen.util 1526 :info :write [28 27] :SyncWriteAmbiguous INFO [2022-03-13 17:11:50,902] jepsen worker 86 - jepsen.util 1616 :invoke :write [28 26] INFO [2022-03-13 17:11:50,964] jepsen worker 43 - jepsen.util 493 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,964] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,965] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:50,965] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:11:50,972] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:50,973] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:11:51,000] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,000] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,001] jepsen worker 76 - jepsen.util 616 :invoke :write [25 17] INFO [2022-03-13 17:11:51,002] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:11:51,030] jepsen worker 48 - jepsen.util 1848 :info :write [16 23] :RequestTimeoutException INFO [2022-03-13 17:11:51,034] jepsen worker 48 - jepsen.util 1938 :invoke :read [16 nil] INFO [2022-03-13 17:11:51,066] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,066] jepsen worker 22 - jepsen.util 652 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,066] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,066] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,066] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,067] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:11:51,067] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:51,068] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:51,068] jepsen worker 35 - jepsen.util 845 :invoke :write [11 29] INFO [2022-03-13 17:11:51,069] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:51,071] jepsen worker 48 - jepsen.util 1938 :ok :read [16 :nil] INFO [2022-03-13 17:11:51,071] jepsen worker 48 - jepsen.util 1938 :invoke :read [16 nil] INFO [2022-03-13 17:11:51,102] jepsen worker 48 - jepsen.util 1938 :ok :read [16 :nil] INFO [2022-03-13 17:11:51,103] jepsen worker 48 - jepsen.util 1938 :invoke :read [16 nil] INFO [2022-03-13 17:11:51,104] jepsen worker 48 - jepsen.util 1938 :ok :read [16 :nil] INFO [2022-03-13 17:11:51,104] jepsen worker 48 - jepsen.util 1938 :invoke :read [16 nil] INFO [2022-03-13 17:11:51,105] jepsen worker 48 - jepsen.util 1938 :ok :read [16 :nil] INFO [2022-03-13 17:11:51,106] jepsen worker 48 - jepsen.util 1938 :invoke :write [16 29] INFO [2022-03-13 17:11:51,225] jepsen worker 44 - jepsen.util 584 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,226] jepsen worker 44 - jepsen.util 584 :invoke :write [14 8] INFO [2022-03-13 17:11:51,229] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,229] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,230] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:11:51,230] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:51,231] jepsen worker 79 - jepsen.util 889 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,232] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:51,321] jepsen worker 76 - jepsen.util 616 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,322] jepsen worker 76 - jepsen.util 616 :invoke :write [25 3] INFO [2022-03-13 17:11:51,356] jepsen worker 80 - jepsen.util 710 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,356] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,356] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,356] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,356] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,356] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,357] jepsen worker 15 - jepsen.util 1905 :info :write [5 16] :RequestTimeoutException INFO [2022-03-13 17:11:51,357] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:11:51,363] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:51,364] jepsen worker 15 - jepsen.util 1995 :invoke :read [5 nil] INFO [2022-03-13 17:11:51,365] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:51,365] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,366] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,366] jepsen worker 41 - jepsen.util 1571 :info :write [13 19] :RequestTimeoutException INFO [2022-03-13 17:11:51,366] jepsen worker 80 - jepsen.util 710 :invoke :write [26 0] INFO [2022-03-13 17:11:51,367] jepsen worker 52 - jepsen.util 592 :invoke :write [17 31] INFO [2022-03-13 17:11:51,368] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:51,369] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:51,369] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,370] jepsen worker 78 - jepsen.util 528 :invoke :write [26 24] INFO [2022-03-13 17:11:51,370] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,371] jepsen worker 13 - jepsen.util 643 :invoke :write [4 45] INFO [2022-03-13 17:11:51,372] jepsen worker 41 - jepsen.util 1661 :invoke :write [13 36] INFO [2022-03-13 17:11:51,373] jepsen worker 57 - jepsen.util 1947 :info :write [19 28] :RequestTimeoutException INFO [2022-03-13 17:11:51,375] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:51,376] jepsen worker 42 - jepsen.util 402 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,376] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,376] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,376] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,376] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,380] jepsen worker 14 - jepsen.util 284 :invoke :write [4 8] INFO [2022-03-13 17:11:51,382] jepsen worker 26 - jepsen.util 476 :invoke :write [8 39] INFO [2022-03-13 17:11:51,383] jepsen worker 61 - jepsen.util 601 :invoke :write [20 11] INFO [2022-03-13 17:11:51,384] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:11:51,385] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:51,386] jepsen worker 67 - jepsen.util 1507 :info :write [22 14] :RequestTimeoutException INFO [2022-03-13 17:11:51,387] jepsen worker 67 - jepsen.util 1597 :invoke :write [22 20] INFO [2022-03-13 17:11:51,387] jepsen worker 57 - jepsen.util 2037 :invoke :read [19 nil] INFO [2022-03-13 17:11:51,403] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,403] jepsen worker 79 - jepsen.util 889 :invoke :write [26 48] INFO [2022-03-13 17:11:51,404] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,404] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:11:51,408] jepsen worker 69 - jepsen.util 609 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,408] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:51,480] jepsen worker 2 - jepsen.util 2162 :info :write [0 5] :RequestTimeoutException INFO [2022-03-13 17:11:51,482] jepsen worker 2 - jepsen.util 2252 :invoke :read [0 nil] INFO [2022-03-13 17:11:51,498] jepsen worker 76 - jepsen.util 616 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,498] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:51,510] jepsen worker 35 - jepsen.util 845 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,511] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:51,530] jepsen worker 21 - jepsen.util 741 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,531] jepsen worker 21 - jepsen.util 741 :invoke :write [7 3] INFO [2022-03-13 17:11:51,531] jepsen worker 77 - jepsen.util 707 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,531] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,531] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,532] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:51,533] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:51,533] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:51,534] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,534] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,534] jepsen worker 12 - jepsen.util 732 :invoke :write [4 33] INFO [2022-03-13 17:11:51,535] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:51,597] jepsen worker 79 - jepsen.util 889 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,598] jepsen worker 79 - jepsen.util 889 :invoke :write [26 23] INFO [2022-03-13 17:11:51,606] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,606] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,607] jepsen worker 70 - jepsen.util 700 :invoke :write [23 37] INFO [2022-03-13 17:11:51,608] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:51,619] jepsen worker 62 - jepsen.util 512 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,619] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:51,651] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:51,651] jepsen worker nemesis - couchbase.util Rebalance check count: 222 INFO [2022-03-13 17:11:51,651] jepsen worker nemesis - couchbase.util Rebalance stuck count: 140 INFO [2022-03-13 17:11:51,663] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,663] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:51,674] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,674] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,675] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:11:51,676] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:11:51,711] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,712] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:11:51,714] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,715] jepsen worker 35 - jepsen.util 845 :invoke :write [11 23] INFO [2022-03-13 17:11:51,755] jepsen worker 44 - jepsen.util 584 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,755] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,755] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,755] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,756] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,756] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:51,757] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:51,758] jepsen worker 51 - jepsen.util 591 :invoke :write [17 31] INFO [2022-03-13 17:11:51,758] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:11:51,759] jepsen worker 77 - jepsen.util 707 :invoke :write [25 6] INFO [2022-03-13 17:11:51,768] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,768] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,769] jepsen worker 71 - jepsen.util 521 :invoke :write [23 37] INFO [2022-03-13 17:11:51,769] jepsen worker 75 - jepsen.util 435 :invoke :write [25 37] INFO [2022-03-13 17:11:51,793] jepsen worker 79 - jepsen.util 889 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,794] jepsen worker 79 - jepsen.util 889 :invoke :write [26 34] INFO [2022-03-13 17:11:51,808] jepsen worker 78 - jepsen.util 528 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,808] jepsen worker 78 - jepsen.util 528 :invoke :write [26 39] INFO [2022-03-13 17:11:51,836] jepsen worker 8 - jepsen.util 728 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,837] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:11:51,872] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,873] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:11:51,937] jepsen worker 85 - jepsen.util 1975 :info :write [28 45] :RequestTimeoutException INFO [2022-03-13 17:11:51,937] jepsen worker 85 - jepsen.util 2065 :invoke :read [28 nil] INFO [2022-03-13 17:11:51,971] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,971] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:51,971] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:11:51,971] jepsen worker 23 - jepsen.util 563 :invoke :write [7 1] INFO [2022-03-13 17:11:51,972] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:51,973] jepsen worker 44 - jepsen.util 584 :invoke :write [14 12] INFO [2022-03-13 17:11:52,110] jepsen worker 46 - jepsen.util 1936 :info :write [15 30] :RequestTimeoutException INFO [2022-03-13 17:11:52,110] jepsen worker 46 - jepsen.util 2026 :invoke :read [15 nil] INFO [2022-03-13 17:11:52,208] jepsen worker 28 - jepsen.util 2008 :info :write [9 33] :RequestTimeoutException INFO [2022-03-13 17:11:52,208] jepsen worker 28 - jepsen.util 2098 :invoke :read [9 nil] INFO [2022-03-13 17:11:52,213] jepsen worker 55 - jepsen.util 1765 :info :write [18 49] :RequestTimeoutException INFO [2022-03-13 17:11:52,214] jepsen worker 55 - jepsen.util 1855 :invoke :read [18 nil] INFO [2022-03-13 17:11:52,220] jepsen worker 20 - jepsen.util 2270 :info :write [6 23] :RequestTimeoutException INFO [2022-03-13 17:11:52,220] jepsen worker 20 - jepsen.util 2360 :invoke :write [6 8] INFO [2022-03-13 17:11:52,253] jepsen worker 9 - jepsen.util 1719 :info :write [3 14] :RequestTimeoutException INFO [2022-03-13 17:11:52,254] jepsen worker 9 - jepsen.util 1809 :invoke :read [3 nil] INFO [2022-03-13 17:11:52,526] jepsen worker 49 - jepsen.util 1579 :info :write [16 29] :SyncWriteAmbiguous INFO [2022-03-13 17:11:52,528] jepsen worker 49 - jepsen.util 1669 :invoke :write [16 44] INFO [2022-03-13 17:11:52,537] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:11:52,537] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,537] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,537] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:52,537] jepsen worker 43 - jepsen.util 493 :invoke :write [14 23] INFO [2022-03-13 17:11:52,540] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:52,540] jepsen worker 6 - jepsen.util 636 :invoke :write [2 33] INFO [2022-03-13 17:11:52,540] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:52,571] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,571] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:52,608] jepsen worker 79 - jepsen.util 889 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,609] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:52,626] jepsen worker 51 - jepsen.util 591 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,627] jepsen worker 51 - jepsen.util 591 :invoke :write [17 11] INFO [2022-03-13 17:11:52,657] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:52,657] jepsen worker nemesis - couchbase.util Rebalance check count: 223 INFO [2022-03-13 17:11:52,658] jepsen worker nemesis - couchbase.util Rebalance stuck count: 141 INFO [2022-03-13 17:11:52,696] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,696] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:52,707] jepsen worker 80 - jepsen.util 710 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,708] jepsen worker 14 - jepsen.util 284 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,708] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:52,708] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:52,712] jepsen worker 36 - jepsen.util 2106 :ok :read [12 20] INFO [2022-03-13 17:11:52,713] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 23] INFO [2022-03-13 17:11:52,713] jepsen worker 2 - jepsen.util 2252 :ok :read [0 41] INFO [2022-03-13 17:11:52,713] jepsen worker 2 - jepsen.util 2252 :invoke :write [0 42] INFO [2022-03-13 17:11:52,721] jepsen worker 83 - jepsen.util 1343 :info :write [27 49] :SyncWriteAmbiguous INFO [2022-03-13 17:11:52,722] jepsen worker 83 - jepsen.util 1433 :invoke :write [27 20] INFO [2022-03-13 17:11:52,733] jepsen worker 75 - jepsen.util 435 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,734] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:52,797] jepsen worker 35 - jepsen.util 845 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,798] jepsen worker 35 - jepsen.util 845 :invoke :write [11 24] INFO [2022-03-13 17:11:52,807] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,807] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,807] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,808] jepsen worker 76 - jepsen.util 616 :invoke :write [25 13] INFO [2022-03-13 17:11:52,809] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:11:52,809] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:52,812] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,812] jepsen worker 79 - jepsen.util 889 :invoke :write [26 8] INFO [2022-03-13 17:11:52,832] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,832] jepsen worker 77 - jepsen.util 707 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,833] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:11:52,834] jepsen worker 60 - jepsen.util 600 :invoke :write [20 47] INFO [2022-03-13 17:11:52,914] jepsen worker 71 - jepsen.util 521 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,915] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:52,944] jepsen worker 61 - jepsen.util 601 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,944] jepsen worker 61 - jepsen.util 601 :invoke :write [20 37] INFO [2022-03-13 17:11:52,963] jepsen worker 76 - jepsen.util 616 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,963] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:52,974] jepsen worker 13 - jepsen.util 643 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,974] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,974] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:52,975] jepsen worker 13 - jepsen.util 643 :invoke :write [4 37] INFO [2022-03-13 17:11:52,975] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:11:52,976] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:53,046] jepsen worker 1 - jepsen.util 1621 :ok :read [0 41] INFO [2022-03-13 17:11:53,047] jepsen worker 1 - jepsen.util 1621 :invoke :read [0 nil] INFO [2022-03-13 17:11:53,060] jepsen worker 31 - jepsen.util 1831 :ok :read [10 :nil] INFO [2022-03-13 17:11:53,061] jepsen worker 31 - jepsen.util 1831 :invoke :write [10 35] INFO [2022-03-13 17:11:53,105] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,105] jepsen worker 1 - jepsen.util 1621 :ok :read [0 41] INFO [2022-03-13 17:11:53,106] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:53,107] jepsen worker 1 - jepsen.util 1621 :invoke :write [0 2] INFO [2022-03-13 17:11:53,126] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,127] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:53,151] jepsen worker 12 - jepsen.util 732 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,151] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:53,174] jepsen worker 9 - jepsen.util 1809 :ok :read [3 25] INFO [2022-03-13 17:11:53,176] jepsen worker 9 - jepsen.util 1809 :invoke :read [3 nil] INFO [2022-03-13 17:11:53,217] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,217] jepsen worker 79 - jepsen.util 889 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,217] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,218] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:53,219] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:53,219] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,220] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,220] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:11:53,220] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,221] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,221] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:11:53,221] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:53,223] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,223] jepsen worker 56 - jepsen.util 1856 :info :write [18 21] :RequestTimeoutException INFO [2022-03-13 17:11:53,224] jepsen worker 76 - jepsen.util 616 :invoke :write [25 23] INFO [2022-03-13 17:11:53,225] jepsen worker 56 - jepsen.util 1946 :invoke :read [18 nil] INFO [2022-03-13 17:11:53,246] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,246] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,246] jepsen worker 26 - jepsen.util 476 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,246] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:53,247] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,247] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:53,248] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:53,249] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:11:53,250] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:53,251] jepsen worker 26 - jepsen.util 476 :invoke :write [8 47] INFO [2022-03-13 17:11:53,254] jepsen worker 56 - jepsen.util 1946 :ok :read [18 21] INFO [2022-03-13 17:11:53,255] jepsen worker 56 - jepsen.util 1946 :invoke :write [18 42] INFO [2022-03-13 17:11:53,286] jepsen worker 9 - jepsen.util 1809 :ok :read [3 25] INFO [2022-03-13 17:11:53,286] jepsen worker 9 - jepsen.util 1809 :invoke :read [3 nil] INFO [2022-03-13 17:11:53,293] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,293] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:53,307] jepsen worker 35 - jepsen.util 845 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,308] jepsen worker 70 - jepsen.util 700 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,308] jepsen worker 35 - jepsen.util 845 :invoke :write [11 40] INFO [2022-03-13 17:11:53,308] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:11:53,375] jepsen worker 28 - jepsen.util 2098 :ok :read [9 26] INFO [2022-03-13 17:11:53,375] jepsen worker 44 - jepsen.util 584 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,375] jepsen worker 28 - jepsen.util 2098 :invoke :write [9 27] INFO [2022-03-13 17:11:53,376] jepsen worker 44 - jepsen.util 584 :invoke :write [14 17] INFO [2022-03-13 17:11:53,387] jepsen worker 54 - jepsen.util 1674 :ok :read [18 21] INFO [2022-03-13 17:11:53,387] jepsen worker 54 - jepsen.util 1674 :invoke :write [18 5] INFO [2022-03-13 17:11:53,390] jepsen worker 9 - jepsen.util 1809 :ok :read [3 25] INFO [2022-03-13 17:11:53,390] jepsen worker 9 - jepsen.util 1809 :invoke :read [3 nil] INFO [2022-03-13 17:11:53,399] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,400] jepsen worker 79 - jepsen.util 889 :invoke :write [26 37] INFO [2022-03-13 17:11:53,477] jepsen worker 7 - jepsen.util 637 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,477] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:11:53,481] jepsen worker 21 - jepsen.util 741 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,481] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,481] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,481] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,481] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,481] jepsen worker 21 - jepsen.util 741 :invoke :write [7 24] INFO [2022-03-13 17:11:53,482] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,483] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:11:53,485] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:11:53,485] jepsen worker 69 - jepsen.util 609 :invoke :write [23 9] INFO [2022-03-13 17:11:53,485] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:53,486] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:11:53,489] jepsen worker 76 - jepsen.util 616 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,490] jepsen worker 76 - jepsen.util 616 :invoke :write [25 18] INFO [2022-03-13 17:11:53,506] jepsen worker 78 - jepsen.util 528 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,507] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,507] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,507] jepsen worker 34 - jepsen.util 754 :invoke :write [11 34] INFO [2022-03-13 17:11:53,508] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,508] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,512] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,513] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:11:53,547] jepsen worker 3 - jepsen.util 1893 :ok :read [1 42] INFO [2022-03-13 17:11:53,548] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:11:53,577] jepsen worker 9 - jepsen.util 1809 :ok :read [3 25] INFO [2022-03-13 17:11:53,578] jepsen worker 45 - jepsen.util 2205 :info :write [15 33] :RequestTimeoutException INFO [2022-03-13 17:11:53,578] jepsen worker 9 - jepsen.util 1809 :invoke :write [3 31] INFO [2022-03-13 17:11:53,579] jepsen worker 45 - jepsen.util 2295 :invoke :write [15 9] INFO [2022-03-13 17:11:53,588] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,589] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:53,598] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,598] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:53,644] jepsen worker 23 - jepsen.util 563 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,644] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:11:53,663] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:53,664] jepsen worker nemesis - couchbase.util Rebalance check count: 224 INFO [2022-03-13 17:11:53,664] jepsen worker nemesis - couchbase.util Rebalance stuck count: 142 INFO [2022-03-13 17:11:53,692] jepsen worker 79 - jepsen.util 889 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,693] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,694] jepsen worker 21 - jepsen.util 741 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,695] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:53,736] jepsen worker 43 - jepsen.util 493 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,737] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:11:53,758] jepsen worker 13 - jepsen.util 643 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,759] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:53,833] jepsen worker 76 - jepsen.util 616 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,833] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,833] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,833] jepsen worker 76 - jepsen.util 616 :invoke :write [25 8] INFO [2022-03-13 17:11:53,834] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,835] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:53,835] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,836] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,837] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,838] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,838] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:11:53,839] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,840] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,840] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:11:53,844] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,845] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,846] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,847] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,849] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,850] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:53,852] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,853] jepsen worker 79 - jepsen.util 889 :invoke :write [26 13] INFO [2022-03-13 17:11:53,879] jepsen worker 35 - jepsen.util 845 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,880] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:53,883] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,883] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:11:53,883] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,884] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:53,890] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,890] jepsen worker 51 - jepsen.util 591 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,891] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:53,892] jepsen worker 51 - jepsen.util 591 :invoke :write [17 37] INFO [2022-03-13 17:11:53,901] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,902] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:53,903] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,904] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:53,954] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,955] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:53,970] jepsen worker 11 - jepsen.util 1991 :info :write [3 44] :RequestTimeoutException INFO [2022-03-13 17:11:53,971] jepsen worker 11 - jepsen.util 2081 :invoke :read [3 nil] INFO [2022-03-13 17:11:53,973] jepsen worker 65 - jepsen.util 2225 :info :write [21 18] :RequestTimeoutException INFO [2022-03-13 17:11:53,974] jepsen worker 65 - jepsen.util 2315 :invoke :write [21 23] INFO [2022-03-13 17:11:53,981] jepsen worker 6 - jepsen.util 636 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,981] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,982] jepsen worker 14 - jepsen.util 284 :invoke :write [4 20] INFO [2022-03-13 17:11:53,983] jepsen worker 6 - jepsen.util 636 :invoke :write [2 38] INFO [2022-03-13 17:11:53,996] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:53,997] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:54,028] jepsen worker 52 - jepsen.util 592 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,029] jepsen worker 52 - jepsen.util 592 :invoke :write [17 27] INFO [2022-03-13 17:11:54,081] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,082] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:11:54,090] jepsen worker 76 - jepsen.util 616 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,091] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:54,121] jepsen worker 60 - jepsen.util 600 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,122] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:11:54,130] jepsen worker 4 - jepsen.util 2074 :info :write [1 17] :RequestTimeoutException INFO [2022-03-13 17:11:54,131] jepsen worker 4 - jepsen.util 2164 :invoke :write [1 14] INFO [2022-03-13 17:11:54,166] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,166] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,167] jepsen worker 22 - jepsen.util 652 :invoke :write [7 32] INFO [2022-03-13 17:11:54,168] jepsen worker 35 - jepsen.util 845 :invoke :write [11 24] INFO [2022-03-13 17:11:54,188] jepsen worker 69 - jepsen.util 609 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,189] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:54,196] jepsen worker 29 - jepsen.util 2009 :info :write [9 44] :RequestTimeoutException INFO [2022-03-13 17:11:54,196] jepsen worker 29 - jepsen.util 2099 :invoke :write [9 10] INFO [2022-03-13 17:11:54,215] jepsen worker 79 - jepsen.util 889 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,215] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,216] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,217] jepsen worker 79 - jepsen.util 889 :invoke :write [26 13] INFO [2022-03-13 17:11:54,217] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:54,218] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:54,219] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,220] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,220] jepsen worker 25 - jepsen.util 565 :invoke :write [8 10] INFO [2022-03-13 17:11:54,220] jepsen worker 76 - jepsen.util 616 :invoke :write [25 1] INFO [2022-03-13 17:11:54,229] jepsen worker 61 - jepsen.util 601 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,230] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,230] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,230] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:11:54,230] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,231] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:11:54,232] jepsen worker 42 - jepsen.util 402 :invoke :write [14 35] INFO [2022-03-13 17:11:54,233] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:54,234] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:11:54,234] jepsen worker 8 - jepsen.util 728 :invoke :write [2 19] INFO [2022-03-13 17:11:54,321] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,321] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,321] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,322] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:54,323] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:54,323] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,323] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,324] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:54,324] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:11:54,325] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:54,386] jepsen worker 10 - jepsen.util 1900 :info :write [3 9] :RequestTimeoutException INFO [2022-03-13 17:11:54,387] jepsen worker 10 - jepsen.util 1990 :invoke :write [3 36] INFO [2022-03-13 17:11:54,393] jepsen worker 79 - jepsen.util 889 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,394] jepsen worker 79 - jepsen.util 889 :invoke :write [26 41] INFO [2022-03-13 17:11:54,399] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,399] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,400] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:54,400] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:54,416] jepsen worker 19 - jepsen.util 1819 :info :write [6 32] :RequestTimeoutException INFO [2022-03-13 17:11:54,417] jepsen worker 19 - jepsen.util 1909 :invoke :read [6 nil] INFO [2022-03-13 17:11:54,445] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,445] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,446] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:11:54,446] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:54,465] jepsen worker 34 - jepsen.util 754 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,465] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,466] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:54,467] jepsen worker 13 - jepsen.util 643 :invoke :write [4 26] INFO [2022-03-13 17:11:54,514] jepsen worker 76 - jepsen.util 616 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,515] jepsen worker 76 - jepsen.util 616 :invoke :write [25 16] INFO [2022-03-13 17:11:54,518] jepsen worker 26 - jepsen.util 476 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,519] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:11:54,542] jepsen worker 22 - jepsen.util 652 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,543] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:54,616] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,616] jepsen worker 44 - jepsen.util 584 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,617] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,617] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,617] jepsen worker 44 - jepsen.util 584 :invoke :write [14 35] INFO [2022-03-13 17:11:54,619] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:54,619] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:11:54,619] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:54,643] jepsen worker 25 - jepsen.util 565 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,645] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:11:54,655] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,655] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,655] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:11:54,656] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:11:54,671] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5087719298245614}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:54,671] jepsen worker nemesis - couchbase.util Rebalance check count: 225 INFO [2022-03-13 17:11:54,671] jepsen worker nemesis - couchbase.util Rebalance stuck count: 143 INFO [2022-03-13 17:11:54,700] jepsen worker 37 - jepsen.util 1927 :info :write [12 4] :RequestTimeoutException INFO [2022-03-13 17:11:54,701] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:11:54,731] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,731] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,731] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,731] jepsen worker 52 - jepsen.util 592 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,732] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,732] jepsen worker 80 - jepsen.util 710 :invoke :write [26 42] INFO [2022-03-13 17:11:54,733] jepsen worker 34 - jepsen.util 754 :invoke :write [11 34] INFO [2022-03-13 17:11:54,733] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:54,734] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:54,735] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:54,745] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,746] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,746] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:54,746] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:54,747] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:11:54,747] jepsen worker 69 - jepsen.util 609 :invoke :write [23 44] INFO [2022-03-13 17:11:54,900] jepsen worker 88 - jepsen.util 1888 :info :write [29 44] :RequestTimeoutException INFO [2022-03-13 17:11:54,903] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:11:55,126] jepsen worker 27 - jepsen.util 2097 :info :write [9 27] :RequestTimeoutException INFO [2022-03-13 17:11:55,127] jepsen worker 27 - jepsen.util 2187 :invoke :write [9 24] INFO [2022-03-13 17:11:55,246] jepsen worker 63 - jepsen.util 1863 :info :write [21 19] :RequestTimeoutException INFO [2022-03-13 17:11:55,247] jepsen worker 63 - jepsen.util 1953 :invoke :read [21 nil] INFO [2022-03-13 17:11:55,250] jepsen worker 30 - jepsen.util 1830 :info :write [10 16] :RequestTimeoutException INFO [2022-03-13 17:11:55,251] jepsen worker 30 - jepsen.util 1920 :invoke :write [10 28] INFO [2022-03-13 17:11:55,258] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,258] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:11:55,259] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,259] jepsen worker 7 - jepsen.util 637 :invoke :write [2 9] INFO [2022-03-13 17:11:55,259] jepsen worker 43 - jepsen.util 493 :invoke :write [14 20] INFO [2022-03-13 17:11:55,264] jepsen worker 35 - jepsen.util 845 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,264] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:55,266] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,269] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,270] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:11:55,271] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:11:55,272] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:55,274] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,274] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:55,276] jepsen worker 79 - jepsen.util 889 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,276] jepsen worker 79 - jepsen.util 889 :invoke :write [26 10] INFO [2022-03-13 17:11:55,277] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,277] jepsen worker 35 - jepsen.util 845 :invoke :write [11 43] INFO [2022-03-13 17:11:55,375] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,376] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:55,378] jepsen worker 13 - jepsen.util 643 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,378] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,378] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,378] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,379] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:55,380] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:55,380] jepsen worker 13 - jepsen.util 643 :invoke :write [4 41] INFO [2022-03-13 17:11:55,382] jepsen worker 78 - jepsen.util 528 :invoke :write [26 9] INFO [2022-03-13 17:11:55,384] jepsen worker 74 - jepsen.util 2324 :info :write [24 4] :RequestTimeoutException INFO [2022-03-13 17:11:55,385] jepsen worker 74 - jepsen.util 2414 :invoke :write [24 15] INFO [2022-03-13 17:11:55,387] jepsen worker 76 - jepsen.util 616 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,388] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:55,390] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,391] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:55,472] jepsen worker 6 - jepsen.util 636 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,472] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:11:55,480] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,480] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,481] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:11:55,481] jepsen worker 21 - jepsen.util 741 :invoke :write [7 22] INFO [2022-03-13 17:11:55,510] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,511] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:55,532] jepsen worker 80 - jepsen.util 710 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,533] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:55,548] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,549] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:11:55,576] jepsen worker 51 - jepsen.util 591 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,577] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:55,584] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,584] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:55,590] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,590] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,591] jepsen worker 79 - jepsen.util 889 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,591] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:11:55,592] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:55,592] jepsen worker 79 - jepsen.util 889 :invoke :write [26 4] INFO [2022-03-13 17:11:55,608] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,609] jepsen worker 76 - jepsen.util 616 :invoke :write [25 41] INFO [2022-03-13 17:11:55,653] jepsen worker 18 - jepsen.util 1728 :info :write [6 14] :RequestTimeoutException INFO [2022-03-13 17:11:55,654] jepsen worker 18 - jepsen.util 1818 :invoke :write [6 35] INFO [2022-03-13 17:11:55,666] jepsen worker 69 - jepsen.util 609 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,666] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:55,667] jepsen worker 14 - jepsen.util 284 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,667] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,667] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:55,667] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,667] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,668] jepsen worker 53 - jepsen.util 503 :invoke :write [17 31] INFO [2022-03-13 17:11:55,669] jepsen worker 61 - jepsen.util 601 :invoke :write [20 30] INFO [2022-03-13 17:11:55,672] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:55,678] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5146198830409356}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:55,679] jepsen worker nemesis - couchbase.util Rebalance check count: 226 INFO [2022-03-13 17:11:55,679] jepsen worker nemesis - couchbase.util Rebalance stuck count: 143 INFO [2022-03-13 17:11:55,706] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,707] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:55,745] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,745] jepsen worker 34 - jepsen.util 754 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,745] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,745] jepsen worker 52 - jepsen.util 592 :invoke :write [17 38] INFO [2022-03-13 17:11:55,746] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:55,746] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:55,791] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,791] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,792] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:11:55,792] jepsen worker 71 - jepsen.util 521 :invoke :write [23 12] INFO [2022-03-13 17:11:55,796] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,796] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,797] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:55,797] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,798] jepsen worker 77 - jepsen.util 707 :invoke :write [25 46] INFO [2022-03-13 17:11:55,798] jepsen worker 12 - jepsen.util 732 :invoke :write [4 16] INFO [2022-03-13 17:11:55,806] jepsen worker 79 - jepsen.util 889 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,807] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:55,843] jepsen worker 35 - jepsen.util 845 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,844] jepsen worker 35 - jepsen.util 845 :invoke :write [11 9] INFO [2022-03-13 17:11:55,846] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,847] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] WARN [2022-03-13 17:11:55,863] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9127888,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10251590,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9557859,"encode_duration_us":24917,"operation_id":"0x5429","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10006508}],"total_count":1}} WARN [2022-03-13 17:11:55,863] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":63379,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9028122,"total_server_duration_us":322828,"encode_duration_us":18770,"operation_id":"0x5641","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10007600},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":21939,"operation_id":"0x57ae","timeout_ms":10000,"total_duration_us":10007335},{"operation_name":"upsert","last_dispatch_duration_us":68505,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1233125,"total_server_duration_us":670934,"encode_duration_us":24385,"operation_id":"0x578b","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10006462},{"operation_name":"upsert","last_dispatch_duration_us":42817,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1254048,"total_server_duration_us":737577,"encode_duration_us":20280,"operation_id":"0x5757","timeout_ms":10000,"last_server_duration_us":23306,"total_duration_us":10002542},{"operation_name":"upsert","last_dispatch_duration_us":41262,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":464387,"total_server_duration_us":206128,"encode_duration_us":22804,"operation_id":"0x56b2","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10001531},{"operation_name":"upsert","last_dispatch_duration_us":9075182,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9075182,"total_server_duration_us":9046921,"encode_duration_us":19925,"operation_id":"0x564e","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9079909},{"operation_name":"upsert","last_dispatch_duration_us":9040874,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9040874,"total_server_duration_us":9023573,"encode_duration_us":21316,"operation_id":"0x5643","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9044709},{"operation_name":"get","last_dispatch_duration_us":6905755,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6905755,"total_server_duration_us":38,"operation_id":"0x5706","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":6906163},{"operation_name":"get","last_dispatch_duration_us":3088399,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":3088399,"total_server_duration_us":32,"operation_id":"0x5869","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3091267},{"operation_name":"upsert","last_dispatch_duration_us":250347,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1093761,"total_server_duration_us":700753,"encode_duration_us":18349,"operation_id":"0x5a46","timeout_ms":10000,"last_server_duration_us":72453,"total_duration_us":1112868}],"total_count":20}} INFO [2022-03-13 17:11:55,906] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,906] jepsen worker 76 - jepsen.util 616 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,907] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:11:55,908] jepsen worker 76 - jepsen.util 616 :invoke :write [25 37] INFO [2022-03-13 17:11:55,910] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,910] jepsen worker 79 - jepsen.util 889 :invoke :write [26 15] INFO [2022-03-13 17:11:55,920] jepsen worker 42 - jepsen.util 402 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,921] jepsen worker 42 - jepsen.util 402 :invoke :write [14 48] INFO [2022-03-13 17:11:55,931] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,931] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:11:55,982] jepsen worker 13 - jepsen.util 643 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:11:55,983] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:56,038] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,039] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:11:56,083] jepsen worker 21 - jepsen.util 741 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,084] jepsen worker 21 - jepsen.util 741 :invoke :write [7 0] INFO [2022-03-13 17:11:56,166] jepsen worker 64 - jepsen.util 1864 :info :write [21 31] :RequestTimeoutException INFO [2022-03-13 17:11:56,166] jepsen worker 64 - jepsen.util 1954 :invoke :read [21 nil] INFO [2022-03-13 17:11:56,168] jepsen worker 64 - jepsen.util 1954 :ok :read [21 :nil] INFO [2022-03-13 17:11:56,169] jepsen worker 64 - jepsen.util 1954 :invoke :write [21 28] INFO [2022-03-13 17:11:56,450] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,450] jepsen worker 44 - jepsen.util 584 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,450] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,450] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,450] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,451] jepsen worker 44 - jepsen.util 584 :invoke :write [14 13] INFO [2022-03-13 17:11:56,452] jepsen worker 60 - jepsen.util 600 :invoke :write [20 47] INFO [2022-03-13 17:11:56,453] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:56,454] jepsen worker 51 - jepsen.util 591 :invoke :write [17 12] INFO [2022-03-13 17:11:56,454] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:11:56,507] jepsen worker 76 - jepsen.util 616 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,507] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:56,535] jepsen worker 78 - jepsen.util 528 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,535] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,535] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,535] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,536] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:56,536] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:56,537] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:56,537] jepsen worker 70 - jepsen.util 700 :invoke :write [23 30] INFO [2022-03-13 17:11:56,547] jepsen worker 8 - jepsen.util 728 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,547] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,547] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:11:56,548] jepsen worker 26 - jepsen.util 476 :invoke :write [8 15] INFO [2022-03-13 17:11:56,556] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,557] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:56,560] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,562] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:56,600] jepsen worker 79 - jepsen.util 889 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,601] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:56,674] jepsen worker 71 - jepsen.util 521 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,674] jepsen worker 35 - jepsen.util 845 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,675] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:56,675] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:56,684] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.52046783625731}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:56,684] jepsen worker nemesis - couchbase.util Rebalance check count: 227 INFO [2022-03-13 17:11:56,684] jepsen worker nemesis - couchbase.util Rebalance stuck count: 143 INFO [2022-03-13 17:11:56,708] jepsen worker 77 - jepsen.util 707 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,708] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:11:56,720] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,720] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,720] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,721] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:11:56,722] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:56,722] jepsen worker 76 - jepsen.util 616 :invoke :write [25 38] INFO [2022-03-13 17:11:56,723] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,723] jepsen worker 79 - jepsen.util 889 :invoke :write [26 17] INFO [2022-03-13 17:11:56,775] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,775] jepsen worker 52 - jepsen.util 592 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,775] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,776] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,776] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,776] jepsen worker 13 - jepsen.util 643 :invoke :write [4 9] INFO [2022-03-13 17:11:56,777] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:56,777] jepsen worker 34 - jepsen.util 754 :invoke :write [11 7] INFO [2022-03-13 17:11:56,778] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:56,779] jepsen worker 78 - jepsen.util 528 :invoke :write [26 11] INFO [2022-03-13 17:11:56,788] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,788] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:11:56,790] jepsen worker 7 - jepsen.util 637 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,791] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:11:56,811] jepsen worker 79 - jepsen.util 889 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,813] jepsen worker 79 - jepsen.util 889 :invoke :write [26 25] INFO [2022-03-13 17:11:56,870] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,870] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,871] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:11:56,871] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:56,876] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,876] jepsen worker 35 - jepsen.util 845 :invoke :write [11 46] INFO [2022-03-13 17:11:56,897] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,898] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:11:56,906] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,907] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:11:56,926] jepsen worker 40 - jepsen.util 1930 :info :write [13 15] :RequestTimeoutException INFO [2022-03-13 17:11:56,927] jepsen worker 40 - jepsen.util 2020 :invoke :read [13 nil] INFO [2022-03-13 17:11:56,945] jepsen worker 12 - jepsen.util 732 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,945] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,946] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:56,946] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:11:56,981] jepsen worker 70 - jepsen.util 700 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,982] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,982] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:56,983] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:56,983] jepsen worker 80 - jepsen.util 710 :invoke :write [26 44] INFO [2022-03-13 17:11:56,984] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:57,002] jepsen worker 76 - jepsen.util 616 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,003] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:57,023] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,023] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,023] jepsen worker 21 - jepsen.util 741 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,023] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,023] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:57,024] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:57,024] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:11:57,025] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:57,043] jepsen worker 43 - jepsen.util 493 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,043] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,043] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,043] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,044] jepsen worker 43 - jepsen.util 493 :invoke :write [14 10] INFO [2022-03-13 17:11:57,044] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:57,045] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:57,047] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:11:57,086] jepsen worker 79 - jepsen.util 889 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,086] jepsen worker 79 - jepsen.util 889 :invoke :write [26 15] INFO [2022-03-13 17:11:57,122] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,122] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:57,130] jepsen worker 13 - jepsen.util 643 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,130] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:57,175] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,175] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,176] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:57,176] jepsen worker 44 - jepsen.util 584 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,176] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:11:57,177] jepsen worker 44 - jepsen.util 584 :invoke :write [14 25] INFO [2022-03-13 17:11:57,211] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,211] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:57,215] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,215] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,215] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,215] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:57,215] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,216] jepsen worker 21 - jepsen.util 741 :invoke :write [7 30] INFO [2022-03-13 17:11:57,217] jepsen worker 75 - jepsen.util 435 :invoke :write [25 36] INFO [2022-03-13 17:11:57,217] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:11:57,328] jepsen worker 61 - jepsen.util 601 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,328] jepsen worker 61 - jepsen.util 601 :invoke :write [20 42] INFO [2022-03-13 17:11:57,338] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,338] jepsen worker 79 - jepsen.util 889 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,339] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:57,339] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] WARN [2022-03-13 17:11:57,364] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18102676,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27414948,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9136210,"encode_duration_us":17996,"operation_id":"0x5079","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10006031},{"operation_name":"upsert","last_dispatch_duration_us":17244830,"last_local_id":"A312307A00000002/00000000E1F76995","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":17244830,"last_local_socket":"172.23.104.79:33490","total_server_duration_us":9034183,"encode_duration_us":30237,"operation_id":"0x552f","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10006776},{"operation_name":"upsert","last_dispatch_duration_us":9057190,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9057190,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9024634,"encode_duration_us":17791,"operation_id":"0x53f5","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10007542},{"operation_name":"upsert","last_dispatch_duration_us":10076019,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10076019,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9028877,"encode_duration_us":17115,"operation_id":"0x54b7","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10007090}],"total_count":4}} WARN [2022-03-13 17:11:57,371] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9054205,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9775423,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9239760,"encode_duration_us":17213,"operation_id":"0x56f5","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10005130}],"total_count":1}} INFO [2022-03-13 17:11:57,373] jepsen worker 60 - jepsen.util 600 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,374] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:57,436] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,436] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,436] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,437] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:57,437] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:11:57,438] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:57,438] jepsen worker 39 - jepsen.util 1929 :info :write [13 27] :RequestTimeoutException INFO [2022-03-13 17:11:57,439] jepsen worker 39 - jepsen.util 2019 :invoke :read [13 nil] INFO [2022-03-13 17:11:57,442] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,442] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,442] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:57,443] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:57,463] jepsen worker 53 - jepsen.util 503 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,464] jepsen worker 53 - jepsen.util 503 :invoke :write [17 28] WARN [2022-03-13 17:11:57,468] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14603243,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":14603243,"total_server_duration_us":11118,"operation_id":"0x551d","timeout_ms":30000,"last_server_duration_us":11118,"total_duration_us":14614882},{"operation_name":"upsert","last_dispatch_duration_us":7726315,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8410308,"total_server_duration_us":325235,"encode_duration_us":21240,"operation_id":"0x5529","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10015938},{"operation_name":"upsert","last_dispatch_duration_us":44499,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1154774,"total_server_duration_us":471592,"encode_duration_us":23795,"operation_id":"0x5761","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10008370},{"operation_name":"upsert","last_dispatch_duration_us":8352148,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8984844,"total_server_duration_us":8633817,"encode_duration_us":17571,"operation_id":"0x5522","timeout_ms":10000,"last_server_duration_us":8340256,"total_duration_us":10008122},{"operation_name":"upsert","last_dispatch_duration_us":35687,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":817237,"total_server_duration_us":371139,"encode_duration_us":20581,"operation_id":"0x57ad","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10005429},{"operation_name":"upsert","last_dispatch_duration_us":33367,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":603587,"total_server_duration_us":375531,"encode_duration_us":18410,"operation_id":"0x5807","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10005284},{"operation_name":"upsert","last_dispatch_duration_us":7258521,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7899095,"total_server_duration_us":349520,"encode_duration_us":17372,"operation_id":"0x5575","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10001174},{"operation_name":"upsert","last_dispatch_duration_us":36870,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1103780,"total_server_duration_us":384911,"encode_duration_us":17580,"operation_id":"0x5762","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10001172},{"operation_name":"get","last_dispatch_duration_us":5310955,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":5310955,"total_server_duration_us":49,"operation_id":"0x5756","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":5313313},{"operation_name":"upsert","last_dispatch_duration_us":910162,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1911556,"total_server_duration_us":708984,"encode_duration_us":20433,"operation_id":"0x5a32","timeout_ms":10000,"last_server_duration_us":92589,"total_duration_us":1928528}],"total_count":64}} WARN [2022-03-13 17:11:57,469] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8352148,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":8984844,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":8633817,"encode_duration_us":17571,"operation_id":"0x5522","timeout_ms":10000,"last_server_duration_us":8340256,"total_duration_us":10008122},{"operation_name":"upsert","last_dispatch_duration_us":17719223,"last_local_id":"A312307A00000005/00000000F4F37B9B","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":18461053,"last_local_socket":"172.23.104.79:33502","total_server_duration_us":9454660,"encode_duration_us":18487,"operation_id":"0x5234","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10010306}],"total_count":2}} WARN [2022-03-13 17:11:57,469] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":39065,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1220224,"total_server_duration_us":672904,"encode_duration_us":39241,"operation_id":"0x565d","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10010987},{"operation_name":"upsert","last_dispatch_duration_us":8362591,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":8362591,"total_server_duration_us":23644,"encode_duration_us":17304,"operation_id":"0x5794","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10010344},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":26815,"operation_id":"0x574b","timeout_ms":10000,"total_duration_us":10008321},{"operation_name":"upsert","last_dispatch_duration_us":8385032,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9161795,"total_server_duration_us":516582,"encode_duration_us":22610,"operation_id":"0x553b","timeout_ms":10000,"last_server_duration_us":51710,"total_duration_us":10007767},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":18742,"operation_id":"0x561a","timeout_ms":10000,"total_duration_us":10003054},{"operation_name":"upsert","last_dispatch_duration_us":32919,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1223554,"total_server_duration_us":623627,"encode_duration_us":17645,"operation_id":"0x5664","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10002138},{"operation_name":"upsert","last_dispatch_duration_us":8861729,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8861729,"total_server_duration_us":21324,"encode_duration_us":25389,"operation_id":"0x56b5","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10002111},{"operation_name":"get","last_dispatch_duration_us":62867,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":62867,"total_server_duration_us":15,"operation_id":"0x57d9","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":7782927},{"operation_name":"get","last_dispatch_duration_us":6046104,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":6046104,"total_server_duration_us":185749,"operation_id":"0x5870","timeout_ms":30000,"last_server_duration_us":185749,"total_duration_us":6051434},{"operation_name":"upsert","last_dispatch_duration_us":410419,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1225762,"total_server_duration_us":645491,"encode_duration_us":18161,"operation_id":"0x5a47","timeout_ms":10000,"last_server_duration_us":63568,"total_duration_us":1243670}],"total_count":26}} INFO [2022-03-13 17:11:57,491] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,491] jepsen worker 35 - jepsen.util 845 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,491] jepsen worker 34 - jepsen.util 754 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,492] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:11:57,492] jepsen worker 35 - jepsen.util 845 :invoke :write [11 8] INFO [2022-03-13 17:11:57,493] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:57,526] jepsen worker 21 - jepsen.util 741 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,527] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:57,529] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,530] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,530] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,531] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:11:57,531] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:57,532] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:57,532] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,533] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:11:57,533] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,534] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] WARN [2022-03-13 17:11:57,564] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17244830,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":17244830,"total_server_duration_us":9034183,"encode_duration_us":30237,"operation_id":"0x552f","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10006776},{"operation_name":"upsert","last_dispatch_duration_us":3891149,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12838060,"total_server_duration_us":49577,"encode_duration_us":19564,"operation_id":"0x5544","timeout_ms":10000,"last_server_duration_us":27764,"total_duration_us":10006554},{"operation_name":"upsert","last_dispatch_duration_us":8008160,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8008160,"total_server_duration_us":43508,"encode_duration_us":22126,"operation_id":"0x56b3","timeout_ms":10000,"last_server_duration_us":43508,"total_duration_us":10006082},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":18428,"operation_id":"0x5726","timeout_ms":10000,"total_duration_us":10003425},{"operation_name":"upsert","last_dispatch_duration_us":9154451,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9154451,"total_server_duration_us":9039489,"encode_duration_us":25779,"operation_id":"0x5624","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":9160901},{"operation_name":"upsert","last_dispatch_duration_us":9035467,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9035467,"total_server_duration_us":9022513,"encode_duration_us":29638,"operation_id":"0x5533","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":9039270},{"operation_name":"get","last_dispatch_duration_us":8383664,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8383664,"total_server_duration_us":117283,"operation_id":"0x55c7","timeout_ms":30000,"last_server_duration_us":117283,"total_duration_us":8385712},{"operation_name":"upsert","last_dispatch_duration_us":4033529,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4033529,"total_server_duration_us":341543,"encode_duration_us":19407,"operation_id":"0x5734","timeout_ms":10000,"last_server_duration_us":341543,"total_duration_us":4039009},{"operation_name":"get","last_dispatch_duration_us":3753788,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3753788,"total_server_duration_us":49,"operation_id":"0x5748","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":3754006},{"operation_name":"get","last_dispatch_duration_us":3753587,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3753587,"total_server_duration_us":19,"operation_id":"0x5749","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":3753744}],"total_count":57}} WARN [2022-03-13 17:11:57,569] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9066875,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":10089063,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9700129,"encode_duration_us":20016,"operation_id":"0x53f7","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10004932},{"operation_name":"upsert","last_dispatch_duration_us":17472125,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17992149,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9219667,"encode_duration_us":18956,"operation_id":"0x53fc","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10005647}],"total_count":2}} WARN [2022-03-13 17:11:57,571] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":316164,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8298700,"total_server_duration_us":138036,"encode_duration_us":21430,"operation_id":"0x57b6","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10010614},{"operation_name":"upsert","last_dispatch_duration_us":8348493,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":14767609,"total_server_duration_us":250353,"encode_duration_us":18264,"operation_id":"0x54d4","timeout_ms":10000,"last_server_duration_us":25362,"total_duration_us":10006723},{"operation_name":"upsert","last_dispatch_duration_us":7454116,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8771421,"total_server_duration_us":317334,"encode_duration_us":22213,"operation_id":"0x56b6","timeout_ms":10000,"last_server_duration_us":24841,"total_duration_us":10006659},{"operation_name":"upsert","last_dispatch_duration_us":33003,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":1205569,"total_server_duration_us":607626,"encode_duration_us":22164,"operation_id":"0x54c4","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10005998},{"operation_name":"upsert","last_dispatch_duration_us":9054205,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9775423,"total_server_duration_us":9239760,"encode_duration_us":17213,"operation_id":"0x56f5","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10005130},{"operation_name":"upsert","last_dispatch_duration_us":36380,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":1272107,"total_server_duration_us":608207,"encode_duration_us":20859,"operation_id":"0x54cc","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10004283},{"operation_name":"upsert","last_dispatch_duration_us":8346950,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11489920,"total_server_duration_us":250869,"encode_duration_us":18458,"operation_id":"0x5635","timeout_ms":10000,"last_server_duration_us":28768,"total_duration_us":10003725},{"operation_name":"upsert","last_dispatch_duration_us":319904,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8143164,"total_server_duration_us":41842,"encode_duration_us":18213,"operation_id":"0x57e8","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10002852},{"operation_name":"upsert","last_dispatch_duration_us":114527,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":700496,"total_server_duration_us":295491,"encode_duration_us":27281,"operation_id":"0x574d","timeout_ms":10000,"last_server_duration_us":63053,"total_duration_us":10002526},{"operation_name":"upsert","last_dispatch_duration_us":7863812,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9140475,"total_server_duration_us":396380,"encode_duration_us":18540,"operation_id":"0x56b4","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10002365}],"total_count":78}} INFO [2022-03-13 17:11:57,603] jepsen worker 87 - jepsen.util 2517 :info :write [29 5] :RequestTimeoutException INFO [2022-03-13 17:11:57,604] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:11:57,641] jepsen worker 51 - jepsen.util 591 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,641] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,641] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,641] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,641] jepsen worker 78 - jepsen.util 528 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,642] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,642] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:57,642] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,643] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:57,644] jepsen worker 51 - jepsen.util 591 :invoke :write [17 4] INFO [2022-03-13 17:11:57,646] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,647] jepsen worker 77 - jepsen.util 707 :invoke :write [25 47] INFO [2022-03-13 17:11:57,648] jepsen worker 12 - jepsen.util 732 :invoke :write [4 16] INFO [2022-03-13 17:11:57,649] jepsen worker 78 - jepsen.util 528 :invoke :write [26 42] INFO [2022-03-13 17:11:57,649] jepsen worker 52 - jepsen.util 592 :invoke :write [17 25] INFO [2022-03-13 17:11:57,650] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] WARN [2022-03-13 17:11:57,664] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11241646,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11241646,"total_server_duration_us":38,"operation_id":"0x56a9","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":11241959},{"operation_name":"upsert","last_dispatch_duration_us":123687,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":123687,"total_server_duration_us":23390,"encode_duration_us":18809,"operation_id":"0x5856","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10010515},{"operation_name":"upsert","last_dispatch_duration_us":126258,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9857905,"total_server_duration_us":178200,"encode_duration_us":24931,"operation_id":"0x5759","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10009465},{"operation_name":"upsert","last_dispatch_duration_us":35588,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1281212,"total_server_duration_us":252843,"encode_duration_us":19951,"operation_id":"0x56ae","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10007235},{"operation_name":"upsert","last_dispatch_duration_us":40587,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":1123670,"total_server_duration_us":604524,"encode_duration_us":18445,"operation_id":"0x5580","timeout_ms":10000,"last_server_duration_us":30256,"total_duration_us":10006291},{"operation_name":"upsert","last_dispatch_duration_us":8735249,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10148026,"total_server_duration_us":268441,"encode_duration_us":21574,"operation_id":"0x565c","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10005849},{"operation_name":"upsert","last_dispatch_duration_us":8520110,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10161366,"total_server_duration_us":303325,"encode_duration_us":30496,"operation_id":"0x565b","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10004439},{"operation_name":"upsert","last_dispatch_duration_us":66597,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9126392,"total_server_duration_us":256512,"encode_duration_us":18939,"operation_id":"0x57a3","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10004334},{"operation_name":"upsert","last_dispatch_duration_us":53954,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1618948,"total_server_duration_us":274150,"encode_duration_us":78647,"operation_id":"0x565f","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10002235},{"operation_name":"upsert","last_dispatch_duration_us":9227253,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9481323,"total_server_duration_us":9068866,"encode_duration_us":24012,"operation_id":"0x5660","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9508718}],"total_count":66}} INFO [2022-03-13 17:11:57,683] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,687] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:11:57,690] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.52046783625731}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:57,690] jepsen worker nemesis - couchbase.util Rebalance check count: 228 INFO [2022-03-13 17:11:57,690] jepsen worker nemesis - couchbase.util Rebalance stuck count: 144 INFO [2022-03-13 17:11:57,732] jepsen worker 75 - jepsen.util 435 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,732] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:57,746] jepsen worker 80 - jepsen.util 710 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,746] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,746] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,746] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,746] jepsen worker 80 - jepsen.util 710 :invoke :write [26 37] INFO [2022-03-13 17:11:57,747] jepsen worker 34 - jepsen.util 754 :invoke :write [11 12] INFO [2022-03-13 17:11:57,748] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:57,748] jepsen worker 70 - jepsen.util 700 :invoke :write [23 43] WARN [2022-03-13 17:11:57,765] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17428182,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18210951,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9567712,"encode_duration_us":16835,"operation_id":"0x52ac","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10000882}],"total_count":1}} INFO [2022-03-13 17:11:57,782] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,782] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,783] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:57,783] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] INFO [2022-03-13 17:11:57,828] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,828] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,828] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,835] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,855] jepsen worker 69 - jepsen.util 609 :invoke :write [23 34] INFO [2022-03-13 17:11:57,855] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:57,855] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:11:57,855] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:11:57,914] jepsen worker 42 - jepsen.util 402 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,914] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,915] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:57,915] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:57,915] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:57,916] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:11:58,071] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,071] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,071] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,072] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,073] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,073] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,073] jepsen worker 78 - jepsen.util 528 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,074] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,078] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,078] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,078] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:58,079] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,080] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,080] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,080] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,081] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:11:58,081] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,081] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,082] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,082] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:58,083] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,083] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:11:58,084] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,085] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,087] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,087] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,088] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,089] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,089] jepsen worker 79 - jepsen.util 889 :invoke :write [26 18] INFO [2022-03-13 17:11:58,090] jepsen worker 76 - jepsen.util 616 :invoke :write [25 32] INFO [2022-03-13 17:11:58,128] jepsen worker 16 - jepsen.util 1726 :info :write [5 46] :RequestTimeoutException INFO [2022-03-13 17:11:58,129] jepsen worker 16 - jepsen.util 1816 :invoke :write [5 38] INFO [2022-03-13 17:11:58,135] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,136] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,153] jepsen worker 35 - jepsen.util 845 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,153] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,191] jepsen worker 79 - jepsen.util 889 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,192] jepsen worker 79 - jepsen.util 889 :invoke :write [26 48] INFO [2022-03-13 17:11:58,220] jepsen worker 69 - jepsen.util 609 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,220] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,220] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,221] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,221] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:58,222] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:11:58,223] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:58,223] jepsen worker 24 - jepsen.util 564 :invoke :write [8 38] INFO [2022-03-13 17:11:58,253] jepsen worker 76 - jepsen.util 616 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,254] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,256] jepsen worker 58 - jepsen.util 1768 :info :write [19 20] :RequestTimeoutException INFO [2022-03-13 17:11:58,256] jepsen worker 58 - jepsen.util 1858 :invoke :read [19 nil] INFO [2022-03-13 17:11:58,319] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,320] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,320] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,320] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:11:58,321] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,321] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,322] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,322] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,323] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:58,323] jepsen worker 21 - jepsen.util 741 :invoke :write [7 25] INFO [2022-03-13 17:11:58,346] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,346] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,346] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,346] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:58,346] jepsen worker 52 - jepsen.util 592 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,347] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:11:58,347] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,348] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,349] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,353] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:11:58,353] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:11:58,355] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,362] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,362] jepsen worker 35 - jepsen.util 845 :invoke :write [11 36] INFO [2022-03-13 17:11:58,370] jepsen worker 82 - jepsen.util 1882 :info :write [27 37] :RequestTimeoutException INFO [2022-03-13 17:11:58,371] jepsen worker 82 - jepsen.util 1972 :invoke :read [27 nil] INFO [2022-03-13 17:11:58,383] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,385] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:11:58,398] jepsen worker 5 - jepsen.util 2255 :info :write [1 24] :RequestTimeoutException INFO [2022-03-13 17:11:58,398] jepsen worker 5 - jepsen.util 2345 :invoke :write [1 12] INFO [2022-03-13 17:11:58,430] jepsen worker 79 - jepsen.util 889 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,430] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,430] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,431] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,432] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,432] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,436] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,436] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,436] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,437] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,437] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,438] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,441] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,441] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,441] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,442] jepsen worker 79 - jepsen.util 889 :invoke :write [26 16] INFO [2022-03-13 17:11:58,442] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,443] jepsen worker 76 - jepsen.util 616 :invoke :write [25 48] INFO [2022-03-13 17:11:58,470] jepsen worker 89 - jepsen.util 2339 :info :write [29 16] :RequestTimeoutException INFO [2022-03-13 17:11:58,471] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:11:58,480] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,482] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,482] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,482] jepsen worker 24 - jepsen.util 564 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,482] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,482] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,483] jepsen worker 71 - jepsen.util 521 :invoke :write [23 1] INFO [2022-03-13 17:11:58,483] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,495] jepsen worker 80 - jepsen.util 710 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,496] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,529] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,530] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:11:58,542] jepsen worker 59 - jepsen.util 1769 :info :write [19 48] :SyncWriteAmbiguous INFO [2022-03-13 17:11:58,543] jepsen worker 59 - jepsen.util 1859 :invoke :write [19 41] INFO [2022-03-13 17:11:58,583] jepsen worker 79 - jepsen.util 889 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,583] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,588] jepsen worker 21 - jepsen.util 741 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,589] jepsen worker 21 - jepsen.util 741 :invoke :write [7 0] INFO [2022-03-13 17:11:58,594] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,594] jepsen worker 22 - jepsen.util 652 :invoke :write [7 7] INFO [2022-03-13 17:11:58,610] jepsen worker 84 - jepsen.util 2064 :ok :read [28 44] INFO [2022-03-13 17:11:58,612] jepsen worker 84 - jepsen.util 2064 :invoke :write [28 18] INFO [2022-03-13 17:11:58,658] jepsen worker 76 - jepsen.util 616 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,659] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,659] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,659] jepsen worker 76 - jepsen.util 616 :invoke :write [25 43] INFO [2022-03-13 17:11:58,659] jepsen worker 79 - jepsen.util 889 :invoke :write [26 11] INFO [2022-03-13 17:11:58,660] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,661] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,661] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,661] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,661] jepsen worker 69 - jepsen.util 609 :invoke :write [23 18] INFO [2022-03-13 17:11:58,662] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:11:58,662] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:11:58,661] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,665] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:58,667] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,668] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:11:58,703] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.52046783625731}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:58,703] jepsen worker nemesis - couchbase.util Rebalance check count: 229 INFO [2022-03-13 17:11:58,703] jepsen worker nemesis - couchbase.util Rebalance stuck count: 145 INFO [2022-03-13 17:11:58,719] jepsen worker 71 - jepsen.util 521 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,720] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:11:58,728] jepsen worker 70 - jepsen.util 700 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,728] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:58,763] jepsen worker 79 - jepsen.util 889 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,764] jepsen worker 79 - jepsen.util 889 :invoke :write [26 23] INFO [2022-03-13 17:11:58,797] jepsen worker 35 - jepsen.util 845 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,798] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,854] jepsen worker 76 - jepsen.util 616 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,854] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,862] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,863] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:11:58,895] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,895] jepsen worker 34 - jepsen.util 754 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,895] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,895] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,895] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,896] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,896] jepsen worker 52 - jepsen.util 592 :invoke :write [17 24] INFO [2022-03-13 17:11:58,897] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,897] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,898] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,899] jepsen worker 70 - jepsen.util 700 :invoke :write [23 17] INFO [2022-03-13 17:11:58,899] jepsen worker 13 - jepsen.util 643 :invoke :write [4 10] INFO [2022-03-13 17:11:58,902] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,903] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,903] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,903] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,904] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:11:58,904] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,904] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,905] jepsen worker 80 - jepsen.util 710 :invoke :write [26 8] INFO [2022-03-13 17:11:58,906] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,907] jepsen worker 78 - jepsen.util 528 :invoke :write [26 45] INFO [2022-03-13 17:11:58,907] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,907] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,952] jepsen worker 21 - jepsen.util 741 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,952] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,953] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:11:58,955] jepsen worker 21 - jepsen.util 741 :invoke :write [7 27] INFO [2022-03-13 17:11:58,960] jepsen worker 22 - jepsen.util 652 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,960] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,961] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:11:58,961] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:11:58,962] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,963] jepsen worker 35 - jepsen.util 845 :invoke :write [11 49] INFO [2022-03-13 17:11:58,970] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:58,976] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:11:59,066] jepsen worker 79 - jepsen.util 889 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,066] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,067] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:59,067] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:59,081] jepsen worker 52 - jepsen.util 592 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,082] jepsen worker 52 - jepsen.util 592 :invoke :write [17 44] INFO [2022-03-13 17:11:59,132] jepsen worker 69 - jepsen.util 609 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,133] jepsen worker 69 - jepsen.util 609 :invoke :write [23 4] INFO [2022-03-13 17:11:59,136] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,136] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,137] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,137] jepsen worker 76 - jepsen.util 616 :invoke :write [25 9] INFO [2022-03-13 17:11:59,138] jepsen worker 79 - jepsen.util 889 :invoke :write [26 44] INFO [2022-03-13 17:11:59,141] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,143] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,143] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,144] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,145] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,146] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,147] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,148] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,149] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,150] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,150] jepsen worker 44 - jepsen.util 584 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,151] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,151] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,151] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,152] jepsen worker 44 - jepsen.util 584 :invoke :write [14 5] INFO [2022-03-13 17:11:59,152] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:11:59,153] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:11:59,155] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,156] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:11:59,183] jepsen worker 13 - jepsen.util 643 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,183] jepsen worker 13 - jepsen.util 643 :invoke :write [4 29] INFO [2022-03-13 17:11:59,195] jepsen worker 32 - jepsen.util 1742 :ok :read [10 :nil] INFO [2022-03-13 17:11:59,196] jepsen worker 32 - jepsen.util 1742 :invoke :write [10 35] INFO [2022-03-13 17:11:59,210] jepsen worker 73 - jepsen.util 1963 :ok :read [24 24] INFO [2022-03-13 17:11:59,211] jepsen worker 73 - jepsen.util 1963 :invoke :write [24 36] INFO [2022-03-13 17:11:59,251] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,252] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,281] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,282] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:59,327] jepsen worker 79 - jepsen.util 889 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,329] jepsen worker 79 - jepsen.util 889 :invoke :write [26 19] INFO [2022-03-13 17:11:59,345] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,346] jepsen worker 8 - jepsen.util 728 :invoke :write [2 24] INFO [2022-03-13 17:11:59,382] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,383] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:11:59,409] jepsen worker 0 - jepsen.util 1890 :info :write [0 16] :SyncWriteAmbiguous INFO [2022-03-13 17:11:59,410] jepsen worker 0 - jepsen.util 1980 :invoke :write [0 23] INFO [2022-03-13 17:11:59,424] jepsen worker 80 - jepsen.util 710 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,424] jepsen worker 80 - jepsen.util 710 :invoke :write [26 42] INFO [2022-03-13 17:11:59,437] jepsen worker 76 - jepsen.util 616 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,437] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:59,450] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,450] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,452] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:59,452] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:11:59,489] jepsen worker 21 - jepsen.util 741 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,491] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:59,496] jepsen worker 35 - jepsen.util 845 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,496] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,496] jepsen worker 35 - jepsen.util 845 :invoke :write [11 31] INFO [2022-03-13 17:11:59,497] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:59,521] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,522] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:11:59,522] jepsen worker 69 - jepsen.util 609 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,522] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,522] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,523] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,523] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:11:59,524] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:11:59,524] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:11:59,525] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:59,534] jepsen worker 17 - jepsen.util 1907 :info :write [5 16] :SyncWriteAmbiguous INFO [2022-03-13 17:11:59,535] jepsen worker 17 - jepsen.util 1997 :invoke :read [5 nil] INFO [2022-03-13 17:11:59,557] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,558] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,558] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,558] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:11:59,559] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,559] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,560] jepsen worker 85 - jepsen.util 2065 :ok :read [28 44] INFO [2022-03-13 17:11:59,561] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:11:59,561] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:11:59,562] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:59,562] jepsen worker 85 - jepsen.util 2065 :invoke :read [28 nil] INFO [2022-03-13 17:11:59,563] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:11:59,617] jepsen worker 81 - jepsen.util 2241 :info :write [27 37] :RequestTimeoutException INFO [2022-03-13 17:11:59,617] jepsen worker 81 - jepsen.util 2331 :invoke :read [27 nil] INFO [2022-03-13 17:11:59,652] jepsen worker 79 - jepsen.util 889 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,652] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,653] jepsen worker 79 - jepsen.util 889 :invoke :write [26 38] INFO [2022-03-13 17:11:59,653] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:59,666] jepsen worker 38 - jepsen.util 2378 :info :write [12 30] :RequestTimeoutException INFO [2022-03-13 17:11:59,667] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:11:59,708] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.52046783625731}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:11:59,709] jepsen worker nemesis - couchbase.util Rebalance check count: 230 INFO [2022-03-13 17:11:59,709] jepsen worker nemesis - couchbase.util Rebalance stuck count: 146 INFO [2022-03-13 17:11:59,720] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,720] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,720] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:59,721] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:11:59,727] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,727] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,729] jepsen worker 70 - jepsen.util 700 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,730] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:11:59,800] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,800] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,800] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,800] jepsen worker 21 - jepsen.util 741 :invoke :write [7 5] INFO [2022-03-13 17:11:59,803] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:11:59,803] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:11:59,808] jepsen worker 51 - jepsen.util 591 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,809] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:11:59,812] jepsen worker 79 - jepsen.util 889 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,812] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,813] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:11:59,813] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:11:59,833] jepsen worker 35 - jepsen.util 845 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,833] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,835] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:11:59,835] jepsen worker 35 - jepsen.util 845 :invoke :write [11 45] INFO [2022-03-13 17:11:59,881] jepsen worker 47 - jepsen.util 2027 :info :write [15 27] :RequestTimeoutException INFO [2022-03-13 17:11:59,881] jepsen worker 47 - jepsen.util 2117 :invoke :read [15 nil] INFO [2022-03-13 17:11:59,891] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,891] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:11:59,900] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,900] jepsen worker 26 - jepsen.util 476 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,900] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,900] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,900] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,901] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,901] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:11:59,902] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:11:59,903] jepsen worker 26 - jepsen.util 476 :invoke :write [8 26] INFO [2022-03-13 17:11:59,903] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,904] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:11:59,904] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,904] jepsen worker 76 - jepsen.util 616 :invoke :write [25 43] INFO [2022-03-13 17:11:59,905] jepsen worker 79 - jepsen.util 889 :invoke :write [26 36] INFO [2022-03-13 17:11:59,921] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,923] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,937] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,937] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:11:59,946] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,947] jepsen worker 78 - jepsen.util 528 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:11:59,947] jepsen worker 78 - jepsen.util 528 :invoke :write [26 16] INFO [2022-03-13 17:11:59,948] jepsen worker 34 - jepsen.util 754 :invoke :write [11 14] INFO [2022-03-13 17:12:00,008] jepsen worker 12 - jepsen.util 732 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,008] jepsen worker 76 - jepsen.util 616 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,008] jepsen worker 12 - jepsen.util 732 :invoke :write [4 34] INFO [2022-03-13 17:12:00,009] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:00,013] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,013] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,013] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,013] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:12:00,014] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:00,014] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:12:00,038] jepsen worker 43 - jepsen.util 493 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,039] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:00,141] jepsen worker 35 - jepsen.util 845 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,142] jepsen worker 35 - jepsen.util 845 :invoke :write [11 19] INFO [2022-03-13 17:12:00,391] jepsen worker 50 - jepsen.util 1940 :info :write [16 46] :RequestTimeoutException INFO [2022-03-13 17:12:00,391] jepsen worker 50 - jepsen.util 2030 :invoke :read [16 nil] INFO [2022-03-13 17:12:00,475] jepsen worker 77 - jepsen.util 707 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,475] jepsen worker 52 - jepsen.util 592 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,476] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:00,477] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:00,478] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,478] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,479] jepsen worker 23 - jepsen.util 563 :invoke :write [7 47] INFO [2022-03-13 17:12:00,479] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:00,483] jepsen worker 21 - jepsen.util 741 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,483] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,484] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,483] jepsen worker 79 - jepsen.util 889 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,485] jepsen worker 79 - jepsen.util 889 :invoke :write [26 24] INFO [2022-03-13 17:12:00,486] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:00,486] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:12:00,490] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:00,500] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,501] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:00,502] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,503] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:00,504] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,505] jepsen worker 76 - jepsen.util 616 :invoke :write [25 11] INFO [2022-03-13 17:12:00,579] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,579] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:12:00,596] jepsen worker 61 - jepsen.util 601 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,597] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,597] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:00,598] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:00,629] jepsen worker 79 - jepsen.util 889 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,630] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:12:00,633] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,634] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:00,676] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,677] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,678] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:12:00,678] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:00,683] jepsen worker 13 - jepsen.util 643 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,683] jepsen worker 13 - jepsen.util 643 :invoke :write [4 1] INFO [2022-03-13 17:12:00,700] jepsen worker 44 - jepsen.util 584 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,700] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,701] jepsen worker 44 - jepsen.util 584 :invoke :write [14 48] INFO [2022-03-13 17:12:00,703] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:00,715] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5263157894736843}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:00,715] jepsen worker nemesis - couchbase.util Rebalance check count: 231 INFO [2022-03-13 17:12:00,715] jepsen worker nemesis - couchbase.util Rebalance stuck count: 146 INFO [2022-03-13 17:12:00,741] jepsen worker 53 - jepsen.util 503 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,742] jepsen worker 53 - jepsen.util 503 :invoke :write [17 5] INFO [2022-03-13 17:12:00,759] jepsen worker 35 - jepsen.util 845 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,759] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,760] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:00,761] jepsen worker 35 - jepsen.util 845 :invoke :write [11 7] INFO [2022-03-13 17:12:00,763] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,764] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:12:00,803] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,804] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:00,812] jepsen worker 80 - jepsen.util 710 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,812] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,812] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,813] jepsen worker 80 - jepsen.util 710 :invoke :write [26 25] INFO [2022-03-13 17:12:00,813] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:00,814] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:00,823] jepsen worker 12 - jepsen.util 732 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,823] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,823] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:00,824] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:00,824] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,824] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:00,895] jepsen worker 8 - jepsen.util 728 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,895] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,896] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,896] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,896] jepsen worker 76 - jepsen.util 616 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,896] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:00,896] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,897] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:00,898] jepsen worker 76 - jepsen.util 616 :invoke :write [25 18] INFO [2022-03-13 17:12:00,899] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:12:00,899] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:00,900] jepsen worker 6 - jepsen.util 636 :invoke :write [2 17] INFO [2022-03-13 17:12:00,916] jepsen worker 68 - jepsen.util 2048 :info :write [22 28] :RequestTimeoutException INFO [2022-03-13 17:12:00,917] jepsen worker 68 - jepsen.util 2138 :invoke :read [22 nil] INFO [2022-03-13 17:12:00,926] jepsen worker 86 - jepsen.util 1616 :info :write [28 26] :RequestTimeoutException INFO [2022-03-13 17:12:00,927] jepsen worker 86 - jepsen.util 1706 :invoke :write [28 37] INFO [2022-03-13 17:12:00,936] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,936] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,936] jepsen worker 21 - jepsen.util 741 :invoke :write [7 6] INFO [2022-03-13 17:12:00,937] jepsen worker 69 - jepsen.util 609 :invoke :write [23 15] INFO [2022-03-13 17:12:00,998] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:00,998] jepsen worker 23 - jepsen.util 563 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,000] jepsen worker 51 - jepsen.util 591 :invoke :write [17 8] INFO [2022-03-13 17:12:01,000] jepsen worker 23 - jepsen.util 563 :invoke :write [7 8] INFO [2022-03-13 17:12:01,004] jepsen worker 34 - jepsen.util 754 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,005] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:01,006] jepsen worker 35 - jepsen.util 845 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,007] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,011] jepsen worker 35 - jepsen.util 845 :invoke :write [11 27] INFO [2022-03-13 17:12:01,013] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:12:01,014] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,014] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,014] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:12:01,014] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:01,060] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,060] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,061] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:12:01,061] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:01,083] jepsen worker 26 - jepsen.util 476 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,083] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:01,084] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,084] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:01,084] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,086] jepsen worker 26 - jepsen.util 476 :invoke :write [8 45] INFO [2022-03-13 17:12:01,087] jepsen worker 7 - jepsen.util 637 :invoke :write [2 9] INFO [2022-03-13 17:12:01,088] jepsen worker 61 - jepsen.util 601 :invoke :write [20 42] INFO [2022-03-13 17:12:01,088] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:01,090] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:01,104] jepsen worker 78 - jepsen.util 528 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,104] jepsen worker 78 - jepsen.util 528 :invoke :write [26 26] INFO [2022-03-13 17:12:01,104] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,105] jepsen worker 52 - jepsen.util 592 :invoke :write [17 13] INFO [2022-03-13 17:12:01,123] jepsen worker 76 - jepsen.util 616 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,123] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,123] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,124] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:01,124] jepsen worker 76 - jepsen.util 616 :invoke :write [25 34] INFO [2022-03-13 17:12:01,126] jepsen worker 79 - jepsen.util 889 :invoke :write [26 42] INFO [2022-03-13 17:12:01,130] jepsen worker 48 - jepsen.util 1938 :info :write [16 29] :RequestTimeoutException INFO [2022-03-13 17:12:01,131] jepsen worker 48 - jepsen.util 2028 :invoke :read [16 nil] INFO [2022-03-13 17:12:01,189] jepsen worker 44 - jepsen.util 584 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,189] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,189] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,190] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,190] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:01,191] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:01,191] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:01,192] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:01,203] jepsen worker 13 - jepsen.util 643 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,203] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,204] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:01,204] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:01,208] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,208] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,208] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:01,209] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:01,267] jepsen worker 76 - jepsen.util 616 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,268] jepsen worker 76 - jepsen.util 616 :invoke :write [25 2] INFO [2022-03-13 17:12:01,278] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,278] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:01,282] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,283] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,283] jepsen worker 62 - jepsen.util 512 :invoke :write [20 37] INFO [2022-03-13 17:12:01,284] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,284] jepsen worker 53 - jepsen.util 503 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,284] jepsen worker 14 - jepsen.util 284 :invoke :write [4 24] INFO [2022-03-13 17:12:01,288] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:01,289] jepsen worker 53 - jepsen.util 503 :invoke :write [17 34] INFO [2022-03-13 17:12:01,313] jepsen worker 21 - jepsen.util 741 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,314] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:01,368] jepsen worker 79 - jepsen.util 889 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,369] jepsen worker 79 - jepsen.util 889 :invoke :write [26 9] INFO [2022-03-13 17:12:01,384] jepsen worker 80 - jepsen.util 710 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,384] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,385] jepsen worker 34 - jepsen.util 754 :invoke :write [11 8] INFO [2022-03-13 17:12:01,385] jepsen worker 80 - jepsen.util 710 :invoke :write [26 17] INFO [2022-03-13 17:12:01,410] jepsen worker 41 - jepsen.util 1661 :info :write [13 36] :RequestTimeoutException INFO [2022-03-13 17:12:01,411] jepsen worker 41 - jepsen.util 1751 :invoke :write [13 39] INFO [2022-03-13 17:12:01,413] jepsen worker 67 - jepsen.util 1597 :info :write [22 20] :RequestTimeoutException INFO [2022-03-13 17:12:01,414] jepsen worker 67 - jepsen.util 1687 :invoke :read [22 nil] INFO [2022-03-13 17:12:01,474] jepsen worker 69 - jepsen.util 609 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,474] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,475] jepsen worker 35 - jepsen.util 845 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,475] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,475] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:12:01,475] jepsen worker 35 - jepsen.util 845 :invoke :write [11 41] INFO [2022-03-13 17:12:01,476] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:12:01,477] jepsen worker 51 - jepsen.util 591 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,479] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:01,480] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:12:01,558] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,559] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:01,561] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,561] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,562] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:12:01,563] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:01,599] jepsen worker 78 - jepsen.util 528 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,599] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:01,606] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:01,606] jepsen worker 6 - jepsen.util 636 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,607] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:01,607] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:01,608] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:01,608] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:01,655] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,655] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,655] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,655] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,655] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,656] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:12:01,656] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 37] INFO [2022-03-13 17:12:01,657] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:01,657] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:12:01,658] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:01,701] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,701] jepsen worker 36 - jepsen.util 2106 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,702] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,702] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:12:01,703] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:12:01,703] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:12:01,715] jepsen worker 76 - jepsen.util 616 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,716] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:01,722] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5263157894736843}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:01,722] jepsen worker nemesis - couchbase.util Rebalance check count: 232 INFO [2022-03-13 17:12:01,722] jepsen worker nemesis - couchbase.util Rebalance stuck count: 147 INFO [2022-03-13 17:12:01,729] jepsen worker 35 - jepsen.util 845 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,731] jepsen worker 35 - jepsen.util 845 :invoke :write [11 9] INFO [2022-03-13 17:12:01,766] jepsen worker 88 - jepsen.util 1978 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,766] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,766] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:12:01,767] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:01,774] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:01,775] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,775] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,774] jepsen worker 23 - jepsen.util 563 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,774] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,775] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,776] jepsen worker 23 - jepsen.util 563 :invoke :write [7 45] INFO [2022-03-13 17:12:01,777] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:01,778] jepsen worker 51 - jepsen.util 591 :invoke :write [17 21] INFO [2022-03-13 17:12:01,779] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:01,780] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:01,790] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:01,799] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,800] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:12:01,821] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,822] jepsen worker 52 - jepsen.util 592 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,822] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,823] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:01,823] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:01,824] jepsen worker 70 - jepsen.util 700 :invoke :write [23 14] INFO [2022-03-13 17:12:01,840] jepsen worker 79 - jepsen.util 889 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,840] jepsen worker 79 - jepsen.util 889 :invoke :write [26 39] INFO [2022-03-13 17:12:01,887] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,888] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:01,892] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,892] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,892] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,893] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 6] INFO [2022-03-13 17:12:01,893] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 8] INFO [2022-03-13 17:12:01,894] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:12:01,896] jepsen worker 61 - jepsen.util 601 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,896] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,897] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:01,897] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,897] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:01,898] jepsen worker 60 - jepsen.util 600 :invoke :write [20 45] INFO [2022-03-13 17:12:01,902] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:01,902] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,903] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:01,903] jepsen worker 77 - jepsen.util 707 :invoke :write [25 10] INFO [2022-03-13 17:12:01,936] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,937] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:01,939] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,939] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:01,943] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,943] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,944] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:01,944] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:12:01,956] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,956] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,956] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:01,957] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:01,987] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,987] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:01,989] jepsen worker 36 - jepsen.util 2106 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:01,990] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,012] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,013] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,029] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,029] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,032] jepsen worker 34 - jepsen.util 754 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,032] jepsen worker 34 - jepsen.util 754 :invoke :write [11 5] INFO [2022-03-13 17:12:02,032] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,033] jepsen worker 33 - jepsen.util 663 :invoke :write [11 1] INFO [2022-03-13 17:12:02,034] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,034] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,047] jepsen worker 26 - jepsen.util 476 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,047] jepsen worker 23 - jepsen.util 563 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,047] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:02,048] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:02,055] jepsen worker 37 - jepsen.util 2017 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,055] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,056] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:12:02,056] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 27] INFO [2022-03-13 17:12:02,060] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,061] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,086] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:02,086] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:12:02,092] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,093] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 18] INFO [2022-03-13 17:12:02,117] jepsen worker 79 - jepsen.util 889 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,117] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,117] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,117] jepsen worker 79 - jepsen.util 889 :invoke :write [26 0] INFO [2022-03-13 17:12:02,118] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:12:02,119] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:02,120] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,121] jepsen worker 76 - jepsen.util 616 :invoke :write [25 32] INFO [2022-03-13 17:12:02,138] jepsen worker 35 - jepsen.util 845 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,138] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,138] jepsen worker 35 - jepsen.util 845 :invoke :write [11 7] INFO [2022-03-13 17:12:02,139] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:02,146] jepsen worker 36 - jepsen.util 2106 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,147] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,159] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,160] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:02,168] jepsen worker 88 - jepsen.util 1978 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,169] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 4] INFO [2022-03-13 17:12:02,236] jepsen worker 80 - jepsen.util 710 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,236] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,236] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,236] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,236] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,236] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,237] jepsen worker 78 - jepsen.util 528 :invoke :write [26 9] INFO [2022-03-13 17:12:02,237] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,237] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,238] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:12:02,239] jepsen worker 52 - jepsen.util 592 :invoke :write [17 17] INFO [2022-03-13 17:12:02,239] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:02,240] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:02,241] jepsen worker 88 - jepsen.util 1978 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,242] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 12] INFO [2022-03-13 17:12:02,242] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:12:02,246] jepsen worker 20 - jepsen.util 2360 :info :write [6 8] :RequestTimeoutException INFO [2022-03-13 17:12:02,247] jepsen worker 20 - jepsen.util 2450 :invoke :read [6 nil] INFO [2022-03-13 17:12:02,262] jepsen worker 51 - jepsen.util 591 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,263] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,263] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:02,263] jepsen worker 44 - jepsen.util 584 :invoke :write [14 36] INFO [2022-03-13 17:12:02,264] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:02,265] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:02,265] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,265] jepsen worker 7 - jepsen.util 637 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,266] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:02,267] jepsen worker 7 - jepsen.util 637 :invoke :write [2 33] INFO [2022-03-13 17:12:02,291] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,291] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,292] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,293] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,310] jepsen worker 79 - jepsen.util 889 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,311] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:12:02,323] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:02,324] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,332] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,332] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,333] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 14] INFO [2022-03-13 17:12:02,334] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,335] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,335] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,336] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,336] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 6] INFO [2022-03-13 17:12:02,337] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:02,338] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:02,341] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,341] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:12:02,343] jepsen worker 88 - jepsen.util 1978 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,343] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 0] INFO [2022-03-13 17:12:02,356] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,357] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:12:02,359] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:02,360] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:12:02,406] jepsen worker 36 - jepsen.util 2106 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,407] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,448] jepsen worker 76 - jepsen.util 616 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,449] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:02,461] jepsen worker 62 - jepsen.util 512 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,461] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:02,461] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,461] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:02,461] jepsen worker 62 - jepsen.util 512 :invoke :write [20 14] INFO [2022-03-13 17:12:02,462] jepsen worker 60 - jepsen.util 600 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,462] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:02,462] jepsen worker 42 - jepsen.util 402 :invoke :write [14 29] INFO [2022-03-13 17:12:02,463] jepsen worker 6 - jepsen.util 636 :invoke :write [2 36] INFO [2022-03-13 17:12:02,466] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,467] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:12:02,468] jepsen worker 60 - jepsen.util 600 :invoke :write [20 1] INFO [2022-03-13 17:12:02,480] jepsen worker 70 - jepsen.util 700 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,480] jepsen worker 35 - jepsen.util 845 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,480] jepsen worker 35 - jepsen.util 845 :invoke :write [11 33] INFO [2022-03-13 17:12:02,481] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:02,483] jepsen worker 88 - jepsen.util 1978 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,484] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,509] jepsen worker 56 - jepsen.util 1946 :info :write [18 42] :SyncWriteAmbiguous INFO [2022-03-13 17:12:02,510] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,510] jepsen worker 56 - jepsen.util 2036 :invoke :write [18 41] INFO [2022-03-13 17:12:02,511] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 33] INFO [2022-03-13 17:12:02,525] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,525] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:02,561] jepsen worker 49 - jepsen.util 1669 :info :write [16 44] :RequestTimeoutException INFO [2022-03-13 17:12:02,562] jepsen worker 49 - jepsen.util 1759 :invoke :write [16 42] INFO [2022-03-13 17:12:02,566] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,566] jepsen worker 37 - jepsen.util 2017 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,566] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,566] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,567] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:12:02,567] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,571] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,571] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 21] INFO [2022-03-13 17:12:02,579] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,580] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:12:02,607] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:02,608] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:12:02,621] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,621] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,621] jepsen worker 33 - jepsen.util 663 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,621] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,621] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:12:02,622] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:02,623] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:12:02,623] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,624] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:12:02,626] jepsen worker 21 - jepsen.util 741 :invoke :write [7 41] INFO [2022-03-13 17:12:02,645] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,645] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 17] INFO [2022-03-13 17:12:02,683] jepsen worker 34 - jepsen.util 754 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,683] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,683] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,684] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,684] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:02,685] jepsen worker 80 - jepsen.util 710 :invoke :write [26 46] INFO [2022-03-13 17:12:02,685] jepsen worker 34 - jepsen.util 754 :invoke :write [11 0] INFO [2022-03-13 17:12:02,686] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:02,709] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,710] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 30] INFO [2022-03-13 17:12:02,723] jepsen worker 3 - jepsen.util 1893 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,727] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:02,736] jepsen worker 2 - jepsen.util 2252 :info :write [0 42] :RequestTimeoutException INFO [2022-03-13 17:12:02,737] jepsen worker 2 - jepsen.util 2342 :invoke :read [0 nil] INFO [2022-03-13 17:12:02,742] jepsen worker 72 - jepsen.util 1962 :ok :read [24 24] INFO [2022-03-13 17:12:02,742] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5321637426900585}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:02,742] jepsen worker nemesis - couchbase.util Rebalance check count: 233 INFO [2022-03-13 17:12:02,742] jepsen worker nemesis - couchbase.util Rebalance stuck count: 147 INFO [2022-03-13 17:12:02,743] jepsen worker 72 - jepsen.util 1962 :invoke :write [24 23] INFO [2022-03-13 17:12:02,748] jepsen worker 83 - jepsen.util 1433 :info :write [27 20] :RequestTimeoutException INFO [2022-03-13 17:12:02,749] jepsen worker 83 - jepsen.util 1523 :invoke :read [27 nil] INFO [2022-03-13 17:12:02,751] jepsen worker 88 - jepsen.util 1978 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,751] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,779] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,780] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,789] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,789] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:12:02,791] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,791] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:02,792] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,793] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:02,794] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,795] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:02,796] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,796] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:02,798] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,799] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:12:02,803] jepsen worker 55 - jepsen.util 1855 :ok :read [18 21] INFO [2022-03-13 17:12:02,803] jepsen worker 55 - jepsen.util 1855 :invoke :write [18 15] INFO [2022-03-13 17:12:02,813] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,818] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:02,830] jepsen worker 19 - jepsen.util 1909 :ok :read [6 16] INFO [2022-03-13 17:12:02,831] jepsen worker 19 - jepsen.util 1909 :invoke :write [6 18] INFO [2022-03-13 17:12:02,844] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:02,851] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 7] INFO [2022-03-13 17:12:02,880] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,881] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:12:02,934] jepsen worker 88 - jepsen.util 1978 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,935] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 16] INFO [2022-03-13 17:12:02,940] jepsen worker 36 - jepsen.util 2106 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,941] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:02,941] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,942] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:12:02,994] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:02,995] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 33] INFO [2022-03-13 17:12:03,011] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,011] jepsen worker 37 - jepsen.util 2017 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,012] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:12:03,015] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:03,027] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,027] jepsen worker 88 - jepsen.util 1978 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,028] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 14] INFO [2022-03-13 17:12:03,028] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,051] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,052] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,052] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,052] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,052] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:03,053] jepsen worker 76 - jepsen.util 616 :invoke :write [25 14] INFO [2022-03-13 17:12:03,053] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:12:03,054] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:12:03,055] jepsen worker 14 - jepsen.util 284 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,055] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,055] jepsen worker 77 - jepsen.util 707 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,056] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,056] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,056] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:03,057] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:03,058] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:12:03,060] jepsen worker 51 - jepsen.util 591 :invoke :write [17 40] INFO [2022-03-13 17:12:03,061] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:03,061] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:12:03,062] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 35] INFO [2022-03-13 17:12:03,067] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,068] jepsen worker 79 - jepsen.util 889 :invoke :read [26 nil] INFO [2022-03-13 17:12:03,069] jepsen worker 79 - jepsen.util 889 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,070] jepsen worker 79 - jepsen.util 889 :invoke :write [26 7] INFO [2022-03-13 17:12:03,071] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,071] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,086] jepsen worker 31 - jepsen.util 1831 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:12:03,087] jepsen worker 31 - jepsen.util 1921 :invoke :write [10 6] INFO [2022-03-13 17:12:03,093] jepsen worker 78 - jepsen.util 528 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,094] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:03,107] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,108] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,115] jepsen worker 35 - jepsen.util 845 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,115] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,116] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:03,117] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:12:03,117] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,118] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,118] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:03,119] jepsen worker 35 - jepsen.util 845 :invoke :write [11 7] INFO [2022-03-13 17:12:03,119] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,120] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:12:03,134] jepsen worker 1 - jepsen.util 1621 :info :write [0 2] :RequestTimeoutException INFO [2022-03-13 17:12:03,135] jepsen worker 1 - jepsen.util 1711 :invoke :write [0 25] INFO [2022-03-13 17:12:03,151] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:03,152] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:12:03,166] jepsen worker 3 - jepsen.util 1893 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,167] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:12:03,188] jepsen worker 88 - jepsen.util 1978 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,189] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:12:03,199] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:03,200] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,208] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,208] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,208] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,208] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:03,209] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:12:03,209] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:12:03,218] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,218] jepsen worker 52 - jepsen.util 592 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,218] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,219] jepsen worker 52 - jepsen.util 592 :invoke :write [17 34] INFO [2022-03-13 17:12:03,219] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:03,220] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:03,222] jepsen worker 76 - jepsen.util 616 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,222] jepsen worker 76 - jepsen.util 616 :invoke :write [25 21] INFO [2022-03-13 17:12:03,261] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,261] jepsen worker 44 - jepsen.util 584 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,262] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,262] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,262] jepsen worker 53 - jepsen.util 503 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,263] jepsen worker 26 - jepsen.util 476 :invoke :write [8 10] INFO [2022-03-13 17:12:03,263] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:03,264] jepsen worker 53 - jepsen.util 503 :invoke :write [17 30] INFO [2022-03-13 17:12:03,264] jepsen worker 44 - jepsen.util 584 :invoke :write [14 45] INFO [2022-03-13 17:12:03,265] jepsen worker 8 - jepsen.util 728 :invoke :write [2 44] INFO [2022-03-13 17:12:03,278] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,279] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:12:03,310] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,310] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,310] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:03,311] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:12:03,313] jepsen worker 69 - jepsen.util 609 :invoke :write [23 39] INFO [2022-03-13 17:12:03,313] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,331] jepsen worker 36 - jepsen.util 2106 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,332] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:03,338] jepsen worker 34 - jepsen.util 754 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,339] jepsen worker 34 - jepsen.util 754 :invoke :write [11 6] INFO [2022-03-13 17:12:03,344] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,344] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:12:03,368] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,368] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 38] INFO [2022-03-13 17:12:03,389] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,390] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,394] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:03,395] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,400] jepsen worker 28 - jepsen.util 2098 :info :write [9 27] :RequestTimeoutException INFO [2022-03-13 17:12:03,400] jepsen worker 28 - jepsen.util 2188 :invoke :write [9 20] INFO [2022-03-13 17:12:03,407] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,408] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:03,420] jepsen worker 54 - jepsen.util 1674 :info :write [18 5] :RequestTimeoutException INFO [2022-03-13 17:12:03,421] jepsen worker 54 - jepsen.util 1764 :invoke :read [18 nil] INFO [2022-03-13 17:12:03,426] jepsen worker 35 - jepsen.util 845 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,427] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:12:03,438] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,438] jepsen worker 79 - jepsen.util 889 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,438] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 38] INFO [2022-03-13 17:12:03,439] jepsen worker 79 - jepsen.util 889 :invoke :write [26 30] INFO [2022-03-13 17:12:03,451] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:03,451] jepsen worker 7 - jepsen.util 637 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,451] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:03,451] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,451] jepsen worker 60 - jepsen.util 600 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,452] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:12:03,452] jepsen worker 12 - jepsen.util 732 :invoke :write [4 0] INFO [2022-03-13 17:12:03,453] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:03,453] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:03,453] jepsen worker 60 - jepsen.util 600 :invoke :write [20 14] INFO [2022-03-13 17:12:03,467] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,467] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,468] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:03,468] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:03,469] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,470] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 40] INFO [2022-03-13 17:12:03,472] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,473] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:12:03,477] jepsen worker 3 - jepsen.util 1893 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,478] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,494] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,495] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,499] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,499] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,501] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,502] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:12:03,503] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,504] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,503] jepsen worker 21 - jepsen.util 741 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,504] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:03,505] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:12:03,505] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:03,509] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:03,510] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:03,528] jepsen worker 88 - jepsen.util 1978 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,529] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 39] INFO [2022-03-13 17:12:03,536] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:03,536] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:12:03,552] jepsen worker 76 - jepsen.util 616 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,552] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:03,596] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,597] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:12:03,603] jepsen worker 45 - jepsen.util 2295 :info :write [15 9] :RequestTimeoutException INFO [2022-03-13 17:12:03,604] jepsen worker 45 - jepsen.util 2385 :invoke :read [15 nil] INFO [2022-03-13 17:12:03,606] jepsen worker 9 - jepsen.util 1809 :info :write [3 31] :RequestTimeoutException INFO [2022-03-13 17:12:03,606] jepsen worker 9 - jepsen.util 1899 :invoke :write [3 26] INFO [2022-03-13 17:12:03,621] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,621] jepsen worker 80 - jepsen.util 710 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,621] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,621] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,621] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,621] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,623] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:12:03,627] jepsen worker 13 - jepsen.util 643 :invoke :write [4 3] INFO [2022-03-13 17:12:03,627] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:03,627] jepsen worker 35 - jepsen.util 845 :invoke :read [11 nil] INFO [2022-03-13 17:12:03,628] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:03,629] jepsen worker 78 - jepsen.util 528 :invoke :write [26 37] INFO [2022-03-13 17:12:03,629] jepsen worker 35 - jepsen.util 845 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,629] jepsen worker 35 - jepsen.util 845 :invoke :write [11 23] INFO [2022-03-13 17:12:03,633] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,634] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,647] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,648] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:12:03,651] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,651] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:03,653] jepsen worker 88 - jepsen.util 1978 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,654] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 4] INFO [2022-03-13 17:12:03,661] jepsen worker 36 - jepsen.util 2106 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,661] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 28] INFO [2022-03-13 17:12:03,699] jepsen worker 42 - jepsen.util 402 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,699] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,700] jepsen worker 23 - jepsen.util 563 :invoke :write [7 19] INFO [2022-03-13 17:12:03,700] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:12:03,711] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,712] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,713] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,714] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,714] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,716] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,717] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,718] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,718] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,718] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:12:03,719] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,720] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:03,720] jepsen worker 52 - jepsen.util 592 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,721] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:03,721] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,722] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:12:03,735] jepsen worker 79 - jepsen.util 889 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,735] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,735] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:03,736] jepsen worker 79 - jepsen.util 889 :invoke :write [26 46] INFO [2022-03-13 17:12:03,749] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5321637426900585}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:03,749] jepsen worker nemesis - couchbase.util Rebalance check count: 234 INFO [2022-03-13 17:12:03,749] jepsen worker nemesis - couchbase.util Rebalance stuck count: 148 INFO [2022-03-13 17:12:03,751] jepsen worker 88 - jepsen.util 1978 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,751] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 48] INFO [2022-03-13 17:12:03,771] jepsen worker 37 - jepsen.util 2017 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,772] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 37] INFO [2022-03-13 17:12:03,794] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,794] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,796] jepsen worker 76 - jepsen.util 616 :invoke :write [25 17] INFO [2022-03-13 17:12:03,796] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:12:03,808] jepsen worker 34 - jepsen.util 754 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,809] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:03,810] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,810] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,810] jepsen worker 69 - jepsen.util 609 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,811] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,811] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,811] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:03,811] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,811] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:03,812] jepsen worker 21 - jepsen.util 741 :invoke :write [7 0] INFO [2022-03-13 17:12:03,812] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:03,813] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:03,817] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:03,823] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,824] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:12:03,856] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,857] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:12:03,858] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,858] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:03,861] jepsen worker 79 - jepsen.util 889 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,861] jepsen worker 79 - jepsen.util 889 :invoke :write [26 15] INFO [2022-03-13 17:12:03,920] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,920] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,920] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,920] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,920] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,921] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:12:03,922] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:03,923] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:03,924] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:03,924] jepsen worker 69 - jepsen.util 609 :invoke :write [23 24] INFO [2022-03-13 17:12:03,965] jepsen worker 76 - jepsen.util 616 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,966] jepsen worker 76 - jepsen.util 616 :invoke :read [25 nil] INFO [2022-03-13 17:12:03,971] jepsen worker 13 - jepsen.util 643 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,971] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:03,993] jepsen worker 21 - jepsen.util 741 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,993] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,993] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,993] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:03,994] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:03,995] jepsen worker 21 - jepsen.util 741 :invoke :write [7 23] INFO [2022-03-13 17:12:03,995] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:03,996] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:04,014] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,014] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:12:04,016] jepsen worker 65 - jepsen.util 2315 :info :write [21 23] :RequestTimeoutException INFO [2022-03-13 17:12:04,017] jepsen worker 65 - jepsen.util 2405 :invoke :write [21 3] INFO [2022-03-13 17:12:04,039] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,040] jepsen worker 35 - jepsen.util 845 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,041] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:04,041] jepsen worker 35 - jepsen.util 845 :invoke :write [11 9] INFO [2022-03-13 17:12:04,046] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,046] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:04,048] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,048] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:12:04,083] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,083] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:04,091] jepsen worker 79 - jepsen.util 889 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,091] jepsen worker 76 - jepsen.util 616 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,092] jepsen worker 79 - jepsen.util 889 :invoke :write [26 39] INFO [2022-03-13 17:12:04,092] jepsen worker 76 - jepsen.util 616 :invoke :write [25 47] INFO [2022-03-13 17:12:04,113] jepsen worker 78 - jepsen.util 528 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,113] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,113] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,113] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,113] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,113] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,114] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:04,115] jepsen worker 80 - jepsen.util 710 :invoke :write [26 8] INFO [2022-03-13 17:12:04,115] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:04,115] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,116] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,117] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:04,117] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:04,117] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:04,117] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:12:04,117] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:04,118] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,118] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,118] jepsen worker 78 - jepsen.util 528 :invoke :write [26 32] INFO [2022-03-13 17:12:04,119] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,119] jepsen worker 13 - jepsen.util 643 :invoke :write [4 8] INFO [2022-03-13 17:12:04,120] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:04,123] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,124] jepsen worker 52 - jepsen.util 592 :invoke :write [17 2] INFO [2022-03-13 17:12:04,126] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,127] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:04,128] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,129] jepsen worker 34 - jepsen.util 754 :invoke :write [11 29] INFO [2022-03-13 17:12:04,146] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,146] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:12:04,161] jepsen worker 4 - jepsen.util 2164 :info :write [1 14] :RequestTimeoutException INFO [2022-03-13 17:12:04,161] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:04,208] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,208] jepsen worker 36 - jepsen.util 2106 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,209] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:04,209] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:04,217] jepsen worker 88 - jepsen.util 1978 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,217] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 10] INFO [2022-03-13 17:12:04,217] jepsen worker 29 - jepsen.util 2099 :info :write [9 10] :RequestTimeoutException INFO [2022-03-13 17:12:04,218] jepsen worker 29 - jepsen.util 2189 :invoke :read [9 nil] INFO [2022-03-13 17:12:04,295] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,295] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,295] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,295] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:12:04,296] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 23] INFO [2022-03-13 17:12:04,296] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:12:04,311] jepsen worker 88 - jepsen.util 1978 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,312] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,333] jepsen worker 37 - jepsen.util 2017 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,333] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,333] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 16] INFO [2022-03-13 17:12:04,334] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 11] INFO [2022-03-13 17:12:04,341] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,342] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,364] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,364] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:04,373] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,374] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,410] jepsen worker 10 - jepsen.util 1990 :info :write [3 36] :RequestTimeoutException INFO [2022-03-13 17:12:04,411] jepsen worker 10 - jepsen.util 2080 :invoke :read [3 nil] INFO [2022-03-13 17:12:04,425] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,426] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,430] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:04,430] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:12:04,453] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,454] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,494] jepsen worker 36 - jepsen.util 2106 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,495] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 4] INFO [2022-03-13 17:12:04,497] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,498] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,504] jepsen worker 3 - jepsen.util 1893 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,505] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:12:04,531] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,531] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 37] INFO [2022-03-13 17:12:04,557] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,558] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:04,589] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:04,589] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 34] INFO [2022-03-13 17:12:04,620] jepsen worker 88 - jepsen.util 1978 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,621] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,626] jepsen worker 37 - jepsen.util 2017 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,627] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:04,649] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,650] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,655] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,656] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:12:04,676] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,677] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,693] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,694] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,710] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,711] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:04,730] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:04,730] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:12:04,750] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,751] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:12:04,759] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5380116959064327}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:04,759] jepsen worker nemesis - couchbase.util Rebalance check count: 235 INFO [2022-03-13 17:12:04,759] jepsen worker nemesis - couchbase.util Rebalance stuck count: 148 INFO [2022-03-13 17:12:04,776] jepsen worker 69 - jepsen.util 609 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,777] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,776] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,776] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,778] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:04,779] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:04,779] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:04,780] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:04,791] jepsen worker 36 - jepsen.util 2106 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,791] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 7] INFO [2022-03-13 17:12:04,794] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,794] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:12:04,822] jepsen worker 87 - jepsen.util 2607 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,823] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,824] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,824] jepsen worker 21 - jepsen.util 741 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,824] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,824] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,824] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,825] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,825] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:12:04,827] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:04,828] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:12:04,829] jepsen worker 21 - jepsen.util 741 :invoke :write [7 42] INFO [2022-03-13 17:12:04,830] jepsen worker 69 - jepsen.util 609 :invoke :write [23 33] INFO [2022-03-13 17:12:04,830] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:12:04,842] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,843] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 18] INFO [2022-03-13 17:12:04,881] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,881] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 39] INFO [2022-03-13 17:12:04,929] jepsen worker 88 - jepsen.util 1978 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,929] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,943] jepsen worker 36 - jepsen.util 2106 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,943] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:04,953] jepsen worker 33 - jepsen.util 663 :ok :read [11 7] INFO [2022-03-13 17:12:04,953] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:04,956] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,957] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:04,967] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,967] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:04,990] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,990] jepsen worker 37 - jepsen.util 2017 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:04,990] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:04,991] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 48] INFO [2022-03-13 17:12:04,992] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:04,992] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 13] INFO [2022-03-13 17:12:05,010] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,011] jepsen worker 89 - jepsen.util 2429 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,011] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:05,012] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,017] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,017] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 17] INFO [2022-03-13 17:12:05,043] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:05,044] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,092] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:05,093] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,097] jepsen worker 36 - jepsen.util 2106 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,098] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:05,123] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:05,124] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,138] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,138] jepsen worker 37 - jepsen.util 2017 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,139] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:05,139] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 23] INFO [2022-03-13 17:12:05,143] jepsen worker 88 - jepsen.util 1978 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,144] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:12:05,149] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,149] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 47] INFO [2022-03-13 17:12:05,151] jepsen worker 27 - jepsen.util 2187 :info :write [9 24] :RequestTimeoutException INFO [2022-03-13 17:12:05,152] jepsen worker 27 - jepsen.util 2277 :invoke :write [9 30] INFO [2022-03-13 17:12:05,165] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,167] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:12:05,180] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:05,181] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:12:05,195] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,195] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:12:05,208] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:05,209] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:12:05,255] jepsen worker 36 - jepsen.util 2106 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,255] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:05,267] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,267] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:05,280] jepsen worker 30 - jepsen.util 1920 :info :write [10 28] :RequestTimeoutException INFO [2022-03-13 17:12:05,281] jepsen worker 30 - jepsen.util 2010 :invoke :write [10 1] INFO [2022-03-13 17:12:05,338] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,338] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:05,368] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,368] jepsen worker 37 - jepsen.util 2017 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,368] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:12:05,369] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 0] INFO [2022-03-13 17:12:05,407] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,408] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:05,413] jepsen worker 74 - jepsen.util 2414 :info :write [24 15] :RequestTimeoutException INFO [2022-03-13 17:12:05,414] jepsen worker 74 - jepsen.util 2504 :invoke :write [24 5] INFO [2022-03-13 17:12:05,431] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,432] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:12:05,509] jepsen worker 21 - jepsen.util 741 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,511] jepsen worker 21 - jepsen.util 741 :invoke :write [7 31] INFO [2022-03-13 17:12:05,528] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,529] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:05,530] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,532] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,536] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,537] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:05,544] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,545] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:05,552] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,553] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:12:05,576] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:05,581] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 32] INFO [2022-03-13 17:12:05,594] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,594] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:12:05,604] jepsen worker 36 - jepsen.util 2106 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,604] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,605] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 47] INFO [2022-03-13 17:12:05,605] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:12:05,621] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,621] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,622] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:12:05,623] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,638] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,639] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:05,640] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,640] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:12:05,643] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,644] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:12:05,644] jepsen worker 13 - jepsen.util 643 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,645] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:05,652] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:05,652] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:12:05,667] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,667] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,676] jepsen worker 18 - jepsen.util 1818 :info :write [6 35] :RequestTimeoutException INFO [2022-03-13 17:12:05,677] jepsen worker 18 - jepsen.util 1908 :invoke :read [6 nil] INFO [2022-03-13 17:12:05,682] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,683] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:12:05,706] jepsen worker 88 - jepsen.util 1978 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,707] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,712] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:05,712] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,722] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,723] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:05,730] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,730] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:12:05,737] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:05,738] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,758] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,759] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 28] INFO [2022-03-13 17:12:05,765] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5380116959064327}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:05,765] jepsen worker nemesis - couchbase.util Rebalance check count: 236 INFO [2022-03-13 17:12:05,765] jepsen worker nemesis - couchbase.util Rebalance stuck count: 149 INFO [2022-03-13 17:12:05,766] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:05,767] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 17] INFO [2022-03-13 17:12:05,772] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,773] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,773] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:12:05,773] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:05,800] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:05,801] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 40] INFO [2022-03-13 17:12:05,810] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,811] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:05,818] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,819] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:12:05,833] jepsen worker 80 - jepsen.util 710 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,835] jepsen worker 80 - jepsen.util 710 :invoke :write [26 16] INFO [2022-03-13 17:12:05,845] jepsen worker 37 - jepsen.util 2017 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,845] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:05,851] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,851] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] WARN [2022-03-13 17:12:05,876] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17779246,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18243633,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9239249,"encode_duration_us":22804,"operation_id":"0x56b2","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10001531}],"total_count":1}} INFO [2022-03-13 17:12:05,877] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,877] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:12:05,887] jepsen worker 88 - jepsen.util 1978 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,888] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,904] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,904] jepsen worker 6 - jepsen.util 636 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,904] jepsen worker 77 - jepsen.util 707 :invoke :write [25 18] INFO [2022-03-13 17:12:05,905] jepsen worker 6 - jepsen.util 636 :invoke :write [2 36] INFO [2022-03-13 17:12:05,920] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:05,920] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,930] jepsen worker 69 - jepsen.util 609 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,930] jepsen worker 69 - jepsen.util 609 :invoke :write [23 9] INFO [2022-03-13 17:12:05,944] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,944] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,944] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:05,945] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:12:05,945] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:05,945] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:05,967] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:05,968] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] WARN [2022-03-13 17:12:05,976] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7465952,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8124481,"total_server_duration_us":305990,"encode_duration_us":22012,"operation_id":"0x59af","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10010948},{"operation_name":"upsert","last_dispatch_duration_us":72802,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1318535,"total_server_duration_us":444913,"encode_duration_us":20142,"operation_id":"0x5b17","timeout_ms":10000,"last_server_duration_us":34911,"total_duration_us":10008897},{"operation_name":"upsert","last_dispatch_duration_us":7535599,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10103908,"total_server_duration_us":292609,"encode_duration_us":16785,"operation_id":"0x58d0","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10008198},{"operation_name":"upsert","last_dispatch_duration_us":7100775,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7100775,"total_server_duration_us":24583,"encode_duration_us":25012,"operation_id":"0x5a77","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10007225},{"operation_name":"upsert","last_dispatch_duration_us":48079,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":475376,"total_server_duration_us":221936,"encode_duration_us":17838,"operation_id":"0x59fd","timeout_ms":10000,"last_server_duration_us":31203,"total_duration_us":10005718},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":19391,"operation_id":"0x598e","timeout_ms":10000,"total_duration_us":10005163},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":21144,"operation_id":"0x5b6c","timeout_ms":10000,"total_duration_us":10001747},{"operation_name":"upsert","last_dispatch_duration_us":9103167,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9103167,"total_server_duration_us":9026756,"encode_duration_us":29448,"operation_id":"0x59b3","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":9110605},{"operation_name":"get","last_dispatch_duration_us":7604900,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7604900,"total_server_duration_us":76,"operation_id":"0x5a5a","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":7607035},{"operation_name":"upsert","last_dispatch_duration_us":99377,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1320214,"total_server_duration_us":316432,"encode_duration_us":16805,"operation_id":"0x5f89","timeout_ms":10000,"last_server_duration_us":71096,"total_duration_us":1380643}],"total_count":17}} INFO [2022-03-13 17:12:05,984] jepsen worker 36 - jepsen.util 2106 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,984] jepsen worker 89 - jepsen.util 2429 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,985] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:05,985] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,985] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:12:05,987] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:05,988] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 20] INFO [2022-03-13 17:12:05,989] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:12:06,005] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,005] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,005] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:06,006] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:12:06,115] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,116] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:12:06,124] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,124] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,161] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,162] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:12:06,169] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,170] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:06,196] jepsen worker 64 - jepsen.util 1954 :info :write [21 28] :RequestTimeoutException INFO [2022-03-13 17:12:06,197] jepsen worker 64 - jepsen.util 2044 :invoke :read [21 nil] INFO [2022-03-13 17:12:06,211] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,212] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:12:06,284] jepsen worker 34 - jepsen.util 754 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,285] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:06,294] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,294] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,305] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,306] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:12:06,306] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,307] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 6] INFO [2022-03-13 17:12:06,330] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,330] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,330] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:06,331] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:06,336] jepsen worker 21 - jepsen.util 741 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,337] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:06,339] jepsen worker 78 - jepsen.util 528 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,340] jepsen worker 78 - jepsen.util 528 :invoke :write [26 21] INFO [2022-03-13 17:12:06,356] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,356] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,357] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:12:06,364] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:12:06,371] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:06,371] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:12:06,385] jepsen worker 36 - jepsen.util 2106 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,386] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:06,387] jepsen worker 52 - jepsen.util 592 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,387] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,387] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,388] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:06,389] jepsen worker 70 - jepsen.util 700 :invoke :write [23 47] INFO [2022-03-13 17:12:06,389] jepsen worker 13 - jepsen.util 643 :invoke :write [4 49] INFO [2022-03-13 17:12:06,417] jepsen worker 88 - jepsen.util 1978 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,417] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,420] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,420] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,421] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:06,422] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:12:06,454] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,455] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 37] INFO [2022-03-13 17:12:06,459] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,460] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:06,490] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,490] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,491] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:06,492] jepsen worker 80 - jepsen.util 710 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,492] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,492] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:06,493] jepsen worker 34 - jepsen.util 754 :invoke :write [11 20] INFO [2022-03-13 17:12:06,493] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:06,518] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,518] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,519] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:12:06,520] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 33] INFO [2022-03-13 17:12:06,555] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,556] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:06,559] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,559] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,559] jepsen worker 69 - jepsen.util 609 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,560] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:06,560] jepsen worker 21 - jepsen.util 741 :invoke :write [7 1] INFO [2022-03-13 17:12:06,561] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:06,562] jepsen worker 88 - jepsen.util 1978 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,562] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,589] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,589] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,591] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,591] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,610] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,611] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:12:06,612] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,613] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,614] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:12:06,614] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,614] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:12:06,615] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,627] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,627] jepsen worker 78 - jepsen.util 528 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,628] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,628] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,628] jepsen worker 78 - jepsen.util 528 :invoke :write [26 3] INFO [2022-03-13 17:12:06,628] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:06,629] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:06,634] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:06,634] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:06,635] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,638] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,638] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,639] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:06,640] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:06,643] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,644] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:12:06,646] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,647] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 19] INFO [2022-03-13 17:12:06,680] jepsen worker 13 - jepsen.util 643 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,680] jepsen worker 13 - jepsen.util 643 :invoke :write [4 40] INFO [2022-03-13 17:12:06,693] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,694] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:06,702] jepsen worker 21 - jepsen.util 741 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,702] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,702] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,702] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,703] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,704] jepsen worker 33 - jepsen.util 663 :invoke :write [11 0] INFO [2022-03-13 17:12:06,704] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:06,705] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:06,706] jepsen worker 24 - jepsen.util 564 :invoke :write [8 8] INFO [2022-03-13 17:12:06,706] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,708] jepsen worker 75 - jepsen.util 435 :invoke :write [25 37] INFO [2022-03-13 17:12:06,708] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:12:06,726] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:06,727] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:12:06,735] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,735] jepsen worker 70 - jepsen.util 700 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,737] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:06,737] jepsen worker 70 - jepsen.util 700 :invoke :write [23 40] INFO [2022-03-13 17:12:06,770] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5380116959064327}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:06,770] jepsen worker nemesis - couchbase.util Rebalance check count: 237 INFO [2022-03-13 17:12:06,770] jepsen worker nemesis - couchbase.util Rebalance stuck count: 150 INFO [2022-03-13 17:12:06,784] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,784] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,784] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,785] jepsen worker 21 - jepsen.util 741 :invoke :write [7 41] INFO [2022-03-13 17:12:06,787] jepsen worker 34 - jepsen.util 754 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,787] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,787] jepsen worker 69 - jepsen.util 609 :invoke :write [23 37] INFO [2022-03-13 17:12:06,788] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:06,788] jepsen worker 34 - jepsen.util 754 :invoke :write [11 27] INFO [2022-03-13 17:12:06,789] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:12:06,803] jepsen worker 88 - jepsen.util 1978 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,804] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,816] jepsen worker 36 - jepsen.util 2106 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,819] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,819] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,819] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:12:06,820] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:12:06,820] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 11] INFO [2022-03-13 17:12:06,832] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,833] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,836] jepsen worker 78 - jepsen.util 528 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,839] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:06,857] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,859] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,870] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,870] jepsen worker 13 - jepsen.util 643 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,871] jepsen worker 80 - jepsen.util 710 :invoke :write [26 16] INFO [2022-03-13 17:12:06,871] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:06,883] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,883] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,906] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,906] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,906] jepsen worker 70 - jepsen.util 700 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,907] jepsen worker 24 - jepsen.util 564 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,907] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,907] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:06,908] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:12:06,908] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:06,930] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,931] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:06,932] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,932] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,938] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,938] jepsen worker 34 - jepsen.util 754 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,938] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,938] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,939] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:06,939] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:06,940] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:06,940] jepsen worker 13 - jepsen.util 643 :invoke :write [4 38] INFO [2022-03-13 17:12:06,968] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,969] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:06,977] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:06,977] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:12:06,997] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:06,997] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:12:07,007] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,007] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,007] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,007] jepsen worker 80 - jepsen.util 710 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,007] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,008] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:07,008] jepsen worker 52 - jepsen.util 592 :invoke :write [17 13] INFO [2022-03-13 17:12:07,009] jepsen worker 70 - jepsen.util 700 :invoke :write [23 9] INFO [2022-03-13 17:12:07,009] jepsen worker 80 - jepsen.util 710 :invoke :write [26 14] INFO [2022-03-13 17:12:07,017] jepsen worker 78 - jepsen.util 528 :invoke :write [26 13] INFO [2022-03-13 17:12:07,028] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,028] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:12:07,034] jepsen worker 36 - jepsen.util 2106 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,035] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 10] INFO [2022-03-13 17:12:07,088] jepsen worker 33 - jepsen.util 663 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,089] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:07,107] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,107] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,108] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,108] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,108] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 19] INFO [2022-03-13 17:12:07,109] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,117] jepsen worker 13 - jepsen.util 643 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,117] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,117] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:07,118] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:07,133] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,134] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,177] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,178] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:12:07,180] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,180] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,186] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,188] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:12:07,205] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:07,206] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,208] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:07,208] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:12:07,224] jepsen worker 88 - jepsen.util 1978 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,225] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,236] jepsen worker 36 - jepsen.util 2106 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,236] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,236] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 37] INFO [2022-03-13 17:12:07,237] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:12:07,241] jepsen worker 80 - jepsen.util 710 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,242] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:07,267] jepsen worker 75 - jepsen.util 435 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,267] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:12:07,268] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:07,269] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:07,270] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:07,270] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,284] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,284] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,330] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,331] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,343] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,344] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,374] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:07,375] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:12:07,376] jepsen worker 21 - jepsen.util 741 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,376] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] WARN [2022-03-13 17:12:07,378] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":12057365,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12057365,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9112856,"encode_duration_us":17378,"operation_id":"0x586f","timeout_ms":10000,"last_server_duration_us":9112856,"total_duration_us":10007258},{"operation_name":"upsert","last_dispatch_duration_us":17598119,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26622033,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9236239,"encode_duration_us":18796,"operation_id":"0x52fa","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10009886}],"total_count":2}} INFO [2022-03-13 17:12:07,383] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,383] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] WARN [2022-03-13 17:12:07,385] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16818548,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17519044,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9340288,"encode_duration_us":27281,"operation_id":"0x574d","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10002526}],"total_count":1}} INFO [2022-03-13 17:12:07,387] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,387] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,388] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,388] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,389] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 20] INFO [2022-03-13 17:12:07,390] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,392] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,392] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,393] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,393] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:12:07,395] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,395] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 7] INFO [2022-03-13 17:12:07,406] jepsen worker 36 - jepsen.util 2106 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,406] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,406] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 48] INFO [2022-03-13 17:12:07,407] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,408] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,409] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,410] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,410] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:12:07,418] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:07,419] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,444] jepsen worker 70 - jepsen.util 700 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,445] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:07,445] jepsen worker 70 - jepsen.util 700 :invoke :write [23 47] INFO [2022-03-13 17:12:07,446] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 7] INFO [2022-03-13 17:12:07,455] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,456] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] WARN [2022-03-13 17:12:07,483] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":130868,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":862368,"total_server_duration_us":290024,"encode_duration_us":23550,"operation_id":"0x5abf","timeout_ms":10000,"last_server_duration_us":30539,"total_duration_us":10008982},{"operation_name":"upsert","last_dispatch_duration_us":7453933,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7453933,"total_server_duration_us":72999,"encode_duration_us":20447,"operation_id":"0x5b70","timeout_ms":10000,"last_server_duration_us":72999,"total_duration_us":10008189},{"operation_name":"upsert","last_dispatch_duration_us":35925,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":120103,"total_server_duration_us":71337,"encode_duration_us":29770,"operation_id":"0x5a12","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10008024},{"operation_name":"upsert","last_dispatch_duration_us":7947923,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8105826,"total_server_duration_us":76918,"encode_duration_us":17342,"operation_id":"0x5b2d","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10005070},{"operation_name":"upsert","last_dispatch_duration_us":7756021,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8573912,"total_server_duration_us":390327,"encode_duration_us":21224,"operation_id":"0x5abe","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10002661},{"operation_name":"upsert","last_dispatch_duration_us":13951705,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":13951705,"total_server_duration_us":9060729,"encode_duration_us":17498,"operation_id":"0x58db","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10001714},{"operation_name":"upsert","last_dispatch_duration_us":9168441,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9206186,"total_server_duration_us":9156106,"encode_duration_us":27411,"operation_id":"0x5ab7","timeout_ms":10000,"last_server_duration_us":9132037,"total_duration_us":9224234},{"operation_name":"upsert","last_dispatch_duration_us":9062395,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9062395,"total_server_duration_us":9050106,"encode_duration_us":16553,"operation_id":"0x59a6","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9066222},{"operation_name":"get","last_dispatch_duration_us":8961457,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":8961457,"total_server_duration_us":69,"operation_id":"0x5adf","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":8961720},{"operation_name":"upsert","last_dispatch_duration_us":4952196,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":4952196,"total_server_duration_us":48334,"encode_duration_us":18211,"operation_id":"0x5c5f","timeout_ms":10000,"last_server_duration_us":48334,"total_duration_us":4956534}],"total_count":17}} WARN [2022-03-13 17:12:07,484] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18620,"operation_id":"0x596d","timeout_ms":10000,"total_duration_us":10009435},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":23063,"operation_id":"0x59ec","timeout_ms":10000,"total_duration_us":10008294},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17850,"operation_id":"0x5a61","timeout_ms":10000,"total_duration_us":10007195},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18133,"operation_id":"0x5bac","timeout_ms":10000,"total_duration_us":10003116},{"operation_name":"upsert","last_dispatch_duration_us":9053272,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9053272,"total_server_duration_us":9042674,"encode_duration_us":19108,"operation_id":"0x594d","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9057442},{"operation_name":"get","last_dispatch_duration_us":8980054,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8980054,"total_server_duration_us":69,"operation_id":"0x596e","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":8985009},{"operation_name":"upsert","last_dispatch_duration_us":461572,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1724695,"total_server_duration_us":704052,"encode_duration_us":22603,"operation_id":"0x5a8e","timeout_ms":10000,"last_server_duration_us":50301,"total_duration_us":8965748},{"operation_name":"upsert","last_dispatch_duration_us":371262,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1623703,"total_server_duration_us":983088,"encode_duration_us":17155,"operation_id":"0x5b5c","timeout_ms":10000,"last_server_duration_us":357510,"total_duration_us":6842237},{"operation_name":"upsert","last_dispatch_duration_us":488915,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1060937,"total_server_duration_us":86615,"encode_duration_us":19660,"operation_id":"0x5fd8","timeout_ms":10000,"last_server_duration_us":29230,"total_duration_us":1079561},{"operation_name":"upsert","last_dispatch_duration_us":570811,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":989675,"total_server_duration_us":69279,"encode_duration_us":18082,"operation_id":"0x5fd7","timeout_ms":10000,"last_server_duration_us":28584,"total_duration_us":1003887}],"total_count":55}} WARN [2022-03-13 17:12:07,485] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9061633,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10165413,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9425462,"encode_duration_us":17580,"operation_id":"0x5762","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10001172}],"total_count":1}} INFO [2022-03-13 17:12:07,520] jepsen worker 87 - jepsen.util 2607 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,520] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,520] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,521] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,526] jepsen worker 88 - jepsen.util 1978 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,527] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,548] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,549] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:07,557] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:07,558] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 4] INFO [2022-03-13 17:12:07,575] jepsen worker 78 - jepsen.util 528 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,575] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,575] jepsen worker 36 - jepsen.util 2106 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,575] jepsen worker 78 - jepsen.util 528 :invoke :write [26 16] INFO [2022-03-13 17:12:07,576] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,576] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] WARN [2022-03-13 17:12:07,579] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14324084,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":14324084,"total_server_duration_us":49,"operation_id":"0x5792","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":14326005},{"operation_name":"get","last_dispatch_duration_us":13643493,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13643493,"total_server_duration_us":56,"operation_id":"0x57cf","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":13643808},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":17227,"operation_id":"0x5a8f","timeout_ms":10000,"total_duration_us":10007958},{"operation_name":"upsert","last_dispatch_duration_us":12057365,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12057365,"total_server_duration_us":9112856,"encode_duration_us":17378,"operation_id":"0x586f","timeout_ms":10000,"last_server_duration_us":9112856,"total_duration_us":10007258},{"operation_name":"upsert","last_dispatch_duration_us":56824,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":8378289,"total_server_duration_us":922153,"encode_duration_us":20437,"operation_id":"0x58d4","timeout_ms":10000,"last_server_duration_us":46175,"total_duration_us":10007068},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":17380,"operation_id":"0x58c1","timeout_ms":10000,"total_duration_us":10004868},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":17350,"operation_id":"0x5b1b","timeout_ms":10000,"total_duration_us":10002892},{"operation_name":"upsert","last_dispatch_duration_us":379875,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":1069381,"total_server_duration_us":446390,"encode_duration_us":18897,"operation_id":"0x5cb7","timeout_ms":10000,"last_server_duration_us":30539,"total_duration_us":3210475},{"operation_name":"upsert","last_dispatch_duration_us":2812409,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2812409,"total_server_duration_us":31776,"encode_duration_us":19921,"operation_id":"0x5ed7","timeout_ms":10000,"last_server_duration_us":31776,"total_duration_us":2822124},{"operation_name":"upsert","last_dispatch_duration_us":2793785,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2793785,"total_server_duration_us":374609,"encode_duration_us":22371,"operation_id":"0x5c66","timeout_ms":10000,"last_server_duration_us":374609,"total_duration_us":2804502}],"total_count":38}} WARN [2022-03-13 17:12:07,583] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13951705,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":13951705,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9060729,"encode_duration_us":17498,"operation_id":"0x58db","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10001714},{"operation_name":"upsert","last_dispatch_duration_us":18090194,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26899405,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9131528,"encode_duration_us":21716,"operation_id":"0x540d","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10002888},{"operation_name":"upsert","last_dispatch_duration_us":21445740,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21445740,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9037366,"encode_duration_us":18742,"operation_id":"0x561a","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10003054},{"operation_name":"upsert","last_dispatch_duration_us":18112027,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18112027,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9036305,"encode_duration_us":26815,"operation_id":"0x574b","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10008321},{"operation_name":"upsert","last_dispatch_duration_us":9244065,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":17606657,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":94334,"encode_duration_us":17304,"operation_id":"0x5794","timeout_ms":10000,"last_server_duration_us":70690,"total_duration_us":10010344}],"total_count":5}} WARN [2022-03-13 17:12:07,586] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9001016,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9001016,"total_server_duration_us":84,"operation_id":"0x59d4","timeout_ms":30000,"last_server_duration_us":84,"total_duration_us":12124964},{"operation_name":"get","last_dispatch_duration_us":8968477,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8968477,"total_server_duration_us":62,"operation_id":"0x5a60","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":10571648},{"operation_name":"upsert","last_dispatch_duration_us":1031998,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":1811808,"total_server_duration_us":444033,"encode_duration_us":17619,"operation_id":"0x5aa6","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10009205},{"operation_name":"upsert","last_dispatch_duration_us":414686,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":1608875,"total_server_duration_us":700225,"encode_duration_us":18453,"operation_id":"0x588d","timeout_ms":10000,"last_server_duration_us":29415,"total_duration_us":10006247},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":24108,"operation_id":"0x5b7f","timeout_ms":10000,"total_duration_us":10003708},{"operation_name":"upsert","last_dispatch_duration_us":9096443,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9096443,"total_server_duration_us":54939,"encode_duration_us":20858,"operation_id":"0x5ae9","timeout_ms":10000,"last_server_duration_us":54939,"total_duration_us":10002734},{"operation_name":"upsert","last_dispatch_duration_us":2759424,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":5241305,"total_server_duration_us":421728,"encode_duration_us":25065,"operation_id":"0x5a14","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10000717},{"operation_name":"get","last_dispatch_duration_us":8393843,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8393843,"total_server_duration_us":27,"operation_id":"0x5b2f","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":8396494},{"operation_name":"get","last_dispatch_duration_us":3240284,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3240284,"total_server_duration_us":56,"operation_id":"0x5c05","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":3804628},{"operation_name":"upsert","last_dispatch_duration_us":3406542,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3406542,"total_server_duration_us":59249,"encode_duration_us":21482,"operation_id":"0x5e7d","timeout_ms":10000,"last_server_duration_us":59249,"total_duration_us":3418420}],"total_count":46}} INFO [2022-03-13 17:12:07,641] jepsen worker 52 - jepsen.util 592 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,641] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,641] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,641] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,641] jepsen worker 69 - jepsen.util 609 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,643] jepsen worker 52 - jepsen.util 592 :invoke :write [17 6] INFO [2022-03-13 17:12:07,643] jepsen worker 34 - jepsen.util 754 :invoke :write [11 28] INFO [2022-03-13 17:12:07,644] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:07,644] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:07,645] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:07,645] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,646] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,662] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,663] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,663] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,663] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 27] INFO [2022-03-13 17:12:07,665] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:12:07,671] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,674] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,675] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,678] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,679] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,679] jepsen worker 88 - jepsen.util 1978 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,680] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 32] INFO [2022-03-13 17:12:07,680] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,681] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,682] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,683] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 0] INFO [2022-03-13 17:12:07,684] jepsen worker 57 - jepsen.util 2037 :ok :read [19 23] INFO [2022-03-13 17:12:07,684] jepsen worker 57 - jepsen.util 2037 :invoke :write [19 41] WARN [2022-03-13 17:12:07,685] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":19401,"operation_id":"0x58c2","timeout_ms":10000,"total_duration_us":10008026},{"operation_name":"upsert","last_dispatch_duration_us":33466,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1521192,"total_server_duration_us":540192,"encode_duration_us":19666,"operation_id":"0x5ae7","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10007639},{"operation_name":"upsert","last_dispatch_duration_us":33892,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":720950,"total_server_duration_us":448284,"encode_duration_us":19584,"operation_id":"0x5bcb","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10006938},{"operation_name":"upsert","last_dispatch_duration_us":47909,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1318812,"total_server_duration_us":547977,"encode_duration_us":34790,"operation_id":"0x5b11","timeout_ms":10000,"last_server_duration_us":33328,"total_duration_us":10006578},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":17238,"operation_id":"0x59ed","timeout_ms":10000,"total_duration_us":10006135},{"operation_name":"upsert","last_dispatch_duration_us":42737,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":1756465,"total_server_duration_us":703651,"encode_duration_us":17872,"operation_id":"0x5975","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10004242},{"operation_name":"upsert","last_dispatch_duration_us":34809,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1695130,"total_server_duration_us":506791,"encode_duration_us":22709,"operation_id":"0x5aa4","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10002099},{"operation_name":"upsert","last_dispatch_duration_us":8942196,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8942196,"total_server_duration_us":20760,"encode_duration_us":20328,"operation_id":"0x5a8c","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10001887},{"operation_name":"upsert","last_dispatch_duration_us":430818,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":1937227,"total_server_duration_us":1498826,"encode_duration_us":19245,"operation_id":"0x5b6b","timeout_ms":10000,"last_server_duration_us":419839,"total_duration_us":6638206},{"operation_name":"upsert","last_dispatch_duration_us":808124,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2135672,"total_server_duration_us":107563,"encode_duration_us":17639,"operation_id":"0x5f86","timeout_ms":10000,"last_server_duration_us":28584,"total_duration_us":2245150}],"total_count":60}} WARN [2022-03-13 17:12:07,685] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17948990,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19567938,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9322133,"encode_duration_us":78647,"operation_id":"0x565f","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10002235},{"operation_name":"upsert","last_dispatch_duration_us":17786409,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19067622,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9294455,"encode_duration_us":19951,"operation_id":"0x56ae","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10007235}],"total_count":2}} INFO [2022-03-13 17:12:07,723] jepsen worker 70 - jepsen.util 700 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,724] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,724] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:07,724] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,724] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:07,725] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:12:07,740] jepsen worker 3 - jepsen.util 1893 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,741] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:12:07,750] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,751] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:12:07,769] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,770] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,769] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,769] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,770] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,771] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:07,771] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:07,772] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:12:07,773] jepsen worker 88 - jepsen.util 1978 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,773] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:12:07,774] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:12:07,777] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:07,777] jepsen worker 69 - jepsen.util 609 :invoke :write [23 47] INFO [2022-03-13 17:12:07,777] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5380116959064327}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:07,778] jepsen worker nemesis - couchbase.util Rebalance check count: 238 INFO [2022-03-13 17:12:07,778] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,778] jepsen worker nemesis - couchbase.util Rebalance stuck count: 151 INFO [2022-03-13 17:12:07,787] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,788] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:12:07,820] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:07,821] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,850] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:07,851] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:07,855] jepsen worker 5 - jepsen.util 2345 :info :write [1 12] :SyncWriteAmbiguous INFO [2022-03-13 17:12:07,864] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:07,878] jepsen worker 78 - jepsen.util 528 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,878] jepsen worker 78 - jepsen.util 528 :invoke :write [26 47] INFO [2022-03-13 17:12:07,879] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,880] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,880] jepsen worker 21 - jepsen.util 741 :invoke :write [7 10] INFO [2022-03-13 17:12:07,880] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:12:07,881] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,881] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,882] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:12:07,882] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:07,883] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,883] jepsen worker 13 - jepsen.util 643 :invoke :write [4 48] INFO [2022-03-13 17:12:07,892] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,893] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] INFO [2022-03-13 17:12:07,931] jepsen worker 36 - jepsen.util 2106 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,931] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:07,951] jepsen worker 52 - jepsen.util 592 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,952] jepsen worker 52 - jepsen.util 592 :invoke :write [17 44] INFO [2022-03-13 17:12:07,968] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,969] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:12:07,969] jepsen worker 11 - jepsen.util 2081 :ok :read [3 25] INFO [2022-03-13 17:12:07,970] jepsen worker 11 - jepsen.util 2081 :invoke :read [3 nil] INFO [2022-03-13 17:12:07,976] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,976] jepsen worker 33 - jepsen.util 663 :invoke :write [11 9] INFO [2022-03-13 17:12:07,988] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:07,989] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:12:08,016] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:08,017] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:12:08,032] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,034] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,047] jepsen worker 37 - jepsen.util 2017 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,047] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,047] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,048] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 5] INFO [2022-03-13 17:12:08,048] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:12:08,049] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 28] INFO [2022-03-13 17:12:08,063] jepsen worker 34 - jepsen.util 754 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,063] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,063] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,063] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,064] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,065] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:12:08,065] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:08,066] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:08,122] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,122] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,125] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,126] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:12:08,127] jepsen worker 69 - jepsen.util 609 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,127] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,132] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,133] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,150] jepsen worker 78 - jepsen.util 528 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,151] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:08,154] jepsen worker 16 - jepsen.util 1816 :info :write [5 38] :RequestTimeoutException INFO [2022-03-13 17:12:08,155] jepsen worker 16 - jepsen.util 1906 :invoke :read [5 nil] INFO [2022-03-13 17:12:08,178] jepsen worker 62 - jepsen.util 512 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,178] jepsen worker 51 - jepsen.util 591 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,179] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:08,179] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,179] jepsen worker 51 - jepsen.util 591 :invoke :write [17 46] INFO [2022-03-13 17:12:08,179] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,180] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:08,181] jepsen worker 14 - jepsen.util 284 :invoke :write [4 45] INFO [2022-03-13 17:12:08,181] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,182] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:08,190] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,190] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:12:08,217] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,217] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,226] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,226] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:08,232] jepsen worker 13 - jepsen.util 643 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,233] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:08,236] jepsen worker 37 - jepsen.util 2017 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,237] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,266] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:08,267] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 39] INFO [2022-03-13 17:12:08,270] jepsen worker 46 - jepsen.util 2026 :ok :read [15 48] INFO [2022-03-13 17:12:08,271] jepsen worker 46 - jepsen.util 2026 :invoke :write [15 24] INFO [2022-03-13 17:12:08,275] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,276] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:08,279] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:08,280] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:12:08,291] jepsen worker 21 - jepsen.util 741 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,292] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:08,327] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,328] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:12:08,364] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,364] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:08,365] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,365] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:12:08,373] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,374] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,375] jepsen worker 52 - jepsen.util 592 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,375] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,375] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,375] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,376] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,376] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:08,375] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,376] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:08,377] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:08,377] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,378] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,378] jepsen worker 52 - jepsen.util 592 :invoke :write [17 4] INFO [2022-03-13 17:12:08,378] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,379] jepsen worker 80 - jepsen.util 710 :invoke :write [26 11] INFO [2022-03-13 17:12:08,380] jepsen worker 78 - jepsen.util 528 :invoke :write [26 5] INFO [2022-03-13 17:12:08,381] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:08,386] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,386] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:08,388] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,388] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,388] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,389] jepsen worker 34 - jepsen.util 754 :invoke :write [11 46] INFO [2022-03-13 17:12:08,391] jepsen worker 13 - jepsen.util 643 :invoke :write [4 2] INFO [2022-03-13 17:12:08,396] jepsen worker 70 - jepsen.util 700 :invoke :write [23 42] INFO [2022-03-13 17:12:08,406] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,406] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,406] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,406] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,407] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:12:08,407] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,408] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,408] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:08,409] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,409] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:08,410] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,410] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,410] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,411] jepsen worker 33 - jepsen.util 663 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,411] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] INFO [2022-03-13 17:12:08,412] jepsen worker 21 - jepsen.util 741 :invoke :write [7 45] INFO [2022-03-13 17:12:08,412] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:08,415] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,416] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:08,418] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,420] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,421] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,421] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,421] jepsen worker 33 - jepsen.util 663 :invoke :write [11 0] INFO [2022-03-13 17:12:08,422] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:08,422] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:12:08,424] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,424] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:12:08,447] jepsen worker 88 - jepsen.util 1978 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,448] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,479] jepsen worker 52 - jepsen.util 592 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,480] jepsen worker 52 - jepsen.util 592 :invoke :write [17 0] INFO [2022-03-13 17:12:08,495] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,495] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,496] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:08,497] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:08,511] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,511] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,512] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:08,512] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:08,518] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,518] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,519] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:12:08,519] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:08,520] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,521] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:12:08,527] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:08,528] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:12:08,532] jepsen worker 36 - jepsen.util 2106 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,532] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,533] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,533] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,534] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:12:08,534] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,544] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,544] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,557] jepsen worker 80 - jepsen.util 710 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,559] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,559] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:08,559] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:08,570] jepsen worker 59 - jepsen.util 1859 :info :write [19 41] :RequestTimeoutException INFO [2022-03-13 17:12:08,571] jepsen worker 59 - jepsen.util 1949 :invoke :write [19 39] INFO [2022-03-13 17:12:08,580] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:08,581] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,585] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,587] jepsen worker 69 - jepsen.util 609 :invoke :write [23 1] INFO [2022-03-13 17:12:08,605] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,605] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:12:08,618] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:08,619] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,624] jepsen worker 33 - jepsen.util 663 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,625] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:08,636] jepsen worker 84 - jepsen.util 2064 :info :write [28 18] :RequestTimeoutException INFO [2022-03-13 17:12:08,637] jepsen worker 84 - jepsen.util 2154 :invoke :write [28 40] INFO [2022-03-13 17:12:08,641] jepsen worker 40 - jepsen.util 2020 :ok :read [13 39] INFO [2022-03-13 17:12:08,641] jepsen worker 40 - jepsen.util 2020 :invoke :write [13 44] INFO [2022-03-13 17:12:08,658] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,659] jepsen worker 78 - jepsen.util 528 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,659] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:08,660] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:08,667] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,667] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,667] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,668] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:12:08,668] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,669] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 44] INFO [2022-03-13 17:12:08,672] jepsen worker 85 - jepsen.util 2065 :ok :read [28 44] INFO [2022-03-13 17:12:08,673] jepsen worker 85 - jepsen.util 2065 :invoke :read [28 nil] INFO [2022-03-13 17:12:08,676] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,677] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:12:08,682] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,682] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,693] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,695] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,725] jepsen worker 21 - jepsen.util 741 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,726] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:08,726] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,727] jepsen worker 34 - jepsen.util 754 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,727] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:12:08,727] jepsen worker 34 - jepsen.util 754 :invoke :write [11 46] INFO [2022-03-13 17:12:08,759] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:08,760] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,760] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:12:08,761] jepsen worker 13 - jepsen.util 643 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,761] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:08,762] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:08,784] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5380116959064327}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:08,785] jepsen worker nemesis - couchbase.util Rebalance check count: 239 INFO [2022-03-13 17:12:08,785] jepsen worker nemesis - couchbase.util Rebalance stuck count: 152 INFO [2022-03-13 17:12:08,789] jepsen worker 70 - jepsen.util 700 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,789] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:08,795] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:08,795] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:12:08,807] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,808] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:12:08,828] jepsen worker 52 - jepsen.util 592 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,828] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,828] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,828] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:08,829] jepsen worker 78 - jepsen.util 528 :invoke :write [26 28] INFO [2022-03-13 17:12:08,830] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:08,845] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,846] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,846] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,847] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:12:08,849] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,850] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:12:08,854] jepsen worker 34 - jepsen.util 754 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,855] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,855] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:08,855] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,855] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,855] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,855] jepsen worker 70 - jepsen.util 700 :invoke :write [23 30] INFO [2022-03-13 17:12:08,856] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:08,857] jepsen worker 13 - jepsen.util 643 :invoke :write [4 15] INFO [2022-03-13 17:12:08,857] jepsen worker 80 - jepsen.util 710 :invoke :write [26 29] INFO [2022-03-13 17:12:08,860] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,861] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,874] jepsen worker 36 - jepsen.util 2106 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,874] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:08,879] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,880] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:12:08,905] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,905] jepsen worker 78 - jepsen.util 528 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,906] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,907] jepsen worker 78 - jepsen.util 528 :invoke :write [26 14] INFO [2022-03-13 17:12:08,908] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:08,908] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:08,912] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,913] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:08,941] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,942] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:08,951] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,951] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:08,964] jepsen worker 70 - jepsen.util 700 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,966] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:12:08,972] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,973] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,973] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,974] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 34] INFO [2022-03-13 17:12:08,975] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:12:08,976] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 32] INFO [2022-03-13 17:12:08,980] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:08,981] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:09,028] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,028] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,028] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,028] jepsen worker 69 - jepsen.util 609 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,028] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:09,029] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,029] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,029] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:09,036] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,036] jepsen worker 13 - jepsen.util 643 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,037] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 16] INFO [2022-03-13 17:12:09,037] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,038] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:09,039] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:12:09,069] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,070] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,090] jepsen worker 80 - jepsen.util 710 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,090] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,090] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,093] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:09,094] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:09,095] jepsen worker 80 - jepsen.util 710 :invoke :write [26 28] INFO [2022-03-13 17:12:09,107] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,107] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,108] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,108] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:12:09,108] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,108] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,109] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:12:09,109] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,109] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:09,110] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,110] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,110] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:09,111] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,111] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,112] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:09,112] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,112] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,112] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,112] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,113] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,113] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:12:09,113] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,114] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,114] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:09,115] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,115] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,116] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,116] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,117] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:12:09,117] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,118] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,118] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,119] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,119] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,120] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,121] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,121] jepsen worker 69 - jepsen.util 609 :invoke :write [23 11] INFO [2022-03-13 17:12:09,122] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,125] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,125] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,126] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,127] jepsen worker 21 - jepsen.util 741 :invoke :write [7 4] INFO [2022-03-13 17:12:09,129] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,130] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:12:09,160] jepsen worker 36 - jepsen.util 2106 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,161] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,180] jepsen worker 78 - jepsen.util 528 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,180] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:09,187] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,188] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,198] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,198] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,201] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,202] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,203] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,204] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,206] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,207] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:12:09,208] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,208] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,216] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,216] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:09,218] jepsen worker 32 - jepsen.util 1742 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:12:09,219] jepsen worker 32 - jepsen.util 1832 :invoke :read [10 nil] INFO [2022-03-13 17:12:09,221] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,221] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,239] jepsen worker 73 - jepsen.util 1963 :info :write [24 36] :RequestTimeoutException INFO [2022-03-13 17:12:09,239] jepsen worker 73 - jepsen.util 2053 :invoke :write [24 5] INFO [2022-03-13 17:12:09,244] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,245] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,248] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,248] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,248] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,248] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,249] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:09,249] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:09,250] jepsen worker 70 - jepsen.util 700 :invoke :write [23 34] INFO [2022-03-13 17:12:09,250] jepsen worker 52 - jepsen.util 592 :invoke :write [17 9] INFO [2022-03-13 17:12:09,266] jepsen worker 89 - jepsen.util 2429 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,266] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,285] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,286] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:09,301] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,302] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:12:09,322] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,323] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:09,324] jepsen worker 80 - jepsen.util 710 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,324] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,324] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,324] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,326] jepsen worker 80 - jepsen.util 710 :invoke :write [26 44] INFO [2022-03-13 17:12:09,328] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:09,329] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:09,329] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:09,344] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,344] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,365] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,365] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,366] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:09,366] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,372] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,373] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,376] jepsen worker 52 - jepsen.util 592 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,376] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,376] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:09,377] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:12:09,380] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,381] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:12:09,417] jepsen worker 70 - jepsen.util 700 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,417] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,417] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,418] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,418] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,418] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,418] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:09,419] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,420] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:09,420] jepsen worker 34 - jepsen.util 754 :invoke :write [11 21] INFO [2022-03-13 17:12:09,434] jepsen worker 0 - jepsen.util 1980 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:12:09,435] jepsen worker 0 - jepsen.util 2070 :invoke :write [0 15] INFO [2022-03-13 17:12:09,447] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,447] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,447] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,447] jepsen worker 80 - jepsen.util 710 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,447] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,448] jepsen worker 70 - jepsen.util 700 :invoke :write [23 18] INFO [2022-03-13 17:12:09,448] jepsen worker 52 - jepsen.util 592 :invoke :write [17 13] INFO [2022-03-13 17:12:09,449] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:09,450] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,451] jepsen worker 78 - jepsen.util 528 :invoke :write [26 0] INFO [2022-03-13 17:12:09,451] jepsen worker 80 - jepsen.util 710 :invoke :write [26 49] INFO [2022-03-13 17:12:09,458] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:09,493] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,494] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,495] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,495] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:09,516] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,517] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:09,521] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,522] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 19] INFO [2022-03-13 17:12:09,532] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,532] jepsen worker 13 - jepsen.util 643 :invoke :write [4 24] INFO [2022-03-13 17:12:09,533] jepsen worker 34 - jepsen.util 754 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,534] jepsen worker 34 - jepsen.util 754 :invoke :write [11 30] INFO [2022-03-13 17:12:09,535] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,535] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,535] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:09,536] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:12:09,554] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,554] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 5] INFO [2022-03-13 17:12:09,560] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,560] jepsen worker 37 - jepsen.util 2017 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,561] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,562] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:12:09,568] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:09,568] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:12:09,586] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:09,587] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:12:09,591] jepsen worker 69 - jepsen.util 609 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,592] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,625] jepsen worker 70 - jepsen.util 700 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,626] jepsen worker 70 - jepsen.util 700 :invoke :write [23 46] INFO [2022-03-13 17:12:09,628] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,629] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,630] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,631] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:12:09,637] jepsen worker 21 - jepsen.util 741 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,637] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,637] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,638] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:12:09,639] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,639] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:09,646] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,647] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:09,691] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,691] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,692] jepsen worker 24 - jepsen.util 564 :invoke :write [8 36] INFO [2022-03-13 17:12:09,693] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:09,694] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,694] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,694] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,695] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,696] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:09,697] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:09,726] jepsen worker 52 - jepsen.util 592 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,727] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:09,736] jepsen worker 88 - jepsen.util 1978 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,737] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,742] jepsen worker 3 - jepsen.util 1893 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,743] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:12:09,756] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,756] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,756] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,756] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,757] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:12:09,757] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,758] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:09,758] jepsen worker 21 - jepsen.util 741 :invoke :write [7 6] INFO [2022-03-13 17:12:09,759] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,770] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,772] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,772] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,792] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.543859649122807}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:09,792] jepsen worker nemesis - couchbase.util Rebalance check count: 240 INFO [2022-03-13 17:12:09,792] jepsen worker nemesis - couchbase.util Rebalance stuck count: 152 INFO [2022-03-13 17:12:09,808] jepsen worker 78 - jepsen.util 528 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,808] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:09,833] jepsen worker 24 - jepsen.util 564 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,833] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,833] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,833] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,835] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:09,835] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:12:09,836] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:09,836] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:09,838] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,841] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:12:09,849] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,849] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,850] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,850] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:12:09,866] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,867] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:12:09,868] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,869] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,897] jepsen worker 44 - jepsen.util 584 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,897] jepsen worker 26 - jepsen.util 476 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,899] jepsen worker 26 - jepsen.util 476 :invoke :write [8 27] INFO [2022-03-13 17:12:09,899] jepsen worker 44 - jepsen.util 584 :invoke :write [14 28] INFO [2022-03-13 17:12:09,904] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,904] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,937] jepsen worker 21 - jepsen.util 741 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,938] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,938] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,938] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,938] jepsen worker 21 - jepsen.util 741 :invoke :write [7 22] INFO [2022-03-13 17:12:09,939] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:12:09,940] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:09,941] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:09,947] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,948] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:09,952] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:09,953] jepsen worker 66 - jepsen.util 1776 :ok :read [22 30] INFO [2022-03-13 17:12:09,954] jepsen worker 66 - jepsen.util 1776 :invoke :read [22 nil] INFO [2022-03-13 17:12:09,955] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:12:09,974] jepsen worker 15 - jepsen.util 1995 :ok :read [5 43] INFO [2022-03-13 17:12:09,975] jepsen worker 15 - jepsen.util 1995 :invoke :read [5 nil] INFO [2022-03-13 17:12:09,995] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:09,996] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:09,999] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,000] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,016] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,017] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:10,027] jepsen worker 68 - jepsen.util 2138 :ok :read [22 30] INFO [2022-03-13 17:12:10,027] jepsen worker 68 - jepsen.util 2138 :invoke :read [22 nil] INFO [2022-03-13 17:12:10,048] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:10,049] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,052] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:10,053] jepsen worker 60 - jepsen.util 600 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,053] jepsen worker 8 - jepsen.util 728 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,053] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 29] INFO [2022-03-13 17:12:10,054] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:10,055] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:10,096] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,096] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,096] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,098] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:12:10,098] jepsen worker 33 - jepsen.util 663 :invoke :write [11 49] INFO [2022-03-13 17:12:10,099] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:12:10,138] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,138] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,144] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,144] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,145] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,145] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,146] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,146] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:12:10,155] jepsen worker 21 - jepsen.util 741 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,156] jepsen worker 21 - jepsen.util 741 :invoke :write [7 9] INFO [2022-03-13 17:12:10,159] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,159] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,160] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:12:10,161] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 36] INFO [2022-03-13 17:12:10,173] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:10,175] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,201] jepsen worker 12 - jepsen.util 732 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,201] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:10,201] jepsen worker 53 - jepsen.util 503 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,201] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,202] jepsen worker 43 - jepsen.util 493 :invoke :write [14 20] INFO [2022-03-13 17:12:10,203] jepsen worker 12 - jepsen.util 732 :invoke :write [4 46] INFO [2022-03-13 17:12:10,204] jepsen worker 53 - jepsen.util 503 :invoke :write [17 3] INFO [2022-03-13 17:12:10,204] jepsen worker 7 - jepsen.util 637 :invoke :write [2 32] INFO [2022-03-13 17:12:10,217] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:10,218] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:12:10,222] jepsen worker 88 - jepsen.util 1978 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,222] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 17] INFO [2022-03-13 17:12:10,255] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,255] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,256] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:12:10,259] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:10,289] jepsen worker 3 - jepsen.util 1893 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,290] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,313] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,314] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:12:10,320] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,320] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:12:10,357] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,357] jepsen worker 23 - jepsen.util 563 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,358] jepsen worker 23 - jepsen.util 563 :invoke :write [7 49] INFO [2022-03-13 17:12:10,358] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:10,381] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,382] jepsen worker 24 - jepsen.util 564 :invoke :write [8 2] INFO [2022-03-13 17:12:10,387] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,388] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,406] jepsen worker 88 - jepsen.util 1978 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,407] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,440] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,440] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,441] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,441] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:12:10,452] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:10,452] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,453] jepsen worker 33 - jepsen.util 663 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,454] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:10,481] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,481] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,481] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,482] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,482] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,483] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,486] jepsen worker 77 - jepsen.util 707 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,486] jepsen worker 6 - jepsen.util 636 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,486] jepsen worker 77 - jepsen.util 707 :invoke :write [25 31] INFO [2022-03-13 17:12:10,487] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:10,487] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,488] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,488] jepsen worker 61 - jepsen.util 601 :invoke :write [20 20] INFO [2022-03-13 17:12:10,488] jepsen worker 62 - jepsen.util 512 :invoke :write [20 14] INFO [2022-03-13 17:12:10,494] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,495] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,497] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,497] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,499] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,499] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,500] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,500] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,501] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,501] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,502] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,502] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,503] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,504] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:12:10,504] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,504] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,508] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,508] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,509] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:10,510] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,510] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:12:10,510] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:12:10,515] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,516] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:10,562] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:10,563] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:12:10,590] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,591] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,595] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,595] jepsen worker 21 - jepsen.util 741 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,596] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:12:10,597] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:10,625] jepsen worker 51 - jepsen.util 591 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,625] jepsen worker 14 - jepsen.util 284 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,626] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:10,626] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:10,630] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:10,630] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,634] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,635] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:10,647] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,648] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,649] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,650] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:12:10,706] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,706] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,707] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 36] INFO [2022-03-13 17:12:10,707] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,711] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,712] jepsen worker 71 - jepsen.util 521 :invoke :write [23 3] INFO [2022-03-13 17:12:10,738] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:10,739] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,739] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 39] INFO [2022-03-13 17:12:10,739] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,744] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,744] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:12:10,768] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,769] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,787] jepsen worker 82 - jepsen.util 1972 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,788] jepsen worker 81 - jepsen.util 2331 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,788] jepsen worker 48 - jepsen.util 2028 :ok :read [16 :nil] INFO [2022-03-13 17:12:10,789] jepsen worker 81 - jepsen.util 2331 :invoke :read [27 nil] INFO [2022-03-13 17:12:10,789] jepsen worker 82 - jepsen.util 1972 :invoke :write [27 48] INFO [2022-03-13 17:12:10,790] jepsen worker 48 - jepsen.util 2028 :invoke :read [16 nil] INFO [2022-03-13 17:12:10,799] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.543859649122807}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:10,799] jepsen worker nemesis - couchbase.util Rebalance check count: 241 INFO [2022-03-13 17:12:10,799] jepsen worker nemesis - couchbase.util Rebalance stuck count: 153 INFO [2022-03-13 17:12:10,821] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,821] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,823] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,824] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,824] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,824] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,824] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:10,825] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,828] jepsen worker 24 - jepsen.util 564 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,828] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,828] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:12:10,829] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,831] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:10,832] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:10,847] jepsen worker 44 - jepsen.util 584 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,847] jepsen worker 26 - jepsen.util 476 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,847] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,849] jepsen worker 26 - jepsen.util 476 :invoke :write [8 48] INFO [2022-03-13 17:12:10,849] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:10,850] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:10,864] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,864] jepsen worker 69 - jepsen.util 609 :invoke :write [23 25] INFO [2022-03-13 17:12:10,896] jepsen worker 71 - jepsen.util 521 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,896] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,900] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:10,901] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:12:10,904] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,904] jepsen worker 36 - jepsen.util 2106 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,904] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,905] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,905] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,906] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:10,925] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:10,925] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,942] jepsen worker 89 - jepsen.util 2429 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,942] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,942] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,942] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,942] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,943] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:12:10,943] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,944] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,945] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,945] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,946] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,946] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 25] INFO [2022-03-13 17:12:10,950] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,952] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:10,954] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,956] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:12:10,956] jepsen worker 86 - jepsen.util 1706 :info :write [28 37] :RequestTimeoutException INFO [2022-03-13 17:12:10,957] jepsen worker 86 - jepsen.util 1796 :invoke :write [28 43] INFO [2022-03-13 17:12:10,967] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,967] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:12:10,978] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:10,978] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:10,979] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:10,979] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 13] INFO [2022-03-13 17:12:10,991] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:10,993] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:12:11,007] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:11,007] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,010] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:11,010] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,022] jepsen worker 80 - jepsen.util 710 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,023] jepsen worker 80 - jepsen.util 710 :invoke :write [26 43] INFO [2022-03-13 17:12:11,027] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,027] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,027] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,027] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:11,028] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:11,028] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:11,078] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,079] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,080] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:12:11,081] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,091] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,091] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,094] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,095] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,095] jepsen worker 2 - jepsen.util 2342 :ok :read [0 41] INFO [2022-03-13 17:12:11,096] jepsen worker 2 - jepsen.util 2342 :invoke :read [0 nil] INFO [2022-03-13 17:12:11,102] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,103] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:12:11,105] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:11,106] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,108] jepsen worker 81 - jepsen.util 2331 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,108] jepsen worker 48 - jepsen.util 2028 :ok :read [16 :nil] INFO [2022-03-13 17:12:11,108] jepsen worker 81 - jepsen.util 2331 :invoke :read [27 nil] INFO [2022-03-13 17:12:11,109] jepsen worker 48 - jepsen.util 2028 :invoke :read [16 nil] INFO [2022-03-13 17:12:11,121] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,122] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,122] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,123] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:12:11,123] jepsen worker 16 - jepsen.util 1906 :ok :read [5 43] INFO [2022-03-13 17:12:11,124] jepsen worker 16 - jepsen.util 1906 :invoke :read [5 nil] INFO [2022-03-13 17:12:11,127] jepsen worker 53 - jepsen.util 503 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,128] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:11,151] jepsen worker 69 - jepsen.util 609 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,152] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,152] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,152] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,152] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,153] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:12:11,153] jepsen worker 21 - jepsen.util 741 :invoke :write [7 37] INFO [2022-03-13 17:12:11,154] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:12:11,157] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:11,159] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:12:11,171] jepsen worker 36 - jepsen.util 2106 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,171] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 42] INFO [2022-03-13 17:12:11,182] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:11,183] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,194] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:11,195] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:12:11,197] jepsen worker 88 - jepsen.util 1978 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,198] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,225] jepsen worker 12 - jepsen.util 732 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,226] jepsen worker 12 - jepsen.util 732 :invoke :write [4 41] INFO [2022-03-13 17:12:11,240] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:11,240] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:12:11,259] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,259] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,260] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 38] INFO [2022-03-13 17:12:11,261] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:11,272] jepsen worker 81 - jepsen.util 2331 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,272] jepsen worker 48 - jepsen.util 2028 :ok :read [16 :nil] INFO [2022-03-13 17:12:11,273] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,273] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,273] jepsen worker 48 - jepsen.util 2028 :invoke :write [16 12] INFO [2022-03-13 17:12:11,276] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:11,277] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:12:11,278] jepsen worker 81 - jepsen.util 2331 :invoke :read [27 nil] INFO [2022-03-13 17:12:11,278] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,280] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:12:11,288] jepsen worker 2 - jepsen.util 2342 :ok :read [0 41] INFO [2022-03-13 17:12:11,289] jepsen worker 2 - jepsen.util 2342 :invoke :read [0 nil] INFO [2022-03-13 17:12:11,292] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:11,293] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,293] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,294] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,296] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,297] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,302] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,302] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:11,342] jepsen worker 16 - jepsen.util 1906 :ok :read [5 43] INFO [2022-03-13 17:12:11,342] jepsen worker 16 - jepsen.util 1906 :invoke :read [5 nil] INFO [2022-03-13 17:12:11,358] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,359] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,369] jepsen worker 7 - jepsen.util 637 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,369] jepsen worker 23 - jepsen.util 563 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,373] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:12:11,375] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:11,393] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:11,395] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:11,404] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,404] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:12:11,425] jepsen worker 3 - jepsen.util 1893 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,426] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:12:11,441] jepsen worker 41 - jepsen.util 1751 :info :write [13 39] :RequestTimeoutException INFO [2022-03-13 17:12:11,441] jepsen worker 16 - jepsen.util 1906 :ok :read [5 43] INFO [2022-03-13 17:12:11,441] jepsen worker 41 - jepsen.util 1841 :invoke :write [13 8] INFO [2022-03-13 17:12:11,442] jepsen worker 16 - jepsen.util 1906 :invoke :read [5 nil] INFO [2022-03-13 17:12:11,446] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,447] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,448] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,449] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,450] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,451] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,452] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,452] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:12:11,470] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,471] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,504] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:11,505] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,514] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,515] jepsen worker 16 - jepsen.util 1906 :ok :read [5 43] INFO [2022-03-13 17:12:11,515] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 21] INFO [2022-03-13 17:12:11,515] jepsen worker 16 - jepsen.util 1906 :invoke :read [5 nil] INFO [2022-03-13 17:12:11,516] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,517] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,521] jepsen worker 77 - jepsen.util 707 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,521] jepsen worker 43 - jepsen.util 493 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,521] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,521] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:11,521] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,521] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:11,521] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,521] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:11,522] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:11,523] jepsen worker 51 - jepsen.util 591 :invoke :write [17 19] INFO [2022-03-13 17:12:11,523] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:11,524] jepsen worker 44 - jepsen.util 584 :invoke :write [14 36] INFO [2022-03-13 17:12:11,524] jepsen worker 60 - jepsen.util 600 :invoke :write [20 25] INFO [2022-03-13 17:12:11,526] jepsen worker 43 - jepsen.util 493 :invoke :write [14 49] INFO [2022-03-13 17:12:11,532] jepsen worker 21 - jepsen.util 741 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,532] jepsen worker 21 - jepsen.util 741 :invoke :write [7 10] INFO [2022-03-13 17:12:11,542] jepsen worker 36 - jepsen.util 2106 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,543] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,553] jepsen worker 81 - jepsen.util 2331 :ok :read [27 4] INFO [2022-03-13 17:12:11,556] jepsen worker 81 - jepsen.util 2331 :invoke :read [27 nil] INFO [2022-03-13 17:12:11,563] jepsen worker 2 - jepsen.util 2342 :ok :read [0 41] INFO [2022-03-13 17:12:11,564] jepsen worker 2 - jepsen.util 2342 :invoke :read [0 nil] INFO [2022-03-13 17:12:11,565] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,566] jepsen worker 71 - jepsen.util 521 :invoke :write [23 27] INFO [2022-03-13 17:12:11,578] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:11,579] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:12:11,580] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,581] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,582] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,583] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:12:11,601] jepsen worker 16 - jepsen.util 1906 :ok :read [5 43] INFO [2022-03-13 17:12:11,601] jepsen worker 16 - jepsen.util 1906 :invoke :read [5 nil] INFO [2022-03-13 17:12:11,616] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,617] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,640] jepsen worker 81 - jepsen.util 2331 :ok :read [27 4] INFO [2022-03-13 17:12:11,641] jepsen worker 81 - jepsen.util 2331 :invoke :write [27 17] INFO [2022-03-13 17:12:11,645] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,645] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,646] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,646] jepsen worker 88 - jepsen.util 1978 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,647] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:11,647] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,647] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,648] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:11,661] jepsen worker 20 - jepsen.util 2450 :ok :read [6 16] INFO [2022-03-13 17:12:11,662] jepsen worker 20 - jepsen.util 2450 :invoke :read [6 nil] INFO [2022-03-13 17:12:11,673] jepsen worker 2 - jepsen.util 2342 :ok :read [0 41] INFO [2022-03-13 17:12:11,674] jepsen worker 2 - jepsen.util 2342 :invoke :read [0 nil] INFO [2022-03-13 17:12:11,694] jepsen worker 16 - jepsen.util 1906 :ok :read [5 43] INFO [2022-03-13 17:12:11,695] jepsen worker 16 - jepsen.util 1906 :invoke :write [5 42] INFO [2022-03-13 17:12:11,717] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,717] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,718] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,718] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 39] INFO [2022-03-13 17:12:11,719] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,719] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:12:11,727] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,727] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:12:11,736] jepsen worker 45 - jepsen.util 2385 :ok :read [15 48] INFO [2022-03-13 17:12:11,736] jepsen worker 45 - jepsen.util 2385 :invoke :write [15 17] INFO [2022-03-13 17:12:11,742] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,749] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:11,750] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,754] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:12:11,773] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,773] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,774] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,774] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,775] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:12:11,777] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,778] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:12:11,779] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:12:11,784] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:11,785] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:12:11,810] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,810] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:11,815] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,816] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:11,825] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5497076023391814}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:11,825] jepsen worker nemesis - couchbase.util Rebalance check count: 242 INFO [2022-03-13 17:12:11,825] jepsen worker nemesis - couchbase.util Rebalance stuck count: 153 INFO [2022-03-13 17:12:11,829] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,830] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,843] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:11,844] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:12:11,849] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,849] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,850] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:11,850] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:11,875] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,876] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,892] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,892] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:12:11,921] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,921] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:12:11,927] jepsen worker 61 - jepsen.util 601 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,927] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,927] jepsen worker 12 - jepsen.util 732 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,927] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,928] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:11,929] jepsen worker 12 - jepsen.util 732 :invoke :write [4 10] INFO [2022-03-13 17:12:11,929] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] INFO [2022-03-13 17:12:11,930] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:11,946] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,947] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:11,956] jepsen worker 36 - jepsen.util 2106 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,957] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:11,980] jepsen worker 21 - jepsen.util 741 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,981] jepsen worker 21 - jepsen.util 741 :invoke :write [7 2] INFO [2022-03-13 17:12:11,990] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:11,992] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 23] INFO [2022-03-13 17:12:12,013] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,013] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,039] jepsen worker 71 - jepsen.util 521 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,040] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,040] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,040] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,041] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,041] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:12,041] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,041] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,042] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:12,043] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:12,043] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,044] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:12,044] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,045] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:12:12,045] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:12:12,046] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,047] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,047] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,048] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,054] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,055] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,056] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,057] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,058] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,059] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,060] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,063] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,065] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:12:12,077] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,079] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:12:12,083] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,083] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,084] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,092] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,094] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,096] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:12,101] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:12:12,102] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 13] INFO [2022-03-13 17:12:12,103] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,104] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,105] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:12:12,106] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 8] INFO [2022-03-13 17:12:12,115] jepsen worker 88 - jepsen.util 1978 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,115] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,116] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:12,117] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,141] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:12,141] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:12:12,155] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,156] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,161] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,161] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:12:12,172] jepsen worker 62 - jepsen.util 512 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,172] jepsen worker 51 - jepsen.util 591 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,173] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:12,173] jepsen worker 51 - jepsen.util 591 :invoke :write [17 43] INFO [2022-03-13 17:12:12,191] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:12,192] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,209] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,209] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,214] jepsen worker 36 - jepsen.util 2106 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,215] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 42] INFO [2022-03-13 17:12:12,219] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:12,222] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,230] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 10] INFO [2022-03-13 17:12:12,231] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:12:12,282] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,282] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,285] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,286] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,287] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,287] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,293] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,294] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,295] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,296] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:12,300] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,300] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:12:12,305] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,306] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:12:12,320] jepsen worker 26 - jepsen.util 476 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,320] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,321] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:12,321] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:12,321] jepsen worker 44 - jepsen.util 584 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,322] jepsen worker 44 - jepsen.util 584 :invoke :write [14 28] INFO [2022-03-13 17:12:12,357] jepsen worker 34 - jepsen.util 754 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,358] jepsen worker 34 - jepsen.util 754 :invoke :write [11 3] INFO [2022-03-13 17:12:12,359] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,360] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:12:12,381] jepsen worker 88 - jepsen.util 1978 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,382] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,385] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,385] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,387] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:12:12,390] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:12:12,415] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:12,416] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,444] jepsen worker 37 - jepsen.util 2017 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,445] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,465] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:12,465] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,478] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,479] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,500] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,501] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:12:12,515] jepsen worker 13 - jepsen.util 643 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,516] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:12,531] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:12,532] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:12:12,534] jepsen worker 56 - jepsen.util 2036 :info :write [18 41] :RequestTimeoutException INFO [2022-03-13 17:12:12,534] jepsen worker 36 - jepsen.util 2106 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,534] jepsen worker 56 - jepsen.util 2126 :invoke :write [18 4] INFO [2022-03-13 17:12:12,535] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 49] INFO [2022-03-13 17:12:12,570] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,571] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:12:12,584] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,584] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:12:12,596] jepsen worker 49 - jepsen.util 1759 :info :write [16 42] :RequestTimeoutException INFO [2022-03-13 17:12:12,597] jepsen worker 49 - jepsen.util 1849 :invoke :write [16 11] INFO [2022-03-13 17:12:12,623] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,623] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,625] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,625] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,629] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,629] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,630] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,631] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,633] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,634] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:12:12,653] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,653] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,655] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:12:12,656] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,658] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,658] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,659] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,660] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,688] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,688] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:12:12,696] jepsen worker 21 - jepsen.util 741 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,696] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,697] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,697] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,697] jepsen worker 21 - jepsen.util 741 :invoke :write [7 33] INFO [2022-03-13 17:12:12,698] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:12,699] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:12:12,713] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:12,731] jepsen worker 70 - jepsen.util 700 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,731] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,731] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,732] jepsen worker 52 - jepsen.util 592 :invoke :write [17 33] INFO [2022-03-13 17:12:12,732] jepsen worker 78 - jepsen.util 528 :invoke :write [26 5] INFO [2022-03-13 17:12:12,734] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:12,755] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,756] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,756] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,756] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:12,761] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,762] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,763] jepsen worker 36 - jepsen.util 2106 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,763] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,764] jepsen worker 72 - jepsen.util 1962 :info :write [24 23] :RequestTimeoutException INFO [2022-03-13 17:12:12,764] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 8] INFO [2022-03-13 17:12:12,765] jepsen worker 72 - jepsen.util 2052 :invoke :write [24 16] INFO [2022-03-13 17:12:12,766] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,766] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:12,767] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,767] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,768] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:12:12,785] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,790] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:12:12,793] jepsen worker 80 - jepsen.util 710 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,795] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,795] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,795] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:12,795] jepsen worker 60 - jepsen.util 600 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,795] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,806] jepsen worker 42 - jepsen.util 402 :invoke :write [14 8] INFO [2022-03-13 17:12:12,807] jepsen worker 77 - jepsen.util 707 :invoke :write [25 8] INFO [2022-03-13 17:12:12,808] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:12,808] jepsen worker 7 - jepsen.util 637 :invoke :write [2 12] INFO [2022-03-13 17:12:12,808] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:12,809] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:12,811] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:12,812] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,823] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,824] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,833] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5497076023391814}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:12,834] jepsen worker nemesis - couchbase.util Rebalance check count: 243 INFO [2022-03-13 17:12:12,834] jepsen worker nemesis - couchbase.util Rebalance stuck count: 154 INFO [2022-03-13 17:12:12,834] jepsen worker 55 - jepsen.util 1855 :info :write [18 15] :RequestTimeoutException INFO [2022-03-13 17:12:12,835] jepsen worker 55 - jepsen.util 1945 :invoke :write [18 49] INFO [2022-03-13 17:12:12,845] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,846] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 12] INFO [2022-03-13 17:12:12,874] jepsen worker 19 - jepsen.util 1909 :info :write [6 18] :RequestTimeoutException INFO [2022-03-13 17:12:12,875] jepsen worker 19 - jepsen.util 1999 :invoke :write [6 31] INFO [2022-03-13 17:12:12,879] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:12,882] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,900] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,936] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:12:12,936] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:12:12,923] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,945] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:12:12,924] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:12,946] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:12,947] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 16] INFO [2022-03-13 17:12:12,962] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,963] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:12:12,980] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,980] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:12,981] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:12,982] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:13,012] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:13,013] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:12:13,037] jepsen worker 88 - jepsen.util 1978 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,038] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 19] INFO [2022-03-13 17:12:13,040] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,040] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:13,098] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,098] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,098] jepsen worker 36 - jepsen.util 2106 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,099] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:12:13,099] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:13,100] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 42] INFO [2022-03-13 17:12:13,104] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,104] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,104] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,105] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:13,105] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:12:13,106] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:13,107] jepsen worker 34 - jepsen.util 754 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,108] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,108] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,108] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:13,108] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:13,109] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:13,115] jepsen worker 43 - jepsen.util 493 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,116] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,116] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,116] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,116] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,116] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,116] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,116] jepsen worker 12 - jepsen.util 732 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,117] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,117] jepsen worker 31 - jepsen.util 1921 :info :write [10 6] :RequestTimeoutException INFO [2022-03-13 17:12:13,118] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:13,121] jepsen worker 43 - jepsen.util 493 :invoke :write [14 24] INFO [2022-03-13 17:12:13,122] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:13,123] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,123] jepsen worker 62 - jepsen.util 512 :invoke :write [20 4] INFO [2022-03-13 17:12:13,125] jepsen worker 31 - jepsen.util 2011 :invoke :read [10 nil] INFO [2022-03-13 17:12:13,126] jepsen worker 6 - jepsen.util 636 :invoke :write [2 44] INFO [2022-03-13 17:12:13,127] jepsen worker 31 - jepsen.util 2011 :ok :read [10 :nil] INFO [2022-03-13 17:12:13,131] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:12:13,131] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,135] jepsen worker 61 - jepsen.util 601 :invoke :write [20 25] INFO [2022-03-13 17:12:13,137] jepsen worker 14 - jepsen.util 284 :invoke :write [4 47] INFO [2022-03-13 17:12:13,137] jepsen worker 31 - jepsen.util 2011 :invoke :write [10 41] INFO [2022-03-13 17:12:13,138] jepsen worker 8 - jepsen.util 728 :invoke :write [2 35] INFO [2022-03-13 17:12:13,145] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:13,157] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:13,161] jepsen worker 1 - jepsen.util 1711 :info :write [0 25] :RequestTimeoutException INFO [2022-03-13 17:12:13,162] jepsen worker 1 - jepsen.util 1801 :invoke :write [0 31] INFO [2022-03-13 17:12:13,166] jepsen worker 21 - jepsen.util 741 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,167] jepsen worker 89 - jepsen.util 2429 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,167] jepsen worker 21 - jepsen.util 741 :invoke :write [7 8] INFO [2022-03-13 17:12:13,167] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:13,172] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,174] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:13,201] jepsen worker 88 - jepsen.util 1978 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,201] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,207] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:13,208] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:12:13,216] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:13,217] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:13,221] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,221] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,222] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:12:13,222] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:12:13,232] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:13,233] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:13,264] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:13,265] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 8] INFO [2022-03-13 17:12:13,315] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,315] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,315] jepsen worker 23 - jepsen.util 563 :invoke :write [7 25] INFO [2022-03-13 17:12:13,316] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,316] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:12:13,316] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:13,319] jepsen worker 78 - jepsen.util 528 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,320] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,346] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,346] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:12:13,359] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,359] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,359] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,359] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,360] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,359] jepsen worker 52 - jepsen.util 592 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,360] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,361] jepsen worker 52 - jepsen.util 592 :invoke :write [17 10] INFO [2022-03-13 17:12:13,361] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,362] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:13,362] jepsen worker 13 - jepsen.util 643 :invoke :write [4 37] INFO [2022-03-13 17:12:13,363] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,363] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,364] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:13,364] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,365] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:12:13,365] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,365] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:12:13,366] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:13,367] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,367] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,367] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:13,368] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,369] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:13,396] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,396] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,397] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:12:13,397] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:12:13,422] jepsen worker 36 - jepsen.util 2106 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,422] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,423] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:12:13,424] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:13,424] jepsen worker 28 - jepsen.util 2188 :info :write [9 20] :RequestTimeoutException INFO [2022-03-13 17:12:13,424] jepsen worker 28 - jepsen.util 2278 :invoke :write [9 42] INFO [2022-03-13 17:12:13,425] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,426] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:13,438] jepsen worker 88 - jepsen.util 1978 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,439] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:12:13,503] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,503] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:13,512] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,512] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,512] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,512] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,512] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:13,513] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,514] jepsen worker 7 - jepsen.util 637 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,514] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,515] jepsen worker 34 - jepsen.util 754 :invoke :write [11 28] INFO [2022-03-13 17:12:13,515] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:13,523] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,524] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:12:13,547] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,547] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,548] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 29] INFO [2022-03-13 17:12:13,548] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:12:13,593] jepsen worker 42 - jepsen.util 402 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,593] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,593] jepsen worker 51 - jepsen.util 591 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,593] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,593] jepsen worker 26 - jepsen.util 476 :invoke :write [8 12] INFO [2022-03-13 17:12:13,594] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:13,595] jepsen worker 53 - jepsen.util 503 :invoke :write [17 2] INFO [2022-03-13 17:12:13,595] jepsen worker 51 - jepsen.util 591 :invoke :write [17 45] INFO [2022-03-13 17:12:13,603] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,604] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:12:13,606] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,607] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:13,610] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,610] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:13,612] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,613] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:12:13,626] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:13,627] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:12:13,631] jepsen worker 9 - jepsen.util 1899 :info :write [3 26] :RequestTimeoutException INFO [2022-03-13 17:12:13,634] jepsen worker 9 - jepsen.util 1989 :invoke :read [3 nil] INFO [2022-03-13 17:12:13,651] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,652] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:13,682] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,683] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 21] INFO [2022-03-13 17:12:13,690] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:13,690] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 9] INFO [2022-03-13 17:12:13,718] jepsen worker 13 - jepsen.util 643 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,718] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:13,725] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,725] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,726] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,726] jepsen worker 71 - jepsen.util 521 :invoke :write [23 1] INFO [2022-03-13 17:12:13,726] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:12:13,727] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:12:13,742] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,743] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:13,786] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,786] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,787] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:12:13,788] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:12:13,797] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,798] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:13,840] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5497076023391814}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:13,840] jepsen worker nemesis - couchbase.util Rebalance check count: 244 INFO [2022-03-13 17:12:13,840] jepsen worker nemesis - couchbase.util Rebalance stuck count: 155 INFO [2022-03-13 17:12:13,843] jepsen worker 36 - jepsen.util 2106 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,843] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:13,869] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,869] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:12:13,873] jepsen worker 88 - jepsen.util 1978 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,874] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 12] INFO [2022-03-13 17:12:13,922] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,922] jepsen worker 52 - jepsen.util 592 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,922] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,922] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,923] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,923] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:13,924] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:13,924] jepsen worker 52 - jepsen.util 592 :invoke :write [17 36] INFO [2022-03-13 17:12:13,932] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,932] jepsen worker 6 - jepsen.util 636 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,932] jepsen worker 44 - jepsen.util 584 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,932] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,932] jepsen worker 69 - jepsen.util 609 :invoke :write [23 35] INFO [2022-03-13 17:12:13,933] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:13,934] jepsen worker 60 - jepsen.util 600 :invoke :write [20 44] INFO [2022-03-13 17:12:13,935] jepsen worker 44 - jepsen.util 584 :invoke :write [14 46] INFO [2022-03-13 17:12:13,951] jepsen worker 79 - jepsen.util 889 :info :write [26 39] :SyncWriteAmbiguous INFO [2022-03-13 17:12:13,952] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,952] jepsen worker 79 - jepsen.util 979 :invoke :write [26 37] INFO [2022-03-13 17:12:13,952] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:12:13,956] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,957] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:13,960] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,961] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:12:13,992] jepsen worker 37 - jepsen.util 2017 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,993] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:13,999] jepsen worker 34 - jepsen.util 754 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:13,999] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,000] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,000] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,000] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,002] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:14,003] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:12:14,004] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:12:14,005] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:14,006] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:14,037] jepsen worker 65 - jepsen.util 2405 :info :write [21 3] :RequestTimeoutException INFO [2022-03-13 17:12:14,038] jepsen worker 65 - jepsen.util 2495 :invoke :write [21 23] INFO [2022-03-13 17:12:14,041] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,041] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,042] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:12:14,042] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:14,054] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,055] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:14,065] jepsen worker 88 - jepsen.util 1978 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,066] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 0] INFO [2022-03-13 17:12:14,070] jepsen worker 35 - jepsen.util 845 :info :write [11 9] :RequestTimeoutException INFO [2022-03-13 17:12:14,071] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:12:14,105] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:14,106] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:14,116] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,116] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,116] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:14,117] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:14,120] jepsen worker 76 - jepsen.util 616 :info :write [25 47] :RequestTimeoutException INFO [2022-03-13 17:12:14,121] jepsen worker 76 - jepsen.util 706 :invoke :write [25 3] INFO [2022-03-13 17:12:14,145] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,145] jepsen worker 21 - jepsen.util 741 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,146] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:14,147] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:12:14,308] jepsen worker 62 - jepsen.util 512 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,308] jepsen worker 77 - jepsen.util 707 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,308] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,308] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,309] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:14,310] jepsen worker 62 - jepsen.util 512 :invoke :write [20 36] INFO [2022-03-13 17:12:14,310] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:14,310] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:12:14,359] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,359] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,359] jepsen worker 52 - jepsen.util 592 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,359] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,360] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:14,360] jepsen worker 80 - jepsen.util 710 :invoke :write [26 16] INFO [2022-03-13 17:12:14,360] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:14,361] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:14,371] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,372] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:14,423] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,424] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:12:14,458] jepsen worker 23 - jepsen.util 563 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,459] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:14,467] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,468] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:14,469] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,470] jepsen worker 22 - jepsen.util 652 :invoke :write [7 8] INFO [2022-03-13 17:12:14,473] jepsen worker 71 - jepsen.util 521 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,474] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:14,511] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,512] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:14,584] jepsen worker 43 - jepsen.util 493 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,584] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:14,845] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5497076023391814}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:14,845] jepsen worker nemesis - couchbase.util Rebalance check count: 245 INFO [2022-03-13 17:12:14,845] jepsen worker nemesis - couchbase.util Rebalance stuck count: 156 INFO [2022-03-13 17:12:14,959] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,959] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:12:14,976] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:14,976] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:12:15,021] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,021] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,022] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,023] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,023] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,024] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,024] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:12:15,025] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:12:15,027] jepsen worker 51 - jepsen.util 591 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,028] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,027] jepsen worker 14 - jepsen.util 284 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,028] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:15,028] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:15,029] jepsen worker 7 - jepsen.util 637 :invoke :write [2 34] INFO [2022-03-13 17:12:15,030] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:15,031] jepsen worker 51 - jepsen.util 591 :invoke :write [17 19] INFO [2022-03-13 17:12:15,085] jepsen worker 22 - jepsen.util 652 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,086] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:12:15,097] jepsen worker 69 - jepsen.util 609 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,098] jepsen worker 69 - jepsen.util 609 :invoke :write [23 4] INFO [2022-03-13 17:12:15,156] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,157] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:12:15,171] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,172] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,172] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,172] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,173] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,173] jepsen worker 52 - jepsen.util 592 :invoke :write [17 21] INFO [2022-03-13 17:12:15,174] jepsen worker 13 - jepsen.util 643 :invoke :write [4 25] INFO [2022-03-13 17:12:15,174] jepsen worker 78 - jepsen.util 528 :invoke :write [26 5] INFO [2022-03-13 17:12:15,178] jepsen worker 27 - jepsen.util 2277 :info :write [9 30] :RequestTimeoutException INFO [2022-03-13 17:12:15,179] jepsen worker 27 - jepsen.util 2367 :invoke :read [9 nil] INFO [2022-03-13 17:12:15,186] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,186] jepsen worker 79 - jepsen.util 979 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,187] jepsen worker 79 - jepsen.util 979 :invoke :write [26 13] INFO [2022-03-13 17:12:15,188] jepsen worker 21 - jepsen.util 741 :invoke :write [7 15] INFO [2022-03-13 17:12:15,250] jepsen worker 60 - jepsen.util 600 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,250] jepsen worker 26 - jepsen.util 476 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,251] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:15,251] jepsen worker 26 - jepsen.util 476 :invoke :write [8 20] INFO [2022-03-13 17:12:15,257] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,257] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,257] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,258] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:15,259] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:12:15,259] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:12:15,281] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,281] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,310] jepsen worker 30 - jepsen.util 2010 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:12:15,311] jepsen worker 30 - jepsen.util 2100 :invoke :write [10 20] INFO [2022-03-13 17:12:15,337] jepsen worker 80 - jepsen.util 710 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,338] jepsen worker 80 - jepsen.util 710 :invoke :write [26 37] INFO [2022-03-13 17:12:15,343] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,344] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,401] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,401] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,402] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,402] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,403] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,403] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,404] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:12:15,404] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,405] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,405] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,406] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,407] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,408] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,408] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:12:15,409] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,410] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:15,411] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,411] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:15,413] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,413] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:15,414] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,415] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:12:15,421] jepsen worker 44 - jepsen.util 584 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,422] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,422] jepsen worker 53 - jepsen.util 503 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,422] jepsen worker 6 - jepsen.util 636 :invoke :write [2 2] INFO [2022-03-13 17:12:15,423] jepsen worker 44 - jepsen.util 584 :invoke :write [14 23] INFO [2022-03-13 17:12:15,430] jepsen worker 53 - jepsen.util 503 :invoke :write [17 34] INFO [2022-03-13 17:12:15,430] jepsen worker 69 - jepsen.util 609 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,430] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:15,437] jepsen worker 74 - jepsen.util 2504 :info :write [24 5] :RequestTimeoutException INFO [2022-03-13 17:12:15,438] jepsen worker 74 - jepsen.util 2594 :invoke :read [24 nil] INFO [2022-03-13 17:12:15,518] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,518] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,519] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:12:15,519] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,520] jepsen worker 21 - jepsen.util 741 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,521] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:15,527] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,527] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:15,538] jepsen worker 76 - jepsen.util 706 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,538] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,539] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,539] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,543] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:15,544] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:15,545] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:15,545] jepsen worker 77 - jepsen.util 707 :invoke :write [25 28] INFO [2022-03-13 17:12:15,611] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,611] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,612] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:12:15,612] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,620] jepsen worker 8 - jepsen.util 728 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,621] jepsen worker 8 - jepsen.util 728 :invoke :write [2 37] INFO [2022-03-13 17:12:15,627] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,628] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,628] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,628] jepsen worker 79 - jepsen.util 979 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,629] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:15,629] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:15,630] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,630] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,631] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,631] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,636] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,637] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:15,691] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,691] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,691] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,691] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 32] INFO [2022-03-13 17:12:15,692] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:12:15,693] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:15,695] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,696] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:12:15,699] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,699] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:15,709] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,710] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 25] INFO [2022-03-13 17:12:15,719] jepsen worker 78 - jepsen.util 528 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,719] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,720] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:15,720] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:15,746] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:15,747] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:15,772] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,772] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,772] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,772] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,773] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:12:15,773] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,773] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,774] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:15,775] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,775] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:12:15,776] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:12:15,777] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:12:15,778] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:15,778] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:15,780] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,781] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:15,782] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,783] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:12:15,792] jepsen worker 88 - jepsen.util 1978 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,793] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:12:15,850] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5497076023391814}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:15,850] jepsen worker nemesis - couchbase.util Rebalance check count: 246 INFO [2022-03-13 17:12:15,851] jepsen worker nemesis - couchbase.util Rebalance stuck count: 157 INFO [2022-03-13 17:12:15,901] jepsen worker 37 - jepsen.util 2017 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,902] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:15,920] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:15,921] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 22] INFO [2022-03-13 17:12:15,942] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,943] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:15,944] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,944] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:15,945] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,946] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:12:15,984] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,985] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:12:15,986] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:15,987] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] WARN [2022-03-13 17:12:15,992] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19799836,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19799836,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9049045,"encode_duration_us":19391,"operation_id":"0x598e","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005163},{"operation_name":"upsert","last_dispatch_duration_us":8811562,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":10130098,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":528181,"encode_duration_us":20142,"operation_id":"0x5b17","timeout_ms":10000,"last_server_duration_us":83268,"total_duration_us":10008897},{"operation_name":"upsert","last_dispatch_duration_us":16622949,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17098326,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9262487,"encode_duration_us":17838,"operation_id":"0x59fd","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10005718},{"operation_name":"upsert","last_dispatch_duration_us":10014921,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10014921,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":9171517,"encode_duration_us":17575,"operation_id":"0x5f79","timeout_ms":10000,"last_server_duration_us":9171517,"total_duration_us":10010352}],"total_count":4}} WARN [2022-03-13 17:12:15,992] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11694803,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11694803,"total_server_duration_us":8393,"operation_id":"0x5c13","timeout_ms":30000,"last_server_duration_us":8393,"total_duration_us":11697616},{"operation_name":"upsert","last_dispatch_duration_us":10014921,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":10014921,"total_server_duration_us":9171517,"encode_duration_us":17575,"operation_id":"0x5f79","timeout_ms":10000,"last_server_duration_us":9171517,"total_duration_us":10010352},{"operation_name":"upsert","last_dispatch_duration_us":34766,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":5248264,"total_server_duration_us":444720,"encode_duration_us":20405,"operation_id":"0x5ff7","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10007855},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":22912,"operation_id":"0x5de8","timeout_ms":10000,"total_duration_us":10005726},{"operation_name":"upsert","last_dispatch_duration_us":34278,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6783663,"total_server_duration_us":362234,"encode_duration_us":19132,"operation_id":"0x5f38","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10004610},{"operation_name":"upsert","last_dispatch_duration_us":9832779,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9832779,"total_server_duration_us":9787759,"encode_duration_us":17414,"operation_id":"0x5f78","timeout_ms":10000,"last_server_duration_us":9787759,"total_duration_us":9836782},{"operation_name":"upsert","last_dispatch_duration_us":9127685,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9385317,"total_server_duration_us":9238442,"encode_duration_us":17479,"operation_id":"0x5ca4","timeout_ms":10000,"last_server_duration_us":9100079,"total_duration_us":9436856},{"operation_name":"get","last_dispatch_duration_us":9084958,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9084958,"total_server_duration_us":10692,"operation_id":"0x5d38","timeout_ms":30000,"last_server_duration_us":10692,"total_duration_us":9087659},{"operation_name":"upsert","last_dispatch_duration_us":8465311,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":8465311,"total_server_duration_us":165497,"encode_duration_us":18913,"operation_id":"0x6041","timeout_ms":10000,"last_server_duration_us":165497,"total_duration_us":8469229},{"operation_name":"get","last_dispatch_duration_us":8319774,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":8319774,"total_server_duration_us":6678,"operation_id":"0x5d4a","timeout_ms":30000,"last_server_duration_us":6678,"total_duration_us":8320148}],"total_count":13}} INFO [2022-03-13 17:12:16,104] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,105] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:16,139] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:16,139] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:16,149] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,149] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,312] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,312] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:12:16,314] jepsen worker 36 - jepsen.util 2106 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,314] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,315] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 19] INFO [2022-03-13 17:12:16,315] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:16,339] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,340] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,373] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:16,373] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:12:16,384] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,384] jepsen worker 75 - jepsen.util 435 :invoke :write [25 11] INFO [2022-03-13 17:12:16,396] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,396] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:12:16,412] jepsen worker 52 - jepsen.util 592 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,412] jepsen worker 52 - jepsen.util 592 :invoke :write [17 48] INFO [2022-03-13 17:12:16,454] jepsen worker 13 - jepsen.util 643 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,454] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:16,474] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,475] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:16,476] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,477] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:12:16,483] jepsen worker 62 - jepsen.util 512 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,483] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:16,483] jepsen worker 51 - jepsen.util 591 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,483] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,484] jepsen worker 43 - jepsen.util 493 :invoke :write [14 15] INFO [2022-03-13 17:12:16,484] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:16,485] jepsen worker 51 - jepsen.util 591 :invoke :write [17 42] INFO [2022-03-13 17:12:16,485] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:16,507] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,507] jepsen worker 80 - jepsen.util 710 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,508] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:12:16,509] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:16,515] jepsen worker 89 - jepsen.util 2429 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,515] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:16,524] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,524] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:16,534] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,534] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,534] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:12:16,535] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,535] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,535] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,536] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,537] jepsen worker 79 - jepsen.util 979 :invoke :write [26 49] INFO [2022-03-13 17:12:16,539] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:16,540] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:12:16,540] jepsen worker 21 - jepsen.util 741 :invoke :write [7 22] INFO [2022-03-13 17:12:16,540] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:16,627] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,627] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,628] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:12:16,628] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:16,630] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,630] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:12:16,632] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,632] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,633] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:16,633] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:16,639] jepsen worker 75 - jepsen.util 435 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,639] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,639] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,641] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:12:16,641] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:16,641] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:16,693] jepsen worker 58 - jepsen.util 1858 :ok :read [19 20] INFO [2022-03-13 17:12:16,693] jepsen worker 58 - jepsen.util 1858 :invoke :write [19 3] INFO [2022-03-13 17:12:16,694] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,695] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:16,696] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,696] jepsen worker 22 - jepsen.util 652 :invoke :write [7 27] INFO [2022-03-13 17:12:16,711] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,711] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,711] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,711] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,711] jepsen worker 61 - jepsen.util 601 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,711] jepsen worker 44 - jepsen.util 584 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,711] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:16,712] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:16,713] jepsen worker 23 - jepsen.util 563 :invoke :write [7 12] INFO [2022-03-13 17:12:16,714] jepsen worker 44 - jepsen.util 584 :invoke :write [14 3] INFO [2022-03-13 17:12:16,714] jepsen worker 61 - jepsen.util 601 :invoke :write [20 44] INFO [2022-03-13 17:12:16,715] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:16,717] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:16,718] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:16,720] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,722] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:16,733] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,734] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:16,742] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,743] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:16,744] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,744] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:12:16,756] jepsen worker 17 - jepsen.util 1997 :ok :read [5 43] INFO [2022-03-13 17:12:16,758] jepsen worker 17 - jepsen.util 1997 :invoke :read [5 nil] INFO [2022-03-13 17:12:16,772] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,772] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,772] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,775] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:12:16,775] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,775] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,783] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:16,783] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:16,785] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,785] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,785] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,786] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,787] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,787] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,787] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,788] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:12:16,788] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,789] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,790] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,790] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:12:16,810] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,810] jepsen worker 52 - jepsen.util 592 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,811] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,812] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:16,813] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:16,813] jepsen worker 13 - jepsen.util 643 :invoke :write [4 4] INFO [2022-03-13 17:12:16,823] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,826] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:16,839] jepsen worker 79 - jepsen.util 979 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,840] jepsen worker 79 - jepsen.util 979 :invoke :write [26 40] INFO [2022-03-13 17:12:16,848] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,848] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,848] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:12:16,849] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:12:16,856] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5555555555555556}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:16,856] jepsen worker nemesis - couchbase.util Rebalance check count: 247 INFO [2022-03-13 17:12:16,856] jepsen worker nemesis - couchbase.util Rebalance stuck count: 157 INFO [2022-03-13 17:12:16,896] jepsen worker 22 - jepsen.util 652 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,896] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:16,898] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,898] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:16,920] jepsen worker 17 - jepsen.util 1997 :ok :read [5 43] INFO [2022-03-13 17:12:16,921] jepsen worker 17 - jepsen.util 1997 :invoke :write [5 3] INFO [2022-03-13 17:12:16,947] jepsen worker 21 - jepsen.util 741 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,948] jepsen worker 21 - jepsen.util 741 :invoke :write [7 28] INFO [2022-03-13 17:12:16,951] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:16,951] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:12:16,963] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,963] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,963] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,965] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 8] INFO [2022-03-13 17:12:16,966] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:16,966] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:16,969] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,970] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:12:16,979] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,979] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,979] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,979] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,979] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,979] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,980] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:16,981] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:16,981] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:16,982] jepsen worker 7 - jepsen.util 637 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,982] jepsen worker 77 - jepsen.util 707 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,982] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,983] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:16,983] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:12:16,984] jepsen worker 52 - jepsen.util 592 :invoke :write [17 5] INFO [2022-03-13 17:12:16,984] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:16,985] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:16,986] jepsen worker 77 - jepsen.util 707 :invoke :write [25 45] INFO [2022-03-13 17:12:16,987] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,988] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:16,996] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:16,997] jepsen worker 22 - jepsen.util 652 :invoke :write [7 27] INFO [2022-03-13 17:12:17,006] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,007] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:12:17,032] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:17,033] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:12:17,067] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,067] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:17,075] jepsen worker 54 - jepsen.util 1764 :ok :read [18 21] INFO [2022-03-13 17:12:17,075] jepsen worker 54 - jepsen.util 1764 :invoke :read [18 nil] INFO [2022-03-13 17:12:17,080] jepsen worker 10 - jepsen.util 2080 :ok :read [3 25] INFO [2022-03-13 17:12:17,081] jepsen worker 10 - jepsen.util 2080 :invoke :read [3 nil] INFO [2022-03-13 17:12:17,086] jepsen worker 13 - jepsen.util 643 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,086] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,086] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,086] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,087] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,087] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,087] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,087] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,088] jepsen worker 80 - jepsen.util 710 :invoke :write [26 11] INFO [2022-03-13 17:12:17,089] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:17,090] jepsen worker 13 - jepsen.util 643 :invoke :write [4 37] INFO [2022-03-13 17:12:17,090] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:17,091] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:17,092] jepsen worker 24 - jepsen.util 564 :invoke :write [8 3] INFO [2022-03-13 17:12:17,093] jepsen worker 70 - jepsen.util 700 :invoke :write [23 6] INFO [2022-03-13 17:12:17,093] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:12:17,102] jepsen worker 11 - jepsen.util 2081 :ok :read [3 25] INFO [2022-03-13 17:12:17,102] jepsen worker 11 - jepsen.util 2081 :invoke :read [3 nil] INFO [2022-03-13 17:12:17,145] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,145] jepsen worker 51 - jepsen.util 591 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,145] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,146] jepsen worker 51 - jepsen.util 591 :invoke :write [17 45] INFO [2022-03-13 17:12:17,146] jepsen worker 76 - jepsen.util 706 :invoke :write [25 46] INFO [2022-03-13 17:12:17,147] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:17,161] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,161] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,162] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:17,163] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:12:17,164] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,164] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:17,166] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,167] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:12:17,188] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,188] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,189] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:12:17,189] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:17,192] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,193] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:12:17,198] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,199] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:12:17,209] jepsen worker 26 - jepsen.util 476 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,210] jepsen worker 26 - jepsen.util 476 :invoke :write [8 35] INFO [2022-03-13 17:12:17,229] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:17,229] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:12:17,320] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,320] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,321] jepsen worker 52 - jepsen.util 592 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,321] jepsen worker 34 - jepsen.util 754 :invoke :write [11 46] INFO [2022-03-13 17:12:17,321] jepsen worker 78 - jepsen.util 528 :invoke :write [26 12] INFO [2022-03-13 17:12:17,322] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:17,327] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,327] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:17,369] jepsen worker 22 - jepsen.util 652 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,369] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:12:17,371] jepsen worker 36 - jepsen.util 2106 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,371] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:17,375] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,376] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:12:17,399] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21786669,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21786669,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9041612,"encode_duration_us":17380,"operation_id":"0x58c1","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10004868},{"operation_name":"upsert","last_dispatch_duration_us":18010381,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26018542,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9075568,"encode_duration_us":22126,"operation_id":"0x56b3","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10006082}],"total_count":2}} WARN [2022-03-13 17:12:17,401] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10841034,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12652842,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9476093,"encode_duration_us":17619,"operation_id":"0x5aa6","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10009205},{"operation_name":"upsert","last_dispatch_duration_us":18014280,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26312980,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9182833,"encode_duration_us":21430,"operation_id":"0x57b6","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10010614}],"total_count":2}} INFO [2022-03-13 17:12:17,412] jepsen worker 79 - jepsen.util 979 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,412] jepsen worker 80 - jepsen.util 710 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,413] jepsen worker 80 - jepsen.util 710 :invoke :write [26 21] INFO [2022-03-13 17:12:17,413] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:17,445] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,446] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 6] INFO [2022-03-13 17:12:17,447] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,447] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,448] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:12:17,451] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:17,452] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,453] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:17,454] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,455] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:17,456] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,457] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:12:17,476] jepsen worker 13 - jepsen.util 643 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,479] jepsen worker 13 - jepsen.util 643 :invoke :write [4 24] INFO [2022-03-13 17:12:17,495] jepsen worker 23 - jepsen.util 563 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,495] jepsen worker 53 - jepsen.util 503 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,496] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:17,497] jepsen worker 53 - jepsen.util 503 :invoke :write [17 28] WARN [2022-03-13 17:12:17,501] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10060797,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":10060797,"total_server_duration_us":9840522,"encode_duration_us":19824,"operation_id":"0x5f75","timeout_ms":10000,"last_server_duration_us":9840522,"total_duration_us":10008329},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":17712,"operation_id":"0x5ee3","timeout_ms":10000,"total_duration_us":10007855},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":20729,"operation_id":"0x5cc7","timeout_ms":10000,"total_duration_us":10005737},{"operation_name":"get","last_dispatch_duration_us":9396458,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9396458,"total_server_duration_us":43,"operation_id":"0x5e59","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9398353},{"operation_name":"upsert","last_dispatch_duration_us":8850775,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":8850775,"total_server_duration_us":84574,"encode_duration_us":17456,"operation_id":"0x601d","timeout_ms":10000,"last_server_duration_us":84574,"total_duration_us":8855085},{"operation_name":"upsert","last_dispatch_duration_us":1699401,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1699401,"total_server_duration_us":1630301,"encode_duration_us":25733,"operation_id":"0x64f0","timeout_ms":10000,"last_server_duration_us":1630301,"total_duration_us":1703473},{"operation_name":"get","last_dispatch_duration_us":1654983,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1654983,"total_server_duration_us":49,"operation_id":"0x64f4","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":1655164},{"operation_name":"get","last_dispatch_duration_us":1654706,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1654706,"total_server_duration_us":27,"operation_id":"0x64f5","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":1654910},{"operation_name":"upsert","last_dispatch_duration_us":754371,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":754371,"total_server_duration_us":411245,"encode_duration_us":26158,"operation_id":"0x65a8","timeout_ms":10000,"last_server_duration_us":411245,"total_duration_us":757721},{"operation_name":"upsert","last_dispatch_duration_us":657864,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":657864,"total_server_duration_us":19030,"encode_duration_us":29915,"operation_id":"0x659e","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":671119}],"total_count":13}} WARN [2022-03-13 17:12:17,510] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18419006,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18419006,"total_server_duration_us":6432,"operation_id":"0x5c95","timeout_ms":30000,"last_server_duration_us":6432,"total_duration_us":18421386},{"operation_name":"get","last_dispatch_duration_us":17195661,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17195661,"total_server_duration_us":6287,"operation_id":"0x5d2d","timeout_ms":30000,"last_server_duration_us":6287,"total_duration_us":17198893},{"operation_name":"get","last_dispatch_duration_us":16266160,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":16266160,"total_server_duration_us":6432,"operation_id":"0x5a0c","timeout_ms":30000,"last_server_duration_us":6432,"total_duration_us":16268599},{"operation_name":"get","last_dispatch_duration_us":13979854,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13979854,"total_server_duration_us":49,"operation_id":"0x5b0e","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":13982040},{"operation_name":"get","last_dispatch_duration_us":13631221,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13631221,"total_server_duration_us":38,"operation_id":"0x5f00","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":13634544},{"operation_name":"get","last_dispatch_duration_us":12650479,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":12650479,"total_server_duration_us":32,"operation_id":"0x5f97","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":12652976},{"operation_name":"upsert","last_dispatch_duration_us":14206403,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14206403,"total_server_duration_us":20921,"encode_duration_us":24223,"operation_id":"0x5e99","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10009387},{"operation_name":"upsert","last_dispatch_duration_us":15812620,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":15812620,"total_server_duration_us":29508,"encode_duration_us":22413,"operation_id":"0x5dbe","timeout_ms":10000,"last_server_duration_us":29508,"total_duration_us":10005952},{"operation_name":"upsert","last_dispatch_duration_us":18070997,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18070997,"total_server_duration_us":21977,"encode_duration_us":21983,"operation_id":"0x5cd7","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10001398},{"operation_name":"get","last_dispatch_duration_us":9104577,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9104577,"total_server_duration_us":27,"operation_id":"0x6155","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9107678}],"total_count":60}} WARN [2022-03-13 17:12:17,511] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21337931,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21337931,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9045859,"encode_duration_us":18133,"operation_id":"0x5bac","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10003116},{"operation_name":"upsert","last_dispatch_duration_us":15699839,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15699839,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9040551,"encode_duration_us":17850,"operation_id":"0x5a61","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10007195},{"operation_name":"upsert","last_dispatch_duration_us":25744883,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25744883,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9039489,"encode_duration_us":23063,"operation_id":"0x59ec","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10008294},{"operation_name":"upsert","last_dispatch_duration_us":17990100,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17990100,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9053293,"encode_duration_us":18620,"operation_id":"0x596d","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10009435}],"total_count":4}} INFO [2022-03-13 17:12:17,517] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,518] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:17,521] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,522] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:17,523] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,523] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:12:17,528] jepsen worker 50 - jepsen.util 2030 :ok :read [16 :nil] INFO [2022-03-13 17:12:17,528] jepsen worker 50 - jepsen.util 2030 :invoke :write [16 45] INFO [2022-03-13 17:12:17,543] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,543] jepsen worker 21 - jepsen.util 741 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,543] jepsen worker 21 - jepsen.util 741 :invoke :write [7 8] INFO [2022-03-13 17:12:17,544] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] WARN [2022-03-13 17:12:17,617] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":19809506,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":19809506,"total_server_duration_us":7129,"operation_id":"0x599c","timeout_ms":30000,"last_server_duration_us":7129,"total_duration_us":19811549},{"operation_name":"get","last_dispatch_duration_us":18586029,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18586029,"total_server_duration_us":13289,"operation_id":"0x5a04","timeout_ms":30000,"last_server_duration_us":13289,"total_duration_us":18594737},{"operation_name":"get","last_dispatch_duration_us":16132934,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":16132934,"total_server_duration_us":32,"operation_id":"0x5a5e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":16137397},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":16819,"operation_id":"0x5d10","timeout_ms":10000,"total_duration_us":10010025},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":20477,"operation_id":"0x5edc","timeout_ms":10000,"total_duration_us":10009498},{"operation_name":"upsert","encode_duration_us":18800,"operation_id":"0x5ffc","timeout_ms":10000,"total_duration_us":10004023},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":19382,"operation_id":"0x5d0f","timeout_ms":10000,"total_duration_us":10001578},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":18473,"operation_id":"0x5f72","timeout_ms":10000,"total_duration_us":10001372},{"operation_name":"upsert","last_dispatch_duration_us":6709219,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":6709219,"total_server_duration_us":50301,"encode_duration_us":23007,"operation_id":"0x5f0a","timeout_ms":10000,"last_server_duration_us":50301,"total_duration_us":6725007},{"operation_name":"upsert","last_dispatch_duration_us":4525000,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4525000,"total_server_duration_us":45278,"encode_duration_us":18245,"operation_id":"0x5f48","timeout_ms":10000,"last_server_duration_us":45278,"total_duration_us":6638148}],"total_count":55}} INFO [2022-03-13 17:12:17,619] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,619] jepsen worker 37 - jepsen.util 2017 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,619] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,622] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:12:17,619] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,623] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:17,625] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 30] INFO [2022-03-13 17:12:17,625] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:12:17,630] jepsen worker 70 - jepsen.util 700 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,630] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,633] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:12:17,635] jepsen worker 52 - jepsen.util 592 :invoke :write [17 19] INFO [2022-03-13 17:12:17,669] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,670] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:12:17,671] jepsen worker 24 - jepsen.util 564 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,672] jepsen worker 24 - jepsen.util 564 :invoke :write [8 39] INFO [2022-03-13 17:12:17,716] jepsen worker 57 - jepsen.util 2037 :info :write [19 41] :RequestTimeoutException WARN [2022-03-13 17:12:17,717] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16901403,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17021506,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9120382,"encode_duration_us":29770,"operation_id":"0x5a12","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10008024},{"operation_name":"upsert","last_dispatch_duration_us":16729213,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17591582,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9333759,"encode_duration_us":23550,"operation_id":"0x5abf","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10008982},{"operation_name":"upsert","last_dispatch_duration_us":10060797,"last_local_id":"A312307A00000003/00000000590563BE","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10060797,"last_local_socket":"172.23.104.79:37454","total_server_duration_us":9840522,"encode_duration_us":19824,"operation_id":"0x5f75","timeout_ms":10000,"last_server_duration_us":9840522,"total_duration_us":10008329}],"total_count":3}} WARN [2022-03-13 17:12:17,717] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":23260,"operation_id":"0x5ec2","timeout_ms":10000,"total_duration_us":10006776},{"operation_name":"upsert","last_dispatch_duration_us":2780531,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":2780531,"total_server_duration_us":23475,"encode_duration_us":18007,"operation_id":"0x5c8d","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10006379},{"operation_name":"upsert","encode_duration_us":16840,"operation_id":"0x5ec3","timeout_ms":10000,"total_duration_us":10006179},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17056,"operation_id":"0x5e88","timeout_ms":10000,"total_duration_us":10005464},{"operation_name":"upsert","last_dispatch_duration_us":1568663,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":1568663,"total_server_duration_us":21405,"encode_duration_us":17638,"operation_id":"0x5d16","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10005238},{"operation_name":"upsert","encode_duration_us":19612,"operation_id":"0x5eff","timeout_ms":10000,"total_duration_us":10002853},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17983,"operation_id":"0x5eb3","timeout_ms":10000,"total_duration_us":10002183},{"operation_name":"get","last_dispatch_duration_us":9086450,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9086450,"total_server_duration_us":43,"operation_id":"0x5dbb","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9089117},{"operation_name":"upsert","last_dispatch_duration_us":6905610,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6905610,"total_server_duration_us":41337,"encode_duration_us":19163,"operation_id":"0x5ef3","timeout_ms":10000,"last_server_duration_us":41337,"total_duration_us":6916938},{"operation_name":"upsert","last_dispatch_duration_us":6762108,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6762108,"total_server_duration_us":48909,"encode_duration_us":17381,"operation_id":"0x5ef1","timeout_ms":10000,"last_server_duration_us":48909,"total_duration_us":6769906}],"total_count":61}} WARN [2022-03-13 17:12:17,717] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12415256,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12415256,"total_server_duration_us":69,"operation_id":"0x5ca1","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":12415642},{"operation_name":"get","last_dispatch_duration_us":11169236,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":11169236,"total_server_duration_us":84,"operation_id":"0x5d48","timeout_ms":30000,"last_server_duration_us":84,"total_duration_us":11169424},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":22073,"operation_id":"0x5fef","timeout_ms":10000,"total_duration_us":10003400},{"operation_name":"get","last_dispatch_duration_us":9655482,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9655482,"total_server_duration_us":32,"operation_id":"0x5dd0","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9655746},{"operation_name":"get","last_dispatch_duration_us":8341431,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8341431,"total_server_duration_us":208692,"operation_id":"0x5eb0","timeout_ms":30000,"last_server_duration_us":208692,"total_duration_us":8343537},{"operation_name":"get","last_dispatch_duration_us":8106445,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8106445,"total_server_duration_us":56,"operation_id":"0x5f30","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8108448},{"operation_name":"upsert","last_dispatch_duration_us":3066228,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3066228,"total_server_duration_us":205072,"encode_duration_us":34085,"operation_id":"0x627b","timeout_ms":10000,"last_server_duration_us":205072,"total_duration_us":3070260},{"operation_name":"get","last_dispatch_duration_us":2984423,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2984423,"total_server_duration_us":43,"operation_id":"0x6282","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3003026},{"operation_name":"upsert","last_dispatch_duration_us":2944364,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2944364,"total_server_duration_us":148091,"encode_duration_us":27081,"operation_id":"0x6263","timeout_ms":10000,"last_server_duration_us":148091,"total_duration_us":2955357},{"operation_name":"get","last_dispatch_duration_us":2948486,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":2948486,"total_server_duration_us":23,"operation_id":"0x6168","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":2951913}],"total_count":39}} WARN [2022-03-13 17:12:17,717] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9067431,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10762562,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9553712,"encode_duration_us":22709,"operation_id":"0x5aa4","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10002099},{"operation_name":"upsert","last_dispatch_duration_us":17756631,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17756631,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9051168,"encode_duration_us":17238,"operation_id":"0x59ed","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10006135},{"operation_name":"upsert","last_dispatch_duration_us":9055990,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10577183,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9578620,"encode_duration_us":19666,"operation_id":"0x5ae7","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10007639},{"operation_name":"upsert","last_dispatch_duration_us":15387039,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15387039,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":6605056,"encode_duration_us":19401,"operation_id":"0x58c2","timeout_ms":10000,"last_server_duration_us":6605056,"total_duration_us":10008026}],"total_count":4}} INFO [2022-03-13 17:12:17,718] jepsen worker 57 - jepsen.util 2127 :invoke :write [19 16] INFO [2022-03-13 17:12:17,733] jepsen worker 6 - jepsen.util 636 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,733] jepsen worker 44 - jepsen.util 584 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,734] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,734] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:17,734] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,735] jepsen worker 6 - jepsen.util 636 :invoke :write [2 21] INFO [2022-03-13 17:12:17,740] jepsen worker 60 - jepsen.util 600 :invoke :write [20 1] INFO [2022-03-13 17:12:17,740] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:17,747] jepsen worker 34 - jepsen.util 754 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,747] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:17,783] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,784] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 24] INFO [2022-03-13 17:12:17,815] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,815] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:17,821] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,821] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,822] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:12:17,823] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:12:17,838] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,838] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:12:17,867] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5555555555555556}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:17,867] jepsen worker nemesis - couchbase.util Rebalance check count: 248 INFO [2022-03-13 17:12:17,867] jepsen worker nemesis - couchbase.util Rebalance stuck count: 158 INFO [2022-03-13 17:12:17,894] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,899] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:17,911] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,912] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:17,914] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,915] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:17,915] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,916] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:12:17,947] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,947] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,948] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:17,948] jepsen worker 79 - jepsen.util 979 :invoke :write [26 49] INFO [2022-03-13 17:12:17,965] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,965] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:17,984] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,984] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,985] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:17,985] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:12:17,986] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,988] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:17,991] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:17,991] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:18,007] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,008] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:12:18,012] jepsen worker 77 - jepsen.util 707 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,012] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:18,012] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,012] jepsen worker 43 - jepsen.util 493 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,012] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,013] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:18,013] jepsen worker 42 - jepsen.util 402 :invoke :write [14 18] INFO [2022-03-13 17:12:18,014] jepsen worker 43 - jepsen.util 493 :invoke :write [14 16] INFO [2022-03-13 17:12:18,015] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:18,022] jepsen worker 12 - jepsen.util 732 :invoke :write [4 16] INFO [2022-03-13 17:12:18,062] jepsen worker 63 - jepsen.util 1953 :ok :read [21 :nil] INFO [2022-03-13 17:12:18,063] jepsen worker 63 - jepsen.util 1953 :invoke :read [21 nil] INFO [2022-03-13 17:12:18,067] jepsen worker 88 - jepsen.util 1978 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,067] jepsen worker 37 - jepsen.util 2017 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,067] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,067] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:12:18,068] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:12:18,068] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 43] INFO [2022-03-13 17:12:18,153] jepsen worker 47 - jepsen.util 2117 :ok :read [15 48] INFO [2022-03-13 17:12:18,153] jepsen worker 47 - jepsen.util 2117 :invoke :write [15 19] INFO [2022-03-13 17:12:18,207] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,207] jepsen worker 3 - jepsen.util 1893 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,208] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:12:18,208] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:18,234] jepsen worker 63 - jepsen.util 1953 :ok :read [21 :nil] INFO [2022-03-13 17:12:18,235] jepsen worker 63 - jepsen.util 1953 :invoke :read [21 nil] INFO [2022-03-13 17:12:18,290] jepsen worker 63 - jepsen.util 1953 :ok :read [21 :nil] INFO [2022-03-13 17:12:18,291] jepsen worker 63 - jepsen.util 1953 :invoke :write [21 42] INFO [2022-03-13 17:12:18,308] jepsen worker 46 - jepsen.util 2026 :info :write [15 24] :RequestTimeoutException INFO [2022-03-13 17:12:18,308] jepsen worker 46 - jepsen.util 2116 :invoke :read [15 nil] INFO [2022-03-13 17:12:18,344] jepsen worker 46 - jepsen.util 2116 :ok :read [15 48] INFO [2022-03-13 17:12:18,344] jepsen worker 46 - jepsen.util 2116 :invoke :write [15 24] INFO [2022-03-13 17:12:18,393] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,393] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,393] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:18,394] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:18,411] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,412] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:18,413] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,413] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:18,414] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,415] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:12:18,433] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,433] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:18,461] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:18,462] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:12:18,471] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,471] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:18,479] jepsen worker 78 - jepsen.util 528 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,479] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:18,516] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,516] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,517] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:18,518] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:12:18,518] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,519] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:18,521] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,521] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:12:18,531] jepsen worker 21 - jepsen.util 741 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,532] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,554] jepsen worker 51 - jepsen.util 591 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,555] jepsen worker 51 - jepsen.util 591 :invoke :write [17 31] INFO [2022-03-13 17:12:18,598] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,598] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,598] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:18,599] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:18,600] jepsen worker 59 - jepsen.util 1949 :info :write [19 39] :RequestTimeoutException INFO [2022-03-13 17:12:18,601] jepsen worker 59 - jepsen.util 2039 :invoke :read [19 nil] INFO [2022-03-13 17:12:18,625] jepsen worker 61 - jepsen.util 601 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,625] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:18,638] jepsen worker 80 - jepsen.util 710 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,638] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:18,652] jepsen worker 24 - jepsen.util 564 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,653] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:12:18,656] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,656] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:18,668] jepsen worker 84 - jepsen.util 2154 :info :write [28 40] :RequestTimeoutException INFO [2022-03-13 17:12:18,668] jepsen worker 84 - jepsen.util 2244 :invoke :read [28 nil] INFO [2022-03-13 17:12:18,670] jepsen worker 40 - jepsen.util 2020 :info :write [13 44] :RequestTimeoutException INFO [2022-03-13 17:12:18,671] jepsen worker 40 - jepsen.util 2110 :invoke :write [13 7] INFO [2022-03-13 17:12:18,674] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,675] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,674] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,675] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 1] INFO [2022-03-13 17:12:18,677] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:12:18,678] jepsen worker 88 - jepsen.util 1978 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,683] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:18,684] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 20] INFO [2022-03-13 17:12:18,711] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,712] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,713] jepsen worker 84 - jepsen.util 2244 :ok :read [28 44] INFO [2022-03-13 17:12:18,714] jepsen worker 84 - jepsen.util 2244 :invoke :read [28 nil] INFO [2022-03-13 17:12:18,717] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,718] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,724] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,725] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,726] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,726] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,728] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,728] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,729] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,730] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:12:18,742] jepsen worker 84 - jepsen.util 2244 :ok :read [28 44] INFO [2022-03-13 17:12:18,743] jepsen worker 84 - jepsen.util 2244 :invoke :read [28 nil] INFO [2022-03-13 17:12:18,757] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,757] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,757] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,757] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:18,758] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:12:18,759] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:18,760] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,760] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:18,767] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,768] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:12:18,772] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:18,772] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,773] jepsen worker 44 - jepsen.util 584 :invoke :write [14 41] INFO [2022-03-13 17:12:18,773] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,774] jepsen worker 84 - jepsen.util 2244 :ok :read [28 44] INFO [2022-03-13 17:12:18,775] jepsen worker 84 - jepsen.util 2244 :invoke :read [28 nil] INFO [2022-03-13 17:12:18,786] jepsen worker 76 - jepsen.util 706 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,787] jepsen worker 76 - jepsen.util 706 :invoke :write [25 2] INFO [2022-03-13 17:12:18,796] jepsen worker 13 - jepsen.util 643 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,796] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:18,805] jepsen worker 84 - jepsen.util 2244 :ok :read [28 44] INFO [2022-03-13 17:12:18,805] jepsen worker 84 - jepsen.util 2244 :invoke :write [28 46] INFO [2022-03-13 17:12:18,850] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,850] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,851] jepsen worker 7 - jepsen.util 637 :invoke :write [2 44] INFO [2022-03-13 17:12:18,852] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:12:18,871] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,871] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:12:18,872] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5555555555555556}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:18,873] jepsen worker nemesis - couchbase.util Rebalance check count: 249 INFO [2022-03-13 17:12:18,873] jepsen worker nemesis - couchbase.util Rebalance stuck count: 159 INFO [2022-03-13 17:12:18,875] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,876] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:12:18,882] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,883] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,883] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:18,884] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:18,899] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,899] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,900] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:18,901] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:18,915] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:18,916] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:18,917] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:18,917] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:18,929] jepsen worker 52 - jepsen.util 592 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,929] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:18,950] jepsen worker 79 - jepsen.util 979 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,950] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,950] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,951] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:18,952] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:18,952] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:18,978] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,979] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:18,985] jepsen worker 39 - jepsen.util 2019 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,985] jepsen worker 67 - jepsen.util 1687 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,986] jepsen worker 39 - jepsen.util 2019 :invoke :read [13 nil] INFO [2022-03-13 17:12:18,986] jepsen worker 67 - jepsen.util 1687 :invoke :read [22 nil] INFO [2022-03-13 17:12:18,990] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:18,990] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:12:19,017] jepsen worker 66 - jepsen.util 1776 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,018] jepsen worker 66 - jepsen.util 1776 :invoke :write [22 30] INFO [2022-03-13 17:12:19,040] jepsen worker 15 - jepsen.util 1995 :ok :read [5 43] INFO [2022-03-13 17:12:19,040] jepsen worker 15 - jepsen.util 1995 :invoke :write [5 19] INFO [2022-03-13 17:12:19,051] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,051] jepsen worker 26 - jepsen.util 476 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,051] jepsen worker 60 - jepsen.util 600 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,052] jepsen worker 26 - jepsen.util 476 :invoke :write [8 45] INFO [2022-03-13 17:12:19,053] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,054] jepsen worker 75 - jepsen.util 435 :invoke :write [25 36] INFO [2022-03-13 17:12:19,055] jepsen worker 60 - jepsen.util 600 :invoke :write [20 14] INFO [2022-03-13 17:12:19,055] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,055] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:12:19,058] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:19,072] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,072] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,073] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:19,073] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,074] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,075] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,075] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:19,076] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,077] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,077] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,077] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:12:19,082] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,084] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,084] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:12:19,088] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,089] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,089] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:19,090] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:19,092] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,092] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,092] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,092] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,092] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,093] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:12:19,094] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:19,095] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:19,095] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:19,096] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:19,101] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,101] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,102] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:19,102] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 5] INFO [2022-03-13 17:12:19,103] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,106] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:12:19,116] jepsen worker 39 - jepsen.util 2019 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,117] jepsen worker 67 - jepsen.util 1687 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,119] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,119] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,119] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,120] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,120] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,120] jepsen worker 34 - jepsen.util 754 :invoke :write [11 27] INFO [2022-03-13 17:12:19,121] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,121] jepsen worker 39 - jepsen.util 2019 :invoke :write [13 7] INFO [2022-03-13 17:12:19,121] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:19,121] jepsen worker 70 - jepsen.util 700 :invoke :write [23 22] INFO [2022-03-13 17:12:19,122] jepsen worker 52 - jepsen.util 592 :invoke :write [17 47] INFO [2022-03-13 17:12:19,124] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:19,125] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:19,125] jepsen worker 67 - jepsen.util 1687 :invoke :read [22 nil] INFO [2022-03-13 17:12:19,125] jepsen worker 13 - jepsen.util 643 :invoke :write [4 10] INFO [2022-03-13 17:12:19,127] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:19,136] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,137] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,139] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:19,139] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,140] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,140] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,140] jepsen worker 78 - jepsen.util 528 :invoke :write [26 7] INFO [2022-03-13 17:12:19,141] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:19,141] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,142] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:19,142] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,143] jepsen worker 80 - jepsen.util 710 :invoke :write [26 28] INFO [2022-03-13 17:12:19,143] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,144] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:19,145] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,146] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:12:19,166] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:19,167] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:19,169] jepsen worker 59 - jepsen.util 2039 :ok :read [19 20] INFO [2022-03-13 17:12:19,171] jepsen worker 59 - jepsen.util 2039 :invoke :write [19 6] INFO [2022-03-13 17:12:19,172] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,172] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:19,188] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:19,189] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:19,203] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,206] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:12:19,222] jepsen worker 68 - jepsen.util 2138 :ok :read [22 30] INFO [2022-03-13 17:12:19,223] jepsen worker 68 - jepsen.util 2138 :invoke :read [22 nil] INFO [2022-03-13 17:12:19,226] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,226] jepsen worker 75 - jepsen.util 435 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,226] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,227] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,227] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:19,227] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,227] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,228] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:19,228] jepsen worker 21 - jepsen.util 741 :invoke :write [7 4] INFO [2022-03-13 17:12:19,229] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:19,229] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:19,230] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:19,241] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,242] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:12:19,263] jepsen worker 73 - jepsen.util 2053 :info :write [24 5] :RequestTimeoutException INFO [2022-03-13 17:12:19,266] jepsen worker 73 - jepsen.util 2143 :invoke :read [24 nil] INFO [2022-03-13 17:12:19,290] jepsen worker 67 - jepsen.util 1687 :ok :read [22 30] INFO [2022-03-13 17:12:19,291] jepsen worker 67 - jepsen.util 1687 :invoke :write [22 43] INFO [2022-03-13 17:12:19,305] jepsen worker 68 - jepsen.util 2138 :ok :read [22 30] INFO [2022-03-13 17:12:19,310] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:19,311] jepsen worker 68 - jepsen.util 2138 :invoke :write [22 8] INFO [2022-03-13 17:12:19,312] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:12:19,321] jepsen worker 12 - jepsen.util 732 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,321] jepsen worker 53 - jepsen.util 503 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,321] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,321] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:19,322] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,322] jepsen worker 53 - jepsen.util 503 :invoke :write [17 41] INFO [2022-03-13 17:12:19,359] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,359] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,359] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,359] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,359] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,359] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,360] jepsen worker 70 - jepsen.util 700 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,361] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:12:19,362] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:12:19,363] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:12:19,363] jepsen worker 79 - jepsen.util 979 :invoke :write [26 39] INFO [2022-03-13 17:12:19,367] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:19,368] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:12:19,369] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:19,431] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,431] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,432] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,432] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:12:19,437] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,437] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,444] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,444] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:12:19,464] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,464] jepsen worker 21 - jepsen.util 741 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,465] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:19,465] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:19,465] jepsen worker 0 - jepsen.util 2070 :info :write [0 15] :RequestTimeoutException INFO [2022-03-13 17:12:19,466] jepsen worker 0 - jepsen.util 2160 :invoke :write [0 43] INFO [2022-03-13 17:12:19,493] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,494] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,494] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:12:19,495] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:19,497] jepsen worker 37 - jepsen.util 2017 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,498] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:12:19,511] jepsen worker 6 - jepsen.util 636 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,511] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,511] jepsen worker 51 - jepsen.util 591 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,512] jepsen worker 6 - jepsen.util 636 :invoke :write [2 47] INFO [2022-03-13 17:12:19,513] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:19,519] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:19,543] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,544] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,548] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,548] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:12:19,576] jepsen worker 80 - jepsen.util 710 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,577] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:19,577] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,577] jepsen worker 22 - jepsen.util 652 :invoke :write [7 8] INFO [2022-03-13 17:12:19,685] jepsen worker 44 - jepsen.util 584 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,686] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:19,697] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,698] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:19,712] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,712] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,721] jepsen worker 13 - jepsen.util 643 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,721] jepsen worker 13 - jepsen.util 643 :invoke :write [4 4] INFO [2022-03-13 17:12:19,727] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,727] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:19,806] jepsen worker 42 - jepsen.util 402 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,806] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,806] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,807] jepsen worker 62 - jepsen.util 512 :invoke :write [20 22] INFO [2022-03-13 17:12:19,807] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:19,807] jepsen worker 42 - jepsen.util 402 :invoke :write [14 15] INFO [2022-03-13 17:12:19,878] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5555555555555556}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:19,879] jepsen worker nemesis - couchbase.util Rebalance check count: 250 INFO [2022-03-13 17:12:19,879] jepsen worker nemesis - couchbase.util Rebalance stuck count: 160 INFO [2022-03-13 17:12:19,956] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,957] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:12:19,963] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,963] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,964] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:19,964] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:19,964] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,965] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,965] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:12:19,965] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:12:19,971] jepsen worker 22 - jepsen.util 652 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,971] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:19,989] jepsen worker 79 - jepsen.util 979 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:19,989] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:20,021] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,022] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,035] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,036] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,065] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:20,066] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,075] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,076] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:12:20,087] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,087] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,087] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,088] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,088] jepsen worker 3 - jepsen.util 1893 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,088] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,088] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:20,089] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:20,089] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,090] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,090] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 30] INFO [2022-03-13 17:12:20,091] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:20,092] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,093] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:12:20,093] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:12:20,094] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:12:20,113] jepsen worker 34 - jepsen.util 754 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,119] jepsen worker 34 - jepsen.util 754 :invoke :write [11 7] INFO [2022-03-13 17:12:20,119] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,121] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:12:20,127] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:20,129] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,132] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:20,133] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:12:20,218] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,219] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:20,220] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,220] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,220] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,220] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,220] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,220] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,221] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,221] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:12:20,222] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:12:20,224] jepsen worker 21 - jepsen.util 741 :invoke :write [7 25] INFO [2022-03-13 17:12:20,224] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,225] jepsen worker 79 - jepsen.util 979 :invoke :write [26 47] INFO [2022-03-13 17:12:20,230] jepsen worker 78 - jepsen.util 528 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,231] jepsen worker 78 - jepsen.util 528 :invoke :write [26 39] INFO [2022-03-13 17:12:20,234] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,234] jepsen worker 76 - jepsen.util 706 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,235] jepsen worker 36 - jepsen.util 2106 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,235] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,235] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:20,237] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:20,238] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:20,238] jepsen worker 43 - jepsen.util 493 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,238] jepsen worker 76 - jepsen.util 706 :invoke :write [25 11] INFO [2022-03-13 17:12:20,239] jepsen worker 43 - jepsen.util 493 :invoke :write [14 9] INFO [2022-03-13 17:12:20,268] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,269] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:20,323] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,324] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:20,339] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,340] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,340] jepsen worker 86 - jepsen.util 1796 :info :write [28 43] :SyncWriteAmbiguous INFO [2022-03-13 17:12:20,340] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,341] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:12:20,342] jepsen worker 86 - jepsen.util 1886 :invoke :write [28 7] INFO [2022-03-13 17:12:20,342] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,342] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,381] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,381] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,381] jepsen worker 52 - jepsen.util 592 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,382] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,383] jepsen worker 80 - jepsen.util 710 :invoke :write [26 35] INFO [2022-03-13 17:12:20,383] jepsen worker 52 - jepsen.util 592 :invoke :write [17 20] INFO [2022-03-13 17:12:20,410] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,410] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,411] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,412] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:12:20,415] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,415] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,416] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,417] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:20,421] jepsen worker 7 - jepsen.util 637 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,421] jepsen worker 60 - jepsen.util 600 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,421] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,421] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,422] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,422] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:20,422] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,422] jepsen worker 51 - jepsen.util 591 :invoke :write [17 47] INFO [2022-03-13 17:12:20,423] jepsen worker 44 - jepsen.util 584 :invoke :write [14 24] INFO [2022-03-13 17:12:20,424] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:20,425] jepsen worker 7 - jepsen.util 637 :invoke :write [2 42] INFO [2022-03-13 17:12:20,425] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:20,426] jepsen worker 60 - jepsen.util 600 :invoke :write [20 6] INFO [2022-03-13 17:12:20,428] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,429] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,430] jepsen worker 77 - jepsen.util 707 :invoke :write [25 31] INFO [2022-03-13 17:12:20,430] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,431] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:12:20,469] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,470] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:12:20,502] jepsen worker 2 - jepsen.util 2342 :ok :read [0 41] INFO [2022-03-13 17:12:20,502] jepsen worker 2 - jepsen.util 2342 :invoke :read [0 nil] INFO [2022-03-13 17:12:20,517] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,517] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,517] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,518] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:20,577] jepsen worker 13 - jepsen.util 643 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,578] jepsen worker 13 - jepsen.util 643 :invoke :write [4 25] INFO [2022-03-13 17:12:20,605] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,605] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,606] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,606] jepsen worker 26 - jepsen.util 476 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,606] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,606] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,606] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:12:20,606] jepsen worker 76 - jepsen.util 706 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,607] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,607] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:20,608] jepsen worker 21 - jepsen.util 741 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,608] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,609] jepsen worker 23 - jepsen.util 563 :invoke :write [7 45] INFO [2022-03-13 17:12:20,609] jepsen worker 12 - jepsen.util 732 :invoke :write [4 41] INFO [2022-03-13 17:12:20,610] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:20,610] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,619] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,620] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,621] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,622] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,623] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,624] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:20,625] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,626] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:12:20,682] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,683] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:12:20,684] jepsen worker 2 - jepsen.util 2342 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,684] jepsen worker 2 - jepsen.util 2342 :invoke :read [0 nil] INFO [2022-03-13 17:12:20,703] jepsen worker 79 - jepsen.util 979 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,703] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,703] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,704] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:20,704] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,705] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,738] jepsen worker 77 - jepsen.util 707 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,739] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:20,742] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,743] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:12:20,759] jepsen worker 34 - jepsen.util 754 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,760] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:20,765] jepsen worker 32 - jepsen.util 1832 :ok :read [10 :nil] INFO [2022-03-13 17:12:20,766] jepsen worker 32 - jepsen.util 1832 :invoke :read [10 nil] INFO [2022-03-13 17:12:20,785] jepsen worker 18 - jepsen.util 1908 :ok :read [6 16] INFO [2022-03-13 17:12:20,785] jepsen worker 18 - jepsen.util 1908 :invoke :read [6 nil] INFO [2022-03-13 17:12:20,787] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,788] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:12:20,796] jepsen worker 20 - jepsen.util 2450 :ok :read [6 16] INFO [2022-03-13 17:12:20,796] jepsen worker 20 - jepsen.util 2450 :invoke :read [6 nil] INFO [2022-03-13 17:12:20,817] jepsen worker 82 - jepsen.util 1972 :info :write [27 48] :RequestTimeoutException INFO [2022-03-13 17:12:20,817] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,817] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,818] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:12:20,818] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,821] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,821] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,821] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,821] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,823] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:12:20,824] jepsen worker 33 - jepsen.util 663 :invoke :write [11 13] INFO [2022-03-13 17:12:20,825] jepsen worker 79 - jepsen.util 979 :invoke :write [26 26] INFO [2022-03-13 17:12:20,826] jepsen worker 53 - jepsen.util 503 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,826] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,826] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,828] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,829] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:20,830] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,831] jepsen worker 82 - jepsen.util 2062 :invoke :read [27 nil] INFO [2022-03-13 17:12:20,832] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,832] jepsen worker 14 - jepsen.util 284 :invoke :write [4 22] INFO [2022-03-13 17:12:20,834] jepsen worker 53 - jepsen.util 503 :invoke :write [17 36] INFO [2022-03-13 17:12:20,834] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,838] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,839] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,838] jepsen worker 32 - jepsen.util 1832 :ok :read [10 :nil] INFO [2022-03-13 17:12:20,844] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:12:20,847] jepsen worker 32 - jepsen.util 1832 :invoke :write [10 23] INFO [2022-03-13 17:12:20,848] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,848] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:20,848] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,853] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:20,861] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,862] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:12:20,871] jepsen worker 27 - jepsen.util 2367 :ok :read [9 26] INFO [2022-03-13 17:12:20,873] jepsen worker 27 - jepsen.util 2367 :invoke :write [9 13] INFO [2022-03-13 17:12:20,882] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:20,883] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,884] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,884] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:20,885] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,887] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5555555555555556}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:20,887] jepsen worker nemesis - couchbase.util Rebalance check count: 251 INFO [2022-03-13 17:12:20,887] jepsen worker nemesis - couchbase.util Rebalance stuck count: 161 INFO [2022-03-13 17:12:20,888] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:20,889] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,890] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,889] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,890] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,891] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 33] INFO [2022-03-13 17:12:20,892] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:20,893] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:20,895] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,895] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:12:20,896] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:20,898] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,898] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:12:20,902] jepsen worker 3 - jepsen.util 1893 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,907] jepsen worker 18 - jepsen.util 1908 :ok :read [6 16] INFO [2022-03-13 17:12:20,907] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 36] INFO [2022-03-13 17:12:20,909] jepsen worker 18 - jepsen.util 1908 :invoke :write [6 29] INFO [2022-03-13 17:12:20,912] jepsen worker 20 - jepsen.util 2450 :ok :read [6 16] INFO [2022-03-13 17:12:20,913] jepsen worker 20 - jepsen.util 2450 :invoke :write [6 30] INFO [2022-03-13 17:12:20,926] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,928] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,928] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:20,933] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,934] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:12:20,934] jepsen worker 2 - jepsen.util 2342 :ok :read [0 41] INFO [2022-03-13 17:12:20,934] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:20,937] jepsen worker 2 - jepsen.util 2342 :invoke :write [0 12] INFO [2022-03-13 17:12:20,945] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:20,946] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,948] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:12:20,949] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:20,964] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,964] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,965] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:12:20,966] jepsen worker 69 - jepsen.util 609 :invoke :write [23 45] INFO [2022-03-13 17:12:20,969] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:20,969] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 12] INFO [2022-03-13 17:12:20,972] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:20,973] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:12:21,008] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:21,009] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:21,020] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,020] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:21,026] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,027] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:12:21,052] jepsen worker 42 - jepsen.util 402 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,052] jepsen worker 51 - jepsen.util 591 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,053] jepsen worker 42 - jepsen.util 402 :invoke :write [14 26] INFO [2022-03-13 17:12:21,054] jepsen worker 51 - jepsen.util 591 :invoke :write [17 31] INFO [2022-03-13 17:12:21,060] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,060] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,060] jepsen worker 78 - jepsen.util 528 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,062] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:21,063] jepsen worker 78 - jepsen.util 528 :invoke :write [26 35] INFO [2022-03-13 17:12:21,064] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:21,065] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,067] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:12:21,078] jepsen worker 3 - jepsen.util 1893 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,079] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:12:21,080] jepsen worker 33 - jepsen.util 663 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,081] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:21,082] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,082] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:21,090] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,091] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:12:21,096] jepsen worker 88 - jepsen.util 1978 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,096] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:21,108] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:21,109] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:12:21,141] jepsen worker 64 - jepsen.util 2044 :ok :read [21 :nil] INFO [2022-03-13 17:12:21,141] jepsen worker 64 - jepsen.util 2044 :invoke :read [21 nil] INFO [2022-03-13 17:12:21,143] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:21,143] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:21,162] jepsen worker 36 - jepsen.util 2106 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,162] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,162] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,162] jepsen worker 80 - jepsen.util 710 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,163] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:21,164] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:21,165] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 33] INFO [2022-03-13 17:12:21,166] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 29] INFO [2022-03-13 17:12:21,195] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,196] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:12:21,199] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,199] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,200] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:12:21,200] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:21,201] jepsen worker 79 - jepsen.util 979 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,201] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,203] jepsen worker 21 - jepsen.util 741 :invoke :write [7 7] INFO [2022-03-13 17:12:21,204] jepsen worker 79 - jepsen.util 979 :invoke :write [26 3] INFO [2022-03-13 17:12:21,215] jepsen worker 64 - jepsen.util 2044 :ok :read [21 :nil] INFO [2022-03-13 17:12:21,217] jepsen worker 64 - jepsen.util 2044 :invoke :write [21 19] INFO [2022-03-13 17:12:21,232] jepsen worker 29 - jepsen.util 2189 :ok :read [9 26] INFO [2022-03-13 17:12:21,232] jepsen worker 29 - jepsen.util 2189 :invoke :write [9 2] INFO [2022-03-13 17:12:21,285] jepsen worker 62 - jepsen.util 512 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,285] jepsen worker 60 - jepsen.util 600 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,286] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:21,287] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:12:21,296] jepsen worker 52 - jepsen.util 592 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,297] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:21,304] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,304] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:12:21,317] jepsen worker 48 - jepsen.util 2028 :info :write [16 12] :RequestTimeoutException INFO [2022-03-13 17:12:21,317] jepsen worker 48 - jepsen.util 2118 :invoke :write [16 26] INFO [2022-03-13 17:12:21,324] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,324] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,325] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:21,326] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,327] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:21,328] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:21,328] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,329] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,329] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:21,330] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:12:21,331] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,338] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:12:21,374] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,375] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:12:21,387] jepsen worker 9 - jepsen.util 1989 :ok :read [3 25] INFO [2022-03-13 17:12:21,387] jepsen worker 9 - jepsen.util 1989 :invoke :write [3 35] INFO [2022-03-13 17:12:21,402] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,403] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:21,444] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,445] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:21,470] jepsen worker 41 - jepsen.util 1841 :info :write [13 8] :RequestTimeoutException INFO [2022-03-13 17:12:21,471] jepsen worker 41 - jepsen.util 1931 :invoke :read [13 nil] INFO [2022-03-13 17:12:21,480] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,480] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,480] jepsen worker 13 - jepsen.util 643 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,481] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:21,482] jepsen worker 34 - jepsen.util 754 :invoke :write [11 1] INFO [2022-03-13 17:12:21,483] jepsen worker 70 - jepsen.util 700 :invoke :write [23 14] INFO [2022-03-13 17:12:21,512] jepsen worker 43 - jepsen.util 493 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,512] jepsen worker 44 - jepsen.util 584 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,512] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,513] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:21,514] jepsen worker 76 - jepsen.util 706 :invoke :write [25 22] INFO [2022-03-13 17:12:21,514] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:21,560] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,560] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,560] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,561] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:21,562] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:21,563] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,563] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:12:21,563] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:21,621] jepsen worker 12 - jepsen.util 732 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,622] jepsen worker 12 - jepsen.util 732 :invoke :write [4 8] INFO [2022-03-13 17:12:21,639] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,640] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:21,660] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,660] jepsen worker 69 - jepsen.util 609 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,661] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:12:21,662] jepsen worker 69 - jepsen.util 609 :invoke :write [23 8] INFO [2022-03-13 17:12:21,666] jepsen worker 81 - jepsen.util 2331 :info :write [27 17] :RequestTimeoutException INFO [2022-03-13 17:12:21,668] jepsen worker 81 - jepsen.util 2421 :invoke :read [27 nil] INFO [2022-03-13 17:12:21,671] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,671] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:12:21,678] jepsen worker 7 - jepsen.util 637 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,679] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,679] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,679] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:21,679] jepsen worker 26 - jepsen.util 476 :invoke :write [8 44] INFO [2022-03-13 17:12:21,680] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:21,681] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,682] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:12:21,703] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,703] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,703] jepsen worker 78 - jepsen.util 528 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,704] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,705] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:21,706] jepsen worker 13 - jepsen.util 643 :invoke :write [4 21] INFO [2022-03-13 17:12:21,706] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:21,707] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:21,727] jepsen worker 16 - jepsen.util 1906 :info :write [5 42] :RequestTimeoutException INFO [2022-03-13 17:12:21,728] jepsen worker 16 - jepsen.util 1996 :invoke :read [5 nil] INFO [2022-03-13 17:12:21,732] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,733] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:12:21,747] jepsen worker 21 - jepsen.util 741 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,748] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:21,750] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,751] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:21,755] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,756] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:12:21,762] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,763] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:21,766] jepsen worker 45 - jepsen.util 2385 :info :write [15 17] :RequestTimeoutException INFO [2022-03-13 17:12:21,767] jepsen worker 45 - jepsen.util 2475 :invoke :write [15 10] INFO [2022-03-13 17:12:21,770] jepsen worker 41 - jepsen.util 1931 :ok :read [13 39] INFO [2022-03-13 17:12:21,771] jepsen worker 41 - jepsen.util 1931 :invoke :read [13 nil] INFO [2022-03-13 17:12:21,797] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,797] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:12:21,800] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:21,801] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:12:21,828] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,828] jepsen worker 37 - jepsen.util 2017 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,828] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:12:21,832] jepsen worker 41 - jepsen.util 1931 :ok :read [13 39] INFO [2022-03-13 17:12:21,832] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:21,833] jepsen worker 41 - jepsen.util 1931 :invoke :write [13 20] INFO [2022-03-13 17:12:21,835] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,838] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:12:21,842] jepsen worker 79 - jepsen.util 979 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,842] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:21,851] jepsen worker 34 - jepsen.util 754 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,851] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:21,883] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,884] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:21,894] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5614035087719298}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:21,894] jepsen worker nemesis - couchbase.util Rebalance check count: 252 INFO [2022-03-13 17:12:21,894] jepsen worker nemesis - couchbase.util Rebalance stuck count: 161 INFO [2022-03-13 17:12:21,900] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,901] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:21,910] jepsen worker 8 - jepsen.util 728 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,910] jepsen worker 23 - jepsen.util 563 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,911] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,912] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:21,912] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:21,913] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:21,931] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,932] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:21,948] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,949] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:12:21,954] jepsen worker 36 - jepsen.util 2106 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:21,955] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:21,961] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:21,962] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,008] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,008] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,009] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:12:22,009] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,010] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:12:22,011] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:22,036] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,037] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:12:22,050] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:22,051] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:12:22,075] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,076] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,080] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,081] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,082] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,082] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:12:22,106] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,107] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,145] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,146] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,146] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,146] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,147] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,147] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,148] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:12:22,149] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:12:22,149] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,150] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,154] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,154] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,154] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,156] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:22,156] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:12:22,159] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:12:22,160] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,160] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,161] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:12:22,161] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:12:22,166] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,167] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,170] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,171] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:22,173] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,173] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:22,183] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:22,184] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,185] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,186] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,233] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,233] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,234] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,234] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,241] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,242] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:22,255] jepsen worker 51 - jepsen.util 591 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,256] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,256] jepsen worker 14 - jepsen.util 284 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,256] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:22,256] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:22,257] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:22,258] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:22,258] jepsen worker 44 - jepsen.util 584 :invoke :write [14 47] INFO [2022-03-13 17:12:22,263] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,263] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,264] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,264] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,264] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:12:22,265] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:12:22,272] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:22,273] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,275] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,279] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:22,282] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:22,282] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,286] jepsen worker 70 - jepsen.util 700 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,286] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,286] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,286] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,286] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,287] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:22,288] jepsen worker 78 - jepsen.util 528 :invoke :write [26 10] INFO [2022-03-13 17:12:22,289] jepsen worker 70 - jepsen.util 700 :invoke :write [23 18] INFO [2022-03-13 17:12:22,290] jepsen worker 80 - jepsen.util 710 :invoke :write [26 12] INFO [2022-03-13 17:12:22,291] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:22,298] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:22,299] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:12:22,301] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,302] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:22,303] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,303] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:12:22,323] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,324] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:12:22,377] jepsen worker 69 - jepsen.util 609 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,377] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,378] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,378] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,378] jepsen worker 69 - jepsen.util 609 :invoke :write [23 26] INFO [2022-03-13 17:12:22,383] jepsen worker 21 - jepsen.util 741 :invoke :write [7 36] INFO [2022-03-13 17:12:22,384] jepsen worker 79 - jepsen.util 979 :invoke :write [26 11] INFO [2022-03-13 17:12:22,385] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:12:22,399] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,399] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,400] jepsen worker 13 - jepsen.util 643 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,402] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:22,404] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:12:22,404] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:22,442] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,443] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:12:22,449] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,449] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,476] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:22,476] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:12:22,484] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:22,484] jepsen worker 76 - jepsen.util 706 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,484] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,485] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:22,486] jepsen worker 43 - jepsen.util 493 :invoke :write [14 35] INFO [2022-03-13 17:12:22,486] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:22,494] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,495] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:12:22,502] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,503] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,503] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:12:22,503] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,503] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,504] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,504] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,505] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:12:22,509] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,509] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,509] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 26] INFO [2022-03-13 17:12:22,510] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,511] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,512] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,513] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,513] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:12:22,521] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,522] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:12:22,538] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:22,539] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,551] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,552] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:12:22,558] jepsen worker 56 - jepsen.util 2126 :info :write [18 4] :RequestTimeoutException INFO [2022-03-13 17:12:22,559] jepsen worker 56 - jepsen.util 2216 :invoke :write [18 47] INFO [2022-03-13 17:12:22,569] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,569] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:22,569] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:22,570] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,600] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,600] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:12:22,618] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,618] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:22,619] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,619] jepsen worker 49 - jepsen.util 1849 :info :write [16 11] :RequestTimeoutException INFO [2022-03-13 17:12:22,619] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:22,620] jepsen worker 70 - jepsen.util 700 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,620] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,620] jepsen worker 49 - jepsen.util 1939 :invoke :read [16 nil] INFO [2022-03-13 17:12:22,622] jepsen worker 70 - jepsen.util 700 :invoke :write [23 47] INFO [2022-03-13 17:12:22,622] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:12:22,649] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,650] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:12:22,661] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,662] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:12:22,672] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,672] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,675] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:22,676] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:12:22,677] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:22,678] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 12] INFO [2022-03-13 17:12:22,684] jepsen worker 36 - jepsen.util 2106 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,684] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 16] INFO [2022-03-13 17:12:22,687] jepsen worker 6 - jepsen.util 636 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,687] jepsen worker 12 - jepsen.util 732 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,687] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,687] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,688] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,688] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,688] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:22,689] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:22,691] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:22,692] jepsen worker 12 - jepsen.util 732 :invoke :write [4 21] INFO [2022-03-13 17:12:22,693] jepsen worker 60 - jepsen.util 600 :invoke :write [20 12] INFO [2022-03-13 17:12:22,694] jepsen worker 51 - jepsen.util 591 :invoke :write [17 38] INFO [2022-03-13 17:12:22,713] jepsen worker 78 - jepsen.util 528 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,719] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:22,763] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,764] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:12:22,770] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,771] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,782] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,783] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:12:22,794] jepsen worker 72 - jepsen.util 2052 :info :write [24 16] :RequestTimeoutException INFO [2022-03-13 17:12:22,794] jepsen worker 72 - jepsen.util 2142 :invoke :read [24 nil] INFO [2022-03-13 17:12:22,829] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,829] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,833] jepsen worker 88 - jepsen.util 1978 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,833] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,861] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:22,862] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,875] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:22,876] jepsen worker 55 - jepsen.util 1945 :info :write [18 49] :RequestTimeoutException INFO [2022-03-13 17:12:22,876] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,877] jepsen worker 55 - jepsen.util 2035 :invoke :read [18 nil] INFO [2022-03-13 17:12:22,885] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,886] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:22,902] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5614035087719298}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:22,902] jepsen worker nemesis - couchbase.util Rebalance check count: 253 INFO [2022-03-13 17:12:22,910] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,915] jepsen worker nemesis - couchbase.util Rebalance stuck count: 162 INFO [2022-03-13 17:12:22,915] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:22,945] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,946] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:22,953] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:22,953] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:12:22,966] jepsen worker 19 - jepsen.util 1999 :info :write [6 31] :RequestTimeoutException INFO [2022-03-13 17:12:22,967] jepsen worker 19 - jepsen.util 2089 :invoke :read [6 nil] INFO [2022-03-13 17:12:22,972] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:22,973] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:22,986] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:22,986] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:12:23,003] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,004] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,027] jepsen worker 36 - jepsen.util 2106 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,027] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,027] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,027] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:12:23,028] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,028] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:12:23,029] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,029] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 21] INFO [2022-03-13 17:12:23,044] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,050] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,050] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,051] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:23,051] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:12:23,052] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,053] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,055] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:23,059] jepsen worker 69 - jepsen.util 609 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,060] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:23,066] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,067] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:23,071] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,072] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:23,086] jepsen worker 80 - jepsen.util 710 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,086] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,086] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,086] jepsen worker 52 - jepsen.util 592 :invoke :write [17 25] INFO [2022-03-13 17:12:23,087] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:23,087] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:23,108] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,108] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,108] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:12:23,109] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,123] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,123] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,123] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:12:23,124] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:12:23,140] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,141] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:12:23,144] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,145] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,148] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,149] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,151] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,151] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,152] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,153] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,154] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,154] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,159] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,159] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:12:23,160] jepsen worker 31 - jepsen.util 2011 :info :write [10 41] :RequestTimeoutException INFO [2022-03-13 17:12:23,160] jepsen worker 31 - jepsen.util 2101 :invoke :write [10 6] INFO [2022-03-13 17:12:23,187] jepsen worker 1 - jepsen.util 1801 :info :write [0 31] :RequestTimeoutException INFO [2022-03-13 17:12:23,188] jepsen worker 1 - jepsen.util 1891 :invoke :read [0 nil] INFO [2022-03-13 17:12:23,189] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:23,189] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:12:23,210] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,211] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:23,221] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,222] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,243] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,244] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,244] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,245] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:12:23,245] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,246] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,246] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:12:23,247] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:12:23,248] jepsen worker 70 - jepsen.util 700 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,248] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,249] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:23,249] jepsen worker 70 - jepsen.util 700 :invoke :write [23 11] INFO [2022-03-13 17:12:23,251] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,251] jepsen worker 78 - jepsen.util 528 :invoke :write [26 13] INFO [2022-03-13 17:12:23,276] jepsen worker 42 - jepsen.util 402 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,277] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,276] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,276] jepsen worker 44 - jepsen.util 584 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,278] jepsen worker 7 - jepsen.util 637 :invoke :write [2 0] INFO [2022-03-13 17:12:23,279] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:23,279] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,279] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,279] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,280] jepsen worker 44 - jepsen.util 584 :invoke :write [14 39] INFO [2022-03-13 17:12:23,281] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,281] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:23,282] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:23,285] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:23,285] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,286] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,301] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,302] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:12:23,314] jepsen worker 36 - jepsen.util 2106 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,314] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 43] INFO [2022-03-13 17:12:23,319] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,319] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,319] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:23,320] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,321] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,322] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,322] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,323] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,323] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,324] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,326] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,326] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,327] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,328] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,330] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,331] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,333] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,333] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,334] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:12:23,335] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:12:23,351] jepsen worker 79 - jepsen.util 979 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,352] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:23,355] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,355] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,379] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,380] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:12:23,388] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,388] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,389] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:12:23,389] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,390] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 36] INFO [2022-03-13 17:12:23,390] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:12:23,395] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,395] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:12:23,450] jepsen worker 28 - jepsen.util 2278 :info :write [9 42] :RequestTimeoutException INFO [2022-03-13 17:12:23,450] jepsen worker 28 - jepsen.util 2368 :invoke :read [9 nil] INFO [2022-03-13 17:12:23,451] jepsen worker 52 - jepsen.util 592 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,452] jepsen worker 52 - jepsen.util 592 :invoke :write [17 19] INFO [2022-03-13 17:12:23,460] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,460] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,461] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:23,461] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:12:23,476] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,476] jepsen worker 26 - jepsen.util 476 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,478] jepsen worker 8 - jepsen.util 728 :invoke :write [2 25] INFO [2022-03-13 17:12:23,478] jepsen worker 26 - jepsen.util 476 :invoke :write [8 44] INFO [2022-03-13 17:12:23,499] jepsen worker 21 - jepsen.util 741 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,499] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,506] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,507] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:23,515] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,515] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,517] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:12:23,517] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,526] jepsen worker 12 - jepsen.util 732 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,527] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:23,538] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,539] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,539] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,539] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,541] jepsen worker 34 - jepsen.util 754 :invoke :write [11 16] INFO [2022-03-13 17:12:23,542] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:23,567] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,568] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,588] jepsen worker 3 - jepsen.util 1893 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,588] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:12:23,599] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,599] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,601] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,601] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:23,602] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,603] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:12:23,605] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,606] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,610] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,610] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:12:23,616] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,616] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,622] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,622] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,623] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,623] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,623] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,623] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:12:23,624] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:12:23,625] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:23,626] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,627] jepsen worker 69 - jepsen.util 609 :invoke :write [23 38] INFO [2022-03-13 17:12:23,629] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,630] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:12:23,648] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,648] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,652] jepsen worker 53 - jepsen.util 503 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,653] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,653] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,653] jepsen worker 36 - jepsen.util 2106 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,653] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,653] jepsen worker 53 - jepsen.util 503 :invoke :write [17 23] INFO [2022-03-13 17:12:23,655] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,656] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 39] INFO [2022-03-13 17:12:23,657] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:23,658] jepsen worker 62 - jepsen.util 512 :invoke :write [20 45] INFO [2022-03-13 17:12:23,675] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:23,676] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 29] INFO [2022-03-13 17:12:23,684] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,684] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,685] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:23,685] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,706] jepsen worker 78 - jepsen.util 528 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,707] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:23,709] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,709] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,722] jepsen worker 60 - jepsen.util 600 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,723] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:23,724] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,724] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,724] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:23,725] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,725] jepsen worker 79 - jepsen.util 979 :invoke :write [26 24] INFO [2022-03-13 17:12:23,725] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,726] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,727] jepsen worker 21 - jepsen.util 741 :invoke :write [7 35] INFO [2022-03-13 17:12:23,727] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,728] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,728] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,728] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:12:23,729] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,729] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:12:23,732] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,732] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,758] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:23,759] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:12:23,764] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,764] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,765] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:23,765] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:23,777] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,777] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,778] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:12:23,779] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 36] INFO [2022-03-13 17:12:23,790] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,790] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,792] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:12:23,792] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 48] INFO [2022-03-13 17:12:23,796] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:23,797] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:12:23,802] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,802] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,803] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:12:23,803] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,814] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,815] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,818] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,818] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,830] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,830] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,831] jepsen worker 88 - jepsen.util 1978 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,832] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:12:23,835] jepsen worker 69 - jepsen.util 609 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,836] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:12:23,845] jepsen worker 70 - jepsen.util 700 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,845] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,845] jepsen worker 70 - jepsen.util 700 :invoke :write [23 32] INFO [2022-03-13 17:12:23,846] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:23,866] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,866] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:23,874] jepsen worker 43 - jepsen.util 493 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,874] jepsen worker 37 - jepsen.util 2017 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,875] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,875] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,875] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:23,875] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:23,876] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:12:23,877] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:23,877] jepsen worker 6 - jepsen.util 636 :invoke :write [2 5] INFO [2022-03-13 17:12:23,879] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:23,908] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,908] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,908] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:23,909] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,912] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,912] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,913] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:23,913] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,914] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,915] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,919] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:12:23,919] jepsen worker 3 - jepsen.util 1893 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,919] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,920] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:12:23,921] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5672514619883041}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:23,921] jepsen worker nemesis - couchbase.util Rebalance check count: 254 INFO [2022-03-13 17:12:23,921] jepsen worker nemesis - couchbase.util Rebalance stuck count: 162 INFO [2022-03-13 17:12:23,924] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,925] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,927] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,927] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,927] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,927] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,927] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:12:23,928] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:23,929] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,931] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,932] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:23,932] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:12:23,933] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,935] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:12:23,950] jepsen worker 51 - jepsen.util 591 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,950] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,950] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,951] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,951] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:12:23,951] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,951] jepsen worker 77 - jepsen.util 707 :invoke :write [25 11] INFO [2022-03-13 17:12:23,952] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:23,953] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:23,955] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:23,974] jepsen worker 21 - jepsen.util 741 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,975] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:23,979] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,979] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:23,980] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:12:23,980] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:24,011] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:24,011] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:24,034] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,035] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:24,042] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,042] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:24,044] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,045] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:24,050] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,051] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:24,066] jepsen worker 65 - jepsen.util 2495 :info :write [21 23] :RequestTimeoutException INFO [2022-03-13 17:12:24,067] jepsen worker 65 - jepsen.util 2585 :invoke :read [21 nil] INFO [2022-03-13 17:12:24,073] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,074] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:12:24,079] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,080] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:24,091] jepsen worker 44 - jepsen.util 584 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,091] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,091] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,091] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,091] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,091] jepsen worker 44 - jepsen.util 584 :invoke :write [14 9] INFO [2022-03-13 17:12:24,091] jepsen worker 7 - jepsen.util 637 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,092] jepsen worker 12 - jepsen.util 732 :invoke :write [4 32] INFO [2022-03-13 17:12:24,092] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:24,093] jepsen worker 7 - jepsen.util 637 :invoke :write [2 12] INFO [2022-03-13 17:12:24,093] jepsen worker 76 - jepsen.util 706 :invoke :write [25 1] INFO [2022-03-13 17:12:24,094] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:24,105] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:24,106] jepsen worker 36 - jepsen.util 2106 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,107] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:24,108] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:24,132] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,133] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:12:24,149] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,150] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,197] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,198] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:12:24,203] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,203] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:12:24,211] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,212] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:12:24,213] jepsen worker 8 - jepsen.util 728 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,214] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:24,217] jepsen worker 79 - jepsen.util 979 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,218] jepsen worker 79 - jepsen.util 979 :invoke :write [26 19] INFO [2022-03-13 17:12:24,236] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:24,237] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:12:24,241] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:24,241] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:24,242] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,242] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,243] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:24,244] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,247] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,248] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,249] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,249] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,249] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,250] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,251] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:24,252] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:12:24,252] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:24,255] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:24,265] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,265] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:24,278] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,279] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:24,279] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,279] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,287] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:24,287] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 43] INFO [2022-03-13 17:12:24,305] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,305] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,305] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:24,306] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:24,341] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,342] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:12:24,345] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,345] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,345] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,346] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:12:24,346] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:24,347] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:12:24,347] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,348] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:12:24,350] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,350] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:12:24,358] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,359] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:24,392] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,392] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,392] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,393] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:12:24,393] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:24,394] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 12] INFO [2022-03-13 17:12:24,410] jepsen worker 26 - jepsen.util 476 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,410] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,410] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:24,411] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:24,424] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,424] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,424] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,424] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:24,425] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:24,425] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:24,426] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,426] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:12:24,496] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,496] jepsen worker 77 - jepsen.util 707 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,496] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,497] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:24,497] jepsen worker 60 - jepsen.util 600 :invoke :write [20 14] INFO [2022-03-13 17:12:24,498] jepsen worker 51 - jepsen.util 591 :invoke :write [17 23] INFO [2022-03-13 17:12:24,499] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,500] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:24,511] jepsen worker 79 - jepsen.util 979 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,512] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,512] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,512] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,512] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,512] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,513] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,513] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:12:24,514] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,515] jepsen worker 21 - jepsen.util 741 :invoke :write [7 7] INFO [2022-03-13 17:12:24,515] jepsen worker 79 - jepsen.util 979 :invoke :write [26 31] INFO [2022-03-13 17:12:24,516] jepsen worker 69 - jepsen.util 609 :invoke :write [23 23] INFO [2022-03-13 17:12:24,516] jepsen worker 24 - jepsen.util 564 :invoke :write [8 19] INFO [2022-03-13 17:12:24,517] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:24,525] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,526] jepsen worker 33 - jepsen.util 663 :invoke :write [11 1] INFO [2022-03-13 17:12:24,527] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,527] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:12:24,565] jepsen worker 53 - jepsen.util 503 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,566] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:24,586] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,586] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,586] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:24,587] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,595] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,595] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,596] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,596] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:24,597] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,597] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,598] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:24,598] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:12:24,598] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,599] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:12:24,614] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,614] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:24,708] jepsen worker 44 - jepsen.util 584 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,708] jepsen worker 44 - jepsen.util 584 :invoke :write [14 6] INFO [2022-03-13 17:12:24,710] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,711] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:24,776] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,776] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,776] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:12:24,777] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:12:24,783] jepsen worker 79 - jepsen.util 979 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,783] jepsen worker 79 - jepsen.util 979 :invoke :write [26 0] INFO [2022-03-13 17:12:24,901] jepsen worker 62 - jepsen.util 512 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,901] jepsen worker 76 - jepsen.util 706 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,902] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:24,902] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:24,908] jepsen worker 69 - jepsen.util 609 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,908] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:12:24,909] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,909] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:12:24,927] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5672514619883041}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:24,927] jepsen worker nemesis - couchbase.util Rebalance check count: 255 INFO [2022-03-13 17:12:24,927] jepsen worker nemesis - couchbase.util Rebalance stuck count: 163 INFO [2022-03-13 17:12:24,952] jepsen worker 88 - jepsen.util 1978 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,953] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:24,984] jepsen worker 24 - jepsen.util 564 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:24,985] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:12:25,010] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,011] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,015] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,015] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,016] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,017] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:12:25,049] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:25,049] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,057] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,057] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:25,058] jepsen worker 21 - jepsen.util 741 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,059] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:25,062] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,063] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:12:25,065] jepsen worker 6 - jepsen.util 636 :fail :write [2 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,065] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:25,129] jepsen worker 36 - jepsen.util 2106 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,130] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,144] jepsen worker 12 - jepsen.util 732 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,144] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,144] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,144] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:25,145] jepsen worker 77 - jepsen.util 707 :invoke :write [25 49] INFO [2022-03-13 17:12:25,146] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:25,153] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,153] jepsen worker 52 - jepsen.util 592 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,153] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,153] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:25,154] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:25,155] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,155] jepsen worker 13 - jepsen.util 643 :invoke :write [4 36] INFO [2022-03-13 17:12:25,155] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:25,156] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:25,157] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:12:25,166] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,166] jepsen worker 33 - jepsen.util 663 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,166] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,167] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:25,167] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,167] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:12:25,172] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:25,175] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:12:25,213] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,214] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,219] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,219] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:12:25,253] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,254] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:25,271] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,272] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,272] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:12:25,273] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:12:25,275] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:25,275] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:12:25,285] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:25,286] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:12:25,334] jepsen worker 30 - jepsen.util 2100 :info :write [10 20] :RequestTimeoutException INFO [2022-03-13 17:12:25,335] jepsen worker 30 - jepsen.util 2190 :invoke :write [10 34] INFO [2022-03-13 17:12:25,356] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,356] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,356] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,357] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 13] INFO [2022-03-13 17:12:25,365] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,365] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,366] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,367] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:12:25,408] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,408] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:12:25,417] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,418] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:25,468] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,468] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,468] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:12:25,469] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:12:25,481] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,482] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,520] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:25,520] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,536] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,537] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:25,591] jepsen worker 36 - jepsen.util 2106 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,592] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,592] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,592] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:12:25,612] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,615] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:25,635] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,636] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,636] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:25,637] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:25,647] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,647] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,648] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:12:25,649] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:25,650] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,651] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:25,651] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,652] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:12:25,652] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,653] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:12:25,675] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,677] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:25,712] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,713] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,738] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:25,739] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:12:25,748] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,749] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,752] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,752] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,753] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,753] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,754] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:25,754] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:25,795] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,795] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 43] INFO [2022-03-13 17:12:25,825] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,825] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,827] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,828] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:25,828] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,829] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:12:25,849] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,849] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:25,863] jepsen worker 88 - jepsen.util 1978 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,863] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:12:25,893] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,894] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:25,927] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,927] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,928] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:12:25,928] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,933] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5730994152046784}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:25,933] jepsen worker nemesis - couchbase.util Rebalance check count: 256 INFO [2022-03-13 17:12:25,933] jepsen worker nemesis - couchbase.util Rebalance stuck count: 163 WARN [2022-03-13 17:12:25,935] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18051815,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25152591,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9102319,"encode_duration_us":25012,"operation_id":"0x5a77","timeout_ms":10000,"last_server_duration_us":9077736,"total_duration_us":10007225}],"total_count":1}} WARN [2022-03-13 17:12:25,935] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17806,"operation_id":"0x61cc","timeout_ms":10000,"total_duration_us":10010356},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":20754,"operation_id":"0x61bf","timeout_ms":10000,"total_duration_us":10007908},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":20945,"operation_id":"0x636b","timeout_ms":10000,"total_duration_us":10007306},{"operation_name":"upsert","last_dispatch_duration_us":33769,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":886730,"total_server_duration_us":501325,"encode_duration_us":18418,"operation_id":"0x646f","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10003677},{"operation_name":"upsert","last_dispatch_duration_us":713344,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":713344,"total_server_duration_us":78979,"encode_duration_us":31170,"operation_id":"0x65a0","timeout_ms":10000,"last_server_duration_us":78979,"total_duration_us":717180},{"operation_name":"upsert","last_dispatch_duration_us":698260,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":698260,"total_server_duration_us":92740,"encode_duration_us":17573,"operation_id":"0x6a30","timeout_ms":10000,"last_server_duration_us":92740,"total_duration_us":701995},{"operation_name":"upsert","last_dispatch_duration_us":695798,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":695798,"total_server_duration_us":221245,"encode_duration_us":17515,"operation_id":"0x66af","timeout_ms":10000,"last_server_duration_us":221245,"total_duration_us":699838},{"operation_name":"upsert","last_dispatch_duration_us":682614,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":682614,"total_server_duration_us":670904,"encode_duration_us":21316,"operation_id":"0x679f","timeout_ms":10000,"last_server_duration_us":670904,"total_duration_us":688735},{"operation_name":"upsert","last_dispatch_duration_us":653205,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":653205,"total_server_duration_us":494060,"encode_duration_us":18947,"operation_id":"0x668e","timeout_ms":10000,"last_server_duration_us":494060,"total_duration_us":657683},{"operation_name":"upsert","last_dispatch_duration_us":618435,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":618435,"total_server_duration_us":540907,"encode_duration_us":18919,"operation_id":"0x6836","timeout_ms":10000,"last_server_duration_us":540907,"total_duration_us":624611}],"total_count":15}} INFO [2022-03-13 17:12:25,941] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,941] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:25,979] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:25,980] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:25,983] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:25,984] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:12:26,002] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,002] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:26,042] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:26,042] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:12:26,054] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,054] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] INFO [2022-03-13 17:12:26,074] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,076] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,079] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,080] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:12:26,083] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,083] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:12:26,086] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,087] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:26,090] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,091] jepsen worker 7 - jepsen.util 637 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,091] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,091] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,091] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,091] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:26,091] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,091] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:26,092] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,092] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,092] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,092] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:12:26,093] jepsen worker 61 - jepsen.util 601 :invoke :write [20 3] INFO [2022-03-13 17:12:26,094] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:26,094] jepsen worker 14 - jepsen.util 284 :invoke :write [4 13] INFO [2022-03-13 17:12:26,095] jepsen worker 8 - jepsen.util 728 :invoke :write [2 29] INFO [2022-03-13 17:12:26,096] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:26,096] jepsen worker 26 - jepsen.util 476 :invoke :write [8 22] INFO [2022-03-13 17:12:26,097] jepsen worker 43 - jepsen.util 493 :invoke :write [14 18] INFO [2022-03-13 17:12:26,099] jepsen worker 54 - jepsen.util 1764 :ok :read [18 21] INFO [2022-03-13 17:12:26,099] jepsen worker 74 - jepsen.util 2594 :ok :read [24 24] INFO [2022-03-13 17:12:26,101] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,101] jepsen worker 42 - jepsen.util 402 :invoke :write [14 31] INFO [2022-03-13 17:12:26,104] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:26,104] jepsen worker 74 - jepsen.util 2594 :invoke :write [24 15] INFO [2022-03-13 17:12:26,110] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:26,111] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:26,112] jepsen worker 54 - jepsen.util 1764 :invoke :read [18 nil] INFO [2022-03-13 17:12:26,119] jepsen worker 51 - jepsen.util 591 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,120] jepsen worker 51 - jepsen.util 591 :invoke :write [17 21] INFO [2022-03-13 17:12:26,128] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,128] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,129] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,129] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,130] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:26,131] jepsen worker 53 - jepsen.util 503 :invoke :write [17 15] INFO [2022-03-13 17:12:26,133] jepsen worker 6 - jepsen.util 636 :invoke :write [2 33] INFO [2022-03-13 17:12:26,134] jepsen worker 62 - jepsen.util 512 :invoke :write [20 49] INFO [2022-03-13 17:12:26,141] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,142] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:26,149] jepsen worker 10 - jepsen.util 2080 :ok :read [3 25] INFO [2022-03-13 17:12:26,149] jepsen worker 10 - jepsen.util 2080 :invoke :read [3 nil] INFO [2022-03-13 17:12:26,153] jepsen worker 11 - jepsen.util 2081 :ok :read [3 25] INFO [2022-03-13 17:12:26,153] jepsen worker 11 - jepsen.util 2081 :invoke :read [3 nil] INFO [2022-03-13 17:12:26,169] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,169] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,170] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:12:26,171] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:12:26,178] jepsen worker 60 - jepsen.util 600 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,179] jepsen worker 60 - jepsen.util 600 :invoke :write [20 42] INFO [2022-03-13 17:12:26,184] jepsen worker 73 - jepsen.util 2143 :ok :read [24 24] INFO [2022-03-13 17:12:26,185] jepsen worker 73 - jepsen.util 2143 :invoke :write [24 24] INFO [2022-03-13 17:12:26,202] jepsen worker 55 - jepsen.util 2035 :ok :read [18 21] INFO [2022-03-13 17:12:26,203] jepsen worker 55 - jepsen.util 2035 :invoke :write [18 32] INFO [2022-03-13 17:12:26,211] jepsen worker 79 - jepsen.util 979 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,212] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:26,215] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:26,216] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:26,218] jepsen worker 26 - jepsen.util 476 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,219] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:26,235] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,235] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,235] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:26,236] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:12:26,244] jepsen worker 54 - jepsen.util 1764 :ok :read [18 21] INFO [2022-03-13 17:12:26,244] jepsen worker 54 - jepsen.util 1764 :invoke :write [18 3] INFO [2022-03-13 17:12:26,268] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,270] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,271] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,273] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:26,284] jepsen worker 44 - jepsen.util 584 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,284] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,284] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,285] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,285] jepsen worker 12 - jepsen.util 732 :invoke :write [4 34] INFO [2022-03-13 17:12:26,287] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:26,288] jepsen worker 76 - jepsen.util 706 :invoke :write [25 13] INFO [2022-03-13 17:12:26,288] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:26,292] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:26,299] jepsen worker 10 - jepsen.util 2080 :ok :read [3 25] INFO [2022-03-13 17:12:26,299] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:26,300] jepsen worker 10 - jepsen.util 2080 :invoke :write [3 38] INFO [2022-03-13 17:12:26,301] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,302] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:12:26,307] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,308] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:26,308] jepsen worker 11 - jepsen.util 2081 :ok :read [3 25] INFO [2022-03-13 17:12:26,309] jepsen worker 11 - jepsen.util 2081 :invoke :write [3 20] INFO [2022-03-13 17:12:26,319] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,320] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:26,334] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,335] jepsen worker 69 - jepsen.util 609 :invoke :write [23 41] INFO [2022-03-13 17:12:26,335] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:26,336] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 8] INFO [2022-03-13 17:12:26,374] jepsen worker 43 - jepsen.util 493 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,374] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,375] jepsen worker 43 - jepsen.util 493 :invoke :write [14 39] INFO [2022-03-13 17:12:26,375] jepsen worker 7 - jepsen.util 637 :invoke :write [2 30] INFO [2022-03-13 17:12:26,378] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,378] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,378] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:26,379] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:26,379] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,380] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:12:26,396] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,396] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,396] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:12:26,397] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:26,398] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,399] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:26,405] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,406] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:12:26,421] jepsen worker 77 - jepsen.util 707 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,422] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:26,426] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,426] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,426] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,427] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:26,428] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:12:26,429] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:26,436] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,436] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,436] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,437] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:26,441] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:12:26,441] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 26] INFO [2022-03-13 17:12:26,452] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,452] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:12:26,460] jepsen worker 88 - jepsen.util 1978 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,463] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 14] INFO [2022-03-13 17:12:26,491] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:26,492] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:26,518] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,519] jepsen worker 33 - jepsen.util 663 :invoke :write [11 11] INFO [2022-03-13 17:12:26,542] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,543] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:26,545] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,546] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,549] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,549] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:12:26,572] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,573] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:12:26,586] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,586] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,587] jepsen worker 79 - jepsen.util 979 :invoke :write [26 32] INFO [2022-03-13 17:12:26,588] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:12:26,616] jepsen worker 3 - jepsen.util 1893 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,616] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:26,636] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,636] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,637] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:12:26,637] jepsen worker 34 - jepsen.util 754 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,637] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,638] jepsen worker 61 - jepsen.util 601 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,638] jepsen worker 88 - jepsen.util 1978 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,640] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:26,641] jepsen worker 34 - jepsen.util 754 :invoke :write [11 29] INFO [2022-03-13 17:12:26,641] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:12:26,643] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:26,643] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:12:26,643] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,648] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:26,649] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:26,649] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:12:26,653] jepsen worker 69 - jepsen.util 609 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,654] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,654] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,655] jepsen worker 21 - jepsen.util 741 :invoke :write [7 41] INFO [2022-03-13 17:12:26,656] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:12:26,659] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:26,687] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,688] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:26,704] jepsen worker 58 - jepsen.util 1858 :info :write [19 3] :SyncWriteAmbiguous INFO [2022-03-13 17:12:26,704] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,705] jepsen worker 58 - jepsen.util 1948 :invoke :write [19 0] INFO [2022-03-13 17:12:26,706] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:12:26,720] jepsen worker 51 - jepsen.util 591 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,721] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:26,729] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,730] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:12:26,755] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,755] jepsen worker 75 - jepsen.util 435 :invoke :write [25 49] INFO [2022-03-13 17:12:26,758] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,758] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,759] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:12:26,759] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:12:26,768] jepsen worker 50 - jepsen.util 2030 :info :write [16 45] :SyncWriteAmbiguous INFO [2022-03-13 17:12:26,769] jepsen worker 50 - jepsen.util 2120 :invoke :write [16 22] INFO [2022-03-13 17:12:26,778] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,779] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:26,781] jepsen worker 42 - jepsen.util 402 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,782] jepsen worker 42 - jepsen.util 402 :invoke :write [14 45] INFO [2022-03-13 17:12:26,793] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,793] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:12:26,855] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,856] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:12:26,862] jepsen worker 33 - jepsen.util 663 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,863] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:26,867] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,868] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,871] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,872] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,877] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,878] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,879] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,880] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,881] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,882] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,884] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,884] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:26,886] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,887] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:12:26,895] jepsen worker 60 - jepsen.util 600 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,896] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:26,896] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:26,896] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,897] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,897] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:26,897] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:12:26,898] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:26,938] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,939] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5730994152046784}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:26,939] jepsen worker nemesis - couchbase.util Rebalance check count: 257 INFO [2022-03-13 17:12:26,939] jepsen worker nemesis - couchbase.util Rebalance stuck count: 164 INFO [2022-03-13 17:12:26,939] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:12:26,946] jepsen worker 17 - jepsen.util 1997 :info :write [5 3] :RequestTimeoutException INFO [2022-03-13 17:12:26,947] jepsen worker 17 - jepsen.util 2087 :invoke :write [5 29] INFO [2022-03-13 17:12:26,968] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,968] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,969] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 41] INFO [2022-03-13 17:12:26,969] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:12:26,969] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,970] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 1] INFO [2022-03-13 17:12:26,973] jepsen worker 79 - jepsen.util 979 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,974] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:26,982] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,983] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:26,995] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,996] jepsen worker 14 - jepsen.util 284 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:26,996] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:26,997] jepsen worker 14 - jepsen.util 284 :invoke :write [4 32] INFO [2022-03-13 17:12:27,018] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:27,018] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:27,055] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,056] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:27,062] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,063] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:12:27,066] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,067] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:27,069] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,070] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:27,073] jepsen worker 12 - jepsen.util 732 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,074] jepsen worker 12 - jepsen.util 732 :invoke :write [4 35] INFO [2022-03-13 17:12:27,077] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,078] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:12:27,082] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,082] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,084] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:27,085] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:12:27,103] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:27,104] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:27,112] jepsen worker 83 - jepsen.util 1523 :ok :read [27 4] INFO [2022-03-13 17:12:27,113] jepsen worker 83 - jepsen.util 1523 :invoke :read [27 nil] INFO [2022-03-13 17:12:27,151] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,152] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:12:27,168] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,168] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:12:27,169] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,169] jepsen worker 8 - jepsen.util 728 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,170] jepsen worker 8 - jepsen.util 728 :invoke :write [2 38] INFO [2022-03-13 17:12:27,170] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:27,182] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,183] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:27,184] jepsen worker 85 - jepsen.util 2065 :ok :read [28 44] INFO [2022-03-13 17:12:27,185] jepsen worker 85 - jepsen.util 2065 :invoke :read [28 nil] INFO [2022-03-13 17:12:27,201] jepsen worker 21 - jepsen.util 741 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,202] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:27,220] jepsen worker 37 - jepsen.util 2017 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,220] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:27,251] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,251] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:27,261] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,261] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,262] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:27,262] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:27,280] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,280] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:27,318] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:27,319] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:12:27,325] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,325] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,326] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,327] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:12:27,328] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:27,329] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:12:27,330] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,330] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:12:27,336] jepsen worker 36 - jepsen.util 2106 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,337] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:12:27,348] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,348] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:27,351] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,352] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:27,423] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,424] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 39] INFO [2022-03-13 17:12:27,426] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,426] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,427] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:12:27,428] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] WARN [2022-03-13 17:12:27,432] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":23078657,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23078657,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9045859,"encode_duration_us":17350,"operation_id":"0x5b1b","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10002892},{"operation_name":"upsert","last_dispatch_duration_us":26238830,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26238830,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9027816,"encode_duration_us":17227,"operation_id":"0x5a8f","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10007958}],"total_count":2}} WARN [2022-03-13 17:12:27,445] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":25471850,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25471850,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9055417,"encode_duration_us":24108,"operation_id":"0x5b7f","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10003708},{"operation_name":"upsert","last_dispatch_duration_us":18159158,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20939689,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9099084,"encode_duration_us":18007,"operation_id":"0x5c8d","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10006379},{"operation_name":"upsert","last_dispatch_duration_us":18102624,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19671287,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9073636,"encode_duration_us":17638,"operation_id":"0x5d16","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10005238}],"total_count":3}} INFO [2022-03-13 17:12:27,486] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,487] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:27,496] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,497] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:27,502] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,503] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:27,504] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,504] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] WARN [2022-03-13 17:12:27,539] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13190390,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13190390,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9035244,"encode_duration_us":30896,"operation_id":"0x644f","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10009624}],"total_count":1}} INFO [2022-03-13 17:12:27,544] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,545] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] WARN [2022-03-13 17:12:27,545] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17135573,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17135573,"total_server_duration_us":27,"operation_id":"0x5d72","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":17135813},{"operation_name":"get","last_dispatch_duration_us":15085090,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":15085090,"total_server_duration_us":38,"operation_id":"0x602e","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":15087637},{"operation_name":"get","last_dispatch_duration_us":11545664,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11545664,"total_server_duration_us":15,"operation_id":"0x6226","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":11545996},{"operation_name":"upsert","last_dispatch_duration_us":118694,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5563158,"total_server_duration_us":294423,"encode_duration_us":17977,"operation_id":"0x6423","timeout_ms":10000,"last_server_duration_us":32355,"total_duration_us":10003909},{"operation_name":"upsert","last_dispatch_duration_us":63041,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8474492,"total_server_duration_us":336329,"encode_duration_us":21238,"operation_id":"0x6493","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10003360},{"operation_name":"upsert","last_dispatch_duration_us":9080117,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9131584,"total_server_duration_us":9067601,"encode_duration_us":89398,"operation_id":"0x6670","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9146809},{"operation_name":"get","last_dispatch_duration_us":9110059,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9110059,"total_server_duration_us":11,"operation_id":"0x63a5","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":9112442},{"operation_name":"get","last_dispatch_duration_us":5647587,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":5647587,"total_server_duration_us":62,"operation_id":"0x653b","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":5649701},{"operation_name":"upsert","last_dispatch_duration_us":1872129,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1872129,"total_server_duration_us":86769,"encode_duration_us":18740,"operation_id":"0x6a1b","timeout_ms":10000,"last_server_duration_us":86769,"total_duration_us":1878381},{"operation_name":"get","last_dispatch_duration_us":914064,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":914064,"total_server_duration_us":69,"operation_id":"0x673b","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":915495}],"total_count":27}} INFO [2022-03-13 17:12:27,572] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,573] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:12:27,584] jepsen worker 88 - jepsen.util 1978 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,585] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:27,613] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,614] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,614] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:12:27,614] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:27,628] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:27,629] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 40] WARN [2022-03-13 17:12:27,633] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":22814174,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":22814174,"total_server_duration_us":763726,"operation_id":"0x5b6d","timeout_ms":30000,"last_server_duration_us":763726,"total_duration_us":22814541},{"operation_name":"get","last_dispatch_duration_us":21544893,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":21544893,"total_server_duration_us":62,"operation_id":"0x5c33","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":21545258},{"operation_name":"get","last_dispatch_duration_us":18255572,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18255572,"total_server_duration_us":43,"operation_id":"0x5d5e","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":18257640},{"operation_name":"get","last_dispatch_duration_us":17570201,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":17570201,"total_server_duration_us":32,"operation_id":"0x5de7","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":17570455},{"operation_name":"upsert","last_dispatch_duration_us":10354833,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10354833,"total_server_duration_us":21082,"encode_duration_us":19630,"operation_id":"0x61ca","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10010246},{"operation_name":"upsert","last_dispatch_duration_us":18935815,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18935815,"total_server_duration_us":9034183,"encode_duration_us":28809,"operation_id":"0x6176","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10007771},{"operation_name":"upsert","last_dispatch_duration_us":8535302,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8591131,"total_server_duration_us":40883,"encode_duration_us":18260,"operation_id":"0x6416","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10005070},{"operation_name":"upsert","last_dispatch_duration_us":108066,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5979447,"total_server_duration_us":47162,"encode_duration_us":19950,"operation_id":"0x6475","timeout_ms":10000,"last_server_duration_us":26241,"total_duration_us":10003686},{"operation_name":"get","last_dispatch_duration_us":9056896,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9056896,"total_server_duration_us":49,"operation_id":"0x62a1","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9062471},{"operation_name":"get","last_dispatch_duration_us":9047691,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9047691,"total_server_duration_us":32,"operation_id":"0x62a4","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9049670}],"total_count":83}} WARN [2022-03-13 17:12:27,640] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10630829,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10630829,"total_server_duration_us":32,"operation_id":"0x6556","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":10632967},{"operation_name":"upsert","last_dispatch_duration_us":101761,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9703625,"total_server_duration_us":148079,"encode_duration_us":20996,"operation_id":"0x6135","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10010548},{"operation_name":"upsert","last_dispatch_duration_us":9151289,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17216217,"total_server_duration_us":172455,"encode_duration_us":19688,"operation_id":"0x6244","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10010222},{"operation_name":"upsert","last_dispatch_duration_us":13190390,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13190390,"total_server_duration_us":9035244,"encode_duration_us":30896,"operation_id":"0x644f","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10009624},{"operation_name":"upsert","last_dispatch_duration_us":9146532,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9754502,"total_server_duration_us":164699,"encode_duration_us":18535,"operation_id":"0x6613","timeout_ms":10000,"last_server_duration_us":29695,"total_duration_us":10006096},{"operation_name":"upsert","last_dispatch_duration_us":9179912,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9878367,"total_server_duration_us":9184589,"encode_duration_us":23549,"operation_id":"0x65ef","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":9983019},{"operation_name":"get","last_dispatch_duration_us":9041516,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9041516,"total_server_duration_us":15,"operation_id":"0x6629","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":9044129},{"operation_name":"get","last_dispatch_duration_us":9019861,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9019861,"total_server_duration_us":11,"operation_id":"0x662d","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":9032684},{"operation_name":"get","last_dispatch_duration_us":8999570,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8999570,"total_server_duration_us":23,"operation_id":"0x6628","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":9004338},{"operation_name":"get","last_dispatch_duration_us":6897463,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6897463,"total_server_duration_us":15,"operation_id":"0x6746","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":6900667}],"total_count":82}} WARN [2022-03-13 17:12:27,645] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":98245,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":98245,"total_server_duration_us":49,"operation_id":"0x5f8c","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":16995856},{"operation_name":"get","last_dispatch_duration_us":9261034,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9261034,"total_server_duration_us":500,"operation_id":"0x6059","timeout_ms":30000,"last_server_duration_us":500,"total_duration_us":14943020},{"operation_name":"upsert","last_dispatch_duration_us":185223,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":185223,"total_server_duration_us":22306,"encode_duration_us":20113,"operation_id":"0x643d","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10008196},{"operation_name":"upsert","last_dispatch_duration_us":163275,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":163275,"total_server_duration_us":22306,"encode_duration_us":16729,"operation_id":"0x6547","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10005803},{"operation_name":"upsert","last_dispatch_duration_us":9260460,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9260460,"total_server_duration_us":46965,"encode_duration_us":20397,"operation_id":"0x622c","timeout_ms":10000,"last_server_duration_us":46965,"total_duration_us":10002643},{"operation_name":"get","last_dispatch_duration_us":9173394,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9173394,"total_server_duration_us":77851,"operation_id":"0x62a9","timeout_ms":30000,"last_server_duration_us":77851,"total_duration_us":9175696},{"operation_name":"get","last_dispatch_duration_us":8988,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8988,"total_server_duration_us":49,"operation_id":"0x64c3","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":7730691},{"operation_name":"upsert","last_dispatch_duration_us":1102419,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1102419,"total_server_duration_us":99502,"encode_duration_us":18514,"operation_id":"0x6588","timeout_ms":10000,"last_server_duration_us":99502,"total_duration_us":6270130},{"operation_name":"upsert","last_dispatch_duration_us":1037598,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1037598,"total_server_duration_us":79545,"encode_duration_us":20970,"operation_id":"0x676c","timeout_ms":10000,"last_server_duration_us":79545,"total_duration_us":3153531},{"operation_name":"get","last_dispatch_duration_us":935380,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":935380,"total_server_duration_us":19,"operation_id":"0x66db","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":3053663}],"total_count":70}} WARN [2022-03-13 17:12:27,646] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18138420,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26244246,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9115346,"encode_duration_us":17342,"operation_id":"0x5b2d","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10005070},{"operation_name":"upsert","last_dispatch_duration_us":18957272,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18957272,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9049045,"encode_duration_us":20729,"operation_id":"0x5cc7","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005737},{"operation_name":"upsert","last_dispatch_duration_us":18010998,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25464931,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9140104,"encode_duration_us":20447,"operation_id":"0x5b70","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":10008189}],"total_count":3}} INFO [2022-03-13 17:12:27,675] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,676] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:27,676] jepsen worker 76 - jepsen.util 706 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,677] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:27,677] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,678] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:27,678] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,678] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:27,678] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,678] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:27,679] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:27,679] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,680] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,680] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:27,680] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:27,680] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:27,685] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,685] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,685] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,685] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,686] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:12:27,686] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:27,686] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:12:27,687] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:27,725] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,726] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:12:27,726] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,727] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:27,729] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,730] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] WARN [2022-03-13 17:12:27,731] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":82061,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9672547,"total_server_duration_us":246417,"encode_duration_us":34152,"operation_id":"0x635d","timeout_ms":10000,"last_server_duration_us":28493,"total_duration_us":10008577},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":20578,"operation_id":"0x64ef","timeout_ms":10000,"total_duration_us":10006577},{"operation_name":"upsert","last_dispatch_duration_us":216150,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9871356,"total_server_duration_us":230482,"encode_duration_us":20644,"operation_id":"0x6317","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10005925},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":23612,"operation_id":"0x63ae","timeout_ms":10000,"total_duration_us":10005661},{"operation_name":"upsert","last_dispatch_duration_us":236831,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9001882,"total_server_duration_us":69194,"encode_duration_us":19176,"operation_id":"0x63a4","timeout_ms":10000,"last_server_duration_us":40057,"total_duration_us":10005206},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":25386,"operation_id":"0x6453","timeout_ms":10000,"total_duration_us":10004253},{"operation_name":"upsert","last_dispatch_duration_us":58352,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9732712,"total_server_duration_us":210563,"encode_duration_us":27733,"operation_id":"0x63a9","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10003750},{"operation_name":"upsert","last_dispatch_duration_us":9107091,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9340363,"total_server_duration_us":9110930,"encode_duration_us":27013,"operation_id":"0x6332","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":9353934},{"operation_name":"get","last_dispatch_duration_us":8805437,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8805437,"total_server_duration_us":6927,"operation_id":"0x63a7","timeout_ms":30000,"last_server_duration_us":6927,"total_duration_us":8811760},{"operation_name":"upsert","last_dispatch_duration_us":3073868,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3073868,"total_server_duration_us":74786,"encode_duration_us":18108,"operation_id":"0x69a6","timeout_ms":10000,"last_server_duration_us":74786,"total_duration_us":3077728}],"total_count":58}} INFO [2022-03-13 17:12:27,746] jepsen worker 57 - jepsen.util 2127 :info :write [19 16] :RequestTimeoutException INFO [2022-03-13 17:12:27,746] jepsen worker 57 - jepsen.util 2217 :invoke :write [19 12] INFO [2022-03-13 17:12:27,789] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,789] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,790] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:12:27,790] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 2] INFO [2022-03-13 17:12:27,815] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,816] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 8] WARN [2022-03-13 17:12:27,831] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18014892,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18735843,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9483528,"encode_duration_us":19584,"operation_id":"0x5bcb","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006938}],"total_count":1}} INFO [2022-03-13 17:12:27,871] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,871] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,871] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,871] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:27,872] jepsen worker 79 - jepsen.util 979 :invoke :write [26 48] INFO [2022-03-13 17:12:27,872] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:12:27,930] jepsen worker 88 - jepsen.util 1978 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,931] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:12:27,933] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,934] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:27,934] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,934] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,021] jepsen worker 77 - jepsen.util 707 :invoke :write [25 2] INFO [2022-03-13 17:12:28,021] jepsen worker 60 - jepsen.util 600 :invoke :write [20 19] INFO [2022-03-13 17:12:27,934] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,934] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:27,945] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5730994152046784}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:27,976] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,007] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,025] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:28,025] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:28,009] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,025] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,009] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,009] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,025] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:28,021] jepsen worker 44 - jepsen.util 584 :invoke :write [14 0] INFO [2022-03-13 17:12:28,025] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 43] INFO [2022-03-13 17:12:28,021] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:28,025] jepsen worker 51 - jepsen.util 591 :invoke :write [17 4] INFO [2022-03-13 17:12:28,025] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:28,025] jepsen worker nemesis - couchbase.util Rebalance check count: 258 INFO [2022-03-13 17:12:28,030] jepsen worker nemesis - couchbase.util Rebalance stuck count: 165 INFO [2022-03-13 17:12:28,041] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,041] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:12:28,067] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,068] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:28,071] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,072] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:28,115] jepsen worker 89 - jepsen.util 2429 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,115] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,115] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,116] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:12:28,117] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:12:28,118] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:12:28,126] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,126] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:12:28,149] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,149] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,151] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:28,152] jepsen worker 76 - jepsen.util 706 :invoke :write [25 40] INFO [2022-03-13 17:12:28,167] jepsen worker 87 - jepsen.util 2607 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,167] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,167] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,168] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:12:28,168] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:12:28,169] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:12:28,178] jepsen worker 47 - jepsen.util 2117 :info :write [15 19] :RequestTimeoutException INFO [2022-03-13 17:12:28,178] jepsen worker 47 - jepsen.util 2207 :invoke :read [15 nil] INFO [2022-03-13 17:12:28,180] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,185] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:12:28,192] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,193] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:28,211] jepsen worker 36 - jepsen.util 2106 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,211] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,241] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,241] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 18] INFO [2022-03-13 17:12:28,256] jepsen worker 66 - jepsen.util 1776 :info :write [22 30] :SyncWriteAmbiguous INFO [2022-03-13 17:12:28,257] jepsen worker 66 - jepsen.util 1866 :invoke :write [22 21] INFO [2022-03-13 17:12:28,288] jepsen worker 77 - jepsen.util 707 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,288] jepsen worker 77 - jepsen.util 707 :invoke :write [25 8] INFO [2022-03-13 17:12:28,303] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,303] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:12:28,307] jepsen worker 42 - jepsen.util 402 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,307] jepsen worker 42 - jepsen.util 402 :invoke :write [14 46] INFO [2022-03-13 17:12:28,312] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,312] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:12:28,318] jepsen worker 63 - jepsen.util 1953 :info :write [21 42] :RequestTimeoutException INFO [2022-03-13 17:12:28,318] jepsen worker 63 - jepsen.util 2043 :invoke :read [21 nil] INFO [2022-03-13 17:12:28,328] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,329] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:12:28,345] jepsen worker 60 - jepsen.util 600 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,346] jepsen worker 60 - jepsen.util 600 :invoke :write [20 43] INFO [2022-03-13 17:12:28,353] jepsen worker 75 - jepsen.util 435 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,354] jepsen worker 75 - jepsen.util 435 :invoke :write [25 49] INFO [2022-03-13 17:12:28,370] jepsen worker 46 - jepsen.util 2116 :info :write [15 24] :RequestTimeoutException INFO [2022-03-13 17:12:28,371] jepsen worker 46 - jepsen.util 2206 :invoke :write [15 46] INFO [2022-03-13 17:12:28,378] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,379] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:12:28,387] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,388] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:28,399] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,399] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:28,412] jepsen worker 88 - jepsen.util 1978 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,413] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:12:28,438] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:28,439] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:28,447] jepsen worker 14 - jepsen.util 284 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,448] jepsen worker 14 - jepsen.util 284 :invoke :write [4 44] INFO [2022-03-13 17:12:28,453] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,453] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:28,469] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,469] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:28,485] jepsen worker 15 - jepsen.util 1995 :info :write [5 19] :SyncWriteAmbiguous INFO [2022-03-13 17:12:28,486] jepsen worker 15 - jepsen.util 2085 :invoke :write [5 21] INFO [2022-03-13 17:12:28,510] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,511] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:28,532] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,532] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,532] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,533] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 35] INFO [2022-03-13 17:12:28,533] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:12:28,534] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,536] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,537] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:28,552] jepsen worker 79 - jepsen.util 979 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,554] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:28,558] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:28,559] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:28,582] jepsen worker 44 - jepsen.util 584 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,582] jepsen worker 44 - jepsen.util 584 :invoke :write [14 4] INFO [2022-03-13 17:12:28,594] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,595] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,595] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:12:28,595] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:28,602] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,603] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:12:28,606] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,606] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:12:28,619] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,620] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 7] INFO [2022-03-13 17:12:28,622] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,623] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:12:28,629] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,629] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,629] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:28,630] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 26] INFO [2022-03-13 17:12:28,644] jepsen worker 6 - jepsen.util 636 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,645] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:28,652] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,653] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:28,654] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,654] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,655] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,655] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:28,656] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:28,656] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:12:28,665] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:28,666] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:12:28,703] jepsen worker 40 - jepsen.util 2110 :info :write [13 7] :RequestTimeoutException INFO [2022-03-13 17:12:28,704] jepsen worker 40 - jepsen.util 2200 :invoke :read [13 nil] INFO [2022-03-13 17:12:28,707] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,708] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:12:28,725] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,726] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:12:28,736] jepsen worker 36 - jepsen.util 2106 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,736] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,739] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,739] jepsen worker 51 - jepsen.util 591 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,740] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:28,741] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:12:28,751] jepsen worker 82 - jepsen.util 2062 :ok :read [27 4] INFO [2022-03-13 17:12:28,752] jepsen worker 82 - jepsen.util 2062 :invoke :read [27 nil] INFO [2022-03-13 17:12:28,763] jepsen worker 1 - jepsen.util 1891 :ok :read [0 41] INFO [2022-03-13 17:12:28,764] jepsen worker 1 - jepsen.util 1891 :invoke :write [0 42] INFO [2022-03-13 17:12:28,778] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,779] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,779] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:28,779] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:12:28,786] jepsen worker 7 - jepsen.util 637 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,787] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:28,794] jepsen worker 88 - jepsen.util 1978 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,794] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 41] INFO [2022-03-13 17:12:28,830] jepsen worker 84 - jepsen.util 2244 :info :write [28 46] :RequestTimeoutException INFO [2022-03-13 17:12:28,830] jepsen worker 84 - jepsen.util 2334 :invoke :write [28 48] INFO [2022-03-13 17:12:28,837] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,838] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:12:28,853] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,854] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,856] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,856] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:12:28,857] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,858] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,859] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,860] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,862] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,863] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:28,869] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,870] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:12:28,904] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,904] jepsen worker 75 - jepsen.util 435 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,905] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:28,905] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:28,905] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,906] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,906] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,906] jepsen worker 69 - jepsen.util 609 :invoke :write [23 31] INFO [2022-03-13 17:12:28,907] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:28,907] jepsen worker 21 - jepsen.util 741 :invoke :write [7 3] INFO [2022-03-13 17:12:28,952] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,952] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:12:28,955] jepsen worker 12 - jepsen.util 732 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,955] jepsen worker 12 - jepsen.util 732 :invoke :write [4 39] INFO [2022-03-13 17:12:28,964] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:28,964] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,018] jepsen worker 43 - jepsen.util 493 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,035] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5730994152046784}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:29,038] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:29,038] jepsen worker nemesis - couchbase.util Rebalance check count: 259 INFO [2022-03-13 17:12:29,039] jepsen worker nemesis - couchbase.util Rebalance stuck count: 166 INFO [2022-03-13 17:12:29,065] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,065] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:29,065] jepsen worker 13 - jepsen.util 643 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,066] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:29,072] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,072] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,072] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,072] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,072] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:12:29,074] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:29,074] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:29,074] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:29,105] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,105] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,106] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,107] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,108] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,109] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,110] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,111] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:12:29,147] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,148] jepsen worker 35 - jepsen.util 935 :invoke :write [11 15] INFO [2022-03-13 17:12:29,154] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,154] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,154] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,155] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,158] jepsen worker 39 - jepsen.util 2019 :info :write [13 7] :RequestTimeoutException INFO [2022-03-13 17:12:29,159] jepsen worker 39 - jepsen.util 2109 :invoke :read [13 nil] INFO [2022-03-13 17:12:29,176] jepsen worker 76 - jepsen.util 706 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,177] jepsen worker 76 - jepsen.util 706 :invoke :write [25 36] INFO [2022-03-13 17:12:29,181] jepsen worker 69 - jepsen.util 609 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,182] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:29,191] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,192] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:29,195] jepsen worker 88 - jepsen.util 1978 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,198] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,198] jepsen worker 59 - jepsen.util 2039 :info :write [19 6] :RequestTimeoutException INFO [2022-03-13 17:12:29,199] jepsen worker 59 - jepsen.util 2129 :invoke :write [19 33] INFO [2022-03-13 17:12:29,200] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:29,200] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:12:29,203] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,204] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,205] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,205] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,205] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,206] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:29,207] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:29,207] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:29,212] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,213] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:12:29,226] jepsen worker 35 - jepsen.util 935 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,230] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,234] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:29,235] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:29,247] jepsen worker 37 - jepsen.util 2017 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,247] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,248] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,249] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,249] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,249] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:12:29,249] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,250] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,250] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,251] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,251] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,252] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:12:29,252] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,252] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:12:29,262] jepsen worker 8 - jepsen.util 728 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,262] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:29,278] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,278] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,279] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:12:29,279] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,287] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,287] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,288] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,288] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:12:29,288] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,289] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:12:29,299] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,300] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,301] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,302] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:12:29,303] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:29,303] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,304] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,304] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:12:29,305] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,306] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:12:29,318] jepsen worker 67 - jepsen.util 1687 :info :write [22 43] :RequestTimeoutException INFO [2022-03-13 17:12:29,319] jepsen worker 67 - jepsen.util 1777 :invoke :write [22 23] INFO [2022-03-13 17:12:29,331] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,332] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:29,343] jepsen worker 68 - jepsen.util 2138 :info :write [22 8] :RequestTimeoutException INFO [2022-03-13 17:12:29,344] jepsen worker 68 - jepsen.util 2228 :invoke :read [22 nil] INFO [2022-03-13 17:12:29,355] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,356] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,374] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,375] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:12:29,379] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,380] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,386] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,387] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:12:29,390] jepsen worker 21 - jepsen.util 741 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,390] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,390] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,391] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:29,391] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,391] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,391] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,392] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:12:29,394] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:29,395] jepsen worker 77 - jepsen.util 707 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,396] jepsen worker 77 - jepsen.util 707 :invoke :write [25 10] INFO [2022-03-13 17:12:29,406] jepsen worker 79 - jepsen.util 979 :invoke :write [26 26] INFO [2022-03-13 17:12:29,414] jepsen worker 34 - jepsen.util 754 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,415] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,431] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,432] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,439] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:29,440] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 18] INFO [2022-03-13 17:12:29,448] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,448] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,488] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,488] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,489] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,490] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,492] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,492] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,492] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,492] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,492] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,492] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,493] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:12:29,493] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:12:29,494] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:29,494] jepsen worker 69 - jepsen.util 609 :invoke :write [23 20] INFO [2022-03-13 17:12:29,495] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:29,495] jepsen worker 42 - jepsen.util 402 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,495] jepsen worker 21 - jepsen.util 741 :invoke :write [7 5] INFO [2022-03-13 17:12:29,496] jepsen worker 0 - jepsen.util 2160 :info :write [0 43] :RequestTimeoutException INFO [2022-03-13 17:12:29,496] jepsen worker 42 - jepsen.util 402 :invoke :write [14 47] INFO [2022-03-13 17:12:29,497] jepsen worker 0 - jepsen.util 2250 :invoke :read [0 nil] INFO [2022-03-13 17:12:29,511] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,512] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,526] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,526] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:29,532] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,532] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,532] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:29,533] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:12:29,543] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,544] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:29,545] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,545] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:12:29,567] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,567] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,567] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:12:29,567] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:12:29,578] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,578] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,579] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,581] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:12:29,593] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:29,593] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,596] jepsen worker 88 - jepsen.util 1978 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,596] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:12:29,635] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,635] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,635] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,636] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:12:29,660] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,660] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,660] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,661] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,661] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 7] INFO [2022-03-13 17:12:29,661] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,667] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,667] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,668] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,668] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,669] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,670] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 45] INFO [2022-03-13 17:12:29,671] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,671] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:12:29,687] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,688] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:29,695] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:29,696] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,731] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:29,731] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:12:29,753] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,753] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,753] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:12:29,754] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,761] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,762] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,808] jepsen worker 36 - jepsen.util 2106 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,809] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 30] INFO [2022-03-13 17:12:29,814] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:29,814] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,841] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,841] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,842] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:12:29,842] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:29,868] jepsen worker 63 - jepsen.util 2043 :ok :read [21 :nil] INFO [2022-03-13 17:12:29,869] jepsen worker 63 - jepsen.util 2043 :invoke :write [21 9] INFO [2022-03-13 17:12:29,882] jepsen worker 19 - jepsen.util 2089 :ok :read [6 16] INFO [2022-03-13 17:12:29,882] jepsen worker 19 - jepsen.util 2089 :invoke :read [6 nil] INFO [2022-03-13 17:12:29,893] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,894] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,899] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,900] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:29,901] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,903] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:29,924] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:29,925] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:29,925] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:29,925] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:12:29,926] jepsen worker 19 - jepsen.util 2089 :ok :read [6 16] INFO [2022-03-13 17:12:29,926] jepsen worker 19 - jepsen.util 2089 :invoke :write [6 40] INFO [2022-03-13 17:12:29,966] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:29,966] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,034] jepsen worker 16 - jepsen.util 1996 :ok :read [5 43] INFO [2022-03-13 17:12:30,035] jepsen worker 16 - jepsen.util 1996 :invoke :read [5 nil] INFO [2022-03-13 17:12:30,048] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5789473684210527}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:30,048] jepsen worker nemesis - couchbase.util Rebalance check count: 260 INFO [2022-03-13 17:12:30,048] jepsen worker nemesis - couchbase.util Rebalance stuck count: 166 INFO [2022-03-13 17:12:30,225] jepsen worker 18 - jepsen.util 1908 :info :write [6 29] :SyncWriteAmbiguous INFO [2022-03-13 17:12:30,226] jepsen worker 18 - jepsen.util 1998 :invoke :read [6 nil] INFO [2022-03-13 17:12:30,293] jepsen worker 64 - jepsen.util 2044 :info :write [21 19] :SyncWriteAmbiguous INFO [2022-03-13 17:12:30,294] jepsen worker 64 - jepsen.util 2134 :invoke :read [21 nil] INFO [2022-03-13 17:12:30,330] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,330] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,337] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,337] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,338] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:30,338] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:30,362] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:30,363] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,370] jepsen worker 86 - jepsen.util 1886 :info :write [28 7] :RequestTimeoutException INFO [2022-03-13 17:12:30,371] jepsen worker 86 - jepsen.util 1976 :invoke :read [28 nil] INFO [2022-03-13 17:12:30,376] jepsen worker 28 - jepsen.util 2368 :ok :read [9 26] INFO [2022-03-13 17:12:30,377] jepsen worker 28 - jepsen.util 2368 :invoke :write [9 4] INFO [2022-03-13 17:12:30,410] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,410] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:30,412] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,413] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:12:30,438] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:30,438] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,443] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,443] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,443] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,444] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:12:30,444] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 38] INFO [2022-03-13 17:12:30,445] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:12:30,453] jepsen worker 9 - jepsen.util 1989 :info :write [3 35] :SyncWriteAmbiguous INFO [2022-03-13 17:12:30,454] jepsen worker 9 - jepsen.util 2079 :invoke :read [3 nil] INFO [2022-03-13 17:12:30,470] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:30,471] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,484] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,485] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:30,491] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,491] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:30,507] jepsen worker 37 - jepsen.util 2017 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,508] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 40] INFO [2022-03-13 17:12:30,510] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:30,511] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 39] INFO [2022-03-13 17:12:30,531] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:30,531] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:12:30,543] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,543] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:12:30,590] jepsen worker 36 - jepsen.util 2106 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,591] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 0] INFO [2022-03-13 17:12:30,646] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,646] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:30,649] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,650] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,650] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:12:30,651] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,735] jepsen worker 88 - jepsen.util 1978 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,736] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,742] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,742] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,743] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:12:30,744] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:30,752] jepsen worker 3 - jepsen.util 1893 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,753] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:30,773] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:30,773] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,802] jepsen worker 37 - jepsen.util 2017 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,803] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:30,814] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,814] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:30,819] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,819] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:12:30,848] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:30,848] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,852] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,853] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:12:30,879] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,883] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:30,884] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:12:30,885] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 24] INFO [2022-03-13 17:12:30,889] jepsen worker 32 - jepsen.util 1832 :info :write [10 23] :RequestTimeoutException INFO [2022-03-13 17:12:30,890] jepsen worker 32 - jepsen.util 1922 :invoke :write [10 42] INFO [2022-03-13 17:12:30,894] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:30,895] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:12:30,906] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,906] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:12:30,910] jepsen worker 27 - jepsen.util 2367 :info :write [9 13] :RequestTimeoutException INFO [2022-03-13 17:12:30,911] jepsen worker 27 - jepsen.util 2457 :invoke :write [9 19] INFO [2022-03-13 17:12:30,914] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,914] jepsen worker 36 - jepsen.util 2106 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,914] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,915] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:12:30,918] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 46] INFO [2022-03-13 17:12:30,918] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:30,947] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,948] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:12:30,950] jepsen worker 20 - jepsen.util 2450 :info :write [6 30] :RequestTimeoutException INFO [2022-03-13 17:12:30,950] jepsen worker 20 - jepsen.util 2540 :invoke :write [6 1] INFO [2022-03-13 17:12:30,976] jepsen worker 2 - jepsen.util 2342 :info :write [0 12] :RequestTimeoutException INFO [2022-03-13 17:12:30,977] jepsen worker 2 - jepsen.util 2432 :invoke :read [0 nil] INFO [2022-03-13 17:12:30,989] jepsen worker 88 - jepsen.util 1978 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,990] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:30,990] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,990] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:30,991] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:12:30,991] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:12:31,018] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:31,019] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:12:31,046] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,047] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:12:31,054] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5789473684210527}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:31,054] jepsen worker nemesis - couchbase.util Rebalance check count: 261 INFO [2022-03-13 17:12:31,054] jepsen worker nemesis - couchbase.util Rebalance stuck count: 167 INFO [2022-03-13 17:12:31,074] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,074] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,088] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,088] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,096] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,096] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,097] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,098] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,099] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,099] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:12:31,140] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,140] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:31,175] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,175] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:31,191] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,191] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:31,208] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:31,209] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:31,228] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,228] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,241] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,241] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,250] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,251] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,252] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,252] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:12:31,253] jepsen worker 29 - jepsen.util 2189 :info :write [9 2] :RequestTimeoutException INFO [2022-03-13 17:12:31,254] jepsen worker 29 - jepsen.util 2279 :invoke :write [9 21] INFO [2022-03-13 17:12:31,256] jepsen worker 36 - jepsen.util 2106 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,259] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,274] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:31,275] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 32] INFO [2022-03-13 17:12:31,300] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,300] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:12:31,303] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:31,304] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:31,310] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,311] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:12:31,324] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:31,325] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:12:31,340] jepsen worker 48 - jepsen.util 2118 :info :write [16 26] :RequestTimeoutException INFO [2022-03-13 17:12:31,341] jepsen worker 48 - jepsen.util 2208 :invoke :read [16 nil] INFO [2022-03-13 17:12:31,355] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,355] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,355] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,356] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:12:31,356] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,357] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 41] INFO [2022-03-13 17:12:31,362] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,363] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 19] INFO [2022-03-13 17:12:31,372] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,373] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:12:31,384] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:31,385] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:12:31,398] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,399] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:12:31,462] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,462] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:31,465] jepsen worker 88 - jepsen.util 1978 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,465] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:12:31,469] jepsen worker 70 - jepsen.util 700 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,470] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,470] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,470] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,470] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,470] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,470] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:31,471] jepsen worker 52 - jepsen.util 592 :invoke :write [17 30] INFO [2022-03-13 17:12:31,472] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:31,472] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,473] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,474] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:31,475] jepsen worker 34 - jepsen.util 754 :invoke :write [11 25] INFO [2022-03-13 17:12:31,475] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,475] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,475] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:31,476] jepsen worker 80 - jepsen.util 710 :invoke :write [26 37] INFO [2022-03-13 17:12:31,477] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,477] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,477] jepsen worker 70 - jepsen.util 700 :invoke :write [23 35] INFO [2022-03-13 17:12:31,478] jepsen worker 60 - jepsen.util 600 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,478] jepsen worker 78 - jepsen.util 528 :invoke :write [26 45] INFO [2022-03-13 17:12:31,482] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:31,483] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,484] jepsen worker 13 - jepsen.util 643 :invoke :write [4 18] INFO [2022-03-13 17:12:31,487] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:31,489] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,491] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:31,523] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,524] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:12:31,541] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,542] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:12:31,560] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,561] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:12:31,563] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,563] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:31,585] jepsen worker 52 - jepsen.util 592 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,586] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:31,587] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,587] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:12:31,591] jepsen worker 14 - jepsen.util 284 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,591] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,591] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,591] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,592] jepsen worker 14 - jepsen.util 284 :invoke :write [4 47] INFO [2022-03-13 17:12:31,595] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:31,596] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:31,596] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:31,614] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,615] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:31,615] jepsen worker 79 - jepsen.util 979 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,615] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,616] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,616] jepsen worker 79 - jepsen.util 979 :invoke :write [26 35] INFO [2022-03-13 17:12:31,618] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:31,618] jepsen worker 33 - jepsen.util 663 :invoke :write [11 0] INFO [2022-03-13 17:12:31,649] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,649] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:12:31,661] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,662] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,673] jepsen worker 37 - jepsen.util 2017 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,673] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,692] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,692] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,693] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,693] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:12:31,694] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:31,694] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:12:31,700] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,701] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:31,702] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,702] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:12:31,708] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:31,709] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:12:31,727] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,728] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:31,743] jepsen worker 36 - jepsen.util 2106 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,743] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,744] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,744] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,747] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,747] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,747] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:12:31,748] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,748] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,749] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,749] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,750] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:12:31,756] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,757] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 41] INFO [2022-03-13 17:12:31,764] jepsen worker 44 - jepsen.util 584 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,764] jepsen worker 44 - jepsen.util 584 :invoke :write [14 26] INFO [2022-03-13 17:12:31,764] jepsen worker 70 - jepsen.util 700 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,765] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,766] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:12:31,771] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:31,790] jepsen worker 45 - jepsen.util 2475 :info :write [15 10] :RequestTimeoutException INFO [2022-03-13 17:12:31,791] jepsen worker 45 - jepsen.util 2565 :invoke :read [15 nil] INFO [2022-03-13 17:12:31,792] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,793] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 43] INFO [2022-03-13 17:12:31,800] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,801] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:12:31,812] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,813] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:12:31,813] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,815] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:12:31,824] jepsen worker 69 - jepsen.util 609 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,824] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:31,851] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,851] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,852] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,852] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:31,856] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,856] jepsen worker 62 - jepsen.util 512 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,857] jepsen worker 62 - jepsen.util 512 :invoke :write [20 7] INFO [2022-03-13 17:12:31,858] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:31,863] jepsen worker 41 - jepsen.util 1931 :info :write [13 20] :RequestTimeoutException INFO [2022-03-13 17:12:31,864] jepsen worker 41 - jepsen.util 2021 :invoke :read [13 nil] INFO [2022-03-13 17:12:31,873] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,874] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:31,912] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,912] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:12:31,919] jepsen worker 34 - jepsen.util 754 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,920] jepsen worker 34 - jepsen.util 754 :invoke :write [11 8] INFO [2022-03-13 17:12:31,924] jepsen worker 21 - jepsen.util 741 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,924] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,924] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,925] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:12:31,929] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:31,929] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:31,937] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,938] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:12:31,953] jepsen worker 36 - jepsen.util 2106 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,954] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 4] INFO [2022-03-13 17:12:31,969] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,969] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:12:31,980] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:31,981] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:32,015] jepsen worker 53 - jepsen.util 503 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,016] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,015] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:32,016] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:32,017] jepsen worker 53 - jepsen.util 503 :invoke :write [17 19] INFO [2022-03-13 17:12:32,017] jepsen worker 8 - jepsen.util 728 :invoke :write [2 1] INFO [2022-03-13 17:12:32,021] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,021] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,027] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,027] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,027] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,028] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:32,029] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,032] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:12:32,039] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,039] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,040] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:32,041] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:12:32,044] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,045] jepsen worker 79 - jepsen.util 979 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,046] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:12:32,047] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:32,056] jepsen worker 78 - jepsen.util 528 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,056] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:32,057] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,057] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:32,062] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:32,062] jepsen worker nemesis - couchbase.util Rebalance check count: 262 INFO [2022-03-13 17:12:32,062] jepsen worker nemesis - couchbase.util Rebalance stuck count: 167 INFO [2022-03-13 17:12:32,067] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,068] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:32,074] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,074] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,075] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,076] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:32,085] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:32,086] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,108] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:32,109] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:12:32,123] jepsen worker 88 - jepsen.util 1978 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,124] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,157] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,158] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:12:32,158] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,158] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:32,159] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,159] jepsen worker 12 - jepsen.util 732 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,160] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,160] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,160] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:32,160] jepsen worker 36 - jepsen.util 2106 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,161] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:12:32,162] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:32,162] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,162] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,166] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,166] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:12:32,173] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,173] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:12:32,177] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,177] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,177] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,178] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,179] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,179] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,179] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,180] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 6] INFO [2022-03-13 17:12:32,180] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,181] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:12:32,185] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,186] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,194] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:32,195] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:12:32,205] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,205] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:12:32,242] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,242] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:12:32,266] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,267] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:32,268] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,268] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:32,272] jepsen worker 42 - jepsen.util 402 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,273] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,273] jepsen worker 42 - jepsen.util 402 :invoke :write [14 10] INFO [2022-03-13 17:12:32,273] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,275] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,275] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,275] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,275] jepsen worker 6 - jepsen.util 636 :invoke :write [2 38] INFO [2022-03-13 17:12:32,276] jepsen worker 33 - jepsen.util 663 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,276] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:32,276] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:12:32,277] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:32,277] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,278] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,279] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,280] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,286] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,286] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,289] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,289] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,292] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,292] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,292] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:32,293] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:32,294] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,295] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:32,296] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,297] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:12:32,313] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,314] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:12:32,315] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,315] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,316] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:12:32,318] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,323] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:32,324] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:12:32,343] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,343] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 7] INFO [2022-03-13 17:12:32,346] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,346] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:32,347] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,348] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:12:32,377] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,378] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:12:32,384] jepsen worker 13 - jepsen.util 643 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,385] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:32,392] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,392] jepsen worker 76 - jepsen.util 706 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,392] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,392] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,393] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,393] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:32,394] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:32,394] jepsen worker 69 - jepsen.util 609 :invoke :write [23 24] INFO [2022-03-13 17:12:32,394] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:32,395] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:32,397] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,398] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:12:32,415] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,415] jepsen worker 14 - jepsen.util 284 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,415] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:32,416] jepsen worker 14 - jepsen.util 284 :invoke :write [4 15] INFO [2022-03-13 17:12:32,429] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,429] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 37] INFO [2022-03-13 17:12:32,468] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,469] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:32,474] jepsen worker 88 - jepsen.util 1978 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,475] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,512] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,512] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,512] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:32,513] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:12:32,513] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,514] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,514] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,515] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:32,516] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,516] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:32,517] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,518] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:32,519] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,519] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:12:32,523] jepsen worker 77 - jepsen.util 707 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,523] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,523] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:32,524] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:32,525] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,526] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:12:32,553] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,553] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:12:32,572] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,572] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:12:32,592] jepsen worker 36 - jepsen.util 2106 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,593] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 23] INFO [2022-03-13 17:12:32,598] jepsen worker 56 - jepsen.util 2216 :info :write [18 47] :RequestTimeoutException INFO [2022-03-13 17:12:32,598] jepsen worker 56 - jepsen.util 2306 :invoke :read [18 nil] INFO [2022-03-13 17:12:32,606] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,606] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,615] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,615] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,617] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:12:32,617] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:12:32,624] jepsen worker 62 - jepsen.util 512 :fail :write [20 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,624] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:32,625] jepsen worker 43 - jepsen.util 493 :invoke :write [14 18] INFO [2022-03-13 17:12:32,625] jepsen worker 62 - jepsen.util 512 :invoke :write [20 23] INFO [2022-03-13 17:12:32,643] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,643] jepsen worker 80 - jepsen.util 710 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,643] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,644] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:32,644] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:32,645] jepsen worker 70 - jepsen.util 700 :invoke :write [23 25] INFO [2022-03-13 17:12:32,682] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,682] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,689] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,690] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:12:32,701] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,702] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:12:32,713] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,714] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,718] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,719] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:12:32,742] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,743] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,785] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,786] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:12:32,788] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,789] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:12:32,804] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,805] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,815] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,816] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,817] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,818] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,820] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,821] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,822] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,823] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:12:32,836] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,836] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:32,860] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:32,860] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:12:32,865] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,865] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:32,893] jepsen worker 37 - jepsen.util 2017 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,894] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,921] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,922] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:32,953] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,954] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:12:32,972] jepsen worker 36 - jepsen.util 2106 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,973] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:32,999] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:32,999] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:12:33,025] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,025] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,048] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,048] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,049] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,049] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:12:33,058] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:33,058] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,069] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:33,069] jepsen worker nemesis - couchbase.util Rebalance check count: 263 INFO [2022-03-13 17:12:33,069] jepsen worker nemesis - couchbase.util Rebalance stuck count: 168 INFO [2022-03-13 17:12:33,095] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,096] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:33,097] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,097] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:33,098] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,099] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:12:33,113] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,113] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,113] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,113] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,114] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 43] INFO [2022-03-13 17:12:33,114] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:33,115] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,115] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:12:33,121] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,122] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:33,124] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,124] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 34] INFO [2022-03-13 17:12:33,141] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,141] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,141] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,141] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:12:33,148] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:33,148] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,162] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,162] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 37] INFO [2022-03-13 17:12:33,190] jepsen worker 31 - jepsen.util 2101 :info :write [10 6] :RequestTimeoutException INFO [2022-03-13 17:12:33,191] jepsen worker 31 - jepsen.util 2191 :invoke :write [10 46] INFO [2022-03-13 17:12:33,195] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:33,196] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,220] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,221] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:33,222] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,223] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:12:33,234] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,234] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,235] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,235] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,235] jepsen worker 36 - jepsen.util 2106 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,236] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 30] INFO [2022-03-13 17:12:33,279] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:33,280] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,313] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,313] jepsen worker 88 - jepsen.util 1978 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,314] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:12:33,315] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:12:33,322] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,323] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:12:33,329] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,329] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,331] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,332] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,335] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,336] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:12:33,343] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:33,344] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,360] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,361] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:33,368] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:33,369] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:12:33,370] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,370] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:12:33,420] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,420] jepsen worker 37 - jepsen.util 2017 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,421] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:12:33,422] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 47] INFO [2022-03-13 17:12:33,435] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,436] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,469] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:33,469] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 16] INFO [2022-03-13 17:12:33,471] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:33,471] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,511] jepsen worker 36 - jepsen.util 2106 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,512] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 32] INFO [2022-03-13 17:12:33,518] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,518] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,542] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,542] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:12:33,554] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,555] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:33,592] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:33,592] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:12:33,626] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,627] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:12:33,638] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:33,639] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:33,650] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:33,650] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,074] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:34,074] jepsen worker nemesis - couchbase.util Rebalance check count: 264 INFO [2022-03-13 17:12:34,074] jepsen worker nemesis - couchbase.util Rebalance stuck count: 169 INFO [2022-03-13 17:12:34,116] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,117] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,119] jepsen worker 88 - jepsen.util 1978 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,119] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,127] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,128] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,168] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:34,169] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 0] INFO [2022-03-13 17:12:34,191] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,191] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:12:34,200] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,200] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,201] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,202] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:12:34,213] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:34,214] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:12:34,216] jepsen worker 37 - jepsen.util 2017 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,216] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,251] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,252] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:12:34,265] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,266] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:34,323] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,324] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:12:34,335] jepsen worker 36 - jepsen.util 2106 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,335] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,335] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,336] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,337] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:12:34,337] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,341] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,341] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,342] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:12:34,342] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:12:34,358] jepsen worker 88 - jepsen.util 1978 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,359] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,412] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:34,412] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 10] INFO [2022-03-13 17:12:34,430] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,431] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,445] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,446] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,447] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,448] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:12:34,455] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,455] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,499] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,500] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,501] jepsen worker 88 - jepsen.util 1978 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,502] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,533] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:34,533] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,536] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,537] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,538] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:34,538] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 41] INFO [2022-03-13 17:12:34,582] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,582] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,582] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,583] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:12:34,584] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,584] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:12:34,602] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,602] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,602] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,603] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,621] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,621] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,622] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:34,622] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:12:34,630] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:34,630] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,635] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:34,635] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,642] jepsen worker 88 - jepsen.util 1978 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,643] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:12:34,676] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:34,677] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:12:34,710] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,711] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:12:34,711] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,711] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:12:34,715] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,715] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,716] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,716] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,769] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:34,769] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 39] INFO [2022-03-13 17:12:34,778] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,778] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:12:34,792] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,793] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,830] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:34,830] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 9] INFO [2022-03-13 17:12:34,840] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,840] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,841] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,841] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,841] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:34,842] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 33] INFO [2022-03-13 17:12:34,842] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:12:34,846] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,846] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 46] INFO [2022-03-13 17:12:34,851] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:34,868] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,868] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:12:34,913] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,913] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:34,947] jepsen worker 89 - jepsen.util 2429 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,947] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:12:34,954] jepsen worker 88 - jepsen.util 1978 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:34,955] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:12:35,003] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,004] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:35,012] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,015] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 9] INFO [2022-03-13 17:12:35,029] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,029] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:12:35,043] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,044] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,045] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:35,045] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:12:35,050] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,051] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,056] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,057] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:12:35,080] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:35,080] jepsen worker nemesis - couchbase.util Rebalance check count: 265 INFO [2022-03-13 17:12:35,080] jepsen worker nemesis - couchbase.util Rebalance stuck count: 170 INFO [2022-03-13 17:12:35,107] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,108] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:12:35,110] jepsen worker 37 - jepsen.util 2017 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,111] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,151] jepsen worker 3 - jepsen.util 1893 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,152] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:12:35,172] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,173] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,174] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,175] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:12:35,175] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,176] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:12:35,241] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,241] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:12:35,252] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,252] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:12:35,272] jepsen worker 36 - jepsen.util 2106 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,272] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,272] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,272] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,273] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:12:35,273] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:12:35,279] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,279] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 34] INFO [2022-03-13 17:12:35,331] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,331] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,332] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,333] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,334] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,334] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:35,339] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,340] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:12:35,366] jepsen worker 30 - jepsen.util 2190 :info :write [10 34] :RequestTimeoutException INFO [2022-03-13 17:12:35,367] jepsen worker 30 - jepsen.util 2280 :invoke :write [10 29] INFO [2022-03-13 17:12:35,378] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,378] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:12:35,413] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,414] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:12:35,420] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,421] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:35,435] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,435] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:12:35,443] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:35,443] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:35,482] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,483] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:12:35,505] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,506] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:12:35,510] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:35,511] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:12:35,511] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,512] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:35,563] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,564] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:35,571] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,572] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:35,598] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:35,599] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:12:35,614] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,614] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:12:35,626] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,626] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:35,635] jepsen worker 36 - jepsen.util 2106 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,636] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 12] INFO [2022-03-13 17:12:35,650] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:35,651] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:12:35,703] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,703] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:12:35,751] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,751] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:12:35,752] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,754] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,755] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,756] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,758] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,759] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,760] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,761] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,763] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,763] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:12:35,790] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,791] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:12:35,795] jepsen worker 81 - jepsen.util 2421 :ok :read [27 4] INFO [2022-03-13 17:12:35,795] jepsen worker 81 - jepsen.util 2421 :invoke :read [27 nil] INFO [2022-03-13 17:12:35,828] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,828] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,828] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:35,829] jepsen worker 79 - jepsen.util 979 :invoke :write [26 42] INFO [2022-03-13 17:12:35,840] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,841] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:35,846] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,847] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 39] INFO [2022-03-13 17:12:35,867] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,867] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:35,879] jepsen worker 39 - jepsen.util 2109 :ok :read [13 39] INFO [2022-03-13 17:12:35,880] jepsen worker 39 - jepsen.util 2109 :invoke :read [13 nil] INFO [2022-03-13 17:12:35,884] jepsen worker 2 - jepsen.util 2432 :ok :read [0 41] INFO [2022-03-13 17:12:35,885] jepsen worker 2 - jepsen.util 2432 :invoke :write [0 2] INFO [2022-03-13 17:12:35,896] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:35,897] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 4] INFO [2022-03-13 17:12:35,898] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,899] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:12:35,921] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,922] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:12:35,925] jepsen worker 81 - jepsen.util 2421 :ok :read [27 4] INFO [2022-03-13 17:12:35,926] jepsen worker 81 - jepsen.util 2421 :invoke :read [27 nil] INFO [2022-03-13 17:12:35,936] jepsen worker 44 - jepsen.util 584 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,936] jepsen worker 44 - jepsen.util 584 :invoke :write [14 28] INFO [2022-03-13 17:12:35,942] jepsen worker 39 - jepsen.util 2109 :ok :read [13 39] INFO [2022-03-13 17:12:35,942] jepsen worker 39 - jepsen.util 2109 :invoke :read [13 nil] WARN [2022-03-13 17:12:35,958] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9052780,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9939511,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9539753,"encode_duration_us":18418,"operation_id":"0x646f","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10003677},{"operation_name":"upsert","last_dispatch_duration_us":25388612,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25388612,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9035244,"encode_duration_us":22912,"operation_id":"0x5de8","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10005726}],"total_count":2}} WARN [2022-03-13 17:12:35,958] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":24341077,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":24341077,"total_server_duration_us":249119,"operation_id":"0x5eb1","timeout_ms":30000,"last_server_duration_us":249119,"total_duration_us":24343658},{"operation_name":"get","last_dispatch_duration_us":18493738,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":18493738,"total_server_duration_us":10692,"operation_id":"0x61cf","timeout_ms":30000,"last_server_duration_us":10692,"total_duration_us":18496597},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":19397,"operation_id":"0x67ba","timeout_ms":10000,"total_duration_us":10008048},{"operation_name":"upsert","last_dispatch_duration_us":6690000,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7441843,"total_server_duration_us":443790,"encode_duration_us":17603,"operation_id":"0x66aa","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10007927},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":25336,"operation_id":"0x6960","timeout_ms":10000,"total_duration_us":10003428},{"operation_name":"upsert","last_dispatch_duration_us":7087962,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7514177,"total_server_duration_us":315763,"encode_duration_us":19653,"operation_id":"0x68a0","timeout_ms":10000,"last_server_duration_us":40589,"total_duration_us":10002454},{"operation_name":"upsert","last_dispatch_duration_us":3304470,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3304470,"total_server_duration_us":3287685,"encode_duration_us":17791,"operation_id":"0x6e4c","timeout_ms":10000,"last_server_duration_us":3287685,"total_duration_us":3308416},{"operation_name":"upsert","last_dispatch_duration_us":2093275,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2093275,"total_server_duration_us":90631,"encode_duration_us":20381,"operation_id":"0x6ce3","timeout_ms":10000,"last_server_duration_us":90631,"total_duration_us":2103878},{"operation_name":"upsert","last_dispatch_duration_us":1970270,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1970270,"total_server_duration_us":1951012,"encode_duration_us":19381,"operation_id":"0x6ce2","timeout_ms":10000,"last_server_duration_us":1951012,"total_duration_us":1975070},{"operation_name":"upsert","last_dispatch_duration_us":889640,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":889640,"total_server_duration_us":57509,"encode_duration_us":19508,"operation_id":"0x6a96","timeout_ms":10000,"last_server_duration_us":57509,"total_duration_us":895190}],"total_count":15}} INFO [2022-03-13 17:12:35,964] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,965] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:12:35,971] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,971] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,976] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,977] jepsen worker 35 - jepsen.util 935 :invoke :write [11 15] INFO [2022-03-13 17:12:35,978] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,978] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,979] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,979] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:35,980] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,981] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:35,982] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,982] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,983] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:35,983] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:35,989] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,991] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:35,997] jepsen worker 55 - jepsen.util 2035 :info :write [18 32] :SyncWriteAmbiguous INFO [2022-03-13 17:12:35,997] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,998] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,998] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,998] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:35,999] jepsen worker 55 - jepsen.util 2125 :invoke :read [18 nil] INFO [2022-03-13 17:12:36,000] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,000] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:12:36,001] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:12:36,002] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:12:36,004] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:12:36,009] jepsen worker 21 - jepsen.util 741 :invoke :write [7 32] INFO [2022-03-13 17:12:36,030] jepsen worker 81 - jepsen.util 2421 :ok :read [27 4] INFO [2022-03-13 17:12:36,032] jepsen worker 81 - jepsen.util 2421 :invoke :read [27 nil] INFO [2022-03-13 17:12:36,042] jepsen worker 8 - jepsen.util 728 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,044] jepsen worker 8 - jepsen.util 728 :invoke :write [2 35] INFO [2022-03-13 17:12:36,058] jepsen worker 88 - jepsen.util 1978 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,058] jepsen worker 39 - jepsen.util 2109 :ok :read [13 39] INFO [2022-03-13 17:12:36,058] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,059] jepsen worker 39 - jepsen.util 2109 :invoke :write [13 30] INFO [2022-03-13 17:12:36,074] jepsen worker 34 - jepsen.util 754 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,074] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,074] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,075] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:36,075] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,076] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,076] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:36,077] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:36,078] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:36,078] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:36,082] jepsen worker 89 - jepsen.util 2429 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,083] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,086] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:36,086] jepsen worker nemesis - couchbase.util Rebalance check count: 266 INFO [2022-03-13 17:12:36,086] jepsen worker nemesis - couchbase.util Rebalance stuck count: 171 INFO [2022-03-13 17:12:36,090] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:36,091] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 49] INFO [2022-03-13 17:12:36,094] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,095] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,135] jepsen worker 81 - jepsen.util 2421 :ok :read [27 4] INFO [2022-03-13 17:12:36,135] jepsen worker 81 - jepsen.util 2421 :invoke :read [27 nil] INFO [2022-03-13 17:12:36,138] jepsen worker 35 - jepsen.util 935 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,138] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:36,146] jepsen worker 69 - jepsen.util 609 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,146] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,147] jepsen worker 69 - jepsen.util 609 :invoke :write [23 13] INFO [2022-03-13 17:12:36,148] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:36,153] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,154] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:36,156] jepsen worker 74 - jepsen.util 2594 :info :write [24 15] :RequestTimeoutException INFO [2022-03-13 17:12:36,157] jepsen worker 74 - jepsen.util 2684 :invoke :read [24 nil] INFO [2022-03-13 17:12:36,176] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,176] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:36,177] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,178] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:36,183] jepsen worker 36 - jepsen.util 2106 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,184] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:36,191] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,192] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:36,192] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,193] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,193] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:36,193] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:12:36,199] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,200] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,200] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:36,201] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:36,205] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,205] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:36,206] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:12:36,207] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:12:36,207] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,208] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:12:36,216] jepsen worker 73 - jepsen.util 2143 :info :write [24 24] :RequestTimeoutException INFO [2022-03-13 17:12:36,221] jepsen worker 73 - jepsen.util 2233 :invoke :write [24 35] INFO [2022-03-13 17:12:36,225] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,225] jepsen worker 70 - jepsen.util 700 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,227] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:36,227] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,227] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,227] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,227] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:12:36,227] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,228] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,228] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,229] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:36,230] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:36,230] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,231] jepsen worker 34 - jepsen.util 754 :invoke :write [11 29] INFO [2022-03-13 17:12:36,231] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,232] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:36,232] jepsen worker 79 - jepsen.util 979 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,233] jepsen worker 81 - jepsen.util 2421 :ok :read [27 4] INFO [2022-03-13 17:12:36,233] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,233] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:36,234] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,235] jepsen worker 81 - jepsen.util 2421 :invoke :write [27 11] INFO [2022-03-13 17:12:36,235] jepsen worker 70 - jepsen.util 700 :invoke :write [23 10] INFO [2022-03-13 17:12:36,236] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:36,236] jepsen worker 13 - jepsen.util 643 :invoke :write [4 32] INFO [2022-03-13 17:12:36,241] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,242] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:36,242] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:36,242] jepsen worker 79 - jepsen.util 979 :invoke :write [26 14] INFO [2022-03-13 17:12:36,244] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,244] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,244] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,244] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,244] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,245] jepsen worker 88 - jepsen.util 1978 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,245] jepsen worker 23 - jepsen.util 563 :invoke :write [7 32] INFO [2022-03-13 17:12:36,245] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,245] jepsen worker 51 - jepsen.util 591 :invoke :write [17 30] INFO [2022-03-13 17:12:36,246] jepsen worker 78 - jepsen.util 528 :invoke :write [26 8] INFO [2022-03-13 17:12:36,246] jepsen worker 76 - jepsen.util 706 :invoke :write [25 8] INFO [2022-03-13 17:12:36,247] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:36,248] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:36,250] jepsen worker 77 - jepsen.util 707 :invoke :write [25 37] INFO [2022-03-13 17:12:36,254] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 37] INFO [2022-03-13 17:12:36,255] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,259] jepsen worker 52 - jepsen.util 592 :invoke :write [17 15] INFO [2022-03-13 17:12:36,260] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,261] jepsen worker 80 - jepsen.util 710 :invoke :write [26 25] INFO [2022-03-13 17:12:36,266] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,266] jepsen worker 54 - jepsen.util 1764 :info :write [18 3] :RequestTimeoutException INFO [2022-03-13 17:12:36,268] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,268] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,269] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,272] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,272] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:12:36,273] jepsen worker 54 - jepsen.util 1854 :invoke :read [18 nil] INFO [2022-03-13 17:12:36,313] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,314] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:12:36,326] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,326] jepsen worker 53 - jepsen.util 503 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,326] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,327] jepsen worker 61 - jepsen.util 601 :invoke :write [20 33] INFO [2022-03-13 17:12:36,328] jepsen worker 53 - jepsen.util 503 :invoke :write [17 27] INFO [2022-03-13 17:12:36,328] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:36,336] jepsen worker 10 - jepsen.util 2080 :info :write [3 38] :RequestTimeoutException INFO [2022-03-13 17:12:36,336] jepsen worker 11 - jepsen.util 2081 :info :write [3 20] :RequestTimeoutException INFO [2022-03-13 17:12:36,336] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,337] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:12:36,339] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,339] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,339] jepsen worker 11 - jepsen.util 2171 :invoke :read [3 nil] INFO [2022-03-13 17:12:36,340] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,340] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,340] jepsen worker 10 - jepsen.util 2170 :invoke :write [3 26] INFO [2022-03-13 17:12:36,341] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,346] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,350] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,350] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,351] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:36,351] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:12:36,352] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,352] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,352] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,353] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:12:36,353] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:36,354] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,355] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,355] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,356] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,356] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:12:36,366] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,366] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,369] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:36,371] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:12:36,371] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:12:36,372] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:12:36,403] jepsen worker 11 - jepsen.util 2171 :ok :read [3 25] INFO [2022-03-13 17:12:36,404] jepsen worker 11 - jepsen.util 2171 :invoke :write [3 25] INFO [2022-03-13 17:12:36,408] jepsen worker 88 - jepsen.util 1978 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,408] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:12:36,415] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,415] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:36,421] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,422] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,454] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,455] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:12:36,475] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,475] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:12:36,483] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:36,484] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 40] INFO [2022-03-13 17:12:36,488] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,488] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,488] jepsen worker 44 - jepsen.util 584 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,489] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:36,490] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:36,490] jepsen worker 44 - jepsen.util 584 :invoke :write [14 17] INFO [2022-03-13 17:12:36,500] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,501] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:12:36,503] jepsen worker 13 - jepsen.util 643 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,504] jepsen worker 13 - jepsen.util 643 :invoke :write [4 30] INFO [2022-03-13 17:12:36,509] jepsen worker 64 - jepsen.util 2134 :ok :read [21 :nil] INFO [2022-03-13 17:12:36,510] jepsen worker 64 - jepsen.util 2134 :invoke :read [21 nil] INFO [2022-03-13 17:12:36,512] jepsen worker 64 - jepsen.util 2134 :ok :read [21 :nil] INFO [2022-03-13 17:12:36,513] jepsen worker 64 - jepsen.util 2134 :invoke :write [21 10] INFO [2022-03-13 17:12:36,546] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,547] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,547] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,548] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:36,550] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,551] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,553] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,553] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,553] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:12:36,554] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,556] jepsen worker 6 - jepsen.util 636 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,557] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:36,589] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:36,590] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 36] INFO [2022-03-13 17:12:36,605] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,605] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:12:36,618] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,619] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:36,625] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,626] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:12:36,630] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,632] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:12:36,666] jepsen worker 21 - jepsen.util 741 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,667] jepsen worker 21 - jepsen.util 741 :invoke :write [7 29] INFO [2022-03-13 17:12:36,671] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,672] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:12:36,676] jepsen worker 88 - jepsen.util 1978 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,676] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,682] jepsen worker 77 - jepsen.util 707 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,682] jepsen worker 77 - jepsen.util 707 :invoke :write [25 9] INFO [2022-03-13 17:12:36,704] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,704] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,706] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:36,706] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:12:36,715] jepsen worker 34 - jepsen.util 754 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,716] jepsen worker 34 - jepsen.util 754 :invoke :write [11 1] INFO [2022-03-13 17:12:36,735] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,736] jepsen worker 58 - jepsen.util 1948 :info :write [19 0] :RequestTimeoutException INFO [2022-03-13 17:12:36,736] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:36,737] jepsen worker 58 - jepsen.util 2038 :invoke :write [19 42] INFO [2022-03-13 17:12:36,758] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,758] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,758] jepsen worker 42 - jepsen.util 402 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,758] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:36,759] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:12:36,759] jepsen worker 42 - jepsen.util 402 :invoke :write [14 28] INFO [2022-03-13 17:12:36,762] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,763] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,763] jepsen worker 89 - jepsen.util 2429 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,763] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:36,764] jepsen worker 69 - jepsen.util 609 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,764] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,764] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,764] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,765] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,766] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,768] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:12:36,769] jepsen worker 69 - jepsen.util 609 :invoke :write [23 49] INFO [2022-03-13 17:12:36,770] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:12:36,774] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:12:36,790] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:36,792] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,807] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:36,807] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,810] jepsen worker 50 - jepsen.util 2120 :info :write [16 22] :RequestTimeoutException INFO [2022-03-13 17:12:36,811] jepsen worker 50 - jepsen.util 2210 :invoke :write [16 36] INFO [2022-03-13 17:12:36,822] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:36,823] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,824] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,824] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,825] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:12:36,825] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:12:36,842] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,843] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,843] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,843] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:12:36,860] jepsen worker 70 - jepsen.util 700 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,860] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:36,871] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,871] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:36,875] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,876] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 23] INFO [2022-03-13 17:12:36,882] jepsen worker 79 - jepsen.util 979 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,883] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:36,887] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,888] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:36,894] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,895] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:12:36,896] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:36,897] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,903] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,904] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:36,906] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,907] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:12:36,917] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,918] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,921] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:36,922] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,923] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,924] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:12:36,949] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:36,949] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,958] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:36,959] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:36,962] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,962] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,962] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 36] INFO [2022-03-13 17:12:36,963] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:36,967] jepsen worker 17 - jepsen.util 2087 :info :write [5 29] :RequestTimeoutException INFO [2022-03-13 17:12:36,968] jepsen worker 17 - jepsen.util 2177 :invoke :write [5 2] INFO [2022-03-13 17:12:36,972] jepsen worker 23 - jepsen.util 563 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,973] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:36,996] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:36,997] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:37,009] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:37,009] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:37,015] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,015] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,016] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,016] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:12:37,016] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,018] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:37,018] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:12:37,019] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:12:37,023] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,023] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,024] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,024] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:37,025] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:12:37,025] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:12:37,030] jepsen worker 80 - jepsen.util 710 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,033] jepsen worker 80 - jepsen.util 710 :invoke :write [26 42] INFO [2022-03-13 17:12:37,044] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:37,044] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:37,061] jepsen worker 21 - jepsen.util 741 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,065] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,065] jepsen worker 21 - jepsen.util 741 :invoke :write [7 13] INFO [2022-03-13 17:12:37,066] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:12:37,076] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:37,076] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:12:37,079] jepsen worker 14 - jepsen.util 284 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,079] jepsen worker 14 - jepsen.util 284 :invoke :write [4 12] INFO [2022-03-13 17:12:37,092] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:37,092] jepsen worker nemesis - couchbase.util Rebalance check count: 267 INFO [2022-03-13 17:12:37,092] jepsen worker nemesis - couchbase.util Rebalance stuck count: 172 INFO [2022-03-13 17:12:37,168] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,168] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:12:37,171] jepsen worker 52 - jepsen.util 592 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,172] jepsen worker 52 - jepsen.util 592 :invoke :write [17 33] INFO [2022-03-13 17:12:37,177] jepsen worker 69 - jepsen.util 609 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,178] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:12:37,200] jepsen worker 76 - jepsen.util 706 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,201] jepsen worker 76 - jepsen.util 706 :invoke :write [25 19] INFO [2022-03-13 17:12:37,207] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,208] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:37,263] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,264] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:37,271] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,272] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:37,273] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,273] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:37,275] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,275] jepsen worker 22 - jepsen.util 652 :invoke :write [7 10] INFO [2022-03-13 17:12:37,288] jepsen worker 62 - jepsen.util 512 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,288] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:37,315] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,316] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:37,316] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,316] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:12:37,319] jepsen worker 78 - jepsen.util 528 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,319] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:37,323] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,324] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:12:37,345] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,345] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:37,346] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,346] jepsen worker 79 - jepsen.util 979 :invoke :write [26 14] INFO [2022-03-13 17:12:37,397] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,398] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] WARN [2022-03-13 17:12:37,459] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9408010,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15387457,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9091959,"encode_duration_us":19950,"operation_id":"0x6475","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10003686},{"operation_name":"upsert","last_dispatch_duration_us":18935815,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18935815,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9034183,"encode_duration_us":28809,"operation_id":"0x6176","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10007771},{"operation_name":"upsert","last_dispatch_duration_us":17964463,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18232535,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9292202,"encode_duration_us":17216,"operation_id":"0x66a4","timeout_ms":10000,"last_server_duration_us":9211071,"total_duration_us":10006912}],"total_count":3}} WARN [2022-03-13 17:12:37,469] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":27278686,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27278686,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9062854,"encode_duration_us":17983,"operation_id":"0x5eb3","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10002183}],"total_count":1}} INFO [2022-03-13 17:12:37,481] jepsen worker 51 - jepsen.util 591 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,481] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,482] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,482] jepsen worker 51 - jepsen.util 591 :invoke :write [17 3] INFO [2022-03-13 17:12:37,483] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:12:37,483] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:37,490] jepsen worker 13 - jepsen.util 643 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,490] jepsen worker 22 - jepsen.util 652 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,490] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,491] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:12:37,492] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:37,492] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] WARN [2022-03-13 17:12:37,560] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":28412,"operation_id":"0x6916","timeout_ms":10000,"total_duration_us":10009466},{"operation_name":"upsert","last_dispatch_duration_us":8755202,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8966730,"total_server_duration_us":84093,"encode_duration_us":17071,"operation_id":"0x66a8","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10009394},{"operation_name":"upsert","last_dispatch_duration_us":8980090,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9428735,"total_server_duration_us":49494,"encode_duration_us":18240,"operation_id":"0x673c","timeout_ms":10000,"last_server_duration_us":28493,"total_duration_us":10008040},{"operation_name":"upsert","last_dispatch_duration_us":17964463,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18232535,"total_server_duration_us":9292202,"encode_duration_us":17216,"operation_id":"0x66a4","timeout_ms":10000,"last_server_duration_us":9211071,"total_duration_us":10006912},{"operation_name":"upsert","last_dispatch_duration_us":8895185,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9530357,"total_server_duration_us":84508,"encode_duration_us":31416,"operation_id":"0x6730","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10005171},{"operation_name":"upsert","last_dispatch_duration_us":8937000,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9350602,"total_server_duration_us":41682,"encode_duration_us":21240,"operation_id":"0x6747","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10005019},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":18932,"operation_id":"0x6b86","timeout_ms":10000,"total_duration_us":10000935},{"operation_name":"upsert","last_dispatch_duration_us":9315185,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9398378,"total_server_duration_us":9094058,"encode_duration_us":18739,"operation_id":"0x6712","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":9423924},{"operation_name":"upsert","last_dispatch_duration_us":9119859,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9197202,"total_server_duration_us":9067415,"encode_duration_us":16913,"operation_id":"0x6710","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9219873},{"operation_name":"get","last_dispatch_duration_us":7895535,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":7895535,"total_server_duration_us":49,"operation_id":"0x680f","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":7900209}],"total_count":43}} WARN [2022-03-13 17:12:37,564] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16203323,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16680388,"total_server_duration_us":9344813,"encode_duration_us":17900,"operation_id":"0x66fd","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10005983},{"operation_name":"upsert","last_dispatch_duration_us":7569550,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8770983,"total_server_duration_us":280149,"encode_duration_us":36463,"operation_id":"0x681a","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10005485},{"operation_name":"upsert","last_dispatch_duration_us":7524435,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8750304,"total_server_duration_us":256422,"encode_duration_us":30794,"operation_id":"0x682b","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10005273},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18597,"operation_id":"0x6858","timeout_ms":10000,"total_duration_us":10003792},{"operation_name":"upsert","last_dispatch_duration_us":59572,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":1042846,"total_server_duration_us":200545,"encode_duration_us":34552,"operation_id":"0x6823","timeout_ms":10000,"last_server_duration_us":27854,"total_duration_us":10001691},{"operation_name":"upsert","last_dispatch_duration_us":9284656,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9284656,"total_server_duration_us":9039489,"encode_duration_us":22777,"operation_id":"0x6827","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":9291216},{"operation_name":"get","last_dispatch_duration_us":6907720,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":6907720,"total_server_duration_us":49,"operation_id":"0x6987","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":6910347},{"operation_name":"upsert","last_dispatch_duration_us":3569345,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3569345,"total_server_duration_us":3275957,"encode_duration_us":29957,"operation_id":"0x6e27","timeout_ms":10000,"last_server_duration_us":3275957,"total_duration_us":3574307},{"operation_name":"upsert","last_dispatch_duration_us":1291884,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2711798,"total_server_duration_us":135120,"encode_duration_us":23182,"operation_id":"0x6b32","timeout_ms":10000,"last_server_duration_us":102959,"total_duration_us":2728946},{"operation_name":"upsert","last_dispatch_duration_us":1951609,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1951609,"total_server_duration_us":24497,"encode_duration_us":18638,"operation_id":"0x6cda","timeout_ms":10000,"last_server_duration_us":24497,"total_duration_us":1958155}],"total_count":15}} WARN [2022-03-13 17:12:37,569] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18136774,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27840399,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9190753,"encode_duration_us":20996,"operation_id":"0x6135","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10010548}],"total_count":1}} INFO [2022-03-13 17:12:37,580] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,580] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:12:37,588] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,588] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,589] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:37,590] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:12:37,590] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,591] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:37,592] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,593] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:12:37,601] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,601] jepsen worker 43 - jepsen.util 493 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,602] jepsen worker 43 - jepsen.util 493 :invoke :write [14 23] INFO [2022-03-13 17:12:37,602] jepsen worker 26 - jepsen.util 476 :invoke :write [8 47] INFO [2022-03-13 17:12:37,612] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,612] jepsen worker 24 - jepsen.util 564 :invoke :write [8 4] INFO [2022-03-13 17:12:37,642] jepsen worker 21 - jepsen.util 741 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,642] jepsen worker 21 - jepsen.util 741 :invoke :write [7 44] WARN [2022-03-13 17:12:37,664] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16056100,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21619259,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9336035,"encode_duration_us":17977,"operation_id":"0x6423","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10003909},{"operation_name":"upsert","last_dispatch_duration_us":16203323,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16680388,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9344813,"encode_duration_us":17900,"operation_id":"0x66fd","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10005983}],"total_count":2}} WARN [2022-03-13 17:12:37,670] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8633770,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9492429,"total_server_duration_us":516882,"encode_duration_us":19856,"operation_id":"0x68a9","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10010465},{"operation_name":"upsert","last_dispatch_duration_us":29979,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":1898566,"total_server_duration_us":621786,"encode_duration_us":22002,"operation_id":"0x66e1","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10009781},{"operation_name":"upsert","last_dispatch_duration_us":9035822,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10426159,"total_server_duration_us":9636178,"encode_duration_us":27234,"operation_id":"0x674b","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10003949},{"operation_name":"upsert","last_dispatch_duration_us":8887914,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9134123,"total_server_duration_us":144684,"encode_duration_us":17707,"operation_id":"0x684f","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10002629},{"operation_name":"upsert","last_dispatch_duration_us":9042980,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9042980,"total_server_duration_us":9032060,"encode_duration_us":27643,"operation_id":"0x684e","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":9046678},{"operation_name":"upsert","last_dispatch_duration_us":9040943,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9040943,"total_server_duration_us":9027816,"encode_duration_us":18928,"operation_id":"0x685d","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":9045035},{"operation_name":"upsert","last_dispatch_duration_us":3068248,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4753762,"total_server_duration_us":114642,"encode_duration_us":27157,"operation_id":"0x6b07","timeout_ms":10000,"last_server_duration_us":40270,"total_duration_us":5856496},{"operation_name":"upsert","last_dispatch_duration_us":3078873,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4592132,"total_server_duration_us":152042,"encode_duration_us":33641,"operation_id":"0x6b08","timeout_ms":10000,"last_server_duration_us":85742,"total_duration_us":5687238},{"operation_name":"get","last_dispatch_duration_us":2838073,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2838073,"total_server_duration_us":165,"operation_id":"0x6b5c","timeout_ms":30000,"last_server_duration_us":165,"total_duration_us":4948141},{"operation_name":"upsert","last_dispatch_duration_us":4637693,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4637693,"total_server_duration_us":86622,"encode_duration_us":19494,"operation_id":"0x6e35","timeout_ms":10000,"last_server_duration_us":86622,"total_duration_us":4642561}],"total_count":39}} WARN [2022-03-13 17:12:37,670] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14106134,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14106134,"total_server_duration_us":43,"operation_id":"0x688a","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":14109162},{"operation_name":"upsert","last_dispatch_duration_us":9454277,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10731699,"total_server_duration_us":120243,"encode_duration_us":20261,"operation_id":"0x6a9f","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10010252},{"operation_name":"upsert","last_dispatch_duration_us":218084,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":413121,"total_server_duration_us":42545,"encode_duration_us":26821,"operation_id":"0x6b1d","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10008490},{"operation_name":"upsert","last_dispatch_duration_us":137140,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10011467,"total_server_duration_us":126942,"encode_duration_us":44332,"operation_id":"0x6b03","timeout_ms":10000,"last_server_duration_us":31489,"total_duration_us":10007001},{"operation_name":"upsert","last_dispatch_duration_us":9456230,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9929123,"total_server_duration_us":98249,"encode_duration_us":24683,"operation_id":"0x6b0c","timeout_ms":10000,"last_server_duration_us":31681,"total_duration_us":10006069},{"operation_name":"upsert","last_dispatch_duration_us":229816,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":229816,"total_server_duration_us":22637,"encode_duration_us":20739,"operation_id":"0x6b1f","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10005962},{"operation_name":"upsert","last_dispatch_duration_us":211211,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":363088,"total_server_duration_us":45052,"encode_duration_us":18933,"operation_id":"0x6b13","timeout_ms":10000,"last_server_duration_us":24928,"total_duration_us":10002203},{"operation_name":"upsert","last_dispatch_duration_us":9758836,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9758836,"total_server_duration_us":9669483,"encode_duration_us":24603,"operation_id":"0x6b10","timeout_ms":10000,"last_server_duration_us":9669483,"total_duration_us":9765442},{"operation_name":"get","last_dispatch_duration_us":6693328,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":6693328,"total_server_duration_us":49,"operation_id":"0x6c87","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":6695649},{"operation_name":"get","last_dispatch_duration_us":32803,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":32803,"total_server_duration_us":1161,"operation_id":"0x6d11","timeout_ms":30000,"last_server_duration_us":1161,"total_duration_us":6214411}],"total_count":58}} INFO [2022-03-13 17:12:37,702] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,702] jepsen worker 34 - jepsen.util 754 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,703] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,703] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:12:37,703] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,704] jepsen worker 70 - jepsen.util 700 :invoke :write [23 39] INFO [2022-03-13 17:12:37,704] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:12:37,705] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:37,726] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,727] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] WARN [2022-03-13 17:12:37,743] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":19995,"operation_id":"0x6b68","timeout_ms":10000,"total_duration_us":10010597},{"operation_name":"upsert","last_dispatch_duration_us":8085672,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8438262,"total_server_duration_us":68741,"encode_duration_us":18749,"operation_id":"0x6760","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10010280},{"operation_name":"upsert","last_dispatch_duration_us":77943,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":628670,"total_server_duration_us":123975,"encode_duration_us":28185,"operation_id":"0x682e","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10009771},{"operation_name":"upsert","last_dispatch_duration_us":8179697,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":11932933,"total_server_duration_us":209851,"encode_duration_us":22043,"operation_id":"0x667f","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10005196},{"operation_name":"get","last_dispatch_duration_us":8288101,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8288101,"total_server_duration_us":8339,"operation_id":"0x6895","timeout_ms":30000,"last_server_duration_us":8339,"total_duration_us":8291848},{"operation_name":"upsert","last_dispatch_duration_us":3449321,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7586455,"total_server_duration_us":1982692,"encode_duration_us":20491,"operation_id":"0x69eb","timeout_ms":10000,"last_server_duration_us":1884590,"total_duration_us":7603467},{"operation_name":"get","last_dispatch_duration_us":6893988,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":6893988,"total_server_duration_us":69,"operation_id":"0x6942","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":6897146},{"operation_name":"upsert","last_dispatch_duration_us":4130828,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4130828,"total_server_duration_us":93043,"encode_duration_us":17904,"operation_id":"0x6dea","timeout_ms":10000,"last_server_duration_us":93043,"total_duration_us":4135562},{"operation_name":"get","last_dispatch_duration_us":4016311,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4016311,"total_server_duration_us":49,"operation_id":"0x6dfa","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":4016539},{"operation_name":"upsert","last_dispatch_duration_us":3874807,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3874807,"total_server_duration_us":118961,"encode_duration_us":30506,"operation_id":"0x6a95","timeout_ms":10000,"last_server_duration_us":118961,"total_duration_us":3879476}],"total_count":35}} INFO [2022-03-13 17:12:37,770] jepsen worker 57 - jepsen.util 2217 :info :write [19 12] :RequestTimeoutException INFO [2022-03-13 17:12:37,770] jepsen worker 57 - jepsen.util 2307 :invoke :read [19 nil] INFO [2022-03-13 17:12:37,808] jepsen worker 44 - jepsen.util 584 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,809] jepsen worker 44 - jepsen.util 584 :invoke :write [14 22] INFO [2022-03-13 17:12:37,815] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,815] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:37,817] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,818] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:12:37,820] jepsen worker 79 - jepsen.util 979 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,821] jepsen worker 79 - jepsen.util 979 :invoke :write [26 10] WARN [2022-03-13 17:12:37,844] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":24508111,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24508111,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9021452,"encode_duration_us":22073,"operation_id":"0x5fef","timeout_ms":10000,"last_server_duration_us":9021452,"total_duration_us":10003400},{"operation_name":"upsert","last_dispatch_duration_us":9207125,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18209007,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9113991,"encode_duration_us":19176,"operation_id":"0x63a4","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10005206}],"total_count":2}} INFO [2022-03-13 17:12:37,859] jepsen worker 53 - jepsen.util 503 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,860] jepsen worker 53 - jepsen.util 503 :invoke :write [17 24] INFO [2022-03-13 17:12:37,920] jepsen worker 80 - jepsen.util 710 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,920] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,920] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,922] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:12:37,922] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:37,923] jepsen worker 80 - jepsen.util 710 :invoke :write [26 32] INFO [2022-03-13 17:12:37,926] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,927] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:12:37,939] jepsen worker 82 - jepsen.util 2062 :ok :read [27 4] INFO [2022-03-13 17:12:37,939] jepsen worker 82 - jepsen.util 2062 :invoke :read [27 nil] INFO [2022-03-13 17:12:37,990] jepsen worker 52 - jepsen.util 592 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,990] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,991] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:37,992] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:38,028] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,034] jepsen worker 82 - jepsen.util 2062 :ok :read [27 4] INFO [2022-03-13 17:12:38,038] jepsen worker 77 - jepsen.util 707 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,038] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,055] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,068] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:38,068] jepsen worker 13 - jepsen.util 643 :invoke :write [4 44] INFO [2022-03-13 17:12:38,068] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:12:38,068] jepsen worker 82 - jepsen.util 2062 :invoke :read [27 nil] INFO [2022-03-13 17:12:38,068] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:38,069] jepsen worker 23 - jepsen.util 563 :invoke :write [7 6] INFO [2022-03-13 17:12:38,069] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:38,098] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:38,098] jepsen worker nemesis - couchbase.util Rebalance check count: 268 INFO [2022-03-13 17:12:38,098] jepsen worker nemesis - couchbase.util Rebalance stuck count: 173 INFO [2022-03-13 17:12:38,114] jepsen worker 82 - jepsen.util 2062 :ok :read [27 4] INFO [2022-03-13 17:12:38,115] jepsen worker 82 - jepsen.util 2062 :invoke :write [27 17] INFO [2022-03-13 17:12:38,124] jepsen worker 68 - jepsen.util 2228 :ok :read [22 30] INFO [2022-03-13 17:12:38,125] jepsen worker 68 - jepsen.util 2228 :invoke :write [22 31] INFO [2022-03-13 17:12:38,159] jepsen worker 61 - jepsen.util 601 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,160] jepsen worker 61 - jepsen.util 601 :invoke :write [20 49] INFO [2022-03-13 17:12:38,169] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,170] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:38,218] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,218] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:38,286] jepsen worker 66 - jepsen.util 1866 :info :write [22 21] :RequestTimeoutException INFO [2022-03-13 17:12:38,287] jepsen worker 66 - jepsen.util 1956 :invoke :read [22 nil] INFO [2022-03-13 17:12:38,300] jepsen worker 24 - jepsen.util 564 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,300] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:38,306] jepsen worker 76 - jepsen.util 706 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,307] jepsen worker 76 - jepsen.util 706 :invoke :write [25 2] INFO [2022-03-13 17:12:38,309] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,309] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,309] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,310] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:38,310] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:12:38,311] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:12:38,367] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,367] jepsen worker 21 - jepsen.util 741 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,368] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:38,368] jepsen worker 69 - jepsen.util 609 :invoke :write [23 33] INFO [2022-03-13 17:12:38,377] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,378] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:38,390] jepsen worker 46 - jepsen.util 2206 :info :write [15 46] :RequestTimeoutException INFO [2022-03-13 17:12:38,391] jepsen worker 46 - jepsen.util 2296 :invoke :read [15 nil] INFO [2022-03-13 17:12:38,396] jepsen worker 70 - jepsen.util 700 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,397] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:38,463] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,464] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:38,486] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,486] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:38,507] jepsen worker 15 - jepsen.util 2085 :info :write [5 21] :RequestTimeoutException INFO [2022-03-13 17:12:38,508] jepsen worker 15 - jepsen.util 2175 :invoke :write [5 2] INFO [2022-03-13 17:12:38,599] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,599] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,599] jepsen worker 37 - jepsen.util 2017 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,600] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:38,600] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,601] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:38,601] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 0] INFO [2022-03-13 17:12:38,602] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:38,606] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,606] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,606] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,607] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:38,607] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:38,608] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:12:38,616] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,617] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:38,618] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,619] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,619] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:12:38,619] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:38,620] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,621] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:38,622] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,623] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:12:38,639] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:38,640] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:38,640] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,640] jepsen worker 14 - jepsen.util 284 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,641] jepsen worker 62 - jepsen.util 512 :invoke :write [20 26] INFO [2022-03-13 17:12:38,642] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:38,651] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,651] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:12:38,653] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:38,655] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:12:38,710] jepsen worker 36 - jepsen.util 2106 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,710] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:38,711] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,712] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:12:38,712] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,713] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:38,735] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,735] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:12:38,747] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:38,747] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:12:38,787] jepsen worker 1 - jepsen.util 1891 :info :write [0 42] :RequestTimeoutException INFO [2022-03-13 17:12:38,788] jepsen worker 1 - jepsen.util 1981 :invoke :read [0 nil] INFO [2022-03-13 17:12:38,797] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,798] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:12:38,805] jepsen worker 37 - jepsen.util 2017 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,805] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,805] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,805] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:12:38,806] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 38] INFO [2022-03-13 17:12:38,806] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:38,817] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,817] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 45] INFO [2022-03-13 17:12:38,825] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,825] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:12:38,854] jepsen worker 84 - jepsen.util 2334 :info :write [28 48] :RequestTimeoutException INFO [2022-03-13 17:12:38,855] jepsen worker 84 - jepsen.util 2424 :invoke :read [28 nil] INFO [2022-03-13 17:12:38,873] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,874] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:38,898] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:38,898] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:38,904] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,905] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:38,910] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,911] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:38,912] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,912] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:12:38,931] jepsen worker 37 - jepsen.util 2017 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,932] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:38,935] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:38,935] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:12:38,961] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:38,962] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,006] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,007] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,025] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,025] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,033] jepsen worker 36 - jepsen.util 2106 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,034] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 35] INFO [2022-03-13 17:12:39,047] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:39,048] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,054] jepsen worker 16 - jepsen.util 1996 :ok :read [5 43] INFO [2022-03-13 17:12:39,055] jepsen worker 16 - jepsen.util 1996 :invoke :write [5 17] INFO [2022-03-13 17:12:39,063] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:39,063] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:12:39,095] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,095] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,095] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:12:39,096] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,100] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,102] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,103] jepsen worker 86 - jepsen.util 1976 :ok :read [28 44] INFO [2022-03-13 17:12:39,104] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,105] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:39,105] jepsen worker 86 - jepsen.util 1976 :invoke :write [28 9] INFO [2022-03-13 17:12:39,105] jepsen worker nemesis - couchbase.util Rebalance check count: 269 INFO [2022-03-13 17:12:39,105] jepsen worker nemesis - couchbase.util Rebalance stuck count: 174 INFO [2022-03-13 17:12:39,105] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,107] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,108] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:12:39,134] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:39,135] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:12:39,143] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,144] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:12:39,215] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,216] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,217] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,218] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:12:39,226] jepsen worker 59 - jepsen.util 2129 :info :write [19 33] :RequestTimeoutException INFO [2022-03-13 17:12:39,227] jepsen worker 59 - jepsen.util 2219 :invoke :write [19 1] INFO [2022-03-13 17:12:39,240] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,240] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,241] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,241] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,250] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,250] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,250] jepsen worker 36 - jepsen.util 2106 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,251] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,251] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:12:39,252] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,260] jepsen worker 19 - jepsen.util 2089 :info :write [6 40] :SyncWriteAmbiguous INFO [2022-03-13 17:12:39,261] jepsen worker 19 - jepsen.util 2179 :invoke :read [6 nil] INFO [2022-03-13 17:12:39,282] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,282] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:12:39,320] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,320] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,320] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,321] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:12:39,321] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:12:39,322] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,325] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,326] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 33] INFO [2022-03-13 17:12:39,330] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,330] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,332] jepsen worker 63 - jepsen.util 2043 :info :write [21 9] :SyncWriteAmbiguous INFO [2022-03-13 17:12:39,333] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,334] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:12:39,334] jepsen worker 63 - jepsen.util 2133 :invoke :read [21 nil] INFO [2022-03-13 17:12:39,335] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,336] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,346] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,347] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:12:39,351] jepsen worker 67 - jepsen.util 1777 :info :write [22 23] :RequestTimeoutException INFO [2022-03-13 17:12:39,352] jepsen worker 67 - jepsen.util 1867 :invoke :read [22 nil] INFO [2022-03-13 17:12:39,375] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:39,375] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,391] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,392] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:12:39,424] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,425] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,460] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,461] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,462] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:39,463] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:12:39,468] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,469] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 13] INFO [2022-03-13 17:12:39,507] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,508] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,535] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,536] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,539] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,539] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,540] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,540] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,541] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,541] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,542] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,543] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,546] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,546] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:12:39,558] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,559] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,559] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,560] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:12:39,560] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,560] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,562] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,562] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,563] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,564] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,565] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,565] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,567] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,567] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,568] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,569] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,570] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,570] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:39,571] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,572] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,579] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,579] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,591] jepsen worker 65 - jepsen.util 2585 :ok :read [21 :nil] INFO [2022-03-13 17:12:39,593] jepsen worker 65 - jepsen.util 2585 :invoke :read [21 nil] INFO [2022-03-13 17:12:39,596] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:39,596] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,597] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,598] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:12:39,609] jepsen worker 72 - jepsen.util 2142 :ok :read [24 24] INFO [2022-03-13 17:12:39,609] jepsen worker 72 - jepsen.util 2142 :invoke :read [24 nil] INFO [2022-03-13 17:12:39,612] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,612] jepsen worker 80 - jepsen.util 710 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,612] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,613] jepsen worker 78 - jepsen.util 528 :invoke :write [26 35] INFO [2022-03-13 17:12:39,613] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:39,614] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:39,616] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,616] jepsen worker 79 - jepsen.util 979 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,616] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,616] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,616] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,617] jepsen worker 21 - jepsen.util 741 :invoke :write [7 4] INFO [2022-03-13 17:12:39,618] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:12:39,618] jepsen worker 51 - jepsen.util 591 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,619] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:39,619] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:39,620] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:12:39,626] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:39,643] jepsen worker 88 - jepsen.util 1978 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,644] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:39,659] jepsen worker 47 - jepsen.util 2207 :ok :read [15 48] INFO [2022-03-13 17:12:39,659] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,660] jepsen worker 47 - jepsen.util 2207 :invoke :write [15 16] INFO [2022-03-13 17:12:39,660] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,675] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,676] jepsen worker 35 - jepsen.util 935 :invoke :write [11 17] INFO [2022-03-13 17:12:39,691] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,691] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,692] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:12:39,693] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,695] jepsen worker 18 - jepsen.util 1998 :ok :read [6 16] INFO [2022-03-13 17:12:39,696] jepsen worker 18 - jepsen.util 1998 :invoke :read [6 nil] INFO [2022-03-13 17:12:39,703] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:39,704] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:12:39,727] jepsen worker 42 - jepsen.util 402 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,728] jepsen worker 42 - jepsen.util 402 :invoke :write [14 23] INFO [2022-03-13 17:12:39,756] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,757] jepsen worker 36 - jepsen.util 2106 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,757] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:12:39,757] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,759] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,759] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 26] INFO [2022-03-13 17:12:39,761] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,762] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,763] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,764] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,765] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,766] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:39,776] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,776] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,776] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,779] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:39,779] jepsen worker 69 - jepsen.util 609 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,779] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:39,780] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:39,780] jepsen worker 79 - jepsen.util 979 :invoke :write [26 3] INFO [2022-03-13 17:12:39,784] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,786] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:12:39,792] jepsen worker 13 - jepsen.util 643 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,792] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,792] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,792] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,792] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,792] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,793] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:39,793] jepsen worker 80 - jepsen.util 710 :invoke :write [26 6] INFO [2022-03-13 17:12:39,794] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:39,795] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,796] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:39,796] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:39,827] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,827] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,827] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:12:39,828] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:12:39,865] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,865] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:39,867] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,867] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:39,868] jepsen worker 21 - jepsen.util 741 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,868] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:39,876] jepsen worker 35 - jepsen.util 935 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,876] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:39,881] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,881] jepsen worker 44 - jepsen.util 584 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,881] jepsen worker 44 - jepsen.util 584 :invoke :write [14 29] INFO [2022-03-13 17:12:39,882] jepsen worker 77 - jepsen.util 707 :invoke :write [25 36] INFO [2022-03-13 17:12:39,884] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,884] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,885] jepsen worker 22 - jepsen.util 652 :invoke :write [7 27] INFO [2022-03-13 17:12:39,886] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:12:39,892] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:39,893] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:12:39,914] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,914] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:39,919] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,920] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:39,921] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,922] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:12:39,940] jepsen worker 8 - jepsen.util 728 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,941] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:39,946] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,947] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:12:39,976] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,977] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:12:39,994] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,994] jepsen worker 78 - jepsen.util 528 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,994] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,995] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,994] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,996] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:39,997] jepsen worker 70 - jepsen.util 700 :invoke :write [23 44] INFO [2022-03-13 17:12:39,998] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:39,998] jepsen worker 34 - jepsen.util 754 :invoke :write [11 30] INFO [2022-03-13 17:12:39,999] jepsen worker 13 - jepsen.util 643 :invoke :write [4 29] INFO [2022-03-13 17:12:40,001] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:12:40,002] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,002] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,003] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:40,004] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,005] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,006] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,006] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:40,007] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,008] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,008] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,009] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,010] jepsen worker 52 - jepsen.util 592 :invoke :write [17 19] INFO [2022-03-13 17:12:40,016] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,019] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,020] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:12:40,031] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,031] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,031] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,031] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,032] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,033] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:40,033] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:40,033] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:12:40,043] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,044] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:12:40,061] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,061] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:12:40,090] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,090] jepsen worker 23 - jepsen.util 563 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,091] jepsen worker 12 - jepsen.util 732 :invoke :write [4 22] INFO [2022-03-13 17:12:40,092] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:40,097] jepsen worker 80 - jepsen.util 710 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,097] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,098] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:40,098] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:40,116] jepsen worker 37 - jepsen.util 2017 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,116] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,117] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:12:40,117] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,120] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:40,120] jepsen worker nemesis - couchbase.util Rebalance check count: 270 INFO [2022-03-13 17:12:40,120] jepsen worker nemesis - couchbase.util Rebalance stuck count: 175 INFO [2022-03-13 17:12:40,128] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,129] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:40,134] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,134] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,135] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:12:40,138] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:12:40,167] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,167] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:12:40,170] jepsen worker 22 - jepsen.util 652 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,171] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:40,172] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,173] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:40,175] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,176] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:40,177] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,178] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:40,179] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,180] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:12:40,202] jepsen worker 79 - jepsen.util 979 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,202] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,203] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:40,203] jepsen worker 79 - jepsen.util 979 :invoke :write [26 30] INFO [2022-03-13 17:12:40,210] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,210] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,210] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:12:40,211] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,213] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:40,214] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:12:40,215] jepsen worker 43 - jepsen.util 493 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,215] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:40,220] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,221] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,222] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,223] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,224] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,225] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,226] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,226] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,226] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:12:40,227] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:12:40,242] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,243] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,262] jepsen worker 34 - jepsen.util 754 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,263] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,265] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,266] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,267] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,268] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,269] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,269] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,270] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,270] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,271] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 0] INFO [2022-03-13 17:12:40,271] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,311] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,312] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:40,323] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,323] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,323] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,323] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,324] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,324] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:12:40,324] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:12:40,325] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:40,325] jepsen worker 69 - jepsen.util 609 :invoke :write [23 31] INFO [2022-03-13 17:12:40,326] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:40,327] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,327] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,327] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:40,328] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:40,361] jepsen worker 76 - jepsen.util 706 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,362] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:40,363] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,363] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,364] jepsen worker 60 - jepsen.util 600 :invoke :write [20 0] INFO [2022-03-13 17:12:40,364] jepsen worker 51 - jepsen.util 591 :invoke :write [17 48] INFO [2022-03-13 17:12:40,365] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,365] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,365] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,366] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,379] jepsen worker 70 - jepsen.util 700 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,380] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:40,400] jepsen worker 28 - jepsen.util 2368 :info :write [9 4] :RequestTimeoutException INFO [2022-03-13 17:12:40,401] jepsen worker 28 - jepsen.util 2458 :invoke :write [9 18] INFO [2022-03-13 17:12:40,406] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,406] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,411] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,412] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,412] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,412] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:40,412] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,413] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:12:40,418] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,420] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,421] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,422] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,423] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,423] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,425] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,425] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,426] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,427] jepsen worker 35 - jepsen.util 935 :invoke :write [11 17] INFO [2022-03-13 17:12:40,429] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,430] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:12:40,452] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:40,453] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:12:40,492] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,492] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,494] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:40,494] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:40,502] jepsen worker 36 - jepsen.util 2106 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,502] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,502] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,502] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,503] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,503] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,503] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,504] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,504] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 40] INFO [2022-03-13 17:12:40,504] jepsen worker 61 - jepsen.util 601 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,504] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,504] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 37] INFO [2022-03-13 17:12:40,505] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,505] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:40,506] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:12:40,509] jepsen worker 7 - jepsen.util 637 :invoke :write [2 4] INFO [2022-03-13 17:12:40,513] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,513] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:40,522] jepsen worker 13 - jepsen.util 643 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,523] jepsen worker 13 - jepsen.util 643 :invoke :write [4 47] INFO [2022-03-13 17:12:40,541] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,542] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:40,561] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,562] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:12:40,578] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,579] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,588] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,589] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:12:40,593] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:40,594] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:12:40,596] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,597] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:12:40,602] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,603] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:12:40,627] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,627] jepsen worker 35 - jepsen.util 935 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,628] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,629] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:12:40,630] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,630] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,630] jepsen worker 52 - jepsen.util 592 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,631] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,631] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,631] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,631] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,631] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,632] jepsen worker 70 - jepsen.util 700 :invoke :write [23 30] INFO [2022-03-13 17:12:40,632] jepsen worker 80 - jepsen.util 710 :invoke :write [26 16] INFO [2022-03-13 17:12:40,633] jepsen worker 78 - jepsen.util 528 :invoke :write [26 9] INFO [2022-03-13 17:12:40,634] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,634] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:40,635] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,637] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,638] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,640] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,640] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,641] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,642] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:40,643] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,644] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:12:40,648] jepsen worker 79 - jepsen.util 979 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,648] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,650] jepsen worker 79 - jepsen.util 979 :invoke :write [26 3] INFO [2022-03-13 17:12:40,651] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:12:40,660] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,661] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,673] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,673] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:40,676] jepsen worker 36 - jepsen.util 2106 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,677] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 42] INFO [2022-03-13 17:12:40,686] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,686] jepsen worker 22 - jepsen.util 652 :invoke :write [7 37] INFO [2022-03-13 17:12:40,703] jepsen worker 77 - jepsen.util 707 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,706] jepsen worker 77 - jepsen.util 707 :invoke :write [25 22] INFO [2022-03-13 17:12:40,736] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,738] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:12:40,744] jepsen worker 37 - jepsen.util 2017 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,746] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:12:40,747] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,748] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,748] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:12:40,748] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,774] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,774] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:12:40,782] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,783] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:12:40,822] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,823] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,838] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,838] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,838] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,839] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 33] INFO [2022-03-13 17:12:40,840] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:12:40,840] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:40,842] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,843] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:12:40,864] jepsen worker 36 - jepsen.util 2106 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,865] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 36] INFO [2022-03-13 17:12:40,887] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,888] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:40,897] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,898] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 8] INFO [2022-03-13 17:12:40,918] jepsen worker 32 - jepsen.util 1922 :info :write [10 42] :RequestTimeoutException INFO [2022-03-13 17:12:40,918] jepsen worker 32 - jepsen.util 2012 :invoke :write [10 16] INFO [2022-03-13 17:12:40,927] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:40,928] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:12:40,940] jepsen worker 27 - jepsen.util 2457 :info :write [9 19] :RequestTimeoutException INFO [2022-03-13 17:12:40,941] jepsen worker 27 - jepsen.util 2547 :invoke :write [9 18] INFO [2022-03-13 17:12:40,941] jepsen worker 3 - jepsen.util 1893 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,946] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:12:40,968] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,969] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,971] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,971] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:40,972] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,973] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:12:40,973] jepsen worker 20 - jepsen.util 2540 :info :write [6 1] :RequestTimeoutException INFO [2022-03-13 17:12:40,974] jepsen worker 20 - jepsen.util 2630 :invoke :read [6 nil] INFO [2022-03-13 17:12:40,978] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,979] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:40,979] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:12:40,980] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,018] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,018] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,047] jepsen worker 88 - jepsen.util 1978 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,048] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:12:41,091] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,092] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:12:41,092] jepsen worker 36 - jepsen.util 2106 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,093] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,093] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 29] INFO [2022-03-13 17:12:41,094] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:12:41,126] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:41,126] jepsen worker nemesis - couchbase.util Rebalance check count: 271 INFO [2022-03-13 17:12:41,126] jepsen worker nemesis - couchbase.util Rebalance stuck count: 176 INFO [2022-03-13 17:12:41,138] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,139] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:41,140] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,141] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,141] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:12:41,142] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:12:41,178] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,179] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:12:41,208] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:41,209] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:12:41,211] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,212] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:12:41,225] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,226] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,252] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,253] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,276] jepsen worker 29 - jepsen.util 2279 :info :write [9 21] :RequestTimeoutException INFO [2022-03-13 17:12:41,276] jepsen worker 29 - jepsen.util 2369 :invoke :write [9 26] INFO [2022-03-13 17:12:41,277] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:41,278] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,292] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,292] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,320] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,320] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:12:41,324] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,325] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:41,340] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,340] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,341] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:41,341] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,344] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,344] jepsen worker 53 - jepsen.util 503 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,345] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:41,346] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:41,348] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,349] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:41,371] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:41,372] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 9] INFO [2022-03-13 17:12:41,372] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:41,373] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,390] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,390] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,391] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:12:41,392] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:12:41,403] jepsen worker 69 - jepsen.util 609 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,403] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:41,426] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:41,427] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:12:41,451] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,451] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,457] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,457] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:12:41,467] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,467] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:12:41,482] jepsen worker 13 - jepsen.util 643 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,482] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,482] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,483] jepsen worker 34 - jepsen.util 754 :invoke :write [11 43] INFO [2022-03-13 17:12:41,484] jepsen worker 13 - jepsen.util 643 :invoke :write [4 16] INFO [2022-03-13 17:12:41,484] jepsen worker 52 - jepsen.util 592 :invoke :write [17 24] INFO [2022-03-13 17:12:41,494] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,494] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,494] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,496] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:41,496] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:41,497] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:41,510] jepsen worker 36 - jepsen.util 2106 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,511] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:41,517] jepsen worker 88 - jepsen.util 1978 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,519] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:12:41,545] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,546] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,555] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,555] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:12:41,583] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,584] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,592] jepsen worker 44 - jepsen.util 584 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,592] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,596] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:41,596] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:41,609] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:41,610] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:12:41,629] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,630] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:41,634] jepsen worker 22 - jepsen.util 652 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,634] jepsen worker 22 - jepsen.util 652 :invoke :write [7 38] INFO [2022-03-13 17:12:41,635] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,635] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,635] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,636] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 1] INFO [2022-03-13 17:12:41,638] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:12:41,638] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:12:41,645] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,645] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,645] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,646] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:12:41,646] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,647] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,647] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,648] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:41,649] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,649] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,649] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:41,650] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:12:41,650] jepsen worker 62 - jepsen.util 512 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,650] jepsen worker 62 - jepsen.util 512 :invoke :write [20 3] INFO [2022-03-13 17:12:41,667] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,668] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:41,681] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,682] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:12:41,720] jepsen worker 70 - jepsen.util 700 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,720] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:12:41,737] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,737] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,738] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:41,738] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:41,739] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,739] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,739] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:41,740] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:12:41,741] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,742] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:12:41,749] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,750] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,750] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,751] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 19] INFO [2022-03-13 17:12:41,757] jepsen worker 79 - jepsen.util 979 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,758] jepsen worker 79 - jepsen.util 979 :invoke :write [26 0] INFO [2022-03-13 17:12:41,775] jepsen worker 36 - jepsen.util 2106 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,776] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:41,791] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,791] jepsen worker 12 - jepsen.util 732 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,792] jepsen worker 76 - jepsen.util 706 :invoke :write [25 36] INFO [2022-03-13 17:12:41,793] jepsen worker 12 - jepsen.util 732 :invoke :write [4 30] INFO [2022-03-13 17:12:41,813] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,813] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:12:41,821] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,822] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:12:41,842] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,843] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,842] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,843] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,843] jepsen worker 21 - jepsen.util 741 :invoke :write [7 44] INFO [2022-03-13 17:12:41,843] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,845] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,846] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:41,846] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:41,847] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:41,848] jepsen worker 24 - jepsen.util 564 :invoke :write [8 36] INFO [2022-03-13 17:12:41,849] jepsen worker 69 - jepsen.util 609 :invoke :write [23 19] INFO [2022-03-13 17:12:41,857] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,857] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:41,867] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:41,867] jepsen worker 42 - jepsen.util 402 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,868] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,867] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,868] jepsen worker 8 - jepsen.util 728 :invoke :write [2 14] INFO [2022-03-13 17:12:41,869] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:41,869] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:12:41,873] jepsen worker 6 - jepsen.util 636 :invoke :write [2 22] INFO [2022-03-13 17:12:41,882] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,883] jepsen worker 88 - jepsen.util 1978 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,884] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,885] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,916] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,917] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:12:41,919] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:41,920] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:12:41,929] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:41,929] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,934] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,935] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:41,936] jepsen worker 78 - jepsen.util 528 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,937] jepsen worker 22 - jepsen.util 652 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,937] jepsen worker 78 - jepsen.util 528 :invoke :write [26 12] INFO [2022-03-13 17:12:41,937] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:41,943] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,943] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,943] jepsen worker 79 - jepsen.util 979 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,944] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,944] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:12:41,944] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,944] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] INFO [2022-03-13 17:12:41,945] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:41,946] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:41,947] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:12:41,949] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,950] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:41,960] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,961] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:12:41,968] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,969] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:12:41,969] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,970] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:12:41,974] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:41,974] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:41,976] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,976] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 39] INFO [2022-03-13 17:12:41,979] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,979] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,980] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:41,980] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 26] INFO [2022-03-13 17:12:41,983] jepsen worker 60 - jepsen.util 600 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:41,984] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:42,016] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:42,017] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:12:42,052] jepsen worker 21 - jepsen.util 741 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,053] jepsen worker 21 - jepsen.util 741 :invoke :write [7 0] INFO [2022-03-13 17:12:42,061] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,062] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,062] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,062] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,070] jepsen worker 26 - jepsen.util 476 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,070] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,070] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,070] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:42,071] jepsen worker 26 - jepsen.util 476 :invoke :write [8 13] INFO [2022-03-13 17:12:42,071] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,073] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,074] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:12:42,075] jepsen worker 80 - jepsen.util 710 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,076] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:42,081] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,081] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,082] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:12:42,083] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:12:42,122] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,123] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:42,132] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:42,132] jepsen worker nemesis - couchbase.util Rebalance check count: 272 INFO [2022-03-13 17:12:42,132] jepsen worker nemesis - couchbase.util Rebalance stuck count: 177 INFO [2022-03-13 17:12:42,136] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,137] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,141] jepsen worker 24 - jepsen.util 564 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,141] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:42,145] jepsen worker 3 - jepsen.util 1893 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,145] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:12:42,178] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:42,179] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 29] INFO [2022-03-13 17:12:42,189] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,189] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,190] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,190] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,190] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,191] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 1] INFO [2022-03-13 17:12:42,203] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,203] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:42,218] jepsen worker 51 - jepsen.util 591 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,218] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:42,224] jepsen worker 34 - jepsen.util 754 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,225] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:12:42,226] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,226] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:12:42,232] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,232] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,233] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:12:42,234] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:42,235] jepsen worker 88 - jepsen.util 1978 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,235] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,236] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:12:42,236] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,245] jepsen worker 69 - jepsen.util 609 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,245] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,245] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,246] jepsen worker 79 - jepsen.util 979 :invoke :write [26 9] INFO [2022-03-13 17:12:42,248] jepsen worker 71 - jepsen.util 521 :invoke :write [23 1] INFO [2022-03-13 17:12:42,252] jepsen worker 69 - jepsen.util 609 :invoke :write [23 7] INFO [2022-03-13 17:12:42,270] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:42,270] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,272] jepsen worker 7 - jepsen.util 637 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,273] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,273] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,273] jepsen worker 7 - jepsen.util 637 :invoke :write [2 40] INFO [2022-03-13 17:12:42,274] jepsen worker 53 - jepsen.util 503 :invoke :write [17 44] INFO [2022-03-13 17:12:42,274] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:42,276] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,277] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:42,282] jepsen worker 89 - jepsen.util 2429 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,282] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,285] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:12:42,286] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,288] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,290] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:12:42,296] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,296] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,297] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:12:42,297] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:12:42,303] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,304] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:12:42,308] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:42,308] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 35] INFO [2022-03-13 17:12:42,326] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:42,327] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,336] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,336] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,368] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,369] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:12:42,390] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,390] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:42,391] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,391] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:12:42,397] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,398] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,417] jepsen worker 36 - jepsen.util 2106 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,419] jepsen worker 77 - jepsen.util 707 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,419] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:42,419] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,419] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,420] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:42,421] jepsen worker 77 - jepsen.util 707 :invoke :write [25 29] INFO [2022-03-13 17:12:42,421] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:42,427] jepsen worker 13 - jepsen.util 643 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,429] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:42,453] jepsen worker 88 - jepsen.util 1978 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,455] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 35] INFO [2022-03-13 17:12:42,464] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:42,464] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,476] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,476] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,476] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,477] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,482] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,483] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:42,489] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,489] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,490] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,490] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,491] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,491] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,491] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,492] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:12:42,492] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,493] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,494] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,494] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:12:42,502] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,502] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:42,505] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:42,506] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:12:42,528] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,528] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:12:42,541] jepsen worker 88 - jepsen.util 1978 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,541] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:12:42,556] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,557] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:12:42,560] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:42,560] jepsen worker 62 - jepsen.util 512 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,560] jepsen worker 43 - jepsen.util 493 :invoke :write [14 1] INFO [2022-03-13 17:12:42,561] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:42,570] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:42,571] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,576] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,576] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,578] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 36] INFO [2022-03-13 17:12:42,579] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 5] INFO [2022-03-13 17:12:42,587] jepsen worker 21 - jepsen.util 741 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,587] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,588] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:42,588] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,588] jepsen worker 24 - jepsen.util 564 :invoke :write [8 30] INFO [2022-03-13 17:12:42,589] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:42,605] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:42,605] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,622] jepsen worker 79 - jepsen.util 979 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,623] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:42,637] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,637] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,642] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,643] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:12:42,672] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,672] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,674] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,674] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:12:42,695] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:42,696] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:12:42,708] jepsen worker 36 - jepsen.util 2106 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,709] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,712] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,712] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:42,713] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,714] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:42,715] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,715] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:12:42,720] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,721] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:12:42,721] jepsen worker 69 - jepsen.util 609 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,722] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:42,743] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,743] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,744] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:12:42,744] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:12:42,780] jepsen worker 37 - jepsen.util 2017 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,781] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,805] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,805] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,809] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,809] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:42,844] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,845] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,845] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:12:42,846] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:12:42,852] jepsen worker 76 - jepsen.util 706 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,852] jepsen worker 76 - jepsen.util 706 :invoke :write [25 1] INFO [2022-03-13 17:12:42,864] jepsen worker 71 - jepsen.util 521 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,865] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:12:42,877] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,877] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,877] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,877] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,878] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,878] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,879] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,880] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,903] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:42,903] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 37] INFO [2022-03-13 17:12:42,927] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,927] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,927] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,928] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,928] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:12:42,928] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,929] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 2] INFO [2022-03-13 17:12:42,929] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,931] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,932] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:42,933] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,933] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:12:42,945] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,946] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,946] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:12:42,947] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:42,959] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:42,959] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:42,985] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,986] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,986] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,987] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:12:42,987] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,987] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:42,988] jepsen worker 21 - jepsen.util 741 :invoke :write [7 30] INFO [2022-03-13 17:12:42,989] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:12:42,995] jepsen worker 6 - jepsen.util 636 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:42,997] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:42,998] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:43,001] jepsen worker 52 - jepsen.util 592 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,005] jepsen worker 88 - jepsen.util 1978 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,060] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,060] jepsen worker 52 - jepsen.util 592 :invoke :write [17 9] INFO [2022-03-13 17:12:43,041] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,060] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 41] INFO [2022-03-13 17:12:43,060] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 0] INFO [2022-03-13 17:12:43,049] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,062] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,050] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,063] jepsen worker 69 - jepsen.util 609 :invoke :write [23 4] INFO [2022-03-13 17:12:43,050] jepsen worker 24 - jepsen.util 564 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,051] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,064] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] INFO [2022-03-13 17:12:43,064] jepsen worker 79 - jepsen.util 979 :invoke :write [26 5] INFO [2022-03-13 17:12:43,074] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,075] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,076] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,077] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,078] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,080] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:43,103] jepsen worker 36 - jepsen.util 2106 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,105] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,105] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,105] jepsen worker 12 - jepsen.util 732 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,105] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,105] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,105] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:43,108] jepsen worker 23 - jepsen.util 563 :invoke :write [7 46] INFO [2022-03-13 17:12:43,109] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:43,110] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:43,111] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:12:43,112] jepsen worker 12 - jepsen.util 732 :invoke :write [4 40] INFO [2022-03-13 17:12:43,131] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:43,133] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 19] INFO [2022-03-13 17:12:43,137] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:43,138] jepsen worker nemesis - couchbase.util Rebalance check count: 273 INFO [2022-03-13 17:12:43,138] jepsen worker nemesis - couchbase.util Rebalance stuck count: 178 INFO [2022-03-13 17:12:43,140] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,140] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:12:43,143] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,144] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:12:43,146] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,147] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:12:43,150] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,150] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,150] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,150] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,151] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:12:43,151] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:43,151] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,152] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,154] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,154] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,155] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:12:43,156] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 5] INFO [2022-03-13 17:12:43,163] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,164] jepsen worker 88 - jepsen.util 1978 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,172] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:12:43,173] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,205] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:43,206] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 36] INFO [2022-03-13 17:12:43,210] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,211] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:12:43,216] jepsen worker 31 - jepsen.util 2191 :info :write [10 46] :RequestTimeoutException INFO [2022-03-13 17:12:43,217] jepsen worker 31 - jepsen.util 2281 :invoke :write [10 1] INFO [2022-03-13 17:12:43,227] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,228] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:12:43,234] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,235] jepsen worker 78 - jepsen.util 528 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,235] jepsen worker 78 - jepsen.util 528 :invoke :write [26 2] INFO [2022-03-13 17:12:43,236] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:43,267] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,267] jepsen worker 37 - jepsen.util 2017 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,267] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,268] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 26] INFO [2022-03-13 17:12:43,282] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,282] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:12:43,291] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,291] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:43,296] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,296] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:12:43,322] jepsen worker 89 - jepsen.util 2429 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,323] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,330] jepsen worker 88 - jepsen.util 1978 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,330] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:12:43,343] jepsen worker 21 - jepsen.util 741 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,343] jepsen worker 21 - jepsen.util 741 :invoke :write [7 6] INFO [2022-03-13 17:12:43,349] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,349] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:12:43,378] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:43,379] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:12:43,383] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,383] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,405] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,405] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:12:43,448] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,448] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,449] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:12:43,449] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 1] INFO [2022-03-13 17:12:43,467] jepsen worker 3 - jepsen.util 1893 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,468] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,468] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,468] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,469] jepsen worker 13 - jepsen.util 643 :invoke :write [4 19] INFO [2022-03-13 17:12:43,470] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 36] INFO [2022-03-13 17:12:43,471] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:43,471] jepsen worker 34 - jepsen.util 754 :invoke :write [11 47] INFO [2022-03-13 17:12:43,478] jepsen worker 77 - jepsen.util 707 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,479] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:12:43,481] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,482] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 12] INFO [2022-03-13 17:12:43,490] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,491] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:12:43,492] jepsen worker 69 - jepsen.util 609 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,493] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:43,552] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,553] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:43,556] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,556] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:12:43,560] jepsen worker 79 - jepsen.util 979 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,560] jepsen worker 79 - jepsen.util 979 :invoke :write [26 23] INFO [2022-03-13 17:12:43,572] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,573] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,596] jepsen worker 37 - jepsen.util 2017 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,597] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 8] INFO [2022-03-13 17:12:43,608] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,609] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,613] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,613] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:12:43,614] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:43,615] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,630] jepsen worker 8 - jepsen.util 728 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,630] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,703] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:43,704] jepsen worker 88 - jepsen.util 1978 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,704] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,704] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,704] jepsen worker 3 - jepsen.util 1893 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,704] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,705] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,707] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,707] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:43,708] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,708] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:12:43,709] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:12:43,710] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:12:43,710] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:12:43,710] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:12:43,711] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:12:43,721] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,721] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:12:43,739] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:43,739] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,754] jepsen worker 36 - jepsen.util 2106 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,755] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,779] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,780] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,781] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,782] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:12:43,816] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,818] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,842] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,843] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,852] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:43,852] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:12:43,859] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,859] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,880] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:43,881] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:43,892] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,892] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,893] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,893] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,906] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,906] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:12:43,920] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,921] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,931] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,932] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:43,933] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,933] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:12:43,953] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,953] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,953] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,954] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,954] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,955] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,966] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,966] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,967] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,968] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 6] INFO [2022-03-13 17:12:43,969] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,969] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:12:43,984] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:43,985] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,985] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:12:43,985] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:43,990] jepsen worker 37 - jepsen.util 2017 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,990] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,991] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:43,991] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:43,992] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:12:43,998] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 30] INFO [2022-03-13 17:12:43,999] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,000] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:12:44,028] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,029] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:12:44,050] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:44,050] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:12:44,087] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,088] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:12:44,113] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,114] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:12:44,128] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,129] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:12:44,145] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5847953216374269}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:44,145] jepsen worker nemesis - couchbase.util Rebalance check count: 274 INFO [2022-03-13 17:12:44,146] jepsen worker nemesis - couchbase.util Rebalance stuck count: 179 INFO [2022-03-13 17:12:44,152] jepsen worker 3 - jepsen.util 1893 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,152] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:12:44,198] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,199] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:12:44,200] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,201] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:12:44,254] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,254] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:12:44,275] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,275] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:12:44,290] jepsen worker 36 - jepsen.util 2106 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,290] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:44,325] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,326] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:12:44,346] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,347] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:12:44,363] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,363] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:44,402] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,403] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:44,425] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,426] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:44,433] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,434] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:44,473] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,474] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,477] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,477] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:12:44,496] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,496] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,520] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:44,521] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:12:44,543] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,543] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,543] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,543] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:44,544] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 13] INFO [2022-03-13 17:12:44,544] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:44,550] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,550] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,554] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:44,555] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,555] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,556] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:12:44,556] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:12:44,557] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:12:44,569] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,569] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,569] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:12:44,570] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:12:44,585] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:44,585] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,609] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,609] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:12:44,639] jepsen worker 36 - jepsen.util 2106 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,640] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:12:44,665] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,666] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:44,667] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,667] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:44,668] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,669] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:44,670] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,670] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:44,675] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:44,676] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,677] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,677] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,712] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,713] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 17] INFO [2022-03-13 17:12:44,736] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,737] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:12:44,746] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:44,746] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,758] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,758] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,789] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:44,789] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:12:44,806] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,807] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:44,832] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,833] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:12:44,859] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:44,859] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,865] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,866] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:12:44,878] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:44,878] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:44,935] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,936] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 30] INFO [2022-03-13 17:12:44,939] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,939] jepsen worker 49 - jepsen.util 1939 :ok :read [16 :nil] INFO [2022-03-13 17:12:44,939] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:44,940] jepsen worker 49 - jepsen.util 1939 :invoke :read [16 nil] INFO [2022-03-13 17:12:44,975] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:44,975] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:12:44,978] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:44,979] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,026] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,027] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:12:45,039] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:45,040] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:12:45,045] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,046] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] INFO [2022-03-13 17:12:45,066] jepsen worker 37 - jepsen.util 2017 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,067] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,068] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,070] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:12:45,077] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,078] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,095] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:45,096] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,137] jepsen worker 36 - jepsen.util 2106 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,137] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,137] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,137] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,138] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,139] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,139] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 6] INFO [2022-03-13 17:12:45,140] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:12:45,151] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:45,151] jepsen worker nemesis - couchbase.util Rebalance check count: 275 INFO [2022-03-13 17:12:45,151] jepsen worker nemesis - couchbase.util Rebalance stuck count: 179 INFO [2022-03-13 17:12:45,178] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,178] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,179] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,179] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:12:45,194] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,194] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:12:45,211] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:45,212] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:12:45,242] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,242] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,242] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,244] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 38] INFO [2022-03-13 17:12:45,244] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,245] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,245] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:12:45,246] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:12:45,249] jepsen worker 52 - jepsen.util 592 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,249] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,250] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,251] jepsen worker 80 - jepsen.util 710 :invoke :write [26 31] INFO [2022-03-13 17:12:45,253] jepsen worker 52 - jepsen.util 592 :invoke :write [17 20] INFO [2022-03-13 17:12:45,258] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:12:45,266] jepsen worker 76 - jepsen.util 706 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,267] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,267] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:45,267] jepsen worker 76 - jepsen.util 706 :invoke :write [25 36] INFO [2022-03-13 17:12:45,268] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:45,268] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:45,274] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,275] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,275] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:45,275] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,276] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:45,276] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:12:45,280] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,281] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,303] jepsen worker 26 - jepsen.util 476 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,304] jepsen worker 26 - jepsen.util 476 :invoke :write [8 29] INFO [2022-03-13 17:12:45,315] jepsen worker 88 - jepsen.util 1978 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,315] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,316] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,317] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,353] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,354] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:12:45,359] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,359] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:12:45,372] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:45,373] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:12:45,385] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:45,385] jepsen worker 30 - jepsen.util 2280 :info :write [10 29] :RequestTimeoutException INFO [2022-03-13 17:12:45,386] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 4] INFO [2022-03-13 17:12:45,386] jepsen worker 30 - jepsen.util 2370 :invoke :read [10 nil] INFO [2022-03-13 17:12:45,403] jepsen worker 21 - jepsen.util 741 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,403] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,403] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:45,404] jepsen worker 21 - jepsen.util 741 :invoke :write [7 23] INFO [2022-03-13 17:12:45,419] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,419] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:45,425] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,426] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:12:45,426] jepsen worker 36 - jepsen.util 2106 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,427] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,446] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,446] jepsen worker 60 - jepsen.util 600 :invoke :write [20 4] INFO [2022-03-13 17:12:45,453] jepsen worker 81 - jepsen.util 2421 :info :write [27 11] :SyncWriteAmbiguous INFO [2022-03-13 17:12:45,454] jepsen worker 81 - jepsen.util 2511 :invoke :write [27 35] INFO [2022-03-13 17:12:45,456] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,456] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,457] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,457] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:12:45,488] jepsen worker 79 - jepsen.util 979 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,488] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:45,498] jepsen worker 88 - jepsen.util 1978 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,499] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,499] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,499] jepsen worker 78 - jepsen.util 528 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,499] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,500] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:45,500] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:45,501] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:45,502] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,502] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:45,503] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,504] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,504] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:12:45,506] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:12:45,511] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,512] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,516] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,517] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:45,519] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,519] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:12:45,524] jepsen worker 53 - jepsen.util 503 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,525] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:45,525] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,525] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,526] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,527] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:12:45,528] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:12:45,534] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:12:45,551] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:45,552] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,565] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,566] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:12:45,577] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,577] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:12:45,599] jepsen worker 40 - jepsen.util 2200 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,602] jepsen worker 40 - jepsen.util 2200 :invoke :read [13 nil] INFO [2022-03-13 17:12:45,603] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,603] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:12:45,612] jepsen worker 83 - jepsen.util 1523 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,613] jepsen worker 83 - jepsen.util 1523 :invoke :write [27 8] INFO [2022-03-13 17:12:45,615] jepsen worker 0 - jepsen.util 2250 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,616] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:45,616] jepsen worker 41 - jepsen.util 2021 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,616] jepsen worker 0 - jepsen.util 2250 :invoke :read [0 nil] INFO [2022-03-13 17:12:45,617] jepsen worker 41 - jepsen.util 2021 :invoke :write [13 14] INFO [2022-03-13 17:12:45,618] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,628] jepsen worker 23 - jepsen.util 563 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,629] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,629] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,629] jepsen worker 23 - jepsen.util 563 :invoke :write [7 38] INFO [2022-03-13 17:12:45,629] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,633] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:12:45,634] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:12:45,638] jepsen worker 66 - jepsen.util 1956 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,639] jepsen worker 66 - jepsen.util 1956 :invoke :write [22 38] INFO [2022-03-13 17:12:45,642] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,643] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:45,643] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,644] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,645] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 16] INFO [2022-03-13 17:12:45,645] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,646] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 15] INFO [2022-03-13 17:12:45,650] jepsen worker 7 - jepsen.util 637 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,650] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:45,679] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,680] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:12:45,686] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,687] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,690] jepsen worker 12 - jepsen.util 732 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,691] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:45,693] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,694] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,695] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,696] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,697] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,698] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:12:45,716] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,716] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:45,723] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,724] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,724] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:45,725] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:45,726] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,726] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,727] jepsen worker 34 - jepsen.util 754 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,728] jepsen worker 34 - jepsen.util 754 :invoke :write [11 37] INFO [2022-03-13 17:12:45,743] jepsen worker 88 - jepsen.util 1978 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,744] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,747] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,747] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:12:45,769] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:45,770] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:12:45,785] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:45,786] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,787] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:45,788] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 40] INFO [2022-03-13 17:12:45,791] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,791] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,791] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:45,791] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:12:45,802] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,803] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:12:45,822] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,822] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,828] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,829] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:12:45,833] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,833] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,833] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,833] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,833] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,834] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:45,835] jepsen worker 8 - jepsen.util 728 :invoke :write [2 7] INFO [2022-03-13 17:12:45,835] jepsen worker 62 - jepsen.util 512 :invoke :write [20 21] INFO [2022-03-13 17:12:45,836] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:45,836] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:45,847] jepsen worker 21 - jepsen.util 741 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,847] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,848] jepsen worker 21 - jepsen.util 741 :invoke :write [7 7] INFO [2022-03-13 17:12:45,848] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:45,849] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,850] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,887] jepsen worker 88 - jepsen.util 1978 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,888] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:45,906] jepsen worker 2 - jepsen.util 2432 :info :write [0 2] :RequestTimeoutException INFO [2022-03-13 17:12:45,906] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,906] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,906] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:45,907] jepsen worker 2 - jepsen.util 2522 :invoke :write [0 19] INFO [2022-03-13 17:12:45,908] jepsen worker 44 - jepsen.util 584 :invoke :write [14 44] INFO [2022-03-13 17:12:45,908] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:45,909] jepsen worker 51 - jepsen.util 591 :invoke :write [17 7] INFO [2022-03-13 17:12:45,910] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,911] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:12:45,919] jepsen worker 13 - jepsen.util 643 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,922] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:45,922] jepsen worker 13 - jepsen.util 643 :invoke :write [4 14] INFO [2022-03-13 17:12:45,922] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:45,929] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,930] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,949] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,949] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,950] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:45,950] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:12:45,951] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,952] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:12:45,960] jepsen worker 43 - jepsen.util 493 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,960] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,961] jepsen worker 43 - jepsen.util 493 :invoke :write [14 36] INFO [2022-03-13 17:12:45,961] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:45,963] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,964] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:12:45,972] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:45,973] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:12:45,978] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,979] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,983] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,984] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,986] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,987] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,988] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,989] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:45,990] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:45,991] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:12:45,999] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,000] jepsen worker 24 - jepsen.util 564 :invoke :write [8 1] INFO [2022-03-13 17:12:46,007] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,008] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 1] INFO [2022-03-13 17:12:46,040] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,040] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,068] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,069] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:46,070] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,071] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:46,072] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,073] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:12:46,086] jepsen worker 39 - jepsen.util 2109 :info :write [13 30] :RequestTimeoutException INFO [2022-03-13 17:12:46,087] jepsen worker 39 - jepsen.util 2199 :invoke :read [13 nil] INFO [2022-03-13 17:12:46,088] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,088] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,088] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:46,089] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:12:46,095] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,096] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] WARN [2022-03-13 17:12:46,108] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16409881,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16409881,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9674945,"encode_duration_us":25336,"operation_id":"0x6960","timeout_ms":10000,"last_server_duration_us":9674945,"total_duration_us":10003428},{"operation_name":"upsert","last_dispatch_duration_us":27507441,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27507441,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9041612,"encode_duration_us":17806,"operation_id":"0x61cc","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10010356},{"operation_name":"upsert","last_dispatch_duration_us":33747397,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":33747397,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9029938,"encode_duration_us":20945,"operation_id":"0x636b","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10007306}],"total_count":3}} WARN [2022-03-13 17:12:46,108] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18498372,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":18498372,"total_server_duration_us":271179,"operation_id":"0x6b96","timeout_ms":30000,"last_server_duration_us":271179,"total_duration_us":18498697},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":29946,"operation_id":"0x6b74","timeout_ms":10000,"total_duration_us":10010065},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":26284,"operation_id":"0x6ca9","timeout_ms":10000,"total_duration_us":10003875},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":16749,"operation_id":"0x6c1a","timeout_ms":10000,"total_duration_us":10001947},{"operation_name":"upsert","last_dispatch_duration_us":3430223,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3430223,"total_server_duration_us":91382,"encode_duration_us":21054,"operation_id":"0x6e53","timeout_ms":10000,"last_server_duration_us":91382,"total_duration_us":3434691},{"operation_name":"upsert","last_dispatch_duration_us":1668719,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1668719,"total_server_duration_us":39109,"encode_duration_us":30516,"operation_id":"0x7401","timeout_ms":10000,"last_server_duration_us":39109,"total_duration_us":1677878},{"operation_name":"upsert","last_dispatch_duration_us":1647544,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1647544,"total_server_duration_us":74096,"encode_duration_us":34396,"operation_id":"0x7069","timeout_ms":10000,"last_server_duration_us":74096,"total_duration_us":1652238},{"operation_name":"upsert","last_dispatch_duration_us":1607446,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1607446,"total_server_duration_us":1588521,"encode_duration_us":29135,"operation_id":"0x73ff","timeout_ms":10000,"last_server_duration_us":1588521,"total_duration_us":1613104},{"operation_name":"upsert","last_dispatch_duration_us":1127390,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1127390,"total_server_duration_us":76591,"encode_duration_us":26645,"operation_id":"0x710a","timeout_ms":10000,"last_server_duration_us":76591,"total_duration_us":1133514},{"operation_name":"upsert","last_dispatch_duration_us":1023637,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1023637,"total_server_duration_us":999578,"encode_duration_us":23663,"operation_id":"0x7109","timeout_ms":10000,"last_server_duration_us":999578,"total_duration_us":1031637}],"total_count":12}} INFO [2022-03-13 17:12:46,114] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,115] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,130] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:46,131] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:12:46,140] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,140] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,140] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,140] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,140] jepsen worker 75 - jepsen.util 435 :invoke :write [25 37] INFO [2022-03-13 17:12:46,141] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:12:46,142] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:46,143] jepsen worker 69 - jepsen.util 609 :invoke :write [23 3] INFO [2022-03-13 17:12:46,157] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:46,157] jepsen worker nemesis - couchbase.util Rebalance check count: 276 INFO [2022-03-13 17:12:46,157] jepsen worker nemesis - couchbase.util Rebalance stuck count: 180 INFO [2022-03-13 17:12:46,189] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,190] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:12:46,197] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,208] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:12:46,210] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,211] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:12:46,213] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,214] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,216] jepsen worker 36 - jepsen.util 2106 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,217] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 25] INFO [2022-03-13 17:12:46,250] jepsen worker 73 - jepsen.util 2233 :info :write [24 35] :RequestTimeoutException INFO [2022-03-13 17:12:46,250] jepsen worker 73 - jepsen.util 2323 :invoke :write [24 22] INFO [2022-03-13 17:12:46,260] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,261] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:12:46,292] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,292] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:12:46,302] jepsen worker 37 - jepsen.util 2017 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,303] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:46,327] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,328] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:46,329] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,329] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:12:46,351] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,352] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,381] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:46,382] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:12:46,386] jepsen worker 10 - jepsen.util 2170 :info :write [3 26] :RequestTimeoutException INFO [2022-03-13 17:12:46,386] jepsen worker 10 - jepsen.util 2260 :invoke :write [3 1] INFO [2022-03-13 17:12:46,389] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,390] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:12:46,399] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,399] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,426] jepsen worker 11 - jepsen.util 2171 :info :write [3 25] :RequestTimeoutException INFO [2022-03-13 17:12:46,427] jepsen worker 11 - jepsen.util 2261 :invoke :write [3 41] INFO [2022-03-13 17:12:46,456] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,456] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:12:46,465] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,466] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:46,482] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,483] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:12:46,502] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,503] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,521] jepsen worker 36 - jepsen.util 2106 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,521] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,521] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:12:46,522] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 8] INFO [2022-03-13 17:12:46,527] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,527] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,542] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,543] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:12:46,546] jepsen worker 64 - jepsen.util 2134 :info :write [21 10] :RequestTimeoutException INFO [2022-03-13 17:12:46,547] jepsen worker 64 - jepsen.util 2224 :invoke :write [21 43] INFO [2022-03-13 17:12:46,562] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,563] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,568] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,569] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,607] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,607] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:46,620] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,620] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,621] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:12:46,621] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:12:46,631] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,631] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,631] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,632] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:12:46,650] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:46,650] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,654] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,655] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,702] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,702] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:12:46,704] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,705] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:46,706] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,707] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:12:46,729] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:46,730] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,732] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,732] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,755] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,756] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 14] INFO [2022-03-13 17:12:46,760] jepsen worker 58 - jepsen.util 2038 :info :write [19 42] :RequestTimeoutException INFO [2022-03-13 17:12:46,760] jepsen worker 58 - jepsen.util 2128 :invoke :write [19 36] INFO [2022-03-13 17:12:46,766] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,766] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,772] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,773] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:12:46,799] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:46,799] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:12:46,812] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,812] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:46,814] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,815] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:12:46,839] jepsen worker 50 - jepsen.util 2210 :info :write [16 36] :RequestTimeoutException INFO [2022-03-13 17:12:46,839] jepsen worker 50 - jepsen.util 2300 :invoke :write [16 6] INFO [2022-03-13 17:12:46,840] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,840] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:46,871] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,872] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:12:46,877] jepsen worker 88 - jepsen.util 1978 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,877] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,889] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:46,891] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,909] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,910] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,935] jepsen worker 36 - jepsen.util 2106 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,937] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:46,940] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,941] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:12:46,942] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,945] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:12:46,959] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:46,960] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:46,966] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:46,967] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 16] INFO [2022-03-13 17:12:46,983] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,984] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:46,984] jepsen worker 76 - jepsen.util 706 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,984] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:46,989] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,989] jepsen worker 52 - jepsen.util 592 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,989] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,990] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:46,992] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,992] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:12:46,992] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:46,993] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:46,998] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:46,998] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:12:46,999] jepsen worker 17 - jepsen.util 2177 :info :write [5 2] :RequestTimeoutException INFO [2022-03-13 17:12:47,000] jepsen worker 17 - jepsen.util 2267 :invoke :read [5 nil] INFO [2022-03-13 17:12:47,010] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:47,011] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,034] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,035] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:12:47,048] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,048] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,048] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,049] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:12:47,055] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,055] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,055] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,056] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,056] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,057] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,063] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,063] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,063] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,064] jepsen worker 21 - jepsen.util 741 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,065] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 6] INFO [2022-03-13 17:12:47,066] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:12:47,067] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:47,067] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,069] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,069] jepsen worker 26 - jepsen.util 476 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,070] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,070] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,070] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,070] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:47,071] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,071] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:47,071] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,072] jepsen worker 7 - jepsen.util 637 :invoke :write [2 33] INFO [2022-03-13 17:12:47,073] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,073] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:12:47,076] jepsen worker 26 - jepsen.util 476 :invoke :write [8 47] INFO [2022-03-13 17:12:47,077] jepsen worker 6 - jepsen.util 636 :invoke :write [2 23] INFO [2022-03-13 17:12:47,078] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,078] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:47,080] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,080] jepsen worker 88 - jepsen.util 1978 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,080] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,081] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,082] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,083] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:12:47,103] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:47,104] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:12:47,151] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,152] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,159] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,160] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:12:47,163] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:47,163] jepsen worker nemesis - couchbase.util Rebalance check count: 277 INFO [2022-03-13 17:12:47,163] jepsen worker nemesis - couchbase.util Rebalance stuck count: 181 INFO [2022-03-13 17:12:47,170] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,171] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:12:47,175] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,176] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:12:47,177] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:47,178] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,179] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,179] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:12:47,206] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:47,206] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:47,223] jepsen worker 60 - jepsen.util 600 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,224] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:47,244] jepsen worker 68 - jepsen.util 2228 :info :write [22 31] :SyncWriteAmbiguous INFO [2022-03-13 17:12:47,245] jepsen worker 68 - jepsen.util 2318 :invoke :write [22 49] INFO [2022-03-13 17:12:47,248] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,248] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,249] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,250] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:12:47,257] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,258] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:12:47,263] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,264] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,265] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,266] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 9] INFO [2022-03-13 17:12:47,279] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:47,280] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,282] jepsen worker 8 - jepsen.util 728 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,283] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:47,290] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,291] jepsen worker 80 - jepsen.util 710 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,291] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,291] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,291] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,292] jepsen worker 70 - jepsen.util 700 :invoke :write [23 0] INFO [2022-03-13 17:12:47,293] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:47,293] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,294] jepsen worker 80 - jepsen.util 710 :invoke :write [26 38] INFO [2022-03-13 17:12:47,294] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:47,295] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,296] jepsen worker 79 - jepsen.util 979 :invoke :write [26 22] INFO [2022-03-13 17:12:47,300] jepsen worker 24 - jepsen.util 564 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,300] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:47,302] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,303] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:12:47,304] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,305] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:47,306] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,307] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:12:47,314] jepsen worker 37 - jepsen.util 2017 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,315] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,354] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,354] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,359] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,360] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:12:47,372] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,373] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:47,393] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:47,393] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,394] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,395] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 29] INFO [2022-03-13 17:12:47,397] jepsen worker 34 - jepsen.util 754 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,398] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] WARN [2022-03-13 17:12:47,410] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10808126,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10808126,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9037366,"encode_duration_us":18932,"operation_id":"0x6b86","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10000935},{"operation_name":"upsert","last_dispatch_duration_us":9163197,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18467046,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9052263,"encode_duration_us":16912,"operation_id":"0x6c29","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10003541}],"total_count":2}} INFO [2022-03-13 17:12:47,411] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:47,412] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:12:47,422] jepsen worker 1 - jepsen.util 1981 :ok :read [0 41] INFO [2022-03-13 17:12:47,423] jepsen worker 1 - jepsen.util 1981 :invoke :write [0 22] INFO [2022-03-13 17:12:47,424] jepsen worker 75 - jepsen.util 435 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,425] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:47,448] jepsen worker 23 - jepsen.util 563 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,448] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,448] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,449] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:12:47,450] jepsen worker 23 - jepsen.util 563 :invoke :write [7 29] INFO [2022-03-13 17:12:47,450] jepsen worker 77 - jepsen.util 707 :invoke :write [25 49] INFO [2022-03-13 17:12:47,458] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,458] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,459] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:12:47,460] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:47,465] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,466] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:12:47,468] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,469] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,475] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,476] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:47,478] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,478] jepsen worker 25 - jepsen.util 565 :invoke :write [8 3] INFO [2022-03-13 17:12:47,497] jepsen worker 3 - jepsen.util 1893 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,498] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,505] jepsen worker 62 - jepsen.util 512 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,505] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,506] jepsen worker 62 - jepsen.util 512 :invoke :write [20 47] INFO [2022-03-13 17:12:47,506] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:47,514] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:47,515] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:12:47,528] jepsen worker 69 - jepsen.util 609 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,528] jepsen worker 69 - jepsen.util 609 :invoke :write [23 17] INFO [2022-03-13 17:12:47,535] jepsen worker 13 - jepsen.util 643 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,535] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,535] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,536] jepsen worker 13 - jepsen.util 643 :invoke :write [4 46] INFO [2022-03-13 17:12:47,536] jepsen worker 78 - jepsen.util 528 :invoke :write [26 35] INFO [2022-03-13 17:12:47,537] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:47,543] jepsen worker 88 - jepsen.util 1978 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,544] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 38] INFO [2022-03-13 17:12:47,560] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,560] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,560] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,561] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,562] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:12:47,562] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 34] INFO [2022-03-13 17:12:47,564] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,564] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:12:47,567] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,568] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,569] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,570] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,572] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,574] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,578] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,579] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 35] INFO [2022-03-13 17:12:47,595] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,595] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,601] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,601] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:47,604] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,604] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] WARN [2022-03-13 17:12:47,609] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17714252,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17714252,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9035244,"encode_duration_us":21129,"operation_id":"0x6c6a","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10001610},{"operation_name":"upsert","last_dispatch_duration_us":17860827,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18046050,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9071351,"encode_duration_us":20113,"operation_id":"0x643d","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10008196},{"operation_name":"upsert","last_dispatch_duration_us":9035822,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10426159,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9636178,"encode_duration_us":27234,"operation_id":"0x674b","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10003949}],"total_count":3}} INFO [2022-03-13 17:12:47,616] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,617] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:47,618] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,619] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:12:47,634] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,634] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,634] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:12:47,635] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:47,638] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,638] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:12:47,664] jepsen worker 44 - jepsen.util 584 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,664] jepsen worker 44 - jepsen.util 584 :invoke :write [14 26] INFO [2022-03-13 17:12:47,679] jepsen worker 25 - jepsen.util 565 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,680] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:47,691] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,692] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,692] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,692] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,692] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,693] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,694] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:12:47,694] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,695] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:12:47,695] jepsen worker 70 - jepsen.util 700 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,696] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:47,699] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] WARN [2022-03-13 17:12:47,710] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":22316826,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":22316826,"total_server_duration_us":38,"operation_id":"0x691f","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":22317225},{"operation_name":"get","last_dispatch_duration_us":16894330,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16894330,"total_server_duration_us":604587,"operation_id":"0x6c58","timeout_ms":30000,"last_server_duration_us":604587,"total_duration_us":16894610},{"operation_name":"get","last_dispatch_duration_us":16117430,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16117430,"total_server_duration_us":619786,"operation_id":"0x6cc3","timeout_ms":30000,"last_server_duration_us":619786,"total_duration_us":16117797},{"operation_name":"get","last_dispatch_duration_us":13750608,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13750608,"total_server_duration_us":43,"operation_id":"0x6de4","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":13750834},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":17914,"operation_id":"0x6d5e","timeout_ms":10000,"total_duration_us":10010369},{"operation_name":"upsert","last_dispatch_duration_us":8654959,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8654959,"total_server_duration_us":24239,"encode_duration_us":20872,"operation_id":"0x7060","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10009771},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":18074,"operation_id":"0x6fcb","timeout_ms":10000,"total_duration_us":10009435},{"operation_name":"upsert","last_dispatch_duration_us":17224704,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17224704,"total_server_duration_us":43070,"encode_duration_us":16729,"operation_id":"0x6be7","timeout_ms":10000,"last_server_duration_us":43070,"total_duration_us":10006066},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":17080,"operation_id":"0x6d17","timeout_ms":10000,"total_duration_us":10005590},{"operation_name":"upsert","last_dispatch_duration_us":1916515,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1916515,"total_server_duration_us":76033,"encode_duration_us":21083,"operation_id":"0x73e6","timeout_ms":10000,"last_server_duration_us":76033,"total_duration_us":1921355}],"total_count":18}} WARN [2022-03-13 17:12:47,710] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18429701,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18792789,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9066504,"encode_duration_us":18933,"operation_id":"0x6b13","timeout_ms":10000,"last_server_duration_us":9021452,"total_duration_us":10002203},{"operation_name":"upsert","last_dispatch_duration_us":18455264,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18685080,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9052575,"encode_duration_us":20739,"operation_id":"0x6b1f","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10005962}],"total_count":2}} WARN [2022-03-13 17:12:47,711] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7874680,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":16040721,"total_server_duration_us":261564,"encode_duration_us":18291,"operation_id":"0x6c11","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10009966},{"operation_name":"upsert","last_dispatch_duration_us":8547222,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8599230,"total_server_duration_us":43362,"encode_duration_us":23150,"operation_id":"0x7010","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10009762},{"operation_name":"upsert","last_dispatch_duration_us":8331690,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8365625,"total_server_duration_us":41363,"encode_duration_us":16719,"operation_id":"0x6e9a","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10007994},{"operation_name":"upsert","last_dispatch_duration_us":32260,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":719174,"total_server_duration_us":238880,"encode_duration_us":20597,"operation_id":"0x6fb1","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10006032},{"operation_name":"upsert","last_dispatch_duration_us":50592,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":160935,"total_server_duration_us":88001,"encode_duration_us":18077,"operation_id":"0x6ff8","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10003993},{"operation_name":"upsert","last_dispatch_duration_us":7785712,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8496622,"total_server_duration_us":272524,"encode_duration_us":18329,"operation_id":"0x6f93","timeout_ms":10000,"last_server_duration_us":37036,"total_duration_us":10001587},{"operation_name":"upsert","last_dispatch_duration_us":9140222,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9140222,"total_server_duration_us":9057542,"encode_duration_us":56868,"operation_id":"0x6fdd","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":9159903},{"operation_name":"get","last_dispatch_duration_us":7349982,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7349982,"total_server_duration_us":53494,"operation_id":"0x70d1","timeout_ms":30000,"last_server_duration_us":53494,"total_duration_us":7350279},{"operation_name":"get","last_dispatch_duration_us":2094843,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2094843,"total_server_duration_us":49,"operation_id":"0x73aa","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":2123203},{"operation_name":"upsert","last_dispatch_duration_us":2094116,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2094116,"total_server_duration_us":1565789,"encode_duration_us":27044,"operation_id":"0x73b0","timeout_ms":10000,"last_server_duration_us":1565789,"total_duration_us":2099822}],"total_count":54}} WARN [2022-03-13 17:12:47,711] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":20523,"operation_id":"0x6d5a","timeout_ms":10000,"total_duration_us":10006269},{"operation_name":"upsert","last_dispatch_duration_us":8831297,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8912349,"total_server_duration_us":67354,"encode_duration_us":19754,"operation_id":"0x704a","timeout_ms":10000,"last_server_duration_us":46513,"total_duration_us":10005508},{"operation_name":"upsert","last_dispatch_duration_us":8790224,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18301405,"total_server_duration_us":197159,"encode_duration_us":18841,"operation_id":"0x6c60","timeout_ms":10000,"last_server_duration_us":35412,"total_duration_us":10004400},{"operation_name":"upsert","last_dispatch_duration_us":9163197,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18467046,"total_server_duration_us":9052263,"encode_duration_us":16912,"operation_id":"0x6c29","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10003541},{"operation_name":"get","last_dispatch_duration_us":9164061,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9164061,"total_server_duration_us":32,"operation_id":"0x6c5d","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9166639},{"operation_name":"upsert","last_dispatch_duration_us":9094672,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9094672,"total_server_duration_us":9034183,"encode_duration_us":18688,"operation_id":"0x70cd","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":9098637},{"operation_name":"get","last_dispatch_duration_us":30667,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":30667,"total_server_duration_us":7645,"operation_id":"0x6ca8","timeout_ms":30000,"last_server_duration_us":7645,"total_duration_us":8765836},{"operation_name":"get","last_dispatch_duration_us":8616328,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8616328,"total_server_duration_us":8069,"operation_id":"0x7116","timeout_ms":30000,"last_server_duration_us":8069,"total_duration_us":8618405},{"operation_name":"upsert","last_dispatch_duration_us":2521689,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2521689,"total_server_duration_us":20362,"encode_duration_us":26344,"operation_id":"0x73a9","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":2551253},{"operation_name":"upsert","last_dispatch_duration_us":2490184,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2490184,"total_server_duration_us":66168,"encode_duration_us":24262,"operation_id":"0x73a8","timeout_ms":10000,"last_server_duration_us":66168,"total_duration_us":2494415}],"total_count":57}} INFO [2022-03-13 17:12:47,713] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,713] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,713] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:47,714] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] WARN [2022-03-13 17:12:47,720] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":35390,"operation_id":"0x6ff1","timeout_ms":10000,"total_duration_us":10009513},{"operation_name":"upsert","last_dispatch_duration_us":8081178,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8081178,"total_server_duration_us":23729,"encode_duration_us":17188,"operation_id":"0x6d77","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10004234},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":22469,"operation_id":"0x6c91","timeout_ms":10000,"total_duration_us":10004215},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":16969,"operation_id":"0x6f4f","timeout_ms":10000,"total_duration_us":10000866},{"operation_name":"upsert","last_dispatch_duration_us":9040549,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9358272,"total_server_duration_us":9172807,"encode_duration_us":18291,"operation_id":"0x6d07","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":9442595},{"operation_name":"upsert","last_dispatch_duration_us":9090136,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9273571,"total_server_duration_us":9124044,"encode_duration_us":27868,"operation_id":"0x6d0d","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9304155},{"operation_name":"get","last_dispatch_duration_us":9001854,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9001854,"total_server_duration_us":9628,"operation_id":"0x6d0f","timeout_ms":30000,"last_server_duration_us":9628,"total_duration_us":9003858},{"operation_name":"get","last_dispatch_duration_us":8714986,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8714986,"total_server_duration_us":6384,"operation_id":"0x6d16","timeout_ms":30000,"last_server_duration_us":6384,"total_duration_us":8717282},{"operation_name":"upsert","last_dispatch_duration_us":2415886,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2415886,"total_server_duration_us":117451,"encode_duration_us":29311,"operation_id":"0x73d4","timeout_ms":10000,"last_server_duration_us":117451,"total_duration_us":2419517},{"operation_name":"upsert","last_dispatch_duration_us":2231351,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2231351,"total_server_duration_us":88098,"encode_duration_us":27930,"operation_id":"0x73bb","timeout_ms":10000,"last_server_duration_us":88098,"total_duration_us":2235119}],"total_count":52}} INFO [2022-03-13 17:12:47,722] jepsen worker 43 - jepsen.util 493 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,723] jepsen worker 43 - jepsen.util 493 :invoke :write [14 29] INFO [2022-03-13 17:12:47,724] jepsen worker 88 - jepsen.util 1978 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,725] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 6] INFO [2022-03-13 17:12:47,725] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,725] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,727] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:47,727] jepsen worker 14 - jepsen.util 284 :invoke :write [4 27] INFO [2022-03-13 17:12:47,757] jepsen worker 37 - jepsen.util 2017 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,758] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,759] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,759] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,760] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:12:47,771] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:47,773] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,774] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:12:47,783] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:47,783] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,796] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,797] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,805] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,806] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] WARN [2022-03-13 17:12:47,810] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16793988,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16793988,"total_server_duration_us":32,"operation_id":"0x6935","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":16795882},{"operation_name":"get","last_dispatch_duration_us":15523574,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":15523574,"total_server_duration_us":500,"operation_id":"0x6a0d","timeout_ms":30000,"last_server_duration_us":500,"total_duration_us":15523862},{"operation_name":"get","last_dispatch_duration_us":11458666,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11458666,"total_server_duration_us":32,"operation_id":"0x6c09","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11461462},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17847,"operation_id":"0x6d62","timeout_ms":10000,"total_duration_us":10004609},{"operation_name":"upsert","last_dispatch_duration_us":17714252,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17714252,"total_server_duration_us":9035244,"encode_duration_us":21129,"operation_id":"0x6c6a","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10001610},{"operation_name":"get","last_dispatch_duration_us":9441221,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9441221,"total_server_duration_us":126,"operation_id":"0x6d10","timeout_ms":30000,"last_server_duration_us":126,"total_duration_us":9443471},{"operation_name":"get","last_dispatch_duration_us":1673622,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1673622,"total_server_duration_us":126,"operation_id":"0x701b","timeout_ms":30000,"last_server_duration_us":126,"total_duration_us":5309706},{"operation_name":"upsert","last_dispatch_duration_us":4945850,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4945850,"total_server_duration_us":79687,"encode_duration_us":20330,"operation_id":"0x6e6b","timeout_ms":10000,"last_server_duration_us":79687,"total_duration_us":4954916},{"operation_name":"upsert","last_dispatch_duration_us":1829912,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1829912,"total_server_duration_us":87802,"encode_duration_us":18278,"operation_id":"0x70aa","timeout_ms":10000,"last_server_duration_us":87802,"total_duration_us":4448712},{"operation_name":"upsert","last_dispatch_duration_us":2262953,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2262953,"total_server_duration_us":56401,"encode_duration_us":19190,"operation_id":"0x6fab","timeout_ms":10000,"last_server_duration_us":56401,"total_duration_us":3875345}],"total_count":55}} WARN [2022-03-13 17:12:47,810] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16675654,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17718501,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9225179,"encode_duration_us":34552,"operation_id":"0x6823","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10001691},{"operation_name":"upsert","last_dispatch_duration_us":23595711,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":23595711,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9046921,"encode_duration_us":18597,"operation_id":"0x6858","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10003792}],"total_count":2}} INFO [2022-03-13 17:12:47,814] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,815] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] WARN [2022-03-13 17:12:47,821] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17284560,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17913230,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9157096,"encode_duration_us":28185,"operation_id":"0x682e","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10009771}],"total_count":1}} INFO [2022-03-13 17:12:47,836] jepsen worker 79 - jepsen.util 979 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,836] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,837] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:12:47,837] jepsen worker 79 - jepsen.util 979 :invoke :write [26 23] INFO [2022-03-13 17:12:47,851] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,852] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:12:47,876] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,876] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,885] jepsen worker 51 - jepsen.util 591 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,885] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:47,889] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,889] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,891] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:12:47,892] jepsen worker 76 - jepsen.util 706 :invoke :write [25 12] INFO [2022-03-13 17:12:47,897] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,898] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:47,899] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:12:47,899] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:12:47,901] jepsen worker 88 - jepsen.util 1978 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,902] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:47,927] jepsen worker 80 - jepsen.util 710 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,928] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,928] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,928] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,929] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:47,930] jepsen worker 52 - jepsen.util 592 :invoke :write [17 22] INFO [2022-03-13 17:12:47,930] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:47,931] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:47,934] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,935] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:12:47,939] jepsen worker 69 - jepsen.util 609 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,939] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,940] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:47,943] jepsen worker 21 - jepsen.util 741 :invoke :write [7 20] INFO [2022-03-13 17:12:47,950] jepsen worker 85 - jepsen.util 2065 :ok :read [28 44] INFO [2022-03-13 17:12:47,951] jepsen worker 85 - jepsen.util 2065 :invoke :write [28 13] INFO [2022-03-13 17:12:47,959] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,959] jepsen worker 36 - jepsen.util 2106 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,959] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:12:47,962] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:47,964] jepsen worker 26 - jepsen.util 476 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,965] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:47,969] jepsen worker 49 - jepsen.util 1939 :ok :read [16 :nil] INFO [2022-03-13 17:12:47,970] jepsen worker 49 - jepsen.util 1939 :invoke :read [16 nil] INFO [2022-03-13 17:12:47,982] jepsen worker 57 - jepsen.util 2307 :ok :read [19 20] INFO [2022-03-13 17:12:47,983] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:47,983] jepsen worker 57 - jepsen.util 2307 :invoke :read [19 nil] INFO [2022-03-13 17:12:47,984] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:12:47,986] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,987] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,987] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,988] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:12:47,989] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,989] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:47,993] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,993] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:47,994] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:12:47,995] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:12:47,997] jepsen worker 84 - jepsen.util 2424 :ok :read [28 44] INFO [2022-03-13 17:12:48,007] jepsen worker 84 - jepsen.util 2424 :invoke :write [28 46] INFO [2022-03-13 17:12:48,029] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,029] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,030] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,031] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,032] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:12:48,033] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:12:48,041] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,042] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:12:48,047] jepsen worker 13 - jepsen.util 643 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,048] jepsen worker 13 - jepsen.util 643 :invoke :write [4 38] INFO [2022-03-13 17:12:48,049] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,050] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:12:48,054] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,054] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,053] jepsen worker 0 - jepsen.util 2250 :ok :read [0 41] INFO [2022-03-13 17:12:48,055] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 25] INFO [2022-03-13 17:12:48,055] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,056] jepsen worker 0 - jepsen.util 2250 :invoke :read [0 nil] INFO [2022-03-13 17:12:48,067] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,067] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:48,088] jepsen worker 40 - jepsen.util 2200 :ok :read [13 39] INFO [2022-03-13 17:12:48,089] jepsen worker 40 - jepsen.util 2200 :invoke :write [13 36] INFO [2022-03-13 17:12:48,090] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,091] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,101] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,103] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:12:48,108] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,108] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:12:48,113] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,114] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:12:48,138] jepsen worker 82 - jepsen.util 2062 :info :write [27 17] :RequestTimeoutException INFO [2022-03-13 17:12:48,138] jepsen worker 82 - jepsen.util 2152 :invoke :read [27 nil] INFO [2022-03-13 17:12:48,142] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,142] jepsen worker 6 - jepsen.util 636 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,142] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,142] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:48,143] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,143] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:48,144] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:12:48,145] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:12:48,153] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,153] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,165] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,166] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:12:48,169] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,170] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:48,170] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,170] jepsen worker nemesis - couchbase.util Rebalance check count: 278 INFO [2022-03-13 17:12:48,170] jepsen worker nemesis - couchbase.util Rebalance stuck count: 182 INFO [2022-03-13 17:12:48,179] jepsen worker 36 - jepsen.util 2106 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,180] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 25] INFO [2022-03-13 17:12:48,199] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,200] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,213] jepsen worker 23 - jepsen.util 563 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,213] jepsen worker 23 - jepsen.util 563 :invoke :write [7 1] INFO [2022-03-13 17:12:48,221] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,222] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:48,223] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:48,223] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,229] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,229] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:48,231] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,232] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,245] jepsen worker 78 - jepsen.util 528 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,246] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,247] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:12:48,247] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,247] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,247] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:12:48,247] jepsen worker 7 - jepsen.util 637 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,248] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,248] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:48,248] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:48,249] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:48,249] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,255] jepsen worker 79 - jepsen.util 979 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,256] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,257] jepsen worker 69 - jepsen.util 609 :invoke :write [23 48] INFO [2022-03-13 17:12:48,258] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:48,263] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:48,264] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,298] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,300] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:12:48,310] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,310] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,311] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,311] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,312] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,313] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:12:48,317] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,318] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:12:48,319] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,320] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:12:48,324] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,326] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:12:48,345] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,346] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,354] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,355] jepsen worker 21 - jepsen.util 741 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,356] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:12:48,357] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:48,364] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:48,364] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:12:48,368] jepsen worker 77 - jepsen.util 707 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,369] jepsen worker 77 - jepsen.util 707 :invoke :write [25 16] INFO [2022-03-13 17:12:48,379] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,380] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,409] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,409] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,414] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:48,415] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 32] INFO [2022-03-13 17:12:48,433] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,434] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:12:48,448] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:48,448] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,448] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 2] INFO [2022-03-13 17:12:48,449] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,456] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,456] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,456] jepsen worker 36 - jepsen.util 2106 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,457] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,458] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,459] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 1] INFO [2022-03-13 17:12:48,462] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,462] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,463] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,464] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,464] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:12:48,465] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,465] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,466] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:48,466] jepsen worker 52 - jepsen.util 592 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,468] jepsen worker 52 - jepsen.util 592 :invoke :write [17 16] INFO [2022-03-13 17:12:48,472] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,472] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,474] jepsen worker 61 - jepsen.util 601 :invoke :write [20 29] INFO [2022-03-13 17:12:48,475] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,476] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:48,476] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,481] jepsen worker 63 - jepsen.util 2133 :ok :read [21 :nil] INFO [2022-03-13 17:12:48,481] jepsen worker 63 - jepsen.util 2133 :invoke :read [21 nil] INFO [2022-03-13 17:12:48,487] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,487] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,488] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,488] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,500] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,500] jepsen worker 46 - jepsen.util 2296 :ok :read [15 48] INFO [2022-03-13 17:12:48,501] jepsen worker 46 - jepsen.util 2296 :invoke :read [15 nil] INFO [2022-03-13 17:12:48,502] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:48,526] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,526] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,527] jepsen worker 79 - jepsen.util 979 :invoke :write [26 13] INFO [2022-03-13 17:12:48,527] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:48,529] jepsen worker 88 - jepsen.util 1978 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,529] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,530] jepsen worker 15 - jepsen.util 2175 :info :write [5 2] :RequestTimeoutException INFO [2022-03-13 17:12:48,531] jepsen worker 15 - jepsen.util 2265 :invoke :write [5 19] INFO [2022-03-13 17:12:48,545] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,546] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,549] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,549] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,549] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,550] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:12:48,552] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,553] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:12:48,585] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:48,586] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:12:48,588] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:48,588] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,608] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,608] jepsen worker 69 - jepsen.util 609 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,608] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,608] jepsen worker 69 - jepsen.util 609 :invoke :write [23 21] INFO [2022-03-13 17:12:48,609] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:48,609] jepsen worker 36 - jepsen.util 2106 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,609] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,610] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,610] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,610] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,660] jepsen worker 13 - jepsen.util 643 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,660] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,660] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,660] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,660] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,660] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,660] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:12:48,661] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:12:48,662] jepsen worker 78 - jepsen.util 528 :invoke :write [26 25] INFO [2022-03-13 17:12:48,662] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,663] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:12:48,663] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:48,667] jepsen worker 44 - jepsen.util 584 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,667] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,668] jepsen worker 44 - jepsen.util 584 :invoke :write [14 27] INFO [2022-03-13 17:12:48,668] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:48,670] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,671] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,674] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,674] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,674] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,675] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,682] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,682] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,690] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,690] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,691] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,691] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,692] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,692] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,692] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,693] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,695] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,695] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,696] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,696] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,696] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,697] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,697] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 9] INFO [2022-03-13 17:12:48,698] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 32] INFO [2022-03-13 17:12:48,702] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:12:48,703] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,704] jepsen worker 87 - jepsen.util 2607 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,704] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,705] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,706] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,706] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,708] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,710] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,710] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,710] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,710] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,711] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,712] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:48,713] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,714] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:12:48,716] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,717] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,718] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,720] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,722] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,723] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:12:48,734] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:48,734] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,735] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,736] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:12:48,736] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,740] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:48,741] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,742] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:12:48,746] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,747] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:12:48,758] jepsen worker 56 - jepsen.util 2306 :ok :read [18 21] INFO [2022-03-13 17:12:48,760] jepsen worker 56 - jepsen.util 2306 :invoke :write [18 33] INFO [2022-03-13 17:12:48,777] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,777] jepsen worker 54 - jepsen.util 1854 :ok :read [18 21] INFO [2022-03-13 17:12:48,777] jepsen worker 37 - jepsen.util 2017 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,778] jepsen worker 53 - jepsen.util 503 :invoke :write [17 5] INFO [2022-03-13 17:12:48,778] jepsen worker 62 - jepsen.util 512 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,779] jepsen worker 54 - jepsen.util 1854 :invoke :write [18 40] INFO [2022-03-13 17:12:48,779] jepsen worker 62 - jepsen.util 512 :invoke :write [20 49] INFO [2022-03-13 17:12:48,780] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:12:48,784] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,784] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,785] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,785] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,789] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:48,790] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:12:48,792] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,792] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,792] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:12:48,793] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,794] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,795] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:48,797] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,797] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:12:48,807] jepsen worker 65 - jepsen.util 2585 :ok :read [21 :nil] INFO [2022-03-13 17:12:48,808] jepsen worker 65 - jepsen.util 2585 :invoke :write [21 0] INFO [2022-03-13 17:12:48,824] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,825] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:48,834] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:48,840] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,840] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,841] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,844] jepsen worker 18 - jepsen.util 1998 :ok :read [6 16] INFO [2022-03-13 17:12:48,845] jepsen worker 18 - jepsen.util 1998 :invoke :read [6 nil] INFO [2022-03-13 17:12:48,845] jepsen worker 72 - jepsen.util 2142 :ok :read [24 24] INFO [2022-03-13 17:12:48,846] jepsen worker 72 - jepsen.util 2142 :invoke :read [24 nil] INFO [2022-03-13 17:12:48,859] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,860] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:12:48,896] jepsen worker 43 - jepsen.util 493 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,896] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:48,902] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,903] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,903] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,904] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:12:48,904] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,905] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,909] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,910] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,914] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,918] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:12:48,930] jepsen worker 79 - jepsen.util 979 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,930] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,934] jepsen worker 79 - jepsen.util 979 :invoke :write [26 0] INFO [2022-03-13 17:12:48,935] jepsen worker 21 - jepsen.util 741 :invoke :write [7 41] INFO [2022-03-13 17:12:48,952] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:48,952] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:48,953] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,953] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:12:48,977] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,977] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:48,979] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,979] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,980] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:48,980] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:48,984] jepsen worker 36 - jepsen.util 2106 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,985] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:48,992] jepsen worker 76 - jepsen.util 706 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,993] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:48,993] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:48,994] jepsen worker 12 - jepsen.util 732 :invoke :write [4 22] INFO [2022-03-13 17:12:49,008] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:49,008] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,011] jepsen worker 69 - jepsen.util 609 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,011] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,011] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,011] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,011] jepsen worker 69 - jepsen.util 609 :invoke :write [23 4] INFO [2022-03-13 17:12:49,012] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,013] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:12:49,014] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:12:49,050] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,050] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,050] jepsen worker 18 - jepsen.util 1998 :ok :read [6 16] INFO [2022-03-13 17:12:49,051] jepsen worker 18 - jepsen.util 1998 :invoke :read [6 nil] INFO [2022-03-13 17:12:49,052] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:12:49,053] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:12:49,061] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,061] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,063] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:12:49,064] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,071] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,071] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,071] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:12:49,071] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,073] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,073] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:12:49,076] jepsen worker 16 - jepsen.util 1996 :info :write [5 17] :RequestTimeoutException INFO [2022-03-13 17:12:49,077] jepsen worker 16 - jepsen.util 2086 :invoke :read [5 nil] INFO [2022-03-13 17:12:49,090] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:49,093] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,095] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,095] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,099] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,100] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:12:49,118] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,119] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,127] jepsen worker 86 - jepsen.util 1976 :info :write [28 9] :RequestTimeoutException INFO [2022-03-13 17:12:49,128] jepsen worker 86 - jepsen.util 2066 :invoke :read [28 nil] INFO [2022-03-13 17:12:49,130] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:49,130] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:12:49,164] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,164] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,164] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,164] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,165] jepsen worker 14 - jepsen.util 284 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,165] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,165] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:49,166] jepsen worker 7 - jepsen.util 637 :invoke :write [2 31] INFO [2022-03-13 17:12:49,167] jepsen worker 42 - jepsen.util 402 :invoke :write [14 14] INFO [2022-03-13 17:12:49,167] jepsen worker 6 - jepsen.util 636 :invoke :write [2 11] INFO [2022-03-13 17:12:49,168] jepsen worker 8 - jepsen.util 728 :invoke :write [2 9] INFO [2022-03-13 17:12:49,170] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:49,185] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,186] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,187] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:49,187] jepsen worker nemesis - couchbase.util Rebalance check count: 279 INFO [2022-03-13 17:12:49,188] jepsen worker nemesis - couchbase.util Rebalance stuck count: 183 INFO [2022-03-13 17:12:49,193] jepsen worker 18 - jepsen.util 1998 :ok :read [6 16] INFO [2022-03-13 17:12:49,194] jepsen worker 18 - jepsen.util 1998 :invoke :read [6 nil] INFO [2022-03-13 17:12:49,208] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,208] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:12:49,213] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,213] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,214] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,214] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 17] INFO [2022-03-13 17:12:49,229] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:49,230] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,243] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,245] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:12:49,253] jepsen worker 59 - jepsen.util 2219 :info :write [19 1] :RequestTimeoutException INFO [2022-03-13 17:12:49,254] jepsen worker 59 - jepsen.util 2309 :invoke :read [19 nil] INFO [2022-03-13 17:12:49,256] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,256] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:49,287] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,288] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,291] jepsen worker 79 - jepsen.util 979 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,293] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:49,297] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:49,298] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,301] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,301] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,301] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,302] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,302] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,303] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:12:49,307] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,307] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,308] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,309] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,310] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,310] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:12:49,315] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:49,316] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:12:49,318] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,319] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:12:49,328] jepsen worker 21 - jepsen.util 741 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,328] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,330] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:49,331] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,345] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,345] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,345] jepsen worker 18 - jepsen.util 1998 :ok :read [6 16] INFO [2022-03-13 17:12:49,347] jepsen worker 18 - jepsen.util 1998 :invoke :write [6 17] INFO [2022-03-13 17:12:49,364] jepsen worker 69 - jepsen.util 609 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,364] jepsen worker 69 - jepsen.util 609 :invoke :write [23 14] INFO [2022-03-13 17:12:49,368] jepsen worker 23 - jepsen.util 563 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,369] jepsen worker 23 - jepsen.util 563 :invoke :write [7 25] INFO [2022-03-13 17:12:49,381] jepsen worker 72 - jepsen.util 2142 :ok :read [24 24] INFO [2022-03-13 17:12:49,382] jepsen worker 72 - jepsen.util 2142 :invoke :write [24 1] INFO [2022-03-13 17:12:49,389] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,390] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,391] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:12:49,392] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:12:49,402] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,403] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,413] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,413] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,416] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,417] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,429] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,429] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:12:49,431] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:49,432] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,433] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,434] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,460] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,460] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,460] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,461] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,462] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,462] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,462] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,463] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:12:49,463] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,464] jepsen worker 61 - jepsen.util 601 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,464] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:12:49,465] jepsen worker 61 - jepsen.util 601 :invoke :write [20 36] INFO [2022-03-13 17:12:49,494] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,494] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,494] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,494] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:12:49,495] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,494] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,496] jepsen worker 21 - jepsen.util 741 :invoke :write [7 32] INFO [2022-03-13 17:12:49,496] jepsen worker 24 - jepsen.util 564 :invoke :write [8 46] INFO [2022-03-13 17:12:49,497] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:49,500] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:49,504] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,505] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,511] jepsen worker 36 - jepsen.util 2106 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,512] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 7] INFO [2022-03-13 17:12:49,515] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,515] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,554] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:49,555] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,565] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:49,565] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 36] INFO [2022-03-13 17:12:49,593] jepsen worker 69 - jepsen.util 609 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,593] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,593] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,593] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,593] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:49,594] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,595] jepsen worker 69 - jepsen.util 609 :invoke :write [23 27] INFO [2022-03-13 17:12:49,595] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:49,596] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,596] jepsen worker 25 - jepsen.util 565 :invoke :write [8 3] INFO [2022-03-13 17:12:49,600] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,601] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,601] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,602] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,603] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,603] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,616] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,616] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,618] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,619] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:12:49,621] jepsen worker 77 - jepsen.util 707 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,622] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,623] jepsen worker 77 - jepsen.util 707 :invoke :write [25 17] INFO [2022-03-13 17:12:49,628] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:49,632] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,632] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,633] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,634] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,635] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,635] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,637] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:12:49,637] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,639] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,640] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,641] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,641] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:12:49,644] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:49,645] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,652] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,652] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,663] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:49,664] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:12:49,692] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:49,692] jepsen worker 25 - jepsen.util 565 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,692] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,693] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:12:49,693] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,694] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:12:49,694] jepsen worker 47 - jepsen.util 2207 :info :write [15 16] :RequestTimeoutException INFO [2022-03-13 17:12:49,695] jepsen worker 47 - jepsen.util 2297 :invoke :read [15 nil] INFO [2022-03-13 17:12:49,719] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,720] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,723] jepsen worker 88 - jepsen.util 1978 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,724] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,739] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,741] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:12:49,743] jepsen worker 24 - jepsen.util 564 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,744] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,758] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,758] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:12:49,761] jepsen worker 53 - jepsen.util 503 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,761] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:49,768] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:49,769] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:12:49,779] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,779] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:49,825] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,825] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,829] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,830] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:12:49,838] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,838] jepsen worker 21 - jepsen.util 741 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,838] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,838] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,838] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:49,839] jepsen worker 71 - jepsen.util 521 :invoke :write [23 40] INFO [2022-03-13 17:12:49,840] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,840] jepsen worker 79 - jepsen.util 979 :invoke :write [26 22] INFO [2022-03-13 17:12:49,846] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,846] jepsen worker 36 - jepsen.util 2106 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,846] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,846] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,847] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,848] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 46] INFO [2022-03-13 17:12:49,856] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,857] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,857] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,858] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,858] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,860] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,865] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,866] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,876] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,876] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,876] jepsen worker 69 - jepsen.util 609 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,876] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,877] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,877] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,877] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,877] jepsen worker 69 - jepsen.util 609 :invoke :write [23 42] INFO [2022-03-13 17:12:49,878] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,879] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:12:49,880] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:12:49,881] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,882] jepsen worker 21 - jepsen.util 741 :invoke :write [7 24] INFO [2022-03-13 17:12:49,882] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:12:49,884] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:49,887] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,888] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,888] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,897] jepsen worker 71 - jepsen.util 521 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,898] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:49,931] jepsen worker 36 - jepsen.util 2106 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,932] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,932] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:49,933] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 45] INFO [2022-03-13 17:12:49,937] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,937] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,938] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:49,938] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,942] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,942] jepsen worker 44 - jepsen.util 584 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,942] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,942] jepsen worker 44 - jepsen.util 584 :invoke :write [14 38] INFO [2022-03-13 17:12:49,943] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:49,945] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:49,946] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,946] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,946] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,947] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:12:49,948] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,949] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,950] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,951] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:12:49,960] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,960] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,960] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,961] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,961] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,961] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:49,977] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:49,978] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,978] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:12:49,979] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,979] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:49,979] jepsen worker 62 - jepsen.util 512 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,979] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,980] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,980] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 7] INFO [2022-03-13 17:12:49,981] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:49,982] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:49,983] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:49,984] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:49,984] jepsen worker 14 - jepsen.util 284 :invoke :write [4 48] INFO [2022-03-13 17:12:49,986] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,987] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:12:49,997] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:49,997] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,997] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:49,998] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,000] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 12] INFO [2022-03-13 17:12:50,000] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:12:50,004] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,005] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 41] INFO [2022-03-13 17:12:50,040] jepsen worker 37 - jepsen.util 2017 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,041] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,041] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:12:50,041] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,051] jepsen worker 79 - jepsen.util 979 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,051] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,052] jepsen worker 79 - jepsen.util 979 :invoke :write [26 4] INFO [2022-03-13 17:12:50,052] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] INFO [2022-03-13 17:12:50,083] jepsen worker 12 - jepsen.util 732 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,083] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:50,092] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,093] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,095] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,096] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,100] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,101] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,102] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,103] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,104] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,105] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,106] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,107] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:12:50,111] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,112] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,135] jepsen worker 3 - jepsen.util 1893 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,136] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:12:50,148] jepsen worker 88 - jepsen.util 1978 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,149] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,153] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,153] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:50,160] jepsen worker 36 - jepsen.util 2106 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,160] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,160] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 4] INFO [2022-03-13 17:12:50,161] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,180] jepsen worker 7 - jepsen.util 637 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,180] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:50,197] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:50,197] jepsen worker nemesis - couchbase.util Rebalance check count: 280 INFO [2022-03-13 17:12:50,197] jepsen worker nemesis - couchbase.util Rebalance stuck count: 184 INFO [2022-03-13 17:12:50,202] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,202] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,207] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:50,207] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,226] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,227] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:50,238] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,239] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:12:50,239] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,239] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,240] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:50,240] jepsen worker 33 - jepsen.util 663 :invoke :write [11 37] INFO [2022-03-13 17:12:50,242] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,243] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,245] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,245] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,272] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:50,273] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,276] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,277] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,278] jepsen worker 23 - jepsen.util 563 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,278] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,279] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:50,279] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:50,318] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,319] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,324] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,325] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:12:50,332] jepsen worker 36 - jepsen.util 2106 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,332] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,332] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,333] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,334] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,334] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,339] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,339] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,339] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,340] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,340] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,341] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 0] INFO [2022-03-13 17:12:50,350] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,350] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,350] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,350] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,351] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:12:50,353] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,354] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,357] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,358] jepsen worker 21 - jepsen.util 741 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,358] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,359] jepsen worker 21 - jepsen.util 741 :invoke :write [7 41] INFO [2022-03-13 17:12:50,359] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,360] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,360] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,360] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:12:50,361] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,361] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,362] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,363] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,364] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,365] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,366] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:12:50,366] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,374] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:12:50,381] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,382] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:12:50,383] jepsen worker 6 - jepsen.util 636 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,383] jepsen worker 6 - jepsen.util 636 :invoke :write [2 34] INFO [2022-03-13 17:12:50,391] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:50,392] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:12:50,397] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,398] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:12:50,428] jepsen worker 28 - jepsen.util 2458 :info :write [9 18] :RequestTimeoutException INFO [2022-03-13 17:12:50,429] jepsen worker 28 - jepsen.util 2548 :invoke :read [9 nil] INFO [2022-03-13 17:12:50,439] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,439] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:50,449] jepsen worker 37 - jepsen.util 2017 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,450] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,468] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,468] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,468] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:12:50,468] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:50,493] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:50,494] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,495] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,496] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,521] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,521] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,525] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,526] jepsen worker 77 - jepsen.util 707 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,526] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,526] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,529] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:12:50,531] jepsen worker 77 - jepsen.util 707 :invoke :write [25 30] INFO [2022-03-13 17:12:50,533] jepsen worker 76 - jepsen.util 706 :invoke :write [25 15] INFO [2022-03-13 17:12:50,535] jepsen worker 60 - jepsen.util 600 :invoke :write [20 37] INFO [2022-03-13 17:12:50,566] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,566] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,566] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:12:50,567] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:12:50,574] jepsen worker 69 - jepsen.util 609 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,575] jepsen worker 69 - jepsen.util 609 :invoke :write [23 5] INFO [2022-03-13 17:12:50,582] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,582] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,604] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,605] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,607] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,607] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:12:50,622] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,622] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,627] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,628] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,629] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,630] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,631] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,631] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,633] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:12:50,634] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 6] INFO [2022-03-13 17:12:50,635] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,635] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:12:50,645] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:50,646] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:12:50,654] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,655] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:50,659] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,659] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:50,659] jepsen worker 8 - jepsen.util 728 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,660] jepsen worker 8 - jepsen.util 728 :invoke :write [2 19] INFO [2022-03-13 17:12:50,690] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,691] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:12:50,693] jepsen worker 79 - jepsen.util 979 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,693] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,694] jepsen worker 79 - jepsen.util 979 :invoke :write [26 14] INFO [2022-03-13 17:12:50,694] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:50,716] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:50,716] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:12:50,721] jepsen worker 3 - jepsen.util 1893 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,722] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 38] INFO [2022-03-13 17:12:50,764] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,765] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:50,767] jepsen worker 33 - jepsen.util 663 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,767] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:12:50,769] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,769] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,769] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:12:50,770] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,771] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,772] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:12:50,773] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,773] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:12:50,827] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,827] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:50,841] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,841] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,842] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:12:50,843] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,846] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,846] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:12:50,851] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,852] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,852] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:50,853] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:12:50,859] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,860] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,860] jepsen worker 44 - jepsen.util 584 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,860] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,861] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,862] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:50,862] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:50,863] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:50,863] jepsen worker 51 - jepsen.util 591 :invoke :write [17 34] INFO [2022-03-13 17:12:50,866] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:12:50,882] jepsen worker 21 - jepsen.util 741 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,883] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:50,925] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,926] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:12:50,933] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,933] jepsen worker 42 - jepsen.util 402 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,933] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:50,933] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,935] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:50,935] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:50,940] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,940] jepsen worker 32 - jepsen.util 2012 :info :write [10 16] :RequestTimeoutException INFO [2022-03-13 17:12:50,941] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:12:50,942] jepsen worker 32 - jepsen.util 2102 :invoke :write [10 34] INFO [2022-03-13 17:12:50,964] jepsen worker 27 - jepsen.util 2547 :info :write [9 18] :RequestTimeoutException INFO [2022-03-13 17:12:50,965] jepsen worker 27 - jepsen.util 2637 :invoke :read [9 nil] INFO [2022-03-13 17:12:50,994] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:50,995] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:51,004] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,004] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,011] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,012] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:12:51,016] jepsen worker 3 - jepsen.util 1893 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,016] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,017] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:12:51,017] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:12:51,030] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,031] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,032] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,033] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:12:51,034] jepsen worker 77 - jepsen.util 707 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,035] jepsen worker 77 - jepsen.util 707 :invoke :write [25 39] INFO [2022-03-13 17:12:51,090] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,090] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:12:51,105] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,106] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:51,106] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,107] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 27] INFO [2022-03-13 17:12:51,116] jepsen worker 69 - jepsen.util 609 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,116] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,116] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,117] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:51,119] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:12:51,119] jepsen worker 71 - jepsen.util 521 :invoke :write [23 47] INFO [2022-03-13 17:12:51,122] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,123] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:12:51,126] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,127] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,128] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,129] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,131] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,131] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,133] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,133] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,135] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,135] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,138] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,138] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,140] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,141] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,142] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,142] jepsen worker 61 - jepsen.util 601 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,142] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,142] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:51,142] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,143] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:51,144] jepsen worker 43 - jepsen.util 493 :invoke :write [14 28] INFO [2022-03-13 17:12:51,144] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:51,145] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,145] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:12:51,174] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,175] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:12:51,184] jepsen worker 79 - jepsen.util 979 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,184] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:51,187] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,187] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,188] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:12:51,189] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:51,203] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:51,203] jepsen worker nemesis - couchbase.util Rebalance check count: 281 INFO [2022-03-13 17:12:51,203] jepsen worker nemesis - couchbase.util Rebalance stuck count: 185 INFO [2022-03-13 17:12:51,219] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,220] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:12:51,228] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,228] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,228] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,229] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,228] jepsen worker 52 - jepsen.util 592 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,229] jepsen worker 52 - jepsen.util 592 :invoke :write [17 18] INFO [2022-03-13 17:12:51,230] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:12:51,231] jepsen worker 13 - jepsen.util 643 :invoke :write [4 5] INFO [2022-03-13 17:12:51,232] jepsen worker 34 - jepsen.util 754 :invoke :write [11 1] INFO [2022-03-13 17:12:51,233] jepsen worker 80 - jepsen.util 710 :invoke :write [26 45] INFO [2022-03-13 17:12:51,240] jepsen worker 60 - jepsen.util 600 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,241] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:51,241] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,242] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,249] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,249] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:12:51,272] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,272] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:12:51,293] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,294] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,296] jepsen worker 29 - jepsen.util 2369 :info :write [9 26] :RequestTimeoutException INFO [2022-03-13 17:12:51,297] jepsen worker 29 - jepsen.util 2459 :invoke :write [9 5] INFO [2022-03-13 17:12:51,331] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,332] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,346] jepsen worker 14 - jepsen.util 284 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,346] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,347] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:12:51,348] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:51,349] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,349] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,349] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,349] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,350] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:51,350] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:12:51,352] jepsen worker 69 - jepsen.util 609 :invoke :write [23 23] INFO [2022-03-13 17:12:51,353] jepsen worker 21 - jepsen.util 741 :invoke :write [7 6] INFO [2022-03-13 17:12:51,360] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:51,360] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,365] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,365] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:12:51,369] jepsen worker 36 - jepsen.util 2106 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,369] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,369] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,370] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:12:51,411] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,411] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:12:51,424] jepsen worker 76 - jepsen.util 706 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,424] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,424] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,424] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:51,425] jepsen worker 23 - jepsen.util 563 :invoke :write [7 3] INFO [2022-03-13 17:12:51,426] jepsen worker 12 - jepsen.util 732 :invoke :write [4 33] INFO [2022-03-13 17:12:51,426] jepsen worker 44 - jepsen.util 584 :invoke :write [14 23] INFO [2022-03-13 17:12:51,427] jepsen worker 76 - jepsen.util 706 :invoke :write [25 23] INFO [2022-03-13 17:12:51,436] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:51,438] jepsen worker 71 - jepsen.util 521 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,439] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] INFO [2022-03-13 17:12:51,440] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 32] INFO [2022-03-13 17:12:51,449] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,449] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] INFO [2022-03-13 17:12:51,465] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,465] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,466] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,466] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,467] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,468] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,468] jepsen worker 6 - jepsen.util 636 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,470] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,470] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,471] jepsen worker 6 - jepsen.util 636 :invoke :write [2 41] INFO [2022-03-13 17:12:51,471] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,471] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,473] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,475] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,476] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,476] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,477] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,478] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,478] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:12:51,483] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:51,484] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,484] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:12:51,504] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,505] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:12:51,523] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,523] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,523] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,524] jepsen worker 79 - jepsen.util 979 :invoke :write [26 16] INFO [2022-03-13 17:12:51,524] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:12:51,525] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,548] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,549] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:51,550] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,550] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,551] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,552] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,554] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,554] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,570] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,571] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,572] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,573] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:12:51,591] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,592] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,595] jepsen worker 51 - jepsen.util 591 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,596] jepsen worker 51 - jepsen.util 591 :invoke :write [17 13] INFO [2022-03-13 17:12:51,612] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,613] jepsen worker 22 - jepsen.util 652 :invoke :write [7 32] INFO [2022-03-13 17:12:51,616] jepsen worker 37 - jepsen.util 2017 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,616] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,617] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:12:51,617] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,620] jepsen worker 88 - jepsen.util 1978 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,621] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:12:51,641] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,642] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:12:51,651] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,652] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:12:51,668] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:51,669] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,669] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:12:51,670] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:12:51,672] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,672] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:51,672] jepsen worker 8 - jepsen.util 728 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,672] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,672] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,672] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,673] jepsen worker 61 - jepsen.util 601 :invoke :write [20 11] INFO [2022-03-13 17:12:51,674] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:51,674] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:51,675] jepsen worker 62 - jepsen.util 512 :invoke :write [20 37] INFO [2022-03-13 17:12:51,675] jepsen worker 42 - jepsen.util 402 :invoke :write [14 7] INFO [2022-03-13 17:12:51,678] jepsen worker 8 - jepsen.util 728 :invoke :write [2 27] INFO [2022-03-13 17:12:51,685] jepsen worker 21 - jepsen.util 741 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,686] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:51,698] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,698] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,699] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,700] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:12:51,715] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,716] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:12:51,725] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,726] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,737] jepsen worker 69 - jepsen.util 609 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,738] jepsen worker 69 - jepsen.util 609 :invoke :write [23 32] INFO [2022-03-13 17:12:51,764] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,765] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:12:51,767] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,768] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:51,768] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 27] INFO [2022-03-13 17:12:51,768] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 17] INFO [2022-03-13 17:12:51,775] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,776] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:12:51,817] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,818] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,834] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,835] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,841] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,842] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:12:51,844] jepsen worker 77 - jepsen.util 707 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,844] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,845] jepsen worker 77 - jepsen.util 707 :invoke :write [25 8] INFO [2022-03-13 17:12:51,845] jepsen worker 60 - jepsen.util 600 :invoke :write [20 46] INFO [2022-03-13 17:12:51,859] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,859] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:51,864] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:51,865] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,889] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,889] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:51,889] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,890] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:12:51,897] jepsen worker 88 - jepsen.util 1978 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,898] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:51,921] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,922] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,930] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,931] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:51,955] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:51,955] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:51,969] jepsen worker 43 - jepsen.util 493 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,969] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,969] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,971] jepsen worker 43 - jepsen.util 493 :invoke :write [14 45] INFO [2022-03-13 17:12:51,971] jepsen worker 14 - jepsen.util 284 :invoke :write [4 28] INFO [2022-03-13 17:12:51,975] jepsen worker 7 - jepsen.util 637 :invoke :write [2 39] INFO [2022-03-13 17:12:51,994] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:51,995] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,003] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,003] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:12:52,006] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,006] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,007] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,007] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:12:52,018] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,019] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,029] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,030] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:12:52,043] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,043] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:52,044] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:12:52,044] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:12:52,048] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:52,048] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:12:52,071] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,072] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,073] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,074] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:12:52,111] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,111] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:12:52,120] jepsen worker 36 - jepsen.util 2106 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,120] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,120] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,121] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,122] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,122] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,123] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,123] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 7] INFO [2022-03-13 17:12:52,124] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,125] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:12:52,125] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,127] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,139] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,139] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,157] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,157] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 7] INFO [2022-03-13 17:12:52,163] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,163] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:52,187] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,187] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,188] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,188] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:52,208] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:52,208] jepsen worker nemesis - couchbase.util Rebalance check count: 282 INFO [2022-03-13 17:12:52,209] jepsen worker nemesis - couchbase.util Rebalance stuck count: 186 INFO [2022-03-13 17:12:52,217] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:52,218] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:12:52,222] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,223] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:52,251] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:52,252] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:12:52,255] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,256] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:12:52,263] jepsen worker 36 - jepsen.util 2106 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,264] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 13] INFO [2022-03-13 17:12:52,288] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,290] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,291] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,292] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:12:52,293] jepsen worker 9 - jepsen.util 2079 :ok :read [3 25] INFO [2022-03-13 17:12:52,294] jepsen worker 9 - jepsen.util 2079 :invoke :write [3 47] INFO [2022-03-13 17:12:52,316] jepsen worker 55 - jepsen.util 2125 :ok :read [18 21] INFO [2022-03-13 17:12:52,317] jepsen worker 55 - jepsen.util 2125 :invoke :write [18 46] INFO [2022-03-13 17:12:52,343] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,343] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,344] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,344] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:12:52,351] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,351] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 21] INFO [2022-03-13 17:12:52,379] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,379] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:12:52,396] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,397] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:12:52,406] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,406] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:12:52,459] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,459] jepsen worker 36 - jepsen.util 2106 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,459] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:12:52,461] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,470] jepsen worker 3 - jepsen.util 1893 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,471] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:12:52,513] jepsen worker 88 - jepsen.util 1978 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,514] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:12:52,529] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,529] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,530] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 46] INFO [2022-03-13 17:12:52,530] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:12:52,569] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,570] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:12:52,581] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,581] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,623] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,623] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,625] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,625] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,626] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,627] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:12:52,639] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,640] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:12:52,671] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,672] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:52,703] jepsen worker 47 - jepsen.util 2297 :ok :read [15 48] INFO [2022-03-13 17:12:52,704] jepsen worker 47 - jepsen.util 2297 :invoke :write [15 42] INFO [2022-03-13 17:12:52,714] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,714] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:12:52,724] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,725] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,770] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,770] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:52,795] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,795] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,797] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,797] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,811] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:52,812] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:52,814] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,814] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,814] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,818] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:12:52,819] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:12:52,819] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:12:52,829] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:52,829] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:12:52,860] jepsen worker 36 - jepsen.util 2106 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,860] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 40] INFO [2022-03-13 17:12:52,862] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,863] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:12:52,898] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:52,898] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:12:52,909] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,909] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,914] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:12:52,915] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,959] jepsen worker 36 - jepsen.util 2106 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,959] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,960] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:52,960] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:52,965] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,965] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:12:52,984] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:52,984] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:12:53,041] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,042] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:12:53,071] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,102] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,102] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,224] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:53,102] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,224] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 40] INFO [2022-03-13 17:12:53,102] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,103] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,224] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:53,120] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,224] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:12:53,224] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:53,149] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,225] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:53,226] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,226] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,227] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:12:53,214] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:53,224] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 17] INFO [2022-03-13 17:12:53,227] jepsen worker nemesis - couchbase.util Rebalance check count: 283 INFO [2022-03-13 17:12:53,224] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:12:53,227] jepsen worker nemesis - couchbase.util Rebalance stuck count: 187 INFO [2022-03-13 17:12:53,231] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:12:53,240] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,243] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:53,250] jepsen worker 31 - jepsen.util 2281 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:12:53,251] jepsen worker 31 - jepsen.util 2371 :invoke :write [10 45] INFO [2022-03-13 17:12:53,264] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:53,265] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:12:53,331] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,331] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,332] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:12:53,332] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:53,335] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,335] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,335] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:12:53,337] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:53,347] jepsen worker 88 - jepsen.util 1978 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,347] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:12:53,369] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,370] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:12:53,403] jepsen worker 76 - jepsen.util 706 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,403] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:53,417] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,418] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:53,422] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,422] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,423] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:53,423] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:12:53,426] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,427] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,427] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:12:53,427] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:53,452] jepsen worker 79 - jepsen.util 979 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,453] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:53,456] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,457] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:12:53,474] jepsen worker 6 - jepsen.util 636 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,474] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,474] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,475] jepsen worker 26 - jepsen.util 476 :invoke :write [8 46] INFO [2022-03-13 17:12:53,475] jepsen worker 53 - jepsen.util 503 :invoke :write [17 35] INFO [2022-03-13 17:12:53,477] jepsen worker 6 - jepsen.util 636 :invoke :write [2 14] INFO [2022-03-13 17:12:53,485] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,486] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:53,520] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:53,521] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:53,553] jepsen worker 36 - jepsen.util 2106 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,554] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:53,556] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,556] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,556] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:12:53,557] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:12:53,559] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,559] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,559] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,560] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:53,561] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:12:53,561] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,562] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:12:53,562] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,562] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:53,562] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,563] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:53,563] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:12:53,564] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,564] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:53,565] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,566] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:12:53,576] jepsen worker 12 - jepsen.util 732 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,578] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:53,598] jepsen worker 8 - jepsen.util 728 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,601] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:12:53,621] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,622] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:53,639] jepsen worker 44 - jepsen.util 584 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,639] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:53,648] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:53,648] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:53,666] jepsen worker 42 - jepsen.util 402 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,667] jepsen worker 42 - jepsen.util 402 :invoke :write [14 1] INFO [2022-03-13 17:12:53,691] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,691] jepsen worker 69 - jepsen.util 609 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,692] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,692] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:12:53,693] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:53,693] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:12:53,780] jepsen worker 23 - jepsen.util 563 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,780] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:53,791] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,792] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:53,814] jepsen worker 62 - jepsen.util 512 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,815] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:53,896] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,896] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,896] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,897] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:53,897] jepsen worker 21 - jepsen.util 741 :invoke :write [7 0] INFO [2022-03-13 17:12:53,899] jepsen worker 79 - jepsen.util 979 :invoke :write [26 45] INFO [2022-03-13 17:12:53,949] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,949] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,950] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:53,951] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:53,997] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:53,998] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:12:54,014] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,014] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,015] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:12:54,015] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:12:54,024] jepsen worker 51 - jepsen.util 591 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,024] jepsen worker 51 - jepsen.util 591 :invoke :write [17 10] INFO [2022-03-13 17:12:54,068] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,068] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,069] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:54,069] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:12:54,136] jepsen worker 61 - jepsen.util 601 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,136] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:54,157] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,158] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:12:54,175] jepsen worker 60 - jepsen.util 600 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,175] jepsen worker 60 - jepsen.util 600 :invoke :write [20 27] INFO [2022-03-13 17:12:54,191] jepsen worker 21 - jepsen.util 741 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,192] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:54,198] jepsen worker 53 - jepsen.util 503 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,200] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:54,234] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:54,235] jepsen worker nemesis - couchbase.util Rebalance check count: 284 INFO [2022-03-13 17:12:54,235] jepsen worker nemesis - couchbase.util Rebalance stuck count: 188 INFO [2022-03-13 17:12:54,238] jepsen worker 19 - jepsen.util 2179 :ok :read [6 16] INFO [2022-03-13 17:12:54,239] jepsen worker 19 - jepsen.util 2179 :invoke :read [6 nil] INFO [2022-03-13 17:12:54,247] jepsen worker 20 - jepsen.util 2630 :ok :read [6 16] INFO [2022-03-13 17:12:54,247] jepsen worker 20 - jepsen.util 2630 :invoke :write [6 49] INFO [2022-03-13 17:12:54,269] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,270] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:12:54,321] jepsen worker 79 - jepsen.util 979 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,321] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,321] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,321] jepsen worker 79 - jepsen.util 979 :invoke :read [26 nil] INFO [2022-03-13 17:12:54,322] jepsen worker 69 - jepsen.util 609 :invoke :write [23 40] INFO [2022-03-13 17:12:54,323] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:12:54,330] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:54,330] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,330] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,330] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,330] jepsen worker 77 - jepsen.util 707 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,331] jepsen worker 77 - jepsen.util 707 :invoke :write [25 23] INFO [2022-03-13 17:12:54,332] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:54,332] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:54,333] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:12:54,333] jepsen worker 44 - jepsen.util 584 :invoke :write [14 18] INFO [2022-03-13 17:12:54,338] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,338] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:12:54,444] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,445] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,445] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:12:54,446] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:54,508] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,508] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:54,513] jepsen worker 6 - jepsen.util 636 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,514] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:12:54,562] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,562] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,563] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:12:54,564] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:54,620] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,620] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:54,630] jepsen worker 51 - jepsen.util 591 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,631] jepsen worker 51 - jepsen.util 591 :invoke :write [17 24] INFO [2022-03-13 17:12:54,646] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,646] jepsen worker 79 - jepsen.util 979 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,646] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,647] jepsen worker 21 - jepsen.util 741 :invoke :read [7 nil] INFO [2022-03-13 17:12:54,648] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:12:54,648] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,648] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,648] jepsen worker 79 - jepsen.util 979 :invoke :write [26 17] INFO [2022-03-13 17:12:54,650] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:54,651] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:12:54,651] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,652] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:12:54,660] jepsen worker 39 - jepsen.util 2199 :ok :read [13 39] INFO [2022-03-13 17:12:54,660] jepsen worker 39 - jepsen.util 2199 :invoke :read [13 nil] INFO [2022-03-13 17:12:54,735] jepsen worker 26 - jepsen.util 476 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,735] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,736] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:54,737] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:54,766] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,766] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:12:54,862] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,862] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:54,882] jepsen worker 60 - jepsen.util 600 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,882] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:54,883] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,883] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,884] jepsen worker 76 - jepsen.util 706 :invoke :write [25 37] INFO [2022-03-13 17:12:54,884] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:54,984] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,984] jepsen worker 69 - jepsen.util 609 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:54,985] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:54,985] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:55,024] jepsen worker 42 - jepsen.util 402 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,024] jepsen worker 42 - jepsen.util 402 :invoke :write [14 23] INFO [2022-03-13 17:12:55,034] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,035] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:55,037] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,038] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:55,038] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,039] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:12:55,039] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,043] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:12:55,062] jepsen worker 21 - jepsen.util 741 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,062] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,062] jepsen worker 21 - jepsen.util 741 :invoke :write [7 45] INFO [2022-03-13 17:12:55,063] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:12:55,228] jepsen worker 77 - jepsen.util 707 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,229] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:55,231] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,231] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:55,237] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,237] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:55,241] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5906432748538012}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:55,241] jepsen worker nemesis - couchbase.util Rebalance check count: 285 INFO [2022-03-13 17:12:55,241] jepsen worker nemesis - couchbase.util Rebalance stuck count: 189 INFO [2022-03-13 17:12:55,283] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,283] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,283] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:12:55,284] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:12:55,284] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,285] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,285] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,285] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,287] jepsen worker 8 - jepsen.util 728 :invoke :write [2 34] INFO [2022-03-13 17:12:55,288] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:55,289] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:55,302] jepsen worker 6 - jepsen.util 636 :invoke :write [2 8] INFO [2022-03-13 17:12:55,364] jepsen worker 79 - jepsen.util 979 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,365] jepsen worker 79 - jepsen.util 979 :invoke :write [26 37] INFO [2022-03-13 17:12:55,428] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,429] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:12:55,446] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,447] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:55,480] jepsen worker 81 - jepsen.util 2511 :info :write [27 35] :RequestTimeoutException INFO [2022-03-13 17:12:55,481] jepsen worker 81 - jepsen.util 2601 :invoke :read [27 nil] INFO [2022-03-13 17:12:55,490] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,491] jepsen worker 35 - jepsen.util 935 :invoke :write [11 3] INFO [2022-03-13 17:12:55,493] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,493] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,493] jepsen worker 69 - jepsen.util 609 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,493] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,494] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:12:55,494] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:12:55,495] jepsen worker 69 - jepsen.util 609 :invoke :write [23 22] INFO [2022-03-13 17:12:55,495] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:12:55,535] jepsen worker 14 - jepsen.util 284 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,536] jepsen worker 14 - jepsen.util 284 :invoke :write [4 18] INFO [2022-03-13 17:12:55,636] jepsen worker 21 - jepsen.util 741 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,636] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,637] jepsen worker 21 - jepsen.util 741 :invoke :write [7 5] INFO [2022-03-13 17:12:55,638] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:12:55,641] jepsen worker 83 - jepsen.util 1523 :info :write [27 8] :RequestTimeoutException INFO [2022-03-13 17:12:55,642] jepsen worker 83 - jepsen.util 1613 :invoke :read [27 nil] INFO [2022-03-13 17:12:55,642] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,643] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:12:55,650] jepsen worker 41 - jepsen.util 2021 :info :write [13 14] :RequestTimeoutException INFO [2022-03-13 17:12:55,651] jepsen worker 41 - jepsen.util 2111 :invoke :write [13 5] INFO [2022-03-13 17:12:55,663] jepsen worker 44 - jepsen.util 584 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,663] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:55,666] jepsen worker 66 - jepsen.util 1956 :info :write [22 38] :RequestTimeoutException INFO [2022-03-13 17:12:55,667] jepsen worker 66 - jepsen.util 2046 :invoke :write [22 21] INFO [2022-03-13 17:12:55,770] jepsen worker 79 - jepsen.util 979 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,771] jepsen worker 79 - jepsen.util 979 :invoke :write [26 33] INFO [2022-03-13 17:12:55,832] jepsen worker 35 - jepsen.util 935 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,832] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:12:55,846] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,846] jepsen worker 43 - jepsen.util 493 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,846] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,847] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:55,847] jepsen worker 26 - jepsen.util 476 :invoke :write [8 12] INFO [2022-03-13 17:12:55,850] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:55,874] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,874] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:12:55,934] jepsen worker 2 - jepsen.util 2522 :info :write [0 19] :RequestTimeoutException INFO [2022-03-13 17:12:55,934] jepsen worker 2 - jepsen.util 2612 :invoke :read [0 nil] INFO [2022-03-13 17:12:55,934] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:55,935] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,048] jepsen worker 69 - jepsen.util 609 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,049] jepsen worker 51 - jepsen.util 591 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,049] jepsen worker 69 - jepsen.util 609 :invoke :read [23 nil] INFO [2022-03-13 17:12:56,049] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,050] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,050] jepsen worker 51 - jepsen.util 591 :invoke :write [17 32] INFO [2022-03-13 17:12:56,051] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:56,051] jepsen worker 12 - jepsen.util 732 :invoke :write [4 49] INFO [2022-03-13 17:12:56,056] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,056] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,057] jepsen worker 35 - jepsen.util 935 :invoke :write [11 2] INFO [2022-03-13 17:12:56,058] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,059] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,060] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] WARN [2022-03-13 17:12:56,122] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":23835783,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23835783,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9055417,"encode_duration_us":16749,"operation_id":"0x6c1a","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10001947},{"operation_name":"upsert","last_dispatch_duration_us":27561026,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27561026,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9068168,"encode_duration_us":29946,"operation_id":"0x6b74","timeout_ms":10000,"last_server_duration_us":9068168,"total_duration_us":10010065}],"total_count":2}} WARN [2022-03-13 17:12:56,122] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":21809400,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":21809400,"total_server_duration_us":38,"operation_id":"0x6d1b","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":21811777},{"operation_name":"get","last_dispatch_duration_us":20741916,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":20741916,"total_server_duration_us":2578446,"operation_id":"0x6b9c","timeout_ms":30000,"last_server_duration_us":2578446,"total_duration_us":20744080},{"operation_name":"get","last_dispatch_duration_us":16293275,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16293275,"total_server_duration_us":15,"operation_id":"0x6fa0","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":16295631},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":18794,"operation_id":"0x74df","timeout_ms":10000,"total_duration_us":10008529},{"operation_name":"upsert","last_dispatch_duration_us":109142,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9568491,"total_server_duration_us":57868,"encode_duration_us":25058,"operation_id":"0x73b9","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10007046},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":18066,"operation_id":"0x70ea","timeout_ms":10000,"total_duration_us":10003001},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":20588,"operation_id":"0x7036","timeout_ms":10000,"total_duration_us":10001952},{"operation_name":"get","last_dispatch_duration_us":2990035,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2990035,"total_server_duration_us":344182,"operation_id":"0x7791","timeout_ms":30000,"last_server_duration_us":344182,"total_duration_us":2992418},{"operation_name":"upsert","last_dispatch_duration_us":1686375,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1686375,"total_server_duration_us":1522735,"encode_duration_us":17816,"operation_id":"0x78ec","timeout_ms":10000,"last_server_duration_us":1522735,"total_duration_us":1692234},{"operation_name":"upsert","last_dispatch_duration_us":1635015,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1635015,"total_server_duration_us":88098,"encode_duration_us":25528,"operation_id":"0x7900","timeout_ms":10000,"last_server_duration_us":88098,"total_duration_us":1639944}],"total_count":12}} INFO [2022-03-13 17:12:56,188] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,188] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:12:56,212] jepsen worker 42 - jepsen.util 402 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,213] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:12:56,214] jepsen worker 35 - jepsen.util 935 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,215] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:56,247] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5964912280701755}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:56,247] jepsen worker nemesis - couchbase.util Rebalance check count: 286 INFO [2022-03-13 17:12:56,247] jepsen worker nemesis - couchbase.util Rebalance stuck count: 189 INFO [2022-03-13 17:12:56,248] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,248] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:56,252] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,252] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:56,252] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,253] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,253] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,254] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:56,255] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,255] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,255] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,256] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:56,259] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,260] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,262] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,263] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 31] INFO [2022-03-13 17:12:56,276] jepsen worker 73 - jepsen.util 2323 :info :write [24 22] :RequestTimeoutException INFO [2022-03-13 17:12:56,277] jepsen worker 73 - jepsen.util 2413 :invoke :write [24 23] INFO [2022-03-13 17:12:56,322] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,323] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:12:56,347] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,348] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,351] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:12:56,351] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:12:56,376] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:56,377] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:56,402] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:56,403] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:56,410] jepsen worker 10 - jepsen.util 2260 :info :write [3 1] :RequestTimeoutException INFO [2022-03-13 17:12:56,411] jepsen worker 10 - jepsen.util 2350 :invoke :write [3 15] INFO [2022-03-13 17:12:56,423] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,423] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,427] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,427] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:12:56,440] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,440] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,441] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 16] INFO [2022-03-13 17:12:56,441] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,448] jepsen worker 11 - jepsen.util 2261 :info :write [3 41] :RequestTimeoutException INFO [2022-03-13 17:12:56,449] jepsen worker 11 - jepsen.util 2351 :invoke :read [3 nil] INFO [2022-03-13 17:12:56,457] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,458] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:56,465] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,466] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:12:56,476] jepsen worker 82 - jepsen.util 2152 :ok :read [27 4] INFO [2022-03-13 17:12:56,476] jepsen worker 82 - jepsen.util 2152 :invoke :write [27 43] INFO [2022-03-13 17:12:56,501] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,501] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,501] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:56,502] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,503] jepsen worker 83 - jepsen.util 1613 :ok :read [27 4] INFO [2022-03-13 17:12:56,503] jepsen worker 83 - jepsen.util 1613 :invoke :read [27 nil] INFO [2022-03-13 17:12:56,507] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:56,508] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:56,534] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:56,535] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:12:56,570] jepsen worker 64 - jepsen.util 2224 :info :write [21 43] :RequestTimeoutException INFO [2022-03-13 17:12:56,571] jepsen worker 64 - jepsen.util 2314 :invoke :read [21 nil] INFO [2022-03-13 17:12:56,582] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,583] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:56,584] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,585] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:56,587] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,588] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:56,589] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,589] jepsen worker 83 - jepsen.util 1613 :ok :read [27 4] INFO [2022-03-13 17:12:56,589] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:56,591] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,591] jepsen worker 83 - jepsen.util 1613 :invoke :read [27 nil] INFO [2022-03-13 17:12:56,591] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:12:56,605] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,606] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:12:56,610] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,611] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,611] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,611] jepsen worker 37 - jepsen.util 2017 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,612] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:12:56,613] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:56,633] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,634] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,647] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:56,648] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:12:56,664] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,665] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:56,686] jepsen worker 83 - jepsen.util 1613 :ok :read [27 4] INFO [2022-03-13 17:12:56,687] jepsen worker 83 - jepsen.util 1613 :invoke :write [27 22] INFO [2022-03-13 17:12:56,703] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:56,703] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:12:56,734] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,735] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,735] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:12:56,735] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:12:56,746] jepsen worker 36 - jepsen.util 2106 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,746] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 31] INFO [2022-03-13 17:12:56,748] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,748] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,748] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,749] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:12:56,751] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,751] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:12:56,761] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,762] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:12:56,763] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,764] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:12:56,768] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,769] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,768] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,770] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:12:56,776] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,776] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,778] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,778] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,779] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,780] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,782] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,782] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,792] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,793] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,793] jepsen worker 58 - jepsen.util 2128 :info :write [19 36] :RequestTimeoutException INFO [2022-03-13 17:12:56,794] jepsen worker 58 - jepsen.util 2218 :invoke :read [19 nil] INFO [2022-03-13 17:12:56,795] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,795] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:56,804] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,804] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:12:56,831] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,832] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:56,843] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,844] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:12:56,858] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:56,858] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 18] INFO [2022-03-13 17:12:56,866] jepsen worker 50 - jepsen.util 2300 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:12:56,867] jepsen worker 50 - jepsen.util 2390 :invoke :read [16 nil] INFO [2022-03-13 17:12:56,871] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,872] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:12:56,876] jepsen worker 24 - jepsen.util 564 :ok :read [8 :nil] INFO [2022-03-13 17:12:56,877] jepsen worker 24 - jepsen.util 564 :invoke :write [8 5] INFO [2022-03-13 17:12:56,887] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:56,887] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,887] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,888] jepsen worker 60 - jepsen.util 600 :invoke :write [20 32] INFO [2022-03-13 17:12:56,889] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:12:56,890] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:56,900] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,901] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:56,906] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,906] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:12:56,928] jepsen worker 69 - jepsen.util 609 :ok :read [23 30] INFO [2022-03-13 17:12:56,929] jepsen worker 69 - jepsen.util 609 :invoke :write [23 10] INFO [2022-03-13 17:12:56,942] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,943] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:56,959] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,959] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,963] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:56,963] jepsen worker 61 - jepsen.util 601 :invoke :write [20 9] INFO [2022-03-13 17:12:56,964] jepsen worker 75 - jepsen.util 435 :ok :read [25 6] INFO [2022-03-13 17:12:56,965] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:12:56,971] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,971] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:12:56,977] jepsen worker 88 - jepsen.util 1978 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:56,978] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:12:57,041] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,041] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:57,051] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,052] jepsen worker 77 - jepsen.util 707 :ok :read [25 6] INFO [2022-03-13 17:12:57,052] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:12:57,052] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:12:57,110] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,111] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:12:57,116] jepsen worker 49 - jepsen.util 1939 :ok :read [16 :nil] INFO [2022-03-13 17:12:57,116] jepsen worker 49 - jepsen.util 1939 :invoke :read [16 nil] INFO [2022-03-13 17:12:57,133] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,133] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:57,142] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,142] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,143] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:57,143] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:12:57,178] jepsen worker 51 - jepsen.util 591 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,179] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:57,200] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,201] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:12:57,217] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,217] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,218] jepsen worker 0 - jepsen.util 2250 :ok :read [0 41] INFO [2022-03-13 17:12:57,218] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:12:57,219] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:57,220] jepsen worker 0 - jepsen.util 2250 :invoke :read [0 nil] INFO [2022-03-13 17:12:57,221] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,221] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,222] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:57,222] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:57,223] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,224] jepsen worker 8 - jepsen.util 728 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,224] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,224] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:57,225] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:57,225] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,225] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:12:57,226] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:12:57,226] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,227] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:57,236] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,237] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:12:57,256] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5964912280701755}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:57,256] jepsen worker nemesis - couchbase.util Rebalance check count: 287 INFO [2022-03-13 17:12:57,256] jepsen worker nemesis - couchbase.util Rebalance stuck count: 190 INFO [2022-03-13 17:12:57,270] jepsen worker 68 - jepsen.util 2318 :info :write [22 49] :RequestTimeoutException INFO [2022-03-13 17:12:57,271] jepsen worker 68 - jepsen.util 2408 :invoke :write [22 30] INFO [2022-03-13 17:12:57,299] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,299] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,299] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] INFO [2022-03-13 17:12:57,300] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:12:57,304] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,304] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:12:57,340] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,341] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:12:57,367] jepsen worker 12 - jepsen.util 732 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,368] jepsen worker 12 - jepsen.util 732 :invoke :write [4 48] INFO [2022-03-13 17:12:57,379] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,379] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:57,380] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,381] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:57,390] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,391] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:12:57,421] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,422] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] WARN [2022-03-13 17:12:57,428] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17857405,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18149087,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9154727,"encode_duration_us":17099,"operation_id":"0x70cc","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10005029},{"operation_name":"upsert","last_dispatch_duration_us":24868602,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":33835332,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9138448,"encode_duration_us":17071,"operation_id":"0x66a8","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10009394}],"total_count":2}} INFO [2022-03-13 17:12:57,448] jepsen worker 1 - jepsen.util 1981 :info :write [0 22] :RequestTimeoutException INFO [2022-03-13 17:12:57,449] jepsen worker 1 - jepsen.util 2071 :invoke :read [0 nil] INFO [2022-03-13 17:12:57,457] jepsen worker 78 - jepsen.util 528 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,458] jepsen worker 78 - jepsen.util 528 :invoke :write [26 15] INFO [2022-03-13 17:12:57,468] jepsen worker 6 - jepsen.util 636 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,469] jepsen worker 6 - jepsen.util 636 :invoke :write [2 12] INFO [2022-03-13 17:12:57,475] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,476] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:12:57,546] jepsen worker 64 - jepsen.util 2314 :ok :read [21 :nil] INFO [2022-03-13 17:12:57,547] jepsen worker 64 - jepsen.util 2314 :invoke :read [21 nil] INFO [2022-03-13 17:12:57,549] jepsen worker 64 - jepsen.util 2314 :ok :read [21 :nil] INFO [2022-03-13 17:12:57,550] jepsen worker 64 - jepsen.util 2314 :invoke :write [21 11] INFO [2022-03-13 17:12:57,559] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,560] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:12:57,591] jepsen worker 63 - jepsen.util 2133 :ok :read [21 :nil] INFO [2022-03-13 17:12:57,592] jepsen worker 63 - jepsen.util 2133 :invoke :read [21 nil] INFO [2022-03-13 17:12:57,606] jepsen worker 46 - jepsen.util 2296 :ok :read [15 48] INFO [2022-03-13 17:12:57,607] jepsen worker 46 - jepsen.util 2296 :invoke :write [15 5] WARN [2022-03-13 17:12:57,622] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17741271,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17741271,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9027816,"encode_duration_us":17847,"operation_id":"0x6d62","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10004609},{"operation_name":"upsert","last_dispatch_duration_us":11183164,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11183164,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9047983,"encode_duration_us":18358,"operation_id":"0x750f","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10008069}],"total_count":2}} WARN [2022-03-13 17:12:57,626] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17568162,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17729097,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9134922,"encode_duration_us":18077,"operation_id":"0x6ff8","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10003993},{"operation_name":"upsert","last_dispatch_duration_us":16639881,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17359055,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9281554,"encode_duration_us":20597,"operation_id":"0x6fb1","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10006032}],"total_count":2}} WARN [2022-03-13 17:12:57,629] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":17564,"operation_id":"0x7204","timeout_ms":10000,"total_duration_us":10008538},{"operation_name":"upsert","last_dispatch_duration_us":37536,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9447674,"total_server_duration_us":237143,"encode_duration_us":18719,"operation_id":"0x75d2","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10005827},{"operation_name":"upsert","last_dispatch_duration_us":17857405,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18149087,"total_server_duration_us":9154727,"encode_duration_us":17099,"operation_id":"0x70cc","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10005029},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":16831,"operation_id":"0x7542","timeout_ms":10000,"total_duration_us":10003585},{"operation_name":"get","last_dispatch_duration_us":8318512,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8318512,"total_server_duration_us":9227,"operation_id":"0x765c","timeout_ms":30000,"last_server_duration_us":9227,"total_duration_us":8320753},{"operation_name":"upsert","last_dispatch_duration_us":2867405,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":2867405,"total_server_duration_us":49140,"encode_duration_us":18000,"operation_id":"0x798e","timeout_ms":10000,"last_server_duration_us":49140,"total_duration_us":2871799},{"operation_name":"upsert","last_dispatch_duration_us":2281573,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2281573,"total_server_duration_us":97638,"encode_duration_us":19873,"operation_id":"0x78fd","timeout_ms":10000,"last_server_duration_us":97638,"total_duration_us":2464838},{"operation_name":"upsert","last_dispatch_duration_us":2301789,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2301789,"total_server_duration_us":178288,"encode_duration_us":18978,"operation_id":"0x78d5","timeout_ms":10000,"last_server_duration_us":178288,"total_duration_us":2408114},{"operation_name":"upsert","last_dispatch_duration_us":2288557,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2288557,"total_server_duration_us":95789,"encode_duration_us":30455,"operation_id":"0x78a1","timeout_ms":10000,"last_server_duration_us":95789,"total_duration_us":2323758},{"operation_name":"upsert","last_dispatch_duration_us":2132244,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2132244,"total_server_duration_us":20045,"encode_duration_us":18314,"operation_id":"0x78fc","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":2310957}],"total_count":70}} INFO [2022-03-13 17:12:57,690] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,690] jepsen worker 76 - jepsen.util 706 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,690] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,690] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:57,691] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:12:57,692] jepsen worker 70 - jepsen.util 700 :invoke :write [23 38] INFO [2022-03-13 17:12:57,693] jepsen worker 44 - jepsen.util 584 :invoke :write [14 14] INFO [2022-03-13 17:12:57,693] jepsen worker 23 - jepsen.util 563 :invoke :write [7 26] WARN [2022-03-13 17:12:57,723] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":2409319,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":2409319,"total_server_duration_us":20521,"encode_duration_us":24877,"operation_id":"0x74e0","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10007444},{"operation_name":"upsert","last_dispatch_duration_us":9112793,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11654610,"total_server_duration_us":54294,"encode_duration_us":18897,"operation_id":"0x74a9","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10006283},{"operation_name":"upsert","last_dispatch_duration_us":8834498,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8834498,"total_server_duration_us":21977,"encode_duration_us":17653,"operation_id":"0x755d","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10004698},{"operation_name":"upsert","last_dispatch_duration_us":16535831,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":16535831,"total_server_duration_us":21001,"encode_duration_us":17239,"operation_id":"0x7266","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10003824},{"operation_name":"get","last_dispatch_duration_us":9144900,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9144900,"total_server_duration_us":38,"operation_id":"0x7640","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9145398},{"operation_name":"get","last_dispatch_duration_us":9138893,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9138893,"total_server_duration_us":56,"operation_id":"0x7650","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9141463},{"operation_name":"get","last_dispatch_duration_us":9110435,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9110435,"total_server_duration_us":2268694,"operation_id":"0x7120","timeout_ms":30000,"last_server_duration_us":2268694,"total_duration_us":9113130},{"operation_name":"get","last_dispatch_duration_us":3028591,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":3028591,"total_server_duration_us":43,"operation_id":"0x7479","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3028857},{"operation_name":"upsert","last_dispatch_duration_us":2926219,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2926219,"total_server_duration_us":85888,"encode_duration_us":17455,"operation_id":"0x76cb","timeout_ms":10000,"last_server_duration_us":85888,"total_duration_us":2931048},{"operation_name":"upsert","last_dispatch_duration_us":2827444,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":2827444,"total_server_duration_us":66431,"encode_duration_us":20885,"operation_id":"0x79b2","timeout_ms":10000,"last_server_duration_us":66431,"total_duration_us":2839466}],"total_count":14}} WARN [2022-03-13 17:12:57,727] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14960639,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":14960639,"total_server_duration_us":6977,"operation_id":"0x713e","timeout_ms":30000,"last_server_duration_us":6977,"total_duration_us":14963095},{"operation_name":"get","last_dispatch_duration_us":13255245,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13255245,"total_server_duration_us":38,"operation_id":"0x726c","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":13257571},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18003,"operation_id":"0x7586","timeout_ms":10000,"total_duration_us":10008267},{"operation_name":"upsert","last_dispatch_duration_us":8954779,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8954779,"total_server_duration_us":21568,"encode_duration_us":20072,"operation_id":"0x74e5","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10006610},{"operation_name":"get","last_dispatch_duration_us":8556015,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8556015,"total_server_duration_us":126,"operation_id":"0x7527","timeout_ms":30000,"last_server_duration_us":126,"total_duration_us":8557831},{"operation_name":"upsert","last_dispatch_duration_us":3046361,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":3046361,"total_server_duration_us":72862,"encode_duration_us":20018,"operation_id":"0x7986","timeout_ms":10000,"last_server_duration_us":72862,"total_duration_us":3049860},{"operation_name":"upsert","last_dispatch_duration_us":2998173,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":2998173,"total_server_duration_us":2637253,"encode_duration_us":19222,"operation_id":"0x797d","timeout_ms":10000,"last_server_duration_us":2637253,"total_duration_us":3014932},{"operation_name":"get","last_dispatch_duration_us":2885621,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":2885621,"total_server_duration_us":38,"operation_id":"0x79a6","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":2885796},{"operation_name":"upsert","last_dispatch_duration_us":2011084,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2011084,"total_server_duration_us":96557,"encode_duration_us":18445,"operation_id":"0x78c5","timeout_ms":10000,"last_server_duration_us":96557,"total_duration_us":2018691},{"operation_name":"upsert","last_dispatch_duration_us":1928343,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1928343,"total_server_duration_us":61139,"encode_duration_us":19762,"operation_id":"0x78ed","timeout_ms":10000,"last_server_duration_us":61139,"total_duration_us":1932290}],"total_count":46}} INFO [2022-03-13 17:12:57,729] jepsen worker 17 - jepsen.util 2267 :ok :read [5 43] INFO [2022-03-13 17:12:57,729] jepsen worker 17 - jepsen.util 2267 :invoke :write [5 32] WARN [2022-03-13 17:12:57,733] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10086973,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10086973,"total_server_duration_us":49,"operation_id":"0x70da","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":10089253},{"operation_name":"upsert","last_dispatch_duration_us":11305462,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11305462,"total_server_duration_us":9026756,"encode_duration_us":17443,"operation_id":"0x7537","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10007712},{"operation_name":"upsert","last_dispatch_duration_us":9077880,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9077880,"total_server_duration_us":55303,"encode_duration_us":16718,"operation_id":"0x712d","timeout_ms":10000,"last_server_duration_us":55303,"total_duration_us":10004208},{"operation_name":"upsert","last_dispatch_duration_us":18114453,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":18114453,"total_server_duration_us":9062854,"encode_duration_us":17982,"operation_id":"0x7132","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10003024},{"operation_name":"upsert","last_dispatch_duration_us":7240475,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7240475,"total_server_duration_us":21082,"encode_duration_us":16669,"operation_id":"0x727f","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10002121},{"operation_name":"get","last_dispatch_duration_us":9145584,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9145584,"total_server_duration_us":1209,"operation_id":"0x7143","timeout_ms":30000,"last_server_duration_us":1209,"total_duration_us":9145837},{"operation_name":"get","last_dispatch_duration_us":9108661,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9108661,"total_server_duration_us":293,"operation_id":"0x7692","timeout_ms":30000,"last_server_duration_us":293,"total_duration_us":9109194},{"operation_name":"get","last_dispatch_duration_us":9086192,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9086192,"total_server_duration_us":19,"operation_id":"0x7699","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":9089300},{"operation_name":"upsert","last_dispatch_duration_us":8750643,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":8750643,"total_server_duration_us":78979,"encode_duration_us":30119,"operation_id":"0x76d0","timeout_ms":10000,"last_server_duration_us":78979,"total_duration_us":8764601},{"operation_name":"upsert","last_dispatch_duration_us":2720723,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2720723,"total_server_duration_us":115449,"encode_duration_us":33698,"operation_id":"0x769b","timeout_ms":10000,"last_server_duration_us":115449,"total_duration_us":2725595}],"total_count":21}} WARN [2022-03-13 17:12:57,823] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16124201,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16124201,"total_server_duration_us":38,"operation_id":"0x6e5d","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":16126795},{"operation_name":"get","last_dispatch_duration_us":12458930,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":12458930,"total_server_duration_us":15,"operation_id":"0x6fd4","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":12461521},{"operation_name":"get","last_dispatch_duration_us":10187275,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10187275,"total_server_duration_us":1373705,"operation_id":"0x70b3","timeout_ms":30000,"last_server_duration_us":1373705,"total_duration_us":10190009},{"operation_name":"upsert","last_dispatch_duration_us":168471,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9770161,"total_server_duration_us":149356,"encode_duration_us":23860,"operation_id":"0x7190","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10009574},{"operation_name":"upsert","last_dispatch_duration_us":11183164,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11183164,"total_server_duration_us":9047983,"encode_duration_us":18358,"operation_id":"0x750f","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10008069},{"operation_name":"upsert","last_dispatch_duration_us":8782589,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8782589,"total_server_duration_us":43727,"encode_duration_us":18304,"operation_id":"0x713b","timeout_ms":10000,"last_server_duration_us":43727,"total_duration_us":10007359},{"operation_name":"upsert","last_dispatch_duration_us":161427,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8540953,"total_server_duration_us":167122,"encode_duration_us":18092,"operation_id":"0x7268","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10004757},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":29783,"operation_id":"0x757d","timeout_ms":10000,"total_duration_us":10002560},{"operation_name":"get","last_dispatch_duration_us":9212876,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9212876,"total_server_duration_us":334,"operation_id":"0x717b","timeout_ms":30000,"last_server_duration_us":334,"total_duration_us":9213352},{"operation_name":"get","last_dispatch_duration_us":9194581,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9194581,"total_server_duration_us":49,"operation_id":"0x717d","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9197228}],"total_count":82}} WARN [2022-03-13 17:12:57,823] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17899701,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17899701,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9022513,"encode_duration_us":17080,"operation_id":"0x6d17","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10005590},{"operation_name":"upsert","last_dispatch_duration_us":11448559,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20103518,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9064790,"encode_duration_us":20872,"operation_id":"0x7060","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10009771},{"operation_name":"upsert","last_dispatch_duration_us":26473437,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26473437,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9030999,"encode_duration_us":17914,"operation_id":"0x6d5e","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10010369}],"total_count":3}} WARN [2022-03-13 17:12:57,834] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13038951,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13038951,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9056479,"encode_duration_us":16969,"operation_id":"0x6f4f","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":10000866},{"operation_name":"upsert","last_dispatch_duration_us":17772710,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17772710,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9035244,"encode_duration_us":22469,"operation_id":"0x6c91","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10004215},{"operation_name":"upsert","last_dispatch_duration_us":18114453,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18114453,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9062854,"encode_duration_us":17982,"operation_id":"0x7132","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10003024},{"operation_name":"upsert","last_dispatch_duration_us":11305462,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11305462,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9026756,"encode_duration_us":17443,"operation_id":"0x7537","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10007712}],"total_count":4}} INFO [2022-03-13 17:12:57,842] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,843] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:12:57,871] jepsen worker 56 - jepsen.util 2306 :info :write [18 33] :SyncWriteAmbiguous INFO [2022-03-13 17:12:57,871] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,872] jepsen worker 56 - jepsen.util 2396 :invoke :write [18 29] INFO [2022-03-13 17:12:57,873] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 36] INFO [2022-03-13 17:12:57,923] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:57,923] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:12:57,990] jepsen worker 85 - jepsen.util 2065 :info :write [28 13] :RequestTimeoutException INFO [2022-03-13 17:12:57,991] jepsen worker 85 - jepsen.util 2155 :invoke :write [28 48] INFO [2022-03-13 17:12:57,999] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,000] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:58,001] jepsen worker 22 - jepsen.util 652 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,002] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:12:58,003] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,007] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:12:58,039] jepsen worker 36 - jepsen.util 2106 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,040] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:58,042] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,042] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:58,048] jepsen worker 14 - jepsen.util 284 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,048] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:58,048] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,049] jepsen worker 14 - jepsen.util 284 :invoke :write [4 6] INFO [2022-03-13 17:12:58,050] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:12:58,050] jepsen worker 84 - jepsen.util 2424 :info :write [28 46] :RequestTimeoutException INFO [2022-03-13 17:12:58,050] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:58,051] jepsen worker 84 - jepsen.util 2514 :invoke :read [28 nil] INFO [2022-03-13 17:12:58,064] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,065] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:12:58,110] jepsen worker 40 - jepsen.util 2200 :info :write [13 36] :RequestTimeoutException INFO [2022-03-13 17:12:58,111] jepsen worker 40 - jepsen.util 2290 :invoke :write [13 20] INFO [2022-03-13 17:12:58,123] jepsen worker 60 - jepsen.util 600 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,123] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,123] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,123] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:12:58,124] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:12:58,124] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:58,129] jepsen worker 88 - jepsen.util 1978 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,130] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,140] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,140] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,141] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,142] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,142] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:58,143] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:12:58,143] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,144] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,144] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:58,145] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:12:58,145] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,146] jepsen worker 26 - jepsen.util 476 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,146] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,146] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:12:58,146] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:12:58,148] jepsen worker 53 - jepsen.util 503 :invoke :write [17 4] INFO [2022-03-13 17:12:58,149] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:58,150] jepsen worker 42 - jepsen.util 402 :invoke :write [14 41] INFO [2022-03-13 17:12:58,163] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,163] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,198] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:58,199] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,206] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,207] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,210] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:58,210] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,231] jepsen worker 12 - jepsen.util 732 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,231] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,231] jepsen worker 12 - jepsen.util 732 :invoke :write [4 6] INFO [2022-03-13 17:12:58,232] jepsen worker 76 - jepsen.util 706 :invoke :write [25 0] INFO [2022-03-13 17:12:58,242] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,242] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:12:58,252] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,253] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:12:58,263] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.5964912280701755}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:58,263] jepsen worker nemesis - couchbase.util Rebalance check count: 288 INFO [2022-03-13 17:12:58,263] jepsen worker nemesis - couchbase.util Rebalance stuck count: 191 INFO [2022-03-13 17:12:58,271] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,271] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,272] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:12:58,272] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,287] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,288] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,289] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,289] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,291] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,292] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:12:58,295] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:58,295] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,301] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:58,302] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,302] jepsen worker 61 - jepsen.util 601 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,303] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,303] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:12:58,303] jepsen worker 8 - jepsen.util 728 :invoke :write [2 32] INFO [2022-03-13 17:12:58,313] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,314] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:12:58,326] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,327] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:58,331] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,331] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:58,336] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,336] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,337] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:12:58,337] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 9] INFO [2022-03-13 17:12:58,337] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,338] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:12:58,344] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:58,344] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 12] INFO [2022-03-13 17:12:58,363] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,364] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,386] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:58,386] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,402] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:58,402] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:12:58,430] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,430] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:12:58,431] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:58,431] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,442] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,442] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,442] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 33] INFO [2022-03-13 17:12:58,443] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:12:58,482] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,482] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:12:58,485] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,485] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:58,488] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,488] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:12:58,489] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,489] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:58,495] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,496] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:12:58,526] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,527] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,540] jepsen worker 18 - jepsen.util 1998 :info :write [6 17] :SyncWriteAmbiguous INFO [2022-03-13 17:12:58,541] jepsen worker 18 - jepsen.util 2088 :invoke :write [6 31] INFO [2022-03-13 17:12:58,552] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:58,552] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:12:58,560] jepsen worker 15 - jepsen.util 2265 :info :write [5 19] :RequestTimeoutException INFO [2022-03-13 17:12:58,561] jepsen worker 3 - jepsen.util 1893 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,561] jepsen worker 15 - jepsen.util 2355 :invoke :read [5 nil] INFO [2022-03-13 17:12:58,562] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 48] INFO [2022-03-13 17:12:58,594] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,594] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,595] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:12:58,595] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:12:58,608] jepsen worker 44 - jepsen.util 584 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,608] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:58,610] jepsen worker 88 - jepsen.util 1978 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,610] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:12:58,642] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,643] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:12:58,643] jepsen worker 7 - jepsen.util 637 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,644] jepsen worker 7 - jepsen.util 637 :invoke :write [2 16] INFO [2022-03-13 17:12:58,682] jepsen worker 36 - jepsen.util 2106 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,682] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:58,714] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,715] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,723] jepsen worker 3 - jepsen.util 1893 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,723] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,750] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:58,750] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 40] INFO [2022-03-13 17:12:58,751] jepsen worker 27 - jepsen.util 2637 :ok :read [9 26] INFO [2022-03-13 17:12:58,752] jepsen worker 27 - jepsen.util 2637 :invoke :read [9 nil] INFO [2022-03-13 17:12:58,775] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,776] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:12:58,777] jepsen worker 27 - jepsen.util 2637 :ok :read [9 26] INFO [2022-03-13 17:12:58,777] jepsen worker 27 - jepsen.util 2637 :invoke :read [9 nil] INFO [2022-03-13 17:12:58,802] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,803] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:58,804] jepsen worker 27 - jepsen.util 2637 :ok :read [9 26] INFO [2022-03-13 17:12:58,805] jepsen worker 27 - jepsen.util 2637 :invoke :write [9 36] INFO [2022-03-13 17:12:58,833] jepsen worker 54 - jepsen.util 1854 :info :write [18 40] :RequestTimeoutException INFO [2022-03-13 17:12:58,834] jepsen worker 54 - jepsen.util 1944 :invoke :write [18 4] INFO [2022-03-13 17:12:58,836] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,837] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:12:58,843] jepsen worker 65 - jepsen.util 2585 :info :write [21 0] :RequestTimeoutException INFO [2022-03-13 17:12:58,844] jepsen worker 65 - jepsen.util 2675 :invoke :read [21 nil] INFO [2022-03-13 17:12:58,846] jepsen worker 65 - jepsen.util 2675 :ok :read [21 :nil] INFO [2022-03-13 17:12:58,846] jepsen worker 65 - jepsen.util 2675 :invoke :read [21 nil] INFO [2022-03-13 17:12:58,847] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,847] jepsen worker 23 - jepsen.util 563 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,847] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,847] jepsen worker 60 - jepsen.util 600 :invoke :write [20 21] INFO [2022-03-13 17:12:58,848] jepsen worker 23 - jepsen.util 563 :invoke :write [7 35] INFO [2022-03-13 17:12:58,848] jepsen worker 65 - jepsen.util 2675 :ok :read [21 :nil] INFO [2022-03-13 17:12:58,848] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:58,848] jepsen worker 65 - jepsen.util 2675 :invoke :write [21 34] INFO [2022-03-13 17:12:58,871] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,872] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,878] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,878] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,878] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,879] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:12:58,880] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,880] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:58,890] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,890] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,891] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,891] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,925] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:58,926] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:12:58,973] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,974] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:58,979] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,979] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,979] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,980] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:12:58,980] jepsen worker 89 - jepsen.util 2429 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,980] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:12:58,981] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,981] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:58,982] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,983] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,985] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,986] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:58,987] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:58,988] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:12:59,004] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,004] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,004] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,004] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,005] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,005] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:12:59,012] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,012] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,012] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,013] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 21] INFO [2022-03-13 17:12:59,013] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:12:59,014] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:12:59,037] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:59,037] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,045] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,045] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 8] INFO [2022-03-13 17:12:59,060] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:59,060] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:12:59,087] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,088] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:59,112] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,113] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,113] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,114] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:59,130] jepsen worker 6 - jepsen.util 636 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,130] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:12:59,130] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,131] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:12:59,131] jepsen worker 6 - jepsen.util 636 :invoke :write [2 43] INFO [2022-03-13 17:12:59,132] jepsen worker 62 - jepsen.util 512 :invoke :write [20 8] INFO [2022-03-13 17:12:59,162] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,162] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,167] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,168] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:12:59,186] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,187] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:12:59,195] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,195] jepsen worker 77 - jepsen.util 707 :invoke :write [25 37] INFO [2022-03-13 17:12:59,204] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,204] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,205] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:12:59,205] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:12:59,261] jepsen worker 88 - jepsen.util 1978 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,262] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,270] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6023391812865497}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:12:59,270] jepsen worker nemesis - couchbase.util Rebalance check count: 289 INFO [2022-03-13 17:12:59,270] jepsen worker nemesis - couchbase.util Rebalance stuck count: 191 INFO [2022-03-13 17:12:59,296] jepsen worker 14 - jepsen.util 284 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,297] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:12:59,297] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,297] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:12:59,300] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,300] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,300] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:12:59,301] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:59,302] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,302] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:12:59,315] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,315] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:12:59,351] jepsen worker 12 - jepsen.util 732 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,352] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:12:59,352] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,352] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,354] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,355] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:12:59,371] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,372] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,389] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:59,390] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,392] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,393] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:59,399] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,400] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,424] jepsen worker 72 - jepsen.util 2142 :info :write [24 1] :RequestTimeoutException INFO [2022-03-13 17:12:59,424] jepsen worker 72 - jepsen.util 2232 :invoke :write [24 29] INFO [2022-03-13 17:12:59,445] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:59,445] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 23] INFO [2022-03-13 17:12:59,454] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,454] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:12:59,467] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,467] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,469] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,470] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:12:59,482] jepsen worker 42 - jepsen.util 402 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,483] jepsen worker 42 - jepsen.util 402 :invoke :write [14 3] INFO [2022-03-13 17:12:59,488] jepsen worker 52 - jepsen.util 592 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,488] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:12:59,493] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,493] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,495] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:12:59,495] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:59,503] jepsen worker 88 - jepsen.util 1978 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,504] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:12:59,532] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,533] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,536] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,537] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:12:59,556] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,556] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:59,591] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:59,591] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,608] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,608] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,609] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:12:59,609] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:12:59,619] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,620] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,634] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:59,635] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:12:59,648] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:59,649] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 37] INFO [2022-03-13 17:12:59,672] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,672] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:12:59,682] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,682] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:12:59,685] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,686] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:12:59,714] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,715] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:12:59,731] jepsen worker 76 - jepsen.util 706 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,733] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,733] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:12:59,734] jepsen worker 76 - jepsen.util 706 :invoke :write [25 22] INFO [2022-03-13 17:12:59,734] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:12:59,735] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:12:59,741] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,741] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,741] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,741] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:12:59,742] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:12:59,742] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:59,747] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,747] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,747] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,747] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 19] INFO [2022-03-13 17:12:59,765] jepsen worker 88 - jepsen.util 1978 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,766] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 21] INFO [2022-03-13 17:12:59,820] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,821] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,823] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,825] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,826] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:59,827] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,827] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,828] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:12:59,835] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,835] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,836] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,837] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,838] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,839] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:12:59,876] jepsen worker 53 - jepsen.util 503 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,876] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,877] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:12:59,877] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,878] jepsen worker 88 - jepsen.util 1978 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,878] jepsen worker 61 - jepsen.util 601 :invoke :write [20 18] INFO [2022-03-13 17:12:59,879] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:12:59,883] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,884] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:12:59,885] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,886] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:12:59,886] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:12:59,887] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,889] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:12:59,905] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,905] jepsen worker 37 - jepsen.util 2017 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,906] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:12:59,907] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 45] INFO [2022-03-13 17:12:59,924] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:12:59,925] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:12:59,927] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:12:59,927] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:12:59,935] jepsen worker 23 - jepsen.util 563 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,936] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:12:59,969] jepsen worker 3 - jepsen.util 1893 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,969] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,971] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:12:59,971] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 30] INFO [2022-03-13 17:12:59,995] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:12:59,996] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:13:00,013] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:00,014] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:13:00,034] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,035] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:00,080] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,080] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:00,088] jepsen worker 8 - jepsen.util 728 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,089] jepsen worker 8 - jepsen.util 728 :invoke :write [2 43] INFO [2022-03-13 17:13:00,124] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,124] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:13:00,129] jepsen worker 3 - jepsen.util 1893 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,130] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:00,143] jepsen worker 34 - jepsen.util 754 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,144] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:00,173] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,174] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,174] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:13:00,174] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 48] INFO [2022-03-13 17:13:00,204] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,205] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,206] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,206] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,210] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,211] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,212] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,213] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,214] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,215] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,216] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,217] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,218] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:13:00,222] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,229] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,229] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:00,256] jepsen worker 36 - jepsen.util 2106 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,257] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 29] INFO [2022-03-13 17:13:00,277] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6023391812865497}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:00,277] jepsen worker nemesis - couchbase.util Rebalance check count: 290 INFO [2022-03-13 17:13:00,277] jepsen worker nemesis - couchbase.util Rebalance stuck count: 192 INFO [2022-03-13 17:13:00,292] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,292] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:00,292] jepsen worker 60 - jepsen.util 600 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,293] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:00,294] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,294] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,294] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:00,295] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:13:00,295] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,296] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:13:00,314] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,315] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,315] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:13:00,315] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,322] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,323] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,333] jepsen worker 7 - jepsen.util 637 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,334] jepsen worker 7 - jepsen.util 637 :invoke :write [2 7] INFO [2022-03-13 17:13:00,353] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:00,353] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 10] INFO [2022-03-13 17:13:00,359] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,360] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,399] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,399] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:13:00,402] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,402] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:13:00,435] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,436] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:13:00,465] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:00,465] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,466] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:00,467] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:13:00,469] jepsen worker 88 - jepsen.util 1978 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,470] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,484] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,484] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:13:00,505] jepsen worker 37 - jepsen.util 2017 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,506] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,516] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:00,517] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,542] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,543] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:13:00,546] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:00,548] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,566] jepsen worker 36 - jepsen.util 2106 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,567] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,567] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 30] INFO [2022-03-13 17:13:00,567] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:00,595] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:00,595] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 23] INFO [2022-03-13 17:13:00,598] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:00,598] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,613] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,614] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,615] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:00,615] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,643] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:00,643] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,653] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,653] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,654] jepsen worker 36 - jepsen.util 2106 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,655] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:13:00,656] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:13:00,657] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 23] INFO [2022-03-13 17:13:00,659] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,660] jepsen worker 13 - jepsen.util 643 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,660] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:13:00,660] jepsen worker 13 - jepsen.util 643 :invoke :write [4 47] INFO [2022-03-13 17:13:00,666] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,667] jepsen worker 77 - jepsen.util 707 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,667] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:00,668] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,668] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,668] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:00,668] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:00,669] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:00,669] jepsen worker 51 - jepsen.util 591 :invoke :write [17 33] INFO [2022-03-13 17:13:00,670] jepsen worker 12 - jepsen.util 732 :invoke :write [4 20] INFO [2022-03-13 17:13:00,695] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:00,695] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,698] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,699] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:00,731] jepsen worker 6 - jepsen.util 636 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,731] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:00,732] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,732] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,732] jepsen worker 6 - jepsen.util 636 :invoke :write [2 42] INFO [2022-03-13 17:13:00,734] jepsen worker 26 - jepsen.util 476 :invoke :write [8 19] INFO [2022-03-13 17:13:00,736] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:00,736] jepsen worker 43 - jepsen.util 493 :invoke :write [14 29] INFO [2022-03-13 17:13:00,777] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,778] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:00,793] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,794] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:13:00,801] jepsen worker 88 - jepsen.util 1978 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,802] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,813] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:00,814] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 40] INFO [2022-03-13 17:13:00,846] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:00,846] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:13:00,889] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,890] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:13:00,892] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,893] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,903] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,904] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,905] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,906] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:00,907] jepsen worker 36 - jepsen.util 2106 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,908] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 0] INFO [2022-03-13 17:13:00,913] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,914] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:13:00,949] jepsen worker 80 - jepsen.util 710 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,950] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,950] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:00,950] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,950] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:00,951] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:00,955] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,955] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,956] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:00,957] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:00,963] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,966] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:00,970] jepsen worker 32 - jepsen.util 2102 :info :write [10 34] :RequestTimeoutException INFO [2022-03-13 17:13:00,971] jepsen worker 32 - jepsen.util 2192 :invoke :write [10 2] INFO [2022-03-13 17:13:00,998] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:00,999] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:13:01,020] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:01,020] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:13:01,021] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,021] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,022] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:13:01,022] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 12] INFO [2022-03-13 17:13:01,056] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,057] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:13:01,091] jepsen worker 89 - jepsen.util 2429 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,092] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,106] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,107] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,117] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,117] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:13:01,121] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:01,121] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,152] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,152] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,152] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,152] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:01,153] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,153] jepsen worker 36 - jepsen.util 2106 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,153] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:13:01,154] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 11] INFO [2022-03-13 17:13:01,155] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,156] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,193] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,193] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,195] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:01,196] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:13:01,197] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,198] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:13:01,254] jepsen worker 45 - jepsen.util 2565 :ok :read [15 48] INFO [2022-03-13 17:13:01,255] jepsen worker 45 - jepsen.util 2565 :invoke :write [15 10] INFO [2022-03-13 17:13:01,255] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,257] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:13:01,257] jepsen worker 37 - jepsen.util 2017 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,258] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,258] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,259] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,263] jepsen worker 74 - jepsen.util 2684 :ok :read [24 24] INFO [2022-03-13 17:13:01,263] jepsen worker 74 - jepsen.util 2684 :invoke :read [24 nil] INFO [2022-03-13 17:13:01,269] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,270] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 30] INFO [2022-03-13 17:13:01,272] jepsen worker 30 - jepsen.util 2370 :ok :read [10 :nil] INFO [2022-03-13 17:13:01,273] jepsen worker 30 - jepsen.util 2370 :invoke :write [10 29] INFO [2022-03-13 17:13:01,283] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6023391812865497}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:01,283] jepsen worker nemesis - couchbase.util Rebalance check count: 291 INFO [2022-03-13 17:13:01,283] jepsen worker nemesis - couchbase.util Rebalance stuck count: 193 INFO [2022-03-13 17:13:01,287] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:01,288] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:13:01,300] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:01,302] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:13:01,312] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,313] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:13:01,316] jepsen worker 29 - jepsen.util 2459 :info :write [9 5] :RequestTimeoutException INFO [2022-03-13 17:13:01,316] jepsen worker 29 - jepsen.util 2549 :invoke :read [9 nil] INFO [2022-03-13 17:13:01,344] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,344] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,351] jepsen worker 48 - jepsen.util 2208 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:13:01,352] jepsen worker 48 - jepsen.util 2208 :invoke :write [16 40] INFO [2022-03-13 17:13:01,356] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,356] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,358] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,359] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,359] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,360] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,360] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:01,360] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,361] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,362] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,363] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,363] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,370] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,371] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,377] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,377] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,407] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:01,407] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:13:01,409] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,410] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,420] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,421] jepsen worker 76 - jepsen.util 706 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,422] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,423] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:01,423] jepsen worker 3 - jepsen.util 1893 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,424] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,429] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,429] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,431] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,432] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,448] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,449] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,450] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,450] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,451] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,452] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,453] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,454] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:13:01,458] jepsen worker 78 - jepsen.util 528 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,458] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,458] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,459] jepsen worker 78 - jepsen.util 528 :invoke :write [26 24] INFO [2022-03-13 17:13:01,459] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:01,460] jepsen worker 22 - jepsen.util 652 :invoke :write [7 12] INFO [2022-03-13 17:13:01,460] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,461] jepsen worker 52 - jepsen.util 592 :invoke :write [17 5] INFO [2022-03-13 17:13:01,493] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,493] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,493] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,494] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,494] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:13:01,495] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,495] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,495] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:13:01,501] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,501] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,513] jepsen worker 36 - jepsen.util 2106 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,513] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,514] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,514] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,514] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:13:01,515] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 33] INFO [2022-03-13 17:13:01,523] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,524] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,525] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,527] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:13:01,539] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:01,540] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:13:01,548] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:01,549] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:13:01,591] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,591] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:13:01,610] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,610] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:13:01,625] jepsen worker 70 - jepsen.util 700 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,625] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,625] jepsen worker 70 - jepsen.util 700 :invoke :write [23 38] INFO [2022-03-13 17:13:01,628] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:01,630] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,631] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,639] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:01,640] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,654] jepsen worker 37 - jepsen.util 2017 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,655] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:13:01,681] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:01,681] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 0] INFO [2022-03-13 17:13:01,703] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,704] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,718] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,719] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,720] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,720] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,721] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,722] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:01,733] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,734] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,766] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:01,767] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,780] jepsen worker 88 - jepsen.util 1978 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,780] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,780] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,780] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,781] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:13:01,782] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:01,783] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:13:01,783] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:13:01,833] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,833] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:01,859] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,860] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:01,873] jepsen worker 47 - jepsen.util 2297 :info :write [15 42] :SyncWriteAmbiguous INFO [2022-03-13 17:13:01,874] jepsen worker 47 - jepsen.util 2387 :invoke :read [15 nil] INFO [2022-03-13 17:13:01,878] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:01,879] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:13:01,910] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,910] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:13:01,915] jepsen worker 62 - jepsen.util 512 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,915] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:01,922] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,922] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,923] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:13:01,923] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:01,927] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,928] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:13:01,929] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,930] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:01,951] jepsen worker 13 - jepsen.util 643 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,951] jepsen worker 22 - jepsen.util 652 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,951] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,952] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:01,953] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,954] jepsen worker 13 - jepsen.util 643 :invoke :write [4 9] INFO [2022-03-13 17:13:01,954] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,956] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:01,957] jepsen worker 80 - jepsen.util 710 :invoke :write [26 41] INFO [2022-03-13 17:13:01,958] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:13:01,981] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:01,981] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:01,989] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:01,989] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,022] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,022] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:02,041] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,041] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:13:02,050] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:02,050] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,068] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,068] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,069] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,069] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,105] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:02,106] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:13:02,106] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,106] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:13:02,121] jepsen worker 78 - jepsen.util 528 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,121] jepsen worker 78 - jepsen.util 528 :invoke :write [26 19] INFO [2022-03-13 17:13:02,136] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,136] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,137] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:13:02,137] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,163] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,163] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,165] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,165] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,167] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,168] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:13:02,198] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,198] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,198] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,199] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:13:02,199] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 6] INFO [2022-03-13 17:13:02,200] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,205] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,206] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:13:02,213] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,213] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,213] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:13:02,221] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:13:02,231] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,231] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:13:02,289] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:02,290] jepsen worker nemesis - couchbase.util Rebalance check count: 292 INFO [2022-03-13 17:13:02,290] jepsen worker nemesis - couchbase.util Rebalance stuck count: 193 INFO [2022-03-13 17:13:02,294] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,294] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,307] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,308] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,320] jepsen worker 9 - jepsen.util 2079 :info :write [3 47] :RequestTimeoutException INFO [2022-03-13 17:13:02,321] jepsen worker 9 - jepsen.util 2169 :invoke :read [3 nil] INFO [2022-03-13 17:13:02,324] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,324] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,324] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:02,325] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,325] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:02,325] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:02,325] jepsen worker 44 - jepsen.util 584 :invoke :write [14 23] INFO [2022-03-13 17:13:02,326] jepsen worker 77 - jepsen.util 707 :invoke :write [25 13] INFO [2022-03-13 17:13:02,327] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,327] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,328] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,328] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:02,330] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,330] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:13:02,350] jepsen worker 55 - jepsen.util 2125 :info :write [18 46] :RequestTimeoutException INFO [2022-03-13 17:13:02,351] jepsen worker 55 - jepsen.util 2215 :invoke :read [18 nil] INFO [2022-03-13 17:13:02,357] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:02,358] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,385] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,386] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:13:02,388] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,389] jepsen worker 36 - jepsen.util 2106 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,389] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,390] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,390] jepsen worker 55 - jepsen.util 2215 :ok :read [18 21] INFO [2022-03-13 17:13:02,391] jepsen worker 55 - jepsen.util 2215 :invoke :read [18 nil] INFO [2022-03-13 17:13:02,405] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,406] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,406] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:02,407] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:13:02,423] jepsen worker 55 - jepsen.util 2215 :ok :read [18 21] INFO [2022-03-13 17:13:02,424] jepsen worker 55 - jepsen.util 2215 :invoke :write [18 45] INFO [2022-03-13 17:13:02,431] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:02,432] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:13:02,436] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:02,437] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 37] INFO [2022-03-13 17:13:02,472] jepsen worker 57 - jepsen.util 2307 :ok :read [19 36] INFO [2022-03-13 17:13:02,473] jepsen worker 57 - jepsen.util 2307 :invoke :read [19 nil] INFO [2022-03-13 17:13:02,484] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,484] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,491] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,491] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,492] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:02,496] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,516] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:02,516] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:13:02,557] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,558] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,558] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,558] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:13:02,577] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,578] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:13:02,608] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,608] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,608] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,609] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,609] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,610] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,622] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,622] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,623] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:13:02,623] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,694] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,694] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:02,697] jepsen worker 89 - jepsen.util 2429 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,697] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,700] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,700] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,700] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,701] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:13:02,701] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,701] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,702] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,703] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,704] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,705] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 0] INFO [2022-03-13 17:13:02,706] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,706] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,711] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,714] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 19] INFO [2022-03-13 17:13:02,729] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:02,729] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:13:02,734] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:02,734] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,752] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,752] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,752] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,753] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:13:02,753] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,754] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,755] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,756] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:13:02,766] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:02,767] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:02,782] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,782] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,783] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,784] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:02,788] jepsen worker 52 - jepsen.util 592 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,788] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,790] jepsen worker 34 - jepsen.util 754 :invoke :write [11 34] INFO [2022-03-13 17:13:02,790] jepsen worker 52 - jepsen.util 592 :invoke :write [17 40] INFO [2022-03-13 17:13:02,799] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,799] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:13:02,800] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,800] jepsen worker 42 - jepsen.util 402 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,800] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:02,801] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:02,812] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,815] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:13:02,831] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,833] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:02,843] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:02,844] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,874] jepsen worker 70 - jepsen.util 700 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,875] jepsen worker 70 - jepsen.util 700 :invoke :write [23 29] INFO [2022-03-13 17:13:02,895] jepsen worker 12 - jepsen.util 732 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,896] jepsen worker 12 - jepsen.util 732 :invoke :write [4 48] INFO [2022-03-13 17:13:02,899] jepsen worker 36 - jepsen.util 2106 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,900] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:02,907] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,908] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:02,910] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,910] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,911] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:13:02,912] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:02,915] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,916] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:02,944] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:02,944] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,975] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,975] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:13:02,979] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,979] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:02,992] jepsen worker 61 - jepsen.util 601 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:02,993] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:13:03,000] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,001] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:13:03,005] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:03,005] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,040] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,040] jepsen worker 51 - jepsen.util 591 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,040] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,041] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:03,041] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,041] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:03,042] jepsen worker 60 - jepsen.util 600 :invoke :write [20 31] INFO [2022-03-13 17:13:03,043] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,043] jepsen worker 76 - jepsen.util 706 :invoke :write [25 42] INFO [2022-03-13 17:13:03,044] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:03,044] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,060] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,062] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,063] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:13:03,074] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,074] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,075] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:13:03,076] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:13:03,079] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:03,080] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 13] INFO [2022-03-13 17:13:03,097] jepsen worker 37 - jepsen.util 2017 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,098] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:13:03,138] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:03,138] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,140] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,140] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:13:03,156] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,157] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,160] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,161] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:03,196] jepsen worker 8 - jepsen.util 728 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,196] jepsen worker 8 - jepsen.util 728 :invoke :write [2 48] INFO [2022-03-13 17:13:03,202] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,202] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,203] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:13:03,204] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 34] INFO [2022-03-13 17:13:03,209] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,210] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 5] INFO [2022-03-13 17:13:03,233] jepsen worker 88 - jepsen.util 1978 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,233] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 38] INFO [2022-03-13 17:13:03,235] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:03,236] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 19] INFO [2022-03-13 17:13:03,276] jepsen worker 31 - jepsen.util 2371 :info :write [10 45] :RequestTimeoutException INFO [2022-03-13 17:13:03,277] jepsen worker 31 - jepsen.util 2461 :invoke :write [10 35] INFO [2022-03-13 17:13:03,288] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,288] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:03,295] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:03,295] jepsen worker nemesis - couchbase.util Rebalance check count: 293 INFO [2022-03-13 17:13:03,295] jepsen worker nemesis - couchbase.util Rebalance stuck count: 194 INFO [2022-03-13 17:13:03,307] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,307] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,309] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,310] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,340] jepsen worker 19 - jepsen.util 2179 :ok :read [6 16] INFO [2022-03-13 17:13:03,341] jepsen worker 19 - jepsen.util 2179 :invoke :write [6 45] INFO [2022-03-13 17:13:03,371] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,372] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,372] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:03,372] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:13:03,385] jepsen worker 77 - jepsen.util 707 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,386] jepsen worker 77 - jepsen.util 707 :invoke :write [25 48] INFO [2022-03-13 17:13:03,390] jepsen worker 88 - jepsen.util 1978 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,391] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:13:03,400] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,400] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:03,407] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,407] jepsen worker 7 - jepsen.util 637 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,407] jepsen worker 7 - jepsen.util 637 :invoke :write [2 6] INFO [2022-03-13 17:13:03,408] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:03,441] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,441] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,450] jepsen worker 44 - jepsen.util 584 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,450] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,451] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:03,452] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,453] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,454] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,455] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,456] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,457] jepsen worker 3 - jepsen.util 1893 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,457] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,458] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:13:03,459] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:03,474] jepsen worker 89 - jepsen.util 2429 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,474] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,493] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:03,494] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,512] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,512] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:13:03,522] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:03,523] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:13:03,567] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,567] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:13:03,568] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,568] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:13:03,582] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,582] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,584] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,584] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:13:03,586] jepsen worker 36 - jepsen.util 2106 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,586] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,586] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,587] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:03,588] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,588] jepsen worker 13 - jepsen.util 643 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,588] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,589] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:03,591] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,591] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:03,596] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,596] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,596] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,596] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,596] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,597] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:13:03,598] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,599] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:03,600] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:03,604] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,605] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,605] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 11] INFO [2022-03-13 17:13:03,607] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:13:03,610] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:03,610] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:03,610] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,636] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,637] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,653] jepsen worker 67 - jepsen.util 1867 :ok :read [22 30] INFO [2022-03-13 17:13:03,654] jepsen worker 67 - jepsen.util 1867 :invoke :write [22 9] INFO [2022-03-13 17:13:03,664] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:03,665] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,683] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,684] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,699] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,699] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:03,707] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,707] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:13:03,721] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,722] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:13:03,724] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:03,726] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,746] jepsen worker 86 - jepsen.util 2066 :ok :read [28 44] INFO [2022-03-13 17:13:03,747] jepsen worker 86 - jepsen.util 2066 :invoke :read [28 nil] INFO [2022-03-13 17:13:03,757] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:03,758] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,783] jepsen worker 1 - jepsen.util 2071 :ok :read [0 41] INFO [2022-03-13 17:13:03,784] jepsen worker 1 - jepsen.util 2071 :invoke :write [0 31] INFO [2022-03-13 17:13:03,786] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,787] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:13:03,788] jepsen worker 39 - jepsen.util 2199 :ok :read [13 39] INFO [2022-03-13 17:13:03,789] jepsen worker 39 - jepsen.util 2199 :invoke :write [13 36] INFO [2022-03-13 17:13:03,798] jepsen worker 50 - jepsen.util 2390 :ok :read [16 :nil] INFO [2022-03-13 17:13:03,800] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,801] jepsen worker 6 - jepsen.util 636 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,801] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,803] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,807] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,807] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,807] jepsen worker 6 - jepsen.util 636 :invoke :write [2 28] INFO [2022-03-13 17:13:03,811] jepsen worker 50 - jepsen.util 2390 :invoke :write [16 10] INFO [2022-03-13 17:13:03,811] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,812] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:03,814] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:03,814] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:03,814] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,815] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:03,815] jepsen worker 2 - jepsen.util 2612 :ok :read [0 41] INFO [2022-03-13 17:13:03,815] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:13:03,816] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,816] jepsen worker 2 - jepsen.util 2612 :invoke :read [0 nil] INFO [2022-03-13 17:13:03,817] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:03,817] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,818] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,820] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:03,822] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:13:03,843] jepsen worker 81 - jepsen.util 2601 :ok :read [27 4] INFO [2022-03-13 17:13:03,844] jepsen worker 81 - jepsen.util 2601 :invoke :write [27 35] INFO [2022-03-13 17:13:03,856] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:03,857] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,861] jepsen worker 15 - jepsen.util 2355 :ok :read [5 43] INFO [2022-03-13 17:13:03,862] jepsen worker 15 - jepsen.util 2355 :invoke :write [5 37] INFO [2022-03-13 17:13:03,885] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:03,886] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:13:03,892] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,893] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:03,903] jepsen worker 86 - jepsen.util 2066 :ok :read [28 44] INFO [2022-03-13 17:13:03,903] jepsen worker 86 - jepsen.util 2066 :invoke :read [28 nil] INFO [2022-03-13 17:13:03,907] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:03,908] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,912] jepsen worker 2 - jepsen.util 2612 :ok :read [0 41] INFO [2022-03-13 17:13:03,913] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,913] jepsen worker 2 - jepsen.util 2612 :invoke :read [0 nil] INFO [2022-03-13 17:13:03,913] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:03,941] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,942] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:03,970] jepsen worker 36 - jepsen.util 2106 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,971] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 32] INFO [2022-03-13 17:13:03,978] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,979] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:13:03,978] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,987] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:03,989] jepsen worker 86 - jepsen.util 2066 :ok :read [28 44] INFO [2022-03-13 17:13:03,989] jepsen worker 86 - jepsen.util 2066 :invoke :read [28 nil] INFO [2022-03-13 17:13:03,992] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:03,993] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:13:04,004] jepsen worker 12 - jepsen.util 732 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,005] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,005] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,005] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:04,006] jepsen worker 12 - jepsen.util 732 :invoke :write [4 12] INFO [2022-03-13 17:13:04,006] jepsen worker 51 - jepsen.util 591 :invoke :write [17 27] INFO [2022-03-13 17:13:04,012] jepsen worker 80 - jepsen.util 710 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,012] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:04,014] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,015] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:13:04,016] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:04,016] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 8] INFO [2022-03-13 17:13:04,041] jepsen worker 2 - jepsen.util 2612 :ok :read [0 41] INFO [2022-03-13 17:13:04,041] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:04,041] jepsen worker 2 - jepsen.util 2612 :invoke :read [0 nil] INFO [2022-03-13 17:13:04,042] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:13:04,047] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,048] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 48] INFO [2022-03-13 17:13:04,060] jepsen worker 43 - jepsen.util 493 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,061] jepsen worker 43 - jepsen.util 493 :invoke :write [14 39] INFO [2022-03-13 17:13:04,069] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,069] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,070] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:04,071] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,103] jepsen worker 86 - jepsen.util 2066 :ok :read [28 44] INFO [2022-03-13 17:13:04,103] jepsen worker 86 - jepsen.util 2066 :invoke :write [28 2] INFO [2022-03-13 17:13:04,126] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,127] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,130] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,130] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:04,139] jepsen worker 2 - jepsen.util 2612 :ok :read [0 41] INFO [2022-03-13 17:13:04,140] jepsen worker 2 - jepsen.util 2612 :invoke :write [0 47] INFO [2022-03-13 17:13:04,174] jepsen worker 88 - jepsen.util 1978 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,175] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,192] jepsen worker 89 - jepsen.util 2429 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,192] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,192] jepsen worker 36 - jepsen.util 2106 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,192] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,193] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,193] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,193] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:13:04,194] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,207] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,208] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:04,209] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,209] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,209] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:13:04,210] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,211] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,211] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:04,212] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:13:04,212] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,219] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:04,219] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,231] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,231] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:04,235] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,236] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:13:04,239] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,240] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:04,252] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:04,252] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:04,259] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,261] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 11] INFO [2022-03-13 17:13:04,262] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:04,262] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:13:04,276] jepsen worker 20 - jepsen.util 2630 :info :write [6 49] :RequestTimeoutException INFO [2022-03-13 17:13:04,276] jepsen worker 20 - jepsen.util 2720 :invoke :read [6 nil] INFO [2022-03-13 17:13:04,300] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:04,301] jepsen worker nemesis - couchbase.util Rebalance check count: 294 INFO [2022-03-13 17:13:04,301] jepsen worker nemesis - couchbase.util Rebalance stuck count: 195 INFO [2022-03-13 17:13:04,326] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,326] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:04,329] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,330] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:04,334] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,335] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:13:04,365] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,365] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,365] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:13:04,366] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,418] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,418] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,419] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:13:04,419] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:13:04,446] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,447] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:13:04,455] jepsen worker 78 - jepsen.util 528 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,456] jepsen worker 78 - jepsen.util 528 :invoke :write [26 0] INFO [2022-03-13 17:13:04,462] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,463] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,464] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,465] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 38] INFO [2022-03-13 17:13:04,478] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,480] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:04,485] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,486] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:04,497] jepsen worker 76 - jepsen.util 706 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,499] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:04,514] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:04,514] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:13:04,524] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,524] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,525] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 21] INFO [2022-03-13 17:13:04,525] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:04,543] jepsen worker 37 - jepsen.util 2017 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,544] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:04,555] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,555] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,556] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:13:04,557] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:04,564] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,564] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:04,570] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,571] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:04,572] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,573] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:13:04,576] jepsen worker 34 - jepsen.util 754 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,576] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:04,583] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,584] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,609] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,609] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,610] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,610] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:13:04,612] jepsen worker 88 - jepsen.util 1978 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,615] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,626] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:04,627] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,649] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:04,650] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,652] jepsen worker 26 - jepsen.util 476 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,652] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,652] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:04,652] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,652] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,653] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:04,654] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:04,655] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:13:04,656] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:13:04,657] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:04,662] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,663] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:13:04,694] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,695] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:04,706] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,707] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:04,707] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,708] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:04,711] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:04,712] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:13:04,735] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:04,735] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 24] INFO [2022-03-13 17:13:04,737] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,737] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,738] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,739] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:13:04,748] jepsen worker 36 - jepsen.util 2106 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,748] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,749] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 21] INFO [2022-03-13 17:13:04,749] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 8] INFO [2022-03-13 17:13:04,767] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,768] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:13:04,798] jepsen worker 88 - jepsen.util 1978 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,799] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,821] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:04,822] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:04,831] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,832] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:04,853] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,853] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,854] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:13:04,855] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:04,864] jepsen worker 52 - jepsen.util 592 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,865] jepsen worker 52 - jepsen.util 592 :invoke :write [17 7] INFO [2022-03-13 17:13:04,865] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:04,871] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 6] INFO [2022-03-13 17:13:04,880] jepsen worker 60 - jepsen.util 600 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,881] jepsen worker 60 - jepsen.util 600 :invoke :write [20 35] INFO [2022-03-13 17:13:04,903] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,904] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:13:04,905] jepsen worker 70 - jepsen.util 700 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,905] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,905] jepsen worker 70 - jepsen.util 700 :invoke :write [23 41] INFO [2022-03-13 17:13:04,906] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:04,910] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,911] jepsen worker 80 - jepsen.util 710 :invoke :write [26 33] INFO [2022-03-13 17:13:04,911] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,912] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:04,920] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,921] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:13:04,920] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,924] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:13:04,971] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,972] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 6] INFO [2022-03-13 17:13:04,987] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:04,988] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:13:05,003] jepsen worker 88 - jepsen.util 1978 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,004] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,007] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,011] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,049] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:05,049] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:13:05,076] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,077] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:13:05,079] jepsen worker 8 - jepsen.util 728 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,079] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,079] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,080] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,080] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:05,081] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:13:05,083] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:13:05,087] jepsen worker 14 - jepsen.util 284 :invoke :write [4 1] INFO [2022-03-13 17:13:05,108] jepsen worker 36 - jepsen.util 2106 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,109] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 40] INFO [2022-03-13 17:13:05,124] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,124] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:05,128] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,129] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:13:05,160] jepsen worker 78 - jepsen.util 528 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,160] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,161] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:05,161] jepsen worker 34 - jepsen.util 754 :invoke :write [11 36] INFO [2022-03-13 17:13:05,226] jepsen worker 3 - jepsen.util 1893 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,227] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:13:05,260] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,260] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,280] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:05,281] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:13:05,282] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,283] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:13:05,284] jepsen worker 77 - jepsen.util 707 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,284] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:05,284] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,285] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:13:05,285] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:05,286] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:05,308] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:05,308] jepsen worker 37 - jepsen.util 2017 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,308] jepsen worker nemesis - couchbase.util Rebalance check count: 295 INFO [2022-03-13 17:13:05,309] jepsen worker nemesis - couchbase.util Rebalance stuck count: 196 INFO [2022-03-13 17:13:05,309] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:13:05,312] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:05,313] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,333] jepsen worker 7 - jepsen.util 637 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,333] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,334] jepsen worker 62 - jepsen.util 512 :invoke :write [20 8] INFO [2022-03-13 17:13:05,334] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:05,374] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,375] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,395] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,396] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:13:05,418] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:05,419] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,419] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:13:05,420] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:05,428] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,428] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:05,445] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:05,446] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,469] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:05,469] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:13:05,473] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,473] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,480] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,481] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,484] jepsen worker 52 - jepsen.util 592 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,484] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,484] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:05,485] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:05,487] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,488] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:05,517] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,517] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,518] jepsen worker 25 - jepsen.util 565 :invoke :write [8 3] INFO [2022-03-13 17:13:05,518] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:13:05,526] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,527] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:05,534] jepsen worker 12 - jepsen.util 732 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,534] jepsen worker 12 - jepsen.util 732 :invoke :write [4 8] INFO [2022-03-13 17:13:05,561] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,562] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:13:05,593] jepsen worker 36 - jepsen.util 2106 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,594] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:05,614] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,615] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,615] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:13:05,617] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,655] jepsen worker 82 - jepsen.util 2152 :info :write [27 43] :SyncWriteAmbiguous INFO [2022-03-13 17:13:05,656] jepsen worker 21 - jepsen.util 741 :info :write [7 5] :RequestTimeoutException INFO [2022-03-13 17:13:05,656] jepsen worker 82 - jepsen.util 2242 :invoke :write [27 36] INFO [2022-03-13 17:13:05,657] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:13:05,668] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,668] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,668] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,669] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:13:05,676] jepsen worker 41 - jepsen.util 2111 :info :write [13 5] :RequestTimeoutException INFO [2022-03-13 17:13:05,677] jepsen worker 41 - jepsen.util 2201 :invoke :read [13 nil] INFO [2022-03-13 17:13:05,683] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,684] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:13:05,687] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,687] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,695] jepsen worker 66 - jepsen.util 2046 :info :write [22 21] :RequestTimeoutException INFO [2022-03-13 17:13:05,695] jepsen worker 66 - jepsen.util 2136 :invoke :write [22 49] INFO [2022-03-13 17:13:05,710] jepsen worker 6 - jepsen.util 636 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,711] jepsen worker 6 - jepsen.util 636 :invoke :write [2 9] INFO [2022-03-13 17:13:05,715] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,716] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:05,717] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,718] jepsen worker 22 - jepsen.util 652 :invoke :write [7 0] INFO [2022-03-13 17:13:05,730] jepsen worker 25 - jepsen.util 565 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,731] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:13:05,748] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,751] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:05,756] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,757] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:05,758] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,758] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:05,759] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,760] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:13:05,765] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:05,766] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,782] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,782] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,791] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:05,791] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 36] INFO [2022-03-13 17:13:05,796] jepsen worker 79 - jepsen.util 979 :info :write [26 33] :RequestTimeoutException INFO [2022-03-13 17:13:05,797] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:05,799] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,799] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,799] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,800] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,800] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 44] INFO [2022-03-13 17:13:05,801] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 45] INFO [2022-03-13 17:13:05,810] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,811] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,814] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,814] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:13:05,827] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:05,828] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,840] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,841] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:05,881] jepsen worker 88 - jepsen.util 1978 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,882] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 24] INFO [2022-03-13 17:13:05,908] jepsen worker 51 - jepsen.util 591 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,909] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:05,911] jepsen worker 70 - jepsen.util 700 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,912] jepsen worker 70 - jepsen.util 700 :invoke :write [23 1] INFO [2022-03-13 17:13:05,913] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,914] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:13:05,915] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,915] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,915] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,916] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,924] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,925] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:05,928] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,928] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,929] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:13:05,930] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,930] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,931] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,931] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:05,932] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:05,933] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,934] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:13:05,946] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:05,946] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:05,952] jepsen worker 88 - jepsen.util 1978 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:05,952] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:13:05,985] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:05,986] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,002] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,003] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:13:06,031] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,032] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:13:06,065] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,066] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,083] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,084] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:13:06,122] jepsen worker 22 - jepsen.util 652 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,123] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:06,124] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,124] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:13:06,126] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,127] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,127] jepsen worker 43 - jepsen.util 493 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,127] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,127] jepsen worker 61 - jepsen.util 601 :invoke :write [20 49] INFO [2022-03-13 17:13:06,131] jepsen worker 26 - jepsen.util 476 :invoke :write [8 41] INFO [2022-03-13 17:13:06,131] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:06,132] jepsen worker 8 - jepsen.util 728 :invoke :write [2 3] INFO [2022-03-13 17:13:06,132] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,132] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:06,133] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:06,134] jepsen worker 53 - jepsen.util 503 :invoke :write [17 40] WARN [2022-03-13 17:13:06,140] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":34100424,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":34100424,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9057542,"encode_duration_us":26284,"operation_id":"0x6ca9","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10003875},{"operation_name":"get","last_dispatch_duration_us":32109629,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":32109629,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":27,"operation_id":"0x6d7c","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":30008496}],"total_count":2}} WARN [2022-03-13 17:13:06,140] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":32109629,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":32109629,"total_server_duration_us":27,"operation_id":"0x6d7c","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":30008496},{"operation_name":"get","last_dispatch_duration_us":18094232,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":18094232,"total_server_duration_us":109047,"operation_id":"0x7148","timeout_ms":30000,"last_server_duration_us":109047,"total_duration_us":24283726},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":31502,"operation_id":"0x7643","timeout_ms":10000,"total_duration_us":10007560},{"operation_name":"upsert","last_dispatch_duration_us":9089291,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9661970,"total_server_duration_us":62860,"encode_duration_us":19011,"operation_id":"0x7926","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10006771},{"operation_name":"upsert","last_dispatch_duration_us":9096232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15408697,"total_server_duration_us":59488,"encode_duration_us":18554,"operation_id":"0x7540","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10004316},{"operation_name":"upsert","last_dispatch_duration_us":9107212,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9525560,"total_server_duration_us":39777,"encode_duration_us":29261,"operation_id":"0x7928","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10003570},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":23713,"operation_id":"0x75b9","timeout_ms":10000,"total_duration_us":10000674},{"operation_name":"upsert","last_dispatch_duration_us":9141599,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9141599,"total_server_duration_us":9026756,"encode_duration_us":17504,"operation_id":"0x7959","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":9149386},{"operation_name":"get","last_dispatch_duration_us":108734,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":108734,"total_server_duration_us":69,"operation_id":"0x7a3d","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":8339466},{"operation_name":"get","last_dispatch_duration_us":127399,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":127399,"total_server_duration_us":76,"operation_id":"0x7b11","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":6320050}],"total_count":24}} INFO [2022-03-13 17:13:06,155] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,157] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,157] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:06,158] jepsen worker 78 - jepsen.util 528 :invoke :write [26 22] INFO [2022-03-13 17:13:06,158] jepsen worker 80 - jepsen.util 710 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,159] jepsen worker 80 - jepsen.util 710 :invoke :write [26 27] INFO [2022-03-13 17:13:06,163] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,164] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:06,179] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,179] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:06,180] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,181] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:13:06,185] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,186] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:13:06,214] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,215] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:13:06,229] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,230] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 9] INFO [2022-03-13 17:13:06,238] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,239] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 40] INFO [2022-03-13 17:13:06,245] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:06,246] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,273] jepsen worker 49 - jepsen.util 1939 :ok :read [16 :nil] INFO [2022-03-13 17:13:06,274] jepsen worker 49 - jepsen.util 1939 :invoke :write [16 16] INFO [2022-03-13 17:13:06,277] jepsen worker 37 - jepsen.util 2017 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,278] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:13:06,306] jepsen worker 73 - jepsen.util 2413 :info :write [24 23] :RequestTimeoutException INFO [2022-03-13 17:13:06,307] jepsen worker 73 - jepsen.util 2503 :invoke :write [24 45] INFO [2022-03-13 17:13:06,317] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,317] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,320] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:06,320] jepsen worker nemesis - couchbase.util Rebalance check count: 296 INFO [2022-03-13 17:13:06,320] jepsen worker nemesis - couchbase.util Rebalance stuck count: 197 INFO [2022-03-13 17:13:06,328] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,329] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,330] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:13:06,332] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:06,333] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,337] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:06,341] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,342] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:13:06,349] jepsen worker 0 - jepsen.util 2250 :ok :read [0 41] INFO [2022-03-13 17:13:06,356] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:06,356] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:06,358] jepsen worker 0 - jepsen.util 2250 :invoke :write [0 38] INFO [2022-03-13 17:13:06,378] jepsen worker 88 - jepsen.util 1978 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,378] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 39] INFO [2022-03-13 17:13:06,409] jepsen worker 36 - jepsen.util 2106 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,410] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 27] INFO [2022-03-13 17:13:06,437] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,437] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:06,440] jepsen worker 10 - jepsen.util 2350 :info :write [3 15] :RequestTimeoutException INFO [2022-03-13 17:13:06,441] jepsen worker 10 - jepsen.util 2440 :invoke :read [3 nil] INFO [2022-03-13 17:13:06,444] jepsen worker 3 - jepsen.util 1893 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,444] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,445] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 13] INFO [2022-03-13 17:13:06,446] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,447] jepsen worker 34 - jepsen.util 754 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,447] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,448] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:06,448] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,448] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:06,449] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:13:06,449] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,450] jepsen worker 52 - jepsen.util 592 :invoke :write [17 7] INFO [2022-03-13 17:13:06,452] jepsen worker 60 - jepsen.util 600 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,452] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,453] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:06,453] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:06,454] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:06,455] jepsen worker 44 - jepsen.util 584 :invoke :write [14 0] INFO [2022-03-13 17:13:06,469] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,469] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,497] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,497] jepsen worker 22 - jepsen.util 652 :invoke :write [7 7] INFO [2022-03-13 17:13:06,502] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:06,502] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,531] jepsen worker 14 - jepsen.util 284 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,531] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,532] jepsen worker 7 - jepsen.util 637 :invoke :write [2 16] INFO [2022-03-13 17:13:06,535] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,535] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:06,535] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:13:06,538] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,539] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:06,541] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,541] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,541] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,542] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:13:06,569] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,570] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:13:06,574] jepsen worker 88 - jepsen.util 1978 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,575] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 21] INFO [2022-03-13 17:13:06,608] jepsen worker 3 - jepsen.util 1893 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,609] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,611] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:06,611] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,643] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:06,644] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,644] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,645] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:06,646] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,647] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:13:06,651] jepsen worker 70 - jepsen.util 700 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,652] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:06,689] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,690] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:06,690] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,691] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:13:06,708] jepsen worker 83 - jepsen.util 1613 :info :write [27 22] :RequestTimeoutException INFO [2022-03-13 17:13:06,709] jepsen worker 83 - jepsen.util 1703 :invoke :write [27 5] INFO [2022-03-13 17:13:06,714] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,714] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,715] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:06,715] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:06,718] jepsen worker 63 - jepsen.util 2133 :ok :read [21 :nil] INFO [2022-03-13 17:13:06,719] jepsen worker 63 - jepsen.util 2133 :invoke :write [21 5] INFO [2022-03-13 17:13:06,723] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:06,724] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,724] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,724] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:13:06,725] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,725] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,725] jepsen worker 22 - jepsen.util 652 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,726] jepsen worker 88 - jepsen.util 1978 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,727] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:13:06,727] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,743] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,743] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,744] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,744] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,745] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,745] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,746] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,747] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,748] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,748] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,749] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,749] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,750] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,750] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:06,750] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,751] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,752] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,752] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,752] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:13:06,753] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:13:06,769] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:06,769] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,775] jepsen worker 59 - jepsen.util 2309 :ok :read [19 36] INFO [2022-03-13 17:13:06,775] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:06,775] jepsen worker 59 - jepsen.util 2309 :invoke :write [19 1] INFO [2022-03-13 17:13:06,776] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,793] jepsen worker 36 - jepsen.util 2106 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,794] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 34] INFO [2022-03-13 17:13:06,833] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,833] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:13:06,838] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,839] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,861] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:06,861] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 29] INFO [2022-03-13 17:13:06,868] jepsen worker 24 - jepsen.util 564 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,868] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:06,873] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,874] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,907] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,908] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,917] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,917] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,917] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,917] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:06,918] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,918] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,918] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:13:06,919] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:13:06,932] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:06,932] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,941] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:06,942] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:06,947] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,948] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:06,956] jepsen worker 69 - jepsen.util 609 :info :write [23 10] :RequestTimeoutException INFO [2022-03-13 17:13:06,956] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:06,974] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:06,974] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:06,975] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:13:06,976] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:13:07,006] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:07,006] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,037] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,038] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:13:07,066] jepsen worker 88 - jepsen.util 1978 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,066] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:13:07,084] jepsen worker 36 - jepsen.util 2106 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,084] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,084] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 35] INFO [2022-03-13 17:13:07,085] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:13:07,106] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:07,107] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,123] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,124] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:13:07,131] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,132] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,133] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,133] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:07,180] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,180] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,190] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,190] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:13:07,221] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,221] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 11] INFO [2022-03-13 17:13:07,239] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:07,239] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,243] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,243] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,252] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,253] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,255] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,255] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,257] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,257] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,266] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,267] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:07,292] jepsen worker 78 - jepsen.util 528 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,292] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:07,296] jepsen worker 68 - jepsen.util 2408 :info :write [22 30] :RequestTimeoutException INFO [2022-03-13 17:13:07,296] jepsen worker 68 - jepsen.util 2498 :invoke :read [22 nil] INFO [2022-03-13 17:13:07,300] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,300] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,300] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,302] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:13:07,302] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:07,303] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:13:07,304] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,305] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,308] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,308] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:07,309] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,310] jepsen worker 62 - jepsen.util 512 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,310] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:07,311] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,312] jepsen worker 62 - jepsen.util 512 :invoke :write [20 48] INFO [2022-03-13 17:13:07,312] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:13:07,325] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,326] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:13:07,326] jepsen worker 36 - jepsen.util 2106 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,327] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:07,327] jepsen worker nemesis - couchbase.util Rebalance check count: 297 INFO [2022-03-13 17:13:07,327] jepsen worker nemesis - couchbase.util Rebalance stuck count: 198 INFO [2022-03-13 17:13:07,327] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,328] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,328] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 48] INFO [2022-03-13 17:13:07,338] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,339] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:13:07,341] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:07,342] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:07,343] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,344] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:13:07,345] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,351] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:13:07,364] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:07,365] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,403] jepsen worker 80 - jepsen.util 710 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,403] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,403] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,404] jepsen worker 80 - jepsen.util 710 :invoke :write [26 11] INFO [2022-03-13 17:13:07,405] jepsen worker 34 - jepsen.util 754 :invoke :write [11 49] INFO [2022-03-13 17:13:07,405] jepsen worker 13 - jepsen.util 643 :invoke :write [4 9] INFO [2022-03-13 17:13:07,422] jepsen worker 37 - jepsen.util 2017 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,423] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,423] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:13:07,430] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,438] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,439] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,440] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,440] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,440] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,440] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:07,440] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:13:07,443] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:13:07,444] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:13:07,454] jepsen worker 24 - jepsen.util 564 :invoke :write [8 39] INFO [2022-03-13 17:13:07,459] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,459] jepsen worker 12 - jepsen.util 732 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,459] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,459] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,459] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,460] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:07,460] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:07,461] jepsen worker 23 - jepsen.util 563 :invoke :write [7 40] INFO [2022-03-13 17:13:07,462] jepsen worker 76 - jepsen.util 706 :invoke :write [25 47] INFO [2022-03-13 17:13:07,462] jepsen worker 12 - jepsen.util 732 :invoke :write [4 16] INFO [2022-03-13 17:13:07,465] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,465] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,476] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,477] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:07,478] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,479] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:07,480] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,483] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:07,496] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,497] jepsen worker 22 - jepsen.util 652 :invoke :write [7 28] INFO [2022-03-13 17:13:07,510] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,511] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] WARN [2022-03-13 17:13:07,543] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15646727,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15646727,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":5238700,"encode_duration_us":29783,"operation_id":"0x757d","timeout_ms":10000,"last_server_duration_us":5238700,"total_duration_us":10002560}],"total_count":1}} WARN [2022-03-13 17:13:07,549] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9110225,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18557899,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9270264,"encode_duration_us":18719,"operation_id":"0x75d2","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10005827},{"operation_name":"upsert","last_dispatch_duration_us":30315585,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":30315585,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9024634,"encode_duration_us":20523,"operation_id":"0x6d5a","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10006269}],"total_count":2}} INFO [2022-03-13 17:13:07,550] jepsen worker 6 - jepsen.util 636 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,551] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:07,551] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,551] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:07,551] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:07,552] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,552] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,553] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:07,554] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:13:07,555] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,555] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:07,555] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 16] INFO [2022-03-13 17:13:07,561] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,561] jepsen worker 52 - jepsen.util 592 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,562] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,564] jepsen worker 70 - jepsen.util 700 :invoke :write [23 13] INFO [2022-03-13 17:13:07,566] jepsen worker 78 - jepsen.util 528 :invoke :write [26 15] INFO [2022-03-13 17:13:07,567] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:07,570] jepsen worker 64 - jepsen.util 2314 :info :write [21 11] :RequestTimeoutException INFO [2022-03-13 17:13:07,570] jepsen worker 64 - jepsen.util 2404 :invoke :read [21 nil] INFO [2022-03-13 17:13:07,592] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:07,592] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,595] jepsen worker 3 - jepsen.util 1893 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,596] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,601] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,602] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:07,610] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,610] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,610] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,610] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,616] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,616] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,616] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:13:07,616] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,618] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,619] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:13:07,624] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:07,624] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,635] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,636] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,636] jepsen worker 46 - jepsen.util 2296 :info :write [15 5] :RequestTimeoutException INFO [2022-03-13 17:13:07,637] jepsen worker 46 - jepsen.util 2386 :invoke :read [15 nil] INFO [2022-03-13 17:13:07,643] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,644] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] WARN [2022-03-13 17:13:07,645] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9710022,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10851678,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":83199,"encode_duration_us":16694,"operation_id":"0x7a61","timeout_ms":10000,"last_server_duration_us":51710,"total_duration_us":10001547},{"operation_name":"upsert","last_dispatch_duration_us":10382321,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10382321,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":459442,"encode_duration_us":22501,"operation_id":"0x7aec","timeout_ms":10000,"last_server_duration_us":459442,"total_duration_us":10003770},{"operation_name":"upsert","last_dispatch_duration_us":16845071,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16845071,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9055417,"encode_duration_us":18003,"operation_id":"0x7586","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10008267},{"operation_name":"upsert","last_dispatch_duration_us":18123279,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26722509,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9092407,"encode_duration_us":23150,"operation_id":"0x7010","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10009762},{"operation_name":"upsert","last_dispatch_duration_us":10319102,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10319102,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9294690,"encode_duration_us":16866,"operation_id":"0x7a69","timeout_ms":10000,"last_server_duration_us":9294690,"total_duration_us":10007836}],"total_count":5}} INFO [2022-03-13 17:13:07,646] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,646] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,647] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:13:07,647] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,648] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:13:07,649] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] WARN [2022-03-13 17:13:07,650] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":29445911,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":29445911,"total_server_duration_us":32,"operation_id":"0x6dd8","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":29448828},{"operation_name":"get","last_dispatch_duration_us":25087504,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":25087504,"total_server_duration_us":32,"operation_id":"0x6fb8","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":25089812},{"operation_name":"get","last_dispatch_duration_us":15885270,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":15885270,"total_server_duration_us":23,"operation_id":"0x749f","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":15885523},{"operation_name":"upsert","last_dispatch_duration_us":32859,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":607826,"total_server_duration_us":288165,"encode_duration_us":18097,"operation_id":"0x7ac6","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10002447},{"operation_name":"upsert","last_dispatch_duration_us":9092145,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9146141,"total_server_duration_us":9045550,"encode_duration_us":16850,"operation_id":"0x7a99","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9160486},{"operation_name":"upsert","last_dispatch_duration_us":796187,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2772123,"total_server_duration_us":744429,"encode_duration_us":17506,"operation_id":"0x7a24","timeout_ms":10000,"last_server_duration_us":143869,"total_duration_us":2787941},{"operation_name":"upsert","last_dispatch_duration_us":2363371,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2363371,"total_server_duration_us":336289,"encode_duration_us":18042,"operation_id":"0x7c4b","timeout_ms":10000,"last_server_duration_us":336289,"total_duration_us":2369669},{"operation_name":"upsert","last_dispatch_duration_us":2336145,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2336145,"total_server_duration_us":21243,"encode_duration_us":27885,"operation_id":"0x7ca6","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":2342170},{"operation_name":"upsert","last_dispatch_duration_us":2195401,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2195401,"total_server_duration_us":37550,"encode_duration_us":24738,"operation_id":"0x7ca4","timeout_ms":10000,"last_server_duration_us":37550,"total_duration_us":2199729},{"operation_name":"get","last_dispatch_duration_us":2030303,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2030303,"total_server_duration_us":49,"operation_id":"0x7c63","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":2030494}],"total_count":61}} INFO [2022-03-13 17:13:07,666] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,666] jepsen worker 34 - jepsen.util 754 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,667] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:07,667] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:07,669] jepsen worker 34 - jepsen.util 754 :invoke :write [11 0] INFO [2022-03-13 17:13:07,669] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,674] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,674] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,674] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:13:07,675] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:07,675] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,679] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:13:07,684] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,684] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:07,685] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,686] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:07,687] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,688] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:13:07,699] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:07,700] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,705] jepsen worker 44 - jepsen.util 584 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,705] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,705] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,705] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,705] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,707] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:07,708] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:07,709] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:13:07,709] jepsen worker 77 - jepsen.util 707 :invoke :write [25 22] INFO [2022-03-13 17:13:07,714] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] WARN [2022-03-13 17:13:07,738] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9111712,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":18239456,"total_server_duration_us":45053,"encode_duration_us":33101,"operation_id":"0x764d","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10009761},{"operation_name":"upsert","last_dispatch_duration_us":34075,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7225566,"total_server_duration_us":366690,"encode_duration_us":18678,"operation_id":"0x7864","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10008911},{"operation_name":"upsert","last_dispatch_duration_us":17714976,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17714976,"total_server_duration_us":24755,"encode_duration_us":22623,"operation_id":"0x76ad","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10006019},{"operation_name":"upsert","last_dispatch_duration_us":9160027,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9160027,"total_server_duration_us":31203,"encode_duration_us":17388,"operation_id":"0x7658","timeout_ms":10000,"last_server_duration_us":31203,"total_duration_us":10003483},{"operation_name":"upsert","last_dispatch_duration_us":9074904,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9074904,"total_server_duration_us":21649,"encode_duration_us":17217,"operation_id":"0x7a64","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10001562},{"operation_name":"get","last_dispatch_duration_us":9142888,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9142888,"total_server_duration_us":43,"operation_id":"0x7af5","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9155931},{"operation_name":"get","last_dispatch_duration_us":9094629,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9094629,"total_server_duration_us":49,"operation_id":"0x7afc","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9098066},{"operation_name":"upsert","last_dispatch_duration_us":6366106,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":6366106,"total_server_duration_us":309026,"encode_duration_us":18398,"operation_id":"0x78db","timeout_ms":10000,"last_server_duration_us":309026,"total_duration_us":6369898},{"operation_name":"upsert","last_dispatch_duration_us":1706463,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1706463,"total_server_duration_us":243386,"encode_duration_us":28510,"operation_id":"0x7b49","timeout_ms":10000,"last_server_duration_us":243386,"total_duration_us":1710051},{"operation_name":"get","last_dispatch_duration_us":974390,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":974390,"total_server_duration_us":1185,"operation_id":"0x7aac","timeout_ms":30000,"last_server_duration_us":1185,"total_duration_us":974660}],"total_count":20}} INFO [2022-03-13 17:13:07,743] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,744] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] WARN [2022-03-13 17:13:07,746] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14597002,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14597002,"total_server_duration_us":10272,"operation_id":"0x7722","timeout_ms":30000,"last_server_duration_us":10272,"total_duration_us":14599608},{"operation_name":"get","last_dispatch_duration_us":1121477,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1121477,"total_server_duration_us":38,"operation_id":"0x7aee","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":10372363},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18889,"operation_id":"0x79e0","timeout_ms":10000,"total_duration_us":10008963},{"operation_name":"upsert","last_dispatch_duration_us":10319102,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10319102,"total_server_duration_us":9294690,"encode_duration_us":16866,"operation_id":"0x7a69","timeout_ms":10000,"last_server_duration_us":9294690,"total_duration_us":10007836},{"operation_name":"upsert","last_dispatch_duration_us":10382321,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10382321,"total_server_duration_us":459442,"encode_duration_us":22501,"operation_id":"0x7aec","timeout_ms":10000,"last_server_duration_us":459442,"total_duration_us":10003770},{"operation_name":"upsert","last_dispatch_duration_us":12001357,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":12001357,"total_server_duration_us":21243,"encode_duration_us":16630,"operation_id":"0x7891","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10001809},{"operation_name":"upsert","last_dispatch_duration_us":9710022,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10851678,"total_server_duration_us":83199,"encode_duration_us":16694,"operation_id":"0x7a61","timeout_ms":10000,"last_server_duration_us":51710,"total_duration_us":10001547},{"operation_name":"upsert","last_dispatch_duration_us":9964731,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9964731,"total_server_duration_us":148831,"encode_duration_us":19190,"operation_id":"0x7ae7","timeout_ms":10000,"last_server_duration_us":148831,"total_duration_us":9971212},{"operation_name":"upsert","last_dispatch_duration_us":9858958,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9858958,"total_server_duration_us":136114,"encode_duration_us":20978,"operation_id":"0x7adf","timeout_ms":10000,"last_server_duration_us":136114,"total_duration_us":9863218},{"operation_name":"upsert","last_dispatch_duration_us":9652313,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9652313,"total_server_duration_us":275494,"encode_duration_us":27492,"operation_id":"0x7ad7","timeout_ms":10000,"last_server_duration_us":275494,"total_duration_us":9657666}],"total_count":27}} INFO [2022-03-13 17:13:07,746] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,747] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:13:07,748] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,749] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 41] INFO [2022-03-13 17:13:07,756] jepsen worker 17 - jepsen.util 2267 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:13:07,756] jepsen worker 17 - jepsen.util 2357 :invoke :read [5 nil] INFO [2022-03-13 17:13:07,760] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,761] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,762] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,763] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,767] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,768] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,770] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,771] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,775] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,775] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,777] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,777] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:07,779] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,779] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:07,788] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:07,788] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,813] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:07,813] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:13:07,829] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,830] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] WARN [2022-03-13 17:13:07,839] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18166186,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20575505,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9048337,"encode_duration_us":24877,"operation_id":"0x74e0","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10007444}],"total_count":1}} INFO [2022-03-13 17:13:07,843] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,844] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:13:07,844] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14472805,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":14472805,"total_server_duration_us":32,"operation_id":"0x7645","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":14473804},{"operation_name":"upsert","last_dispatch_duration_us":18033032,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":18226634,"total_server_duration_us":9059142,"encode_duration_us":32285,"operation_id":"0x7748","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10008702},{"operation_name":"upsert","last_dispatch_duration_us":18016025,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":18775316,"total_server_duration_us":9146586,"encode_duration_us":47637,"operation_id":"0x76f7","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006487},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":19513,"operation_id":"0x7a65","timeout_ms":10000,"total_duration_us":10006181},{"operation_name":"upsert","last_dispatch_duration_us":35715,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9770799,"total_server_duration_us":151436,"encode_duration_us":32744,"operation_id":"0x76fa","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10002205},{"operation_name":"upsert","last_dispatch_duration_us":9164301,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9164301,"total_server_duration_us":9056479,"encode_duration_us":22059,"operation_id":"0x7741","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":9168528},{"operation_name":"upsert","last_dispatch_duration_us":9068764,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9068764,"total_server_duration_us":9047983,"encode_duration_us":31500,"operation_id":"0x76ec","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9074869},{"operation_name":"get","last_dispatch_duration_us":51772,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":51772,"total_server_duration_us":8015,"operation_id":"0x7866","timeout_ms":30000,"last_server_duration_us":8015,"total_duration_us":7770677},{"operation_name":"upsert","last_dispatch_duration_us":1175750,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2997678,"total_server_duration_us":97387,"encode_duration_us":18583,"operation_id":"0x7907","timeout_ms":10000,"last_server_duration_us":33819,"total_duration_us":6648566},{"operation_name":"upsert","last_dispatch_duration_us":3877147,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3877147,"total_server_duration_us":126636,"encode_duration_us":19993,"operation_id":"0x7cac","timeout_ms":10000,"last_server_duration_us":126636,"total_duration_us":3897372}],"total_count":62}} INFO [2022-03-13 17:13:07,844] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,845] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:07,845] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,846] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,847] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:13:07,847] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] WARN [2022-03-13 17:13:07,848] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17492211,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17492211,"total_server_duration_us":43,"operation_id":"0x7730","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":17493502},{"operation_name":"get","last_dispatch_duration_us":10691806,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10691806,"total_server_duration_us":436375,"operation_id":"0x7595","timeout_ms":30000,"last_server_duration_us":436375,"total_duration_us":10694397},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18119,"operation_id":"0x7b37","timeout_ms":10000,"total_duration_us":10010667},{"operation_name":"upsert","last_dispatch_duration_us":10414178,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10414178,"total_server_duration_us":30633,"encode_duration_us":18469,"operation_id":"0x7a86","timeout_ms":10000,"last_server_duration_us":30633,"total_duration_us":10009725},{"operation_name":"upsert","last_dispatch_duration_us":13395696,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":13395696,"total_server_duration_us":20441,"encode_duration_us":18263,"operation_id":"0x797f","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10006210},{"operation_name":"upsert","last_dispatch_duration_us":9484425,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9484425,"total_server_duration_us":26330,"encode_duration_us":17948,"operation_id":"0x7b07","timeout_ms":10000,"last_server_duration_us":26330,"total_duration_us":10005890},{"operation_name":"upsert","last_dispatch_duration_us":9507267,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":9507267,"total_server_duration_us":202112,"encode_duration_us":25480,"operation_id":"0x788d","timeout_ms":10000,"last_server_duration_us":202112,"total_duration_us":9690302},{"operation_name":"upsert","last_dispatch_duration_us":9397652,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":9397652,"total_server_duration_us":167439,"encode_duration_us":25666,"operation_id":"0x788c","timeout_ms":10000,"last_server_duration_us":167439,"total_duration_us":9402482},{"operation_name":"get","last_dispatch_duration_us":9125355,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9125355,"total_server_duration_us":4404,"operation_id":"0x7b30","timeout_ms":30000,"last_server_duration_us":4404,"total_duration_us":9125608},{"operation_name":"upsert","last_dispatch_duration_us":8885098,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":8885098,"total_server_duration_us":170957,"encode_duration_us":21528,"operation_id":"0x788b","timeout_ms":10000,"last_server_duration_us":170957,"total_duration_us":8889477}],"total_count":48}} INFO [2022-03-13 17:13:07,863] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:07,863] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,893] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,894] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:07,897] jepsen worker 56 - jepsen.util 2396 :info :write [18 29] :RequestTimeoutException INFO [2022-03-13 17:13:07,898] jepsen worker 56 - jepsen.util 2486 :invoke :write [18 15] INFO [2022-03-13 17:13:07,905] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:07,906] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:07,942] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,942] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,943] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:13:07,944] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:07,946] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,947] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] WARN [2022-03-13 17:13:07,948] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18087682,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25328158,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9062694,"encode_duration_us":16669,"operation_id":"0x727f","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10002121},{"operation_name":"upsert","last_dispatch_duration_us":18565219,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27643100,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9092669,"encode_duration_us":16718,"operation_id":"0x712d","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10004208}],"total_count":2}} INFO [2022-03-13 17:13:07,982] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:07,983] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:13:08,020] jepsen worker 85 - jepsen.util 2155 :info :write [28 48] :RequestTimeoutException INFO [2022-03-13 17:13:08,021] jepsen worker 85 - jepsen.util 2245 :invoke :write [28 38] INFO [2022-03-13 17:13:08,023] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,023] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 9] INFO [2022-03-13 17:13:08,043] jepsen worker 36 - jepsen.util 2106 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,057] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:08,060] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,060] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:13:08,096] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,097] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,098] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,098] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,100] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,100] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:08,123] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,123] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 0] INFO [2022-03-13 17:13:08,136] jepsen worker 40 - jepsen.util 2290 :info :write [13 20] :RequestTimeoutException INFO [2022-03-13 17:13:08,137] jepsen worker 40 - jepsen.util 2380 :invoke :write [13 7] INFO [2022-03-13 17:13:08,146] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,146] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:08,166] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,166] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:08,204] jepsen worker 3 - jepsen.util 1893 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,204] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:08,217] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,217] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,218] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:13:08,218] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,218] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,219] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,239] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,240] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:08,248] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,248] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:08,251] jepsen worker 61 - jepsen.util 601 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,252] jepsen worker 61 - jepsen.util 601 :invoke :write [20 2] INFO [2022-03-13 17:13:08,253] jepsen worker 22 - jepsen.util 652 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,253] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:08,255] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:08,255] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,258] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,258] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:13:08,320] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,320] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,320] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,321] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:13:08,321] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:08,322] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:13:08,322] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,323] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:13:08,333] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:08,333] jepsen worker nemesis - couchbase.util Rebalance check count: 298 INFO [2022-03-13 17:13:08,333] jepsen worker nemesis - couchbase.util Rebalance stuck count: 199 INFO [2022-03-13 17:13:08,340] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,341] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,343] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,344] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,345] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,346] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:13:08,366] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,366] jepsen worker 37 - jepsen.util 2017 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,366] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,368] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:13:08,368] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,369] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 26] INFO [2022-03-13 17:13:08,379] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:08,380] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:13:08,393] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,393] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,394] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:13:08,395] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:13:08,399] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,400] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:13:08,427] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,427] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,427] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:13:08,428] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,444] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,445] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,467] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,468] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:08,506] jepsen worker 37 - jepsen.util 2017 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,506] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,506] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,507] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 41] INFO [2022-03-13 17:13:08,508] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:13:08,508] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 8] INFO [2022-03-13 17:13:08,566] jepsen worker 18 - jepsen.util 2088 :info :write [6 31] :RequestTimeoutException INFO [2022-03-13 17:13:08,567] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,567] jepsen worker 18 - jepsen.util 2178 :invoke :read [6 nil] INFO [2022-03-13 17:13:08,568] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:13:08,579] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,579] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:08,621] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,622] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,659] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:08,660] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,660] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 26] INFO [2022-03-13 17:13:08,660] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:13:08,672] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,672] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:13:08,679] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,679] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:08,680] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,681] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:13:08,687] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:08,691] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,724] jepsen worker 36 - jepsen.util 2106 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,724] jepsen worker 89 - jepsen.util 2429 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,725] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,725] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,763] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,764] jepsen worker 88 - jepsen.util 1978 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,764] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:13:08,765] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,768] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:08,769] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,804] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:08,805] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 22] INFO [2022-03-13 17:13:08,834] jepsen worker 27 - jepsen.util 2637 :info :write [9 36] :RequestTimeoutException INFO [2022-03-13 17:13:08,835] jepsen worker 27 - jepsen.util 2727 :invoke :write [9 27] INFO [2022-03-13 17:13:08,868] jepsen worker 54 - jepsen.util 1944 :info :write [18 4] :RequestTimeoutException INFO [2022-03-13 17:13:08,869] jepsen worker 54 - jepsen.util 2034 :invoke :write [18 0] INFO [2022-03-13 17:13:08,871] jepsen worker 65 - jepsen.util 2675 :info :write [21 34] :RequestTimeoutException INFO [2022-03-13 17:13:08,871] jepsen worker 65 - jepsen.util 2765 :invoke :read [21 nil] INFO [2022-03-13 17:13:08,881] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,881] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:13:08,890] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,891] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,899] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,900] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:08,907] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:08,907] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:08,911] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:08,911] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:08,940] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:08,940] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,001] jepsen worker 88 - jepsen.util 1978 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,002] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 13] INFO [2022-03-13 17:13:09,018] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,018] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,018] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,019] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 4] INFO [2022-03-13 17:13:09,019] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,019] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:09,020] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,020] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,027] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,028] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,028] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 33] INFO [2022-03-13 17:13:09,029] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,031] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,032] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:13:09,059] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:09,060] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:09,067] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,069] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:13:09,079] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,079] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:13:09,081] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,082] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,096] jepsen worker 37 - jepsen.util 2017 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,096] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,097] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:13:09,097] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:09,097] jepsen worker 78 - jepsen.util 528 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,098] jepsen worker 78 - jepsen.util 528 :invoke :write [26 5] INFO [2022-03-13 17:13:09,118] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,118] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,119] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,120] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:13:09,122] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,123] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,127] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,127] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,128] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:13:09,129] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:13:09,130] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:09,130] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,147] jepsen worker 36 - jepsen.util 2106 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,147] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 37] INFO [2022-03-13 17:13:09,157] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:09,157] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,231] jepsen worker 88 - jepsen.util 1978 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,231] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,232] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,232] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,233] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,233] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,234] jepsen worker 24 - jepsen.util 564 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,234] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,235] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:13:09,236] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,245] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,246] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,259] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,259] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:09,272] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:09,273] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:13:09,281] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:09,282] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,287] jepsen worker 70 - jepsen.util 700 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,287] jepsen worker 70 - jepsen.util 700 :invoke :write [23 32] INFO [2022-03-13 17:13:09,317] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,317] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,317] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,317] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:13:09,318] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:13:09,319] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:09,325] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,326] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:09,327] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,328] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:13:09,328] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,329] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:13:09,343] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:09,344] jepsen worker nemesis - couchbase.util Rebalance check count: 299 INFO [2022-03-13 17:13:09,344] jepsen worker nemesis - couchbase.util Rebalance stuck count: 200 INFO [2022-03-13 17:13:09,359] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,359] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:09,362] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,363] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:09,367] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,367] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:13:09,379] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:09,379] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 48] INFO [2022-03-13 17:13:09,397] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,398] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,399] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,400] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:13:09,404] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,405] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 39] INFO [2022-03-13 17:13:09,409] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,410] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,410] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,410] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,411] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:13:09,411] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:09,412] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,412] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,413] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:09,413] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,413] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:09,414] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,414] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,414] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,415] jepsen worker 75 - jepsen.util 435 :invoke :write [25 21] INFO [2022-03-13 17:13:09,415] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,415] jepsen worker 33 - jepsen.util 663 :invoke :write [11 37] INFO [2022-03-13 17:13:09,422] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:13:09,450] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,450] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:13:09,453] jepsen worker 72 - jepsen.util 2232 :info :write [24 29] :RequestTimeoutException INFO [2022-03-13 17:13:09,454] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:09,455] jepsen worker 72 - jepsen.util 2322 :invoke :write [24 34] INFO [2022-03-13 17:13:09,455] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 40] INFO [2022-03-13 17:13:09,470] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,471] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:13:09,528] jepsen worker 34 - jepsen.util 754 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,528] jepsen worker 33 - jepsen.util 663 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,529] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:09,529] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:13:09,538] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,539] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:09,541] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,542] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:13:09,554] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,554] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,555] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,556] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:09,573] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,573] jepsen worker 36 - jepsen.util 2106 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,574] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 35] INFO [2022-03-13 17:13:09,576] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,603] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,604] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:13:09,624] jepsen worker 75 - jepsen.util 435 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,624] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:09,627] jepsen worker 88 - jepsen.util 1978 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,628] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,661] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,662] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:13:09,664] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,664] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:13:09,676] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,676] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,677] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,677] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,678] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:13:09,678] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,678] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:13:09,680] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:13:09,691] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:09,692] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,695] jepsen worker 89 - jepsen.util 2429 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,696] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,743] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,743] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:09,745] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,746] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:13:09,751] jepsen worker 37 - jepsen.util 2017 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,751] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,751] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,752] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,753] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:13:09,753] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:13:09,766] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,766] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:13:09,772] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:09,777] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,783] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,784] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:09,790] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:09,791] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:13:09,800] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,800] jepsen worker 13 - jepsen.util 643 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,801] jepsen worker 52 - jepsen.util 592 :invoke :write [17 33] INFO [2022-03-13 17:13:09,801] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:09,814] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:09,814] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:13:09,819] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,819] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:13:09,847] jepsen worker 36 - jepsen.util 2106 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,848] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,848] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 18] INFO [2022-03-13 17:13:09,849] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,854] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,854] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,879] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,880] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:09,883] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,884] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:09,894] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,894] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,894] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:09,894] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:13:09,911] jepsen worker 80 - jepsen.util 710 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,911] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:09,949] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,949] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,965] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,965] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,966] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:09,966] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:13:09,965] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,967] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,967] jepsen worker 21 - jepsen.util 831 :invoke :write [7 14] INFO [2022-03-13 17:13:09,967] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,967] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:09,968] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:13:09,969] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,971] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:09,973] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,973] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:13:09,976] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,977] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:13:09,979] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:09,980] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 31] INFO [2022-03-13 17:13:09,982] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,982] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,982] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,985] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:09,985] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:13:09,985] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:09,987] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:09,987] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:13:10,031] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:10,032] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,040] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,040] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:10,056] jepsen worker 78 - jepsen.util 528 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,056] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:10,074] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,074] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,075] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,075] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:10,075] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:13:10,076] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:13:10,080] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,080] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:10,087] jepsen worker 36 - jepsen.util 2106 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,087] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,088] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 9] INFO [2022-03-13 17:13:10,089] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:13:10,123] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:10,123] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:13:10,130] jepsen worker 88 - jepsen.util 1978 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,131] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,131] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,131] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,132] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:10,133] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:10,169] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:10,169] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 49] INFO [2022-03-13 17:13:10,171] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,172] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:13:10,177] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,177] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:13:10,184] jepsen worker 70 - jepsen.util 700 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,184] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,185] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,186] jepsen worker 34 - jepsen.util 754 :invoke :write [11 21] INFO [2022-03-13 17:13:10,191] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:10,191] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:10,211] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,212] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:13:10,235] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,236] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:10,237] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,238] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:13:10,262] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,262] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,263] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:13:10,264] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:10,272] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,272] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:10,274] jepsen worker 21 - jepsen.util 831 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,274] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:13:10,286] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,286] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:10,293] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,293] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:10,295] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,296] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:13:10,304] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,305] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:10,334] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,336] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:13:10,347] jepsen worker 28 - jepsen.util 2548 :ok :read [9 26] INFO [2022-03-13 17:13:10,348] jepsen worker 28 - jepsen.util 2548 :invoke :read [9 nil] INFO [2022-03-13 17:13:10,350] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:10,350] jepsen worker nemesis - couchbase.util Rebalance check count: 300 INFO [2022-03-13 17:13:10,350] jepsen worker nemesis - couchbase.util Rebalance stuck count: 201 INFO [2022-03-13 17:13:10,353] jepsen worker 88 - jepsen.util 1978 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,353] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,359] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,360] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:13:10,365] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,365] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,366] jepsen worker 11 - jepsen.util 2351 :ok :read [3 25] INFO [2022-03-13 17:13:10,367] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:13:10,367] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,368] jepsen worker 11 - jepsen.util 2351 :invoke :write [3 7] INFO [2022-03-13 17:13:10,394] jepsen worker 52 - jepsen.util 592 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,394] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,394] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,394] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,395] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,395] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,395] jepsen worker 52 - jepsen.util 592 :invoke :write [17 13] INFO [2022-03-13 17:13:10,395] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,395] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,396] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,397] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:10,398] jepsen worker 78 - jepsen.util 528 :invoke :write [26 29] INFO [2022-03-13 17:13:10,398] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,399] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:13:10,400] jepsen worker 13 - jepsen.util 643 :invoke :write [4 26] INFO [2022-03-13 17:13:10,401] jepsen worker 24 - jepsen.util 564 :invoke :write [8 38] INFO [2022-03-13 17:13:10,402] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:10,403] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:13:10,406] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:10,409] jepsen worker 80 - jepsen.util 710 :invoke :write [26 34] INFO [2022-03-13 17:13:10,414] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,418] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,419] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:13:10,421] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,421] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,423] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:10,423] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:13:10,424] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:10,449] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:10,450] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,475] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,477] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:13:10,486] jepsen worker 36 - jepsen.util 2106 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,486] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 8] INFO [2022-03-13 17:13:10,506] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,506] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:10,507] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:10,508] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,519] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,519] jepsen worker 23 - jepsen.util 563 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,520] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:10,520] jepsen worker 23 - jepsen.util 563 :invoke :write [7 46] INFO [2022-03-13 17:13:10,527] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,527] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:10,545] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,545] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,545] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:10,546] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:13:10,548] jepsen worker 24 - jepsen.util 564 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,548] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:13:10,549] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,549] jepsen worker 22 - jepsen.util 652 :invoke :write [7 37] INFO [2022-03-13 17:13:10,555] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:10,556] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,562] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,563] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:10,605] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:10,605] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,607] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,607] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 37] INFO [2022-03-13 17:13:10,643] jepsen worker 34 - jepsen.util 754 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,643] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,644] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:13:10,644] jepsen worker 70 - jepsen.util 700 :invoke :write [23 33] INFO [2022-03-13 17:13:10,648] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,649] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,649] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:10,649] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,649] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:13:10,649] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:13:10,658] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,659] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,666] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,666] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,667] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:10,668] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,668] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,669] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:10,670] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:13:10,670] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 26] INFO [2022-03-13 17:13:10,676] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,678] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:10,684] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:10,685] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,691] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,692] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:10,702] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:10,702] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 21] INFO [2022-03-13 17:13:10,710] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:10,710] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:13:10,717] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:10,717] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:13:10,733] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,733] jepsen worker 26 - jepsen.util 476 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,734] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:10,735] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:10,758] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,758] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:10,768] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,769] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:13:10,794] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,794] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,795] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:13:10,795] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,795] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:13:10,797] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:13:10,857] jepsen worker 88 - jepsen.util 1978 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,857] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,859] jepsen worker 52 - jepsen.util 592 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,860] jepsen worker 52 - jepsen.util 592 :invoke :write [17 14] INFO [2022-03-13 17:13:10,867] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,867] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,868] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,868] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:13:10,869] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:10,869] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:10,870] jepsen worker 3 - jepsen.util 1893 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,871] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:13:10,879] jepsen worker 36 - jepsen.util 2106 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,880] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 19] INFO [2022-03-13 17:13:10,896] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:10,897] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:13:10,931] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,931] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,931] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,931] jepsen worker 75 - jepsen.util 435 :invoke :write [25 20] INFO [2022-03-13 17:13:10,932] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:13:10,932] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:10,934] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,935] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:13:10,939] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,939] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:10,943] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,943] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:10,968] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,968] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,969] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:10,969] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:13:10,974] jepsen worker 22 - jepsen.util 652 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,974] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,975] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:10,976] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:13:10,976] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,977] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:10,978] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,979] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:10,980] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:10,981] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:13:10,991] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:10,992] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:10,996] jepsen worker 32 - jepsen.util 2192 :info :write [10 2] :RequestTimeoutException INFO [2022-03-13 17:13:10,997] jepsen worker 32 - jepsen.util 2282 :invoke :read [10 nil] INFO [2022-03-13 17:13:11,030] jepsen worker 37 - jepsen.util 2017 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,030] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,031] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,031] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,044] jepsen worker 47 - jepsen.util 2387 :ok :read [15 48] INFO [2022-03-13 17:13:11,044] jepsen worker 47 - jepsen.util 2387 :invoke :read [15 nil] INFO [2022-03-13 17:13:11,061] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,062] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:11,070] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,071] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:11,091] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:11,092] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:13:11,096] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,096] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:13:11,105] jepsen worker 13 - jepsen.util 643 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,105] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:11,106] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,107] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:11,129] jepsen worker 36 - jepsen.util 2106 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,129] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,129] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,129] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,130] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:13:11,130] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,146] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,147] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,149] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,150] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:11,161] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,162] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:13:11,163] jepsen worker 75 - jepsen.util 435 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,163] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,164] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:13:11,164] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:11,180] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,180] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,180] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 11] INFO [2022-03-13 17:13:11,181] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,181] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:13:11,181] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:13:11,241] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,241] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:13:11,242] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,243] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,243] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,243] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:11,243] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:13:11,244] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:13:11,248] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,249] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,249] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:13:11,250] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:13:11,253] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,253] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,253] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,254] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,256] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:13:11,257] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:13:11,262] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:13:11,266] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:13:11,278] jepsen worker 45 - jepsen.util 2565 :info :write [15 10] :RequestTimeoutException INFO [2022-03-13 17:13:11,278] jepsen worker 45 - jepsen.util 2655 :invoke :read [15 nil] INFO [2022-03-13 17:13:11,298] jepsen worker 30 - jepsen.util 2370 :info :write [10 29] :RequestTimeoutException INFO [2022-03-13 17:13:11,299] jepsen worker 30 - jepsen.util 2460 :invoke :read [10 nil] INFO [2022-03-13 17:13:11,305] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,305] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:13:11,322] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,322] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:13:11,345] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,346] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,355] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:11,355] jepsen worker nemesis - couchbase.util Rebalance check count: 301 INFO [2022-03-13 17:13:11,355] jepsen worker nemesis - couchbase.util Rebalance stuck count: 202 INFO [2022-03-13 17:13:11,359] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,359] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:13:11,370] jepsen worker 48 - jepsen.util 2208 :info :write [16 40] :RequestTimeoutException INFO [2022-03-13 17:13:11,371] jepsen worker 48 - jepsen.util 2298 :invoke :read [16 nil] INFO [2022-03-13 17:13:11,371] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,372] jepsen worker 78 - jepsen.util 528 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,372] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:11,373] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:13:11,386] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,387] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,412] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,412] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:11,426] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,427] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:13:11,428] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,429] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,430] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,430] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,431] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,432] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,436] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,437] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:11,439] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,440] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,441] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,441] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:11,474] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,474] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:11,493] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,493] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:11,497] jepsen worker 36 - jepsen.util 2106 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,498] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 44] INFO [2022-03-13 17:13:11,517] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,518] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:11,520] jepsen worker 80 - jepsen.util 710 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,521] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:11,528] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,528] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,529] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:13:11,529] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,541] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,541] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:13:11,555] jepsen worker 16 - jepsen.util 2086 :ok :read [5 43] INFO [2022-03-13 17:13:11,556] jepsen worker 16 - jepsen.util 2086 :invoke :read [5 nil] INFO [2022-03-13 17:13:11,560] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:11,560] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 45] INFO [2022-03-13 17:13:11,561] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:11,562] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:13:11,602] jepsen worker 84 - jepsen.util 2514 :ok :read [28 44] INFO [2022-03-13 17:13:11,603] jepsen worker 84 - jepsen.util 2514 :invoke :read [28 nil] INFO [2022-03-13 17:13:11,603] jepsen worker 58 - jepsen.util 2218 :ok :read [19 36] INFO [2022-03-13 17:13:11,604] jepsen worker 58 - jepsen.util 2218 :invoke :read [19 nil] INFO [2022-03-13 17:13:11,606] jepsen worker 57 - jepsen.util 2307 :ok :read [19 36] INFO [2022-03-13 17:13:11,607] jepsen worker 57 - jepsen.util 2307 :invoke :read [19 nil] INFO [2022-03-13 17:13:11,607] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,608] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,608] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,609] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,615] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,616] jepsen worker 16 - jepsen.util 2086 :ok :read [5 43] INFO [2022-03-13 17:13:11,616] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:11,616] jepsen worker 16 - jepsen.util 2086 :invoke :read [5 nil] INFO [2022-03-13 17:13:11,637] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,637] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,639] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,640] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:13:11,645] jepsen worker 84 - jepsen.util 2514 :ok :read [28 44] INFO [2022-03-13 17:13:11,645] jepsen worker 57 - jepsen.util 2307 :ok :read [19 36] INFO [2022-03-13 17:13:11,646] jepsen worker 84 - jepsen.util 2514 :invoke :write [28 34] INFO [2022-03-13 17:13:11,646] jepsen worker 57 - jepsen.util 2307 :invoke :read [19 nil] INFO [2022-03-13 17:13:11,649] jepsen worker 58 - jepsen.util 2218 :ok :read [19 36] INFO [2022-03-13 17:13:11,649] jepsen worker 58 - jepsen.util 2218 :invoke :read [19 nil] INFO [2022-03-13 17:13:11,652] jepsen worker 16 - jepsen.util 2086 :ok :read [5 43] INFO [2022-03-13 17:13:11,652] jepsen worker 16 - jepsen.util 2086 :invoke :read [5 nil] INFO [2022-03-13 17:13:11,656] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,656] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,656] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:11,657] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:13:11,660] jepsen worker 12 - jepsen.util 732 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,660] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:11,661] jepsen worker 44 - jepsen.util 584 :invoke :write [14 5] INFO [2022-03-13 17:13:11,661] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,662] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,667] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,670] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:13:11,671] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:11,672] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:11,672] jepsen worker 51 - jepsen.util 591 :invoke :write [17 43] INFO [2022-03-13 17:13:11,676] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,681] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:11,681] jepsen worker 57 - jepsen.util 2307 :ok :read [19 36] INFO [2022-03-13 17:13:11,682] jepsen worker 57 - jepsen.util 2307 :invoke :read [19 nil] INFO [2022-03-13 17:13:11,684] jepsen worker 58 - jepsen.util 2218 :ok :read [19 36] INFO [2022-03-13 17:13:11,684] jepsen worker 58 - jepsen.util 2218 :invoke :read [19 nil] INFO [2022-03-13 17:13:11,696] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,697] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,697] jepsen worker 16 - jepsen.util 2086 :ok :read [5 43] INFO [2022-03-13 17:13:11,698] jepsen worker 16 - jepsen.util 2086 :invoke :write [5 9] INFO [2022-03-13 17:13:11,737] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,737] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,738] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:11,738] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:13:11,748] jepsen worker 88 - jepsen.util 1978 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,749] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:11,749] jepsen worker 57 - jepsen.util 2307 :ok :read [19 36] INFO [2022-03-13 17:13:11,751] jepsen worker 57 - jepsen.util 2307 :invoke :read [19 nil] INFO [2022-03-13 17:13:11,758] jepsen worker 36 - jepsen.util 2106 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,758] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 2] INFO [2022-03-13 17:13:11,760] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,760] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,761] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,762] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:13:11,769] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,770] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:11,771] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,771] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:13:11,786] jepsen worker 58 - jepsen.util 2218 :ok :read [19 36] INFO [2022-03-13 17:13:11,787] jepsen worker 58 - jepsen.util 2218 :invoke :write [19 12] INFO [2022-03-13 17:13:11,800] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:11,801] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:13:11,806] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,806] jepsen worker 70 - jepsen.util 700 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,807] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:11,808] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:11,808] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,809] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:11,810] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,810] jepsen worker 22 - jepsen.util 652 :invoke :write [7 8] INFO [2022-03-13 17:13:11,817] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,817] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,817] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,818] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:13:11,821] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:13:11,821] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:11,842] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,843] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:11,849] jepsen worker 57 - jepsen.util 2307 :ok :read [19 36] INFO [2022-03-13 17:13:11,850] jepsen worker 57 - jepsen.util 2307 :invoke :write [19 38] INFO [2022-03-13 17:13:11,854] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,855] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,855] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:13:11,856] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:11,858] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,859] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,862] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,863] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:11,865] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,866] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:11,868] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,869] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 23] INFO [2022-03-13 17:13:11,882] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,882] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,884] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:13:11,889] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:11,893] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:11,894] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:13:11,903] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,903] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:11,903] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:11,904] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:13:11,905] jepsen worker 7 - jepsen.util 637 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,906] jepsen worker 7 - jepsen.util 637 :invoke :write [2 44] INFO [2022-03-13 17:13:11,919] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,919] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,920] jepsen worker 24 - jepsen.util 564 :invoke :write [8 30] INFO [2022-03-13 17:13:11,920] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:11,949] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,950] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:13:11,965] jepsen worker 36 - jepsen.util 2106 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,965] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 17] INFO [2022-03-13 17:13:11,998] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:11,999] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:13:12,031] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:12,032] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,055] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,056] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,059] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,059] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,061] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,062] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:13:12,068] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,069] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,096] jepsen worker 3 - jepsen.util 1893 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,097] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,127] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,128] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,158] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,159] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,171] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:12,171] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,191] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:12,192] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:13:12,193] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,193] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:13:12,203] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,204] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:12,220] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:12,221] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:13:12,240] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,241] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,271] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,271] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:12,275] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:12,276] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:13:12,300] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,300] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:13:12,337] jepsen worker 36 - jepsen.util 2106 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,337] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,337] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,337] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:13:12,339] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,339] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,344] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,345] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:13:12,347] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,348] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,348] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:13:12,349] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:13:12,362] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:12,362] jepsen worker nemesis - couchbase.util Rebalance check count: 302 INFO [2022-03-13 17:13:12,362] jepsen worker nemesis - couchbase.util Rebalance stuck count: 203 INFO [2022-03-13 17:13:12,399] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,399] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,403] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,403] jepsen worker 21 - jepsen.util 831 :invoke :write [7 36] INFO [2022-03-13 17:13:12,429] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,430] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,430] jepsen worker 52 - jepsen.util 592 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,430] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:12,431] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:12,431] jepsen worker 78 - jepsen.util 528 :invoke :write [26 24] INFO [2022-03-13 17:13:12,449] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,450] jepsen worker 55 - jepsen.util 2215 :info :write [18 45] :RequestTimeoutException INFO [2022-03-13 17:13:12,450] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,450] jepsen worker 55 - jepsen.util 2305 :invoke :write [18 22] INFO [2022-03-13 17:13:12,452] jepsen worker 22 - jepsen.util 652 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,452] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,453] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:12,453] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:13:12,455] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,456] jepsen worker 80 - jepsen.util 710 :invoke :write [26 26] INFO [2022-03-13 17:13:12,458] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,459] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,460] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,461] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,462] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,463] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,463] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,463] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,463] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,464] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,465] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:13:12,465] jepsen worker 13 - jepsen.util 643 :invoke :write [4 5] INFO [2022-03-13 17:13:12,466] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:12,467] jepsen worker 34 - jepsen.util 754 :invoke :write [11 4] INFO [2022-03-13 17:13:12,485] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,486] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:12,503] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,503] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,504] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 41] INFO [2022-03-13 17:13:12,504] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,510] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,511] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:12,521] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,521] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,522] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:12,523] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:13:12,523] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,523] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:13:12,546] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,546] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:12,546] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:12,547] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 47] INFO [2022-03-13 17:13:12,558] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,558] jepsen worker 24 - jepsen.util 564 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,558] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,559] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:12,559] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:12,560] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:13:12,571] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,571] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 48] INFO [2022-03-13 17:13:12,599] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,599] jepsen worker 78 - jepsen.util 528 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,599] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:12,600] jepsen worker 78 - jepsen.util 528 :invoke :write [26 40] INFO [2022-03-13 17:13:12,603] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,604] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,612] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,612] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:12,619] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,619] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,619] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,621] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,622] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,622] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,629] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,630] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:12,662] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,662] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,662] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,663] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:13:12,664] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:13:12,665] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:12,670] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:12,670] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,678] jepsen worker 62 - jepsen.util 512 :fail :write [20 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,678] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,678] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:12,679] jepsen worker 62 - jepsen.util 512 :invoke :write [20 5] INFO [2022-03-13 17:13:12,679] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:12,680] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:12,685] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,685] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,686] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:13:12,687] jepsen worker 22 - jepsen.util 652 :invoke :write [7 7] INFO [2022-03-13 17:13:12,711] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,713] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:13:12,718] jepsen worker 88 - jepsen.util 1978 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,720] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,720] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,720] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,721] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 11] INFO [2022-03-13 17:13:12,722] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:13:12,724] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,727] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,729] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,730] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:13:12,741] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:12,742] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:13:12,755] jepsen worker 80 - jepsen.util 710 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,756] jepsen worker 80 - jepsen.util 710 :invoke :write [26 3] INFO [2022-03-13 17:13:12,757] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,758] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:12,760] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,764] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:12,765] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:12,766] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:12,773] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,773] jepsen worker 21 - jepsen.util 831 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,775] jepsen worker 21 - jepsen.util 831 :invoke :write [7 1] INFO [2022-03-13 17:13:12,775] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:12,801] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,802] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:12,810] jepsen worker 36 - jepsen.util 2106 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,810] jepsen worker 77 - jepsen.util 707 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,811] jepsen worker 77 - jepsen.util 707 :invoke :write [25 26] INFO [2022-03-13 17:13:12,812] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:12,828] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,828] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:13:12,867] jepsen worker 79 - jepsen.util 1069 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,867] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:12,872] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,873] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:13:12,882] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,882] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,882] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:12,883] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:12,890] jepsen worker 88 - jepsen.util 1978 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,890] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 4] INFO [2022-03-13 17:13:12,913] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,914] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:12,952] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:12,953] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:13:12,956] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,956] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:12,994] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,994] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,995] jepsen worker 22 - jepsen.util 652 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:12,995] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:13:12,999] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:12,999] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,015] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,016] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,020] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,021] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,033] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,033] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,034] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,035] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,036] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,036] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,044] jepsen worker 37 - jepsen.util 2017 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,044] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,045] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,045] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:13:13,061] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,061] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,062] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:13:13,062] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:13:13,111] jepsen worker 76 - jepsen.util 706 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,111] jepsen worker 21 - jepsen.util 831 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,112] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,112] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,112] jepsen worker 21 - jepsen.util 831 :invoke :write [7 9] INFO [2022-03-13 17:13:13,113] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:13:13,114] jepsen worker 76 - jepsen.util 706 :invoke :write [25 10] INFO [2022-03-13 17:13:13,118] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:13:13,118] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,118] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,118] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,119] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:13:13,120] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:13:13,133] jepsen worker 88 - jepsen.util 1978 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,133] jepsen worker 24 - jepsen.util 564 :invoke :write [8 3] INFO [2022-03-13 17:13:13,133] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,133] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,133] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,134] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:13:13,134] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:13:13,135] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:13:13,137] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,141] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:13:13,146] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:13,149] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,160] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:13,194] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,194] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,196] jepsen worker 34 - jepsen.util 754 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,197] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:13,224] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,225] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:13,246] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,247] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,265] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,265] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,266] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:13:13,266] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,286] jepsen worker 21 - jepsen.util 831 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,287] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:13:13,294] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:13,294] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,302] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,302] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:13:13,304] jepsen worker 31 - jepsen.util 2461 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:13:13,305] jepsen worker 31 - jepsen.util 2551 :invoke :read [10 nil] INFO [2022-03-13 17:13:13,320] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,320] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,320] jepsen worker 23 - jepsen.util 563 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,321] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:13,321] jepsen worker 12 - jepsen.util 732 :invoke :write [4 35] INFO [2022-03-13 17:13:13,322] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:13,328] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,328] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,329] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,329] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,338] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,338] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,339] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,339] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,346] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,347] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,347] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:13:13,347] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,351] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:13,352] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:13:13,352] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,353] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,354] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,354] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,355] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,356] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:13:13,362] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,362] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 2] INFO [2022-03-13 17:13:13,366] jepsen worker 19 - jepsen.util 2179 :info :write [6 45] :RequestTimeoutException INFO [2022-03-13 17:13:13,368] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:13,368] jepsen worker nemesis - couchbase.util Rebalance check count: 303 INFO [2022-03-13 17:13:13,368] jepsen worker nemesis - couchbase.util Rebalance stuck count: 204 INFO [2022-03-13 17:13:13,368] jepsen worker 19 - jepsen.util 2269 :invoke :write [6 25] INFO [2022-03-13 17:13:13,369] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,370] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,411] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:13,412] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:13,417] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:13,418] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,423] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,424] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:13:13,424] jepsen worker 53 - jepsen.util 503 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,424] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,425] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,425] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:13,426] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:13,426] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:13,428] jepsen worker 13 - jepsen.util 643 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,428] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,428] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,429] jepsen worker 13 - jepsen.util 643 :invoke :write [4 0] INFO [2022-03-13 17:13:13,429] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:13,429] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:13,431] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,431] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:13:13,443] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,444] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:13:13,460] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,460] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:13:13,484] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,484] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,490] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,491] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:13,493] jepsen worker 78 - jepsen.util 528 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,493] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:13,494] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,497] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:13,524] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:13,524] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:13:13,559] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,559] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,564] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,565] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,611] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,612] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:13,617] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,617] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,621] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,622] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:13,651] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,652] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:13:13,666] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:13,667] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,681] jepsen worker 67 - jepsen.util 1867 :info :write [22 9] :RequestTimeoutException INFO [2022-03-13 17:13:13,681] jepsen worker 67 - jepsen.util 1957 :invoke :read [22 nil] INFO [2022-03-13 17:13:13,684] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,684] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 33] INFO [2022-03-13 17:13:13,687] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,688] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,700] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,701] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:13:13,711] jepsen worker 36 - jepsen.util 2106 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,711] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,711] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,712] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,712] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:13:13,713] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,713] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,713] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 20] INFO [2022-03-13 17:13:13,715] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,716] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,717] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,717] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,717] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,719] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:13:13,719] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:13:13,728] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:13:13,738] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:13,738] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:13:13,743] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,743] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,743] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,744] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:13,745] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,745] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,747] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,747] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,749] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,750] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:13,750] jepsen worker 80 - jepsen.util 710 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,750] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,750] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,750] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,751] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:13,752] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:13,753] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:13,753] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:13,758] jepsen worker 3 - jepsen.util 1893 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,758] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,766] jepsen worker 44 - jepsen.util 584 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,766] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,767] jepsen worker 60 - jepsen.util 600 :invoke :write [20 48] INFO [2022-03-13 17:13:13,774] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:13,793] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,794] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:13:13,811] jepsen worker 1 - jepsen.util 2071 :info :write [0 31] :RequestTimeoutException INFO [2022-03-13 17:13:13,811] jepsen worker 1 - jepsen.util 2161 :invoke :read [0 nil] INFO [2022-03-13 17:13:13,812] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,813] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 7] INFO [2022-03-13 17:13:13,819] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,819] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:13,820] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,821] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:13,822] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,823] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:13:13,823] jepsen worker 36 - jepsen.util 2106 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,824] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 9] INFO [2022-03-13 17:13:13,824] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,825] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,825] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,825] jepsen worker 39 - jepsen.util 2199 :info :write [13 36] :RequestTimeoutException INFO [2022-03-13 17:13:13,826] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:13,826] jepsen worker 39 - jepsen.util 2289 :invoke :write [13 22] INFO [2022-03-13 17:13:13,833] jepsen worker 24 - jepsen.util 564 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,834] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:13:13,846] jepsen worker 8 - jepsen.util 728 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,846] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,847] jepsen worker 8 - jepsen.util 728 :invoke :write [2 23] INFO [2022-03-13 17:13:13,849] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:13,849] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:13,850] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:13,856] jepsen worker 50 - jepsen.util 2390 :info :write [16 10] :RequestTimeoutException INFO [2022-03-13 17:13:13,857] jepsen worker 50 - jepsen.util 2480 :invoke :write [16 34] INFO [2022-03-13 17:13:13,858] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,858] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:13,860] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,860] jepsen worker 21 - jepsen.util 831 :invoke :write [7 28] INFO [2022-03-13 17:13:13,862] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,862] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,862] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:13:13,864] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:13:13,870] jepsen worker 81 - jepsen.util 2601 :info :write [27 35] :RequestTimeoutException INFO [2022-03-13 17:13:13,871] jepsen worker 81 - jepsen.util 2691 :invoke :read [27 nil] INFO [2022-03-13 17:13:13,876] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,877] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,890] jepsen worker 15 - jepsen.util 2355 :info :write [5 37] :RequestTimeoutException INFO [2022-03-13 17:13:13,891] jepsen worker 15 - jepsen.util 2445 :invoke :read [5 nil] INFO [2022-03-13 17:13:13,897] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,897] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,897] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,898] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:13,905] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,906] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:13,919] jepsen worker 13 - jepsen.util 643 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,920] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,920] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,920] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,920] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,919] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,921] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:13,921] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:13,922] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:13,922] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,923] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,923] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:13,923] jepsen worker 34 - jepsen.util 754 :invoke :write [11 14] INFO [2022-03-13 17:13:13,924] jepsen worker 70 - jepsen.util 700 :invoke :write [23 9] INFO [2022-03-13 17:13:13,924] jepsen worker 78 - jepsen.util 528 :invoke :write [26 10] INFO [2022-03-13 17:13:13,926] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:13,929] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,929] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,929] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,930] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,930] jepsen worker 51 - jepsen.util 591 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,931] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:13,931] jepsen worker 80 - jepsen.util 710 :invoke :write [26 40] INFO [2022-03-13 17:13:13,932] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:13,932] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:13,933] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:13,937] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,938] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:13,954] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:13,955] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 16] INFO [2022-03-13 17:13:13,956] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,956] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,957] jepsen worker 52 - jepsen.util 592 :invoke :write [17 23] INFO [2022-03-13 17:13:13,958] jepsen worker 13 - jepsen.util 643 :invoke :write [4 44] INFO [2022-03-13 17:13:13,960] jepsen worker 62 - jepsen.util 512 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,961] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,960] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,961] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 48] INFO [2022-03-13 17:13:13,961] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:13,966] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:13:13,970] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,976] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,978] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,979] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:13:13,989] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,990] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:13,992] jepsen worker 88 - jepsen.util 1978 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:13,993] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 40] INFO [2022-03-13 17:13:14,000] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:14,001] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:13:14,056] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,056] jepsen worker 36 - jepsen.util 2106 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,056] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,057] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:13:14,057] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,058] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:14,059] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,059] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:14,067] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,068] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:13:14,076] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,076] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:14,078] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,078] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:13:14,090] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,091] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,092] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,092] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:13:14,102] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,102] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,102] jepsen worker 89 - jepsen.util 2429 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,103] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,104] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 45] INFO [2022-03-13 17:13:14,104] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,105] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:13:14,106] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:14,118] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,119] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,119] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,120] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:14,120] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:13:14,121] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:14,126] jepsen worker 86 - jepsen.util 2066 :info :write [28 2] :RequestTimeoutException INFO [2022-03-13 17:13:14,127] jepsen worker 86 - jepsen.util 2156 :invoke :write [28 7] INFO [2022-03-13 17:13:14,139] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:14,139] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,156] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,157] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 7] INFO [2022-03-13 17:13:14,166] jepsen worker 2 - jepsen.util 2612 :info :write [0 47] :RequestTimeoutException INFO [2022-03-13 17:13:14,167] jepsen worker 2 - jepsen.util 2702 :invoke :read [0 nil] INFO [2022-03-13 17:13:14,173] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,174] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,191] jepsen worker 88 - jepsen.util 1978 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,191] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,196] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:14,196] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:13:14,199] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,199] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,200] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,200] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:13:14,201] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:13:14,201] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:14,202] jepsen worker 77 - jepsen.util 707 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,203] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:14,203] jepsen worker 34 - jepsen.util 754 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,204] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:14,212] jepsen worker 21 - jepsen.util 831 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,213] jepsen worker 21 - jepsen.util 831 :invoke :write [7 48] INFO [2022-03-13 17:13:14,229] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,230] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:13:14,253] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:14,254] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 17] INFO [2022-03-13 17:13:14,260] jepsen worker 37 - jepsen.util 2017 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,260] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 1] INFO [2022-03-13 17:13:14,291] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,291] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,291] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:14,292] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:13:14,298] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,298] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:13:14,303] jepsen worker 61 - jepsen.util 601 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,304] jepsen worker 61 - jepsen.util 601 :invoke :write [20 43] INFO [2022-03-13 17:13:14,315] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,316] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:13:14,323] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,323] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,323] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:14,324] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:14,353] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,354] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:13:14,356] jepsen worker 70 - jepsen.util 700 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,356] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:14,370] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,370] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,370] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:14,371] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,374] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:14,374] jepsen worker nemesis - couchbase.util Rebalance check count: 304 INFO [2022-03-13 17:13:14,374] jepsen worker nemesis - couchbase.util Rebalance stuck count: 205 INFO [2022-03-13 17:13:14,387] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,387] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,388] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:13:14,390] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:14,406] jepsen worker 3 - jepsen.util 1893 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,407] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:14,412] jepsen worker 88 - jepsen.util 1978 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,412] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:13:14,436] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,437] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,463] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:14,464] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,501] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,501] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,501] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,501] jepsen worker 21 - jepsen.util 831 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,501] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,501] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:13:14,502] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:14,503] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:13:14,503] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:13:14,503] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,504] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:13:14,505] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:13:14,509] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,509] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,513] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,514] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:13:14,518] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,518] jepsen worker 37 - jepsen.util 2017 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,519] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,519] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,519] jepsen worker 76 - jepsen.util 706 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,519] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,520] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:14,522] jepsen worker 76 - jepsen.util 706 :invoke :write [25 10] INFO [2022-03-13 17:13:14,522] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:14,523] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,523] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,523] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:14,525] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,526] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,526] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 12] INFO [2022-03-13 17:13:14,526] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,527] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:13:14,534] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:13:14,568] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:14,569] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,580] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,580] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:13:14,581] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,581] jepsen worker 80 - jepsen.util 710 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,582] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:14,582] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:14,605] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:14,605] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 37] INFO [2022-03-13 17:13:14,611] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,611] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:13:14,656] jepsen worker 7 - jepsen.util 637 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,656] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,656] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:14,656] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,656] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:14,656] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,656] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,657] jepsen worker 42 - jepsen.util 402 :invoke :write [14 12] INFO [2022-03-13 17:13:14,658] jepsen worker 14 - jepsen.util 284 :invoke :write [4 0] INFO [2022-03-13 17:13:14,659] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,659] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:14,659] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,660] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:14,660] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:14,660] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:14,661] jepsen worker 26 - jepsen.util 476 :invoke :write [8 38] INFO [2022-03-13 17:13:14,661] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:14,666] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:14,671] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,672] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,692] jepsen worker 78 - jepsen.util 528 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,693] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:14,696] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,696] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:13:14,736] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,736] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,737] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:14,737] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,738] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,739] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:13:14,753] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,753] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,753] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:13:14,755] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:14,780] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:14,780] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,796] jepsen worker 36 - jepsen.util 2106 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,796] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:13:14,806] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,807] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,811] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,812] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,815] jepsen worker 13 - jepsen.util 643 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,815] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,816] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,817] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:14,818] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,818] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 34] INFO [2022-03-13 17:13:14,819] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,820] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:14,821] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,822] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,824] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,825] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:14,832] jepsen worker 12 - jepsen.util 732 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,832] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:14,832] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:14,833] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:14,844] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:14,845] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 2] INFO [2022-03-13 17:13:14,854] jepsen worker 89 - jepsen.util 2429 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,854] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:13:14,920] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,921] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,923] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,924] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,925] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,925] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:14,939] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,940] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:14,941] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,942] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:14,958] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,958] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:14,968] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,968] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,968] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,968] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:13:14,969] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:13:14,969] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:13:14,974] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,974] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,975] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,975] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:13:14,977] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:13:14,979] jepsen worker 35 - jepsen.util 935 :invoke :write [11 15] INFO [2022-03-13 17:13:14,984] jepsen worker 88 - jepsen.util 1978 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,985] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:14,987] jepsen worker 58 - jepsen.util 2218 :ok :write [19 12] INFO [2022-03-13 17:13:14,988] jepsen worker 58 - jepsen.util 2218 :invoke :read [19 nil] INFO [2022-03-13 17:13:14,988] jepsen worker 8 - jepsen.util 728 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,988] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:14,990] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:14,991] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:15,035] jepsen worker 52 - jepsen.util 592 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,035] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,035] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,036] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,036] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,036] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,037] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:15,038] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:15,039] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:15,039] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,040] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:15,040] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 24] INFO [2022-03-13 17:13:15,041] jepsen worker 78 - jepsen.util 528 :invoke :write [26 20] INFO [2022-03-13 17:13:15,041] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:15,059] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,060] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,068] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,068] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,069] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,069] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,070] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 31] INFO [2022-03-13 17:13:15,070] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:13:15,097] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,098] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,111] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,114] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:15,134] jepsen worker 67 - jepsen.util 1957 :ok :read [22 30] INFO [2022-03-13 17:13:15,135] jepsen worker 67 - jepsen.util 1957 :invoke :write [22 33] INFO [2022-03-13 17:13:15,137] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,137] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:15,150] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,150] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,150] jepsen worker 87 - jepsen.util 2607 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,151] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,151] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,152] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,152] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:13:15,153] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,153] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,153] jepsen worker 60 - jepsen.util 600 :fail :write [20 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,154] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,155] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:13:15,155] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:15,156] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,157] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,157] jepsen worker 51 - jepsen.util 591 :invoke :write [17 24] INFO [2022-03-13 17:13:15,157] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,157] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,158] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:13:15,159] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,159] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:13:15,160] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:15,161] jepsen worker 69 - jepsen.util 699 :invoke :write [23 42] INFO [2022-03-13 17:13:15,162] jepsen worker 60 - jepsen.util 600 :invoke :write [20 36] INFO [2022-03-13 17:13:15,163] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,164] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 7] INFO [2022-03-13 17:13:15,164] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,164] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,164] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:15,165] jepsen worker 52 - jepsen.util 592 :invoke :write [17 47] INFO [2022-03-13 17:13:15,165] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:15,166] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,167] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,167] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,168] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:15,169] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:15,169] jepsen worker 70 - jepsen.util 700 :invoke :write [23 31] INFO [2022-03-13 17:13:15,169] jepsen worker 80 - jepsen.util 710 :invoke :write [26 11] INFO [2022-03-13 17:13:15,184] jepsen worker 88 - jepsen.util 1978 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,185] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,186] jepsen worker 79 - jepsen.util 1069 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,187] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 31] INFO [2022-03-13 17:13:15,198] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:15,198] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:13:15,207] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,208] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:13:15,242] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,243] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,245] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,246] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:15,254] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:15,254] jepsen worker 76 - jepsen.util 706 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,255] jepsen worker 76 - jepsen.util 706 :invoke :write [25 9] INFO [2022-03-13 17:13:15,255] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,255] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,255] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:15,256] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:15,256] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:13:15,257] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,258] jepsen worker 12 - jepsen.util 732 :invoke :write [4 14] INFO [2022-03-13 17:13:15,277] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,278] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,300] jepsen worker 37 - jepsen.util 2017 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,301] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,307] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,308] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:13:15,310] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,311] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:13:15,311] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,311] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,317] jepsen worker 78 - jepsen.util 528 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,317] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,317] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,317] jepsen worker 78 - jepsen.util 528 :invoke :write [26 49] INFO [2022-03-13 17:13:15,318] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:15,318] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:15,322] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,322] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,338] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,338] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,342] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,342] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:15,370] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,370] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:13:15,374] jepsen worker 61 - jepsen.util 601 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,376] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,376] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:15,376] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,377] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,377] jepsen worker 61 - jepsen.util 601 :invoke :write [20 18] INFO [2022-03-13 17:13:15,378] jepsen worker 43 - jepsen.util 493 :invoke :write [14 34] INFO [2022-03-13 17:13:15,378] jepsen worker 53 - jepsen.util 503 :invoke :write [17 15] INFO [2022-03-13 17:13:15,379] jepsen worker 6 - jepsen.util 636 :invoke :write [2 43] INFO [2022-03-13 17:13:15,380] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:15,381] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:15,381] jepsen worker nemesis - couchbase.util Rebalance check count: 305 INFO [2022-03-13 17:13:15,381] jepsen worker nemesis - couchbase.util Rebalance stuck count: 206 INFO [2022-03-13 17:13:15,420] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,421] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,441] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,441] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:13:15,443] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,444] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,446] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,446] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:13:15,459] jepsen worker 41 - jepsen.util 2201 :ok :read [13 39] INFO [2022-03-13 17:13:15,460] jepsen worker 41 - jepsen.util 2201 :invoke :write [13 2] INFO [2022-03-13 17:13:15,484] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,484] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,485] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:13:15,485] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:13:15,491] jepsen worker 35 - jepsen.util 935 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,492] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:15,515] jepsen worker 81 - jepsen.util 2691 :ok :read [27 4] INFO [2022-03-13 17:13:15,516] jepsen worker 81 - jepsen.util 2691 :invoke :write [27 47] INFO [2022-03-13 17:13:15,524] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,524] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,525] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,526] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:13:15,557] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,557] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:13:15,587] jepsen worker 68 - jepsen.util 2498 :ok :read [22 30] INFO [2022-03-13 17:13:15,588] jepsen worker 68 - jepsen.util 2498 :invoke :read [22 nil] INFO [2022-03-13 17:13:15,591] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,591] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,591] jepsen worker 36 - jepsen.util 2106 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,591] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:13:15,592] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:13:15,600] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 23] INFO [2022-03-13 17:13:15,609] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,609] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,610] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,610] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,632] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,632] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:15,672] jepsen worker 52 - jepsen.util 592 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,672] jepsen worker 60 - jepsen.util 600 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,672] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,673] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:13:15,673] jepsen worker 52 - jepsen.util 592 :invoke :write [17 0] INFO [2022-03-13 17:13:15,674] jepsen worker 60 - jepsen.util 600 :invoke :write [20 25] INFO [2022-03-13 17:13:15,675] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,675] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,676] jepsen worker 82 - jepsen.util 2242 :info :write [27 36] :RequestTimeoutException INFO [2022-03-13 17:13:15,677] jepsen worker 82 - jepsen.util 2332 :invoke :write [27 8] INFO [2022-03-13 17:13:15,682] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,682] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,683] jepsen worker 33 - jepsen.util 663 :invoke :write [11 9] INFO [2022-03-13 17:13:15,684] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:15,696] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,696] jepsen worker 42 - jepsen.util 402 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,697] jepsen worker 42 - jepsen.util 402 :invoke :write [14 4] INFO [2022-03-13 17:13:15,698] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:13:15,699] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,699] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,699] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,699] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,700] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,701] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,710] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,711] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,712] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,712] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,713] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:13:15,716] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,729] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:15,730] jepsen worker 66 - jepsen.util 2136 :info :write [22 49] :RequestTimeoutException INFO [2022-03-13 17:13:15,731] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,731] jepsen worker 66 - jepsen.util 2226 :invoke :read [22 nil] INFO [2022-03-13 17:13:15,735] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:15,735] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,744] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,744] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,749] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,749] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,751] jepsen worker 1 - jepsen.util 2161 :ok :read [0 41] INFO [2022-03-13 17:13:15,751] jepsen worker 1 - jepsen.util 2161 :invoke :write [0 28] INFO [2022-03-13 17:13:15,757] jepsen worker 2 - jepsen.util 2702 :ok :read [0 41] INFO [2022-03-13 17:13:15,758] jepsen worker 2 - jepsen.util 2702 :invoke :read [0 nil] INFO [2022-03-13 17:13:15,772] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,773] jepsen worker 24 - jepsen.util 564 :invoke :write [8 30] INFO [2022-03-13 17:13:15,776] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,776] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,783] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,785] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:15,793] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:15,797] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:13:15,797] jepsen worker 2 - jepsen.util 2702 :ok :read [0 41] INFO [2022-03-13 17:13:15,799] jepsen worker 2 - jepsen.util 2702 :invoke :write [0 36] INFO [2022-03-13 17:13:15,810] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,812] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:13:15,819] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,820] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,831] jepsen worker 29 - jepsen.util 2549 :ok :read [9 26] INFO [2022-03-13 17:13:15,831] jepsen worker 29 - jepsen.util 2549 :invoke :read [9 nil] INFO [2022-03-13 17:13:15,833] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,833] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,833] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:13:15,834] jepsen worker 71 - jepsen.util 521 :invoke :write [23 47] INFO [2022-03-13 17:13:15,856] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,856] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,856] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,857] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,870] jepsen worker 10 - jepsen.util 2440 :ok :read [3 25] INFO [2022-03-13 17:13:15,871] jepsen worker 10 - jepsen.util 2440 :invoke :write [3 24] INFO [2022-03-13 17:13:15,875] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,875] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,876] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:15,877] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:15,892] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,892] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,893] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:15,893] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:13:15,895] jepsen worker 80 - jepsen.util 710 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,896] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:15,924] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,925] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,927] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,928] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,934] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,934] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,934] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,934] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,935] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,936] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:13:15,937] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,937] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:15,938] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,939] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:13:15,939] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,940] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,941] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,941] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,942] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:13:15,942] jepsen worker 33 - jepsen.util 663 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,942] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,943] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:15,944] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,944] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,944] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:13:15,945] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,946] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,946] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,948] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,949] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:15,950] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,951] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:13:15,969] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:15,969] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 7] INFO [2022-03-13 17:13:15,993] jepsen worker 36 - jepsen.util 2106 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:15,994] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,020] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:16,021] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:13:16,045] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,046] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:16,069] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,070] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,088] jepsen worker 69 - jepsen.util 699 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,088] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,089] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:16,089] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:13:16,101] jepsen worker 88 - jepsen.util 1978 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,103] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:13:16,105] jepsen worker 48 - jepsen.util 2298 :ok :read [16 :nil] INFO [2022-03-13 17:13:16,106] jepsen worker 24 - jepsen.util 564 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,106] jepsen worker 48 - jepsen.util 2298 :invoke :write [16 24] INFO [2022-03-13 17:13:16,106] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:16,120] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,121] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:13:16,125] jepsen worker 17 - jepsen.util 2357 :ok :read [5 43] INFO [2022-03-13 17:13:16,125] jepsen worker 17 - jepsen.util 2357 :invoke :read [5 nil] INFO [2022-03-13 17:13:16,129] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,130] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] WARN [2022-03-13 17:13:16,159] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":25415062,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25415062,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9045859,"encode_duration_us":23713,"operation_id":"0x75b9","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10000674},{"operation_name":"upsert","last_dispatch_duration_us":9279045,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9279045,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9038428,"encode_duration_us":17090,"operation_id":"0x7ce8","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10001066},{"operation_name":"upsert","last_dispatch_duration_us":18034234,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18034234,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9045859,"encode_duration_us":31502,"operation_id":"0x7643","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007560},{"operation_name":"upsert","last_dispatch_duration_us":18129554,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27698046,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9087806,"encode_duration_us":25058,"operation_id":"0x73b9","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10007046}],"total_count":4}} WARN [2022-03-13 17:13:16,160] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":18405,"operation_id":"0x7a8d","timeout_ms":10000,"total_duration_us":10010398},{"operation_name":"upsert","last_dispatch_duration_us":9000242,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9000242,"total_server_duration_us":21324,"encode_duration_us":19226,"operation_id":"0x7e4a","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10008626},{"operation_name":"upsert","last_dispatch_duration_us":8980364,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8980364,"total_server_duration_us":21487,"encode_duration_us":19439,"operation_id":"0x7e39","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10006564},{"operation_name":"upsert","last_dispatch_duration_us":8986367,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8986367,"total_server_duration_us":20680,"encode_duration_us":19507,"operation_id":"0x7e49","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10006431},{"operation_name":"upsert","last_dispatch_duration_us":122937,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":122937,"total_server_duration_us":25537,"encode_duration_us":20241,"operation_id":"0x7e2b","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10005549},{"operation_name":"upsert","last_dispatch_duration_us":9279045,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9279045,"total_server_duration_us":9038428,"encode_duration_us":17090,"operation_id":"0x7ce8","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10001066},{"operation_name":"get","last_dispatch_duration_us":9147334,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9147334,"total_server_duration_us":76,"operation_id":"0x7d3e","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":9154452},{"operation_name":"upsert","last_dispatch_duration_us":494091,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1507814,"total_server_duration_us":170369,"encode_duration_us":21309,"operation_id":"0x805a","timeout_ms":10000,"last_server_duration_us":47647,"total_duration_us":1557506},{"operation_name":"upsert","last_dispatch_duration_us":517094,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1498590,"total_server_duration_us":219281,"encode_duration_us":17755,"operation_id":"0x805b","timeout_ms":10000,"last_server_duration_us":73136,"total_duration_us":1525568},{"operation_name":"upsert","last_dispatch_duration_us":484438,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":801551,"total_server_duration_us":371972,"encode_duration_us":25439,"operation_id":"0x8093","timeout_ms":10000,"last_server_duration_us":308017,"total_duration_us":812890}],"total_count":22}} INFO [2022-03-13 17:13:16,165] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,165] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:13:16,190] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,190] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,191] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,191] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,191] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,191] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:16,192] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,192] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,193] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,193] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:13:16,193] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,194] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,194] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,195] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,196] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,196] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,197] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,197] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:13:16,198] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,199] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 19] INFO [2022-03-13 17:13:16,201] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,202] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:13:16,215] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,216] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 21] INFO [2022-03-13 17:13:16,258] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,258] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 23] INFO [2022-03-13 17:13:16,288] jepsen worker 71 - jepsen.util 521 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,288] jepsen worker 71 - jepsen.util 521 :invoke :write [23 13] INFO [2022-03-13 17:13:16,295] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,295] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,295] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:16,295] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:13:16,299] jepsen worker 14 - jepsen.util 284 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,299] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,299] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,300] jepsen worker 49 - jepsen.util 1939 :info :write [16 16] :RequestTimeoutException INFO [2022-03-13 17:13:16,300] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:16,300] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:16,301] jepsen worker 49 - jepsen.util 2029 :invoke :write [16 25] INFO [2022-03-13 17:13:16,301] jepsen worker 8 - jepsen.util 728 :invoke :write [2 38] INFO [2022-03-13 17:13:16,313] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,314] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,319] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,319] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:13:16,326] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,326] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,326] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,327] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:13:16,330] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:13:16,331] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:13:16,337] jepsen worker 73 - jepsen.util 2503 :info :write [24 45] :RequestTimeoutException INFO [2022-03-13 17:13:16,337] jepsen worker 73 - jepsen.util 2593 :invoke :write [24 26] INFO [2022-03-13 17:13:16,371] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,372] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:16,380] jepsen worker 0 - jepsen.util 2250 :info :write [0 38] :RequestTimeoutException INFO [2022-03-13 17:13:16,380] jepsen worker 0 - jepsen.util 2340 :invoke :read [0 nil] INFO [2022-03-13 17:13:16,386] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:16,386] jepsen worker nemesis - couchbase.util Rebalance check count: 306 INFO [2022-03-13 17:13:16,386] jepsen worker nemesis - couchbase.util Rebalance stuck count: 207 INFO [2022-03-13 17:13:16,387] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,388] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:16,389] jepsen worker 17 - jepsen.util 2357 :ok :read [5 43] INFO [2022-03-13 17:13:16,389] jepsen worker 17 - jepsen.util 2357 :invoke :read [5 nil] INFO [2022-03-13 17:13:16,423] jepsen worker 88 - jepsen.util 1978 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,424] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 27] INFO [2022-03-13 17:13:16,477] jepsen worker 36 - jepsen.util 2106 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,478] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,488] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,488] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:16,505] jepsen worker 70 - jepsen.util 700 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,506] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,506] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,505] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,506] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:16,507] jepsen worker 13 - jepsen.util 643 :invoke :write [4 1] INFO [2022-03-13 17:13:16,507] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:16,508] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:16,509] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,509] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:13:16,514] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,515] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:13:16,521] jepsen worker 71 - jepsen.util 521 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,521] jepsen worker 71 - jepsen.util 521 :invoke :write [23 35] INFO [2022-03-13 17:13:16,528] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,528] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:16,546] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,547] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:16,572] jepsen worker 51 - jepsen.util 591 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,572] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,572] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:16,573] jepsen worker 77 - jepsen.util 707 :invoke :write [25 32] INFO [2022-03-13 17:13:16,574] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:16,575] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:16,581] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,581] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,582] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,583] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:13:16,583] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 2] INFO [2022-03-13 17:13:16,584] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,586] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,586] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,590] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,591] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,592] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,593] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,619] jepsen worker 3 - jepsen.util 1893 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,620] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 5] INFO [2022-03-13 17:13:16,620] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,621] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:13:16,641] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,641] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,642] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:13:16,642] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 19] INFO [2022-03-13 17:13:16,657] jepsen worker 17 - jepsen.util 2357 :ok :read [5 43] INFO [2022-03-13 17:13:16,659] jepsen worker 17 - jepsen.util 2357 :invoke :read [5 nil] INFO [2022-03-13 17:13:16,672] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:16,673] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 20] INFO [2022-03-13 17:13:16,676] jepsen worker 88 - jepsen.util 1978 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,677] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:16,700] jepsen worker 37 - jepsen.util 2017 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,701] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:13:16,703] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,703] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:13:16,704] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,705] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:16,715] jepsen worker 79 - jepsen.util 1069 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,716] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:16,719] jepsen worker 78 - jepsen.util 528 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,721] jepsen worker 78 - jepsen.util 528 :invoke :write [26 1] INFO [2022-03-13 17:13:16,740] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:16,740] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:13:16,744] jepsen worker 83 - jepsen.util 1703 :info :write [27 5] :RequestTimeoutException INFO [2022-03-13 17:13:16,745] jepsen worker 83 - jepsen.util 1793 :invoke :read [27 nil] INFO [2022-03-13 17:13:16,756] jepsen worker 63 - jepsen.util 2133 :info :write [21 5] :RequestTimeoutException INFO [2022-03-13 17:13:16,757] jepsen worker 63 - jepsen.util 2223 :invoke :write [21 13] INFO [2022-03-13 17:13:16,759] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,760] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:13:16,765] jepsen worker 3 - jepsen.util 1893 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,766] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:13:16,801] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,802] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:13:16,806] jepsen worker 59 - jepsen.util 2309 :info :write [19 1] :RequestTimeoutException INFO [2022-03-13 17:13:16,807] jepsen worker 59 - jepsen.util 2399 :invoke :read [19 nil] INFO [2022-03-13 17:13:16,812] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,812] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:16,821] jepsen worker 64 - jepsen.util 2404 :ok :read [21 :nil] INFO [2022-03-13 17:13:16,822] jepsen worker 64 - jepsen.util 2404 :invoke :write [21 1] INFO [2022-03-13 17:13:16,824] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,824] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:16,835] jepsen worker 18 - jepsen.util 2178 :ok :read [6 16] INFO [2022-03-13 17:13:16,835] jepsen worker 18 - jepsen.util 2178 :invoke :write [6 43] INFO [2022-03-13 17:13:16,837] jepsen worker 83 - jepsen.util 1793 :ok :read [27 4] INFO [2022-03-13 17:13:16,838] jepsen worker 83 - jepsen.util 1793 :invoke :write [27 21] INFO [2022-03-13 17:13:16,846] jepsen worker 45 - jepsen.util 2655 :ok :read [15 48] INFO [2022-03-13 17:13:16,847] jepsen worker 45 - jepsen.util 2655 :invoke :read [15 nil] INFO [2022-03-13 17:13:16,848] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,850] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:16,880] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,881] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:13:16,885] jepsen worker 17 - jepsen.util 2357 :ok :read [5 43] INFO [2022-03-13 17:13:16,886] jepsen worker 17 - jepsen.util 2357 :invoke :write [5 8] INFO [2022-03-13 17:13:16,907] jepsen worker 12 - jepsen.util 732 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,908] jepsen worker 12 - jepsen.util 732 :invoke :write [4 26] INFO [2022-03-13 17:13:16,915] jepsen worker 87 - jepsen.util 2607 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,915] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,916] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:16,916] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:16,917] jepsen worker 45 - jepsen.util 2655 :ok :read [15 48] INFO [2022-03-13 17:13:16,918] jepsen worker 45 - jepsen.util 2655 :invoke :read [15 nil] INFO [2022-03-13 17:13:16,930] jepsen worker 26 - jepsen.util 476 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,930] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,931] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:13:16,931] jepsen worker 26 - jepsen.util 476 :invoke :write [8 49] INFO [2022-03-13 17:13:16,966] jepsen worker 36 - jepsen.util 2106 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,968] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:13:16,970] jepsen worker 76 - jepsen.util 706 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:16,971] jepsen worker 76 - jepsen.util 706 :invoke :write [25 45] INFO [2022-03-13 17:13:16,987] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:16,988] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 43] INFO [2022-03-13 17:13:16,998] jepsen worker 45 - jepsen.util 2655 :ok :read [15 48] INFO [2022-03-13 17:13:16,999] jepsen worker 45 - jepsen.util 2655 :invoke :write [15 25] INFO [2022-03-13 17:13:17,048] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,048] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,049] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,049] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:13:17,051] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:17,051] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:17,065] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,065] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,066] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:13:17,066] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:13:17,082] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,082] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,083] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:17,083] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:13:17,083] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,084] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,086] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,086] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,092] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,092] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:17,108] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,108] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,108] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,108] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,109] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:17,110] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:13:17,110] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,111] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:13:17,111] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:13:17,113] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:13:17,121] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,124] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:13:17,147] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,148] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:17,173] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:17,174] jepsen worker 6 - jepsen.util 636 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,174] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,175] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:17,193] jepsen worker 88 - jepsen.util 1978 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,193] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,203] jepsen worker 71 - jepsen.util 521 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,203] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:13:17,211] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,211] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 13] INFO [2022-03-13 17:13:17,227] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,229] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:17,229] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:13:17,230] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:13:17,250] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:17,251] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,269] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,270] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,277] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,278] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:17,301] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,301] jepsen worker 52 - jepsen.util 592 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,301] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,301] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,301] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,302] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,306] jepsen worker 70 - jepsen.util 700 :invoke :write [23 47] INFO [2022-03-13 17:13:17,316] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:17,317] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:13:17,317] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:13:17,318] jepsen worker 52 - jepsen.util 592 :invoke :write [17 8] INFO [2022-03-13 17:13:17,318] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:17,325] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,326] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,328] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,329] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:13:17,369] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:17,370] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:13:17,386] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,387] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,393] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:17,393] jepsen worker nemesis - couchbase.util Rebalance check count: 307 INFO [2022-03-13 17:13:17,393] jepsen worker nemesis - couchbase.util Rebalance stuck count: 208 INFO [2022-03-13 17:13:17,394] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,394] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,395] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,396] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:13:17,423] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,424] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:13:17,449] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:17,450] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,460] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,461] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:13:17,462] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,463] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:13:17,463] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,464] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:13:17,476] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,476] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,477] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,477] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:13:17,487] jepsen worker 13 - jepsen.util 643 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,487] jepsen worker 13 - jepsen.util 643 :invoke :write [4 21] INFO [2022-03-13 17:13:17,500] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,501] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:17,501] jepsen worker 3 - jepsen.util 1893 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,502] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:17,543] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:17,543] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,545] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,546] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:17,555] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,555] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] WARN [2022-03-13 17:13:17,559] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18016025,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18775316,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9146586,"encode_duration_us":47637,"operation_id":"0x76f7","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006487},{"operation_name":"upsert","last_dispatch_duration_us":14679922,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15164996,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9335061,"encode_duration_us":20913,"operation_id":"0x7bf6","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10007291},{"operation_name":"upsert","last_dispatch_duration_us":23476711,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":32259300,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9100206,"encode_duration_us":18304,"operation_id":"0x713b","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":10007359},{"operation_name":"upsert","last_dispatch_duration_us":18033032,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18226634,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9059142,"encode_duration_us":32285,"operation_id":"0x7748","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10008702},{"operation_name":"upsert","last_dispatch_duration_us":14681765,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15295146,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9361747,"encode_duration_us":19396,"operation_id":"0x7bbb","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009611}],"total_count":5}} WARN [2022-03-13 17:13:17,573] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16438077,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17045903,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9340396,"encode_duration_us":18097,"operation_id":"0x7ac6","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10002447},{"operation_name":"upsert","last_dispatch_duration_us":23871357,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23871357,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9032060,"encode_duration_us":16831,"operation_id":"0x7542","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10003585}],"total_count":2}} INFO [2022-03-13 17:13:17,608] jepsen worker 61 - jepsen.util 601 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,609] jepsen worker 61 - jepsen.util 601 :invoke :write [20 35] INFO [2022-03-13 17:13:17,615] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,616] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,616] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:13:17,617] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:17,628] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,628] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:17,642] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,642] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,643] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 6] INFO [2022-03-13 17:13:17,643] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,648] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,650] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,650] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,651] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,655] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,656] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:17,667] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:17,668] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 38] WARN [2022-03-13 17:13:17,674] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":19899203,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":19899203,"total_server_duration_us":27,"operation_id":"0x7809","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":19901507},{"operation_name":"get","last_dispatch_duration_us":13897568,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13897568,"total_server_duration_us":13355,"operation_id":"0x7a90","timeout_ms":30000,"last_server_duration_us":13355,"total_duration_us":13900026},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":25697,"operation_id":"0x7bc0","timeout_ms":10000,"total_duration_us":10007699},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":18871,"operation_id":"0x7b45","timeout_ms":10000,"total_duration_us":10005388},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":20037,"operation_id":"0x7cdc","timeout_ms":10000,"total_duration_us":10004476},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":20514,"operation_id":"0x7cd9","timeout_ms":10000,"total_duration_us":10001463},{"operation_name":"get","last_dispatch_duration_us":9763594,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9763594,"total_server_duration_us":15693,"operation_id":"0x7f29","timeout_ms":30000,"last_server_duration_us":15693,"total_duration_us":9765651},{"operation_name":"upsert","last_dispatch_duration_us":4589244,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5614912,"total_server_duration_us":318218,"encode_duration_us":17777,"operation_id":"0x8018","timeout_ms":10000,"last_server_duration_us":201269,"total_duration_us":5630989},{"operation_name":"upsert","last_dispatch_duration_us":5062567,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5062567,"total_server_duration_us":60128,"encode_duration_us":34313,"operation_id":"0x8060","timeout_ms":10000,"last_server_duration_us":60128,"total_duration_us":5066229},{"operation_name":"upsert","last_dispatch_duration_us":4897973,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4897973,"total_server_duration_us":41337,"encode_duration_us":17741,"operation_id":"0x805f","timeout_ms":10000,"last_server_duration_us":41337,"total_duration_us":4902548}],"total_count":44}} WARN [2022-03-13 17:13:17,676] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9036523,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10191553,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9262003,"encode_duration_us":21326,"operation_id":"0x7e66","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10004324},{"operation_name":"upsert","last_dispatch_duration_us":18100069,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18100069,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9039489,"encode_duration_us":18889,"operation_id":"0x79e0","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10008963}],"total_count":2}} INFO [2022-03-13 17:13:17,684] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:17,684] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:17,689] jepsen worker 78 - jepsen.util 528 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,689] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,689] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,689] jepsen worker 78 - jepsen.util 528 :invoke :write [26 8] INFO [2022-03-13 17:13:17,690] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:17,690] jepsen worker 80 - jepsen.util 710 :invoke :write [26 10] INFO [2022-03-13 17:13:17,703] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,704] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:17,706] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,707] jepsen worker 24 - jepsen.util 564 :invoke :write [8 29] INFO [2022-03-13 17:13:17,733] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,733] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,733] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,735] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,736] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:13:17,737] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:17,740] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,740] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,740] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,741] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,741] jepsen worker 60 - jepsen.util 600 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,741] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:17,741] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:17,742] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:17,743] jepsen worker 44 - jepsen.util 584 :invoke :write [14 36] INFO [2022-03-13 17:13:17,743] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:13:17,747] jepsen worker 60 - jepsen.util 600 :invoke :write [20 46] INFO [2022-03-13 17:13:17,747] jepsen worker 51 - jepsen.util 591 :invoke :write [17 35] INFO [2022-03-13 17:13:17,755] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException WARN [2022-03-13 17:13:17,755] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":16542,"operation_id":"0x7f6d","timeout_ms":10000,"total_duration_us":10009488},{"operation_name":"upsert","last_dispatch_duration_us":17410265,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17410265,"total_server_duration_us":25800,"encode_duration_us":19074,"operation_id":"0x7b48","timeout_ms":10000,"last_server_duration_us":25800,"total_duration_us":10008807},{"operation_name":"upsert","last_dispatch_duration_us":9169362,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9169362,"total_server_duration_us":25188,"encode_duration_us":27746,"operation_id":"0x7f2e","timeout_ms":10000,"last_server_duration_us":25188,"total_duration_us":10005932},{"operation_name":"upsert","last_dispatch_duration_us":8341094,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8767365,"total_server_duration_us":264016,"encode_duration_us":20605,"operation_id":"0x7d73","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10004690},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":17757,"operation_id":"0x7f76","timeout_ms":10000,"total_duration_us":10003258},{"operation_name":"upsert","last_dispatch_duration_us":9053410,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9758321,"total_server_duration_us":9431925,"encode_duration_us":18760,"operation_id":"0x7bc3","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10003159},{"operation_name":"upsert","last_dispatch_duration_us":32407,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":851480,"total_server_duration_us":435689,"encode_duration_us":18657,"operation_id":"0x7b28","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10000876},{"operation_name":"get","last_dispatch_duration_us":8269114,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8269114,"total_server_duration_us":293,"operation_id":"0x7ff5","timeout_ms":30000,"last_server_duration_us":293,"total_duration_us":8271269},{"operation_name":"upsert","last_dispatch_duration_us":503781,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":779634,"total_server_duration_us":365965,"encode_duration_us":18377,"operation_id":"0x8095","timeout_ms":10000,"last_server_duration_us":293784,"total_duration_us":790032},{"operation_name":"upsert","last_dispatch_duration_us":733368,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":733368,"total_server_duration_us":574670,"encode_duration_us":18528,"operation_id":"0x8026","timeout_ms":10000,"last_server_duration_us":574670,"total_duration_us":737550}],"total_count":13}} INFO [2022-03-13 17:13:17,756] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:17,759] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,762] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:13:17,773] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] WARN [2022-03-13 17:13:17,777] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8903487,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10135495,"total_server_duration_us":293802,"encode_duration_us":35740,"operation_id":"0x7e46","timeout_ms":10000,"last_server_duration_us":36119,"total_duration_us":10008996},{"operation_name":"upsert","last_dispatch_duration_us":8880226,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10100056,"total_server_duration_us":312397,"encode_duration_us":27162,"operation_id":"0x7e41","timeout_ms":10000,"last_server_duration_us":37036,"total_duration_us":10008711},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":16905,"operation_id":"0x7de7","timeout_ms":10000,"total_duration_us":10007811},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":23060,"operation_id":"0x7f70","timeout_ms":10000,"total_duration_us":10006095},{"operation_name":"upsert","last_dispatch_duration_us":9036523,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10191553,"total_server_duration_us":9262003,"encode_duration_us":21326,"operation_id":"0x7e66","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10004324},{"operation_name":"upsert","last_dispatch_duration_us":8731689,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9880117,"total_server_duration_us":274594,"encode_duration_us":19311,"operation_id":"0x7e63","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10003256},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":22305,"operation_id":"0x7cc4","timeout_ms":10000,"total_duration_us":10002099},{"operation_name":"upsert","last_dispatch_duration_us":8599150,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":15419358,"total_server_duration_us":318636,"encode_duration_us":23109,"operation_id":"0x7b55","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10001373},{"operation_name":"get","last_dispatch_duration_us":1915607,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1915607,"total_server_duration_us":9227,"operation_id":"0x83bb","timeout_ms":30000,"last_server_duration_us":9227,"total_duration_us":1919446},{"operation_name":"upsert","last_dispatch_duration_us":734182,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1586641,"total_server_duration_us":153133,"encode_duration_us":19897,"operation_id":"0x8012","timeout_ms":10000,"last_server_duration_us":36017,"total_duration_us":1759672}],"total_count":32}} INFO [2022-03-13 17:13:17,778] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:13:17,810] jepsen worker 88 - jepsen.util 1978 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,810] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 44] INFO [2022-03-13 17:13:17,814] jepsen worker 36 - jepsen.util 2106 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,814] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,815] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 4] INFO [2022-03-13 17:13:17,815] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,816] jepsen worker 71 - jepsen.util 521 :invoke :write [23 26] INFO [2022-03-13 17:13:17,817] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 42] INFO [2022-03-13 17:13:17,831] jepsen worker 43 - jepsen.util 493 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,831] jepsen worker 43 - jepsen.util 493 :invoke :write [14 17] INFO [2022-03-13 17:13:17,848] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,848] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,848] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,850] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:17,851] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:17,852] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,852] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:17,852] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,853] jepsen worker 70 - jepsen.util 700 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,853] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,854] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:13:17,855] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] WARN [2022-03-13 17:13:17,855] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9053410,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9758321,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9431925,"encode_duration_us":18760,"operation_id":"0x7bc3","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10003159},{"operation_name":"upsert","last_dispatch_duration_us":18103248,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27263275,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9067508,"encode_duration_us":17388,"operation_id":"0x7658","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10003483}],"total_count":2}} INFO [2022-03-13 17:13:17,856] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:13:17,856] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:17,856] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,857] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:17,857] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,858] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,859] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,860] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] WARN [2022-03-13 17:13:17,860] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":22457062,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":22457062,"total_server_duration_us":32,"operation_id":"0x7718","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":22459435},{"operation_name":"get","last_dispatch_duration_us":14778234,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":14778234,"total_server_duration_us":27,"operation_id":"0x7ad9","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14779405},{"operation_name":"get","last_dispatch_duration_us":13527502,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13527502,"total_server_duration_us":15,"operation_id":"0x7b4f","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":13530284},{"operation_name":"upsert","last_dispatch_duration_us":14681765,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":15295146,"total_server_duration_us":9361747,"encode_duration_us":19396,"operation_id":"0x7bbb","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009611},{"operation_name":"upsert","last_dispatch_duration_us":14679922,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":15164996,"total_server_duration_us":9335061,"encode_duration_us":20913,"operation_id":"0x7bf6","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10007291},{"operation_name":"upsert","last_dispatch_duration_us":13506955,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":13506955,"total_server_duration_us":22059,"encode_duration_us":22096,"operation_id":"0x7de3","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10004596},{"operation_name":"get","last_dispatch_duration_us":9249191,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9249191,"total_server_duration_us":958,"operation_id":"0x8032","timeout_ms":30000,"last_server_duration_us":958,"total_duration_us":9249482},{"operation_name":"get","last_dispatch_duration_us":9113448,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9113448,"total_server_duration_us":27,"operation_id":"0x7d76","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9115282},{"operation_name":"get","last_dispatch_duration_us":8249384,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8249384,"total_server_duration_us":32,"operation_id":"0x80bd","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8251431},{"operation_name":"upsert","last_dispatch_duration_us":2507947,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2507947,"total_server_duration_us":81254,"encode_duration_us":31221,"operation_id":"0x7f62","timeout_ms":10000,"last_server_duration_us":81254,"total_duration_us":7682503}],"total_count":54}} INFO [2022-03-13 17:13:17,861] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,861] jepsen worker 21 - jepsen.util 831 :invoke :write [7 49] WARN [2022-03-13 17:13:17,863] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14491398,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":14491398,"total_server_duration_us":27,"operation_id":"0x7cde","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14494679},{"operation_name":"upsert","last_dispatch_duration_us":18113131,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":18113131,"total_server_duration_us":9076673,"encode_duration_us":16978,"operation_id":"0x7b3c","timeout_ms":10000,"last_server_duration_us":9076673,"total_duration_us":10009147},{"operation_name":"upsert","last_dispatch_duration_us":245234,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9964052,"total_server_duration_us":85840,"encode_duration_us":20820,"operation_id":"0x7fd0","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10009099},{"operation_name":"upsert","last_dispatch_duration_us":8218879,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8218879,"total_server_duration_us":22804,"encode_duration_us":17353,"operation_id":"0x7b97","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10007407},{"operation_name":"upsert","last_dispatch_duration_us":10265745,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10265745,"total_server_duration_us":23390,"encode_duration_us":16633,"operation_id":"0x7f27","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10002355},{"operation_name":"upsert","last_dispatch_duration_us":9094688,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9094688,"total_server_duration_us":23983,"encode_duration_us":34369,"operation_id":"0x7fc4","timeout_ms":10000,"last_server_duration_us":23983,"total_duration_us":10001958},{"operation_name":"get","last_dispatch_duration_us":9407637,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9407637,"total_server_duration_us":15,"operation_id":"0x7f79","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":9411028},{"operation_name":"get","last_dispatch_duration_us":8344040,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8344040,"total_server_duration_us":43,"operation_id":"0x8062","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8347420},{"operation_name":"get","last_dispatch_duration_us":4733059,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4733059,"total_server_duration_us":62,"operation_id":"0x8235","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":4733611},{"operation_name":"upsert","last_dispatch_duration_us":1218238,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2457972,"total_server_duration_us":168713,"encode_duration_us":23010,"operation_id":"0x8008","timeout_ms":10000,"last_server_duration_us":78979,"total_duration_us":2482829}],"total_count":54}} WARN [2022-03-13 17:13:17,863] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18113131,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18113131,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9076673,"encode_duration_us":16978,"operation_id":"0x7b3c","timeout_ms":10000,"last_server_duration_us":9076673,"total_duration_us":10009147},{"operation_name":"upsert","last_dispatch_duration_us":18148146,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18148146,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9052231,"encode_duration_us":18119,"operation_id":"0x7b37","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10010667}],"total_count":2}} INFO [2022-03-13 17:13:17,864] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,865] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:17,873] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,874] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:17,890] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,891] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:13:17,901] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:17,901] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:17,931] jepsen worker 56 - jepsen.util 2486 :info :write [18 15] :RequestTimeoutException INFO [2022-03-13 17:13:17,931] jepsen worker 56 - jepsen.util 2576 :invoke :read [18 nil] INFO [2022-03-13 17:13:17,946] jepsen worker 88 - jepsen.util 1978 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,947] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:13:17,948] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,949] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:13:17,956] jepsen worker 52 - jepsen.util 592 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,957] jepsen worker 52 - jepsen.util 592 :invoke :write [17 44] INFO [2022-03-13 17:13:17,984] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:17,985] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:13:17,993] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:17,994] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,015] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,016] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:13:18,024] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,024] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,024] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:18,025] jepsen worker 33 - jepsen.util 663 :invoke :write [11 42] INFO [2022-03-13 17:13:18,028] jepsen worker 36 - jepsen.util 2106 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,028] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 21] INFO [2022-03-13 17:13:18,048] jepsen worker 77 - jepsen.util 707 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,048] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,048] jepsen worker 85 - jepsen.util 2245 :info :write [28 38] :RequestTimeoutException INFO [2022-03-13 17:13:18,062] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:18,062] jepsen worker 85 - jepsen.util 2335 :invoke :read [28 nil] INFO [2022-03-13 17:13:18,062] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:13:18,068] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,069] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 41] INFO [2022-03-13 17:13:18,088] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,088] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,088] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,089] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 48] INFO [2022-03-13 17:13:18,090] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:13:18,090] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:18,090] jepsen worker 37 - jepsen.util 2017 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,091] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 47] INFO [2022-03-13 17:13:18,112] jepsen worker 13 - jepsen.util 643 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,112] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,113] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,114] jepsen worker 13 - jepsen.util 643 :invoke :write [4 25] INFO [2022-03-13 17:13:18,119] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,119] jepsen worker 53 - jepsen.util 503 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,120] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 17] INFO [2022-03-13 17:13:18,120] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:18,127] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,127] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,128] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,128] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:18,160] jepsen worker 40 - jepsen.util 2380 :info :write [13 7] :RequestTimeoutException INFO [2022-03-13 17:13:18,161] jepsen worker 40 - jepsen.util 2470 :invoke :write [13 26] INFO [2022-03-13 17:13:18,171] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:18,171] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:18,202] jepsen worker 88 - jepsen.util 1978 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,203] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 34] INFO [2022-03-13 17:13:18,207] jepsen worker 3 - jepsen.util 1893 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,208] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:13:18,211] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,211] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,256] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,256] jepsen worker 35 - jepsen.util 935 :invoke :write [11 22] INFO [2022-03-13 17:13:18,262] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,262] jepsen worker 24 - jepsen.util 564 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,263] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:13:18,264] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:18,265] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,266] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,268] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,269] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:13:18,321] jepsen worker 88 - jepsen.util 1978 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,321] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,323] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,323] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,326] jepsen worker 21 - jepsen.util 831 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,327] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,329] jepsen worker 36 - jepsen.util 2106 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,329] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,329] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 47] INFO [2022-03-13 17:13:18,331] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:18,365] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:18,366] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,392] jepsen worker 37 - jepsen.util 2017 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,393] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,395] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:18,396] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,397] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,398] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:18,398] jepsen worker nemesis - couchbase.util Rebalance check count: 308 INFO [2022-03-13 17:13:18,398] jepsen worker nemesis - couchbase.util Rebalance stuck count: 209 INFO [2022-03-13 17:13:18,399] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:18,401] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,401] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:18,427] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:18,428] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,440] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,441] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:13:18,460] jepsen worker 78 - jepsen.util 528 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,460] jepsen worker 78 - jepsen.util 528 :invoke :write [26 49] INFO [2022-03-13 17:13:18,463] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,463] jepsen worker 71 - jepsen.util 521 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,464] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:18,465] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:18,472] jepsen worker 12 - jepsen.util 732 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,473] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:18,488] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,488] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:18,490] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,490] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:18,496] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,497] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,500] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,500] jepsen worker 36 - jepsen.util 2106 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,501] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,501] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,501] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:13:18,502] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 34] INFO [2022-03-13 17:13:18,502] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,503] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 41] INFO [2022-03-13 17:13:18,508] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:18,508] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 0] INFO [2022-03-13 17:13:18,512] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,515] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:13:18,515] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,516] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:13:18,519] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,520] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:13:18,564] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,565] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,600] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,600] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,600] jepsen worker 80 - jepsen.util 710 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,601] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:18,601] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,602] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:18,602] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,603] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:13:18,607] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,608] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:18,611] jepsen worker 42 - jepsen.util 402 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,612] jepsen worker 42 - jepsen.util 402 :invoke :write [14 25] INFO [2022-03-13 17:13:18,615] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,616] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,634] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,635] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:18,646] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,647] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:13:18,654] jepsen worker 52 - jepsen.util 592 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,655] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,656] jepsen worker 52 - jepsen.util 592 :invoke :write [17 40] INFO [2022-03-13 17:13:18,656] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,662] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,662] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,662] jepsen worker 35 - jepsen.util 935 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,663] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,663] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:18,664] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,665] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,665] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,671] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,672] jepsen worker 21 - jepsen.util 831 :invoke :write [7 35] INFO [2022-03-13 17:13:18,672] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,672] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,673] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:18,673] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:13:18,678] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,679] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,680] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,680] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,680] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,681] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,681] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,682] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:13:18,682] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,683] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:13:18,684] jepsen worker 36 - jepsen.util 2106 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,684] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 31] INFO [2022-03-13 17:13:18,697] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:18,698] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,722] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:18,723] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,739] jepsen worker 37 - jepsen.util 2017 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,739] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:13:18,743] jepsen worker 33 - jepsen.util 663 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,744] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,747] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,748] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:13:18,755] jepsen worker 88 - jepsen.util 1978 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,756] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 33] INFO [2022-03-13 17:13:18,774] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,775] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:13:18,784] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,785] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,791] jepsen worker 13 - jepsen.util 643 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,792] jepsen worker 13 - jepsen.util 643 :invoke :write [4 32] INFO [2022-03-13 17:13:18,803] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,804] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,815] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:18,815] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,840] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,841] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:13:18,856] jepsen worker 27 - jepsen.util 2727 :info :write [9 27] :RequestTimeoutException INFO [2022-03-13 17:13:18,857] jepsen worker 27 - jepsen.util 2817 :invoke :write [9 4] INFO [2022-03-13 17:13:18,860] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,861] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:13:18,866] jepsen worker 76 - jepsen.util 706 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,866] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:18,871] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,872] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,872] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,872] jepsen worker 21 - jepsen.util 831 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,872] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:18,873] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:13:18,879] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,879] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,880] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,879] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,880] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:13:18,881] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:13:18,881] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:13:18,884] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,885] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,886] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,890] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,891] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,892] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,893] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:13:18,896] jepsen worker 54 - jepsen.util 2034 :info :write [18 0] :RequestTimeoutException INFO [2022-03-13 17:13:18,897] jepsen worker 54 - jepsen.util 2124 :invoke :read [18 nil] INFO [2022-03-13 17:13:18,909] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:18,909] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,913] jepsen worker 78 - jepsen.util 528 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,913] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,913] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,913] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,913] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,914] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:18,915] jepsen worker 34 - jepsen.util 754 :invoke :write [11 6] INFO [2022-03-13 17:13:18,916] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:18,919] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,920] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,921] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:18,921] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:18,925] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,925] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,926] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,927] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:18,928] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,928] jepsen worker 22 - jepsen.util 652 :invoke :write [7 28] INFO [2022-03-13 17:13:18,936] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,936] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,937] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:18,938] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,940] jepsen worker 88 - jepsen.util 1978 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,940] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,952] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,953] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,953] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:18,953] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:13:18,971] jepsen worker 36 - jepsen.util 2106 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,972] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:18,979] jepsen worker 52 - jepsen.util 592 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,979] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:18,982] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:18,984] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:18,985] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,985] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,986] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,987] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:18,987] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,988] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:13:18,992] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,992] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,992] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:13:18,993] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,995] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,996] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:18,997] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:18,998] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:13:19,015] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,016] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:13:19,020] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:19,021] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,052] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,052] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,053] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:19,053] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 19] INFO [2022-03-13 17:13:19,054] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,055] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:19,074] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,074] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,074] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,075] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:19,076] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:13:19,076] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:19,077] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,078] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,078] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 44] INFO [2022-03-13 17:13:19,078] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:19,079] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 29] INFO [2022-03-13 17:13:19,089] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:19,090] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,091] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:13:19,104] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,106] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:19,106] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,107] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:13:19,136] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,136] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,137] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:19,138] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:19,153] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,153] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:19,173] jepsen worker 3 - jepsen.util 1893 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,174] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,184] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,184] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,185] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,185] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,186] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,186] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,187] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,188] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,188] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,189] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,190] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,190] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,190] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,191] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:13:19,191] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,192] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,193] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,194] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:13:19,200] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,200] jepsen worker 26 - jepsen.util 476 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,200] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:19,200] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:13:19,201] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:13:19,201] jepsen worker 26 - jepsen.util 476 :invoke :write [8 15] INFO [2022-03-13 17:13:19,204] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,204] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,205] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,205] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:19,214] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,222] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,250] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:19,250] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,270] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,271] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:13:19,287] jepsen worker 13 - jepsen.util 643 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,287] jepsen worker 22 - jepsen.util 652 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,287] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,287] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,287] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,288] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:19,289] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:19,290] jepsen worker 70 - jepsen.util 700 :invoke :write [23 37] INFO [2022-03-13 17:13:19,290] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,290] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:19,291] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:13:19,291] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:19,298] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,298] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,303] jepsen worker 37 - jepsen.util 2017 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,304] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:19,315] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,315] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:13:19,318] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,319] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:13:19,348] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:19,349] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 14] INFO [2022-03-13 17:13:19,358] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,358] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:13:19,361] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,361] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,367] jepsen worker 44 - jepsen.util 584 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,368] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:19,386] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,387] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:13:19,405] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:19,405] jepsen worker nemesis - couchbase.util Rebalance check count: 309 INFO [2022-03-13 17:13:19,405] jepsen worker nemesis - couchbase.util Rebalance stuck count: 210 INFO [2022-03-13 17:13:19,434] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,434] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:19,439] jepsen worker 34 - jepsen.util 754 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,440] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,440] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,440] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,440] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,440] jepsen worker 52 - jepsen.util 592 :invoke :write [17 25] INFO [2022-03-13 17:13:19,441] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:19,441] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:19,442] jepsen worker 80 - jepsen.util 710 :invoke :write [26 15] INFO [2022-03-13 17:13:19,442] jepsen worker 78 - jepsen.util 528 :invoke :write [26 1] INFO [2022-03-13 17:13:19,455] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,456] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,456] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:19,457] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:13:19,466] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,466] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,467] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,467] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:13:19,468] jepsen worker 43 - jepsen.util 493 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,468] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,468] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,468] jepsen worker 43 - jepsen.util 493 :invoke :write [14 8] INFO [2022-03-13 17:13:19,469] jepsen worker 62 - jepsen.util 512 :invoke :write [20 21] INFO [2022-03-13 17:13:19,469] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:19,487] jepsen worker 72 - jepsen.util 2322 :info :write [24 34] :RequestTimeoutException INFO [2022-03-13 17:13:19,488] jepsen worker 72 - jepsen.util 2412 :invoke :write [24 1] INFO [2022-03-13 17:13:19,489] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,489] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:19,516] jepsen worker 88 - jepsen.util 1978 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,517] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,526] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,527] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,527] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:13:19,528] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,561] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:19,562] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,562] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,562] jepsen worker 70 - jepsen.util 700 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,562] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,563] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,563] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,564] jepsen worker 34 - jepsen.util 754 :invoke :write [11 26] INFO [2022-03-13 17:13:19,564] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:19,565] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:19,565] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:19,566] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:13:19,576] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,576] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,577] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,577] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,609] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,610] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:19,613] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,614] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:19,628] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,628] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,628] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,629] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,629] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:13:19,629] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 36] INFO [2022-03-13 17:13:19,644] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:19,645] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 30] INFO [2022-03-13 17:13:19,664] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:19,665] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:13:19,671] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,672] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:19,677] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,677] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,678] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:19,678] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:19,679] jepsen worker 60 - jepsen.util 600 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,679] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,679] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:19,680] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:19,690] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,691] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:13:19,707] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,707] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:13:19,729] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,730] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,749] jepsen worker 88 - jepsen.util 1978 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,750] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 1] INFO [2022-03-13 17:13:19,786] jepsen worker 3 - jepsen.util 1893 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,787] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,811] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,811] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:19,822] jepsen worker 8 - jepsen.util 728 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,822] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:19,822] jepsen worker 8 - jepsen.util 728 :invoke :write [2 42] INFO [2022-03-13 17:13:19,823] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,832] jepsen worker 52 - jepsen.util 592 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,833] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:19,833] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,833] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,834] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,834] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:19,835] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:13:19,835] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] INFO [2022-03-13 17:13:19,854] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,854] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:19,855] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,856] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:19,857] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,857] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:19,857] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,858] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,858] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,859] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:13:19,859] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:19,860] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:13:19,866] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,867] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:19,869] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,870] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:13:19,871] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:19,872] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:13:19,909] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,910] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:19,913] jepsen worker 88 - jepsen.util 1978 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,914] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:13:19,922] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,923] jepsen worker 36 - jepsen.util 2106 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,923] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,924] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,925] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 3] INFO [2022-03-13 17:13:19,925] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,926] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,930] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 48] INFO [2022-03-13 17:13:19,933] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,934] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,934] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,935] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,935] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,935] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,935] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,935] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:13:19,937] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:13:19,937] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:13:19,942] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,945] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,946] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:19,946] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:13:19,947] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,948] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:19,949] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,951] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:19,953] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:19,953] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:19,954] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,955] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:19,968] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,969] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:19,969] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,970] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:19,997] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:19,998] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:20,027] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,027] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,027] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,027] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:13:20,028] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:20,029] jepsen worker 78 - jepsen.util 528 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,029] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:13:20,029] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:20,029] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,030] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:13:20,045] jepsen worker 36 - jepsen.util 2106 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,045] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,073] jepsen worker 65 - jepsen.util 2765 :ok :read [21 :nil] INFO [2022-03-13 17:13:20,074] jepsen worker 65 - jepsen.util 2765 :invoke :read [21 nil] INFO [2022-03-13 17:13:20,078] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,079] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:20,080] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,080] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,081] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:13:20,082] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:13:20,088] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,089] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:13:20,108] jepsen worker 47 - jepsen.util 2387 :ok :read [15 48] INFO [2022-03-13 17:13:20,109] jepsen worker 47 - jepsen.util 2387 :invoke :read [15 nil] INFO [2022-03-13 17:13:20,127] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,128] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:20,147] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,147] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:13:20,156] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,156] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:20,166] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,167] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:13:20,200] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,201] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:20,201] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,202] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:13:20,223] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,224] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,251] jepsen worker 37 - jepsen.util 2017 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,251] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,255] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:20,255] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,266] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,266] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:13:20,275] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,276] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,275] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,277] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,278] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:20,278] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,278] jepsen worker 80 - jepsen.util 710 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,278] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,278] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,279] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,279] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:13:20,280] jepsen worker 13 - jepsen.util 643 :invoke :write [4 29] INFO [2022-03-13 17:13:20,281] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:13:20,281] jepsen worker 21 - jepsen.util 831 :invoke :write [7 40] INFO [2022-03-13 17:13:20,282] jepsen worker 80 - jepsen.util 710 :invoke :write [26 17] INFO [2022-03-13 17:13:20,286] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:20,290] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,290] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,292] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,293] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,294] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,295] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,296] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,297] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,297] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,297] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,298] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:13:20,298] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:13:20,311] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,312] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,312] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,312] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:20,313] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,315] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:13:20,318] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:20,319] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,339] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,339] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,339] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,340] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:20,341] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 21] INFO [2022-03-13 17:13:20,342] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:13:20,346] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:20,346] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,347] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,348] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:20,362] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,363] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:13:20,373] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:20,373] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,388] jepsen worker 11 - jepsen.util 2351 :info :write [3 7] :RequestTimeoutException INFO [2022-03-13 17:13:20,389] jepsen worker 11 - jepsen.util 2441 :invoke :read [3 nil] INFO [2022-03-13 17:13:20,390] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,390] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:13:20,392] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,392] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,393] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:20,393] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:13:20,398] jepsen worker 51 - jepsen.util 591 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,398] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,398] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,398] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,398] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,398] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,398] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,398] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:20,400] jepsen worker 23 - jepsen.util 563 :invoke :write [7 10] INFO [2022-03-13 17:13:20,404] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:20,405] jepsen worker 69 - jepsen.util 699 :invoke :write [23 41] INFO [2022-03-13 17:13:20,406] jepsen worker 60 - jepsen.util 600 :invoke :write [20 38] INFO [2022-03-13 17:13:20,407] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:20,408] jepsen worker 44 - jepsen.util 584 :invoke :write [14 17] INFO [2022-03-13 17:13:20,408] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:20,410] jepsen worker 76 - jepsen.util 706 :invoke :write [25 31] INFO [2022-03-13 17:13:20,412] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:20,412] jepsen worker nemesis - couchbase.util Rebalance check count: 310 INFO [2022-03-13 17:13:20,412] jepsen worker nemesis - couchbase.util Rebalance stuck count: 211 INFO [2022-03-13 17:13:20,419] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,419] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:13:20,456] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,457] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,463] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,463] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,469] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,470] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,472] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,473] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:13:20,474] jepsen worker 36 - jepsen.util 2106 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,475] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 5] INFO [2022-03-13 17:13:20,483] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,484] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:20,492] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:20,493] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 28] INFO [2022-03-13 17:13:20,515] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,516] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:20,541] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,542] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,542] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:20,542] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,542] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:13:20,543] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:13:20,545] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,545] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,546] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:13:20,546] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:13:20,577] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,578] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,580] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,580] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,582] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,583] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:13:20,592] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,592] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:20,613] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,614] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,640] jepsen worker 34 - jepsen.util 754 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,640] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,640] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,640] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,641] jepsen worker 78 - jepsen.util 528 :invoke :write [26 15] INFO [2022-03-13 17:13:20,642] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:20,643] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:20,644] jepsen worker 52 - jepsen.util 592 :invoke :write [17 26] INFO [2022-03-13 17:13:20,647] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,647] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,647] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:20,648] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:20,654] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,655] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:13:20,658] jepsen worker 21 - jepsen.util 831 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,659] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,667] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,668] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:20,669] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,669] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,670] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:13:20,671] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:13:20,674] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:20,674] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,678] jepsen worker 88 - jepsen.util 1978 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,679] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 40] INFO [2022-03-13 17:13:20,686] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,688] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:13:20,708] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,708] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,709] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:13:20,709] jepsen worker 36 - jepsen.util 2106 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,709] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:13:20,710] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:20,713] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,714] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,716] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,716] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,718] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,718] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,720] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,720] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:13:20,721] jepsen worker 61 - jepsen.util 601 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,721] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,722] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:13:20,723] jepsen worker 6 - jepsen.util 636 :invoke :write [2 43] INFO [2022-03-13 17:13:20,735] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:20,736] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,749] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,749] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 0] INFO [2022-03-13 17:13:20,752] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,752] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,752] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,753] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:20,753] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:13:20,754] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,773] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:20,775] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,776] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:13:20,776] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,777] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,777] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,777] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:13:20,777] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:20,778] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:13:20,779] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,784] jepsen worker 13 - jepsen.util 643 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,785] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:20,787] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,787] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,788] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 12] INFO [2022-03-13 17:13:20,789] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:13:20,824] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,824] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:20,846] jepsen worker 37 - jepsen.util 2017 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,847] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 41] INFO [2022-03-13 17:13:20,849] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,850] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,850] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:20,850] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:20,869] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,870] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:13:20,896] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,896] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:13:20,906] jepsen worker 88 - jepsen.util 1978 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,907] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 14] INFO [2022-03-13 17:13:20,920] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,920] jepsen worker 80 - jepsen.util 710 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,920] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,921] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:20,921] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:20,921] jepsen worker 80 - jepsen.util 710 :invoke :write [26 48] INFO [2022-03-13 17:13:20,935] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,935] jepsen worker 71 - jepsen.util 521 :invoke :write [23 42] INFO [2022-03-13 17:13:20,940] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,941] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:13:20,975] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,976] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:20,977] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:20,977] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:13:20,992] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,992] jepsen worker 36 - jepsen.util 2106 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:20,993] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 19] INFO [2022-03-13 17:13:20,999] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 19] INFO [2022-03-13 17:13:21,020] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,021] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:21,023] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,024] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,027] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,027] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,027] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,028] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:13:21,028] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:21,029] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:21,030] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,030] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,030] jepsen worker 22 - jepsen.util 652 :invoke :write [7 38] INFO [2022-03-13 17:13:21,031] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,032] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:13:21,033] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:21,034] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,035] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:13:21,046] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,047] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,068] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,069] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,073] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,073] jepsen worker 88 - jepsen.util 1978 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,073] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:13:21,074] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,100] jepsen worker 52 - jepsen.util 592 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,101] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:21,111] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,111] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,122] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,123] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,159] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,159] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:13:21,160] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,160] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,160] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:13:21,161] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,162] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,162] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:13:21,167] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,167] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,168] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,169] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:21,170] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,172] jepsen worker 37 - jepsen.util 2017 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,172] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:13:21,177] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 2] INFO [2022-03-13 17:13:21,190] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,191] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:21,194] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:21,195] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 18] INFO [2022-03-13 17:13:21,199] jepsen worker 89 - jepsen.util 2429 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,200] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:13:21,258] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,258] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,262] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,262] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,263] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,264] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:13:21,266] jepsen worker 36 - jepsen.util 2106 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,267] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,267] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,267] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:13:21,280] jepsen worker 78 - jepsen.util 528 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,280] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,280] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,280] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,280] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,281] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:21,282] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:21,282] jepsen worker 78 - jepsen.util 528 :invoke :write [26 2] INFO [2022-03-13 17:13:21,283] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,283] jepsen worker 34 - jepsen.util 754 :invoke :write [11 49] INFO [2022-03-13 17:13:21,342] jepsen worker 88 - jepsen.util 1978 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,342] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 41] INFO [2022-03-13 17:13:21,352] jepsen worker 32 - jepsen.util 2282 :ok :read [10 :nil] INFO [2022-03-13 17:13:21,352] jepsen worker 32 - jepsen.util 2282 :invoke :read [10 nil] INFO [2022-03-13 17:13:21,355] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,356] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,356] jepsen worker 22 - jepsen.util 652 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,357] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:13:21,412] jepsen worker 37 - jepsen.util 2017 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,413] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,413] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,413] jepsen worker 32 - jepsen.util 2282 :ok :read [10 :nil] INFO [2022-03-13 17:13:21,413] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 48] INFO [2022-03-13 17:13:21,414] jepsen worker 32 - jepsen.util 2282 :invoke :write [10 19] INFO [2022-03-13 17:13:21,415] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,415] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 28] INFO [2022-03-13 17:13:21,423] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:21,423] jepsen worker nemesis - couchbase.util Rebalance check count: 311 INFO [2022-03-13 17:13:21,423] jepsen worker nemesis - couchbase.util Rebalance stuck count: 212 INFO [2022-03-13 17:13:21,425] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,425] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,426] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,426] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,427] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:21,427] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:21,428] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,428] jepsen worker 71 - jepsen.util 521 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,428] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,429] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,429] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,429] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,429] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,430] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,431] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:13:21,431] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,432] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:21,433] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,433] jepsen worker 21 - jepsen.util 831 :invoke :write [7 14] INFO [2022-03-13 17:13:21,434] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,440] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,441] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,445] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,445] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,446] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:21,447] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,447] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,448] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,450] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,451] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:13:21,462] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,462] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,464] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,466] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,466] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,467] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:13:21,467] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,468] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,471] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,471] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:21,471] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:13:21,472] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:13:21,473] jepsen worker 88 - jepsen.util 1978 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,474] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:21,474] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 42] INFO [2022-03-13 17:13:21,474] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,499] jepsen worker 57 - jepsen.util 2307 :ok :write [19 38] INFO [2022-03-13 17:13:21,500] jepsen worker 57 - jepsen.util 2307 :invoke :write [19 32] INFO [2022-03-13 17:13:21,516] jepsen worker 20 - jepsen.util 2720 :ok :read [6 16] INFO [2022-03-13 17:13:21,518] jepsen worker 20 - jepsen.util 2720 :invoke :write [6 41] INFO [2022-03-13 17:13:21,522] jepsen worker 80 - jepsen.util 710 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,522] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,522] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,522] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,523] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:21,523] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:21,524] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:21,524] jepsen worker 13 - jepsen.util 643 :invoke :write [4 35] INFO [2022-03-13 17:13:21,545] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,546] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:13:21,557] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,557] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,557] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,557] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,557] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,557] jepsen worker 46 - jepsen.util 2386 :ok :read [15 48] INFO [2022-03-13 17:13:21,557] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:21,560] jepsen worker 46 - jepsen.util 2386 :invoke :write [15 1] INFO [2022-03-13 17:13:21,560] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,561] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:13:21,562] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,563] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:13:21,564] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:21,565] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:13:21,565] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,566] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,566] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,567] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:21,571] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,572] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,572] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:21,573] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:13:21,580] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,581] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,586] jepsen worker 21 - jepsen.util 831 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,588] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:13:21,605] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:21,605] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:21,647] jepsen worker 37 - jepsen.util 2017 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,648] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:13:21,657] jepsen worker 88 - jepsen.util 1978 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,657] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,665] jepsen worker 78 - jepsen.util 528 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,666] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:21,673] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,674] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:13:21,683] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,683] jepsen worker 84 - jepsen.util 2514 :info :write [28 34] :RequestTimeoutException INFO [2022-03-13 17:13:21,684] jepsen worker 84 - jepsen.util 2604 :invoke :write [28 46] INFO [2022-03-13 17:13:21,687] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:13:21,687] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,688] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:13:21,704] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,705] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:21,708] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,709] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:13:21,713] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:21,714] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,724] jepsen worker 16 - jepsen.util 2086 :info :write [5 9] :RequestTimeoutException INFO [2022-03-13 17:13:21,725] jepsen worker 16 - jepsen.util 2176 :invoke :read [5 nil] INFO [2022-03-13 17:13:21,753] jepsen worker 36 - jepsen.util 2106 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,754] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,786] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,787] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:21,787] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,788] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,789] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,790] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 19] INFO [2022-03-13 17:13:21,809] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,810] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:13:21,813] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:21,815] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,823] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,826] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:13:21,865] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:21,866] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:21,874] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,874] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,874] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,875] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:13:21,875] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,876] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:21,888] jepsen worker 34 - jepsen.util 754 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,888] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,888] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,888] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,890] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:21,890] jepsen worker 34 - jepsen.util 754 :invoke :write [11 24] INFO [2022-03-13 17:13:21,891] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:21,891] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:21,920] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,920] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:13:21,948] jepsen worker 3 - jepsen.util 1893 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,948] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,949] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:21,950] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:21,950] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,951] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,952] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,953] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:21,959] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,959] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:13:21,968] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:21,969] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:13:21,985] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:21,987] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 36] INFO [2022-03-13 17:13:22,011] jepsen worker 13 - jepsen.util 643 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,011] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,011] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,011] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,012] jepsen worker 80 - jepsen.util 710 :invoke :write [26 43] INFO [2022-03-13 17:13:22,013] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:22,014] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,014] jepsen worker 70 - jepsen.util 700 :invoke :write [23 0] INFO [2022-03-13 17:13:22,015] jepsen worker 78 - jepsen.util 528 :invoke :write [26 19] INFO [2022-03-13 17:13:22,015] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:22,034] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,035] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:13:22,041] jepsen worker 59 - jepsen.util 2399 :ok :read [19 38] INFO [2022-03-13 17:13:22,041] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,041] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,042] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,042] jepsen worker 59 - jepsen.util 2399 :invoke :read [19 nil] INFO [2022-03-13 17:13:22,042] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,057] jepsen worker 66 - jepsen.util 2226 :ok :read [22 30] INFO [2022-03-13 17:13:22,058] jepsen worker 66 - jepsen.util 2226 :invoke :read [22 nil] INFO [2022-03-13 17:13:22,081] jepsen worker 59 - jepsen.util 2399 :ok :read [19 38] INFO [2022-03-13 17:13:22,082] jepsen worker 59 - jepsen.util 2399 :invoke :read [19 nil] INFO [2022-03-13 17:13:22,085] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,086] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:22,090] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,091] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,091] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,091] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,091] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,092] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:13:22,092] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,093] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,095] jepsen worker 66 - jepsen.util 2226 :ok :read [22 30] INFO [2022-03-13 17:13:22,096] jepsen worker 66 - jepsen.util 2226 :invoke :read [22 nil] INFO [2022-03-13 17:13:22,110] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,111] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:22,120] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,121] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,122] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:13:22,122] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,122] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:22,123] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,135] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:22,136] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,141] jepsen worker 59 - jepsen.util 2399 :ok :read [19 38] INFO [2022-03-13 17:13:22,142] jepsen worker 59 - jepsen.util 2399 :invoke :write [19 39] INFO [2022-03-13 17:13:22,146] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,147] jepsen worker 33 - jepsen.util 663 :invoke :write [11 19] INFO [2022-03-13 17:13:22,149] jepsen worker 66 - jepsen.util 2226 :ok :read [22 30] INFO [2022-03-13 17:13:22,150] jepsen worker 66 - jepsen.util 2226 :invoke :write [22 31] INFO [2022-03-13 17:13:22,155] jepsen worker 88 - jepsen.util 1978 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,155] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:13:22,171] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,171] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,171] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,171] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 22] INFO [2022-03-13 17:13:22,172] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,172] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:13:22,182] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,182] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,183] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,183] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,205] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,205] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,206] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,206] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,207] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,207] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,208] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:13:22,208] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:13:22,220] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,220] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:22,220] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,221] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:13:22,222] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,222] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:22,245] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,245] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,247] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:13:22,248] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,269] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,269] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,270] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,271] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:13:22,272] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:22,273] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:22,275] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,275] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:13:22,282] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,283] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:22,286] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,287] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:22,289] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,290] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:13:22,321] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,321] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,322] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 15] INFO [2022-03-13 17:13:22,324] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:13:22,328] jepsen worker 34 - jepsen.util 754 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,328] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,329] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,329] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:22,330] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:22,330] jepsen worker 13 - jepsen.util 643 :invoke :write [4 48] INFO [2022-03-13 17:13:22,343] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,343] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,387] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,387] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,404] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,404] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,405] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:22,405] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:22,413] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,414] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:13:22,419] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,420] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:13:22,428] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:22,429] jepsen worker nemesis - couchbase.util Rebalance check count: 312 INFO [2022-03-13 17:13:22,429] jepsen worker nemesis - couchbase.util Rebalance stuck count: 213 INFO [2022-03-13 17:13:22,462] jepsen worker 36 - jepsen.util 2106 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,463] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,462] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,463] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,464] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 2] INFO [2022-03-13 17:13:22,464] jepsen worker 33 - jepsen.util 663 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,464] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,464] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,464] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,464] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,465] jepsen worker 71 - jepsen.util 521 :invoke :write [23 40] INFO [2022-03-13 17:13:22,466] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:13:22,471] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,473] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:22,474] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:13:22,478] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:22,481] jepsen worker 55 - jepsen.util 2305 :info :write [18 22] :RequestTimeoutException INFO [2022-03-13 17:13:22,482] jepsen worker 55 - jepsen.util 2395 :invoke :read [18 nil] INFO [2022-03-13 17:13:22,494] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,495] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,497] jepsen worker 88 - jepsen.util 1978 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,498] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,529] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,529] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,529] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,531] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:13:22,533] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,533] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,545] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:22,546] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,551] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,552] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,552] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:13:22,553] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,555] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,556] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:13:22,560] jepsen worker 80 - jepsen.util 710 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,561] jepsen worker 80 - jepsen.util 710 :invoke :write [26 15] INFO [2022-03-13 17:13:22,568] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,569] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,571] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,572] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 48] INFO [2022-03-13 17:13:22,584] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:22,585] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:22,616] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,617] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:22,627] jepsen worker 36 - jepsen.util 2106 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,627] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,627] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,627] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,628] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,628] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,632] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,632] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,632] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,632] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,633] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:13:22,633] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,636] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,636] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,636] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:13:22,637] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,637] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 33] INFO [2022-03-13 17:13:22,638] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,640] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,641] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:13:22,641] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,642] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:13:22,655] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:22,656] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,682] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,683] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:13:22,709] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,710] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,715] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,717] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:22,724] jepsen worker 23 - jepsen.util 563 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,725] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:22,751] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:22,752] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,760] jepsen worker 70 - jepsen.util 700 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,761] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:22,761] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,761] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,762] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,762] jepsen worker 21 - jepsen.util 831 :invoke :write [7 33] INFO [2022-03-13 17:13:22,763] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:22,763] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,773] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,774] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,777] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,777] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,782] jepsen worker 3 - jepsen.util 1893 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,783] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,786] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,786] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,788] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,789] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,791] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,791] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:22,795] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:22,795] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,800] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,801] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,819] jepsen worker 42 - jepsen.util 402 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,820] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,820] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:13:22,821] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,821] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:22,823] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,824] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:22,825] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:22,826] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 3] INFO [2022-03-13 17:13:22,826] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:22,834] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:22,834] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,868] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,868] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,868] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:22,868] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:22,871] jepsen worker 78 - jepsen.util 528 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,872] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,871] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,872] jepsen worker 34 - jepsen.util 754 :invoke :write [11 37] INFO [2022-03-13 17:13:22,873] jepsen worker 52 - jepsen.util 592 :invoke :write [17 11] INFO [2022-03-13 17:13:22,874] jepsen worker 78 - jepsen.util 528 :invoke :write [26 43] INFO [2022-03-13 17:13:22,879] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,880] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:13:22,895] jepsen worker 36 - jepsen.util 2106 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,896] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,931] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,931] jepsen worker 71 - jepsen.util 521 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,931] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:22,932] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:22,933] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,933] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:22,933] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:22,934] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:22,945] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,946] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:22,979] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,979] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,979] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,979] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,980] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,980] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 32] INFO [2022-03-13 17:13:22,982] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,983] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:13:22,984] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 47] INFO [2022-03-13 17:13:22,987] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:13:22,993] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:22,995] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:22,997] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,000] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:13:23,023] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,024] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:23,024] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:23,025] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:13:23,028] jepsen worker 60 - jepsen.util 600 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,029] jepsen worker 60 - jepsen.util 600 :invoke :write [20 20] INFO [2022-03-13 17:13:23,032] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,033] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:23,042] jepsen worker 21 - jepsen.util 831 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,043] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,043] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,044] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:23,045] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:13:23,045] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,050] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,050] jepsen worker 88 - jepsen.util 1978 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,051] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:13:23,051] jepsen worker 21 - jepsen.util 831 :invoke :write [7 44] INFO [2022-03-13 17:13:23,074] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:13:23,052] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:13:23,082] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,082] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,082] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,082] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,083] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:23,083] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:23,084] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:23,084] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,084] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:23,084] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:23,087] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,087] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,087] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,087] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,089] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:13:23,089] jepsen worker 24 - jepsen.util 564 :invoke :write [8 15] INFO [2022-03-13 17:13:23,089] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:23,090] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:23,091] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,092] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:13:23,092] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,092] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:13:23,093] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,094] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:13:23,094] jepsen worker 13 - jepsen.util 643 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,095] jepsen worker 13 - jepsen.util 643 :invoke :write [4 0] INFO [2022-03-13 17:13:23,133] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,133] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:23,147] jepsen worker 36 - jepsen.util 2106 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,147] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,152] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,152] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:23,157] jepsen worker 26 - jepsen.util 476 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,158] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,158] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:23,159] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:23,164] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,164] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,169] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,170] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,178] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,179] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:23,194] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,195] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:13:23,198] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,199] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:13:23,205] jepsen worker 21 - jepsen.util 831 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,207] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:23,247] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,248] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:23,267] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,268] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:23,285] jepsen worker 37 - jepsen.util 2017 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,287] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,299] jepsen worker 80 - jepsen.util 710 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,300] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,300] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:23,301] jepsen worker 70 - jepsen.util 700 :invoke :write [23 39] INFO [2022-03-13 17:13:23,303] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,303] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,304] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:13:23,304] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:23,332] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:23,333] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 5] INFO [2022-03-13 17:13:23,363] jepsen worker 69 - jepsen.util 699 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,363] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,365] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:23,365] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:23,371] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,371] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,371] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,372] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:23,373] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:13:23,373] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:23,374] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,375] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,375] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:23,375] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:23,376] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,376] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:23,377] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,377] jepsen worker 35 - jepsen.util 935 :invoke :write [11 15] INFO [2022-03-13 17:13:23,378] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,378] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:13:23,391] jepsen worker 19 - jepsen.util 2269 :info :write [6 25] :RequestTimeoutException INFO [2022-03-13 17:13:23,391] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,392] jepsen worker 19 - jepsen.util 2359 :invoke :read [6 nil] INFO [2022-03-13 17:13:23,393] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:23,399] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,399] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,399] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,400] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:23,400] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 41] INFO [2022-03-13 17:13:23,403] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,406] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,406] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,407] jepsen worker 78 - jepsen.util 528 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,407] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:23,408] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:23,410] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:13:23,435] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:23,435] jepsen worker nemesis - couchbase.util Rebalance check count: 313 INFO [2022-03-13 17:13:23,435] jepsen worker nemesis - couchbase.util Rebalance stuck count: 214 INFO [2022-03-13 17:13:23,436] jepsen worker 24 - jepsen.util 564 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,437] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:23,438] jepsen worker 62 - jepsen.util 512 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,439] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:23,463] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,463] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,463] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:13:23,464] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:13:23,466] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,466] jepsen worker 71 - jepsen.util 521 :invoke :write [23 42] INFO [2022-03-13 17:13:23,483] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,483] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,483] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:23,484] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:23,484] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,485] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,540] jepsen worker 88 - jepsen.util 1978 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,541] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 0] INFO [2022-03-13 17:13:23,549] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:23,550] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:23,564] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,565] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:23,591] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,591] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:13:23,611] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,611] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:23,619] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,619] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,619] jepsen worker 37 - jepsen.util 2017 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,620] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:13:23,621] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,622] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 15] INFO [2022-03-13 17:13:23,626] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,627] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 41] INFO [2022-03-13 17:13:23,665] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,665] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:13:23,686] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,687] jepsen worker 88 - jepsen.util 1978 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,687] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:13:23,689] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 4] INFO [2022-03-13 17:13:23,764] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,765] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:23,781] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,781] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:23,783] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,783] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:13:23,789] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,789] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:23,839] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,839] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:13:23,839] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,840] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:23,846] jepsen worker 39 - jepsen.util 2289 :info :write [13 22] :RequestTimeoutException INFO [2022-03-13 17:13:23,847] jepsen worker 39 - jepsen.util 2379 :invoke :read [13 nil] INFO [2022-03-13 17:13:23,852] jepsen worker 88 - jepsen.util 1978 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,853] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:23,890] jepsen worker 50 - jepsen.util 2480 :info :write [16 34] :RequestTimeoutException INFO [2022-03-13 17:13:23,891] jepsen worker 50 - jepsen.util 2570 :invoke :write [16 1] INFO [2022-03-13 17:13:23,906] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:23,907] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:23,913] jepsen worker 36 - jepsen.util 2106 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,914] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 1] INFO [2022-03-13 17:13:23,915] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,917] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:13:23,926] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:23,927] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 6] INFO [2022-03-13 17:13:23,943] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:23,944] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:13:24,003] jepsen worker 37 - jepsen.util 2017 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,003] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,003] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,003] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:24,004] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,004] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:13:24,030] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:24,031] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 20] INFO [2022-03-13 17:13:24,050] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,051] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,075] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,076] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:13:24,103] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:24,103] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,115] jepsen worker 58 - jepsen.util 2218 :ok :read [19 38] INFO [2022-03-13 17:13:24,116] jepsen worker 58 - jepsen.util 2218 :invoke :read [19 nil] INFO [2022-03-13 17:13:24,126] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,127] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:13:24,135] jepsen worker 36 - jepsen.util 2106 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,135] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,136] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,136] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,136] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:13:24,137] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:24,143] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,143] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,144] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,144] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:24,145] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,147] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,147] jepsen worker 36 - jepsen.util 2106 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,148] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:13:24,148] jepsen worker 36 - jepsen.util 2106 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,151] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 8] INFO [2022-03-13 17:13:24,154] jepsen worker 86 - jepsen.util 2156 :info :write [28 7] :RequestTimeoutException INFO [2022-03-13 17:13:24,155] jepsen worker 86 - jepsen.util 2246 :invoke :write [28 18] INFO [2022-03-13 17:13:24,167] jepsen worker 3 - jepsen.util 1893 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,167] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:24,170] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:24,171] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:24,175] jepsen worker 88 - jepsen.util 1978 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,176] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,220] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:24,220] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 10] INFO [2022-03-13 17:13:24,233] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,234] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:13:24,254] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,254] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:13:24,288] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,289] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,292] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,293] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,294] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,295] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:24,303] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,303] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,303] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:13:24,304] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] INFO [2022-03-13 17:13:24,339] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,340] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:24,341] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,342] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:13:24,351] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,352] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:24,355] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,356] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,356] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,356] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:13:24,356] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:13:24,357] jepsen worker 24 - jepsen.util 564 :invoke :write [8 30] INFO [2022-03-13 17:13:24,366] jepsen worker 52 - jepsen.util 592 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,366] jepsen worker 52 - jepsen.util 592 :invoke :write [17 4] INFO [2022-03-13 17:13:24,374] jepsen worker 88 - jepsen.util 1978 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,376] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,381] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,381] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:13:24,400] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,401] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,401] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,402] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:24,437] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:24,438] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 25] INFO [2022-03-13 17:13:24,441] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6081871345029239}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:24,441] jepsen worker nemesis - couchbase.util Rebalance check count: 314 INFO [2022-03-13 17:13:24,441] jepsen worker nemesis - couchbase.util Rebalance stuck count: 215 INFO [2022-03-13 17:13:24,452] jepsen worker 34 - jepsen.util 754 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,452] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:24,454] jepsen worker 44 - jepsen.util 584 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,455] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:24,460] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:24,460] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,461] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:24,462] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:24,484] jepsen worker 36 - jepsen.util 2106 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,484] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 41] INFO [2022-03-13 17:13:24,486] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,486] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,486] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,487] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:24,492] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,492] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:13:24,504] jepsen worker 71 - jepsen.util 521 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,505] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:24,522] jepsen worker 43 - jepsen.util 493 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,522] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,523] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:13:24,524] jepsen worker 43 - jepsen.util 493 :invoke :write [14 42] INFO [2022-03-13 17:13:24,551] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,552] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:13:24,556] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,557] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:13:24,568] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,568] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,569] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,569] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:13:24,572] jepsen worker 35 - jepsen.util 935 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,572] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:24,575] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,575] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,575] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,575] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:24,576] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:13:24,577] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,577] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:24,577] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:24,578] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,579] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:13:24,584] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,584] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:13:24,594] jepsen worker 88 - jepsen.util 1978 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,596] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,606] jepsen worker 13 - jepsen.util 643 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,606] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,608] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:24,609] jepsen worker 13 - jepsen.util 643 :invoke :write [4 12] INFO [2022-03-13 17:13:24,631] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,632] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:13:24,642] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,642] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,644] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:24,654] jepsen worker 85 - jepsen.util 2335 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,655] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,655] jepsen worker 85 - jepsen.util 2335 :invoke :write [28 41] INFO [2022-03-13 17:13:24,656] jepsen worker 15 - jepsen.util 2445 :ok :read [5 43] INFO [2022-03-13 17:13:24,657] jepsen worker 15 - jepsen.util 2445 :invoke :read [5 nil] INFO [2022-03-13 17:13:24,665] jepsen worker 68 - jepsen.util 2498 :ok :read [22 30] INFO [2022-03-13 17:13:24,666] jepsen worker 68 - jepsen.util 2498 :invoke :write [22 21] INFO [2022-03-13 17:13:24,667] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:24,668] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,668] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,669] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,694] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,695] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:13:24,706] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,706] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,706] jepsen worker 76 - jepsen.util 706 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,706] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,708] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:24,709] jepsen worker 51 - jepsen.util 591 :invoke :write [17 13] INFO [2022-03-13 17:13:24,709] jepsen worker 76 - jepsen.util 706 :invoke :write [25 28] INFO [2022-03-13 17:13:24,718] jepsen worker 41 - jepsen.util 2201 :info :write [13 2] :SyncWriteAmbiguous INFO [2022-03-13 17:13:24,719] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,719] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,719] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:24,719] jepsen worker 41 - jepsen.util 2291 :invoke :write [13 38] INFO [2022-03-13 17:13:24,720] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:13:24,721] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:13:24,725] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,725] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,728] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,729] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:24,736] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,736] jepsen worker 36 - jepsen.util 2106 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,736] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:13:24,737] jepsen worker 36 - jepsen.util 2106 :invoke :write [12 38] INFO [2022-03-13 17:13:24,741] jepsen worker 70 - jepsen.util 700 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,741] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,742] jepsen worker 78 - jepsen.util 528 :invoke :write [26 36] INFO [2022-03-13 17:13:24,743] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:24,754] jepsen worker 81 - jepsen.util 2691 :info :write [27 47] :SyncWriteAmbiguous INFO [2022-03-13 17:13:24,759] jepsen worker 81 - jepsen.util 2781 :invoke :write [27 48] INFO [2022-03-13 17:13:24,767] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,768] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:13:24,786] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:24,786] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,798] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:24,799] jepsen worker 15 - jepsen.util 2445 :ok :read [5 43] INFO [2022-03-13 17:13:24,800] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:24,800] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,800] jepsen worker 15 - jepsen.util 2445 :invoke :read [5 nil] INFO [2022-03-13 17:13:24,801] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,802] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,802] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:24,804] jepsen worker 6 - jepsen.util 636 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,805] jepsen worker 6 - jepsen.util 636 :invoke :write [2 23] INFO [2022-03-13 17:13:24,813] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,814] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:13:24,826] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:24,826] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 46] INFO [2022-03-13 17:13:24,851] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,852] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:13:24,887] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,888] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:13:24,889] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,889] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,889] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:24,890] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:13:24,891] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,892] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:13:24,900] jepsen worker 24 - jepsen.util 564 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,901] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,901] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:13:24,902] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:24,903] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,903] jepsen worker 52 - jepsen.util 592 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,903] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,904] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:24,904] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:24,905] jepsen worker 52 - jepsen.util 592 :invoke :write [17 17] INFO [2022-03-13 17:13:24,955] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:24,956] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:24,958] jepsen worker 88 - jepsen.util 1978 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,959] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,971] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,971] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,972] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:24,973] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:13:24,985] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:24,986] jepsen worker 77 - jepsen.util 707 :invoke :write [25 19] INFO [2022-03-13 17:13:24,987] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:24,988] jepsen worker 88 - jepsen.util 1978 :invoke :read [29 nil] INFO [2022-03-13 17:13:24,999] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,000] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:13:25,023] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,026] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:13:25,043] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:25,044] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:13:25,053] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,054] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:25,056] jepsen worker 88 - jepsen.util 1978 :ok :read [29 10] INFO [2022-03-13 17:13:25,057] jepsen worker 88 - jepsen.util 1978 :invoke :write [29 29] INFO [2022-03-13 17:13:25,081] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,081] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,081] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:25,082] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:25,087] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,088] jepsen worker 79 - jepsen.util 1069 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,088] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,088] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,089] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:25,090] jepsen worker 53 - jepsen.util 503 :invoke :write [17 17] INFO [2022-03-13 17:13:25,091] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:25,092] jepsen worker 7 - jepsen.util 637 :invoke :write [2 32] INFO [2022-03-13 17:13:25,099] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,099] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,099] jepsen worker 13 - jepsen.util 643 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,100] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:13:25,100] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:25,100] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:25,132] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,133] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:25,137] jepsen worker 15 - jepsen.util 2445 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,138] jepsen worker 15 - jepsen.util 2445 :invoke :write [5 4] INFO [2022-03-13 17:13:25,153] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,154] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:25,164] jepsen worker 67 - jepsen.util 1957 :info :write [22 33] :RequestTimeoutException INFO [2022-03-13 17:13:25,164] jepsen worker 67 - jepsen.util 2047 :invoke :write [22 18] INFO [2022-03-13 17:13:25,174] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,174] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,174] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:13:25,174] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,174] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,175] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:13:25,175] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,175] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:13:25,176] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:25,176] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:13:25,186] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,186] jepsen worker 78 - jepsen.util 528 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,186] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,187] jepsen worker 80 - jepsen.util 710 :invoke :write [26 3] INFO [2022-03-13 17:13:25,187] jepsen worker 78 - jepsen.util 528 :invoke :write [26 33] INFO [2022-03-13 17:13:25,188] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:25,284] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,285] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:13:25,296] jepsen worker 60 - jepsen.util 600 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,296] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,296] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:25,297] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,297] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,298] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:25,299] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:25,300] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:25,301] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:25,301] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:25,323] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,323] jepsen worker 52 - jepsen.util 592 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,323] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,323] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:25,323] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,324] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:25,325] jepsen worker 34 - jepsen.util 754 :invoke :write [11 47] INFO [2022-03-13 17:13:25,325] jepsen worker 52 - jepsen.util 592 :invoke :write [17 47] INFO [2022-03-13 17:13:25,336] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,336] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,336] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,337] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:25,338] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:25,339] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:25,384] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,385] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:13:25,387] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,388] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,388] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,388] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,388] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:25,389] jepsen worker 14 - jepsen.util 284 :invoke :write [4 31] INFO [2022-03-13 17:13:25,389] jepsen worker 26 - jepsen.util 476 :invoke :write [8 24] INFO [2022-03-13 17:13:25,390] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:25,393] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,394] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:25,402] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,403] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:13:25,448] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:25,449] jepsen worker nemesis - couchbase.util Rebalance check count: 315 INFO [2022-03-13 17:13:25,449] jepsen worker nemesis - couchbase.util Rebalance stuck count: 215 INFO [2022-03-13 17:13:25,471] jepsen worker 3 - jepsen.util 1893 :ok :read [1 42] INFO [2022-03-13 17:13:25,472] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:13:25,474] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,474] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:13:25,492] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,492] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,493] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,493] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,494] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:13:25,494] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:13:25,495] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:13:25,498] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:25,500] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,501] jepsen worker 80 - jepsen.util 710 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,501] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:13:25,503] jepsen worker 80 - jepsen.util 710 :invoke :write [26 1] INFO [2022-03-13 17:13:25,592] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,592] jepsen worker 51 - jepsen.util 591 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,593] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:13:25,594] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:25,597] jepsen worker 4 - jepsen.util 2254 :ok :read [1 42] INFO [2022-03-13 17:13:25,598] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:13:25,680] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,680] jepsen worker 8 - jepsen.util 728 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,680] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,681] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,681] jepsen worker 61 - jepsen.util 601 :invoke :write [20 47] INFO [2022-03-13 17:13:25,682] jepsen worker 8 - jepsen.util 728 :invoke :write [2 2] INFO [2022-03-13 17:13:25,682] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:25,683] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:25,706] jepsen worker 82 - jepsen.util 2332 :info :write [27 8] :RequestTimeoutException INFO [2022-03-13 17:13:25,707] jepsen worker 82 - jepsen.util 2422 :invoke :read [27 nil] INFO [2022-03-13 17:13:25,709] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,709] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,709] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,710] jepsen worker 78 - jepsen.util 528 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,711] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:25,711] jepsen worker 78 - jepsen.util 528 :invoke :write [26 18] INFO [2022-03-13 17:13:25,712] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:13:25,712] jepsen worker 70 - jepsen.util 700 :invoke :write [23 7] INFO [2022-03-13 17:13:25,755] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,756] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:25,776] jepsen worker 1 - jepsen.util 2161 :info :write [0 28] :RequestTimeoutException INFO [2022-03-13 17:13:25,776] jepsen worker 1 - jepsen.util 2251 :invoke :read [0 nil] INFO [2022-03-13 17:13:25,795] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,795] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:25,801] jepsen worker 52 - jepsen.util 592 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,802] jepsen worker 52 - jepsen.util 592 :invoke :write [17 34] INFO [2022-03-13 17:13:25,836] jepsen worker 2 - jepsen.util 2702 :info :write [0 36] :RequestTimeoutException INFO [2022-03-13 17:13:25,837] jepsen worker 2 - jepsen.util 2792 :invoke :write [0 40] INFO [2022-03-13 17:13:25,839] jepsen worker 76 - jepsen.util 706 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,840] jepsen worker 76 - jepsen.util 706 :invoke :write [25 45] INFO [2022-03-13 17:13:25,865] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,866] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,866] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:13:25,870] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,870] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:13:25,878] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:25,879] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,880] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:13:25,896] jepsen worker 10 - jepsen.util 2440 :info :write [3 24] :RequestTimeoutException INFO [2022-03-13 17:13:25,897] jepsen worker 10 - jepsen.util 2530 :invoke :write [3 10] INFO [2022-03-13 17:13:25,918] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,918] jepsen worker 34 - jepsen.util 754 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,919] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:25,920] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:25,924] jepsen worker 43 - jepsen.util 493 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,925] jepsen worker 43 - jepsen.util 493 :invoke :write [14 22] INFO [2022-03-13 17:13:25,974] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:25,975] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:13:26,009] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,011] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,011] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:26,011] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,044] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,045] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:26,058] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,058] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,059] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:26,060] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:26,062] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,062] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,062] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,062] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,062] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:13:26,063] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:13:26,064] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:13:26,064] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:13:26,077] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,077] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:26,089] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:26,089] jepsen worker 77 - jepsen.util 707 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,090] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,090] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:13:26,090] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17997111,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18120048,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9049110,"encode_duration_us":20241,"operation_id":"0x7e2b","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10005549},{"operation_name":"upsert","last_dispatch_duration_us":23596716,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23596716,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9039489,"encode_duration_us":18405,"operation_id":"0x7a8d","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10010398}],"total_count":2}} INFO [2022-03-13 17:13:26,090] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,090] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:13:26,090] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11200700,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11200700,"total_server_duration_us":676,"operation_id":"0x80e0","timeout_ms":30000,"last_server_duration_us":676,"total_duration_us":11201034},{"operation_name":"upsert","last_dispatch_duration_us":12147010,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":12147010,"total_server_duration_us":26774,"encode_duration_us":21062,"operation_id":"0x807c","timeout_ms":10000,"last_server_duration_us":26774,"total_duration_us":10010707},{"operation_name":"upsert","last_dispatch_duration_us":38685,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8579870,"total_server_duration_us":292404,"encode_duration_us":23605,"operation_id":"0x83cc","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10008630},{"operation_name":"upsert","last_dispatch_duration_us":9036447,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9036447,"total_server_duration_us":21977,"encode_duration_us":23156,"operation_id":"0x7fba","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10007695},{"operation_name":"upsert","last_dispatch_duration_us":7634658,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7634658,"total_server_duration_us":22306,"encode_duration_us":27299,"operation_id":"0x82d2","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10002208},{"operation_name":"get","last_dispatch_duration_us":9041303,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9041303,"total_server_duration_us":135,"operation_id":"0x8214","timeout_ms":30000,"last_server_duration_us":135,"total_duration_us":9043847},{"operation_name":"get","last_dispatch_duration_us":111988,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":111988,"total_server_duration_us":10272,"operation_id":"0x8505","timeout_ms":30000,"last_server_duration_us":10272,"total_duration_us":6305178},{"operation_name":"get","last_dispatch_duration_us":37401,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":37401,"total_server_duration_us":62,"operation_id":"0x85aa","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":5201331},{"operation_name":"upsert","last_dispatch_duration_us":1156264,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1156264,"total_server_duration_us":47876,"encode_duration_us":25681,"operation_id":"0x89be","timeout_ms":10000,"last_server_duration_us":47876,"total_duration_us":1168124},{"operation_name":"upsert","last_dispatch_duration_us":1085024,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1085024,"total_server_duration_us":73410,"encode_duration_us":17064,"operation_id":"0x89bc","timeout_ms":10000,"last_server_duration_us":73410,"total_duration_us":1090403}],"total_count":18}} INFO [2022-03-13 17:13:26,090] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,092] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:26,092] jepsen worker 12 - jepsen.util 732 :invoke :write [4 30] INFO [2022-03-13 17:13:26,093] jepsen worker 60 - jepsen.util 600 :invoke :write [20 1] INFO [2022-03-13 17:13:26,095] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:13:26,096] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:26,097] jepsen worker 77 - jepsen.util 707 :invoke :write [25 47] INFO [2022-03-13 17:13:26,099] jepsen worker 69 - jepsen.util 699 :invoke :write [23 32] INFO [2022-03-13 17:13:26,105] jepsen worker 80 - jepsen.util 710 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,106] jepsen worker 80 - jepsen.util 710 :invoke :write [26 5] INFO [2022-03-13 17:13:26,113] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,115] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:26,115] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,117] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:26,124] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,126] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,127] jepsen worker 48 - jepsen.util 2298 :info :write [16 24] :RequestTimeoutException INFO [2022-03-13 17:13:26,128] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:26,132] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:26,133] jepsen worker 48 - jepsen.util 2388 :invoke :read [16 nil] INFO [2022-03-13 17:13:26,136] jepsen worker 45 - jepsen.util 2655 :info :write [15 25] :SyncWriteAmbiguous INFO [2022-03-13 17:13:26,138] jepsen worker 45 - jepsen.util 2745 :invoke :write [15 5] INFO [2022-03-13 17:13:26,139] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,139] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,142] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:13:26,142] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:26,144] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,145] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:13:26,147] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,147] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:13:26,147] jepsen worker 6 - jepsen.util 636 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,148] jepsen worker 6 - jepsen.util 636 :invoke :write [2 13] INFO [2022-03-13 17:13:26,154] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,155] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 45] INFO [2022-03-13 17:13:26,157] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,158] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,181] jepsen worker 87 - jepsen.util 2607 :ok :read [29 10] INFO [2022-03-13 17:13:26,181] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:13:26,199] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,200] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:26,252] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,252] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,253] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:13:26,254] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:26,256] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,256] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,257] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:26,258] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:13:26,266] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,267] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 33] INFO [2022-03-13 17:13:26,297] jepsen worker 76 - jepsen.util 706 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,297] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,298] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,298] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:13:26,299] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:26,300] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:26,310] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:26,311] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:26,311] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:26,311] jepsen worker 29 - jepsen.util 2549 :ok :read [9 26] INFO [2022-03-13 17:13:26,312] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,312] jepsen worker 29 - jepsen.util 2549 :invoke :write [9 16] INFO [2022-03-13 17:13:26,320] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,320] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:26,328] jepsen worker 49 - jepsen.util 2029 :info :write [16 25] :RequestTimeoutException INFO [2022-03-13 17:13:26,328] jepsen worker 49 - jepsen.util 2119 :invoke :read [16 nil] INFO [2022-03-13 17:13:26,340] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,341] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:26,342] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,343] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:26,355] jepsen worker 89 - jepsen.util 2429 :ok :read [29 10] INFO [2022-03-13 17:13:26,355] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:26,360] jepsen worker 73 - jepsen.util 2593 :info :write [24 26] :RequestTimeoutException INFO [2022-03-13 17:13:26,361] jepsen worker 73 - jepsen.util 2683 :invoke :read [24 nil] INFO [2022-03-13 17:13:26,363] jepsen worker 17 - jepsen.util 2357 :info :write [5 8] :SyncWriteAmbiguous INFO [2022-03-13 17:13:26,364] jepsen worker 17 - jepsen.util 2447 :invoke :write [5 24] INFO [2022-03-13 17:13:26,368] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,368] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,369] jepsen worker 21 - jepsen.util 831 :invoke :write [7 1] INFO [2022-03-13 17:13:26,370] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:13:26,377] jepsen worker 53 - jepsen.util 503 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,378] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,378] jepsen worker 70 - jepsen.util 700 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,378] jepsen worker 53 - jepsen.util 503 :invoke :write [17 20] INFO [2022-03-13 17:13:26,379] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:26,381] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,382] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:26,383] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,383] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:26,386] jepsen worker 70 - jepsen.util 700 :invoke :write [23 13] INFO [2022-03-13 17:13:26,436] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,437] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:13:26,455] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,455] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:26,455] jepsen worker nemesis - couchbase.util Rebalance check count: 316 INFO [2022-03-13 17:13:26,455] jepsen worker nemesis - couchbase.util Rebalance stuck count: 216 INFO [2022-03-13 17:13:26,455] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:13:26,472] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,473] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 9] INFO [2022-03-13 17:13:26,504] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,505] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,505] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:13:26,506] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:13:26,511] jepsen worker 78 - jepsen.util 528 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,511] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,511] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:26,512] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:13:26,533] jepsen worker 37 - jepsen.util 2017 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,534] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:26,558] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,558] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,559] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:26,559] jepsen worker 33 - jepsen.util 663 :invoke :write [11 11] INFO [2022-03-13 17:13:26,574] jepsen worker 5 - jepsen.util 2435 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,574] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,575] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:13:26,576] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:13:26,582] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,583] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,583] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:13:26,584] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:26,610] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,610] jepsen worker 35 - jepsen.util 935 :invoke :write [11 17] INFO [2022-03-13 17:13:26,612] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,612] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:26,641] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,642] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:26,652] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,653] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,654] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,655] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:26,656] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,657] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 33] INFO [2022-03-13 17:13:26,659] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,660] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:13:26,691] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,691] jepsen worker 52 - jepsen.util 592 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,691] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,691] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:26,692] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:26,693] jepsen worker 13 - jepsen.util 643 :invoke :write [4 17] INFO [2022-03-13 17:13:26,699] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,701] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:13:26,724] jepsen worker 21 - jepsen.util 831 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,725] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:13:26,731] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,732] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:13:26,739] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,739] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:26,762] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,763] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,764] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,765] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,768] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,768] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,775] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,775] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,777] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,778] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:26,786] jepsen worker 63 - jepsen.util 2223 :info :write [21 13] :RequestTimeoutException INFO [2022-03-13 17:13:26,787] jepsen worker 63 - jepsen.util 2313 :invoke :write [21 20] INFO [2022-03-13 17:13:26,796] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,796] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:26,796] jepsen worker 7 - jepsen.util 637 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,797] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:26,797] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:26,798] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:26,843] jepsen worker 64 - jepsen.util 2404 :info :write [21 1] :RequestTimeoutException INFO [2022-03-13 17:13:26,844] jepsen worker 64 - jepsen.util 2494 :invoke :read [21 nil] INFO [2022-03-13 17:13:26,847] jepsen worker 3 - jepsen.util 1893 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,848] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:13:26,855] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,855] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,856] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:26,857] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:26,858] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,859] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:26,860] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,862] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:26,863] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,864] jepsen worker 18 - jepsen.util 2178 :info :write [6 43] :RequestTimeoutException INFO [2022-03-13 17:13:26,864] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:26,864] jepsen worker 18 - jepsen.util 2268 :invoke :read [6 nil] INFO [2022-03-13 17:13:26,866] jepsen worker 83 - jepsen.util 1793 :info :write [27 21] :RequestTimeoutException INFO [2022-03-13 17:13:26,866] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,867] jepsen worker 83 - jepsen.util 1883 :invoke :read [27 nil] INFO [2022-03-13 17:13:26,868] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:26,869] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,869] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,869] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:13:26,870] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:26,871] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,873] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:26,886] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,887] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:26,889] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,890] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:13:26,910] jepsen worker 12 - jepsen.util 732 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,910] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:26,931] jepsen worker 80 - jepsen.util 710 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,932] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:26,935] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,936] jepsen worker 22 - jepsen.util 652 :invoke :write [7 32] INFO [2022-03-13 17:13:26,946] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,947] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:26,986] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,986] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:13:26,990] jepsen worker 14 - jepsen.util 284 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:26,991] jepsen worker 14 - jepsen.util 284 :invoke :write [4 27] INFO [2022-03-13 17:13:27,003] jepsen worker 35 - jepsen.util 935 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,004] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,004] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:13:27,005] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:13:27,015] jepsen worker 33 - jepsen.util 663 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,016] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:27,068] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,069] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,075] jepsen worker 70 - jepsen.util 700 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,076] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:27,077] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,077] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,077] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,078] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:27,078] jepsen worker 52 - jepsen.util 592 :invoke :write [17 41] INFO [2022-03-13 17:13:27,079] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:27,097] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,099] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:13:27,110] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,110] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,114] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,115] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,115] jepsen worker 21 - jepsen.util 831 :invoke :write [7 35] INFO [2022-03-13 17:13:27,115] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:27,116] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,117] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,119] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,119] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:27,128] jepsen worker 77 - jepsen.util 707 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,129] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:27,162] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,162] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,162] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,162] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:13:27,163] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,164] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:13:27,164] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,165] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,166] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,167] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,168] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,169] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,170] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,171] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:13:27,171] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,172] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:13:27,174] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,174] jepsen worker 26 - jepsen.util 476 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,175] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:13:27,179] jepsen worker 26 - jepsen.util 476 :invoke :write [8 11] INFO [2022-03-13 17:13:27,202] jepsen worker 22 - jepsen.util 652 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,203] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:13:27,204] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,204] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,205] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:27,205] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,208] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,208] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,209] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 37] INFO [2022-03-13 17:13:27,210] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:27,212] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,213] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:27,241] jepsen worker 69 - jepsen.util 699 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,241] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:27,271] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,271] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:13:27,276] jepsen worker 43 - jepsen.util 493 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,276] jepsen worker 43 - jepsen.util 493 :invoke :write [14 4] INFO [2022-03-13 17:13:27,282] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,283] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,283] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:13:27,284] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:13:27,288] jepsen worker 13 - jepsen.util 643 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,288] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,288] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,289] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:27,290] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:27,290] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,290] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:27,290] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,291] jepsen worker 70 - jepsen.util 700 :invoke :write [23 33] INFO [2022-03-13 17:13:27,296] jepsen worker 78 - jepsen.util 528 :invoke :write [26 35] INFO [2022-03-13 17:13:27,296] jepsen worker 89 - jepsen.util 2429 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,296] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,297] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,298] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:27,298] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,299] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,300] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,300] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:27,312] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,312] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,313] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:13:27,318] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:13:27,351] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,352] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:13:27,400] jepsen worker 60 - jepsen.util 600 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,401] jepsen worker 60 - jepsen.util 600 :invoke :write [20 2] INFO [2022-03-13 17:13:27,401] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,401] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,402] jepsen worker 76 - jepsen.util 706 :invoke :write [25 41] INFO [2022-03-13 17:13:27,403] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:27,407] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:27,407] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,407] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,407] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,407] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,408] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:27,409] jepsen worker 77 - jepsen.util 707 :invoke :write [25 16] INFO [2022-03-13 17:13:27,410] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:27,410] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:27,411] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:27,420] jepsen worker 52 - jepsen.util 592 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,420] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,420] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,420] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,420] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,420] jepsen worker 52 - jepsen.util 592 :invoke :write [17 49] INFO [2022-03-13 17:13:27,421] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,422] jepsen worker 80 - jepsen.util 710 :invoke :write [26 23] INFO [2022-03-13 17:13:27,422] jepsen worker 34 - jepsen.util 754 :invoke :write [11 18] INFO [2022-03-13 17:13:27,426] jepsen worker 21 - jepsen.util 831 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,427] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:13:27,433] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:27,435] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,435] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,435] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:13:27,435] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,436] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:13:27,437] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:27,438] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,439] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:13:27,448] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,448] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:27,462] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:27,462] jepsen worker nemesis - couchbase.util Rebalance check count: 317 INFO [2022-03-13 17:13:27,462] jepsen worker nemesis - couchbase.util Rebalance stuck count: 217 INFO [2022-03-13 17:13:27,476] jepsen worker 79 - jepsen.util 1069 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,476] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,477] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,477] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,477] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,477] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:27,479] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:27,480] jepsen worker 51 - jepsen.util 591 :invoke :write [17 8] INFO [2022-03-13 17:13:27,481] jepsen worker 44 - jepsen.util 584 :invoke :write [14 39] INFO [2022-03-13 17:13:27,482] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:27,483] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:27,483] jepsen worker 23 - jepsen.util 563 :invoke :write [7 23] INFO [2022-03-13 17:13:27,502] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,503] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:27,514] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,515] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:13:27,517] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,518] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:27,560] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,561] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:13:27,566] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,566] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:13:27,569] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,570] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] WARN [2022-03-13 17:13:27,594] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21495302,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21495302,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9034183,"encode_duration_us":18871,"operation_id":"0x7b45","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10005388}],"total_count":1}} INFO [2022-03-13 17:13:27,601] jepsen worker 78 - jepsen.util 528 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,602] jepsen worker 78 - jepsen.util 528 :invoke :write [26 27] INFO [2022-03-13 17:13:27,606] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,606] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 26] INFO [2022-03-13 17:13:27,620] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,621] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:27,639] jepsen worker 70 - jepsen.util 700 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,639] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,639] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:27,640] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:27,665] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,666] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:27,672] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,673] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:27,675] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,675] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] WARN [2022-03-13 17:13:27,678] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11429054,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12045552,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9252988,"encode_duration_us":34345,"operation_id":"0x827a","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10002929}],"total_count":1}} INFO [2022-03-13 17:13:27,693] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,693] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] WARN [2022-03-13 17:13:27,695] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8354421,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":16264188,"total_server_duration_us":299589,"encode_duration_us":16967,"operation_id":"0x8081","timeout_ms":10000,"last_server_duration_us":55790,"total_duration_us":10010210},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":22731,"operation_id":"0x8121","timeout_ms":10000,"total_duration_us":10009007},{"operation_name":"upsert","last_dispatch_duration_us":29488,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":130534,"total_server_duration_us":88116,"encode_duration_us":24183,"operation_id":"0x8555","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10001586},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":18550,"operation_id":"0x81a0","timeout_ms":10000,"total_duration_us":10001278},{"operation_name":"upsert","last_dispatch_duration_us":9123270,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9216420,"total_server_duration_us":9057387,"encode_duration_us":17798,"operation_id":"0x84c9","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":9236438},{"operation_name":"upsert","last_dispatch_duration_us":9120150,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9200220,"total_server_duration_us":9066829,"encode_duration_us":17827,"operation_id":"0x84da","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":9216771},{"operation_name":"upsert","last_dispatch_duration_us":2190998,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2190998,"total_server_duration_us":93498,"encode_duration_us":23979,"operation_id":"0x880e","timeout_ms":10000,"last_server_duration_us":93498,"total_duration_us":4305090},{"operation_name":"upsert","last_dispatch_duration_us":4083892,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4083892,"total_server_duration_us":82115,"encode_duration_us":34605,"operation_id":"0x87d4","timeout_ms":10000,"last_server_duration_us":82115,"total_duration_us":4260662},{"operation_name":"get","last_dispatch_duration_us":1948643,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1948643,"total_server_duration_us":19,"operation_id":"0x8807","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":4058197},{"operation_name":"get","last_dispatch_duration_us":1948786,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1948786,"total_server_duration_us":15,"operation_id":"0x8808","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":4057636}],"total_count":57}} INFO [2022-03-13 17:13:27,695] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,696] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,697] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,697] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,698] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:13:27,699] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,700] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,701] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 8] INFO [2022-03-13 17:13:27,718] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,720] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] WARN [2022-03-13 17:13:27,725] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":20421383,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20421383,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9033121,"encode_duration_us":22305,"operation_id":"0x7cc4","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10002099}],"total_count":1}} INFO [2022-03-13 17:13:27,725] jepsen worker 6 - jepsen.util 636 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,726] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:27,726] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:13:27,726] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17215904,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":17215904,"total_server_duration_us":7489,"operation_id":"0x7e76","timeout_ms":30000,"last_server_duration_us":7489,"total_duration_us":17219011},{"operation_name":"get","last_dispatch_duration_us":13894660,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13894660,"total_server_duration_us":49,"operation_id":"0x804d","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":13897608},{"operation_name":"upsert","last_dispatch_duration_us":54751,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1627289,"total_server_duration_us":604425,"encode_duration_us":18662,"operation_id":"0x850b","timeout_ms":10000,"last_server_duration_us":31489,"total_duration_us":10005567},{"operation_name":"upsert","last_dispatch_duration_us":113807,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1836074,"total_server_duration_us":537974,"encode_duration_us":31272,"operation_id":"0x8517","timeout_ms":10000,"last_server_duration_us":31585,"total_duration_us":10005520},{"operation_name":"upsert","last_dispatch_duration_us":114070,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1502052,"total_server_duration_us":614747,"encode_duration_us":23013,"operation_id":"0x85b2","timeout_ms":10000,"last_server_duration_us":72999,"total_duration_us":10004909},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17442,"operation_id":"0x80df","timeout_ms":10000,"total_duration_us":10003624},{"operation_name":"upsert","last_dispatch_duration_us":555830,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1471443,"total_server_duration_us":203491,"encode_duration_us":34498,"operation_id":"0x8a7e","timeout_ms":10000,"last_server_duration_us":96865,"total_duration_us":1484933},{"operation_name":"upsert","last_dispatch_duration_us":1179165,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1179165,"total_server_duration_us":77851,"encode_duration_us":22624,"operation_id":"0x8a91","timeout_ms":10000,"last_server_duration_us":77851,"total_duration_us":1182843},{"operation_name":"upsert","last_dispatch_duration_us":1131383,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1131383,"total_server_duration_us":877299,"encode_duration_us":22723,"operation_id":"0x89aa","timeout_ms":10000,"last_server_duration_us":877299,"total_duration_us":1137665},{"operation_name":"upsert","last_dispatch_duration_us":1011773,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1011773,"total_server_duration_us":86622,"encode_duration_us":19855,"operation_id":"0x89d1","timeout_ms":10000,"last_server_duration_us":86622,"total_duration_us":1016725}],"total_count":26}} INFO [2022-03-13 17:13:27,726] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,727] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,727] jepsen worker 62 - jepsen.util 512 :invoke :write [20 10] INFO [2022-03-13 17:13:27,728] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,729] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:27,729] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:27,730] jepsen worker 42 - jepsen.util 402 :invoke :write [14 21] INFO [2022-03-13 17:13:27,731] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:13:27,732] jepsen worker 7 - jepsen.util 637 :invoke :write [2 44] INFO [2022-03-13 17:13:27,757] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,758] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] WARN [2022-03-13 17:13:27,777] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9130608,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9130608,"total_server_duration_us":7282,"operation_id":"0x83cf","timeout_ms":30000,"last_server_duration_us":7282,"total_duration_us":10741256},{"operation_name":"get","last_dispatch_duration_us":10353343,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10353343,"total_server_duration_us":27,"operation_id":"0x820e","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":10353656},{"operation_name":"upsert","last_dispatch_duration_us":46945,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":5587170,"total_server_duration_us":238696,"encode_duration_us":18423,"operation_id":"0x855a","timeout_ms":10000,"last_server_duration_us":29045,"total_duration_us":10003404},{"operation_name":"upsert","last_dispatch_duration_us":45290,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8525495,"total_server_duration_us":239871,"encode_duration_us":18996,"operation_id":"0x8364","timeout_ms":10000,"last_server_duration_us":32840,"total_duration_us":10002543},{"operation_name":"upsert","last_dispatch_duration_us":16457411,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16457411,"total_server_duration_us":22141,"encode_duration_us":20030,"operation_id":"0x808a","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10002533},{"operation_name":"get","last_dispatch_duration_us":9051295,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9051295,"total_server_duration_us":38,"operation_id":"0x84e9","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9053746},{"operation_name":"get","last_dispatch_duration_us":6579360,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":6579360,"total_server_duration_us":49,"operation_id":"0x8652","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":6579621},{"operation_name":"upsert","last_dispatch_duration_us":186317,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":887823,"total_server_duration_us":328420,"encode_duration_us":20399,"operation_id":"0x8aa4","timeout_ms":10000,"last_server_duration_us":66825,"total_duration_us":945865},{"operation_name":"get","last_dispatch_duration_us":771552,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":771552,"total_server_duration_us":770791,"operation_id":"0x89e0","timeout_ms":30000,"last_server_duration_us":770791,"total_duration_us":773876},{"operation_name":"upsert","last_dispatch_duration_us":219775,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":608619,"total_server_duration_us":124312,"encode_duration_us":17744,"operation_id":"0x8ad5","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":636572}],"total_count":10}} INFO [2022-03-13 17:13:27,784] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,784] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,784] jepsen worker 80 - jepsen.util 710 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,785] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:27,785] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:27,786] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:27,787] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,787] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:13:27,804] jepsen worker 3 - jepsen.util 1893 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,804] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,805] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:13:27,805] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:13:27,819] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,820] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:27,830] jepsen worker 89 - jepsen.util 2429 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,831] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,835] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,836] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,837] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,838] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,839] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,840] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,841] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,841] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:13:27,846] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,847] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:27,865] jepsen worker 76 - jepsen.util 706 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,866] jepsen worker 76 - jepsen.util 706 :invoke :write [25 14] WARN [2022-03-13 17:13:27,877] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10458258,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10458258,"total_server_duration_us":1345723,"operation_id":"0x8518","timeout_ms":30000,"last_server_duration_us":1345723,"total_duration_us":10462061},{"operation_name":"upsert","last_dispatch_duration_us":16029638,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16029638,"total_server_duration_us":20841,"encode_duration_us":17980,"operation_id":"0x80e1","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10008941},{"operation_name":"upsert","last_dispatch_duration_us":8847031,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10694512,"total_server_duration_us":222489,"encode_duration_us":19853,"operation_id":"0x84f9","timeout_ms":10000,"last_server_duration_us":69214,"total_duration_us":10008607},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":21575,"operation_id":"0x85a1","timeout_ms":10000,"total_duration_us":10006972},{"operation_name":"upsert","last_dispatch_duration_us":9119123,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9119123,"total_server_duration_us":48909,"encode_duration_us":19492,"operation_id":"0x8520","timeout_ms":10000,"last_server_duration_us":48909,"total_duration_us":10005220},{"operation_name":"upsert","last_dispatch_duration_us":8787014,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10327941,"total_server_duration_us":167200,"encode_duration_us":17063,"operation_id":"0x853d","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10003381},{"operation_name":"upsert","last_dispatch_duration_us":8788850,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12481365,"total_server_duration_us":212214,"encode_duration_us":17309,"operation_id":"0x83c1","timeout_ms":10000,"last_server_duration_us":58874,"total_duration_us":10002223},{"operation_name":"upsert","last_dispatch_duration_us":9239192,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9438458,"total_server_duration_us":9077149,"encode_duration_us":20358,"operation_id":"0x85b4","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9455701},{"operation_name":"upsert","last_dispatch_duration_us":1536943,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1536943,"total_server_duration_us":70960,"encode_duration_us":33486,"operation_id":"0x897c","timeout_ms":10000,"last_server_duration_us":70960,"total_duration_us":1544962},{"operation_name":"upsert","last_dispatch_duration_us":1486939,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1486939,"total_server_duration_us":37036,"encode_duration_us":17165,"operation_id":"0x899c","timeout_ms":10000,"last_server_duration_us":37036,"total_duration_us":1493545}],"total_count":57}} WARN [2022-03-13 17:13:27,877] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17706216,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18557696,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9469872,"encode_duration_us":18657,"operation_id":"0x7b28","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10000876},{"operation_name":"upsert","last_dispatch_duration_us":18241714,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18241714,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9059667,"encode_duration_us":16542,"operation_id":"0x7f6d","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":10009488},{"operation_name":"upsert","last_dispatch_duration_us":18193922,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18193922,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9049045,"encode_duration_us":17757,"operation_id":"0x7f76","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10003258}],"total_count":3}} WARN [2022-03-13 17:13:27,878] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8839999,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11884192,"total_server_duration_us":259604,"encode_duration_us":17665,"operation_id":"0x8283","timeout_ms":10000,"last_server_duration_us":31393,"total_duration_us":10008171},{"operation_name":"upsert","last_dispatch_duration_us":993364,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":993364,"total_server_duration_us":19340,"encode_duration_us":18078,"operation_id":"0x83fd","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10008001},{"operation_name":"upsert","last_dispatch_duration_us":54388,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9055647,"total_server_duration_us":261991,"encode_duration_us":20298,"operation_id":"0x85b1","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10007769},{"operation_name":"upsert","last_dispatch_duration_us":9006728,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9006728,"total_server_duration_us":23475,"encode_duration_us":20727,"operation_id":"0x8498","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10007458},{"operation_name":"upsert","last_dispatch_duration_us":49284,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9027866,"total_server_duration_us":272815,"encode_duration_us":17594,"operation_id":"0x85ad","timeout_ms":10000,"last_server_duration_us":33721,"total_duration_us":10002972},{"operation_name":"upsert","last_dispatch_duration_us":11429054,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12045552,"total_server_duration_us":9252988,"encode_duration_us":34345,"operation_id":"0x827a","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10002929},{"operation_name":"upsert","last_dispatch_duration_us":8901483,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9482350,"total_server_duration_us":6637160,"encode_duration_us":23520,"operation_id":"0x82a0","timeout_ms":10000,"last_server_duration_us":6493993,"total_duration_us":9625091},{"operation_name":"get","last_dispatch_duration_us":9110949,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9110949,"total_server_duration_us":43,"operation_id":"0x847e","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9112348},{"operation_name":"upsert","last_dispatch_duration_us":9095637,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9095637,"total_server_duration_us":9047983,"encode_duration_us":28632,"operation_id":"0x85be","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9100498},{"operation_name":"upsert","last_dispatch_duration_us":2279548,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2279548,"total_server_duration_us":98879,"encode_duration_us":27079,"operation_id":"0x865d","timeout_ms":10000,"last_server_duration_us":98879,"total_duration_us":6940606}],"total_count":50}} INFO [2022-03-13 17:13:27,891] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,891] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:13:27,896] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,897] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:13:27,916] jepsen worker 34 - jepsen.util 754 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,917] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:27,924] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,924] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:13:27,937] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,938] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:27,952] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,953] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,953] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:13:27,953] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:27,955] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,955] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,956] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:27,956] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:13:27,963] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,964] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:13:27,970] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,970] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:13:27,974] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,975] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] WARN [2022-03-13 17:13:27,977] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18097150,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26316030,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9051681,"encode_duration_us":17353,"operation_id":"0x7b97","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10007407}],"total_count":1}} INFO [2022-03-13 17:13:27,983] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,983] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:27,985] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,986] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:27,998] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:27,999] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:13:28,002] jepsen worker 53 - jepsen.util 503 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,002] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:28,034] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,034] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,036] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,036] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:28,045] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,046] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:13:28,067] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,068] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,068] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,068] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,113] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:13:28,069] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:13:28,113] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:28,072] jepsen worker 60 - jepsen.util 600 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,078] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,113] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:28,078] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,113] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:28,113] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:28,084] jepsen worker 52 - jepsen.util 592 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,114] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:28,113] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:28,118] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,119] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:13:28,123] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,124] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:13:28,147] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,148] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,159] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,159] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,162] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,163] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,165] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,165] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:13:28,165] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,166] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:13:28,180] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,180] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:13:28,183] jepsen worker 40 - jepsen.util 2470 :info :write [13 26] :RequestTimeoutException INFO [2022-03-13 17:13:28,183] jepsen worker 40 - jepsen.util 2560 :invoke :read [13 nil] INFO [2022-03-13 17:13:28,194] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,194] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,194] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,195] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:13:28,195] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:28,196] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:13:28,204] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,204] jepsen worker 21 - jepsen.util 831 :invoke :write [7 15] INFO [2022-03-13 17:13:28,212] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,213] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,220] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,220] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:28,245] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,246] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:13:28,258] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,259] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:28,259] jepsen worker 14 - jepsen.util 284 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,260] jepsen worker 14 - jepsen.util 284 :invoke :write [4 40] INFO [2022-03-13 17:13:28,267] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,267] jepsen worker 77 - jepsen.util 707 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,267] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,267] jepsen worker 12 - jepsen.util 732 :invoke :write [4 45] INFO [2022-03-13 17:13:28,268] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:28,268] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:28,281] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,281] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,281] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,282] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:28,290] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,290] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,291] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:13:28,292] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:13:28,295] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,295] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:13:28,308] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,308] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:28,316] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,316] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,316] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,316] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,316] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,316] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,316] jepsen worker 78 - jepsen.util 528 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,318] jepsen worker 52 - jepsen.util 592 :invoke :write [17 43] INFO [2022-03-13 17:13:28,319] jepsen worker 13 - jepsen.util 643 :invoke :write [4 47] INFO [2022-03-13 17:13:28,320] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:28,320] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,321] jepsen worker 80 - jepsen.util 710 :invoke :write [26 35] INFO [2022-03-13 17:13:28,321] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,331] jepsen worker 70 - jepsen.util 700 :invoke :write [23 34] INFO [2022-03-13 17:13:28,332] jepsen worker 78 - jepsen.util 528 :invoke :write [26 15] INFO [2022-03-13 17:13:28,333] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,333] jepsen worker 24 - jepsen.util 564 :invoke :write [8 2] INFO [2022-03-13 17:13:28,334] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:28,338] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,345] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,347] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,349] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,350] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,351] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,352] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,352] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,354] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,354] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:28,372] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,373] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,373] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,374] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:13:28,387] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,387] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,388] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,388] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:28,389] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,389] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,390] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,390] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,390] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:28,391] jepsen worker 22 - jepsen.util 652 :invoke :write [7 31] INFO [2022-03-13 17:13:28,391] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,392] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:28,420] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,421] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:13:28,424] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,424] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,424] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,425] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:13:28,426] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:28,427] jepsen worker 33 - jepsen.util 663 :invoke :write [11 37] INFO [2022-03-13 17:13:28,429] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,430] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:13:28,459] jepsen worker 26 - jepsen.util 476 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,459] jepsen worker 44 - jepsen.util 584 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,460] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:28,461] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:28,468] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:28,468] jepsen worker nemesis - couchbase.util Rebalance check count: 318 INFO [2022-03-13 17:13:28,468] jepsen worker nemesis - couchbase.util Rebalance stuck count: 218 INFO [2022-03-13 17:13:28,502] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,502] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,506] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,507] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:28,513] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,513] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:13:28,521] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,522] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 22] INFO [2022-03-13 17:13:28,553] jepsen worker 80 - jepsen.util 710 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,554] jepsen worker 80 - jepsen.util 710 :invoke :write [26 48] INFO [2022-03-13 17:13:28,572] jepsen worker 21 - jepsen.util 831 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,572] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,572] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:28,573] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:28,579] jepsen worker 24 - jepsen.util 564 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,579] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,580] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:28,581] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:13:28,626] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,627] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:28,632] jepsen worker 61 - jepsen.util 601 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,632] jepsen worker 61 - jepsen.util 601 :invoke :write [20 22] INFO [2022-03-13 17:13:28,639] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,640] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,649] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,649] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,654] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,655] jepsen worker 89 - jepsen.util 2429 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,655] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:13:28,659] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:13:28,680] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,680] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,681] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:28,682] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:28,685] jepsen worker 33 - jepsen.util 663 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,685] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:28,714] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,715] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:13:28,744] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,745] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:28,746] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,747] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:28,765] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,765] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,765] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:28,766] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:28,776] jepsen worker 13 - jepsen.util 643 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,776] jepsen worker 51 - jepsen.util 591 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,776] jepsen worker 51 - jepsen.util 591 :invoke :write [17 21] INFO [2022-03-13 17:13:28,777] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:28,824] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,824] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,825] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,826] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:28,835] jepsen worker 8 - jepsen.util 728 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,835] jepsen worker 8 - jepsen.util 728 :invoke :write [2 34] INFO [2022-03-13 17:13:28,850] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,851] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:13:28,852] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,853] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,853] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:13:28,853] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,858] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,859] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,859] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,860] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:13:28,860] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,862] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:13:28,863] jepsen worker 23 - jepsen.util 563 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,864] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:13:28,886] jepsen worker 27 - jepsen.util 2817 :info :write [9 4] :RequestTimeoutException INFO [2022-03-13 17:13:28,887] jepsen worker 27 - jepsen.util 2907 :invoke :write [9 39] INFO [2022-03-13 17:13:28,911] jepsen worker 22 - jepsen.util 652 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,912] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:13:28,920] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,920] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,921] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:13:28,921] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,921] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:13:28,922] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:13:28,928] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,929] jepsen worker 75 - jepsen.util 435 :invoke :write [25 40] INFO [2022-03-13 17:13:28,930] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,931] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:28,949] jepsen worker 76 - jepsen.util 706 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,949] jepsen worker 43 - jepsen.util 493 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,949] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,949] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,950] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,950] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,951] jepsen worker 53 - jepsen.util 503 :invoke :write [17 10] INFO [2022-03-13 17:13:28,952] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,953] jepsen worker 60 - jepsen.util 600 :invoke :write [20 2] INFO [2022-03-13 17:13:28,954] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:28,955] jepsen worker 43 - jepsen.util 493 :invoke :write [14 48] INFO [2022-03-13 17:13:28,955] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,956] jepsen worker 69 - jepsen.util 699 :invoke :write [23 38] INFO [2022-03-13 17:13:28,957] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:13:28,958] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:28,958] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:28,978] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,979] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:13:28,987] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:28,988] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:29,000] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,000] jepsen worker 52 - jepsen.util 592 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,001] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:29,001] jepsen worker 34 - jepsen.util 754 :invoke :write [11 49] INFO [2022-03-13 17:13:29,031] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,031] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,031] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,032] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:29,033] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:13:29,034] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:29,073] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,074] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:29,081] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,081] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:29,087] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,088] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:13:29,107] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,108] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:13:29,160] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,160] jepsen worker 75 - jepsen.util 435 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,160] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,161] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:29,162] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:29,162] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:29,167] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,167] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:29,181] jepsen worker 72 - jepsen.util 2412 :info :write [24 1] :SyncWriteAmbiguous INFO [2022-03-13 17:13:29,181] jepsen worker 78 - jepsen.util 528 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,181] jepsen worker 72 - jepsen.util 2502 :invoke :read [24 nil] INFO [2022-03-13 17:13:29,182] jepsen worker 78 - jepsen.util 528 :invoke :write [26 22] INFO [2022-03-13 17:13:29,211] jepsen worker 14 - jepsen.util 284 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,211] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,212] jepsen worker 12 - jepsen.util 732 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,212] jepsen worker 6 - jepsen.util 636 :invoke :write [2 31] INFO [2022-03-13 17:13:29,212] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:29,213] jepsen worker 12 - jepsen.util 732 :invoke :write [4 10] INFO [2022-03-13 17:13:29,214] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:29,214] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:29,230] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,231] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:29,238] jepsen worker 74 - jepsen.util 2684 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,239] jepsen worker 74 - jepsen.util 2684 :invoke :read [24 nil] INFO [2022-03-13 17:13:29,242] jepsen worker 65 - jepsen.util 2765 :ok :read [21 :nil] INFO [2022-03-13 17:13:29,243] jepsen worker 65 - jepsen.util 2765 :invoke :read [21 nil] INFO [2022-03-13 17:13:29,250] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,250] jepsen worker 47 - jepsen.util 2387 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,250] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:29,251] jepsen worker 47 - jepsen.util 2387 :invoke :read [15 nil] INFO [2022-03-13 17:13:29,251] jepsen worker 65 - jepsen.util 2765 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,252] jepsen worker 65 - jepsen.util 2765 :invoke :read [21 nil] INFO [2022-03-13 17:13:29,252] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,253] jepsen worker 65 - jepsen.util 2765 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,253] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:29,254] jepsen worker 65 - jepsen.util 2765 :invoke :write [21 3] INFO [2022-03-13 17:13:29,257] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,258] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,258] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,258] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,259] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] INFO [2022-03-13 17:13:29,260] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:29,261] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:29,262] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,262] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:29,262] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,263] jepsen worker 71 - jepsen.util 521 :invoke :write [23 12] INFO [2022-03-13 17:13:29,264] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,264] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:29,266] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:13:29,266] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,266] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:29,267] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,268] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:29,270] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,270] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:29,271] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,272] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:29,273] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,274] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:13:29,290] jepsen worker 47 - jepsen.util 2387 :ok :read [15 48] INFO [2022-03-13 17:13:29,291] jepsen worker 47 - jepsen.util 2387 :invoke :write [15 20] INFO [2022-03-13 17:13:29,299] jepsen worker 70 - jepsen.util 700 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,300] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:13:29,348] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,348] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,351] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,352] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:13:29,353] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,353] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,353] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,355] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:13:29,356] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:13:29,357] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,358] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,358] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,359] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:29,360] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:13:29,377] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,378] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:29,379] jepsen worker 71 - jepsen.util 521 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,380] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:29,444] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,445] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:13:29,460] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,461] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:13:29,474] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:29,474] jepsen worker nemesis - couchbase.util Rebalance check count: 319 INFO [2022-03-13 17:13:29,474] jepsen worker nemesis - couchbase.util Rebalance stuck count: 219 INFO [2022-03-13 17:13:29,485] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,485] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:29,488] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,489] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:29,490] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,491] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:29,492] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,492] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:29,493] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,493] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:29,514] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,514] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:29,528] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,529] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:13:29,589] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,590] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,590] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:13:29,590] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 22] INFO [2022-03-13 17:13:29,594] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,594] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:29,655] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,656] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,681] jepsen worker 89 - jepsen.util 2429 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,682] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,685] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,686] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:13:29,686] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,687] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:29,702] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,702] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,702] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,703] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:13:29,703] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:13:29,704] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:13:29,761] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,761] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:29,784] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,785] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:13:29,829] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,830] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:13:29,831] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,831] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,874] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,875] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 22] INFO [2022-03-13 17:13:29,884] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,884] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 30] INFO [2022-03-13 17:13:29,927] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,928] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:29,941] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,941] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,941] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:13:29,942] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,969] jepsen worker 89 - jepsen.util 2429 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,969] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,971] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,972] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,973] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,973] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:29,978] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,978] jepsen worker 42 - jepsen.util 402 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,978] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,978] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:29,979] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:29,979] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:29,979] jepsen worker 80 - jepsen.util 710 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,980] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,980] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,980] jepsen worker 80 - jepsen.util 710 :invoke :write [26 24] INFO [2022-03-13 17:13:29,980] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:29,981] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:29,985] jepsen worker 3 - jepsen.util 1893 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,985] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,985] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:13:29,986] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:29,998] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,998] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,999] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:29,999] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:13:30,000] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,000] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:30,009] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,010] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,010] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:13:30,015] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,020] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,020] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,029] jepsen worker 51 - jepsen.util 591 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,029] jepsen worker 51 - jepsen.util 591 :invoke :write [17 47] INFO [2022-03-13 17:13:30,035] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,036] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,044] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,045] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:30,066] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,067] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,067] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,068] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,084] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,085] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,087] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,088] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:13:30,122] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,122] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,128] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,128] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,128] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,128] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,129] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:13:30,130] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:13:30,130] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,131] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,132] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,132] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,132] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,133] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:13:30,134] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,134] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:13:30,143] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,144] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:13:30,145] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,145] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,145] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:13:30,146] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:13:30,144] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,146] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:30,155] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,156] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:13:30,168] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,169] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,172] jepsen worker 61 - jepsen.util 601 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,173] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,173] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:13:30,173] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,175] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,176] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,178] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,178] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:13:30,235] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,235] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,236] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:13:30,236] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:13:30,239] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,239] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,246] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,247] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,247] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,247] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:30,248] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:30,248] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,257] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,258] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,258] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:30,259] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:30,261] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,262] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:30,265] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,266] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,279] jepsen worker 34 - jepsen.util 754 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,279] jepsen worker 34 - jepsen.util 754 :invoke :write [11 11] INFO [2022-03-13 17:13:30,291] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,292] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,299] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,299] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:13:30,315] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,315] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:13:30,335] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,335] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:30,336] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,336] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,349] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,349] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,350] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:13:30,350] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:13:30,363] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,364] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,384] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,385] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:13:30,391] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,392] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,396] jepsen worker 62 - jepsen.util 512 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,396] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,397] jepsen worker 62 - jepsen.util 512 :invoke :write [20 9] INFO [2022-03-13 17:13:30,397] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,400] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,401] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,402] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,402] jepsen worker 78 - jepsen.util 528 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,403] jepsen worker 78 - jepsen.util 528 :invoke :write [26 16] INFO [2022-03-13 17:13:30,403] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:30,404] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,405] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,449] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,451] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,453] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,453] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:30,460] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,460] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,462] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,462] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,475] jepsen worker 60 - jepsen.util 600 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,475] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:30,480] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,480] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:30,480] jepsen worker nemesis - couchbase.util Rebalance check count: 320 INFO [2022-03-13 17:13:30,480] jepsen worker nemesis - couchbase.util Rebalance stuck count: 220 INFO [2022-03-13 17:13:30,481] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:13:30,495] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,495] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,495] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,496] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:13:30,496] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,497] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:13:30,498] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,498] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,506] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,508] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,509] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,511] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,515] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,516] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:13:30,519] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,519] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,520] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,520] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,520] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,520] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:13:30,521] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,522] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:30,522] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,523] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,523] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,523] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,524] jepsen worker 71 - jepsen.util 521 :invoke :write [23 12] INFO [2022-03-13 17:13:30,525] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,527] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:30,527] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,528] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:30,529] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,531] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,532] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,532] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,532] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,532] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,533] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,533] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,534] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:13:30,536] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,538] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,538] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,538] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:30,538] jepsen worker 33 - jepsen.util 663 :invoke :write [11 10] INFO [2022-03-13 17:13:30,540] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,540] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:13:30,540] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,556] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,557] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:13:30,557] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:13:30,559] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:30,561] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,561] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,562] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,563] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:30,565] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,566] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,567] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,568] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,570] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,570] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,571] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,572] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:30,573] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,575] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:13:30,578] jepsen worker 30 - jepsen.util 2460 :ok :read [10 :nil] INFO [2022-03-13 17:13:30,579] jepsen worker 30 - jepsen.util 2460 :invoke :read [10 nil] INFO [2022-03-13 17:13:30,607] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,608] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,609] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,609] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,609] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,610] jepsen worker 70 - jepsen.util 700 :invoke :write [23 46] INFO [2022-03-13 17:13:30,610] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,610] jepsen worker 13 - jepsen.util 643 :invoke :write [4 26] INFO [2022-03-13 17:13:30,611] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:30,611] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,612] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,613] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,614] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,614] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:30,618] jepsen worker 69 - jepsen.util 699 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,618] jepsen worker 7 - jepsen.util 637 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,618] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:30,618] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,619] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:30,619] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:30,620] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:30,620] jepsen worker 14 - jepsen.util 284 :invoke :read [4 nil] INFO [2022-03-13 17:13:30,654] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,654] jepsen worker 71 - jepsen.util 521 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,655] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,655] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:13:30,673] jepsen worker 32 - jepsen.util 2282 :info :write [10 19] :SyncWriteAmbiguous INFO [2022-03-13 17:13:30,675] jepsen worker 32 - jepsen.util 2372 :invoke :write [10 4] INFO [2022-03-13 17:13:30,706] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,707] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:30,714] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,714] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:13:30,715] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,716] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:30,734] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,735] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,753] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,754] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 39] INFO [2022-03-13 17:13:30,796] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,797] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:13:30,809] jepsen worker 6 - jepsen.util 636 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,810] jepsen worker 6 - jepsen.util 636 :invoke :write [2 5] INFO [2022-03-13 17:13:30,810] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,810] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,811] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,811] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,812] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:13:30,812] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,812] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,813] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:13:30,815] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,816] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,818] jepsen worker 80 - jepsen.util 710 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,818] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,818] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:30,819] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:30,821] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,821] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,822] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:13:30,824] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:13:30,858] jepsen worker 89 - jepsen.util 2429 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,859] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:13:30,859] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,860] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,884] jepsen worker 12 - jepsen.util 732 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,885] jepsen worker 12 - jepsen.util 732 :invoke :write [4 15] INFO [2022-03-13 17:13:30,891] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,891] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:30,941] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,942] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:30,947] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,948] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:30,948] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,948] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:13:30,949] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,949] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,949] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:30,950] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:30,964] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,965] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,971] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,973] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:30,974] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,975] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:30,987] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:30,988] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:13:31,000] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,000] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,001] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,001] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,024] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,025] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,032] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,032] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,034] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,034] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,035] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,036] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,037] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,037] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:13:31,041] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,041] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,041] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,041] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:13:31,042] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:13:31,042] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,046] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,047] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:13:31,054] jepsen worker 8 - jepsen.util 728 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,055] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:31,070] jepsen worker 33 - jepsen.util 663 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,071] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:31,096] jepsen worker 34 - jepsen.util 754 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,096] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,097] jepsen worker 34 - jepsen.util 754 :invoke :write [11 35] INFO [2022-03-13 17:13:31,098] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:31,101] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,102] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:13:31,108] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,108] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,108] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,109] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:13:31,110] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:31,111] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,111] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:31,112] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,113] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:31,113] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:31,114] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,114] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,115] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:13:31,115] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:31,119] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,120] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:13:31,123] jepsen worker 51 - jepsen.util 591 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,123] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,123] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,123] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,123] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,125] jepsen worker 76 - jepsen.util 706 :invoke :write [25 16] INFO [2022-03-13 17:13:31,126] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:31,127] jepsen worker 60 - jepsen.util 600 :invoke :write [20 37] INFO [2022-03-13 17:13:31,127] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,127] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:31,129] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:31,129] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:13:31,132] jepsen worker 44 - jepsen.util 584 :invoke :write [14 37] INFO [2022-03-13 17:13:31,134] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:31,142] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,142] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:13:31,144] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,144] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:13:31,155] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,157] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:31,171] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,172] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:31,208] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,208] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,228] jepsen worker 59 - jepsen.util 2399 :info :write [19 39] :SyncWriteAmbiguous INFO [2022-03-13 17:13:31,228] jepsen worker 78 - jepsen.util 528 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,229] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,229] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:31,230] jepsen worker 59 - jepsen.util 2489 :invoke :read [19 nil] INFO [2022-03-13 17:13:31,230] jepsen worker 52 - jepsen.util 592 :invoke :write [17 5] INFO [2022-03-13 17:13:31,253] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,253] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,253] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,254] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,254] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:31,255] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:13:31,256] jepsen worker 71 - jepsen.util 521 :invoke :write [23 12] INFO [2022-03-13 17:13:31,260] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,261] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:13:31,261] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:13:31,262] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,264] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:13:31,265] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,266] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:13:31,275] jepsen worker 59 - jepsen.util 2489 :ok :read [19 38] INFO [2022-03-13 17:13:31,275] jepsen worker 59 - jepsen.util 2489 :invoke :write [19 1] INFO [2022-03-13 17:13:31,283] jepsen worker 53 - jepsen.util 503 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,284] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,285] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:31,285] jepsen worker 53 - jepsen.util 503 :invoke :write [17 46] INFO [2022-03-13 17:13:31,292] jepsen worker 66 - jepsen.util 2226 :info :write [22 31] :SyncWriteAmbiguous INFO [2022-03-13 17:13:31,294] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,294] jepsen worker 66 - jepsen.util 2316 :invoke :read [22 nil] INFO [2022-03-13 17:13:31,294] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:13:31,301] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,302] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:13:31,333] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,334] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:31,336] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,336] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,337] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,338] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,342] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,343] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:13:31,367] jepsen worker 12 - jepsen.util 732 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,367] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,367] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,367] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,368] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:31,368] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:31,369] jepsen worker 23 - jepsen.util 563 :invoke :write [7 35] INFO [2022-03-13 17:13:31,370] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:31,380] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,381] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:31,396] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,396] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:13:31,404] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,405] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:13:31,408] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,408] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,408] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,409] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:13:31,410] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,410] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,414] jepsen worker 70 - jepsen.util 700 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,415] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,415] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,415] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:31,415] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,416] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,416] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,417] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,417] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,417] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:13:31,418] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,419] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,420] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,420] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,421] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,422] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:13:31,436] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,437] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:31,449] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,450] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:31,487] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:31,487] jepsen worker nemesis - couchbase.util Rebalance check count: 321 INFO [2022-03-13 17:13:31,487] jepsen worker nemesis - couchbase.util Rebalance stuck count: 221 INFO [2022-03-13 17:13:31,497] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,498] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,498] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:13:31,499] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:31,507] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,510] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,512] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,513] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,514] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,514] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,515] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,516] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 37] INFO [2022-03-13 17:13:31,517] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,518] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:13:31,533] jepsen worker 57 - jepsen.util 2307 :info :write [19 32] :RequestTimeoutException INFO [2022-03-13 17:13:31,534] jepsen worker 57 - jepsen.util 2397 :invoke :write [19 13] INFO [2022-03-13 17:13:31,541] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,542] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:31,555] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,555] jepsen worker 20 - jepsen.util 2720 :info :write [6 41] :RequestTimeoutException INFO [2022-03-13 17:13:31,556] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:31,557] jepsen worker 20 - jepsen.util 2810 :invoke :write [6 36] INFO [2022-03-13 17:13:31,572] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,573] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:31,596] jepsen worker 46 - jepsen.util 2386 :info :write [15 1] :RequestTimeoutException INFO [2022-03-13 17:13:31,597] jepsen worker 46 - jepsen.util 2476 :invoke :read [15 nil] INFO [2022-03-13 17:13:31,600] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,601] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:13:31,610] jepsen worker 13 - jepsen.util 643 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,610] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,610] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:31,611] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:31,616] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,616] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,616] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:13:31,617] jepsen worker 35 - jepsen.util 935 :invoke :write [11 15] INFO [2022-03-13 17:13:31,625] jepsen worker 89 - jepsen.util 2429 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,626] jepsen worker 43 - jepsen.util 493 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,626] jepsen worker 76 - jepsen.util 706 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,627] jepsen worker 43 - jepsen.util 493 :invoke :write [14 35] INFO [2022-03-13 17:13:31,631] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,632] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,633] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,634] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,635] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:31,635] jepsen worker 76 - jepsen.util 706 :invoke :write [25 17] INFO [2022-03-13 17:13:31,644] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,645] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:31,647] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,648] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:31,655] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,656] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:31,658] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,659] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:31,690] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,690] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,691] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:13:31,691] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:13:31,692] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,693] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:13:31,704] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,704] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,710] jepsen worker 84 - jepsen.util 2604 :info :write [28 46] :RequestTimeoutException INFO [2022-03-13 17:13:31,711] jepsen worker 84 - jepsen.util 2694 :invoke :write [28 26] INFO [2022-03-13 17:13:31,732] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,733] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:31,780] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,781] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:31,782] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,782] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:13:31,786] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,787] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,837] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,838] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,838] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:13:31,838] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,839] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:31,839] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:13:31,856] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,857] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:13:31,867] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,867] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,868] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,869] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,870] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,870] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,879] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,879] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,881] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,881] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,886] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,887] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:31,888] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,888] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:13:31,930] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,931] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:31,942] jepsen worker 14 - jepsen.util 284 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,942] jepsen worker 14 - jepsen.util 284 :invoke :write [4 16] INFO [2022-03-13 17:13:31,948] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,951] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:31,956] jepsen worker 7 - jepsen.util 637 :ok :read [2 5] INFO [2022-03-13 17:13:31,957] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:31,981] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,981] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,981] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,982] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:31,983] jepsen worker 70 - jepsen.util 700 :invoke :write [23 46] INFO [2022-03-13 17:13:31,983] jepsen worker 13 - jepsen.util 643 :invoke :write [4 47] INFO [2022-03-13 17:13:31,984] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:31,985] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:32,010] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,011] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:13:32,024] jepsen worker 61 - jepsen.util 601 :ok :read [20 12] INFO [2022-03-13 17:13:32,024] jepsen worker 61 - jepsen.util 601 :invoke :read [20 nil] INFO [2022-03-13 17:13:32,031] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,032] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,032] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,033] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,033] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,034] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,045] jepsen worker 39 - jepsen.util 2379 :ok :read [13 39] INFO [2022-03-13 17:13:32,045] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,046] jepsen worker 39 - jepsen.util 2379 :invoke :write [13 2] INFO [2022-03-13 17:13:32,046] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,048] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,048] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:32,051] jepsen worker 49 - jepsen.util 2119 :ok :read [16 :nil] INFO [2022-03-13 17:13:32,053] jepsen worker 49 - jepsen.util 2119 :invoke :read [16 nil] INFO [2022-03-13 17:13:32,067] jepsen worker 82 - jepsen.util 2422 :ok :read [27 4] INFO [2022-03-13 17:13:32,067] jepsen worker 82 - jepsen.util 2422 :invoke :write [27 17] INFO [2022-03-13 17:13:32,077] jepsen worker 49 - jepsen.util 2119 :ok :read [16 :nil] INFO [2022-03-13 17:13:32,078] jepsen worker 49 - jepsen.util 2119 :invoke :write [16 47] INFO [2022-03-13 17:13:32,079] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,079] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,079] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,080] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,080] jepsen worker 83 - jepsen.util 1883 :ok :read [27 4] INFO [2022-03-13 17:13:32,080] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 20] INFO [2022-03-13 17:13:32,081] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,081] jepsen worker 83 - jepsen.util 1883 :invoke :read [27 nil] INFO [2022-03-13 17:13:32,091] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,091] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,091] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,092] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:13:32,092] jepsen worker 66 - jepsen.util 2316 :ok :read [22 30] INFO [2022-03-13 17:13:32,092] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,092] jepsen worker 66 - jepsen.util 2316 :invoke :write [22 16] INFO [2022-03-13 17:13:32,093] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:13:32,098] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,099] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,100] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,100] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,101] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,102] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,103] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,103] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,104] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,105] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,106] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,107] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:13:32,173] jepsen worker 87 - jepsen.util 2607 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,174] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:13:32,249] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,249] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,249] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,252] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,252] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:13:32,253] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:32,254] jepsen worker 83 - jepsen.util 1883 :ok :read [27 4] INFO [2022-03-13 17:13:32,255] jepsen worker 83 - jepsen.util 1883 :invoke :read [27 nil] INFO [2022-03-13 17:13:32,283] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,284] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:13:32,285] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,285] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,306] jepsen worker 83 - jepsen.util 1883 :ok :read [27 4] INFO [2022-03-13 17:13:32,306] jepsen worker 83 - jepsen.util 1883 :invoke :read [27 nil] INFO [2022-03-13 17:13:32,329] jepsen worker 9 - jepsen.util 2169 :fail :read [3 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:13:32,329] jepsen worker 9 - jepsen.util 2169 :invoke :read [3 nil] INFO [2022-03-13 17:13:32,337] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,338] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:13:32,341] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,342] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,342] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,343] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,356] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,356] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,360] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,360] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:32,391] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,391] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:13:32,399] jepsen worker 35 - jepsen.util 935 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,400] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:32,404] jepsen worker 83 - jepsen.util 1883 :ok :read [27 4] INFO [2022-03-13 17:13:32,404] jepsen worker 83 - jepsen.util 1883 :invoke :write [27 5] INFO [2022-03-13 17:13:32,411] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,411] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,411] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,411] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:13:32,412] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,412] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,413] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,413] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,414] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,414] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,415] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,416] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,416] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,416] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,423] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,423] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,425] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:13:32,426] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:13:32,452] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,453] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:32,456] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,456] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:32,492] jepsen worker 6 - jepsen.util 636 :fail :write [2 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,493] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:13:32,498] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:32,498] jepsen worker nemesis - couchbase.util Rebalance check count: 322 INFO [2022-03-13 17:13:32,498] jepsen worker nemesis - couchbase.util Rebalance stuck count: 222 INFO [2022-03-13 17:13:32,500] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,500] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:32,505] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,505] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:13:32,507] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,507] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:32,541] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,542] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:32,554] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,555] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,587] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,587] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,588] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,588] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:32,590] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,591] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,594] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,595] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:32,604] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,605] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,606] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,606] jepsen worker 33 - jepsen.util 663 :invoke :write [11 13] INFO [2022-03-13 17:13:32,607] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,607] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:32,615] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,615] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,615] jepsen worker 34 - jepsen.util 754 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,616] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:32,617] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:32,618] jepsen worker 78 - jepsen.util 528 :invoke :write [26 37] INFO [2022-03-13 17:13:32,637] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,637] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,637] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,638] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,638] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,638] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:32,638] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,639] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,640] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:13:32,640] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:32,641] jepsen worker 26 - jepsen.util 476 :invoke :write [8 8] INFO [2022-03-13 17:13:32,644] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 43] INFO [2022-03-13 17:13:32,645] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:13:32,645] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,646] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,647] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:32,648] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,649] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:13:32,653] jepsen worker 8 - jepsen.util 728 :ok :read [2 5] INFO [2022-03-13 17:13:32,655] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:32,695] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,696] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,699] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,700] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:32,719] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,720] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:13:32,732] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,733] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:13:32,736] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,737] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:13:32,764] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,764] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,764] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,765] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:32,765] jepsen worker 53 - jepsen.util 503 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,765] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:13:32,766] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:32,766] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:32,767] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,767] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,767] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:13:32,768] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:13:32,797] jepsen worker 52 - jepsen.util 592 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,798] jepsen worker 52 - jepsen.util 592 :invoke :write [17 5] INFO [2022-03-13 17:13:32,800] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,800] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,820] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,820] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,820] jepsen worker 44 - jepsen.util 584 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,820] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,821] jepsen worker 51 - jepsen.util 591 :invoke :write [17 21] INFO [2022-03-13 17:13:32,822] jepsen worker 69 - jepsen.util 699 :invoke :write [23 9] INFO [2022-03-13 17:13:32,822] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:13:32,823] jepsen worker 44 - jepsen.util 584 :invoke :write [14 22] INFO [2022-03-13 17:13:32,829] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,829] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:32,830] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,832] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:32,854] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,854] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,854] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:32,855] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,856] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,856] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:13:32,857] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,858] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,867] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,867] jepsen worker 71 - jepsen.util 521 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,867] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:13:32,868] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:32,869] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,870] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:13:32,883] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,884] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,898] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,899] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:13:32,948] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,949] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:32,949] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,950] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,952] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,952] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,953] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,954] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,957] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,958] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,959] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,960] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:32,978] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,978] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,978] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,980] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,980] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,981] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,981] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:32,981] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,981] jepsen worker 43 - jepsen.util 493 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,982] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,982] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,983] jepsen worker 43 - jepsen.util 493 :invoke :write [14 11] INFO [2022-03-13 17:13:32,983] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:13:32,984] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:13:32,988] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,988] jepsen worker 23 - jepsen.util 563 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,990] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,990] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:32,992] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,992] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:32,993] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:32,994] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:13:33,008] jepsen worker 33 - jepsen.util 663 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,009] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:33,031] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,032] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:33,054] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,054] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,057] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,058] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:13:33,068] jepsen worker 13 - jepsen.util 643 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,069] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:33,069] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,103] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:13:33,080] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,103] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,103] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,105] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:33,105] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:13:33,105] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:13:33,111] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,112] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:33,123] jepsen worker 62 - jepsen.util 512 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,123] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,123] jepsen worker 7 - jepsen.util 637 :invoke :write [2 31] INFO [2022-03-13 17:13:33,124] jepsen worker 62 - jepsen.util 512 :invoke :write [20 46] INFO [2022-03-13 17:13:33,139] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,140] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:13:33,141] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,141] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:33,146] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,147] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:13:33,169] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,169] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,171] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:33,171] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,179] jepsen worker 0 - jepsen.util 2340 :ok :read [0 41] INFO [2022-03-13 17:13:33,180] jepsen worker 0 - jepsen.util 2340 :invoke :write [0 16] INFO [2022-03-13 17:13:33,203] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,203] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,203] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,203] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,203] jepsen worker 76 - jepsen.util 706 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,205] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 39] INFO [2022-03-13 17:13:33,206] jepsen worker 75 - jepsen.util 435 :invoke :write [25 37] INFO [2022-03-13 17:13:33,207] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:33,208] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:13:33,209] jepsen worker 76 - jepsen.util 706 :invoke :write [25 22] INFO [2022-03-13 17:13:33,225] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,226] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,229] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,230] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:33,237] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,238] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,237] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,238] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:33,239] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:13:33,239] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:13:33,244] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,245] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:33,247] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,247] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:13:33,284] jepsen worker 58 - jepsen.util 2218 :ok :read [19 38] INFO [2022-03-13 17:13:33,285] jepsen worker 58 - jepsen.util 2218 :invoke :read [19 nil] INFO [2022-03-13 17:13:33,300] jepsen worker 16 - jepsen.util 2176 :ok :read [5 43] INFO [2022-03-13 17:13:33,300] jepsen worker 16 - jepsen.util 2176 :invoke :read [5 nil] INFO [2022-03-13 17:13:33,318] jepsen worker 89 - jepsen.util 2429 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,318] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,321] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,322] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,323] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,323] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,325] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,326] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,327] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,327] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:33,344] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,345] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:13:33,354] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,355] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:13:33,379] jepsen worker 58 - jepsen.util 2218 :ok :read [19 38] INFO [2022-03-13 17:13:33,382] jepsen worker 58 - jepsen.util 2218 :invoke :write [19 47] INFO [2022-03-13 17:13:33,383] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,383] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,383] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,384] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:13:33,392] jepsen worker 70 - jepsen.util 700 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,392] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,392] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,393] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:33,393] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:33,394] jepsen worker 34 - jepsen.util 754 :invoke :write [11 47] INFO [2022-03-13 17:13:33,412] jepsen worker 16 - jepsen.util 2176 :ok :read [5 43] INFO [2022-03-13 17:13:33,413] jepsen worker 16 - jepsen.util 2176 :invoke :write [5 39] INFO [2022-03-13 17:13:33,428] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,428] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,433] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,434] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:33,438] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,439] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,441] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,441] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,441] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,441] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:13:33,442] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:13:33,442] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:33,453] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,454] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:13:33,470] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,470] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,471] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:13:33,473] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:13:33,485] jepsen worker 14 - jepsen.util 284 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,485] jepsen worker 61 - jepsen.util 601 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,485] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,486] jepsen worker 14 - jepsen.util 284 :invoke :write [4 4] INFO [2022-03-13 17:13:33,490] jepsen worker 61 - jepsen.util 601 :invoke :write [20 11] INFO [2022-03-13 17:13:33,501] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,502] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,503] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:33,504] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,505] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:33,505] jepsen worker nemesis - couchbase.util Rebalance check count: 323 INFO [2022-03-13 17:13:33,505] jepsen worker nemesis - couchbase.util Rebalance stuck count: 223 INFO [2022-03-13 17:13:33,506] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,508] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,508] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,510] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,511] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:13:33,529] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,530] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:33,579] jepsen worker 78 - jepsen.util 528 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,579] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:33,596] jepsen worker 75 - jepsen.util 435 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,596] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:13:33,599] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,599] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:33,603] jepsen worker 60 - jepsen.util 600 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,603] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,603] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:33,604] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,604] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,604] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:33,605] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,605] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,605] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:33,606] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:13:33,606] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,606] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,606] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:13:33,607] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 48] INFO [2022-03-13 17:13:33,615] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,616] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,625] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,626] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:33,687] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,688] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,689] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,689] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,690] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,691] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:13:33,691] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,692] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:13:33,744] jepsen worker 3 - jepsen.util 1893 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,745] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:13:33,749] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,749] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,752] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,753] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,754] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,754] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,755] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,756] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,757] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,757] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,759] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,759] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,760] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,760] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,761] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,762] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:33,770] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,771] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,777] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,778] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:33,779] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,780] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:13:33,792] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,793] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:13:33,811] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,811] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,812] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,812] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,812] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,813] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:33,814] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:33,814] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:33,824] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,824] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,824] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,824] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:13:33,825] jepsen worker 21 - jepsen.util 831 :invoke :write [7 1] INFO [2022-03-13 17:13:33,825] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:33,826] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,827] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:13:33,839] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,839] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,839] jepsen worker 52 - jepsen.util 592 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,839] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,839] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,839] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:33,839] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,839] jepsen worker 80 - jepsen.util 710 :invoke :write [26 28] INFO [2022-03-13 17:13:33,839] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,840] jepsen worker 52 - jepsen.util 592 :invoke :write [17 48] INFO [2022-03-13 17:13:33,841] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,842] jepsen worker 42 - jepsen.util 402 :invoke :write [14 22] INFO [2022-03-13 17:13:33,843] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,844] jepsen worker 6 - jepsen.util 636 :invoke :write [2 13] INFO [2022-03-13 17:13:33,844] jepsen worker 13 - jepsen.util 643 :invoke :write [4 31] INFO [2022-03-13 17:13:33,845] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,846] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,846] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:33,846] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,848] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,849] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,850] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:33,852] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,853] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,867] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,868] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:33,872] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,873] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,874] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,876] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:33,886] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,887] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:33,889] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,889] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:33,894] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,895] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:13:33,909] jepsen worker 81 - jepsen.util 2781 :info :write [27 48] :SyncWriteAmbiguous INFO [2022-03-13 17:13:33,911] jepsen worker 81 - jepsen.util 2871 :invoke :write [27 49] INFO [2022-03-13 17:13:33,915] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,916] jepsen worker 71 - jepsen.util 521 :invoke :write [23 15] INFO [2022-03-13 17:13:33,918] jepsen worker 50 - jepsen.util 2570 :info :write [16 1] :RequestTimeoutException INFO [2022-03-13 17:13:33,918] jepsen worker 50 - jepsen.util 2660 :invoke :read [16 nil] INFO [2022-03-13 17:13:33,927] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,927] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:33,936] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,936] jepsen worker 34 - jepsen.util 754 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,937] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,937] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,937] jepsen worker 22 - jepsen.util 652 :invoke :write [7 27] INFO [2022-03-13 17:13:33,938] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:13:33,939] jepsen worker 78 - jepsen.util 528 :invoke :write [26 9] INFO [2022-03-13 17:13:33,940] jepsen worker 34 - jepsen.util 754 :invoke :write [11 19] INFO [2022-03-13 17:13:33,946] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,946] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,947] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:13:33,947] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:33,953] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,953] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,954] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:13:33,954] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:13:33,967] jepsen worker 40 - jepsen.util 2560 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,969] jepsen worker 40 - jepsen.util 2560 :invoke :write [13 12] INFO [2022-03-13 17:13:33,991] jepsen worker 1 - jepsen.util 2251 :ok :read [0 41] INFO [2022-03-13 17:13:33,992] jepsen worker 1 - jepsen.util 2251 :invoke :read [0 nil] INFO [2022-03-13 17:13:33,994] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:33,995] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:13:34,030] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,030] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:13:34,058] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,059] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:34,060] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,061] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:34,062] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,063] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:13:34,068] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,068] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:13:34,103] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,103] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:34,115] jepsen worker 1 - jepsen.util 2251 :ok :read [0 41] INFO [2022-03-13 17:13:34,116] jepsen worker 1 - jepsen.util 2251 :invoke :read [0 nil] INFO [2022-03-13 17:13:34,133] jepsen worker 52 - jepsen.util 592 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,134] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:34,145] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,146] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:34,147] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,148] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:34,149] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,149] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:34,150] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,151] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:13:34,153] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,153] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:34,154] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,154] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:13:34,155] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,156] jepsen worker 79 - jepsen.util 1069 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,156] jepsen worker 69 - jepsen.util 699 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,156] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:34,157] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:34,157] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:34,190] jepsen worker 86 - jepsen.util 2246 :info :write [28 18] :RequestTimeoutException INFO [2022-03-13 17:13:34,191] jepsen worker 86 - jepsen.util 2336 :invoke :read [28 nil] INFO [2022-03-13 17:13:34,198] jepsen worker 26 - jepsen.util 476 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,199] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:34,216] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,216] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:13:34,217] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,217] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,217] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:13:34,218] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:34,219] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,219] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 38] INFO [2022-03-13 17:13:34,220] jepsen worker 1 - jepsen.util 2251 :ok :read [0 41] INFO [2022-03-13 17:13:34,220] jepsen worker 1 - jepsen.util 2251 :invoke :read [0 nil] INFO [2022-03-13 17:13:34,237] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,238] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:34,242] jepsen worker 44 - jepsen.util 584 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,242] jepsen worker 44 - jepsen.util 584 :invoke :write [14 45] INFO [2022-03-13 17:13:34,248] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,248] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,249] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:13:34,250] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:34,260] jepsen worker 13 - jepsen.util 643 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,260] jepsen worker 13 - jepsen.util 643 :invoke :write [4 46] INFO [2022-03-13 17:13:34,261] jepsen worker 21 - jepsen.util 831 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,262] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,262] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:34,263] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:13:34,284] jepsen worker 88 - jepsen.util 1978 :info :write [29 29] :SyncWriteAmbiguous INFO [2022-03-13 17:13:34,285] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:13:34,323] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,323] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:13:34,331] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,332] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,332] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,332] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:34,333] jepsen worker 21 - jepsen.util 831 :invoke :write [7 48] INFO [2022-03-13 17:13:34,334] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:13:34,337] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,337] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:34,339] jepsen worker 43 - jepsen.util 493 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,346] jepsen worker 36 - jepsen.util 2106 :info :write [12 38] :SyncWriteAmbiguous INFO [2022-03-13 17:13:34,346] jepsen worker 86 - jepsen.util 2336 :ok :read [28 44] INFO [2022-03-13 17:13:34,347] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:13:34,347] jepsen worker 86 - jepsen.util 2336 :invoke :read [28 nil] INFO [2022-03-13 17:13:34,352] jepsen worker 22 - jepsen.util 652 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,352] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,339] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,346] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:34,353] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:34,353] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:13:34,354] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:34,355] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,355] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:34,358] jepsen worker 71 - jepsen.util 521 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,360] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:34,367] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,367] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:34,422] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,423] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:34,431] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,432] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:13:34,434] jepsen worker 1 - jepsen.util 2251 :ok :read [0 41] INFO [2022-03-13 17:13:34,436] jepsen worker 1 - jepsen.util 2251 :invoke :read [0 nil] INFO [2022-03-13 17:13:34,441] jepsen worker 86 - jepsen.util 2336 :ok :read [28 44] INFO [2022-03-13 17:13:34,442] jepsen worker 86 - jepsen.util 2336 :invoke :read [28 nil] INFO [2022-03-13 17:13:34,461] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,461] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,461] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:34,462] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:13:34,468] jepsen worker 80 - jepsen.util 710 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,469] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:34,472] jepsen worker 51 - jepsen.util 591 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,473] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:34,490] jepsen worker 3 - jepsen.util 1893 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,490] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:34,510] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:34,510] jepsen worker nemesis - couchbase.util Rebalance check count: 324 INFO [2022-03-13 17:13:34,510] jepsen worker nemesis - couchbase.util Rebalance stuck count: 224 INFO [2022-03-13 17:13:34,513] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,514] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:13:34,535] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,535] jepsen worker 21 - jepsen.util 831 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,535] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:13:34,536] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:34,570] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,570] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,570] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,570] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:34,571] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:13:34,571] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:13:34,574] jepsen worker 1 - jepsen.util 2251 :ok :read [0 41] INFO [2022-03-13 17:13:34,574] jepsen worker 1 - jepsen.util 2251 :invoke :write [0 18] INFO [2022-03-13 17:13:34,580] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,581] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:13:34,584] jepsen worker 78 - jepsen.util 528 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,585] jepsen worker 78 - jepsen.util 528 :invoke :write [26 31] INFO [2022-03-13 17:13:34,587] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,589] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:13:34,592] jepsen worker 48 - jepsen.util 2388 :ok :read [16 :nil] INFO [2022-03-13 17:13:34,593] jepsen worker 48 - jepsen.util 2388 :invoke :write [16 16] INFO [2022-03-13 17:13:34,620] jepsen worker 86 - jepsen.util 2336 :ok :read [28 44] INFO [2022-03-13 17:13:34,621] jepsen worker 86 - jepsen.util 2336 :invoke :write [28 14] INFO [2022-03-13 17:13:34,625] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,626] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:34,643] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,644] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:34,655] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,657] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:34,672] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,673] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:13:34,674] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,674] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,675] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:13:34,676] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:34,679] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,680] jepsen worker 85 - jepsen.util 2335 :info :write [28 41] :RequestTimeoutException INFO [2022-03-13 17:13:34,680] jepsen worker 22 - jepsen.util 652 :invoke :write [7 45] INFO [2022-03-13 17:13:34,681] jepsen worker 85 - jepsen.util 2425 :invoke :read [28 nil] INFO [2022-03-13 17:13:34,700] jepsen worker 68 - jepsen.util 2498 :info :write [22 21] :RequestTimeoutException INFO [2022-03-13 17:13:34,701] jepsen worker 68 - jepsen.util 2588 :invoke :read [22 nil] INFO [2022-03-13 17:13:34,710] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,711] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:34,711] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,711] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,712] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:13:34,712] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:13:34,724] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,724] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,726] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:34,727] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:34,737] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,737] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:13:34,764] jepsen worker 41 - jepsen.util 2291 :info :write [13 38] :RequestTimeoutException INFO [2022-03-13 17:13:34,765] jepsen worker 41 - jepsen.util 2381 :invoke :read [13 nil] INFO [2022-03-13 17:13:34,780] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,780] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,781] jepsen worker 70 - jepsen.util 700 :invoke :write [23 19] INFO [2022-03-13 17:13:34,781] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:34,785] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,786] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:34,787] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,788] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 30] INFO [2022-03-13 17:13:34,790] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,791] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:13:34,792] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,792] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,793] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,793] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:13:34,796] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,798] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:13:34,803] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:13:34,806] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,806] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 40] INFO [2022-03-13 17:13:34,825] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:34,826] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,828] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:34,833] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,833] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,834] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:13:34,835] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:34,854] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,854] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:34,857] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,857] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:34,867] jepsen worker 34 - jepsen.util 754 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,868] jepsen worker 34 - jepsen.util 754 :invoke :write [11 14] INFO [2022-03-13 17:13:34,873] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,873] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:34,899] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,899] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:34,914] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,914] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,914] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,914] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:13:34,915] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:13:34,916] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:13:34,927] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,927] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:34,932] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,933] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,933] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,933] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:34,933] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:34,935] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,935] jepsen worker 21 - jepsen.util 831 :invoke :write [7 21] INFO [2022-03-13 17:13:34,936] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:13:34,938] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,939] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:13:34,956] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,957] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:34,980] jepsen worker 37 - jepsen.util 2017 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:34,981] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,016] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,016] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:13:35,039] jepsen worker 22 - jepsen.util 652 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,040] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:35,042] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,042] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,043] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,043] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:35,044] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,045] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:13:35,048] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,049] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,051] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,052] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:13:35,053] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,054] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:13:35,058] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,059] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 20] INFO [2022-03-13 17:13:35,110] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,110] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,112] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:13:35,112] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,141] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,141] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,142] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:13:35,142] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:13:35,145] jepsen worker 88 - jepsen.util 2068 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,145] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,146] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,146] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:13:35,147] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:35,148] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:13:35,153] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,153] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,155] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:35,155] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,160] jepsen worker 15 - jepsen.util 2445 :info :write [5 4] :RequestTimeoutException INFO [2022-03-13 17:13:35,161] jepsen worker 15 - jepsen.util 2535 :invoke :write [5 32] INFO [2022-03-13 17:13:35,187] jepsen worker 67 - jepsen.util 2047 :info :write [22 18] :RequestTimeoutException INFO [2022-03-13 17:13:35,187] jepsen worker 67 - jepsen.util 2137 :invoke :write [22 38] INFO [2022-03-13 17:13:35,192] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,193] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:35,228] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,228] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:13:35,241] jepsen worker 13 - jepsen.util 643 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,241] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,242] jepsen worker 21 - jepsen.util 831 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,242] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:35,242] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:35,243] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:35,252] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,252] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:13:35,271] jepsen worker 59 - jepsen.util 2489 :ok :write [19 1] INFO [2022-03-13 17:13:35,272] jepsen worker 59 - jepsen.util 2489 :invoke :read [19 nil] INFO [2022-03-13 17:13:35,281] jepsen worker 87 - jepsen.util 2607 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,281] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,282] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:35,283] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:13:35,294] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,294] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,295] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:13:35,295] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,298] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,299] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:13:35,349] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,350] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:35,354] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,354] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,356] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:13:35,357] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,358] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,358] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,362] jepsen worker 78 - jepsen.util 528 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,362] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,363] jepsen worker 78 - jepsen.util 528 :invoke :write [26 17] INFO [2022-03-13 17:13:35,366] jepsen worker 52 - jepsen.util 592 :invoke :write [17 41] INFO [2022-03-13 17:13:35,368] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,369] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:13:35,376] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,377] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:35,397] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,398] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:35,417] jepsen worker 10 - jepsen.util 2530 :info :write [3 10] :SyncWriteAmbiguous INFO [2022-03-13 17:13:35,418] jepsen worker 10 - jepsen.util 2620 :invoke :read [3 nil] INFO [2022-03-13 17:13:35,424] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,425] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:13:35,428] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,428] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,430] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,430] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,432] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,432] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,433] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:35,433] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:13:35,436] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,437] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:13:35,459] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,460] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:13:35,476] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,477] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:35,484] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,485] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:13:35,499] jepsen worker 72 - jepsen.util 2502 :ok :read [24 24] INFO [2022-03-13 17:13:35,500] jepsen worker 72 - jepsen.util 2502 :invoke :read [24 nil] INFO [2022-03-13 17:13:35,512] jepsen worker 46 - jepsen.util 2476 :ok :read [15 48] INFO [2022-03-13 17:13:35,515] jepsen worker 46 - jepsen.util 2476 :invoke :read [15 nil] INFO [2022-03-13 17:13:35,519] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:35,519] jepsen worker nemesis - couchbase.util Rebalance check count: 325 INFO [2022-03-13 17:13:35,519] jepsen worker nemesis - couchbase.util Rebalance stuck count: 225 INFO [2022-03-13 17:13:35,521] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,522] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,522] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,522] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:13:35,523] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:13:35,524] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:35,528] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,528] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,528] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,530] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:13:35,530] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:35,531] jepsen worker 21 - jepsen.util 831 :invoke :write [7 25] INFO [2022-03-13 17:13:35,538] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,539] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:13:35,548] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,549] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 44] INFO [2022-03-13 17:13:35,564] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,565] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,567] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,568] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,570] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,571] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:13:35,584] jepsen worker 72 - jepsen.util 2502 :ok :read [24 24] INFO [2022-03-13 17:13:35,584] jepsen worker 72 - jepsen.util 2502 :invoke :write [24 24] INFO [2022-03-13 17:13:35,594] jepsen worker 46 - jepsen.util 2476 :ok :read [15 48] INFO [2022-03-13 17:13:35,595] jepsen worker 46 - jepsen.util 2476 :invoke :write [15 25] INFO [2022-03-13 17:13:35,602] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,603] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:35,603] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,603] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,604] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:35,605] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:13:35,631] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,632] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:13:35,658] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,659] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:13:35,663] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,663] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:35,666] jepsen worker 70 - jepsen.util 700 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,667] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:35,685] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,686] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:13:35,693] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,693] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:35,734] jepsen worker 21 - jepsen.util 831 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,735] jepsen worker 21 - jepsen.util 831 :invoke :write [7 15] INFO [2022-03-13 17:13:35,739] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,740] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:13:35,761] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,761] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,762] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:35,762] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:35,780] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,780] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,780] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,781] jepsen worker 22 - jepsen.util 652 :invoke :write [7 0] INFO [2022-03-13 17:13:35,782] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:13:35,783] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:13:35,793] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,794] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:35,798] jepsen worker 37 - jepsen.util 2017 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,799] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,800] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,800] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:35,822] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,824] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:35,842] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,843] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:35,860] jepsen worker 2 - jepsen.util 2792 :info :write [0 40] :RequestTimeoutException INFO [2022-03-13 17:13:35,860] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,860] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,860] jepsen worker 34 - jepsen.util 754 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,861] jepsen worker 2 - jepsen.util 2882 :invoke :write [0 22] INFO [2022-03-13 17:13:35,861] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:35,862] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:35,863] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:35,884] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,884] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,884] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:35,884] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:35,891] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,892] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:13:35,898] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,899] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:13:35,901] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,902] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:35,903] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,903] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,903] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,904] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:13:35,905] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:13:35,910] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,911] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,912] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:13:35,976] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,976] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,976] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,976] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:35,977] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:13:35,977] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:35,984] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,985] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:35,986] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,987] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:13:35,992] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,993] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:13:35,998] jepsen worker 21 - jepsen.util 831 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,999] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,999] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:35,999] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:13:36,000] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:36,000] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:36,001] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,002] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:36,002] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,002] jepsen worker 78 - jepsen.util 528 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,003] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:13:36,003] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,004] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:36,004] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:13:36,028] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,031] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:36,038] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,039] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 37] INFO [2022-03-13 17:13:36,082] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,082] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:36,084] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,084] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,085] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,085] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,086] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:13:36,086] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,115] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,115] jepsen worker 22 - jepsen.util 652 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,115] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,116] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:36,118] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,118] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:36,121] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,122] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:36,135] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,136] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:36,137] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,138] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:13:36,143] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,144] jepsen worker 21 - jepsen.util 831 :invoke :write [7 23] WARN [2022-03-13 17:13:36,155] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9669498,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9669498,"total_server_duration_us":9044797,"encode_duration_us":17821,"operation_id":"0x873a","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9673112},{"operation_name":"get","last_dispatch_duration_us":9167054,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9167054,"total_server_duration_us":62540,"operation_id":"0x879f","timeout_ms":30000,"last_server_duration_us":62540,"total_duration_us":9167517},{"operation_name":"get","last_dispatch_duration_us":9140128,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9140128,"total_server_duration_us":70556,"operation_id":"0x87a3","timeout_ms":30000,"last_server_duration_us":70556,"total_duration_us":9140504},{"operation_name":"upsert","last_dispatch_duration_us":9097722,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9097722,"total_server_duration_us":9037366,"encode_duration_us":33366,"operation_id":"0x88fa","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":9103579},{"operation_name":"upsert","last_dispatch_duration_us":9060150,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9060150,"total_server_duration_us":9046921,"encode_duration_us":20366,"operation_id":"0x88f4","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9064120},{"operation_name":"get","last_dispatch_duration_us":8177155,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8177155,"total_server_duration_us":727364,"operation_id":"0x8a02","timeout_ms":30000,"last_server_duration_us":727364,"total_duration_us":8179253},{"operation_name":"get","last_dispatch_duration_us":6339936,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6339936,"total_server_duration_us":24928,"operation_id":"0x8afa","timeout_ms":30000,"last_server_duration_us":24928,"total_duration_us":6343369},{"operation_name":"get","last_dispatch_duration_us":5722106,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":5722106,"total_server_duration_us":49,"operation_id":"0x8b42","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":5722388},{"operation_name":"get","last_dispatch_duration_us":5188530,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":5188530,"total_server_duration_us":23,"operation_id":"0x8ba0","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":5191498},{"operation_name":"upsert","last_dispatch_duration_us":786543,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1008665,"total_server_duration_us":184335,"encode_duration_us":17169,"operation_id":"0x8e79","timeout_ms":10000,"last_server_duration_us":101067,"total_duration_us":1025596}],"total_count":17}} INFO [2022-03-13 17:13:36,163] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,163] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,164] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,164] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,164] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:13:36,165] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:13:36,176] jepsen worker 45 - jepsen.util 2745 :info :write [15 5] :RequestTimeoutException INFO [2022-03-13 17:13:36,176] jepsen worker 45 - jepsen.util 2835 :invoke :write [15 14] INFO [2022-03-13 17:13:36,177] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,178] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,178] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,178] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,179] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,179] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:13:36,180] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,181] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,195] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,196] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,197] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,197] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,198] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:13:36,198] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,215] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,215] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:36,223] jepsen worker 52 - jepsen.util 592 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,224] jepsen worker 52 - jepsen.util 592 :invoke :write [17 38] INFO [2022-03-13 17:13:36,224] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,225] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,225] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,225] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,225] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,225] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:13:36,226] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:36,227] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:36,227] jepsen worker 13 - jepsen.util 643 :invoke :write [4 46] INFO [2022-03-13 17:13:36,227] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:36,237] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,237] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,237] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,237] jepsen worker 78 - jepsen.util 528 :invoke :write [26 3] INFO [2022-03-13 17:13:36,238] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:36,238] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:36,244] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,245] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:13:36,245] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,246] jepsen worker 80 - jepsen.util 710 :invoke :write [26 22] INFO [2022-03-13 17:13:36,247] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,247] jepsen worker 89 - jepsen.util 2429 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,248] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,248] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,250] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,251] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:13:36,257] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,257] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,257] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,258] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,258] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:13:36,259] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:13:36,260] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,260] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,262] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:13:36,262] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,263] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,264] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,266] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,267] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,268] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,269] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,271] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,271] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:13:36,274] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,275] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:13:36,300] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,300] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,301] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:13:36,302] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:13:36,316] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,316] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,317] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:36,317] jepsen worker 71 - jepsen.util 521 :invoke :write [23 1] INFO [2022-03-13 17:13:36,336] jepsen worker 29 - jepsen.util 2549 :info :write [9 16] :RequestTimeoutException INFO [2022-03-13 17:13:36,337] jepsen worker 29 - jepsen.util 2639 :invoke :read [9 nil] INFO [2022-03-13 17:13:36,359] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,359] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,360] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:13:36,362] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,363] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,364] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:13:36,371] jepsen worker 29 - jepsen.util 2639 :ok :read [9 26] INFO [2022-03-13 17:13:36,372] jepsen worker 29 - jepsen.util 2639 :invoke :write [9 13] INFO [2022-03-13 17:13:36,388] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,389] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:13:36,400] jepsen worker 17 - jepsen.util 2447 :info :write [5 24] :RequestTimeoutException INFO [2022-03-13 17:13:36,401] jepsen worker 17 - jepsen.util 2537 :invoke :write [5 7] INFO [2022-03-13 17:13:36,427] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,427] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,428] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:13:36,428] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,429] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,430] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:13:36,459] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,460] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,462] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,463] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:13:36,493] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,494] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,525] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:36,526] jepsen worker nemesis - couchbase.util Rebalance check count: 326 INFO [2022-03-13 17:13:36,526] jepsen worker nemesis - couchbase.util Rebalance stuck count: 226 INFO [2022-03-13 17:13:36,542] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,542] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,543] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,544] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:36,580] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,581] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,590] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,591] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:13:36,631] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,632] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,634] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,635] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,639] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,639] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,641] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,641] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,644] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,645] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,647] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,647] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:13:36,659] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,659] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,698] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,699] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:36,713] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,713] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,714] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,715] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,745] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,745] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,745] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,745] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,746] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:13:36,746] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,747] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,747] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,747] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:36,748] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,748] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,748] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,749] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,750] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,750] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,750] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:13:36,751] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,751] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,752] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,753] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:13:36,777] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,778] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,791] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,792] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,796] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,797] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:13:36,800] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,801] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,814] jepsen worker 63 - jepsen.util 2313 :info :write [21 20] :RequestTimeoutException INFO [2022-03-13 17:13:36,814] jepsen worker 63 - jepsen.util 2403 :invoke :read [21 nil] INFO [2022-03-13 17:13:36,843] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,844] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:13:36,858] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,859] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:36,919] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,919] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,923] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,924] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,931] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,932] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:36,934] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,934] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 19] INFO [2022-03-13 17:13:36,950] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,950] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,950] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,951] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:13:36,952] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,952] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:36,955] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:36,956] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:13:37,042] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,043] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 44] INFO [2022-03-13 17:13:37,044] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,045] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:13:37,045] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,046] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:13:37,058] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,058] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,058] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,060] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,060] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,061] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:13:37,104] jepsen worker 10 - jepsen.util 2620 :ok :read [3 25] INFO [2022-03-13 17:13:37,104] jepsen worker 10 - jepsen.util 2620 :invoke :read [3 nil] INFO [2022-03-13 17:13:37,136] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,136] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:13:37,152] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,152] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,152] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,153] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,153] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:13:37,154] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:13:37,171] jepsen worker 89 - jepsen.util 2429 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,172] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,175] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,175] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,180] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,181] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:13:37,198] jepsen worker 10 - jepsen.util 2620 :ok :read [3 25] INFO [2022-03-13 17:13:37,199] jepsen worker 10 - jepsen.util 2620 :invoke :read [3 nil] INFO [2022-03-13 17:13:37,224] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,224] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:13:37,249] jepsen worker 10 - jepsen.util 2620 :ok :read [3 25] INFO [2022-03-13 17:13:37,250] jepsen worker 10 - jepsen.util 2620 :invoke :write [3 11] INFO [2022-03-13 17:13:37,258] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,258] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,259] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:13:37,259] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,280] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,281] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:13:37,287] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,288] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,333] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,333] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:13:37,350] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,350] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,353] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,353] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,359] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,359] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:37,393] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,393] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:37,395] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,395] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,395] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:13:37,396] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,404] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,404] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,415] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,415] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:13:37,446] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,446] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:13:37,462] jepsen worker 37 - jepsen.util 2017 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,462] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,463] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,463] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,467] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,467] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,467] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,467] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,469] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,469] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,469] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,470] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:13:37,471] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,471] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,473] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,473] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,480] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,480] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:37,483] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,484] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:37,485] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,485] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:13:37,485] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,486] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:13:37,494] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,494] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:13:37,495] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,496] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:37,525] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,525] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,525] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,525] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,526] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,527] jepsen worker 24 - jepsen.util 564 :invoke :write [8 29] INFO [2022-03-13 17:13:37,529] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,529] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:13:37,531] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:37,531] jepsen worker nemesis - couchbase.util Rebalance check count: 327 INFO [2022-03-13 17:13:37,531] jepsen worker nemesis - couchbase.util Rebalance stuck count: 227 INFO [2022-03-13 17:13:37,553] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,553] jepsen worker 71 - jepsen.util 521 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,554] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:13:37,555] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] WARN [2022-03-13 17:13:37,558] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17374692,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17505226,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9200972,"encode_duration_us":24183,"operation_id":"0x8555","timeout_ms":10000,"last_server_duration_us":9112856,"total_duration_us":10001586},{"operation_name":"upsert","last_dispatch_duration_us":29579485,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":29579485,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9036305,"encode_duration_us":25697,"operation_id":"0x7bc0","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10007699}],"total_count":2}} INFO [2022-03-13 17:13:37,570] jepsen worker 28 - jepsen.util 2548 :ok :read [9 26] INFO [2022-03-13 17:13:37,571] jepsen worker 28 - jepsen.util 2548 :invoke :read [9 nil] INFO [2022-03-13 17:13:37,571] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,572] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,574] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,575] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:13:37,582] jepsen worker 21 - jepsen.util 831 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,583] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,583] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:37,583] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:13:37,589] jepsen worker 31 - jepsen.util 2551 :ok :read [10 :nil] INFO [2022-03-13 17:13:37,590] jepsen worker 31 - jepsen.util 2551 :invoke :write [10 39] INFO [2022-03-13 17:13:37,594] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,594] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,603] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,604] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:13:37,616] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,617] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:13:37,624] jepsen worker 7 - jepsen.util 637 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,624] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:37,637] jepsen worker 52 - jepsen.util 592 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,638] jepsen worker 52 - jepsen.util 592 :invoke :write [17 30] INFO [2022-03-13 17:13:37,640] jepsen worker 11 - jepsen.util 2441 :ok :read [3 25] INFO [2022-03-13 17:13:37,641] jepsen worker 11 - jepsen.util 2441 :invoke :write [3 11] INFO [2022-03-13 17:13:37,674] jepsen worker 24 - jepsen.util 564 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,675] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,675] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:13:37,676] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:37,681] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,682] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,682] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,682] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,683] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,684] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:13:37,684] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,685] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,701] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,702] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:37,714] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,714] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,714] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:13:37,715] jepsen worker 21 - jepsen.util 831 :invoke :write [7 15] INFO [2022-03-13 17:13:37,728] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,728] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] WARN [2022-03-13 17:13:37,758] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17975921,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18969285,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9046096,"encode_duration_us":18078,"operation_id":"0x83fd","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008001}],"total_count":1}} WARN [2022-03-13 17:13:37,758] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":130476,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1846273,"total_server_duration_us":298650,"encode_duration_us":22489,"operation_id":"0x8a72","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10010608},{"operation_name":"upsert","last_dispatch_duration_us":7794530,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10313830,"total_server_duration_us":320808,"encode_duration_us":27584,"operation_id":"0x8a1e","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10007143},{"operation_name":"upsert","last_dispatch_duration_us":45717,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9063445,"total_server_duration_us":366953,"encode_duration_us":29630,"operation_id":"0x8b4a","timeout_ms":10000,"last_server_duration_us":30918,"total_duration_us":10005592},{"operation_name":"upsert","last_dispatch_duration_us":7595715,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":12274573,"total_server_duration_us":338443,"encode_duration_us":19601,"operation_id":"0x88ba","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10005527},{"operation_name":"upsert","last_dispatch_duration_us":7923477,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9642344,"total_server_duration_us":381259,"encode_duration_us":19895,"operation_id":"0x8a71","timeout_ms":10000,"last_server_duration_us":51121,"total_duration_us":10003879},{"operation_name":"upsert","last_dispatch_duration_us":9198682,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10175321,"total_server_duration_us":9159803,"encode_duration_us":18309,"operation_id":"0x8ab0","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10003127},{"operation_name":"upsert","last_dispatch_duration_us":71092,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9402797,"total_server_duration_us":320282,"encode_duration_us":19982,"operation_id":"0x8b0b","timeout_ms":10000,"last_server_duration_us":42417,"total_duration_us":10002058},{"operation_name":"upsert","last_dispatch_duration_us":9117747,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9199881,"total_server_duration_us":9046777,"encode_duration_us":22741,"operation_id":"0x8871","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9234586},{"operation_name":"upsert","last_dispatch_duration_us":1319453,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1319453,"total_server_duration_us":1230344,"encode_duration_us":17961,"operation_id":"0x9141","timeout_ms":10000,"last_server_duration_us":1230344,"total_duration_us":1323086},{"operation_name":"upsert","last_dispatch_duration_us":652402,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":919279,"total_server_duration_us":516109,"encode_duration_us":21262,"operation_id":"0x8e84","timeout_ms":10000,"last_server_duration_us":419552,"total_duration_us":942827}],"total_count":11}} WARN [2022-03-13 17:13:37,759] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":35181981,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":35181981,"total_server_duration_us":38,"operation_id":"0x7d63","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":30006055},{"operation_name":"get","last_dispatch_duration_us":27973125,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":27973125,"total_server_duration_us":792904,"operation_id":"0x7cd8","timeout_ms":30000,"last_server_duration_us":792904,"total_duration_us":27973463},{"operation_name":"get","last_dispatch_duration_us":27195339,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":27195339,"total_server_duration_us":32,"operation_id":"0x819b","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":27206690},{"operation_name":"get","last_dispatch_duration_us":18101350,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18101350,"total_server_duration_us":23,"operation_id":"0x8353","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":24283765},{"operation_name":"get","last_dispatch_duration_us":17230837,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17230837,"total_server_duration_us":23,"operation_id":"0x87c8","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":17233475},{"operation_name":"upsert","last_dispatch_duration_us":10031883,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10031883,"total_server_duration_us":22554,"encode_duration_us":18959,"operation_id":"0x8a06","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10006919},{"operation_name":"get","last_dispatch_duration_us":8187070,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8187070,"total_server_duration_us":43,"operation_id":"0x8b02","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8189711},{"operation_name":"get","last_dispatch_duration_us":5782686,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5782686,"total_server_duration_us":23,"operation_id":"0x8c64","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":5782984},{"operation_name":"upsert","last_dispatch_duration_us":1687737,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2425998,"total_server_duration_us":227089,"encode_duration_us":27482,"operation_id":"0x8e39","timeout_ms":10000,"last_server_duration_us":106947,"total_duration_us":2448366},{"operation_name":"upsert","last_dispatch_duration_us":1638747,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1638747,"total_server_duration_us":51238,"encode_duration_us":30533,"operation_id":"0x8e3d","timeout_ms":10000,"last_server_duration_us":51238,"total_duration_us":1656696}],"total_count":49}} INFO [2022-03-13 17:13:37,765] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,765] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,765] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,766] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:13:37,766] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:13:37,771] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,799] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,799] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,801] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,801] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,803] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,803] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,804] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,804] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,809] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,809] jepsen worker 70 - jepsen.util 700 :invoke :write [23 36] INFO [2022-03-13 17:13:37,816] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,817] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:37,818] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,818] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,818] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,819] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,819] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:37,819] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:37,824] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,824] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:13:37,830] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,831] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] WARN [2022-03-13 17:13:37,856] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21721500,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21721500,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9038428,"encode_duration_us":17442,"operation_id":"0x80df","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10003624},{"operation_name":"upsert","last_dispatch_duration_us":9069687,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10905762,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9584895,"encode_duration_us":31272,"operation_id":"0x8517","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10005520}],"total_count":2}} WARN [2022-03-13 17:13:37,856] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":19278075,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":19278075,"total_server_duration_us":117,"operation_id":"0x8231","timeout_ms":30000,"last_server_duration_us":117,"total_duration_us":19278400},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":19365,"operation_id":"0x86d0","timeout_ms":10000,"total_duration_us":10008786},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":32876,"operation_id":"0x889a","timeout_ms":10000,"total_duration_us":10004137},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":32567,"operation_id":"0x88aa","timeout_ms":10000,"total_duration_us":10002670},{"operation_name":"upsert","last_dispatch_duration_us":9574498,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":9574498,"total_server_duration_us":9103272,"encode_duration_us":22693,"operation_id":"0x8a7a","timeout_ms":10000,"last_server_duration_us":9103272,"total_duration_us":9580133},{"operation_name":"upsert","last_dispatch_duration_us":9091246,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9091246,"total_server_duration_us":9063917,"encode_duration_us":49386,"operation_id":"0x8a8d","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":9097058},{"operation_name":"get","last_dispatch_duration_us":8454444,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8454444,"total_server_duration_us":165,"operation_id":"0x8b28","timeout_ms":30000,"last_server_duration_us":165,"total_duration_us":8457994},{"operation_name":"upsert","last_dispatch_duration_us":7910410,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":7910410,"total_server_duration_us":41660,"encode_duration_us":22535,"operation_id":"0x8b74","timeout_ms":10000,"last_server_duration_us":41660,"total_duration_us":7914319},{"operation_name":"upsert","last_dispatch_duration_us":7831809,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":7831809,"total_server_duration_us":66562,"encode_duration_us":19483,"operation_id":"0x8b73","timeout_ms":10000,"last_server_duration_us":66562,"total_duration_us":7835602},{"operation_name":"upsert","last_dispatch_duration_us":917680,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1536548,"total_server_duration_us":297495,"encode_duration_us":42130,"operation_id":"0x8e4c","timeout_ms":10000,"last_server_duration_us":77430,"total_duration_us":1568936}],"total_count":30}} WARN [2022-03-13 17:13:37,857] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":27754,"operation_id":"0x8b2f","timeout_ms":10000,"total_duration_us":10008728},{"operation_name":"upsert","last_dispatch_duration_us":37256,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":2157220,"total_server_duration_us":504616,"encode_duration_us":18111,"operation_id":"0x8ab3","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10003444},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":21769,"operation_id":"0x8b43","timeout_ms":10000,"total_duration_us":10001881},{"operation_name":"upsert","last_dispatch_duration_us":9193199,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":9193199,"total_server_duration_us":9136301,"encode_duration_us":24369,"operation_id":"0x8aa7","timeout_ms":10000,"last_server_duration_us":9136301,"total_duration_us":9198301},{"operation_name":"upsert","last_dispatch_duration_us":3956301,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":3956301,"total_server_duration_us":3945709,"encode_duration_us":34499,"operation_id":"0x8e4d","timeout_ms":10000,"last_server_duration_us":3945709,"total_duration_us":3961362},{"operation_name":"upsert","last_dispatch_duration_us":803250,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1530184,"total_server_duration_us":156155,"encode_duration_us":18699,"operation_id":"0x8e43","timeout_ms":10000,"last_server_duration_us":44500,"total_duration_us":1547055},{"operation_name":"upsert","last_dispatch_duration_us":873833,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1486262,"total_server_duration_us":168075,"encode_duration_us":18482,"operation_id":"0x8e30","timeout_ms":10000,"last_server_duration_us":88246,"total_duration_us":1499009},{"operation_name":"upsert","last_dispatch_duration_us":1401457,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1401457,"total_server_duration_us":656223,"encode_duration_us":19874,"operation_id":"0x8cae","timeout_ms":10000,"last_server_duration_us":656223,"total_duration_us":1405600},{"operation_name":"upsert","last_dispatch_duration_us":1378011,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1378011,"total_server_duration_us":61901,"encode_duration_us":23346,"operation_id":"0x9154","timeout_ms":10000,"last_server_duration_us":61901,"total_duration_us":1389665},{"operation_name":"upsert","last_dispatch_duration_us":1373198,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1373198,"total_server_duration_us":197914,"encode_duration_us":29508,"operation_id":"0x8ec3","timeout_ms":10000,"last_server_duration_us":197914,"total_duration_us":1378441}],"total_count":60}} INFO [2022-03-13 17:13:37,858] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,858] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:13:37,858] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16775009,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16775009,"total_server_duration_us":7437,"operation_id":"0x855d","timeout_ms":30000,"last_server_duration_us":7437,"total_duration_us":16778135},{"operation_name":"get","last_dispatch_duration_us":8939615,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8939615,"total_server_duration_us":8557,"operation_id":"0x88bf","timeout_ms":30000,"last_server_duration_us":8557,"total_duration_us":11560463},{"operation_name":"upsert","last_dispatch_duration_us":11690860,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11690860,"total_server_duration_us":21324,"encode_duration_us":26234,"operation_id":"0x8892","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10006747},{"operation_name":"upsert","last_dispatch_duration_us":1552069,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10274562,"total_server_duration_us":85543,"encode_duration_us":19039,"operation_id":"0x8b98","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10006237},{"operation_name":"upsert","last_dispatch_duration_us":211253,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9689848,"total_server_duration_us":193849,"encode_duration_us":39531,"operation_id":"0x8a7b","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10004149},{"operation_name":"upsert","last_dispatch_duration_us":15007480,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":15007480,"total_server_duration_us":23390,"encode_duration_us":17783,"operation_id":"0x865f","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10003606},{"operation_name":"upsert","last_dispatch_duration_us":9068804,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9464721,"total_server_duration_us":9109516,"encode_duration_us":19280,"operation_id":"0x8b11","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":9498673},{"operation_name":"get","last_dispatch_duration_us":9149950,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9149950,"total_server_duration_us":11427,"operation_id":"0x8a12","timeout_ms":30000,"last_server_duration_us":11427,"total_duration_us":9152692},{"operation_name":"get","last_dispatch_duration_us":6296330,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6296330,"total_server_duration_us":7028,"operation_id":"0x8cfb","timeout_ms":30000,"last_server_duration_us":7028,"total_duration_us":6300804},{"operation_name":"upsert","last_dispatch_duration_us":4366197,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4366197,"total_server_duration_us":112969,"encode_duration_us":20756,"operation_id":"0x8f6e","timeout_ms":10000,"last_server_duration_us":112969,"total_duration_us":4478875}],"total_count":56}} INFO [2022-03-13 17:13:37,859] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:13:37,860] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:37,860] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,861] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,861] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 8] INFO [2022-03-13 17:13:37,863] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:13:37,875] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,875] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 26] INFO [2022-03-13 17:13:37,879] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,879] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:37,886] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,886] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:37,901] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,902] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:13:37,905] jepsen worker 13 - jepsen.util 643 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,905] jepsen worker 13 - jepsen.util 643 :invoke :write [4 41] INFO [2022-03-13 17:13:37,925] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,925] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,926] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,926] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,926] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:37,927] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,927] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:37,928] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,928] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:37,929] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,929] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:13:37,930] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:37,931] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:37,931] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:13:37,932] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,933] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:13:37,933] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,933] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:37,934] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,935] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:37,936] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,937] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,937] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,937] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:13:37,938] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:13:37,938] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] WARN [2022-03-13 17:13:37,959] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9198682,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10175321,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9159803,"encode_duration_us":18309,"operation_id":"0x8ab0","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10003127}],"total_count":1}} INFO [2022-03-13 17:13:37,979] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:37,980] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:38,008] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,009] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,009] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:13:38,010] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:38,012] jepsen worker 3 - jepsen.util 1893 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,013] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:13:38,013] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,014] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:13:38,022] jepsen worker 76 - jepsen.util 706 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,022] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,022] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,023] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,023] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,023] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,024] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:13:38,025] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:38,026] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:38,027] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:38,028] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:38,028] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:38,035] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,036] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:38,036] jepsen worker 37 - jepsen.util 2017 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,037] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,037] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 21] INFO [2022-03-13 17:13:38,038] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:13:38,048] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,049] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:38,050] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,051] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] WARN [2022-03-13 17:13:38,059] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18123041,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27217730,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9065595,"encode_duration_us":34369,"operation_id":"0x7fc4","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10001958}],"total_count":1}} INFO [2022-03-13 17:13:38,076] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,076] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:38,095] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,096] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:38,098] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,099] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,100] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:38,100] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:13:38,107] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,107] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,108] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:13:38,109] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:13:38,129] jepsen worker 56 - jepsen.util 2576 :ok :read [18 21] INFO [2022-03-13 17:13:38,133] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,133] jepsen worker 21 - jepsen.util 831 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,134] jepsen worker 56 - jepsen.util 2576 :invoke :write [18 48] INFO [2022-03-13 17:13:38,132] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,134] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:38,130] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,134] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:13:38,136] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,137] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:13:38,143] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,144] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:13:38,149] jepsen worker 73 - jepsen.util 2683 :ok :read [24 24] INFO [2022-03-13 17:13:38,149] jepsen worker 73 - jepsen.util 2683 :invoke :write [24 7] INFO [2022-03-13 17:13:38,162] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,163] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:13:38,205] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,205] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,205] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,206] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:38,206] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,207] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:13:38,211] jepsen worker 78 - jepsen.util 528 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,212] jepsen worker 78 - jepsen.util 528 :invoke :write [26 23] INFO [2022-03-13 17:13:38,246] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,247] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:13:38,260] jepsen worker 74 - jepsen.util 2684 :ok :read [24 24] INFO [2022-03-13 17:13:38,261] jepsen worker 74 - jepsen.util 2684 :invoke :read [24 nil] INFO [2022-03-13 17:13:38,272] jepsen worker 9 - jepsen.util 2169 :ok :read [3 25] INFO [2022-03-13 17:13:38,273] jepsen worker 9 - jepsen.util 2169 :invoke :read [3 nil] INFO [2022-03-13 17:13:38,276] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,276] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,276] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,277] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:38,278] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,279] jepsen worker 24 - jepsen.util 564 :invoke :write [8 30] INFO [2022-03-13 17:13:38,311] jepsen worker 28 - jepsen.util 2548 :ok :read [9 26] INFO [2022-03-13 17:13:38,312] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,312] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,312] jepsen worker 28 - jepsen.util 2548 :invoke :write [9 9] INFO [2022-03-13 17:13:38,312] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,313] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:38,313] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:38,313] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,339] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,339] jepsen worker 24 - jepsen.util 564 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,339] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,340] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,340] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:38,341] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:13:38,341] jepsen worker 75 - jepsen.util 435 :invoke :write [25 49] INFO [2022-03-13 17:13:38,342] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:13:38,362] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,363] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:13:38,379] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,380] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:38,392] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,392] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:13:38,404] jepsen worker 74 - jepsen.util 2684 :ok :read [24 24] INFO [2022-03-13 17:13:38,405] jepsen worker 74 - jepsen.util 2684 :invoke :write [24 22] INFO [2022-03-13 17:13:38,414] jepsen worker 9 - jepsen.util 2169 :ok :read [3 25] INFO [2022-03-13 17:13:38,415] jepsen worker 9 - jepsen.util 2169 :invoke :read [3 nil] INFO [2022-03-13 17:13:38,417] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,417] jepsen worker 80 - jepsen.util 710 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,418] jepsen worker 80 - jepsen.util 710 :invoke :write [26 23] INFO [2022-03-13 17:13:38,419] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:38,424] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,424] jepsen worker 22 - jepsen.util 652 :invoke :write [7 37] INFO [2022-03-13 17:13:38,460] jepsen worker 75 - jepsen.util 435 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,460] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:13:38,464] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,465] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:13:38,530] jepsen worker 52 - jepsen.util 592 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,531] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:38,534] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,535] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:13:38,538] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:38,538] jepsen worker nemesis - couchbase.util Rebalance check count: 328 INFO [2022-03-13 17:13:38,538] jepsen worker nemesis - couchbase.util Rebalance stuck count: 228 INFO [2022-03-13 17:13:38,552] jepsen worker 9 - jepsen.util 2169 :ok :read [3 25] INFO [2022-03-13 17:13:38,552] jepsen worker 9 - jepsen.util 2169 :invoke :read [3 nil] INFO [2022-03-13 17:13:38,554] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,555] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:38,557] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,558] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:13:38,589] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,589] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,590] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:13:38,590] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:38,596] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,596] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:38,621] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,622] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:38,629] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,630] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:13:38,637] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,637] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:13:38,647] jepsen worker 70 - jepsen.util 700 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,647] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:38,648] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,648] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,654] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,654] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:13:38,676] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,677] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:13:38,702] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,703] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:38,712] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,713] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:38,720] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,721] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:38,722] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,723] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:38,724] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,725] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:13:38,745] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,745] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,746] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:13:38,751] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:13:38,757] jepsen worker 22 - jepsen.util 652 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,758] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,758] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:38,759] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,763] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,765] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:13:38,768] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,769] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:38,806] jepsen worker 37 - jepsen.util 2017 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,807] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 2] INFO [2022-03-13 17:13:38,814] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,815] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:13:38,832] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,832] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,833] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,833] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:38,836] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,836] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,837] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:38,837] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:38,852] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,853] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:13:38,893] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,894] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:38,896] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,897] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:38,910] jepsen worker 27 - jepsen.util 2907 :info :write [9 39] :RequestTimeoutException INFO [2022-03-13 17:13:38,911] jepsen worker 27 - jepsen.util 2997 :invoke :write [9 47] INFO [2022-03-13 17:13:38,923] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,923] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,923] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:13:38,924] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,924] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:13:38,924] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:38,928] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,928] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,928] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,928] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,929] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:38,930] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:13:38,930] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:38,932] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,932] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:13:38,934] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:13:38,938] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,939] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:13:38,961] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,962] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:38,963] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,964] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:13:38,970] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,971] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:13:38,975] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,976] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:38,980] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,981] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:13:38,991] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,992] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:13:38,997] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:38,998] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:13:39,040] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,041] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,043] jepsen worker 37 - jepsen.util 2017 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,043] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,045] jepsen worker 78 - jepsen.util 528 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,045] jepsen worker 78 - jepsen.util 528 :invoke :write [26 32] INFO [2022-03-13 17:13:39,047] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,047] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,049] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,050] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:39,061] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,061] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,061] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,062] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:39,062] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:13:39,063] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:39,089] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,090] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:39,101] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,101] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,107] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,107] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:13:39,115] jepsen worker 13 - jepsen.util 643 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,116] jepsen worker 13 - jepsen.util 643 :invoke :write [4 31] INFO [2022-03-13 17:13:39,149] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,150] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,160] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,160] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,160] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,161] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:39,161] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:39,162] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:13:39,163] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,163] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,163] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:13:39,164] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,164] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,164] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,165] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,165] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,165] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,165] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,166] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,166] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,167] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,167] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,168] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,168] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:39,168] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,169] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:13:39,170] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,170] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:39,171] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:13:39,172] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:13:39,206] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,207] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,207] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:13:39,209] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,211] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,212] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,214] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,214] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,214] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,214] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,215] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,216] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:13:39,216] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,216] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:13:39,217] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:13:39,226] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:13:39,252] jepsen worker 80 - jepsen.util 710 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,253] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,253] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,252] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,254] jepsen worker 80 - jepsen.util 710 :invoke :write [26 16] INFO [2022-03-13 17:13:39,254] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,255] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:13:39,256] jepsen worker 52 - jepsen.util 592 :invoke :write [17 46] INFO [2022-03-13 17:13:39,264] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,264] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,264] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,265] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,266] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,266] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,266] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:39,267] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,267] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,267] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,268] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:13:39,269] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:39,272] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,273] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:39,274] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,274] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,274] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,275] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:13:39,275] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,276] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,277] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,278] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,279] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,279] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:13:39,288] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,290] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:13:39,303] jepsen worker 65 - jepsen.util 2765 :info :write [21 3] :RequestTimeoutException INFO [2022-03-13 17:13:39,304] jepsen worker 65 - jepsen.util 2855 :invoke :read [21 nil] INFO [2022-03-13 17:13:39,318] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,318] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,319] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:39,320] jepsen worker 47 - jepsen.util 2387 :info :write [15 20] :RequestTimeoutException INFO [2022-03-13 17:13:39,320] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:39,321] jepsen worker 47 - jepsen.util 2477 :invoke :write [15 12] INFO [2022-03-13 17:13:39,351] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,351] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,352] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:39,353] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:39,355] jepsen worker 78 - jepsen.util 528 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,356] jepsen worker 78 - jepsen.util 528 :invoke :write [26 29] INFO [2022-03-13 17:13:39,376] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,376] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,399] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,399] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,399] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,399] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,400] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:13:39,400] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:13:39,401] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:39,401] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:13:39,405] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,406] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:13:39,413] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,414] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:39,415] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,415] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:39,423] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,424] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:13:39,430] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,431] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:39,464] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,464] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,465] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,466] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:13:39,468] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,469] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,478] jepsen worker 13 - jepsen.util 643 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,478] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,478] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,479] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,479] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,480] jepsen worker 13 - jepsen.util 643 :invoke :write [4 20] INFO [2022-03-13 17:13:39,481] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,481] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:13:39,518] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,519] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 27] INFO [2022-03-13 17:13:39,523] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,524] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:39,536] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,536] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:13:39,544] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:39,544] jepsen worker nemesis - couchbase.util Rebalance check count: 329 INFO [2022-03-13 17:13:39,544] jepsen worker nemesis - couchbase.util Rebalance stuck count: 229 INFO [2022-03-13 17:13:39,564] jepsen worker 80 - jepsen.util 710 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,564] jepsen worker 80 - jepsen.util 710 :invoke :write [26 23] INFO [2022-03-13 17:13:39,565] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,566] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:13:39,574] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,574] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,575] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,575] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,582] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,582] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,582] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,583] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:39,586] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,587] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,587] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,587] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:13:39,591] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,591] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 45] INFO [2022-03-13 17:13:39,594] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,595] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:13:39,620] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,620] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,620] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,621] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:39,621] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:13:39,622] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:39,651] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,652] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:39,653] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,654] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:39,658] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,659] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:39,660] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,662] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:39,665] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,666] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 40] INFO [2022-03-13 17:13:39,668] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,668] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:13:39,673] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,674] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:13:39,685] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,685] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,685] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,686] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,686] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:39,687] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:13:39,687] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,692] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,694] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,695] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:13:39,696] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,697] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:13:39,713] jepsen worker 30 - jepsen.util 2460 :ok :read [10 :nil] INFO [2022-03-13 17:13:39,713] jepsen worker 64 - jepsen.util 2494 :ok :read [21 :nil] INFO [2022-03-13 17:13:39,715] jepsen worker 30 - jepsen.util 2460 :invoke :read [10 nil] INFO [2022-03-13 17:13:39,715] jepsen worker 64 - jepsen.util 2494 :invoke :write [21 14] INFO [2022-03-13 17:13:39,727] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,728] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,728] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,728] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,728] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:39,729] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:39,730] jepsen worker 55 - jepsen.util 2395 :ok :read [18 21] INFO [2022-03-13 17:13:39,730] jepsen worker 55 - jepsen.util 2395 :invoke :read [18 nil] INFO [2022-03-13 17:13:39,741] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,741] jepsen worker 30 - jepsen.util 2460 :ok :read [10 :nil] INFO [2022-03-13 17:13:39,741] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:39,742] jepsen worker 30 - jepsen.util 2460 :invoke :write [10 13] INFO [2022-03-13 17:13:39,745] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,746] jepsen worker 70 - jepsen.util 700 :invoke :write [23 0] INFO [2022-03-13 17:13:39,747] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,747] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:13:39,763] jepsen worker 65 - jepsen.util 2855 :ok :read [21 :nil] INFO [2022-03-13 17:13:39,763] jepsen worker 65 - jepsen.util 2855 :invoke :write [21 15] INFO [2022-03-13 17:13:39,784] jepsen worker 55 - jepsen.util 2395 :ok :read [18 21] INFO [2022-03-13 17:13:39,785] jepsen worker 55 - jepsen.util 2395 :invoke :write [18 29] INFO [2022-03-13 17:13:39,787] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,787] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,789] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,789] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:13:39,820] jepsen worker 88 - jepsen.util 2068 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,821] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:39,822] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,822] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,822] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,822] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,823] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,824] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:13:39,824] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,824] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:13:39,825] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,826] jepsen worker 52 - jepsen.util 592 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,826] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:13:39,827] jepsen worker 52 - jepsen.util 592 :invoke :write [17 37] INFO [2022-03-13 17:13:39,882] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,883] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:39,883] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,884] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:13:39,898] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,898] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:39,900] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,900] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:13:39,928] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,929] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:13:39,940] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,940] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:39,947] jepsen worker 78 - jepsen.util 528 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,947] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,947] jepsen worker 34 - jepsen.util 754 :invoke :write [11 46] INFO [2022-03-13 17:13:39,949] jepsen worker 78 - jepsen.util 528 :invoke :write [26 23] INFO [2022-03-13 17:13:39,969] jepsen worker 89 - jepsen.util 2429 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,970] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:13:39,993] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,995] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 5] INFO [2022-03-13 17:13:39,996] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:39,997] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:13:39,993] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,000] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,001] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,001] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,001] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,002] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,003] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:40,003] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,004] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,004] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,007] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,007] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,007] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,008] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:13:40,008] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:13:40,009] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:13:40,039] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,040] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:13:40,048] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,049] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,050] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,050] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,053] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,055] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,058] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,060] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:13:40,066] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,066] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,068] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:13:40,068] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:40,071] jepsen worker 37 - jepsen.util 2017 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,072] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 1] INFO [2022-03-13 17:13:40,123] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,123] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,123] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,123] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,124] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:13:40,125] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:40,132] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,133] jepsen worker 13 - jepsen.util 643 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,133] jepsen worker 13 - jepsen.util 643 :invoke :write [4 13] INFO [2022-03-13 17:13:40,133] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,134] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:13:40,135] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,136] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,137] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,150] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,150] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,151] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:40,151] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:40,152] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,153] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:13:40,170] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,171] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,210] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,210] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,210] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,211] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:13:40,212] jepsen worker 24 - jepsen.util 564 :invoke :write [8 1] INFO [2022-03-13 17:13:40,212] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:40,222] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,223] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:13:40,239] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,240] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:13:40,244] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,244] jepsen worker 3 - jepsen.util 1893 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,244] jepsen worker 80 - jepsen.util 710 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,246] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:13:40,247] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:40,248] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,261] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,262] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,292] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,292] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,293] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:13:40,293] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,321] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,322] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:13:40,334] jepsen worker 37 - jepsen.util 2017 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,334] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,334] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,335] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,344] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,344] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,344] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,344] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,345] jepsen worker 70 - jepsen.util 700 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,345] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:13:40,346] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,346] jepsen worker 70 - jepsen.util 700 :invoke :write [23 44] INFO [2022-03-13 17:13:40,347] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:13:40,348] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,349] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:40,355] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:13:40,356] jepsen worker 24 - jepsen.util 564 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,357] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,362] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,362] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,362] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,362] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,363] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,363] jepsen worker 21 - jepsen.util 831 :invoke :write [7 29] INFO [2022-03-13 17:13:40,367] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,367] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,368] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,369] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,369] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,370] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,370] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:13:40,371] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:13:40,404] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,405] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:13:40,421] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,421] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,421] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:13:40,427] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:13:40,433] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,433] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,433] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,434] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:40,435] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 41] INFO [2022-03-13 17:13:40,435] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:13:40,469] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,469] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,469] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:13:40,470] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,505] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,506] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,507] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,507] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:13:40,515] jepsen worker 21 - jepsen.util 831 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,515] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:13:40,527] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,527] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,528] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:13:40,528] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:13:40,536] jepsen worker 52 - jepsen.util 592 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,537] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:40,538] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,541] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,550] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:40,550] jepsen worker nemesis - couchbase.util Rebalance check count: 330 INFO [2022-03-13 17:13:40,550] jepsen worker nemesis - couchbase.util Rebalance stuck count: 230 INFO [2022-03-13 17:13:40,585] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,585] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,587] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,587] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,588] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,589] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,590] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,591] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:13:40,594] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,595] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:13:40,623] jepsen worker 3 - jepsen.util 1893 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,624] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,631] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,631] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,632] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,633] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,640] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,641] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:40,642] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,642] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,642] jepsen worker 24 - jepsen.util 564 :invoke :write [8 34] INFO [2022-03-13 17:13:40,643] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,646] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,647] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:40,673] jepsen worker 68 - jepsen.util 2588 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,674] jepsen worker 68 - jepsen.util 2588 :invoke :read [22 nil] INFO [2022-03-13 17:13:40,676] jepsen worker 34 - jepsen.util 754 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,676] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,678] jepsen worker 22 - jepsen.util 652 :invoke :write [7 7] INFO [2022-03-13 17:13:40,678] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,679] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,680] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:40,684] jepsen worker 59 - jepsen.util 2489 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,686] jepsen worker 59 - jepsen.util 2489 :invoke :read [19 nil] INFO [2022-03-13 17:13:40,686] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,687] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,687] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,688] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:40,692] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,692] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,692] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,693] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 19] INFO [2022-03-13 17:13:40,693] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,693] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:40,695] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,695] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,696] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:40,697] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,700] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,700] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,701] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:13:40,701] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:13:40,706] jepsen worker 32 - jepsen.util 2372 :info :write [10 4] :RequestTimeoutException INFO [2022-03-13 17:13:40,708] jepsen worker 32 - jepsen.util 2462 :invoke :read [10 nil] INFO [2022-03-13 17:13:40,710] jepsen worker 32 - jepsen.util 2462 :ok :read [10 :nil] INFO [2022-03-13 17:13:40,711] jepsen worker 32 - jepsen.util 2462 :invoke :read [10 nil] INFO [2022-03-13 17:13:40,712] jepsen worker 32 - jepsen.util 2462 :ok :read [10 :nil] INFO [2022-03-13 17:13:40,713] jepsen worker 32 - jepsen.util 2462 :invoke :read [10 nil] INFO [2022-03-13 17:13:40,715] jepsen worker 32 - jepsen.util 2462 :ok :read [10 :nil] INFO [2022-03-13 17:13:40,716] jepsen worker 32 - jepsen.util 2462 :invoke :read [10 nil] INFO [2022-03-13 17:13:40,717] jepsen worker 32 - jepsen.util 2462 :ok :read [10 :nil] INFO [2022-03-13 17:13:40,718] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,719] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:13:40,719] jepsen worker 32 - jepsen.util 2462 :invoke :read [10 nil] INFO [2022-03-13 17:13:40,721] jepsen worker 32 - jepsen.util 2462 :ok :read [10 :nil] INFO [2022-03-13 17:13:40,721] jepsen worker 32 - jepsen.util 2462 :invoke :read [10 nil] INFO [2022-03-13 17:13:40,722] jepsen worker 32 - jepsen.util 2462 :ok :read [10 :nil] INFO [2022-03-13 17:13:40,723] jepsen worker 32 - jepsen.util 2462 :invoke :write [10 38] INFO [2022-03-13 17:13:40,749] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,751] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,751] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,751] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,755] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:40,756] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:40,756] jepsen worker 33 - jepsen.util 663 :invoke :write [11 44] INFO [2022-03-13 17:13:40,757] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:40,762] jepsen worker 20 - jepsen.util 2810 :info :write [6 36] :SyncWriteAmbiguous INFO [2022-03-13 17:13:40,762] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,762] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,762] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,764] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:13:40,765] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:13:40,765] jepsen worker 20 - jepsen.util 2900 :invoke :write [6 40] INFO [2022-03-13 17:13:40,765] jepsen worker 78 - jepsen.util 528 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,766] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:40,767] jepsen worker 78 - jepsen.util 528 :invoke :write [26 29] INFO [2022-03-13 17:13:40,767] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,768] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:13:40,774] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,777] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:40,836] jepsen worker 24 - jepsen.util 564 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,836] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,836] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,837] jepsen worker 59 - jepsen.util 2489 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,838] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:40,839] jepsen worker 59 - jepsen.util 2489 :invoke :read [19 nil] INFO [2022-03-13 17:13:40,840] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:13:40,840] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:13:40,845] jepsen worker 37 - jepsen.util 2017 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,846] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:13:40,851] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,851] jepsen worker 63 - jepsen.util 2403 :ok :read [21 :nil] INFO [2022-03-13 17:13:40,851] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:13:40,852] jepsen worker 63 - jepsen.util 2403 :invoke :read [21 nil] INFO [2022-03-13 17:13:40,855] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,855] jepsen worker 63 - jepsen.util 2403 :ok :read [21 :nil] INFO [2022-03-13 17:13:40,856] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:13:40,857] jepsen worker 68 - jepsen.util 2588 :ok :read [22 30] INFO [2022-03-13 17:13:40,858] jepsen worker 68 - jepsen.util 2588 :invoke :read [22 nil] INFO [2022-03-13 17:13:40,862] jepsen worker 63 - jepsen.util 2403 :invoke :write [21 24] INFO [2022-03-13 17:13:40,863] jepsen worker 59 - jepsen.util 2489 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,864] jepsen worker 59 - jepsen.util 2489 :invoke :write [19 1] INFO [2022-03-13 17:13:40,885] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,887] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:40,889] jepsen worker 68 - jepsen.util 2588 :ok :read [22 30] INFO [2022-03-13 17:13:40,890] jepsen worker 68 - jepsen.util 2588 :invoke :read [22 nil] INFO [2022-03-13 17:13:40,927] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,927] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:13:40,937] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,937] jepsen worker 33 - jepsen.util 663 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,937] jepsen worker 13 - jepsen.util 643 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,937] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,938] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:40,939] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:40,939] jepsen worker 13 - jepsen.util 643 :invoke :write [4 45] INFO [2022-03-13 17:13:40,940] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:40,950] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,950] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,951] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:40,951] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:13:40,952] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,952] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:40,953] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:40,954] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:13:40,998] jepsen worker 68 - jepsen.util 2588 :ok :read [22 30] INFO [2022-03-13 17:13:40,999] jepsen worker 68 - jepsen.util 2588 :invoke :read [22 nil] INFO [2022-03-13 17:13:41,002] jepsen worker 22 - jepsen.util 652 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,003] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,004] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,005] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:13:41,018] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,018] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,019] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:41,019] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,028] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,030] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,036] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,036] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,038] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,038] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,038] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,039] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,045] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,046] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:13:41,072] jepsen worker 68 - jepsen.util 2588 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,073] jepsen worker 68 - jepsen.util 2588 :invoke :read [22 nil] INFO [2022-03-13 17:13:41,096] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,096] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,096] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,097] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:41,097] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:41,098] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,098] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,099] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,099] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:41,099] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:41,110] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,111] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,110] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,111] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,111] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,111] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:41,112] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:13:41,112] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,112] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:41,113] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:13:41,113] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,113] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,116] jepsen worker 70 - jepsen.util 700 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,116] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,116] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,117] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:13:41,117] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,117] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:41,118] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:41,118] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:41,119] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,120] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:13:41,120] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,121] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,122] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,123] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,125] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,125] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,126] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,127] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 39] INFO [2022-03-13 17:13:41,127] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:13:41,128] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:13:41,137] jepsen worker 39 - jepsen.util 2379 :info :write [13 2] :SyncWriteAmbiguous INFO [2022-03-13 17:13:41,140] jepsen worker 39 - jepsen.util 2469 :invoke :read [13 nil] INFO [2022-03-13 17:13:41,150] jepsen worker 68 - jepsen.util 2588 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,150] jepsen worker 68 - jepsen.util 2588 :invoke :read [22 nil] INFO [2022-03-13 17:13:41,168] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,169] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,170] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,171] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,186] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,186] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,188] jepsen worker 39 - jepsen.util 2469 :ok :read [13 39] INFO [2022-03-13 17:13:41,188] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,189] jepsen worker 39 - jepsen.util 2469 :invoke :read [13 nil] INFO [2022-03-13 17:13:41,189] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,191] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,192] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,195] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,195] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,197] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,197] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,205] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,205] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:13:41,210] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,211] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,212] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,213] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:41,215] jepsen worker 39 - jepsen.util 2469 :ok :read [13 39] INFO [2022-03-13 17:13:41,215] jepsen worker 39 - jepsen.util 2469 :invoke :write [13 28] INFO [2022-03-13 17:13:41,217] jepsen worker 68 - jepsen.util 2588 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,218] jepsen worker 68 - jepsen.util 2588 :invoke :write [22 44] INFO [2022-03-13 17:13:41,249] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,249] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,249] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,250] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,250] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,251] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 30] INFO [2022-03-13 17:13:41,251] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,252] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:13:41,253] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,259] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,259] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,260] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,260] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,261] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,262] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,262] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,262] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,262] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,262] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,263] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:41,264] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:41,265] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,266] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,266] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:41,267] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:13:41,267] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,268] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,268] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,269] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:13:41,270] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:13:41,271] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:13:41,271] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:13:41,273] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:13:41,275] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:41,278] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,284] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:13:41,320] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,323] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,335] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,335] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,335] jepsen worker 78 - jepsen.util 528 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,336] jepsen worker 78 - jepsen.util 528 :invoke :write [26 0] INFO [2022-03-13 17:13:41,336] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:41,337] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,344] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,344] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,346] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,346] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,347] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,348] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:13:41,348] jepsen worker 62 - jepsen.util 512 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,348] jepsen worker 44 - jepsen.util 584 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,348] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,349] jepsen worker 62 - jepsen.util 512 :invoke :write [20 43] INFO [2022-03-13 17:13:41,349] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:41,350] jepsen worker 51 - jepsen.util 591 :invoke :write [17 30] INFO [2022-03-13 17:13:41,355] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,355] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,358] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,359] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,368] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,368] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,369] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,369] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:13:41,375] jepsen worker 3 - jepsen.util 1893 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,375] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,376] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 13] INFO [2022-03-13 17:13:41,377] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,380] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,380] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,383] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,384] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,397] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,398] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:41,409] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,410] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:13:41,415] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,415] jepsen worker 37 - jepsen.util 2017 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,415] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,416] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,448] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,449] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:13:41,461] jepsen worker 50 - jepsen.util 2660 :ok :read [16 :nil] INFO [2022-03-13 17:13:41,463] jepsen worker 50 - jepsen.util 2660 :invoke :read [16 nil] INFO [2022-03-13 17:13:41,465] jepsen worker 41 - jepsen.util 2381 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,465] jepsen worker 41 - jepsen.util 2381 :invoke :write [13 31] INFO [2022-03-13 17:13:41,467] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,468] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,470] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,470] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,470] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,471] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:13:41,472] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,472] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,474] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,474] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,474] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,474] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,475] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,476] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,476] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,477] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:13:41,478] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,479] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,481] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,485] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,486] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,487] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,488] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,489] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,490] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,491] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,491] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,491] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:13:41,492] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,492] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:13:41,528] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,529] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:13:41,530] jepsen worker 13 - jepsen.util 643 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,530] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,531] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,531] jepsen worker 34 - jepsen.util 754 :invoke :write [11 22] INFO [2022-03-13 17:13:41,533] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:41,533] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:41,539] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,539] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:41,539] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,540] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,540] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,541] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,543] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:41,544] jepsen worker 69 - jepsen.util 699 :invoke :write [23 37] INFO [2022-03-13 17:13:41,545] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:41,545] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:41,558] jepsen worker 57 - jepsen.util 2397 :info :write [19 13] :RequestTimeoutException INFO [2022-03-13 17:13:41,559] jepsen worker 57 - jepsen.util 2487 :invoke :write [19 44] INFO [2022-03-13 17:13:41,559] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:41,559] jepsen worker nemesis - couchbase.util Rebalance check count: 331 INFO [2022-03-13 17:13:41,559] jepsen worker nemesis - couchbase.util Rebalance stuck count: 231 INFO [2022-03-13 17:13:41,564] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,565] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,565] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:41,567] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:13:41,579] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,580] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,583] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,584] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,628] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,628] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,628] jepsen worker 80 - jepsen.util 710 :invoke :write [26 43] INFO [2022-03-13 17:13:41,629] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:41,647] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,648] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:41,660] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,661] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:13:41,692] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,692] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:41,694] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,695] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,696] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,697] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,698] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,699] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:41,716] jepsen worker 50 - jepsen.util 2660 :ok :read [16 :nil] INFO [2022-03-13 17:13:41,717] jepsen worker 50 - jepsen.util 2660 :invoke :read [16 nil] INFO [2022-03-13 17:13:41,722] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,723] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:41,724] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,724] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,736] jepsen worker 84 - jepsen.util 2694 :info :write [28 26] :RequestTimeoutException INFO [2022-03-13 17:13:41,737] jepsen worker 84 - jepsen.util 2784 :invoke :read [28 nil] INFO [2022-03-13 17:13:41,763] jepsen worker 78 - jepsen.util 528 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,763] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,763] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,764] jepsen worker 13 - jepsen.util 643 :invoke :write [4 21] INFO [2022-03-13 17:13:41,765] jepsen worker 78 - jepsen.util 528 :invoke :write [26 20] INFO [2022-03-13 17:13:41,765] jepsen worker 52 - jepsen.util 592 :invoke :write [17 49] INFO [2022-03-13 17:13:41,772] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,773] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:13:41,801] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,801] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,802] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,802] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,802] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,802] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:13:41,803] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:13:41,803] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:13:41,805] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:13:41,805] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 12] INFO [2022-03-13 17:13:41,808] jepsen worker 3 - jepsen.util 1893 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,809] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,809] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,809] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,810] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,812] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,812] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,813] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,813] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:13:41,816] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:13:41,816] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,817] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:41,819] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,820] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,821] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:13:41,821] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:13:41,870] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,871] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,871] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,871] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:13:41,904] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,904] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:41,907] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,908] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:13:41,931] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,931] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:41,942] jepsen worker 37 - jepsen.util 2017 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,942] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,965] jepsen worker 50 - jepsen.util 2660 :ok :read [16 :nil] INFO [2022-03-13 17:13:41,966] jepsen worker 50 - jepsen.util 2660 :invoke :read [16 nil] INFO [2022-03-13 17:13:41,978] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,979] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:41,985] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,986] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,986] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:41,987] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,987] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:13:41,987] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:13:41,992] jepsen worker 34 - jepsen.util 754 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,992] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,992] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:41,993] jepsen worker 70 - jepsen.util 700 :invoke :write [23 20] INFO [2022-03-13 17:13:41,994] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:13:41,996] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:42,023] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,024] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,027] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,028] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,029] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,029] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,029] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,029] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:42,030] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:42,031] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:13:42,061] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,062] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:13:42,100] jepsen worker 82 - jepsen.util 2422 :info :write [27 17] :RequestTimeoutException INFO [2022-03-13 17:13:42,100] jepsen worker 49 - jepsen.util 2119 :info :write [16 47] :RequestTimeoutException INFO [2022-03-13 17:13:42,101] jepsen worker 49 - jepsen.util 2209 :invoke :write [16 33] INFO [2022-03-13 17:13:42,102] jepsen worker 82 - jepsen.util 2512 :invoke :write [27 20] INFO [2022-03-13 17:13:42,104] jepsen worker 80 - jepsen.util 710 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,105] jepsen worker 80 - jepsen.util 710 :invoke :write [26 33] INFO [2022-03-13 17:13:42,114] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,114] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,115] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,115] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:13:42,118] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,118] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,119] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:42,119] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:13:42,123] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,124] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:42,126] jepsen worker 66 - jepsen.util 2316 :info :write [22 16] :RequestTimeoutException INFO [2022-03-13 17:13:42,127] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,127] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,127] jepsen worker 66 - jepsen.util 2406 :invoke :write [22 5] INFO [2022-03-13 17:13:42,129] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,129] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,129] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,129] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,130] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,131] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:42,131] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:42,132] jepsen worker 75 - jepsen.util 435 :invoke :write [25 49] INFO [2022-03-13 17:13:42,133] jepsen worker 33 - jepsen.util 663 :invoke :write [11 42] INFO [2022-03-13 17:13:42,142] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:42,143] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,144] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,146] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,147] jepsen worker 50 - jepsen.util 2660 :ok :read [16 :nil] INFO [2022-03-13 17:13:42,147] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:42,148] jepsen worker 50 - jepsen.util 2660 :invoke :write [16 16] INFO [2022-03-13 17:13:42,184] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,184] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:13:42,185] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,186] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:42,191] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,191] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,192] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:42,193] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:42,200] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,200] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,200] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,200] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,202] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,202] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,203] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,203] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:13:42,204] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,205] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:13:42,206] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,206] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,207] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,208] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,208] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,208] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,209] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:13:42,209] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,209] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,210] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,210] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:13:42,210] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:13:42,226] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,227] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,227] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,227] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,228] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 40] INFO [2022-03-13 17:13:42,229] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,229] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,230] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,230] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,231] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,232] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:13:42,232] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,233] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,235] jepsen worker 78 - jepsen.util 528 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,236] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,237] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,238] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:13:42,238] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:42,248] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,249] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:13:42,284] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,284] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,284] jepsen worker 75 - jepsen.util 435 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,284] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,285] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:13:42,286] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,286] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:42,287] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:13:42,288] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:42,288] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:13:42,290] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,290] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,291] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:42,291] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:42,292] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,293] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:13:42,300] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,301] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:42,302] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,303] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:13:42,325] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,326] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,341] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,342] jepsen worker 52 - jepsen.util 592 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,342] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:13:42,342] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:42,387] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,388] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,388] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,388] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:13:42,391] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,392] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:42,398] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,399] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:42,400] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,401] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,401] jepsen worker 33 - jepsen.util 663 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,402] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:42,402] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:42,403] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:13:42,424] jepsen worker 13 - jepsen.util 643 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,424] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,425] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:42,426] jepsen worker 34 - jepsen.util 754 :invoke :write [11 6] INFO [2022-03-13 17:13:42,430] jepsen worker 83 - jepsen.util 1883 :info :write [27 5] :RequestTimeoutException INFO [2022-03-13 17:13:42,431] jepsen worker 83 - jepsen.util 1973 :invoke :read [27 nil] INFO [2022-03-13 17:13:42,467] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,468] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,470] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,470] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 32] INFO [2022-03-13 17:13:42,477] jepsen worker 37 - jepsen.util 2017 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,477] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:42,496] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,496] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:42,500] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,501] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:13:42,512] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,512] jepsen worker 22 - jepsen.util 652 :invoke :write [7 5] INFO [2022-03-13 17:13:42,556] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,556] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,556] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,556] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,557] jepsen worker 75 - jepsen.util 435 :invoke :write [25 21] INFO [2022-03-13 17:13:42,558] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:42,559] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:13:42,559] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:42,560] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,560] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,560] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,561] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,561] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 8] INFO [2022-03-13 17:13:42,561] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,561] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:13:42,562] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,563] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,563] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,565] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,566] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,566] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:42,567] jepsen worker nemesis - couchbase.util Rebalance check count: 332 INFO [2022-03-13 17:13:42,567] jepsen worker nemesis - couchbase.util Rebalance stuck count: 232 INFO [2022-03-13 17:13:42,567] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,567] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,567] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:13:42,568] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:13:42,568] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,569] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,575] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,575] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,576] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:13:42,582] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,582] jepsen worker 83 - jepsen.util 1973 :ok :read [27 4] INFO [2022-03-13 17:13:42,582] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,583] jepsen worker 83 - jepsen.util 1973 :invoke :write [27 7] INFO [2022-03-13 17:13:42,584] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 17] INFO [2022-03-13 17:13:42,605] jepsen worker 70 - jepsen.util 700 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,606] jepsen worker 70 - jepsen.util 700 :invoke :write [23 25] INFO [2022-03-13 17:13:42,616] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,616] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,617] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] INFO [2022-03-13 17:13:42,618] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:13:42,631] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,631] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,631] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:13:42,632] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:13:42,639] jepsen worker 36 - jepsen.util 2196 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,640] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:13:42,646] jepsen worker 75 - jepsen.util 435 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,647] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:13:42,657] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,659] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,667] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,668] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:13:42,705] jepsen worker 3 - jepsen.util 1893 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,705] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,706] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,706] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,711] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,711] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:42,712] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,712] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:13:42,725] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,726] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:42,732] jepsen worker 80 - jepsen.util 710 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,732] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,732] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,732] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,733] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:42,734] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:42,735] jepsen worker 78 - jepsen.util 528 :invoke :write [26 23] INFO [2022-03-13 17:13:42,735] jepsen worker 52 - jepsen.util 592 :invoke :write [17 32] INFO [2022-03-13 17:13:42,767] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,767] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,783] jepsen worker 6 - jepsen.util 636 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,783] jepsen worker 6 - jepsen.util 636 :invoke :write [2 0] INFO [2022-03-13 17:13:42,801] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,802] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:13:42,806] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,807] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:13:42,807] jepsen worker 22 - jepsen.util 652 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,808] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:42,809] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,810] jepsen worker 22 - jepsen.util 652 :invoke :write [7 45] INFO [2022-03-13 17:13:42,839] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,839] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:13:42,876] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,877] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,879] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:42,880] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:13:42,883] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,884] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:42,908] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,908] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,909] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,909] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:42,909] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,910] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,940] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,940] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:13:42,944] jepsen worker 37 - jepsen.util 2017 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,944] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,944] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,945] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:42,947] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,947] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,948] jepsen worker 34 - jepsen.util 754 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,948] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:42,949] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:13:42,950] jepsen worker 34 - jepsen.util 754 :invoke :write [11 15] INFO [2022-03-13 17:13:42,950] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,951] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:13:42,951] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:42,952] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:42,957] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,958] jepsen worker 23 - jepsen.util 563 :invoke :write [7 16] INFO [2022-03-13 17:13:42,975] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,976] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:42,978] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:42,978] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:13:43,025] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,026] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,026] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,025] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,026] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:13:43,027] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,029] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:13:43,034] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,034] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,035] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,036] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,036] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,036] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:13:43,037] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:13:43,058] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,059] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:43,073] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,073] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,074] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:13:43,075] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:13:43,083] jepsen worker 54 - jepsen.util 2124 :ok :read [18 21] INFO [2022-03-13 17:13:43,088] jepsen worker 54 - jepsen.util 2124 :invoke :write [18 49] INFO [2022-03-13 17:13:43,087] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,089] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,092] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,093] jepsen worker 19 - jepsen.util 2359 :ok :read [6 16] INFO [2022-03-13 17:13:43,093] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:13:43,093] jepsen worker 19 - jepsen.util 2359 :invoke :read [6 nil] INFO [2022-03-13 17:13:43,105] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,106] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:43,125] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,125] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:13:43,146] jepsen worker 22 - jepsen.util 652 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,147] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:43,150] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,151] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:13:43,158] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,159] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:13:43,186] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,186] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,187] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,187] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,187] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,188] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:43,189] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:43,190] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:13:43,190] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:43,191] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,204] jepsen worker 0 - jepsen.util 2340 :info :write [0 16] :RequestTimeoutException INFO [2022-03-13 17:13:43,204] jepsen worker 0 - jepsen.util 2430 :invoke :read [0 nil] INFO [2022-03-13 17:13:43,209] jepsen worker 18 - jepsen.util 2268 :ok :read [6 16] INFO [2022-03-13 17:13:43,209] jepsen worker 18 - jepsen.util 2268 :invoke :write [6 12] INFO [2022-03-13 17:13:43,221] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,222] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:13:43,260] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,261] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,293] jepsen worker 19 - jepsen.util 2359 :ok :read [6 16] INFO [2022-03-13 17:13:43,293] jepsen worker 19 - jepsen.util 2359 :invoke :write [6 49] INFO [2022-03-13 17:13:43,302] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,302] jepsen worker 70 - jepsen.util 700 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,303] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,303] jepsen worker 70 - jepsen.util 700 :invoke :write [23 27] INFO [2022-03-13 17:13:43,303] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:43,304] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:43,321] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,322] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:13:43,322] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,323] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,324] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,330] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,331] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 4] INFO [2022-03-13 17:13:43,331] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,333] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,334] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,335] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,335] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,336] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,337] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,340] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,341] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:43,349] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,350] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:13:43,389] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,390] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,390] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 37] INFO [2022-03-13 17:13:43,394] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:13:43,414] jepsen worker 58 - jepsen.util 2218 :info :write [19 47] :RequestTimeoutException INFO [2022-03-13 17:13:43,415] jepsen worker 58 - jepsen.util 2308 :invoke :read [19 nil] INFO [2022-03-13 17:13:43,428] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,428] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,429] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:43,429] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,432] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,433] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,434] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,435] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:13:43,436] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,436] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,436] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,436] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,437] jepsen worker 33 - jepsen.util 663 :invoke :write [11 42] INFO [2022-03-13 17:13:43,437] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:13:43,438] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:43,439] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:43,441] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,441] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,442] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:43,442] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:13:43,443] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,444] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:43,445] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,446] jepsen worker 71 - jepsen.util 521 :invoke :write [23 40] INFO [2022-03-13 17:13:43,446] jepsen worker 16 - jepsen.util 2176 :info :write [5 39] :RequestTimeoutException INFO [2022-03-13 17:13:43,447] jepsen worker 16 - jepsen.util 2266 :invoke :write [5 20] INFO [2022-03-13 17:13:43,454] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,454] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,472] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,472] jepsen worker 21 - jepsen.util 831 :invoke :write [7 22] INFO [2022-03-13 17:13:43,514] jepsen worker 14 - jepsen.util 284 :info :write [4 4] :RequestTimeoutException INFO [2022-03-13 17:13:43,515] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,515] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:13:43,515] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,515] jepsen worker 25 - jepsen.util 565 :invoke :write [8 3] INFO [2022-03-13 17:13:43,516] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:43,518] jepsen worker 37 - jepsen.util 2017 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,519] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 4] INFO [2022-03-13 17:13:43,524] jepsen worker 61 - jepsen.util 601 :info :write [20 11] :RequestTimeoutException INFO [2022-03-13 17:13:43,524] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:43,548] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,550] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,552] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,552] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,552] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,552] jepsen worker 21 - jepsen.util 831 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,554] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,554] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:13:43,555] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,555] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:43,556] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,556] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:43,573] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:43,573] jepsen worker nemesis - couchbase.util Rebalance check count: 333 INFO [2022-03-13 17:13:43,573] jepsen worker nemesis - couchbase.util Rebalance stuck count: 233 INFO [2022-03-13 17:13:43,591] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,592] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:43,593] jepsen worker 78 - jepsen.util 528 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,593] jepsen worker 78 - jepsen.util 528 :invoke :write [26 3] INFO [2022-03-13 17:13:43,594] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,594] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:43,642] jepsen worker 89 - jepsen.util 2429 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,643] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,645] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,645] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:13:43,653] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,654] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:13:43,659] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,659] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,659] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,659] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,660] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 16] INFO [2022-03-13 17:13:43,660] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,661] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:13:43,661] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,667] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,668] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:43,673] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,676] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,684] jepsen worker 25 - jepsen.util 565 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,685] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:43,696] jepsen worker 33 - jepsen.util 663 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,696] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:43,697] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,697] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,699] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:13:43,700] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:13:43,707] jepsen worker 85 - jepsen.util 2425 :ok :read [28 44] INFO [2022-03-13 17:13:43,707] jepsen worker 85 - jepsen.util 2425 :invoke :read [28 nil] INFO [2022-03-13 17:13:43,738] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,739] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:13:43,755] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,755] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,755] jepsen worker 14 - jepsen.util 374 :invoke :write [4 4] INFO [2022-03-13 17:13:43,756] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:13:43,757] jepsen worker 52 - jepsen.util 592 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,758] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:43,770] jepsen worker 88 - jepsen.util 2068 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,771] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:43,771] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,772] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,773] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,774] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:43,789] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,789] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:13:43,804] jepsen worker 42 - jepsen.util 402 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,804] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,804] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:43,804] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,804] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,804] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,804] jepsen worker 26 - jepsen.util 476 :invoke :write [8 9] INFO [2022-03-13 17:13:43,805] jepsen worker 8 - jepsen.util 728 :invoke :write [2 1] INFO [2022-03-13 17:13:43,806] jepsen worker 53 - jepsen.util 503 :invoke :write [17 21] INFO [2022-03-13 17:13:43,806] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:43,806] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:43,807] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:43,819] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,819] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,820] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:13:43,821] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:13:43,834] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,835] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:43,856] jepsen worker 14 - jepsen.util 374 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,857] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:13:43,873] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,874] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:43,877] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,878] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:43,884] jepsen worker 37 - jepsen.util 2017 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,884] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 17] INFO [2022-03-13 17:13:43,885] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,885] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,885] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:43,886] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:13:43,886] jepsen worker 51 - jepsen.util 591 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,887] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,887] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,887] jepsen worker 51 - jepsen.util 591 :invoke :write [17 18] INFO [2022-03-13 17:13:43,892] jepsen worker 34 - jepsen.util 754 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,892] jepsen worker 76 - jepsen.util 706 :invoke :write [25 8] INFO [2022-03-13 17:13:43,892] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,893] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,893] jepsen worker 60 - jepsen.util 600 :invoke :write [20 24] INFO [2022-03-13 17:13:43,893] jepsen worker 80 - jepsen.util 710 :invoke :write [26 23] INFO [2022-03-13 17:13:43,907] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:43,909] jepsen worker 13 - jepsen.util 643 :invoke :write [4 5] INFO [2022-03-13 17:13:43,920] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,921] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:43,928] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,928] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:43,947] jepsen worker 81 - jepsen.util 2871 :info :write [27 49] :RequestTimeoutException INFO [2022-03-13 17:13:43,947] jepsen worker 81 - jepsen.util 2961 :invoke :read [27 nil] INFO [2022-03-13 17:13:43,957] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,957] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 22] INFO [2022-03-13 17:13:43,970] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,971] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:43,980] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,981] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:13:43,990] jepsen worker 71 - jepsen.util 521 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,990] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,990] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,990] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:43,991] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:13:43,992] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:13:43,992] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:13:43,993] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:44,007] jepsen worker 40 - jepsen.util 2560 :info :write [13 12] :RequestTimeoutException INFO [2022-03-13 17:13:44,008] jepsen worker 40 - jepsen.util 2650 :invoke :read [13 nil] INFO [2022-03-13 17:13:44,040] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,040] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:44,041] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,042] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:44,049] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,049] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:13:44,056] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,057] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:13:44,070] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,070] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,071] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,073] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,076] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,076] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,093] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,093] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,095] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,096] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,096] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,097] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,097] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,097] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,098] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 3] INFO [2022-03-13 17:13:44,098] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,099] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:13:44,100] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,103] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,104] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:44,157] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,158] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:44,168] jepsen worker 70 - jepsen.util 700 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,168] jepsen worker 69 - jepsen.util 699 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,168] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:44,169] jepsen worker 69 - jepsen.util 699 :invoke :write [23 32] INFO [2022-03-13 17:13:44,170] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:44,170] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:44,184] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,184] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,184] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:13:44,184] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,185] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,185] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,186] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,186] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:44,204] jepsen worker 89 - jepsen.util 2429 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,205] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:13:44,211] jepsen worker 86 - jepsen.util 2336 :info :write [28 14] :SyncWriteAmbiguous INFO [2022-03-13 17:13:44,212] jepsen worker 86 - jepsen.util 2426 :invoke :read [28 nil] INFO [2022-03-13 17:13:44,247] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,248] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:13:44,249] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,250] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:13:44,260] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,261] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:44,266] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,266] jepsen worker 88 - jepsen.util 2068 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,267] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,267] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 38] INFO [2022-03-13 17:13:44,268] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:13:44,269] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,286] jepsen worker 37 - jepsen.util 2017 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,286] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,287] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,287] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,294] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,294] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,295] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:13:44,296] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,297] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,298] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,300] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,301] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 33] INFO [2022-03-13 17:13:44,309] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,309] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,311] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,311] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:44,316] jepsen worker 23 - jepsen.util 563 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,316] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,316] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,317] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:44,317] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:44,317] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:44,323] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,323] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,323] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:13:44,323] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:13:44,345] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,346] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:13:44,365] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,365] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:44,398] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,399] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:13:44,404] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,404] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,405] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,405] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 27] INFO [2022-03-13 17:13:44,418] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,418] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:13:44,431] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,432] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,433] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:44,433] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:13:44,437] jepsen worker 62 - jepsen.util 512 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,438] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:44,440] jepsen worker 78 - jepsen.util 528 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,440] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,440] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,441] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:44,441] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:44,442] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:44,444] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,445] jepsen worker 22 - jepsen.util 652 :invoke :write [7 32] INFO [2022-03-13 17:13:44,458] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,458] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,460] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,460] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,461] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:44,461] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:44,464] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,464] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:13:44,515] jepsen worker 3 - jepsen.util 1893 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,515] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:13:44,522] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,523] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:13:44,544] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,545] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:44,548] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,549] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:44,563] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,564] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:44,579] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:44,579] jepsen worker nemesis - couchbase.util Rebalance check count: 334 INFO [2022-03-13 17:13:44,579] jepsen worker nemesis - couchbase.util Rebalance stuck count: 234 INFO [2022-03-13 17:13:44,592] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,593] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,593] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,594] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,594] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,594] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,594] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,595] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,595] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:44,596] jepsen worker 75 - jepsen.util 435 :invoke :write [25 20] INFO [2022-03-13 17:13:44,596] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:44,597] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:13:44,611] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,611] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,612] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:13:44,613] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:13:44,618] jepsen worker 1 - jepsen.util 2251 :info :write [0 18] :RequestTimeoutException INFO [2022-03-13 17:13:44,619] jepsen worker 1 - jepsen.util 2341 :invoke :read [0 nil] INFO [2022-03-13 17:13:44,621] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,621] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,622] jepsen worker 14 - jepsen.util 374 :invoke :write [4 5] INFO [2022-03-13 17:13:44,623] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:44,623] jepsen worker 80 - jepsen.util 710 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,625] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:44,636] jepsen worker 48 - jepsen.util 2388 :info :write [16 16] :RequestTimeoutException INFO [2022-03-13 17:13:44,637] jepsen worker 48 - jepsen.util 2478 :invoke :read [16 nil] INFO [2022-03-13 17:13:44,642] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,645] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,648] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,649] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,656] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,657] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:13:44,669] jepsen worker 37 - jepsen.util 2017 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,669] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:13:44,688] jepsen worker 6 - jepsen.util 636 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,688] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,688] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:44,688] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:44,688] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 13] INFO [2022-03-13 17:13:44,689] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:44,689] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:44,690] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:44,692] jepsen worker 75 - jepsen.util 435 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,692] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:13:44,692] jepsen worker 88 - jepsen.util 2068 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,693] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,714] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,715] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,719] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,719] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:44,723] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,724] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:44,752] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,753] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,759] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,760] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,762] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,763] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,762] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,762] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,763] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:13:44,763] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,764] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:13:44,765] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,765] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,765] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,766] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,766] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,766] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:44,767] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,767] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,768] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,768] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,769] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,769] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:44,770] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 19] INFO [2022-03-13 17:13:44,775] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,776] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:13:44,778] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,779] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:13:44,804] jepsen worker 51 - jepsen.util 591 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,805] jepsen worker 51 - jepsen.util 591 :invoke :write [17 31] INFO [2022-03-13 17:13:44,807] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,807] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,807] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:44,808] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:44,810] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,811] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:44,812] jepsen worker 13 - jepsen.util 643 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,812] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,812] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,813] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,813] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,813] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,814] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:44,814] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:44,815] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:44,816] jepsen worker 34 - jepsen.util 754 :invoke :write [11 42] INFO [2022-03-13 17:13:44,818] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:44,818] jepsen worker 78 - jepsen.util 528 :invoke :write [26 28] INFO [2022-03-13 17:13:44,843] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,846] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,852] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,852] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,855] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,855] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,859] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,860] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:13:44,865] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,865] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,868] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,869] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:44,891] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,892] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:44,928] jepsen worker 14 - jepsen.util 374 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,928] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,928] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,928] jepsen worker 8 - jepsen.util 728 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,929] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,929] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:44,930] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:44,930] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,931] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:13:44,931] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,932] jepsen worker 22 - jepsen.util 652 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,932] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,932] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:44,932] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,932] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,933] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:44,933] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,934] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,934] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:13:44,935] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:13:44,936] jepsen worker 70 - jepsen.util 700 :invoke :write [23 20] INFO [2022-03-13 17:13:44,937] jepsen worker 52 - jepsen.util 592 :invoke :write [17 20] INFO [2022-03-13 17:13:44,938] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,939] jepsen worker 21 - jepsen.util 831 :invoke :write [7 49] INFO [2022-03-13 17:13:44,939] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:44,942] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:44,946] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,946] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,946] jepsen worker 13 - jepsen.util 643 :invoke :write [4 47] INFO [2022-03-13 17:13:44,948] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:44,948] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:44,950] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,954] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:44,966] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:44,971] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:44,974] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,005] jepsen worker 76 - jepsen.util 706 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,005] jepsen worker 76 - jepsen.util 706 :invoke :write [25 0] INFO [2022-03-13 17:13:45,020] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,022] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:13:45,029] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,030] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,031] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:13:45,031] jepsen worker 78 - jepsen.util 528 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,033] jepsen worker 78 - jepsen.util 528 :invoke :write [26 30] INFO [2022-03-13 17:13:45,035] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:13:45,064] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,064] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,065] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:13:45,065] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:45,076] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,076] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,079] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,079] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,080] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,081] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:13:45,097] jepsen worker 3 - jepsen.util 1893 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,097] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,131] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,131] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,132] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,132] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,132] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:13:45,133] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,134] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:45,134] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:13:45,138] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,138] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,139] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,140] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:13:45,150] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,151] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:45,188] jepsen worker 15 - jepsen.util 2535 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:13:45,188] jepsen worker 15 - jepsen.util 2625 :invoke :write [5 23] INFO [2022-03-13 17:13:45,192] jepsen worker 26 - jepsen.util 476 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,193] jepsen worker 26 - jepsen.util 476 :invoke :write [8 20] INFO [2022-03-13 17:13:45,200] jepsen worker 34 - jepsen.util 754 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,200] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,200] jepsen worker 80 - jepsen.util 710 :invoke :write [26 12] INFO [2022-03-13 17:13:45,202] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:45,208] jepsen worker 21 - jepsen.util 831 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,208] jepsen worker 21 - jepsen.util 831 :invoke :write [7 35] INFO [2022-03-13 17:13:45,210] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,210] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,210] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,210] jepsen worker 67 - jepsen.util 2137 :info :write [22 38] :RequestTimeoutException INFO [2022-03-13 17:13:45,210] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,213] jepsen worker 67 - jepsen.util 2227 :invoke :write [22 42] INFO [2022-03-13 17:13:45,215] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:13:45,216] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:13:45,217] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,218] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,219] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,220] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,221] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:13:45,222] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,223] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,224] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,225] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,226] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,227] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,228] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,230] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,230] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,231] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,232] jepsen worker 60 - jepsen.util 600 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,231] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,234] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:13:45,235] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:45,236] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:13:45,239] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,240] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:13:45,243] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,244] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:45,310] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,311] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,313] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,313] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,316] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,317] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,318] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,318] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,320] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,321] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:13:45,336] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,336] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:13:45,337] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,338] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:13:45,343] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,343] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:13:45,353] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,353] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,354] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:13:45,354] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:45,365] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,366] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,381] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,382] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:45,385] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,386] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,419] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,420] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:13:45,437] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,438] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,461] jepsen worker 52 - jepsen.util 592 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,462] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:45,476] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,476] jepsen worker 25 - jepsen.util 565 :invoke :write [8 15] INFO [2022-03-13 17:13:45,484] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,484] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:13:45,499] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,499] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:45,525] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,525] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 23] INFO [2022-03-13 17:13:45,545] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,545] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,545] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:13:45,546] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,557] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,558] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,559] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,560] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:13:45,587] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:45,587] jepsen worker nemesis - couchbase.util Rebalance check count: 335 INFO [2022-03-13 17:13:45,587] jepsen worker nemesis - couchbase.util Rebalance stuck count: 235 INFO [2022-03-13 17:13:45,600] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,600] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,601] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,601] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:13:45,602] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:13:45,603] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,609] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,611] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,612] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,613] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,613] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 4] INFO [2022-03-13 17:13:45,613] jepsen worker 72 - jepsen.util 2502 :info :write [24 24] :RequestTimeoutException INFO [2022-03-13 17:13:45,614] jepsen worker 72 - jepsen.util 2592 :invoke :read [24 nil] INFO [2022-03-13 17:13:45,619] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:45,623] jepsen worker 46 - jepsen.util 2476 :info :write [15 25] :RequestTimeoutException INFO [2022-03-13 17:13:45,624] jepsen worker 46 - jepsen.util 2566 :invoke :read [15 nil] INFO [2022-03-13 17:13:45,634] jepsen worker 69 - jepsen.util 699 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,634] jepsen worker 53 - jepsen.util 503 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,634] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,634] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,635] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,635] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,635] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,635] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:45,635] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,635] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:45,635] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:45,635] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,636] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:45,637] jepsen worker 6 - jepsen.util 636 :invoke :write [2 37] INFO [2022-03-13 17:13:45,638] jepsen worker 43 - jepsen.util 493 :invoke :write [14 39] INFO [2022-03-13 17:13:45,638] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:45,639] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:45,641] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:45,641] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:45,642] jepsen worker 12 - jepsen.util 732 :invoke :write [4 46] INFO [2022-03-13 17:13:45,643] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:45,643] jepsen worker 69 - jepsen.util 699 :invoke :write [23 27] INFO [2022-03-13 17:13:45,644] jepsen worker 62 - jepsen.util 512 :invoke :write [20 49] INFO [2022-03-13 17:13:45,646] jepsen worker 61 - jepsen.util 691 :invoke :write [20 26] INFO [2022-03-13 17:13:45,653] jepsen worker 21 - jepsen.util 831 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,653] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,654] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:45,655] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:45,656] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,657] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:13:45,674] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,674] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:45,682] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,682] jepsen worker 3 - jepsen.util 1893 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,684] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:13:45,684] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,725] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,726] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,734] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,735] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:13:45,774] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,775] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,777] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,778] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,780] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,780] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:13:45,803] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,803] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,804] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,805] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,822] jepsen worker 79 - jepsen.util 1069 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,822] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,823] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 25] INFO [2022-03-13 17:13:45,823] jepsen worker 8 - jepsen.util 728 :invoke :write [2 29] INFO [2022-03-13 17:13:45,832] jepsen worker 70 - jepsen.util 700 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,832] jepsen worker 70 - jepsen.util 700 :invoke :write [23 37] INFO [2022-03-13 17:13:45,837] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,838] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,844] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,844] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:13:45,849] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,849] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,862] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,863] jepsen worker 71 - jepsen.util 521 :invoke :write [23 34] INFO [2022-03-13 17:13:45,868] jepsen worker 51 - jepsen.util 591 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,868] jepsen worker 51 - jepsen.util 591 :invoke :write [17 20] INFO [2022-03-13 17:13:45,883] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,883] jepsen worker 37 - jepsen.util 2017 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,884] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:13:45,884] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:13:45,886] jepsen worker 2 - jepsen.util 2882 :info :write [0 22] :RequestTimeoutException INFO [2022-03-13 17:13:45,886] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,886] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,887] jepsen worker 2 - jepsen.util 2972 :invoke :read [0 nil] INFO [2022-03-13 17:13:45,887] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:13:45,887] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:45,893] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,894] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:13:45,903] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,903] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:13:45,929] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,930] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:45,932] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,933] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:45,934] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,934] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,935] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:45,935] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,935] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:13:45,936] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:13:45,936] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,937] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:13:45,947] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,947] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:13:45,968] jepsen worker 26 - jepsen.util 476 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,968] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:45,968] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,968] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,969] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:45,970] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:13:45,970] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:45,971] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:45,976] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,976] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,976] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:45,977] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,978] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:45,978] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,005] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,005] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,005] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 39] INFO [2022-03-13 17:13:46,007] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:13:46,052] jepsen worker 25 - jepsen.util 565 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,052] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,052] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,052] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,052] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,052] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,053] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,053] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:13:46,053] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:13:46,054] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:13:46,055] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:13:46,055] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,056] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:13:46,062] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:13:46,066] jepsen worker 71 - jepsen.util 521 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,066] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,068] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,068] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:13:46,071] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:13:46,075] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,103] jepsen worker 76 - jepsen.util 706 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,103] jepsen worker 13 - jepsen.util 643 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,103] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:46,103] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,103] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,103] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,103] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:46,105] jepsen worker 13 - jepsen.util 643 :invoke :write [4 18] INFO [2022-03-13 17:13:46,105] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:46,107] jepsen worker 76 - jepsen.util 706 :invoke :write [25 21] INFO [2022-03-13 17:13:46,107] jepsen worker 77 - jepsen.util 707 :invoke :write [25 7] INFO [2022-03-13 17:13:46,109] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:13:46,137] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,137] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,139] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:13:46,140] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:13:46,143] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,143] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] WARN [2022-03-13 17:13:46,181] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":169743,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9741329,"total_server_duration_us":312137,"encode_duration_us":17386,"operation_id":"0x8ed9","timeout_ms":10000,"last_server_duration_us":43289,"total_duration_us":10016017},{"operation_name":"upsert","last_dispatch_duration_us":320525,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9627500,"total_server_duration_us":314520,"encode_duration_us":21448,"operation_id":"0x8ed0","timeout_ms":10000,"last_server_duration_us":41016,"total_duration_us":10010606},{"operation_name":"upsert","last_dispatch_duration_us":40239,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1364751,"total_server_duration_us":673149,"encode_duration_us":21059,"operation_id":"0x8d16","timeout_ms":10000,"last_server_duration_us":28493,"total_duration_us":10007604},{"operation_name":"upsert","last_dispatch_duration_us":31448,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1337806,"total_server_duration_us":689768,"encode_duration_us":38771,"operation_id":"0x8d12","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10007437},{"operation_name":"upsert","last_dispatch_duration_us":219449,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10003599,"total_server_duration_us":300932,"encode_duration_us":21552,"operation_id":"0x8ef3","timeout_ms":10000,"last_server_duration_us":29788,"total_duration_us":10003863},{"operation_name":"upsert","last_dispatch_duration_us":278663,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9622309,"total_server_duration_us":305702,"encode_duration_us":19079,"operation_id":"0x8ed2","timeout_ms":10000,"last_server_duration_us":34017,"total_duration_us":10002517},{"operation_name":"upsert","last_dispatch_duration_us":9053593,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9053593,"total_server_duration_us":9035244,"encode_duration_us":26738,"operation_id":"0x8ecc","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":9062352},{"operation_name":"get","last_dispatch_duration_us":7541502,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7541502,"total_server_duration_us":62,"operation_id":"0x8ffd","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":7541866},{"operation_name":"get","last_dispatch_duration_us":6699034,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6699034,"total_server_duration_us":15339,"operation_id":"0x907d","timeout_ms":30000,"last_server_duration_us":15339,"total_duration_us":6699348},{"operation_name":"upsert","last_dispatch_duration_us":1413324,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1413324,"total_server_duration_us":1303978,"encode_duration_us":20049,"operation_id":"0x9143","timeout_ms":10000,"last_server_duration_us":1303978,"total_duration_us":1417504}],"total_count":14}} INFO [2022-03-13 17:13:46,187] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,187] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,187] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:46,188] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:13:46,196] jepsen worker 45 - jepsen.util 2835 :info :write [15 14] :RequestTimeoutException INFO [2022-03-13 17:13:46,197] jepsen worker 45 - jepsen.util 2925 :invoke :read [15 nil] INFO [2022-03-13 17:13:46,198] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,198] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:46,213] jepsen worker 62 - jepsen.util 512 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,213] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,214] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:46,215] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,222] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,223] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,224] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,225] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:46,233] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,234] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:13:46,235] jepsen worker 61 - jepsen.util 691 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,235] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:46,275] jepsen worker 37 - jepsen.util 2017 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,276] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,282] jepsen worker 78 - jepsen.util 528 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,282] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,283] jepsen worker 78 - jepsen.util 528 :invoke :write [26 6] INFO [2022-03-13 17:13:46,284] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:46,294] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,294] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,301] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,303] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,305] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,306] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:13:46,335] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,336] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:13:46,344] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,345] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:46,349] jepsen worker 10 - jepsen.util 2620 :info :write [3 11] :SyncWriteAmbiguous INFO [2022-03-13 17:13:46,349] jepsen worker 10 - jepsen.util 2710 :invoke :read [3 nil] INFO [2022-03-13 17:13:46,350] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,351] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:13:46,381] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,382] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:13:46,390] jepsen worker 29 - jepsen.util 2639 :info :write [9 13] :RequestTimeoutException INFO [2022-03-13 17:13:46,391] jepsen worker 29 - jepsen.util 2729 :invoke :write [9 16] INFO [2022-03-13 17:13:46,396] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,396] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,398] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:13:46,398] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,400] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,401] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:13:46,426] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,426] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,426] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:46,427] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,427] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,428] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:13:46,428] jepsen worker 17 - jepsen.util 2537 :info :write [5 7] :RequestTimeoutException INFO [2022-03-13 17:13:46,430] jepsen worker 17 - jepsen.util 2627 :invoke :write [5 37] INFO [2022-03-13 17:13:46,442] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,442] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,443] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,444] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,445] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,446] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:13:46,464] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,464] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:46,465] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,465] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:46,471] jepsen worker 6 - jepsen.util 636 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,472] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:46,476] jepsen worker 69 - jepsen.util 699 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,476] jepsen worker 69 - jepsen.util 699 :invoke :write [23 37] INFO [2022-03-13 17:13:46,495] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,496] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:46,534] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,534] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,534] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:13:46,535] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,535] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 31] INFO [2022-03-13 17:13:46,535] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,536] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:46,536] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:13:46,541] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,541] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,542] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,542] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:13:46,543] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,543] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 20] INFO [2022-03-13 17:13:46,554] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,554] jepsen worker 80 - jepsen.util 710 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,555] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:46,556] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:46,586] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,587] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:13:46,592] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:46,593] jepsen worker nemesis - couchbase.util Rebalance check count: 336 INFO [2022-03-13 17:13:46,593] jepsen worker nemesis - couchbase.util Rebalance stuck count: 236 INFO [2022-03-13 17:13:46,604] jepsen worker 72 - jepsen.util 2592 :ok :read [24 24] INFO [2022-03-13 17:13:46,604] jepsen worker 43 - jepsen.util 493 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,605] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:46,605] jepsen worker 72 - jepsen.util 2592 :invoke :write [24 47] INFO [2022-03-13 17:13:46,609] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,611] jepsen worker 21 - jepsen.util 831 :invoke :write [7 25] INFO [2022-03-13 17:13:46,618] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,618] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,619] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,619] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:46,629] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:13:46,632] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:13:46,652] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,652] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,652] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,653] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:13:46,653] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:46,654] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:46,660] jepsen worker 12 - jepsen.util 732 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,661] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:46,696] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,696] jepsen worker 88 - jepsen.util 2068 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,697] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,697] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,702] jepsen worker 37 - jepsen.util 2017 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,703] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 17] INFO [2022-03-13 17:13:46,706] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,707] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,708] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,708] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:13:46,759] jepsen worker 70 - jepsen.util 700 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,760] jepsen worker 70 - jepsen.util 700 :invoke :write [23 32] INFO [2022-03-13 17:13:46,760] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,761] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,763] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,763] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,764] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:13:46,765] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:13:46,766] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,766] jepsen worker 14 - jepsen.util 374 :invoke :write [4 43] INFO [2022-03-13 17:13:46,769] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,770] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:13:46,793] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,794] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:13:46,799] jepsen worker 79 - jepsen.util 1069 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,800] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:46,800] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,800] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,800] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,801] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:13:46,801] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:46,802] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:46,831] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,832] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:46,832] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,833] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:13:46,837] jepsen worker 21 - jepsen.util 831 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,837] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,837] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:13:46,838] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:13:46,871] jepsen worker 51 - jepsen.util 591 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,872] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,872] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:46,872] jepsen worker 51 - jepsen.util 591 :invoke :write [17 11] INFO [2022-03-13 17:13:46,873] jepsen worker 60 - jepsen.util 600 :invoke :write [20 49] INFO [2022-03-13 17:13:46,873] jepsen worker 44 - jepsen.util 584 :invoke :write [14 14] INFO [2022-03-13 17:13:46,905] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,905] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,905] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,906] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:46,907] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:46,907] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,907] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] INFO [2022-03-13 17:13:46,907] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:46,911] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,912] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:46,955] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,956] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:46,959] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,960] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:13:46,976] jepsen worker 13 - jepsen.util 643 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,976] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,976] jepsen worker 13 - jepsen.util 643 :invoke :write [4 21] INFO [2022-03-13 17:13:46,977] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:46,993] jepsen worker 14 - jepsen.util 374 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:46,993] jepsen worker 14 - jepsen.util 374 :invoke :write [4 27] INFO [2022-03-13 17:13:47,000] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,000] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,001] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:13:47,001] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:13:47,034] jepsen worker 8 - jepsen.util 728 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,035] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:47,035] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,035] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,035] jepsen worker 8 - jepsen.util 728 :invoke :write [2 0] INFO [2022-03-13 17:13:47,037] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:13:47,037] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:47,038] jepsen worker 26 - jepsen.util 476 :invoke :write [8 27] INFO [2022-03-13 17:13:47,038] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,039] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:47,040] jepsen worker 37 - jepsen.util 2017 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,041] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:13:47,046] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,046] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:13:47,100] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,100] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,100] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,101] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:47,101] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,102] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:47,103] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,104] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,104] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,104] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:13:47,104] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:47,105] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:47,107] jepsen worker 76 - jepsen.util 706 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,108] jepsen worker 76 - jepsen.util 706 :invoke :write [25 32] INFO [2022-03-13 17:13:47,114] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,115] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,116] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,117] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,118] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,119] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,120] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,121] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:47,161] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,162] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:47,166] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,166] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,167] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:13:47,167] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:13:47,180] jepsen worker 31 - jepsen.util 2551 :info :write [10 39] :SyncWriteAmbiguous INFO [2022-03-13 17:13:47,182] jepsen worker 31 - jepsen.util 2641 :invoke :write [10 43] INFO [2022-03-13 17:13:47,188] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,189] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:47,200] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,201] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:13:47,214] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,214] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,215] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:13:47,217] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:13:47,224] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,225] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:13:47,248] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,248] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,248] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,249] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:13:47,249] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:13:47,251] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,251] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,252] jepsen worker 77 - jepsen.util 707 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,252] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,252] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:47,253] jepsen worker 62 - jepsen.util 512 :invoke :write [20 4] INFO [2022-03-13 17:13:47,253] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,253] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,253] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:47,254] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:47,255] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:47,255] jepsen worker 7 - jepsen.util 637 :invoke :write [2 9] INFO [2022-03-13 17:13:47,256] jepsen worker 53 - jepsen.util 503 :invoke :write [17 32] INFO [2022-03-13 17:13:47,258] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:13:47,258] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,258] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,258] jepsen worker 78 - jepsen.util 528 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,258] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:47,259] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:47,259] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:47,262] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:47,267] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,268] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:47,276] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,277] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:13:47,283] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,284] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:13:47,309] jepsen worker 56 - jepsen.util 2576 :info :write [18 48] :SyncWriteAmbiguous INFO [2022-03-13 17:13:47,310] jepsen worker 56 - jepsen.util 2666 :invoke :write [18 6] INFO [2022-03-13 17:13:47,323] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,324] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:13:47,339] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,340] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:47,348] jepsen worker 14 - jepsen.util 374 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,348] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,348] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,348] jepsen worker 14 - jepsen.util 374 :invoke :write [4 20] INFO [2022-03-13 17:13:47,351] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:47,352] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:47,370] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,371] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,375] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,376] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,377] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,377] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:13:47,399] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,400] jepsen worker 42 - jepsen.util 402 :invoke :write [14 7] INFO [2022-03-13 17:13:47,406] jepsen worker 9 - jepsen.util 2169 :ok :read [3 25] INFO [2022-03-13 17:13:47,406] jepsen worker 70 - jepsen.util 700 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,406] jepsen worker 9 - jepsen.util 2169 :invoke :read [3 nil] INFO [2022-03-13 17:13:47,406] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,407] jepsen worker 70 - jepsen.util 700 :invoke :write [23 22] INFO [2022-03-13 17:13:47,407] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:47,414] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,415] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,416] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,417] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:47,426] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,427] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:13:47,446] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,446] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:13:47,473] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,473] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,473] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,474] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:47,474] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:47,475] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:13:47,478] jepsen worker 69 - jepsen.util 699 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,478] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,478] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,479] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:47,479] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:47,480] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:47,480] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,481] jepsen worker 12 - jepsen.util 732 :invoke :write [4 4] INFO [2022-03-13 17:13:47,510] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,510] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,512] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,512] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,517] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,517] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,518] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,518] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:47,519] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,520] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:47,541] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,541] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,554] jepsen worker 13 - jepsen.util 643 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,554] jepsen worker 14 - jepsen.util 374 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,554] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,554] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,554] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,555] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,555] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,555] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,556] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:47,557] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:47,558] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:47,559] jepsen worker 80 - jepsen.util 710 :invoke :write [26 28] INFO [2022-03-13 17:13:47,559] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:47,560] jepsen worker 78 - jepsen.util 528 :invoke :write [26 41] INFO [2022-03-13 17:13:47,561] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:47,561] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:13:47,581] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,581] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:13:47,593] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,594] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,597] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,597] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,599] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:47,599] jepsen worker nemesis - couchbase.util Rebalance check count: 337 INFO [2022-03-13 17:13:47,599] jepsen worker nemesis - couchbase.util Rebalance stuck count: 237 INFO [2022-03-13 17:13:47,600] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,600] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,601] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,601] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,601] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,602] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,603] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,604] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:13:47,604] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,605] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,605] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,606] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:13:47,606] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,607] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:13:47,628] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,628] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,628] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,629] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:13:47,633] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:13:47,634] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:47,653] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,653] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:13:47,655] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,656] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,656] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:47,657] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:13:47,659] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,660] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:47,662] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,663] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,663] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,663] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,665] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,666] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,666] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:13:47,667] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,667] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,667] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,668] jepsen worker 11 - jepsen.util 2441 :info :write [3 11] :RequestTimeoutException INFO [2022-03-13 17:13:47,668] jepsen worker 11 - jepsen.util 2531 :invoke :read [3 nil] INFO [2022-03-13 17:13:47,669] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,669] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,672] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,673] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:13:47,676] jepsen worker 26 - jepsen.util 476 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,677] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,677] jepsen worker 26 - jepsen.util 476 :invoke :write [8 18] WARN [2022-03-13 17:13:47,678] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":36209171,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":36209171,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9029938,"encode_duration_us":20514,"operation_id":"0x7cd9","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10001463},{"operation_name":"get","last_dispatch_duration_us":35181981,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":35181981,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":38,"operation_id":"0x7d63","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":30006055}],"total_count":2}} INFO [2022-03-13 17:13:47,679] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,681] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,681] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:13:47,735] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,736] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:47,745] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,746] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,746] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,747] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:13:47,751] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,752] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] WARN [2022-03-13 17:13:47,782] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9146205,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10726892,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9263462,"encode_duration_us":29880,"operation_id":"0x8f8e","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10000848},{"operation_name":"upsert","last_dispatch_duration_us":10543693,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10543693,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9032060,"encode_duration_us":20352,"operation_id":"0x90fa","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10008007},{"operation_name":"upsert","last_dispatch_duration_us":9449122,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9449122,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":96711,"encode_duration_us":25347,"operation_id":"0x8fab","timeout_ms":10000,"last_server_duration_us":96711,"total_duration_us":10001853},{"operation_name":"upsert","last_dispatch_duration_us":8122523,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":8122523,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":46287,"encode_duration_us":22648,"operation_id":"0x8fac","timeout_ms":10000,"last_server_duration_us":46287,"total_duration_us":10010529},{"operation_name":"upsert","last_dispatch_duration_us":9050992,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10813839,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9167394,"encode_duration_us":19862,"operation_id":"0x90fb","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10008515}],"total_count":5}} INFO [2022-03-13 17:13:47,788] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,788] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,789] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] WARN [2022-03-13 17:13:47,790] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17950694,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19452747,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9655298,"encode_duration_us":23013,"operation_id":"0x85b2","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10004909},{"operation_name":"upsert","last_dispatch_duration_us":20748618,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20748618,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9036305,"encode_duration_us":19365,"operation_id":"0x86d0","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10008786},{"operation_name":"upsert","last_dispatch_duration_us":17893093,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19520382,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9641791,"encode_duration_us":18662,"operation_id":"0x850b","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10005567}],"total_count":3}} WARN [2022-03-13 17:13:47,790] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17227926,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":17227926,"total_server_duration_us":76,"operation_id":"0x8923","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":17230248},{"operation_name":"get","last_dispatch_duration_us":12867981,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":12867981,"total_server_duration_us":1644,"operation_id":"0x8b99","timeout_ms":30000,"last_server_duration_us":1644,"total_duration_us":12868312},{"operation_name":"upsert","last_dispatch_duration_us":148660,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9951443,"total_server_duration_us":159496,"encode_duration_us":17609,"operation_id":"0x8df9","timeout_ms":10000,"last_server_duration_us":24497,"total_duration_us":10012557},{"operation_name":"upsert","last_dispatch_duration_us":9028960,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9028960,"total_server_duration_us":19887,"encode_duration_us":36086,"operation_id":"0x906d","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10006009},{"operation_name":"upsert","last_dispatch_duration_us":11897761,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11897761,"total_server_duration_us":18953,"encode_duration_us":20013,"operation_id":"0x8ea6","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10004156},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":22137,"operation_id":"0x911f","timeout_ms":10000,"total_duration_us":10002103},{"operation_name":"get","last_dispatch_duration_us":9133600,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9133600,"total_server_duration_us":135,"operation_id":"0x8ddc","timeout_ms":30000,"last_server_duration_us":135,"total_duration_us":9133788},{"operation_name":"get","last_dispatch_duration_us":8999982,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8999982,"total_server_duration_us":8557,"operation_id":"0x9072","timeout_ms":30000,"last_server_duration_us":8557,"total_duration_us":9002523},{"operation_name":"upsert","last_dispatch_duration_us":252251,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":719567,"total_server_duration_us":182400,"encode_duration_us":18509,"operation_id":"0x9261","timeout_ms":10000,"last_server_duration_us":74924,"total_duration_us":749472},{"operation_name":"upsert","last_dispatch_duration_us":258236,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":503963,"total_server_duration_us":138724,"encode_duration_us":24315,"operation_id":"0x9267","timeout_ms":10000,"last_server_duration_us":49140,"total_duration_us":528408}],"total_count":11}} INFO [2022-03-13 17:13:47,791] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:47,796] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,796] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,797] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,797] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:47,797] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:47,798] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:13:47,812] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,813] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:13:47,821] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,821] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,823] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:13:47,823] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:13:47,851] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,852] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:13:47,863] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,864] jepsen worker 23 - jepsen.util 563 :invoke :write [7 6] INFO [2022-03-13 17:13:47,865] jepsen worker 70 - jepsen.util 700 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,865] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,865] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,865] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,867] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:47,868] jepsen worker 52 - jepsen.util 592 :invoke :write [17 24] WARN [2022-03-13 17:13:47,874] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":69297,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":665311,"total_server_duration_us":345147,"encode_duration_us":18783,"operation_id":"0x90b4","timeout_ms":10000,"last_server_duration_us":44389,"total_duration_us":10002821},{"operation_name":"upsert","last_dispatch_duration_us":7945539,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10312194,"total_server_duration_us":400475,"encode_duration_us":20484,"operation_id":"0x8cde","timeout_ms":10000,"last_server_duration_us":67617,"total_duration_us":10002187},{"operation_name":"upsert","last_dispatch_duration_us":9104411,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9165942,"total_server_duration_us":9104490,"encode_duration_us":17950,"operation_id":"0x8e85","timeout_ms":10000,"last_server_duration_us":9068168,"total_duration_us":9184954},{"operation_name":"get","last_dispatch_duration_us":4035720,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":4035720,"total_server_duration_us":657,"operation_id":"0x91b7","timeout_ms":30000,"last_server_duration_us":657,"total_duration_us":4035984},{"operation_name":"upsert","last_dispatch_duration_us":324089,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":782723,"total_server_duration_us":162517,"encode_duration_us":23010,"operation_id":"0x9252","timeout_ms":10000,"last_server_duration_us":61774,"total_duration_us":806533},{"operation_name":"upsert","last_dispatch_duration_us":312004,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":780401,"total_server_duration_us":144022,"encode_duration_us":17241,"operation_id":"0x925b","timeout_ms":10000,"last_server_duration_us":54576,"total_duration_us":805531},{"operation_name":"upsert","last_dispatch_duration_us":571260,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":571260,"total_server_duration_us":100753,"encode_duration_us":21695,"operation_id":"0x9733","timeout_ms":10000,"last_server_duration_us":100753,"total_duration_us":575372},{"operation_name":"upsert","last_dispatch_duration_us":221039,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":535946,"total_server_duration_us":153342,"encode_duration_us":28849,"operation_id":"0x925e","timeout_ms":10000,"last_server_duration_us":63310,"total_duration_us":552686}],"total_count":8}} INFO [2022-03-13 17:13:47,876] jepsen worker 34 - jepsen.util 754 :invoke :write [11 14] INFO [2022-03-13 17:13:47,876] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] WARN [2022-03-13 17:13:47,878] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":24162434,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":24162434,"total_server_duration_us":43,"operation_id":"0x86d6","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":24166781},{"operation_name":"get","last_dispatch_duration_us":19678854,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":19678854,"total_server_duration_us":9919,"operation_id":"0x89bb","timeout_ms":30000,"last_server_duration_us":9919,"total_duration_us":19683611},{"operation_name":"get","last_dispatch_duration_us":16327249,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16327249,"total_server_duration_us":49,"operation_id":"0x8b9f","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":16329590},{"operation_name":"upsert","last_dispatch_duration_us":58018,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7817153,"total_server_duration_us":458558,"encode_duration_us":27703,"operation_id":"0x9000","timeout_ms":10000,"last_server_duration_us":42525,"total_duration_us":10007446},{"operation_name":"upsert","last_dispatch_duration_us":141884,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7451496,"total_server_duration_us":100925,"encode_duration_us":17303,"operation_id":"0x9149","timeout_ms":10000,"last_server_duration_us":26864,"total_duration_us":10001683},{"operation_name":"get","last_dispatch_duration_us":5971109,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5971109,"total_server_duration_us":24669,"operation_id":"0x9074","timeout_ms":30000,"last_server_duration_us":24669,"total_duration_us":5971426},{"operation_name":"get","last_dispatch_duration_us":5410015,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5410015,"total_server_duration_us":34017,"operation_id":"0x90bb","timeout_ms":30000,"last_server_duration_us":34017,"total_duration_us":5410269},{"operation_name":"upsert","last_dispatch_duration_us":1063349,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1063349,"total_server_duration_us":105664,"encode_duration_us":18649,"operation_id":"0x915c","timeout_ms":10000,"last_server_duration_us":105664,"total_duration_us":2659643},{"operation_name":"upsert","last_dispatch_duration_us":1947276,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1947276,"total_server_duration_us":109371,"encode_duration_us":49094,"operation_id":"0x9164","timeout_ms":10000,"last_server_duration_us":109371,"total_duration_us":2122141},{"operation_name":"upsert","last_dispatch_duration_us":1935015,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1935015,"total_server_duration_us":77150,"encode_duration_us":18805,"operation_id":"0x9159","timeout_ms":10000,"last_server_duration_us":77150,"total_duration_us":1954766}],"total_count":66}} WARN [2022-03-13 17:13:47,878] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18582381,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18582381,"total_server_duration_us":510204,"operation_id":"0x8647","timeout_ms":30000,"last_server_duration_us":510204,"total_duration_us":20181128},{"operation_name":"get","last_dispatch_duration_us":11758903,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11758903,"total_server_duration_us":27,"operation_id":"0x8b45","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":11765800},{"operation_name":"upsert","last_dispatch_duration_us":8628385,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10035450,"total_server_duration_us":225507,"encode_duration_us":28056,"operation_id":"0x900b","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10010134},{"operation_name":"upsert","last_dispatch_duration_us":33021,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9629888,"total_server_duration_us":229282,"encode_duration_us":18404,"operation_id":"0x90b2","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10007755},{"operation_name":"upsert","last_dispatch_duration_us":35402,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8710081,"total_server_duration_us":260425,"encode_duration_us":19200,"operation_id":"0x9171","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10007407},{"operation_name":"upsert","last_dispatch_duration_us":47073,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9876577,"total_server_duration_us":205786,"encode_duration_us":38109,"operation_id":"0x9063","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10005097},{"operation_name":"upsert","last_dispatch_duration_us":151946,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":928202,"total_server_duration_us":78933,"encode_duration_us":21401,"operation_id":"0x921e","timeout_ms":10000,"last_server_duration_us":31108,"total_duration_us":10004703},{"operation_name":"upsert","last_dispatch_duration_us":8409623,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12436286,"total_server_duration_us":222086,"encode_duration_us":20924,"operation_id":"0x8e82","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10001889},{"operation_name":"upsert","last_dispatch_duration_us":9565203,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9565203,"total_server_duration_us":9053293,"encode_duration_us":16774,"operation_id":"0x9214","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9571253},{"operation_name":"upsert","last_dispatch_duration_us":9314574,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9553497,"total_server_duration_us":9066446,"encode_duration_us":20493,"operation_id":"0x906e","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9566720}],"total_count":64}} WARN [2022-03-13 17:13:47,882] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9437380,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9437380,"total_server_duration_us":15,"operation_id":"0x8f37","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":11037721},{"operation_name":"upsert","last_dispatch_duration_us":8122523,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":8122523,"total_server_duration_us":46287,"encode_duration_us":22648,"operation_id":"0x8fac","timeout_ms":10000,"last_server_duration_us":46287,"total_duration_us":10010529},{"operation_name":"upsert","last_dispatch_duration_us":9050992,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10813839,"total_server_duration_us":9167394,"encode_duration_us":19862,"operation_id":"0x90fb","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10008515},{"operation_name":"upsert","last_dispatch_duration_us":10543693,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10543693,"total_server_duration_us":9032060,"encode_duration_us":20352,"operation_id":"0x90fa","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10008007},{"operation_name":"upsert","last_dispatch_duration_us":7741010,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9751874,"total_server_duration_us":418501,"encode_duration_us":23368,"operation_id":"0x8f93","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10007217},{"operation_name":"upsert","last_dispatch_duration_us":8233512,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9951375,"total_server_duration_us":331418,"encode_duration_us":17437,"operation_id":"0x8f75","timeout_ms":10000,"last_server_duration_us":31393,"total_duration_us":10005752},{"operation_name":"upsert","last_dispatch_duration_us":9449122,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9449122,"total_server_duration_us":96711,"encode_duration_us":25347,"operation_id":"0x8fab","timeout_ms":10000,"last_server_duration_us":96711,"total_duration_us":10001853},{"operation_name":"upsert","last_dispatch_duration_us":9146205,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10726892,"total_server_duration_us":9263462,"encode_duration_us":29880,"operation_id":"0x8f8e","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10000848},{"operation_name":"get","last_dispatch_duration_us":9390900,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9390900,"total_server_duration_us":15,"operation_id":"0x8fec","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":9957661},{"operation_name":"upsert","last_dispatch_duration_us":9817847,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9817847,"total_server_duration_us":44833,"encode_duration_us":27871,"operation_id":"0x8ff4","timeout_ms":10000,"last_server_duration_us":44833,"total_duration_us":9932722}],"total_count":48}} INFO [2022-03-13 17:13:47,906] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,906] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,907] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,907] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:13:47,916] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,916] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:47,921] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,922] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,922] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,922] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,923] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 40] INFO [2022-03-13 17:13:47,924] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:47,924] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:13:47,926] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:47,954] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,954] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,962] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,963] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,964] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,965] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:47,966] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,967] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:13:47,984] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,985] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,989] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,990] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,991] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,991] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:47,991] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,992] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:47,994] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:47,995] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:48,016] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,017] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:48,017] jepsen worker 8 - jepsen.util 728 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,017] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,018] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,018] jepsen worker 22 - jepsen.util 652 :invoke :write [7 27] INFO [2022-03-13 17:13:48,018] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,019] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:48,020] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:48,021] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:48,021] jepsen worker 8 - jepsen.util 728 :invoke :write [2 43] INFO [2022-03-13 17:13:48,023] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,024] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:13:48,026] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,026] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:48,027] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:48,037] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,037] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,037] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,038] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:13:48,039] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:13:48,039] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:13:48,049] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,049] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:13:48,078] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9065786,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11223006,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9556847,"encode_duration_us":18111,"operation_id":"0x8ab3","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10003444},{"operation_name":"upsert","last_dispatch_duration_us":26268007,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26268007,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9053293,"encode_duration_us":21575,"operation_id":"0x85a1","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10006972}],"total_count":2}} INFO [2022-03-13 17:13:48,090] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,090] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,135] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,091] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,135] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,135] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:13:48,138] jepsen worker 57 - jepsen.util 2487 :ok :write [19 44] INFO [2022-03-13 17:13:48,139] jepsen worker 57 - jepsen.util 2487 :invoke :read [19 nil] INFO [2022-03-13 17:13:48,153] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,153] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,154] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,154] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,154] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,154] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:13:48,155] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,155] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,159] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,159] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,159] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,160] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,161] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,161] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,161] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:13:48,161] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,162] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,163] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,163] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,164] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,166] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,167] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,167] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,167] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,168] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,168] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,169] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,170] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:13:48,171] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,171] jepsen worker 0 - jepsen.util 2430 :ok :read [0 41] INFO [2022-03-13 17:13:48,171] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,172] jepsen worker 0 - jepsen.util 2430 :invoke :read [0 nil] INFO [2022-03-13 17:13:48,172] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,173] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:13:48,177] jepsen worker 73 - jepsen.util 2683 :info :write [24 7] :RequestTimeoutException INFO [2022-03-13 17:13:48,178] jepsen worker 73 - jepsen.util 2773 :invoke :write [24 42] INFO [2022-03-13 17:13:48,190] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,190] jepsen worker 37 - jepsen.util 2017 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,191] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:13:48,192] jepsen worker 51 - jepsen.util 591 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,192] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,192] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,192] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,192] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:48,193] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:13:48,194] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:48,195] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:48,209] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,210] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:48,210] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:48,217] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,218] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:13:48,226] jepsen worker 40 - jepsen.util 2650 :ok :read [13 39] INFO [2022-03-13 17:13:48,227] jepsen worker 40 - jepsen.util 2650 :invoke :write [13 44] INFO [2022-03-13 17:13:48,256] jepsen worker 57 - jepsen.util 2487 :ok :read [19 44] INFO [2022-03-13 17:13:48,257] jepsen worker 57 - jepsen.util 2487 :invoke :read [19 nil] INFO [2022-03-13 17:13:48,259] jepsen worker 80 - jepsen.util 710 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,260] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:48,263] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,264] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,264] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,264] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:48,265] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:48,266] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 30] INFO [2022-03-13 17:13:48,269] jepsen worker 0 - jepsen.util 2430 :ok :read [0 41] INFO [2022-03-13 17:13:48,270] jepsen worker 0 - jepsen.util 2430 :invoke :write [0 33] INFO [2022-03-13 17:13:48,294] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,295] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:13:48,297] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,298] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:13:48,302] jepsen worker 57 - jepsen.util 2487 :ok :read [19 44] INFO [2022-03-13 17:13:48,302] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,302] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,303] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,303] jepsen worker 57 - jepsen.util 2487 :invoke :read [19 nil] INFO [2022-03-13 17:13:48,303] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:13:48,307] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:48,307] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:48,322] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,323] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,338] jepsen worker 28 - jepsen.util 2548 :info :write [9 9] :RequestTimeoutException INFO [2022-03-13 17:13:48,339] jepsen worker 28 - jepsen.util 2638 :invoke :write [9 22] INFO [2022-03-13 17:13:48,356] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,356] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:48,368] jepsen worker 57 - jepsen.util 2487 :ok :read [19 44] INFO [2022-03-13 17:13:48,368] jepsen worker 57 - jepsen.util 2487 :invoke :read [19 nil] INFO [2022-03-13 17:13:48,371] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,372] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:48,396] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,397] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:13:48,427] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,427] jepsen worker 74 - jepsen.util 2684 :info :write [24 22] :RequestTimeoutException INFO [2022-03-13 17:13:48,427] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:48,429] jepsen worker 74 - jepsen.util 2774 :invoke :read [24 nil] INFO [2022-03-13 17:13:48,429] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,430] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:48,432] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,432] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:13:48,450] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,450] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,451] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:13:48,451] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:48,450] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,452] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:48,460] jepsen worker 78 - jepsen.util 528 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,461] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:48,463] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,463] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,463] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:48,464] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:48,472] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,473] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,473] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,473] jepsen worker 37 - jepsen.util 2017 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,473] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,474] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 12] INFO [2022-03-13 17:13:48,475] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:13:48,475] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:13:48,488] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,489] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,489] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:48,490] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:13:48,494] jepsen worker 57 - jepsen.util 2487 :ok :read [19 44] INFO [2022-03-13 17:13:48,494] jepsen worker 57 - jepsen.util 2487 :invoke :write [19 47] INFO [2022-03-13 17:13:48,566] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,566] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,566] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,567] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:13:48,573] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,574] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,576] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,576] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,577] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,578] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,579] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,579] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 30] INFO [2022-03-13 17:13:48,580] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,580] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:48,588] jepsen worker 22 - jepsen.util 652 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,588] jepsen worker 62 - jepsen.util 512 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,588] jepsen worker 60 - jepsen.util 600 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,588] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:13:48,589] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:48,589] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:48,605] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:48,606] jepsen worker nemesis - couchbase.util Rebalance check count: 338 INFO [2022-03-13 17:13:48,606] jepsen worker nemesis - couchbase.util Rebalance stuck count: 238 INFO [2022-03-13 17:13:48,644] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,645] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,645] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,645] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,645] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,646] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,646] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,646] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:48,646] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:13:48,647] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:13:48,647] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:48,648] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:48,648] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:48,649] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:13:48,649] jepsen worker 37 - jepsen.util 2017 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,650] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:13:48,655] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,655] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,656] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:13:48,658] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:48,671] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,672] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:48,679] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,680] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:48,688] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,689] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,712] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,712] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:48,720] jepsen worker 53 - jepsen.util 503 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,720] jepsen worker 34 - jepsen.util 754 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,722] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:48,722] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:48,734] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,735] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:13:48,762] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,763] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,764] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:13:48,764] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,764] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,764] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:48,765] jepsen worker 21 - jepsen.util 831 :invoke :write [7 36] INFO [2022-03-13 17:13:48,766] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:48,779] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,779] jepsen worker 76 - jepsen.util 706 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,779] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:48,780] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:13:48,796] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,796] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,797] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:13:48,797] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:13:48,799] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,800] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:13:48,812] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,813] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:48,845] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,846] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:48,879] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,879] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,880] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,880] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:48,880] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:48,881] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:13:48,890] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,891] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:48,892] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,893] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:13:48,902] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,904] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 37] INFO [2022-03-13 17:13:48,904] jepsen worker 3 - jepsen.util 1893 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,905] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:48,905] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,906] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:48,934] jepsen worker 27 - jepsen.util 2997 :info :write [9 47] :RequestTimeoutException INFO [2022-03-13 17:13:48,935] jepsen worker 27 - jepsen.util 3087 :invoke :write [9 5] INFO [2022-03-13 17:13:48,968] jepsen worker 7 - jepsen.util 637 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,968] jepsen worker 7 - jepsen.util 637 :invoke :write [2 20] INFO [2022-03-13 17:13:48,976] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,976] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:13:48,997] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,997] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:48,999] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:48,999] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:49,020] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,021] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:13:49,025] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,027] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:13:49,068] jepsen worker 52 - jepsen.util 592 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,068] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,068] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,068] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,068] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,069] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,069] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,069] jepsen worker 52 - jepsen.util 592 :invoke :write [17 34] INFO [2022-03-13 17:13:49,071] jepsen worker 34 - jepsen.util 754 :invoke :write [11 18] INFO [2022-03-13 17:13:49,071] jepsen worker 22 - jepsen.util 652 :invoke :write [7 28] INFO [2022-03-13 17:13:49,078] jepsen worker 80 - jepsen.util 710 :invoke :write [26 11] INFO [2022-03-13 17:13:49,078] jepsen worker 78 - jepsen.util 528 :invoke :write [26 13] INFO [2022-03-13 17:13:49,078] jepsen worker 70 - jepsen.util 700 :invoke :write [23 38] INFO [2022-03-13 17:13:49,080] jepsen worker 13 - jepsen.util 643 :invoke :write [4 15] INFO [2022-03-13 17:13:49,084] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,084] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,085] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,085] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:13:49,086] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:49,087] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:49,089] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,090] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:13:49,095] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,096] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:13:49,116] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,117] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,117] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,118] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,118] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:13:49,119] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:49,127] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,127] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,128] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:13:49,129] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:13:49,173] jepsen worker 37 - jepsen.util 2017 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,173] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,207] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,207] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,215] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,216] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,221] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,221] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,222] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,223] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:13:49,225] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,226] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:13:49,264] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,264] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:49,269] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,269] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:49,274] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,275] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:13:49,312] jepsen worker 21 - jepsen.util 831 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,312] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,312] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,313] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:49,314] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:13:49,316] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:49,320] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,321] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:13:49,322] jepsen worker 80 - jepsen.util 710 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,323] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:49,337] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,339] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:13:49,339] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,339] jepsen worker 44 - jepsen.util 584 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,340] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,341] jepsen worker 44 - jepsen.util 584 :invoke :write [14 10] INFO [2022-03-13 17:13:49,341] jepsen worker 69 - jepsen.util 699 :invoke :write [23 1] INFO [2022-03-13 17:13:49,342] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:49,342] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,342] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,346] jepsen worker 47 - jepsen.util 2477 :info :write [15 12] :RequestTimeoutException INFO [2022-03-13 17:13:49,346] jepsen worker 47 - jepsen.util 2567 :invoke :read [15 nil] INFO [2022-03-13 17:13:49,350] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,351] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,351] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,352] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:13:49,359] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,360] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:13:49,392] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,392] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,392] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,393] jepsen worker 14 - jepsen.util 374 :invoke :write [4 0] INFO [2022-03-13 17:13:49,394] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:49,394] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:13:49,400] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,401] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,404] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,404] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,405] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,405] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:49,405] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,406] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:49,407] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:13:49,407] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:13:49,433] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,433] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:13:49,451] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,452] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:49,456] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,457] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:13:49,487] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,488] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,489] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,490] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:49,493] jepsen worker 42 - jepsen.util 402 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,494] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:49,499] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,500] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:13:49,510] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,510] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,511] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:49,511] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:49,519] jepsen worker 70 - jepsen.util 700 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,520] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:49,524] jepsen worker 22 - jepsen.util 652 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,524] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:49,526] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,526] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,527] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,527] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:49,527] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:49,528] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:13:49,528] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,529] jepsen worker 22 - jepsen.util 652 :invoke :write [7 12] INFO [2022-03-13 17:13:49,548] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,550] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 2] INFO [2022-03-13 17:13:49,565] jepsen worker 12 - jepsen.util 732 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,566] jepsen worker 12 - jepsen.util 732 :invoke :write [4 4] INFO [2022-03-13 17:13:49,567] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,568] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,574] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,575] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,577] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,577] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:13:49,600] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,601] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:13:49,611] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:49,611] jepsen worker nemesis - couchbase.util Rebalance check count: 339 INFO [2022-03-13 17:13:49,612] jepsen worker nemesis - couchbase.util Rebalance stuck count: 239 INFO [2022-03-13 17:13:49,627] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,627] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,627] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,628] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:49,629] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:49,629] jepsen worker 33 - jepsen.util 663 :invoke :write [11 42] INFO [2022-03-13 17:13:49,653] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,653] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:13:49,671] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,671] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:49,677] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,677] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:13:49,683] jepsen worker 78 - jepsen.util 528 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,684] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:49,693] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,693] jepsen worker 14 - jepsen.util 374 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,695] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:13:49,695] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:13:49,722] jepsen worker 26 - jepsen.util 476 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,722] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,723] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,723] jepsen worker 23 - jepsen.util 563 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,723] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,723] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:49,725] jepsen worker 23 - jepsen.util 563 :invoke :write [7 19] INFO [2022-03-13 17:13:49,726] jepsen worker 60 - jepsen.util 600 :invoke :write [20 27] INFO [2022-03-13 17:13:49,727] jepsen worker 76 - jepsen.util 706 :invoke :write [25 36] INFO [2022-03-13 17:13:49,729] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:49,738] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,739] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:49,756] jepsen worker 64 - jepsen.util 2494 :info :write [21 14] :RequestTimeoutException INFO [2022-03-13 17:13:49,757] jepsen worker 64 - jepsen.util 2584 :invoke :read [21 nil] INFO [2022-03-13 17:13:49,759] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,760] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:49,762] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,763] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,763] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:13:49,763] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:13:49,776] jepsen worker 30 - jepsen.util 2460 :info :write [10 13] :RequestTimeoutException INFO [2022-03-13 17:13:49,776] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,776] jepsen worker 30 - jepsen.util 2550 :invoke :read [10 nil] INFO [2022-03-13 17:13:49,777] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,778] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,779] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:13:49,794] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,795] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,796] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,796] jepsen worker 65 - jepsen.util 2855 :info :write [21 15] :RequestTimeoutException INFO [2022-03-13 17:13:49,797] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,797] jepsen worker 65 - jepsen.util 2945 :invoke :write [21 11] INFO [2022-03-13 17:13:49,798] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,798] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:13:49,819] jepsen worker 22 - jepsen.util 652 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,820] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:13:49,823] jepsen worker 55 - jepsen.util 2395 :info :write [18 29] :RequestTimeoutException INFO [2022-03-13 17:13:49,824] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,825] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,825] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,826] jepsen worker 55 - jepsen.util 2485 :invoke :read [18 nil] INFO [2022-03-13 17:13:49,826] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,828] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:13:49,831] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:13:49,835] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,835] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:13:49,837] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,838] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,839] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,839] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,844] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,845] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,846] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,846] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:13:49,848] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,848] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:13:49,857] jepsen worker 8 - jepsen.util 728 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,857] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,857] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,857] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,859] jepsen worker 62 - jepsen.util 512 :invoke :write [20 6] INFO [2022-03-13 17:13:49,859] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:13:49,860] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:49,860] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:49,872] jepsen worker 37 - jepsen.util 2017 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,873] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 23] INFO [2022-03-13 17:13:49,901] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,902] jepsen worker 33 - jepsen.util 663 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,902] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:13:49,903] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:49,955] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,955] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,956] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:49,956] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:49,957] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,957] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:13:49,977] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:49,977] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,003] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,003] jepsen worker 13 - jepsen.util 643 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,004] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,004] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:50,005] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,006] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:50,016] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,016] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,017] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:50,018] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:50,024] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,025] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:13:50,041] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,041] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:13:50,060] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,060] jepsen worker 37 - jepsen.util 2017 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,061] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:13:50,061] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,063] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,064] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,067] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,068] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,070] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,071] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 11] INFO [2022-03-13 17:13:50,094] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,095] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:50,104] jepsen worker 34 - jepsen.util 754 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,105] jepsen worker 34 - jepsen.util 754 :invoke :write [11 17] INFO [2022-03-13 17:13:50,110] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,111] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:50,115] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,116] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:50,137] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,138] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:13:50,171] jepsen worker 7 - jepsen.util 637 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,171] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,172] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,172] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,172] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,172] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,171] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:50,172] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:50,172] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:50,173] jepsen worker 26 - jepsen.util 476 :invoke :write [8 28] INFO [2022-03-13 17:13:50,174] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 21] INFO [2022-03-13 17:13:50,174] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:50,175] jepsen worker 8 - jepsen.util 728 :invoke :write [2 0] INFO [2022-03-13 17:13:50,179] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:50,181] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:50,181] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,181] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:50,182] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,182] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:50,183] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,188] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,189] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:13:50,198] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,198] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,198] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,199] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,199] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,200] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:50,209] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,210] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:13:50,250] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,250] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,251] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:13:50,252] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:13:50,252] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,252] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,253] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:50,255] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:50,282] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,282] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,282] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,282] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,284] jepsen worker 44 - jepsen.util 584 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,284] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:13:50,284] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:13:50,285] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:13:50,288] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:50,289] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,291] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,291] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 20] INFO [2022-03-13 17:13:50,316] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,318] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:13:50,322] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,322] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,323] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,324] jepsen worker 52 - jepsen.util 592 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,324] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,324] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,324] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,324] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,324] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:13:50,326] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:13:50,326] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:13:50,328] jepsen worker 13 - jepsen.util 643 :invoke :write [4 39] INFO [2022-03-13 17:13:50,329] jepsen worker 80 - jepsen.util 710 :invoke :write [26 15] INFO [2022-03-13 17:13:50,329] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:50,330] jepsen worker 78 - jepsen.util 528 :invoke :write [26 33] INFO [2022-03-13 17:13:50,334] jepsen worker 37 - jepsen.util 2017 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,335] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:13:50,335] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:50,383] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,384] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,385] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,385] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,386] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,387] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,390] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,391] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,392] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,392] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:13:50,396] jepsen worker 62 - jepsen.util 512 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,397] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,397] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,397] jepsen worker 62 - jepsen.util 512 :invoke :write [20 33] INFO [2022-03-13 17:13:50,397] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:50,397] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:50,397] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,398] jepsen worker 53 - jepsen.util 503 :invoke :write [17 47] INFO [2022-03-13 17:13:50,399] jepsen worker 6 - jepsen.util 636 :invoke :write [2 41] INFO [2022-03-13 17:13:50,400] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:50,400] jepsen worker 43 - jepsen.util 493 :invoke :write [14 19] INFO [2022-03-13 17:13:50,401] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:50,409] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,410] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:50,434] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,435] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:13:50,435] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,436] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:13:50,442] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,443] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,443] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:13:50,444] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:13:50,465] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,465] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:50,474] jepsen worker 69 - jepsen.util 699 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,475] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:50,497] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,498] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:50,502] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,502] jepsen worker 34 - jepsen.util 754 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,503] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,503] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:50,503] jepsen worker 52 - jepsen.util 592 :invoke :write [17 22] INFO [2022-03-13 17:13:50,504] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:50,506] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,506] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,507] jepsen worker 21 - jepsen.util 831 :invoke :write [7 35] INFO [2022-03-13 17:13:50,507] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:13:50,534] jepsen worker 87 - jepsen.util 2607 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,535] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:13:50,550] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,551] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:13:50,562] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,563] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:13:50,613] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,614] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:13:50,614] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,614] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,615] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:50,616] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:50,618] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:50,618] jepsen worker nemesis - couchbase.util Rebalance check count: 340 INFO [2022-03-13 17:13:50,618] jepsen worker nemesis - couchbase.util Rebalance stuck count: 240 INFO [2022-03-13 17:13:50,624] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,624] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,625] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 30] INFO [2022-03-13 17:13:50,625] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:13:50,661] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,663] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,665] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,666] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:50,672] jepsen worker 80 - jepsen.util 710 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,673] jepsen worker 80 - jepsen.util 710 :invoke :write [26 44] INFO [2022-03-13 17:13:50,682] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,683] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,685] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,685] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:50,686] jepsen worker 26 - jepsen.util 476 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,687] jepsen worker 12 - jepsen.util 732 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,687] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,687] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,688] jepsen worker 26 - jepsen.util 476 :invoke :write [8 31] INFO [2022-03-13 17:13:50,689] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,690] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:50,691] jepsen worker 12 - jepsen.util 732 :invoke :write [4 5] INFO [2022-03-13 17:13:50,692] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,692] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:13:50,697] jepsen worker 21 - jepsen.util 831 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,697] jepsen worker 21 - jepsen.util 831 :invoke :write [7 22] INFO [2022-03-13 17:13:50,704] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,705] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,752] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,753] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,754] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,755] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,759] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,759] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:13:50,760] jepsen worker 32 - jepsen.util 2462 :info :write [10 38] :RequestTimeoutException INFO [2022-03-13 17:13:50,760] jepsen worker 32 - jepsen.util 2552 :invoke :write [10 20] INFO [2022-03-13 17:13:50,769] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,769] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:50,780] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,781] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:13:50,788] jepsen worker 20 - jepsen.util 2900 :info :write [6 40] :RequestTimeoutException INFO [2022-03-13 17:13:50,788] jepsen worker 20 - jepsen.util 2990 :invoke :write [6 48] INFO [2022-03-13 17:13:50,815] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,816] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,817] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,818] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:13:50,820] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,820] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:13:50,823] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,823] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,824] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:13:50,824] jepsen worker 14 - jepsen.util 374 :invoke :write [4 16] INFO [2022-03-13 17:13:50,837] jepsen worker 79 - jepsen.util 1069 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,837] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,837] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,838] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,838] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:50,839] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:50,840] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:50,840] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:13:50,878] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,879] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,880] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,880] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,881] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,882] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:50,883] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,884] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,885] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,886] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,891] jepsen worker 37 - jepsen.util 2017 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,891] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,892] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,892] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:13:50,892] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:13:50,892] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,896] jepsen worker 59 - jepsen.util 2489 :info :write [19 1] :RequestTimeoutException INFO [2022-03-13 17:13:50,896] jepsen worker 59 - jepsen.util 2579 :invoke :read [19 nil] INFO [2022-03-13 17:13:50,910] jepsen worker 63 - jepsen.util 2403 :info :write [21 24] :RequestTimeoutException INFO [2022-03-13 17:13:50,911] jepsen worker 63 - jepsen.util 2493 :invoke :write [21 13] INFO [2022-03-13 17:13:50,929] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,929] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,929] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:13:50,930] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 20] INFO [2022-03-13 17:13:50,932] jepsen worker 76 - jepsen.util 706 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,933] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:50,935] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,935] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,935] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,936] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,936] jepsen worker 13 - jepsen.util 643 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,937] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:13:50,939] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:50,939] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:13:50,940] jepsen worker 33 - jepsen.util 663 :invoke :write [11 11] INFO [2022-03-13 17:13:50,941] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:13:50,942] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,947] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:13:50,972] jepsen worker 21 - jepsen.util 831 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,973] jepsen worker 21 - jepsen.util 831 :invoke :write [7 23] INFO [2022-03-13 17:13:50,984] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,985] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:50,990] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:50,991] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:51,015] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,015] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:51,017] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,017] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,017] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 38] INFO [2022-03-13 17:13:51,018] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:13:51,031] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,032] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:13:51,038] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,039] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:13:51,084] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,084] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,089] jepsen worker 88 - jepsen.util 2068 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,090] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:51,090] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,091] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:13:51,106] jepsen worker 14 - jepsen.util 374 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,106] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,107] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:51,107] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:13:51,111] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,112] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:51,133] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,134] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:13:51,140] jepsen worker 8 - jepsen.util 728 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,140] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,140] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:51,141] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:51,141] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:51,142] jepsen worker 8 - jepsen.util 728 :invoke :write [2 14] INFO [2022-03-13 17:13:51,143] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,144] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:13:51,150] jepsen worker 78 - jepsen.util 528 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,150] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,150] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,151] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:51,152] jepsen worker 34 - jepsen.util 754 :invoke :write [11 37] INFO [2022-03-13 17:13:51,152] jepsen worker 70 - jepsen.util 700 :invoke :write [23 35] INFO [2022-03-13 17:13:51,156] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,157] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,161] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,162] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:13:51,178] jepsen worker 52 - jepsen.util 592 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,180] jepsen worker 52 - jepsen.util 592 :invoke :write [17 18] INFO [2022-03-13 17:13:51,209] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,209] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,209] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:51,209] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,210] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:13:51,210] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:13:51,210] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,211] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:13:51,224] jepsen worker 21 - jepsen.util 831 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,224] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:13:51,240] jepsen worker 39 - jepsen.util 2469 :info :write [13 28] :RequestTimeoutException INFO [2022-03-13 17:13:51,240] jepsen worker 39 - jepsen.util 2559 :invoke :write [13 10] INFO [2022-03-13 17:13:51,246] jepsen worker 68 - jepsen.util 2588 :info :write [22 44] :RequestTimeoutException INFO [2022-03-13 17:13:51,247] jepsen worker 68 - jepsen.util 2678 :invoke :write [22 45] INFO [2022-03-13 17:13:51,285] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,285] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,290] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,291] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,296] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,296] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,299] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,300] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,306] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,307] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,361] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,361] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 9] INFO [2022-03-13 17:13:51,373] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,374] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:51,405] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,405] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,406] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:51,407] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,411] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,412] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,413] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,413] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,414] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,415] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,416] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,416] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,417] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,418] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,419] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,419] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:51,490] jepsen worker 41 - jepsen.util 2381 :info :write [13 31] :RequestTimeoutException INFO [2022-03-13 17:13:51,490] jepsen worker 41 - jepsen.util 2471 :invoke :write [13 12] INFO [2022-03-13 17:13:51,491] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,492] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:13:51,526] jepsen worker 3 - jepsen.util 1893 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,526] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,527] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:13:51,527] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:51,546] jepsen worker 23 - jepsen.util 563 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,547] jepsen worker 23 - jepsen.util 563 :invoke :write [7 19] INFO [2022-03-13 17:13:51,553] jepsen worker 49 - jepsen.util 2209 :info :write [16 33] :SyncWriteAmbiguous INFO [2022-03-13 17:13:51,554] jepsen worker 49 - jepsen.util 2299 :invoke :write [16 38] INFO [2022-03-13 17:13:51,556] jepsen worker 33 - jepsen.util 663 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,556] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:51,560] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,560] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:13:51,574] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,575] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:51,580] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,580] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,580] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:13:51,581] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:13:51,617] jepsen worker 81 - jepsen.util 2961 :ok :read [27 4] INFO [2022-03-13 17:13:51,618] jepsen worker 81 - jepsen.util 2961 :invoke :write [27 34] INFO [2022-03-13 17:13:51,619] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,619] jepsen worker 80 - jepsen.util 710 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,620] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,620] jepsen worker 78 - jepsen.util 528 :invoke :write [26 11] INFO [2022-03-13 17:13:51,621] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:51,621] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:51,624] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:51,624] jepsen worker nemesis - couchbase.util Rebalance check count: 341 INFO [2022-03-13 17:13:51,624] jepsen worker nemesis - couchbase.util Rebalance stuck count: 241 INFO [2022-03-13 17:13:51,644] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,644] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:51,644] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,645] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,645] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:13:51,646] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:51,666] jepsen worker 5 - jepsen.util 2435 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,666] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:13:51,672] jepsen worker 62 - jepsen.util 512 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,673] jepsen worker 62 - jepsen.util 512 :invoke :write [20 31] INFO [2022-03-13 17:13:51,680] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,680] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,681] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:51,681] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:51,686] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,686] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,687] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:13:51,687] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:13:51,695] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,696] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:51,727] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,728] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:13:51,739] jepsen worker 60 - jepsen.util 600 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,740] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,740] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,739] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:51,740] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,740] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,740] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:51,741] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:51,742] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:51,743] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:51,743] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:51,744] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:51,751] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,751] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:51,757] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,758] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:51,759] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,759] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:51,760] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,760] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,760] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:13:51,760] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,760] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,761] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:13:51,762] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:13:51,762] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:51,764] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,764] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:13:51,782] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,783] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:13:51,797] jepsen worker 34 - jepsen.util 754 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,798] jepsen worker 34 - jepsen.util 754 :invoke :write [11 49] INFO [2022-03-13 17:13:51,813] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,813] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,814] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:51,815] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:51,816] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,817] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:51,817] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,818] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,818] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:51,819] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,819] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,819] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:13:51,820] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:13:51,822] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:13:51,823] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,824] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:51,825] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,826] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:13:51,830] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,831] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:13:51,836] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,836] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:13:51,890] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,891] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:51,901] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,901] jepsen worker 22 - jepsen.util 652 :invoke :write [7 7] INFO [2022-03-13 17:13:51,911] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,912] jepsen worker 14 - jepsen.util 374 :invoke :write [4 49] INFO [2022-03-13 17:13:51,919] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,920] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:13:51,924] jepsen worker 53 - jepsen.util 503 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,924] jepsen worker 12 - jepsen.util 732 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,924] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,924] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:51,925] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:51,925] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,925] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:51,926] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:13:51,929] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,929] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:13:51,935] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,936] jepsen worker 71 - jepsen.util 521 :invoke :write [23 35] INFO [2022-03-13 17:13:51,974] jepsen worker 70 - jepsen.util 700 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,975] jepsen worker 70 - jepsen.util 700 :invoke :write [23 17] INFO [2022-03-13 17:13:51,978] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,978] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,986] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,986] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:51,990] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,991] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:51,992] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:51,993] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,003] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,004] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:13:52,013] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,014] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 19] INFO [2022-03-13 17:13:52,031] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,031] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,032] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:13:52,033] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:52,033] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,034] jepsen worker 21 - jepsen.util 831 :invoke :write [7 22] INFO [2022-03-13 17:13:52,055] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,055] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:13:52,079] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,080] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,081] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,081] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:13:52,087] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,087] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,111] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,111] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,117] jepsen worker 52 - jepsen.util 592 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,117] jepsen worker 52 - jepsen.util 592 :invoke :write [17 8] INFO [2022-03-13 17:13:52,119] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,119] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,119] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:52,120] jepsen worker 13 - jepsen.util 643 :invoke :write [4 15] INFO [2022-03-13 17:13:52,126] jepsen worker 14 - jepsen.util 374 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,126] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,127] jepsen worker 14 - jepsen.util 374 :invoke :write [4 41] INFO [2022-03-13 17:13:52,127] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:13:52,130] jepsen worker 82 - jepsen.util 2512 :info :write [27 20] :RequestTimeoutException INFO [2022-03-13 17:13:52,131] jepsen worker 82 - jepsen.util 2602 :invoke :read [27 nil] INFO [2022-03-13 17:13:52,132] jepsen worker 6 - jepsen.util 636 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,132] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,132] jepsen worker 23 - jepsen.util 563 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,132] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:52,132] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,133] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,133] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,133] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,133] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:52,133] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:52,133] jepsen worker 6 - jepsen.util 636 :invoke :write [2 18] INFO [2022-03-13 17:13:52,134] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,134] jepsen worker 44 - jepsen.util 584 :invoke :write [14 28] INFO [2022-03-13 17:13:52,134] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,135] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:52,136] jepsen worker 12 - jepsen.util 732 :invoke :write [4 6] INFO [2022-03-13 17:13:52,136] jepsen worker 76 - jepsen.util 706 :invoke :write [25 12] INFO [2022-03-13 17:13:52,138] jepsen worker 61 - jepsen.util 691 :invoke :write [20 2] INFO [2022-03-13 17:13:52,139] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:52,142] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:52,143] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,143] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,143] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:13:52,143] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,143] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:52,144] jepsen worker 77 - jepsen.util 707 :invoke :write [25 0] INFO [2022-03-13 17:13:52,148] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,148] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,149] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,150] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,151] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:52,151] jepsen worker 60 - jepsen.util 600 :invoke :write [20 15] INFO [2022-03-13 17:13:52,151] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:52,152] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:52,157] jepsen worker 66 - jepsen.util 2406 :info :write [22 5] :RequestTimeoutException INFO [2022-03-13 17:13:52,158] jepsen worker 66 - jepsen.util 2496 :invoke :write [22 14] INFO [2022-03-13 17:13:52,167] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,168] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,169] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,170] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:13:52,177] jepsen worker 22 - jepsen.util 652 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,177] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,178] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,178] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:13:52,179] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 39] INFO [2022-03-13 17:13:52,180] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:13:52,184] jepsen worker 50 - jepsen.util 2660 :info :write [16 16] :RequestTimeoutException INFO [2022-03-13 17:13:52,185] jepsen worker 50 - jepsen.util 2750 :invoke :read [16 nil] INFO [2022-03-13 17:13:52,194] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,195] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,234] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,234] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,235] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,236] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:52,242] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,242] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,247] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,248] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:13:52,265] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,265] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,294] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,294] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,307] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,307] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,308] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,309] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,310] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,311] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:13:52,320] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,320] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,359] jepsen worker 18 - jepsen.util 2268 :info :write [6 12] :SyncWriteAmbiguous INFO [2022-03-13 17:13:52,359] jepsen worker 3 - jepsen.util 1893 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,360] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,359] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,359] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,360] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,361] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:13:52,362] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,363] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,363] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,363] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,364] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,365] jepsen worker 18 - jepsen.util 2358 :invoke :read [6 nil] INFO [2022-03-13 17:13:52,365] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,366] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,367] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,367] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,367] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,368] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,368] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,368] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 34] INFO [2022-03-13 17:13:52,369] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,369] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,369] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:13:52,370] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,370] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:13:52,372] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,373] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:13:52,384] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,384] jepsen worker 37 - jepsen.util 2017 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,385] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,386] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 37] INFO [2022-03-13 17:13:52,438] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,439] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 6] INFO [2022-03-13 17:13:52,445] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,445] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,446] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,446] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:13:52,456] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,462] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:13:52,475] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,476] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:13:52,521] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,522] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:13:52,541] jepsen worker 16 - jepsen.util 2266 :info :write [5 20] :SyncWriteAmbiguous INFO [2022-03-13 17:13:52,543] jepsen worker 16 - jepsen.util 2356 :invoke :write [5 5] INFO [2022-03-13 17:13:52,559] jepsen worker 37 - jepsen.util 2017 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,559] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,559] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,560] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,610] jepsen worker 83 - jepsen.util 1973 :info :write [27 7] :RequestTimeoutException INFO [2022-03-13 17:13:52,611] jepsen worker 83 - jepsen.util 2063 :invoke :read [27 nil] INFO [2022-03-13 17:13:52,618] jepsen worker 36 - jepsen.util 2196 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,619] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:13:52,629] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:52,630] jepsen worker nemesis - couchbase.util Rebalance check count: 342 INFO [2022-03-13 17:13:52,630] jepsen worker nemesis - couchbase.util Rebalance stuck count: 242 INFO [2022-03-13 17:13:52,646] jepsen worker 88 - jepsen.util 2068 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,646] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 47] INFO [2022-03-13 17:13:52,668] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,668] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,668] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,669] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:13:52,672] jepsen worker 48 - jepsen.util 2478 :ok :read [16 :nil] INFO [2022-03-13 17:13:52,673] jepsen worker 48 - jepsen.util 2478 :invoke :write [16 11] INFO [2022-03-13 17:13:52,696] jepsen worker 59 - jepsen.util 2579 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,697] jepsen worker 59 - jepsen.util 2579 :invoke :write [19 36] INFO [2022-03-13 17:13:52,714] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,714] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,716] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,716] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,717] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,718] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,729] jepsen worker 82 - jepsen.util 2602 :ok :read [27 4] INFO [2022-03-13 17:13:52,729] jepsen worker 82 - jepsen.util 2602 :invoke :write [27 20] INFO [2022-03-13 17:13:52,732] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,733] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,740] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,741] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,741] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,742] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,742] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,742] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,744] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:13:52,744] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:13:52,763] jepsen worker 84 - jepsen.util 2784 :ok :read [28 44] INFO [2022-03-13 17:13:52,764] jepsen worker 84 - jepsen.util 2784 :invoke :write [28 22] INFO [2022-03-13 17:13:52,815] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,816] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,817] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,817] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:13:52,818] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,819] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,819] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,819] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,832] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,833] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:13:52,837] jepsen worker 37 - jepsen.util 2017 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,837] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,839] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,839] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,840] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,840] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,841] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:13:52,841] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:52,847] jepsen worker 58 - jepsen.util 2308 :ok :read [19 44] INFO [2022-03-13 17:13:52,847] jepsen worker 58 - jepsen.util 2308 :invoke :read [19 nil] INFO [2022-03-13 17:13:52,858] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,858] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:13:52,899] jepsen worker 85 - jepsen.util 2425 :ok :read [28 44] INFO [2022-03-13 17:13:52,899] jepsen worker 85 - jepsen.util 2425 :invoke :read [28 nil] INFO [2022-03-13 17:13:52,905] jepsen worker 86 - jepsen.util 2426 :ok :read [28 44] INFO [2022-03-13 17:13:52,906] jepsen worker 86 - jepsen.util 2426 :invoke :read [28 nil] INFO [2022-03-13 17:13:52,911] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,911] jepsen worker 88 - jepsen.util 2068 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,911] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,912] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:13:52,913] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,913] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:52,914] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,917] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,918] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,919] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,920] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,920] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:13:52,921] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,922] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:13:52,988] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,989] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:52,997] jepsen worker 37 - jepsen.util 2017 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:52,998] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:13:53,000] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,001] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:13:53,021] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,021] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,021] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,022] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:13:53,023] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:13:53,024] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,067] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,068] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:53,095] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,095] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:13:53,116] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,125] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:13:53,116] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,125] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,126] jepsen worker 54 - jepsen.util 2124 :info :write [18 49] :RequestTimeoutException INFO [2022-03-13 17:13:53,127] jepsen worker 54 - jepsen.util 2214 :invoke :read [18 nil] INFO [2022-03-13 17:13:53,145] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,145] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,146] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:13:53,151] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,168] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,169] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,173] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,173] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,174] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,174] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,177] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,178] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,180] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,180] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,184] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,184] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,186] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,187] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:13:53,217] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,217] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:13:53,259] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,259] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,259] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:13:53,260] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:53,268] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,268] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,269] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:13:53,269] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,305] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,305] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:13:53,316] jepsen worker 19 - jepsen.util 2359 :info :write [6 49] :RequestTimeoutException INFO [2022-03-13 17:13:53,317] jepsen worker 19 - jepsen.util 2449 :invoke :write [6 33] INFO [2022-03-13 17:13:53,350] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,350] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,350] jepsen worker 26 - jepsen.util 476 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,350] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,351] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,351] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,352] jepsen worker 26 - jepsen.util 476 :invoke :write [8 7] INFO [2022-03-13 17:13:53,353] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:53,354] jepsen worker 69 - jepsen.util 699 :invoke :write [23 17] INFO [2022-03-13 17:13:53,355] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:53,355] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:13:53,364] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:53,364] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,365] jepsen worker 24 - jepsen.util 564 :invoke :write [8 38] INFO [2022-03-13 17:13:53,376] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,377] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,382] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,383] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,383] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,383] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,383] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,384] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,384] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:13:53,385] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,393] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,394] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,395] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,396] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,397] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,397] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,398] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,399] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,401] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,402] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:53,425] jepsen worker 78 - jepsen.util 528 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,426] jepsen worker 78 - jepsen.util 528 :invoke :write [26 16] INFO [2022-03-13 17:13:53,433] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,434] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:13:53,460] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,460] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,460] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,460] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,461] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,461] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:13:53,469] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,469] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,472] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,472] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,474] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,475] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:13:53,480] jepsen worker 21 - jepsen.util 831 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,480] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:13:53,493] jepsen worker 71 - jepsen.util 521 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,493] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:13:53,520] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,521] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,521] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:13:53,521] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,521] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,522] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,538] jepsen worker 43 - jepsen.util 493 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,539] jepsen worker 43 - jepsen.util 493 :invoke :write [14 36] INFO [2022-03-13 17:13:53,545] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,546] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:13:53,562] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,563] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,563] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,563] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,564] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:13:53,564] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,592] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,592] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:13:53,593] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,593] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,598] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,598] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:13:53,619] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,620] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:53,635] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:53,635] jepsen worker nemesis - couchbase.util Rebalance check count: 343 INFO [2022-03-13 17:13:53,635] jepsen worker nemesis - couchbase.util Rebalance stuck count: 243 INFO [2022-03-13 17:13:53,647] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,647] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,647] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,648] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,648] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,648] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:13:53,649] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,649] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,649] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,650] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,653] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,653] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:13:53,661] jepsen worker 14 - jepsen.util 374 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,661] jepsen worker 34 - jepsen.util 754 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,661] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,663] jepsen worker 14 - jepsen.util 374 :invoke :write [4 24] INFO [2022-03-13 17:13:53,663] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:13:53,664] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:13:53,711] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,711] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:13:53,715] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,716] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,716] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,716] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,716] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,717] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:13:53,722] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,723] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:13:53,734] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,735] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:13:53,756] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,756] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,758] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,759] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:13:53,763] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,764] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,765] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,766] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,770] jepsen worker 76 - jepsen.util 706 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,771] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,771] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:53,772] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:13:53,781] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,781] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,788] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,788] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,788] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,788] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,789] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:13:53,789] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,795] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,796] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:53,807] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,808] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:53,809] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,811] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:13:53,821] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,822] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:13:53,833] jepsen worker 24 - jepsen.util 564 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,834] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:13:53,847] jepsen worker 70 - jepsen.util 700 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,847] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,847] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,848] jepsen worker 80 - jepsen.util 710 :invoke :write [26 13] INFO [2022-03-13 17:13:53,849] jepsen worker 70 - jepsen.util 700 :invoke :write [23 16] INFO [2022-03-13 17:13:53,850] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:13:53,856] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,856] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,880] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,880] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,882] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,882] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,883] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,884] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,885] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,886] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:13:53,897] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,897] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,897] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,899] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,899] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:13:53,900] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,917] jepsen worker 14 - jepsen.util 374 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,918] jepsen worker 14 - jepsen.util 374 :invoke :write [4 14] INFO [2022-03-13 17:13:53,943] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,943] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,943] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 19] INFO [2022-03-13 17:13:53,944] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,947] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,948] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:53,949] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,949] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:53,953] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,953] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:53,954] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,955] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:53,976] jepsen worker 13 - jepsen.util 643 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,976] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:53,994] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,994] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,995] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,995] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,995] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:13:53,996] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:53,996] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:53,997] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:13:53,997] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:13:53,998] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:54,010] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,011] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,015] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,015] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:13:54,044] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,045] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:13:54,056] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,056] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:54,085] jepsen worker 37 - jepsen.util 2017 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,085] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,086] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 26] INFO [2022-03-13 17:13:54,086] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:13:54,094] jepsen worker 52 - jepsen.util 592 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,095] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:54,105] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,105] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,106] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 12] INFO [2022-03-13 17:13:54,108] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,110] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:54,116] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 34] INFO [2022-03-13 17:13:54,157] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,158] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,161] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,161] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,163] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,163] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,164] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:54,165] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,167] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,167] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,168] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,168] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:13:54,169] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,171] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:13:54,172] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:54,172] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:13:54,185] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,186] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:54,189] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,190] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,191] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:13:54,195] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:13:54,195] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,199] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:13:54,256] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,257] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:13:54,264] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,264] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,265] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:54,265] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:54,276] jepsen worker 37 - jepsen.util 2017 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,277] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:13:54,322] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,323] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:13:54,323] jepsen worker 3 - jepsen.util 1893 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,324] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:54,384] jepsen worker 87 - jepsen.util 2607 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,384] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,390] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,391] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:13:54,403] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,404] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:13:54,459] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,460] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:13:54,475] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,475] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:13:54,485] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,486] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,488] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,489] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,491] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,492] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,494] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,494] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,496] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,496] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,499] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,500] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:54,519] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,520] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,579] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,579] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,579] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,580] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:13:54,580] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:54,581] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:13:54,604] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,605] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:13:54,624] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,625] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,641] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:54,641] jepsen worker nemesis - couchbase.util Rebalance check count: 344 INFO [2022-03-13 17:13:54,641] jepsen worker nemesis - couchbase.util Rebalance stuck count: 244 INFO [2022-03-13 17:13:54,644] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,645] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,647] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,648] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:13:54,656] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,656] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:54,659] jepsen worker 78 - jepsen.util 528 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,660] jepsen worker 78 - jepsen.util 528 :invoke :write [26 49] INFO [2022-03-13 17:13:54,677] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,677] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,678] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,678] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:13:54,682] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 30] INFO [2022-03-13 17:13:54,682] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,684] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,685] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,686] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,686] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:13:54,705] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,705] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,727] jepsen worker 14 - jepsen.util 374 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,727] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,728] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,728] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,728] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,729] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,729] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,730] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:54,731] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:13:54,731] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,732] jepsen worker 21 - jepsen.util 831 :invoke :write [7 14] INFO [2022-03-13 17:13:54,733] jepsen worker 34 - jepsen.util 754 :invoke :write [11 8] INFO [2022-03-13 17:13:54,733] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:13:54,734] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:13:54,742] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,743] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,744] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,745] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,752] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,755] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:54,762] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,763] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:54,789] jepsen worker 80 - jepsen.util 710 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,789] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,789] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,790] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,790] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:54,791] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,791] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,792] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:54,793] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:13:54,793] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:54,798] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,799] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,799] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,799] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:13:54,800] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,800] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:13:54,801] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:54,801] jepsen worker 24 - jepsen.util 564 :invoke :write [8 19] INFO [2022-03-13 17:13:54,829] jepsen worker 37 - jepsen.util 2017 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,830] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:13:54,830] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,831] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:13:54,853] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,853] jepsen worker 70 - jepsen.util 700 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,853] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,854] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:54,855] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:54,856] jepsen worker 13 - jepsen.util 643 :invoke :write [4 17] INFO [2022-03-13 17:13:54,860] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,860] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,860] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,861] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,861] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:13:54,862] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,884] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,885] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,889] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,889] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,891] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,891] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,893] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,893] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,894] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,895] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:13:54,896] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,896] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:54,912] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,913] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,913] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:54,913] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:54,913] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,914] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:13:54,916] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,917] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,919] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,920] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:54,921] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,921] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:13:54,944] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,944] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,944] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,944] jepsen worker 78 - jepsen.util 528 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,945] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 23] INFO [2022-03-13 17:13:54,946] jepsen worker 78 - jepsen.util 528 :invoke :write [26 31] INFO [2022-03-13 17:13:54,946] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:54,947] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:13:54,958] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,958] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:13:54,974] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,974] jepsen worker 24 - jepsen.util 564 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,975] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,975] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:54,976] jepsen worker 33 - jepsen.util 663 :invoke :write [11 45] INFO [2022-03-13 17:13:54,977] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:54,987] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:54,988] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,008] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,009] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,010] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,010] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:13:55,012] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,015] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:13:55,026] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,026] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,027] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,027] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,035] jepsen worker 21 - jepsen.util 831 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,036] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:55,084] jepsen worker 34 - jepsen.util 754 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,084] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,085] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,085] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,085] jepsen worker 80 - jepsen.util 710 :invoke :write [26 34] INFO [2022-03-13 17:13:55,086] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:55,087] jepsen worker 70 - jepsen.util 700 :invoke :write [23 7] INFO [2022-03-13 17:13:55,087] jepsen worker 52 - jepsen.util 592 :invoke :write [17 30] INFO [2022-03-13 17:13:55,089] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,090] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:13:55,095] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,095] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,096] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,097] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:13:55,104] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,104] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,105] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,106] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:13:55,107] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:55,108] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,124] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,124] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,124] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:13:55,125] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:55,155] jepsen worker 3 - jepsen.util 1893 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,155] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:13:55,163] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,163] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,163] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,164] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 31] INFO [2022-03-13 17:13:55,166] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,166] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,168] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,168] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,172] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,173] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,174] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,175] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,175] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,175] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,176] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,177] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,178] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,178] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,179] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,179] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,180] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,181] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,181] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,181] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:13:55,184] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,185] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,191] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,192] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,200] jepsen worker 33 - jepsen.util 663 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,200] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,200] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,201] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:13:55,202] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,202] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:55,218] jepsen worker 15 - jepsen.util 2625 :info :write [5 23] :RequestTimeoutException INFO [2022-03-13 17:13:55,219] jepsen worker 15 - jepsen.util 2715 :invoke :read [5 nil] INFO [2022-03-13 17:13:55,237] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,237] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,237] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,238] jepsen worker 21 - jepsen.util 831 :invoke :write [7 36] INFO [2022-03-13 17:13:55,238] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:55,239] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:13:55,250] jepsen worker 67 - jepsen.util 2227 :info :write [22 42] :RequestTimeoutException INFO [2022-03-13 17:13:55,250] jepsen worker 67 - jepsen.util 2317 :invoke :read [22 nil] INFO [2022-03-13 17:13:55,263] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,263] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,263] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,263] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,264] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:13:55,264] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,271] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,271] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,272] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:13:55,273] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,280] jepsen worker 13 - jepsen.util 643 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,280] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,281] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:55,281] jepsen worker 22 - jepsen.util 652 :invoke :write [7 37] INFO [2022-03-13 17:13:55,290] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,291] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:13:55,326] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,326] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,326] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,327] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:55,328] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:13:55,328] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,340] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,340] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,340] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:13:55,341] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,365] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,365] jepsen worker 37 - jepsen.util 2017 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,366] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,366] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:13:55,372] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,373] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:13:55,386] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,386] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,387] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,387] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:13:55,387] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,388] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,410] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,410] jepsen worker 78 - jepsen.util 528 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,412] jepsen worker 78 - jepsen.util 528 :invoke :write [26 20] INFO [2022-03-13 17:13:55,412] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:55,414] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,414] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:13:55,453] jepsen worker 21 - jepsen.util 831 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,453] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:55,455] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,455] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:13:55,455] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,456] jepsen worker 33 - jepsen.util 663 :invoke :write [11 11] INFO [2022-03-13 17:13:55,460] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,460] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,488] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,489] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:13:55,490] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,491] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:13:55,507] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,507] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,507] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,507] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,508] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,508] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,523] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,523] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,524] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:55,524] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,537] jepsen worker 22 - jepsen.util 652 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,539] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:13:55,549] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,549] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,562] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,562] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,563] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:13:55,563] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:13:55,575] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,575] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,575] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,575] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,576] jepsen worker 35 - jepsen.util 935 :invoke :write [11 48] INFO [2022-03-13 17:13:55,577] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:55,578] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:13:55,579] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:13:55,584] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,585] jepsen worker 14 - jepsen.util 374 :invoke :write [4 21] INFO [2022-03-13 17:13:55,594] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,595] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,601] jepsen worker 80 - jepsen.util 710 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,601] jepsen worker 80 - jepsen.util 710 :invoke :write [26 43] INFO [2022-03-13 17:13:55,616] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,616] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,616] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,616] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,616] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,617] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,618] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,618] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:13:55,619] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,620] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,620] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:55,624] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,625] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,626] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:13:55,626] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:13:55,627] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:13:55,634] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,634] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,635] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,635] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:13:55,647] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:55,647] jepsen worker nemesis - couchbase.util Rebalance check count: 345 INFO [2022-03-13 17:13:55,647] jepsen worker nemesis - couchbase.util Rebalance stuck count: 245 INFO [2022-03-13 17:13:55,672] jepsen worker 33 - jepsen.util 663 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,674] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:55,688] jepsen worker 52 - jepsen.util 592 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,689] jepsen worker 52 - jepsen.util 592 :invoke :write [17 11] INFO [2022-03-13 17:13:55,690] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,690] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,694] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,695] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:13:55,697] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,697] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,699] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,699] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,700] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,700] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,701] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,702] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:13:55,707] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,708] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,722] jepsen worker 72 - jepsen.util 2592 :info :write [24 47] :SyncWriteAmbiguous INFO [2022-03-13 17:13:55,723] jepsen worker 72 - jepsen.util 2682 :invoke :write [24 43] INFO [2022-03-13 17:13:55,743] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,743] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,744] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,745] jepsen worker 22 - jepsen.util 652 :invoke :write [7 8] INFO [2022-03-13 17:13:55,785] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,785] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,786] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:13:55,786] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,789] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,790] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:55,794] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,796] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,796] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,797] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:13:55,800] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,801] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,804] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,805] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:13:55,813] jepsen worker 74 - jepsen.util 2774 :ok :read [24 24] INFO [2022-03-13 17:13:55,814] jepsen worker 74 - jepsen.util 2774 :invoke :read [24 nil] INFO [2022-03-13 17:13:55,817] jepsen worker 55 - jepsen.util 2485 :ok :read [18 21] INFO [2022-03-13 17:13:55,818] jepsen worker 55 - jepsen.util 2485 :invoke :read [18 nil] INFO [2022-03-13 17:13:55,819] jepsen worker 18 - jepsen.util 2358 :ok :read [6 16] INFO [2022-03-13 17:13:55,820] jepsen worker 18 - jepsen.util 2358 :invoke :write [6 32] INFO [2022-03-13 17:13:55,820] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,821] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,843] jepsen worker 74 - jepsen.util 2774 :ok :read [24 24] INFO [2022-03-13 17:13:55,844] jepsen worker 74 - jepsen.util 2774 :invoke :read [24 nil] INFO [2022-03-13 17:13:55,844] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,845] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,852] jepsen worker 55 - jepsen.util 2485 :ok :read [18 21] INFO [2022-03-13 17:13:55,853] jepsen worker 55 - jepsen.util 2485 :invoke :read [18 nil] INFO [2022-03-13 17:13:55,863] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,863] jepsen worker 35 - jepsen.util 935 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,864] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:13:55,866] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:13:55,868] jepsen worker 70 - jepsen.util 700 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,868] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,869] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:13:55,874] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,875] jepsen worker 13 - jepsen.util 643 :invoke :write [4 29] INFO [2022-03-13 17:13:55,875] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:13:55,886] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,886] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:13:55,908] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,908] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,919] jepsen worker 74 - jepsen.util 2774 :ok :read [24 24] INFO [2022-03-13 17:13:55,919] jepsen worker 74 - jepsen.util 2774 :invoke :write [24 43] INFO [2022-03-13 17:13:55,938] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,939] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:55,942] jepsen worker 55 - jepsen.util 2485 :ok :read [18 21] INFO [2022-03-13 17:13:55,943] jepsen worker 55 - jepsen.util 2485 :invoke :write [18 34] INFO [2022-03-13 17:13:55,950] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,951] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:13:55,959] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,959] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,959] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,959] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,960] jepsen worker 14 - jepsen.util 374 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,960] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,960] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,961] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,961] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:13:55,962] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:55,963] jepsen worker 14 - jepsen.util 374 :invoke :write [4 39] INFO [2022-03-13 17:13:55,963] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,963] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:55,964] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:13:55,964] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:13:55,966] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,975] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,981] jepsen worker 78 - jepsen.util 528 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:55,982] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:55,983] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:56,021] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,021] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,022] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:13:56,022] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,045] jepsen worker 22 - jepsen.util 652 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,046] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:56,048] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,049] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:56,050] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,051] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:13:56,052] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,052] jepsen worker 21 - jepsen.util 831 :invoke :write [7 25] INFO [2022-03-13 17:13:56,055] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,055] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:13:56,055] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,056] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:56,062] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,062] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,062] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,063] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,063] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:13:56,064] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,064] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:13:56,065] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 10] INFO [2022-03-13 17:13:56,148] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,148] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,148] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,148] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,149] jepsen worker 80 - jepsen.util 710 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,153] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,153] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:56,153] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:56,154] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:56,154] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:13:56,154] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,154] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,157] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,157] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:13:56,165] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,165] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,166] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,167] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:13:56,170] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:56,170] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:13:56,173] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,175] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:13:56,180] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,180] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,181] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:13:56,181] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:13:56,232] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,232] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,232] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,233] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:56,234] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:56,234] jepsen worker 24 - jepsen.util 564 :invoke :write [8 19] INFO [2022-03-13 17:13:56,247] jepsen worker 14 - jepsen.util 374 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,248] jepsen worker 14 - jepsen.util 374 :invoke :write [4 44] INFO [2022-03-13 17:13:56,261] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,262] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 34] INFO [2022-03-13 17:13:56,267] jepsen worker 52 - jepsen.util 592 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,267] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:56,282] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,282] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:13:56,294] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,295] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,305] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,306] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:13:56,319] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,319] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:56,321] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,321] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:13:56,322] jepsen worker 21 - jepsen.util 831 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,322] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,322] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:56,322] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:56,329] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,329] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,330] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,330] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:56,330] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:13:56,333] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] WARN [2022-03-13 17:13:56,347] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9058236,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10107949,"total_server_duration_us":9268252,"encode_duration_us":17599,"operation_id":"0x950e","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10010238},{"operation_name":"upsert","last_dispatch_duration_us":96973,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":96973,"total_server_duration_us":79120,"encode_duration_us":26488,"operation_id":"0x9418","timeout_ms":10000,"last_server_duration_us":79120,"total_duration_us":10009965},{"operation_name":"upsert","last_dispatch_duration_us":8696996,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9820727,"total_server_duration_us":312373,"encode_duration_us":27227,"operation_id":"0x951d","timeout_ms":10000,"last_server_duration_us":27583,"total_duration_us":10007869},{"operation_name":"upsert","last_dispatch_duration_us":8396125,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10039085,"total_server_duration_us":383233,"encode_duration_us":19564,"operation_id":"0x94a7","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10004333},{"operation_name":"upsert","last_dispatch_duration_us":9063794,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9836864,"total_server_duration_us":9511903,"encode_duration_us":17172,"operation_id":"0x916f","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10000761},{"operation_name":"get","last_dispatch_duration_us":8033745,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8033745,"total_server_duration_us":43,"operation_id":"0x96be","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8034096},{"operation_name":"get","last_dispatch_duration_us":1798638,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1798638,"total_server_duration_us":20601,"operation_id":"0x9ac9","timeout_ms":30000,"last_server_duration_us":20601,"total_duration_us":1798966},{"operation_name":"upsert","last_dispatch_duration_us":1496288,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1496288,"total_server_duration_us":55303,"encode_duration_us":31774,"operation_id":"0x9b86","timeout_ms":10000,"last_server_duration_us":55303,"total_duration_us":1501327},{"operation_name":"upsert","last_dispatch_duration_us":1438650,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1438650,"total_server_duration_us":48679,"encode_duration_us":21275,"operation_id":"0x9b81","timeout_ms":10000,"last_server_duration_us":48679,"total_duration_us":1442588},{"operation_name":"upsert","last_dispatch_duration_us":1418434,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1418434,"total_server_duration_us":86182,"encode_duration_us":22674,"operation_id":"0x9b71","timeout_ms":10000,"last_server_duration_us":86182,"total_duration_us":1422229}],"total_count":24}} WARN [2022-03-13 17:13:56,348] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9063794,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9836864,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9511903,"encode_duration_us":17172,"operation_id":"0x916f","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10000761},{"operation_name":"upsert","last_dispatch_duration_us":9058236,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10107949,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9268252,"encode_duration_us":17599,"operation_id":"0x950e","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10010238}],"total_count":2}} INFO [2022-03-13 17:13:56,354] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,355] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,356] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,357] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:13:56,387] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,388] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:56,389] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,389] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:56,399] jepsen worker 88 - jepsen.util 2068 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,399] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:13:56,419] jepsen worker 29 - jepsen.util 2729 :info :write [9 16] :RequestTimeoutException INFO [2022-03-13 17:13:56,420] jepsen worker 29 - jepsen.util 2819 :invoke :write [9 2] INFO [2022-03-13 17:13:56,424] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,424] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,424] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:56,424] jepsen worker 24 - jepsen.util 564 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,425] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:13:56,425] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:56,435] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,436] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,437] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,437] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:13:56,454] jepsen worker 17 - jepsen.util 2627 :info :write [5 37] :RequestTimeoutException INFO [2022-03-13 17:13:56,455] jepsen worker 17 - jepsen.util 2717 :invoke :write [5 15] INFO [2022-03-13 17:13:56,455] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,456] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,459] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,460] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:56,461] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,461] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,467] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,468] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:13:56,482] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,483] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:13:56,523] jepsen worker 14 - jepsen.util 374 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,523] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,523] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,523] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,524] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:56,525] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:13:56,525] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,525] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:13:56,526] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:56,527] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,528] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,529] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:56,529] jepsen worker 70 - jepsen.util 700 :invoke :write [23 35] INFO [2022-03-13 17:13:56,530] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,531] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:56,532] jepsen worker 14 - jepsen.util 374 :invoke :write [4 43] INFO [2022-03-13 17:13:56,532] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,533] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:13:56,542] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,542] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,542] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,543] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,543] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:56,544] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:13:56,545] jepsen worker 33 - jepsen.util 663 :invoke :write [11 9] INFO [2022-03-13 17:13:56,546] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:13:56,579] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,579] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,580] jepsen worker 37 - jepsen.util 2017 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,581] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,583] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:56,583] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,586] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,587] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:13:56,591] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,591] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:56,591] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,592] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,592] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 8] INFO [2022-03-13 17:13:56,592] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:56,593] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,593] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:13:56,629] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,629] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,630] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:13:56,630] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:56,635] jepsen worker 9 - jepsen.util 2169 :ok :read [3 25] INFO [2022-03-13 17:13:56,636] jepsen worker 9 - jepsen.util 2169 :invoke :write [3 7] INFO [2022-03-13 17:13:56,643] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,643] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,643] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:56,644] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:56,652] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,653] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:13:56,654] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,655] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:13:56,658] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:56,658] jepsen worker nemesis - couchbase.util Rebalance check count: 346 INFO [2022-03-13 17:13:56,658] jepsen worker nemesis - couchbase.util Rebalance stuck count: 246 INFO [2022-03-13 17:13:56,670] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,671] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,673] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,674] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 37] INFO [2022-03-13 17:13:56,676] jepsen worker 10 - jepsen.util 2710 :ok :read [3 25] INFO [2022-03-13 17:13:56,676] jepsen worker 10 - jepsen.util 2710 :invoke :read [3 nil] INFO [2022-03-13 17:13:56,688] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,689] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:56,739] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,740] jepsen worker 13 - jepsen.util 643 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,740] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,740] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,740] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,740] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,740] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,741] jepsen worker 52 - jepsen.util 592 :invoke :write [17 10] INFO [2022-03-13 17:13:56,741] jepsen worker 34 - jepsen.util 754 :invoke :write [11 40] INFO [2022-03-13 17:13:56,742] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:56,742] jepsen worker 80 - jepsen.util 710 :invoke :write [26 18] INFO [2022-03-13 17:13:56,743] jepsen worker 78 - jepsen.util 528 :invoke :write [26 33] INFO [2022-03-13 17:13:56,750] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,751] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:13:56,751] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,752] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:56,757] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,757] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:56,792] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,793] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:56,800] jepsen worker 10 - jepsen.util 2710 :ok :read [3 25] INFO [2022-03-13 17:13:56,800] jepsen worker 10 - jepsen.util 2710 :invoke :read [3 nil] INFO [2022-03-13 17:13:56,838] jepsen worker 62 - jepsen.util 512 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,838] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,839] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,839] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:56,840] jepsen worker 53 - jepsen.util 503 :invoke :write [17 41] INFO [2022-03-13 17:13:56,840] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:13:56,858] jepsen worker 12 - jepsen.util 732 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,859] jepsen worker 12 - jepsen.util 732 :invoke :write [4 6] INFO [2022-03-13 17:13:56,859] jepsen worker 37 - jepsen.util 2017 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,859] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,860] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,861] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:13:56,865] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,866] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:56,872] jepsen worker 33 - jepsen.util 663 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,872] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,872] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,873] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:56,874] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:13:56,874] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:13:56,927] jepsen worker 89 - jepsen.util 2429 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,927] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,928] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,929] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,929] jepsen worker 10 - jepsen.util 2710 :ok :read [3 25] INFO [2022-03-13 17:13:56,930] jepsen worker 10 - jepsen.util 2710 :invoke :write [3 20] INFO [2022-03-13 17:13:56,930] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,931] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,934] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,935] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,936] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,937] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,940] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,941] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:13:56,948] jepsen worker 70 - jepsen.util 700 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,948] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,948] jepsen worker 13 - jepsen.util 643 :invoke :write [4 34] INFO [2022-03-13 17:13:56,949] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:56,955] jepsen worker 14 - jepsen.util 374 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,955] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:56,957] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,959] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,966] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,966] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,966] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,966] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,966] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,967] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 28] INFO [2022-03-13 17:13:56,967] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,968] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,968] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:56,969] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,969] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:13:56,969] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:13:56,975] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,975] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,976] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:56,976] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:13:56,977] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,978] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:13:56,991] jepsen worker 6 - jepsen.util 636 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,991] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,991] jepsen worker 77 - jepsen.util 707 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:56,991] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:56,993] jepsen worker 6 - jepsen.util 636 :invoke :write [2 25] INFO [2022-03-13 17:13:56,993] jepsen worker 77 - jepsen.util 707 :invoke :write [25 7] INFO [2022-03-13 17:13:56,994] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:13:56,995] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:57,029] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,029] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,031] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:13:57,031] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:13:57,050] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,051] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:57,052] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,052] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,052] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,053] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:57,053] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:13:57,055] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:13:57,056] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,056] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,056] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,057] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,057] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:57,058] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,058] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:13:57,059] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,059] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:57,059] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,059] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:57,060] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:57,060] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,061] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:13:57,070] jepsen worker 30 - jepsen.util 2550 :ok :read [10 :nil] INFO [2022-03-13 17:13:57,071] jepsen worker 30 - jepsen.util 2550 :invoke :read [10 nil] INFO [2022-03-13 17:13:57,077] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,077] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,077] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,078] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:57,078] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:13:57,079] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,082] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,083] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:13:57,090] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,090] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,097] jepsen worker 3 - jepsen.util 1893 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,098] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:57,120] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,120] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,122] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,123] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,133] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,134] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,135] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,135] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,137] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,138] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,139] jepsen worker 30 - jepsen.util 2550 :ok :read [10 :nil] INFO [2022-03-13 17:13:57,139] jepsen worker 30 - jepsen.util 2550 :invoke :read [10 nil] INFO [2022-03-13 17:13:57,141] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,142] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:13:57,155] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,156] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:13:57,169] jepsen worker 30 - jepsen.util 2550 :ok :read [10 :nil] INFO [2022-03-13 17:13:57,170] jepsen worker 30 - jepsen.util 2550 :invoke :write [10 27] INFO [2022-03-13 17:13:57,180] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,182] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:13:57,203] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,204] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 40] INFO [2022-03-13 17:13:57,206] jepsen worker 31 - jepsen.util 2641 :info :write [10 43] :RequestTimeoutException INFO [2022-03-13 17:13:57,207] jepsen worker 31 - jepsen.util 2731 :invoke :read [10 nil] INFO [2022-03-13 17:13:57,218] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,218] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,219] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:57,219] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:13:57,234] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,235] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:13:57,266] jepsen worker 31 - jepsen.util 2731 :ok :read [10 :nil] INFO [2022-03-13 17:13:57,267] jepsen worker 31 - jepsen.util 2731 :invoke :write [10 8] INFO [2022-03-13 17:13:57,272] jepsen worker 8 - jepsen.util 728 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,272] jepsen worker 61 - jepsen.util 691 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,272] jepsen worker 8 - jepsen.util 728 :invoke :write [2 32] INFO [2022-03-13 17:13:57,273] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,274] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:57,274] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:13:57,276] jepsen worker 34 - jepsen.util 754 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,276] jepsen worker 34 - jepsen.util 754 :invoke :write [11 18] INFO [2022-03-13 17:13:57,283] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,284] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,284] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:57,285] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,302] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,303] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:13:57,319] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,319] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:13:57,336] jepsen worker 56 - jepsen.util 2666 :info :write [18 6] :RequestTimeoutException INFO [2022-03-13 17:13:57,337] jepsen worker 56 - jepsen.util 2756 :invoke :write [18 28] INFO [2022-03-13 17:13:57,347] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,348] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,349] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,350] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,383] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,384] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:57,386] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,387] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:13:57,387] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,387] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,391] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,391] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,391] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,391] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,392] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:57,393] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:13:57,395] jepsen worker 52 - jepsen.util 592 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,396] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,396] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:57,397] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:13:57,420] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,421] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:13:57,460] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,460] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:13:57,493] jepsen worker 37 - jepsen.util 2017 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,493] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:57,496] jepsen worker 26 - jepsen.util 476 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,496] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,497] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:57,497] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:57,499] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,500] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:57,501] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,502] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:13:57,511] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,511] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,511] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,511] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:57,512] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:13:57,513] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,561] jepsen worker 78 - jepsen.util 528 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,562] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:57,566] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,566] jepsen worker 44 - jepsen.util 584 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,566] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,568] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:57,569] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:13:57,570] jepsen worker 51 - jepsen.util 591 :invoke :write [17 35] INFO [2022-03-13 17:13:57,578] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,578] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:13:57,597] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,598] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:57,600] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,600] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,601] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:57,602] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,608] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,608] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,609] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:13:57,610] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 13] INFO [2022-03-13 17:13:57,614] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,614] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,615] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:13:57,615] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:57,620] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,620] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,620] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,620] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,621] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:13:57,621] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:13:57,622] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:13:57,634] jepsen worker 21 - jepsen.util 831 :invoke :write [7 48] WARN [2022-03-13 17:13:57,652] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17864464,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18792666,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9129039,"encode_duration_us":21401,"operation_id":"0x921e","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10004703}],"total_count":1}} INFO [2022-03-13 17:13:57,662] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,662] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:13:57,664] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:57,664] jepsen worker nemesis - couchbase.util Rebalance check count: 347 INFO [2022-03-13 17:13:57,664] jepsen worker nemesis - couchbase.util Rebalance stuck count: 247 INFO [2022-03-13 17:13:57,671] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,671] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,671] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,671] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,672] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,673] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:13:57,674] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 4] INFO [2022-03-13 17:13:57,675] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:13:57,676] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:13:57,679] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,679] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:57,681] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,681] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,682] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,683] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,684] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:57,687] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,687] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 22] INFO [2022-03-13 17:13:57,695] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,696] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,701] jepsen worker 43 - jepsen.util 493 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,701] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,702] jepsen worker 43 - jepsen.util 493 :invoke :write [14 9] INFO [2022-03-13 17:13:57,703] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:13:57,734] jepsen worker 69 - jepsen.util 699 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,734] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:13:57,740] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,741] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:13:57,752] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,752] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:13:57,762] jepsen worker 37 - jepsen.util 2017 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,763] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:57,793] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,793] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:13:57,817] jepsen worker 57 - jepsen.util 2487 :info :write [19 47] :SyncWriteAmbiguous INFO [2022-03-13 17:13:57,818] jepsen worker 57 - jepsen.util 2577 :invoke :read [19 nil] INFO [2022-03-13 17:13:57,832] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,832] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:13:57,845] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,845] jepsen worker 12 - jepsen.util 732 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,845] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,845] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:57,846] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:57,846] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:57,849] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,849] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,849] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,850] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:57,850] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:57,850] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:57,852] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:13:57,852] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9217547,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9217547,"total_server_duration_us":38,"operation_id":"0x97cd","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":10308461},{"operation_name":"upsert","last_dispatch_duration_us":32987,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":738769,"total_server_duration_us":268095,"encode_duration_us":19829,"operation_id":"0x951a","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10009482},{"operation_name":"upsert","last_dispatch_duration_us":9216771,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18214082,"total_server_duration_us":74865,"encode_duration_us":17395,"operation_id":"0x927f","timeout_ms":10000,"last_server_duration_us":32451,"total_duration_us":10008079},{"operation_name":"upsert","last_dispatch_duration_us":30724,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":864609,"total_server_duration_us":456350,"encode_duration_us":24483,"operation_id":"0x9719","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10004427},{"operation_name":"upsert","last_dispatch_duration_us":9223651,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18380571,"total_server_duration_us":88551,"encode_duration_us":23074,"operation_id":"0x926f","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10004376},{"operation_name":"upsert","last_dispatch_duration_us":15732219,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":15732219,"total_server_duration_us":21001,"encode_duration_us":17674,"operation_id":"0x941d","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10004026},{"operation_name":"upsert","last_dispatch_duration_us":9214733,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18195672,"total_server_duration_us":79988,"encode_duration_us":20235,"operation_id":"0x9289","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10001587},{"operation_name":"get","last_dispatch_duration_us":9209341,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9209341,"total_server_duration_us":38,"operation_id":"0x9870","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9211905},{"operation_name":"get","last_dispatch_duration_us":81956,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":81956,"total_server_duration_us":49,"operation_id":"0x9a0a","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":7292652},{"operation_name":"upsert","last_dispatch_duration_us":5377190,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5377190,"total_server_duration_us":58252,"encode_duration_us":36507,"operation_id":"0x9b8b","timeout_ms":10000,"last_server_duration_us":58252,"total_duration_us":5394731}],"total_count":50}} INFO [2022-03-13 17:13:57,852] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,852] jepsen worker 80 - jepsen.util 710 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,852] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,854] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,854] jepsen worker 80 - jepsen.util 710 :invoke :write [26 25] INFO [2022-03-13 17:13:57,855] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:57,855] jepsen worker 70 - jepsen.util 700 :invoke :write [23 3] INFO [2022-03-13 17:13:57,868] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,868] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:57,869] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,870] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:13:57,903] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,903] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:57,949] jepsen worker 28 - jepsen.util 2638 :info :write [9 22] :SyncWriteAmbiguous INFO [2022-03-13 17:13:57,949] jepsen worker 28 - jepsen.util 2728 :invoke :read [9 nil] WARN [2022-03-13 17:13:57,950] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11007785,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11007785,"total_server_duration_us":7437,"operation_id":"0x94df","timeout_ms":30000,"last_server_duration_us":7437,"total_duration_us":11009755},{"operation_name":"upsert","last_dispatch_duration_us":43962,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1791360,"total_server_duration_us":380124,"encode_duration_us":19519,"operation_id":"0x937f","timeout_ms":10000,"last_server_duration_us":28676,"total_duration_us":10013983},{"operation_name":"upsert","last_dispatch_duration_us":9044442,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10403206,"total_server_duration_us":9325799,"encode_duration_us":30094,"operation_id":"0x9373","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10010562},{"operation_name":"upsert","last_dispatch_duration_us":36906,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1550477,"total_server_duration_us":378772,"encode_duration_us":24789,"operation_id":"0x937b","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10007481},{"operation_name":"upsert","last_dispatch_duration_us":7372359,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8877595,"total_server_duration_us":400897,"encode_duration_us":26213,"operation_id":"0x9374","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10006855},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18119,"operation_id":"0x9852","timeout_ms":10000,"total_duration_us":10005479},{"operation_name":"get","last_dispatch_duration_us":9412958,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9412958,"total_server_duration_us":81684,"operation_id":"0x95dd","timeout_ms":30000,"last_server_duration_us":81684,"total_duration_us":9415593},{"operation_name":"get","last_dispatch_duration_us":9171893,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9171893,"total_server_duration_us":8393,"operation_id":"0x9625","timeout_ms":30000,"last_server_duration_us":8393,"total_duration_us":9176005},{"operation_name":"get","last_dispatch_duration_us":8676435,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8676435,"total_server_duration_us":8393,"operation_id":"0x966c","timeout_ms":30000,"last_server_duration_us":8393,"total_duration_us":8678844},{"operation_name":"upsert","last_dispatch_duration_us":1542139,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1542139,"total_server_duration_us":92438,"encode_duration_us":17423,"operation_id":"0x9b72","timeout_ms":10000,"last_server_duration_us":92438,"total_duration_us":1545813}],"total_count":27}} WARN [2022-03-13 17:13:57,951] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":6376896,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":6376896,"total_server_duration_us":22223,"encode_duration_us":26250,"operation_id":"0x9726","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10009915},{"operation_name":"upsert","last_dispatch_duration_us":38599,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":654330,"total_server_duration_us":375001,"encode_duration_us":19090,"operation_id":"0x97d2","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10008569},{"operation_name":"upsert","last_dispatch_duration_us":7787734,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8337855,"total_server_duration_us":353349,"encode_duration_us":18399,"operation_id":"0x957b","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10007373},{"operation_name":"upsert","last_dispatch_duration_us":9106518,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9951484,"total_server_duration_us":9601427,"encode_duration_us":17187,"operation_id":"0x946c","timeout_ms":10000,"last_server_duration_us":9081990,"total_duration_us":10006924},{"operation_name":"upsert","last_dispatch_duration_us":32997,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":966130,"total_server_duration_us":590495,"encode_duration_us":42323,"operation_id":"0x9436","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10005548},{"operation_name":"get","last_dispatch_duration_us":7648222,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7648222,"total_server_duration_us":6678,"operation_id":"0x964d","timeout_ms":30000,"last_server_duration_us":6678,"total_duration_us":7650653},{"operation_name":"upsert","last_dispatch_duration_us":1335510,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1335510,"total_server_duration_us":69481,"encode_duration_us":27088,"operation_id":"0x9b95","timeout_ms":10000,"last_server_duration_us":69481,"total_duration_us":1339706},{"operation_name":"upsert","last_dispatch_duration_us":545672,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":545672,"total_server_duration_us":125257,"encode_duration_us":19805,"operation_id":"0x98f3","timeout_ms":10000,"last_server_duration_us":125257,"total_duration_us":549250},{"operation_name":"upsert","last_dispatch_duration_us":527791,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":527791,"total_server_duration_us":21568,"encode_duration_us":18120,"operation_id":"0x9cb3","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":533419},{"operation_name":"upsert","last_dispatch_duration_us":520926,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":520926,"total_server_duration_us":51474,"encode_duration_us":18002,"operation_id":"0x9aea","timeout_ms":10000,"last_server_duration_us":51474,"total_duration_us":524330}],"total_count":10}} WARN [2022-03-13 17:13:57,951] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8724672,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10204315,"total_server_duration_us":544305,"encode_duration_us":17032,"operation_id":"0x946d","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10011102},{"operation_name":"upsert","last_dispatch_duration_us":8771251,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9248504,"total_server_duration_us":93412,"encode_duration_us":27981,"operation_id":"0x95c0","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10009467},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":17376,"operation_id":"0x9867","timeout_ms":10000,"total_duration_us":10008259},{"operation_name":"upsert","last_dispatch_duration_us":8756275,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":13103224,"total_server_duration_us":136476,"encode_duration_us":17906,"operation_id":"0x931c","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10006464},{"operation_name":"upsert","last_dispatch_duration_us":144773,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":322170,"total_server_duration_us":45859,"encode_duration_us":18655,"operation_id":"0x95d1","timeout_ms":10000,"last_server_duration_us":23222,"total_duration_us":10004031},{"operation_name":"upsert","last_dispatch_duration_us":8597192,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10122956,"total_server_duration_us":562444,"encode_duration_us":28065,"operation_id":"0x9434","timeout_ms":10000,"last_server_duration_us":42852,"total_duration_us":10002997},{"operation_name":"upsert","last_dispatch_duration_us":9201795,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9398695,"total_server_duration_us":9094405,"encode_duration_us":32270,"operation_id":"0x9513","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9417086},{"operation_name":"upsert","last_dispatch_duration_us":9127152,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9127152,"total_server_duration_us":9038428,"encode_duration_us":17267,"operation_id":"0x95cc","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":9130965},{"operation_name":"upsert","last_dispatch_duration_us":9055014,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9055014,"total_server_duration_us":9039489,"encode_duration_us":29712,"operation_id":"0x95ec","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":9062371},{"operation_name":"upsert","last_dispatch_duration_us":1939161,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1939161,"total_server_duration_us":82403,"encode_duration_us":28063,"operation_id":"0x9b80","timeout_ms":10000,"last_server_duration_us":82403,"total_duration_us":1948273}],"total_count":62}} WARN [2022-03-13 17:13:57,951] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16844019,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16844019,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9043735,"encode_duration_us":22137,"operation_id":"0x911f","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10002103},{"operation_name":"upsert","last_dispatch_duration_us":9044442,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10403206,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9325799,"encode_duration_us":30094,"operation_id":"0x9373","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10010562}],"total_count":2}} WARN [2022-03-13 17:13:57,956] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":6962656,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9786452,"total_server_duration_us":302833,"encode_duration_us":17745,"operation_id":"0x97d9","timeout_ms":10000,"last_server_duration_us":18570,"total_duration_us":10005689},{"operation_name":"upsert","last_dispatch_duration_us":7702993,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":15703425,"total_server_duration_us":254084,"encode_duration_us":18137,"operation_id":"0x92d6","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10004395},{"operation_name":"upsert","last_dispatch_duration_us":9147119,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9276803,"total_server_duration_us":9068084,"encode_duration_us":26586,"operation_id":"0x9948","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9294127},{"operation_name":"upsert","last_dispatch_duration_us":9084540,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9084540,"total_server_duration_us":9049045,"encode_duration_us":25974,"operation_id":"0x980a","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":9088588},{"operation_name":"get","last_dispatch_duration_us":7347195,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7347195,"total_server_duration_us":8612,"operation_id":"0x992d","timeout_ms":30000,"last_server_duration_us":8612,"total_duration_us":7349269},{"operation_name":"upsert","last_dispatch_duration_us":6556145,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":6556145,"total_server_duration_us":3252555,"encode_duration_us":17845,"operation_id":"0x94c4","timeout_ms":10000,"last_server_duration_us":3252555,"total_duration_us":6560212},{"operation_name":"upsert","last_dispatch_duration_us":5024781,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":5024781,"total_server_duration_us":104387,"encode_duration_us":18553,"operation_id":"0x9ae9","timeout_ms":10000,"last_server_duration_us":104387,"total_duration_us":6110958},{"operation_name":"get","last_dispatch_duration_us":4914658,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4914658,"total_server_duration_us":69,"operation_id":"0x9abe","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":5998342},{"operation_name":"get","last_dispatch_duration_us":5957792,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":5957792,"total_server_duration_us":12247,"operation_id":"0x9a19","timeout_ms":30000,"last_server_duration_us":12247,"total_duration_us":5960183},{"operation_name":"get","last_dispatch_duration_us":4767609,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4767609,"total_server_duration_us":186,"operation_id":"0x9ae4","timeout_ms":30000,"last_server_duration_us":186,"total_duration_us":5849767}],"total_count":48}} INFO [2022-03-13 17:13:57,964] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,964] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,964] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,964] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,965] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:13:57,966] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:13:57,967] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:13:57,967] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:13:57,977] jepsen worker 13 - jepsen.util 643 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:57,977] jepsen worker 13 - jepsen.util 643 :invoke :write [4 47] INFO [2022-03-13 17:13:58,026] jepsen worker 21 - jepsen.util 831 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,026] jepsen worker 21 - jepsen.util 831 :invoke :write [7 14] WARN [2022-03-13 17:13:58,051] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9181913,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9847225,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9428201,"encode_duration_us":18783,"operation_id":"0x90b4","timeout_ms":10000,"last_server_duration_us":9083054,"total_duration_us":10002821},{"operation_name":"upsert","last_dispatch_duration_us":9106518,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9951484,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9601427,"encode_duration_us":17187,"operation_id":"0x946c","timeout_ms":10000,"last_server_duration_us":9081990,"total_duration_us":10006924}],"total_count":2}} INFO [2022-03-13 17:13:58,099] jepsen worker 77 - jepsen.util 707 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,099] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,099] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:58,099] jepsen worker 53 - jepsen.util 503 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,099] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,110] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,128] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,164] jepsen worker 77 - jepsen.util 707 :invoke :write [25 48] INFO [2022-03-13 17:13:58,164] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:58,164] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:58,164] jepsen worker 42 - jepsen.util 402 :invoke :write [14 41] INFO [2022-03-13 17:13:58,164] jepsen worker 7 - jepsen.util 637 :invoke :write [2 26] INFO [2022-03-13 17:13:58,164] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:58,164] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:58,207] jepsen worker 73 - jepsen.util 2773 :info :write [24 42] :RequestTimeoutException INFO [2022-03-13 17:13:58,208] jepsen worker 73 - jepsen.util 2863 :invoke :write [24 12] INFO [2022-03-13 17:13:58,212] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,212] jepsen worker 34 - jepsen.util 754 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,212] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,212] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,213] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:58,214] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:58,214] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:58,215] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:58,231] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,232] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:13:58,263] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,263] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,263] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:58,264] jepsen worker 40 - jepsen.util 2650 :info :write [13 44] :RequestTimeoutException INFO [2022-03-13 17:13:58,264] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:13:58,265] jepsen worker 40 - jepsen.util 2740 :invoke :write [13 19] INFO [2022-03-13 17:13:58,271] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,271] jepsen worker 71 - jepsen.util 521 :invoke :write [23 10] INFO [2022-03-13 17:13:58,303] jepsen worker 0 - jepsen.util 2430 :info :write [0 33] :RequestTimeoutException INFO [2022-03-13 17:13:58,304] jepsen worker 0 - jepsen.util 2520 :invoke :read [0 nil] INFO [2022-03-13 17:13:58,341] jepsen worker 6 - jepsen.util 636 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,341] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:13:58,344] jepsen worker 70 - jepsen.util 700 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,345] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:58,433] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,433] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:58,436] jepsen worker 60 - jepsen.util 600 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,437] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:13:58,437] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:58,437] jepsen worker 44 - jepsen.util 584 :invoke :write [14 42] INFO [2022-03-13 17:13:58,461] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,461] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,461] jepsen worker 21 - jepsen.util 831 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,462] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:13:58,462] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:58,464] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:13:58,498] jepsen worker 80 - jepsen.util 710 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,499] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:13:58,548] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,548] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,548] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:13:58,549] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:13:58,553] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,554] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:58,555] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,556] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:13:58,560] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,561] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:13:58,584] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,584] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,585] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:58,585] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:13:58,588] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,589] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:58,591] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,591] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:13:58,595] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,595] jepsen worker 8 - jepsen.util 728 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,595] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,596] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:58,597] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 4] INFO [2022-03-13 17:13:58,598] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:13:58,610] jepsen worker 89 - jepsen.util 2429 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,610] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:13:58,611] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,611] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,611] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,612] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 0] INFO [2022-03-13 17:13:58,616] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,617] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:58,618] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,618] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:58,619] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,620] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:13:58,623] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,623] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,623] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:13:58,624] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,627] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,628] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,629] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,630] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,631] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,631] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,631] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,632] jepsen worker 23 - jepsen.util 563 :invoke :write [7 15] INFO [2022-03-13 17:13:58,637] jepsen worker 3 - jepsen.util 1893 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,637] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,638] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:13:58,639] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,642] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,642] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,642] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,642] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,642] jepsen worker 34 - jepsen.util 754 :invoke :write [11 23] INFO [2022-03-13 17:13:58,643] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,643] jepsen worker 13 - jepsen.util 643 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,643] jepsen worker 52 - jepsen.util 592 :invoke :write [17 42] INFO [2022-03-13 17:13:58,644] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:58,644] jepsen worker 80 - jepsen.util 710 :invoke :write [26 10] INFO [2022-03-13 17:13:58,645] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:58,646] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:58,652] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,652] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,652] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,653] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:13:58,653] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:58,653] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:58,654] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,654] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,654] jepsen worker 78 - jepsen.util 528 :invoke :write [26 21] INFO [2022-03-13 17:13:58,655] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:58,655] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,655] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:13:58,671] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:58,671] jepsen worker nemesis - couchbase.util Rebalance check count: 348 INFO [2022-03-13 17:13:58,671] jepsen worker nemesis - couchbase.util Rebalance stuck count: 248 INFO [2022-03-13 17:13:58,672] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,672] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,673] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:13:58,674] jepsen worker 70 - jepsen.util 700 :invoke :write [23 25] INFO [2022-03-13 17:13:58,691] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,692] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:13:58,722] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,723] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:58,724] jepsen worker 71 - jepsen.util 521 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,724] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,725] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:58,725] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:13:58,727] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,728] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:13:58,752] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,753] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:58,763] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,763] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:13:58,767] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,768] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:58,795] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,795] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,795] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,795] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,796] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:58,796] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,796] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:58,797] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:58,797] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,797] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,798] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:13:58,798] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,798] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:13:58,800] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:58,801] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,801] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,802] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,802] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:13:58,803] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:13:58,807] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,809] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,809] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,810] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,811] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:13:58,818] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,819] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,819] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,819] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:13:58,822] jepsen worker 14 - jepsen.util 374 :invoke :write [4 6] INFO [2022-03-13 17:13:58,822] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:13:58,827] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,828] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:13:58,853] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,853] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,854] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:58,854] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:58,857] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,857] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:13:58,865] jepsen worker 43 - jepsen.util 493 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,866] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,866] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,866] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,866] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,866] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,866] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,866] jepsen worker 51 - jepsen.util 591 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,868] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:13:58,868] jepsen worker 12 - jepsen.util 732 :invoke :write [4 33] INFO [2022-03-13 17:13:58,869] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:58,870] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:13:58,871] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:58,872] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,872] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:58,873] jepsen worker 69 - jepsen.util 699 :invoke :write [23 0] INFO [2022-03-13 17:13:58,874] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:13:58,875] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,881] jepsen worker 34 - jepsen.util 754 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,882] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:58,920] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,921] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:58,923] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,924] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:58,934] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,934] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:13:58,956] jepsen worker 27 - jepsen.util 3087 :info :write [9 5] :RequestTimeoutException INFO [2022-03-13 17:13:58,956] jepsen worker 27 - jepsen.util 3177 :invoke :write [9 37] INFO [2022-03-13 17:13:58,969] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,970] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:58,970] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,970] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:13:58,999] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:58,999] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,000] jepsen worker 71 - jepsen.util 521 :invoke :write [23 3] INFO [2022-03-13 17:13:59,000] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:59,004] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,005] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:13:59,042] jepsen worker 14 - jepsen.util 374 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,043] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:13:59,046] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,046] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,046] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,047] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,047] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,048] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:13:59,055] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,056] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:13:59,075] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,075] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,075] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,075] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,076] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,076] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,117] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,117] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,117] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,117] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,117] jepsen worker 80 - jepsen.util 710 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,118] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,118] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,119] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,119] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,119] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,119] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:13:59,120] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,123] jepsen worker 77 - jepsen.util 707 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,123] jepsen worker 7 - jepsen.util 637 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,123] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,124] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:13:59,125] jepsen worker 80 - jepsen.util 710 :invoke :write [26 17] INFO [2022-03-13 17:13:59,128] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:59,129] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:13:59,129] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:13:59,130] jepsen worker 13 - jepsen.util 643 :invoke :write [4 41] INFO [2022-03-13 17:13:59,130] jepsen worker 7 - jepsen.util 637 :invoke :write [2 38] INFO [2022-03-13 17:13:59,131] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:13:59,134] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:13:59,136] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 25] INFO [2022-03-13 17:13:59,139] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,151] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:13:59,152] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,180] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,180] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,181] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,182] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,187] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,188] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,192] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,192] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:13:59,199] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,199] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:13:59,203] jepsen worker 71 - jepsen.util 521 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,203] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,204] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:59,204] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:13:59,208] jepsen worker 3 - jepsen.util 1893 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,208] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:13:59,218] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,219] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:59,261] jepsen worker 42 - jepsen.util 402 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,261] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,262] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,262] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,262] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,262] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:59,263] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,263] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,264] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:13:59,267] jepsen worker 6 - jepsen.util 636 :invoke :write [2 37] INFO [2022-03-13 17:13:59,268] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 48] INFO [2022-03-13 17:13:59,268] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,269] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:13:59,270] jepsen worker 8 - jepsen.util 728 :invoke :write [2 12] INFO [2022-03-13 17:13:59,292] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,293] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:59,300] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,301] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,302] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,304] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,307] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,308] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,313] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,314] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,316] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,320] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,322] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,323] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:13:59,331] jepsen worker 45 - jepsen.util 2925 :ok :read [15 48] INFO [2022-03-13 17:13:59,331] jepsen worker 45 - jepsen.util 2925 :invoke :read [15 nil] INFO [2022-03-13 17:13:59,340] jepsen worker 64 - jepsen.util 2584 :ok :read [21 :nil] INFO [2022-03-13 17:13:59,340] jepsen worker 64 - jepsen.util 2584 :invoke :read [21 nil] INFO [2022-03-13 17:13:59,343] jepsen worker 11 - jepsen.util 2531 :ok :read [3 25] INFO [2022-03-13 17:13:59,344] jepsen worker 11 - jepsen.util 2531 :invoke :read [3 nil] INFO [2022-03-13 17:13:59,373] jepsen worker 64 - jepsen.util 2584 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,373] jepsen worker 64 - jepsen.util 2584 :invoke :read [21 nil] INFO [2022-03-13 17:13:59,376] jepsen worker 52 - jepsen.util 592 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,376] jepsen worker 44 - jepsen.util 584 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,377] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:13:59,377] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:13:59,380] jepsen worker 54 - jepsen.util 2214 :ok :read [18 21] INFO [2022-03-13 17:13:59,380] jepsen worker 54 - jepsen.util 2214 :invoke :write [18 23] INFO [2022-03-13 17:13:59,387] jepsen worker 45 - jepsen.util 2925 :ok :read [15 48] INFO [2022-03-13 17:13:59,388] jepsen worker 45 - jepsen.util 2925 :invoke :read [15 nil] INFO [2022-03-13 17:13:59,390] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,390] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,391] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,392] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,392] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,393] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:13:59,404] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,404] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,405] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,405] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,406] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:13:59,408] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:13:59,409] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,410] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:13:59,410] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:13:59,414] jepsen worker 64 - jepsen.util 2584 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,414] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:13:59,416] jepsen worker 64 - jepsen.util 2584 :invoke :write [21 24] INFO [2022-03-13 17:13:59,438] jepsen worker 11 - jepsen.util 2531 :ok :read [3 25] INFO [2022-03-13 17:13:59,439] jepsen worker 11 - jepsen.util 2531 :invoke :write [3 45] INFO [2022-03-13 17:13:59,448] jepsen worker 45 - jepsen.util 2925 :ok :read [15 48] INFO [2022-03-13 17:13:59,449] jepsen worker 45 - jepsen.util 2925 :invoke :read [15 nil] INFO [2022-03-13 17:13:59,455] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,456] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,480] jepsen worker 79 - jepsen.util 1069 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,481] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,481] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,481] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:13:59,481] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 31] INFO [2022-03-13 17:13:59,482] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:13:59,483] jepsen worker 53 - jepsen.util 503 :invoke :write [17 14] INFO [2022-03-13 17:13:59,483] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:13:59,491] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,491] jepsen worker 37 - jepsen.util 2017 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,492] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:13:59,492] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,501] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,501] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,503] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:13:59,508] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:13:59,513] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,514] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,526] jepsen worker 45 - jepsen.util 2925 :ok :read [15 48] INFO [2022-03-13 17:13:59,526] jepsen worker 45 - jepsen.util 2925 :invoke :write [15 24] INFO [2022-03-13 17:13:59,531] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,531] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,534] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,535] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,538] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,538] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:13:59,566] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,566] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:59,570] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,571] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:13:59,572] jepsen worker 88 - jepsen.util 2068 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,572] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,573] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,574] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,574] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:13:59,574] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,574] jepsen worker 23 - jepsen.util 563 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,575] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,575] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,575] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,575] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,576] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 34] INFO [2022-03-13 17:13:59,578] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:13:59,579] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:13:59,580] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,580] jepsen worker 51 - jepsen.util 591 :invoke :write [17 31] INFO [2022-03-13 17:13:59,581] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:13:59,581] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,582] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,583] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,584] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:13:59,585] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,586] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:13:59,590] jepsen worker 76 - jepsen.util 706 :invoke :write [25 18] INFO [2022-03-13 17:13:59,597] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,599] jepsen worker 21 - jepsen.util 831 :invoke :write [7 22] INFO [2022-03-13 17:13:59,604] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,604] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,604] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,606] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,613] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,613] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:13:59,614] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,614] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:13:59,656] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,657] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,659] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,659] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,660] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,661] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,667] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,667] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,668] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,668] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:13:59,669] jepsen worker 24 - jepsen.util 564 :invoke :write [8 4] INFO [2022-03-13 17:13:59,669] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:59,673] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,673] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,674] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:13:59,674] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:13:59,677] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:13:59,678] jepsen worker nemesis - couchbase.util Rebalance check count: 349 INFO [2022-03-13 17:13:59,678] jepsen worker nemesis - couchbase.util Rebalance stuck count: 249 INFO [2022-03-13 17:13:59,684] jepsen worker 78 - jepsen.util 528 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,685] jepsen worker 78 - jepsen.util 528 :invoke :write [26 24] INFO [2022-03-13 17:13:59,726] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,727] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,755] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,756] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:13:59,810] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,810] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:13:59,818] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,818] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:13:59,820] jepsen worker 65 - jepsen.util 2945 :info :write [21 11] :RequestTimeoutException INFO [2022-03-13 17:13:59,820] jepsen worker 65 - jepsen.util 3035 :invoke :read [21 nil] INFO [2022-03-13 17:13:59,821] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:13:59,821] jepsen worker 12 - jepsen.util 732 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,821] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,822] jepsen worker 26 - jepsen.util 476 :invoke :write [8 18] INFO [2022-03-13 17:13:59,822] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:13:59,823] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:13:59,826] jepsen worker 7 - jepsen.util 637 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,827] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,828] jepsen worker 7 - jepsen.util 637 :invoke :write [2 25] INFO [2022-03-13 17:13:59,829] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:13:59,830] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,831] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:13:59,846] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,846] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,846] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:13:59,846] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:13:59,879] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,880] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:13:59,905] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,906] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:13:59,912] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,913] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:13:59,961] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,961] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:13:59,963] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,963] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:13:59,976] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,976] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:13:59,980] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,981] jepsen worker 14 - jepsen.util 374 :invoke :write [4 46] INFO [2022-03-13 17:13:59,983] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,984] jepsen worker 70 - jepsen.util 700 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,984] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,984] jepsen worker 70 - jepsen.util 700 :invoke :write [23 30] INFO [2022-03-13 17:13:59,985] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:13:59,986] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:13:59,987] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,988] jepsen worker 22 - jepsen.util 652 :invoke :write [7 31] INFO [2022-03-13 17:13:59,996] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,996] jepsen worker 87 - jepsen.util 2607 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,997] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:13:59,997] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:13:59,998] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:14:00,008] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:14:00,016] jepsen worker 24 - jepsen.util 564 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,016] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,016] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,016] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,017] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:00,017] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:00,018] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,018] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:14:00,019] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:14:00,020] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,021] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,021] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,021] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:14:00,022] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:14:00,038] jepsen worker 69 - jepsen.util 699 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,039] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,039] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,039] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:00,039] jepsen worker 6 - jepsen.util 636 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,039] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,039] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,039] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:00,041] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:00,042] jepsen worker 44 - jepsen.util 584 :invoke :write [14 17] INFO [2022-03-13 17:14:00,042] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:14:00,043] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:00,044] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:14:00,048] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:00,080] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,080] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,081] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:14:00,082] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:14:00,094] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,095] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:00,095] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,096] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 12] INFO [2022-03-13 17:14:00,119] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,119] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:14:00,135] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,136] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:14:00,154] jepsen worker 13 - jepsen.util 643 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,154] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:00,164] jepsen worker 8 - jepsen.util 728 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,164] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:14:00,164] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,165] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:00,166] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:00,166] jepsen worker 43 - jepsen.util 493 :invoke :write [14 16] INFO [2022-03-13 17:14:00,175] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,175] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,175] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:00,176] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,179] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,180] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,181] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,182] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:14:00,201] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,201] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,203] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,204] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:14:00,223] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,223] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:14:00,267] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,267] jepsen worker 37 - jepsen.util 2017 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,267] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,268] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:14:00,269] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,269] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:14:00,272] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,272] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,272] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,273] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,273] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,274] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:14:00,277] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,278] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,279] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,279] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,281] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,281] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,283] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,284] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:00,302] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,302] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 10] INFO [2022-03-13 17:14:00,316] jepsen worker 76 - jepsen.util 706 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,317] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:00,332] jepsen worker 21 - jepsen.util 831 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,332] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,333] jepsen worker 21 - jepsen.util 831 :invoke :write [7 29] INFO [2022-03-13 17:14:00,334] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:00,336] jepsen worker 22 - jepsen.util 652 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,336] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:00,338] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,338] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:00,343] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,343] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:14:00,367] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,368] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:14:00,389] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,389] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:00,432] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,433] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:14:00,459] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,460] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,461] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,461] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:14:00,500] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,501] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,509] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,510] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,510] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,510] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:00,511] jepsen worker 14 - jepsen.util 374 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,512] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:00,528] jepsen worker 80 - jepsen.util 710 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,528] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,528] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,529] jepsen worker 52 - jepsen.util 592 :invoke :write [17 39] INFO [2022-03-13 17:14:00,529] jepsen worker 80 - jepsen.util 710 :invoke :write [26 25] INFO [2022-03-13 17:14:00,534] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:00,556] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,557] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,559] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,559] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,560] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,563] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:00,586] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,586] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:14:00,591] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,591] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:00,593] jepsen worker 53 - jepsen.util 503 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,594] jepsen worker 53 - jepsen.util 503 :invoke :write [17 47] INFO [2022-03-13 17:14:00,596] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,596] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,597] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:14:00,597] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:14:00,606] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,607] jepsen worker 78 - jepsen.util 528 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,607] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,607] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,607] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:14:00,607] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,608] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:00,608] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,609] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:00,609] jepsen worker 34 - jepsen.util 754 :invoke :write [11 24] INFO [2022-03-13 17:14:00,612] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,616] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,617] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:14:00,617] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,626] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,626] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,633] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,633] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,655] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,655] jepsen worker 3 - jepsen.util 1893 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,656] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,656] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:14:00,657] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:14:00,657] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:00,660] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,662] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,664] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,664] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:00,673] jepsen worker 51 - jepsen.util 591 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,673] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,674] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,674] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,674] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,675] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:00,675] jepsen worker 60 - jepsen.util 600 :invoke :write [20 23] INFO [2022-03-13 17:14:00,676] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:00,677] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:00,677] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:00,687] jepsen worker 39 - jepsen.util 2559 :info :write [13 10] :SyncWriteAmbiguous INFO [2022-03-13 17:14:00,688] jepsen worker 39 - jepsen.util 2649 :invoke :read [13 nil] INFO [2022-03-13 17:14:00,691] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:00,691] jepsen worker nemesis - couchbase.util Rebalance check count: 350 INFO [2022-03-13 17:14:00,691] jepsen worker nemesis - couchbase.util Rebalance stuck count: 250 INFO [2022-03-13 17:14:00,692] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,693] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:00,695] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,696] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:14:00,710] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,711] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:00,715] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,715] jepsen worker 21 - jepsen.util 831 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,716] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:14:00,717] jepsen worker 14 - jepsen.util 374 :invoke :write [4 27] INFO [2022-03-13 17:14:00,720] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,720] jepsen worker 70 - jepsen.util 700 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,721] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:00,721] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:00,761] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,761] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,762] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,762] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:14:00,765] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,765] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,766] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,766] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,767] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:14:00,767] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,775] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,775] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:14:00,782] jepsen worker 50 - jepsen.util 2750 :ok :read [16 :nil] INFO [2022-03-13 17:14:00,782] jepsen worker 15 - jepsen.util 2715 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,783] jepsen worker 15 - jepsen.util 2715 :invoke :write [5 3] INFO [2022-03-13 17:14:00,788] jepsen worker 32 - jepsen.util 2552 :info :write [10 20] :RequestTimeoutException INFO [2022-03-13 17:14:00,788] jepsen worker 50 - jepsen.util 2750 :invoke :write [16 15] INFO [2022-03-13 17:14:00,788] jepsen worker 32 - jepsen.util 2642 :invoke :write [10 45] INFO [2022-03-13 17:14:00,794] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,795] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:14:00,816] jepsen worker 20 - jepsen.util 2990 :info :write [6 48] :RequestTimeoutException INFO [2022-03-13 17:14:00,817] jepsen worker 20 - jepsen.util 3080 :invoke :write [6 0] INFO [2022-03-13 17:14:00,839] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,840] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,845] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,845] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,847] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,847] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,849] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,849] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,855] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,855] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:00,857] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,857] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:14:00,858] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,858] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 48] INFO [2022-03-13 17:14:00,873] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,874] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,876] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,876] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:14:00,900] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,901] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:14:00,924] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,924] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,924] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,924] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,925] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:00,926] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:14:00,926] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:00,927] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,927] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:14:00,928] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:00,928] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,929] jepsen worker 24 - jepsen.util 564 :invoke :write [8 13] INFO [2022-03-13 17:14:00,929] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,930] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:00,931] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,932] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:00,932] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,933] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,933] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:14:00,936] jepsen worker 80 - jepsen.util 710 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,938] jepsen worker 63 - jepsen.util 2493 :info :write [21 13] :RequestTimeoutException INFO [2022-03-13 17:14:00,939] jepsen worker 63 - jepsen.util 2583 :invoke :write [21 41] INFO [2022-03-13 17:14:00,940] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:00,944] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:00,963] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,963] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,964] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:14:00,966] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:00,976] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,977] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,979] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,979] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,980] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,981] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,982] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,983] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,987] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,989] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:00,992] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:00,993] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,033] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,033] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:14:01,041] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,041] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,042] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,043] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:14:01,043] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,043] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,044] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,045] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,046] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,047] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:14:01,057] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,058] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,070] jepsen worker 88 - jepsen.util 2068 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,071] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,071] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:14:01,072] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:14:01,120] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,120] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:14:01,133] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,133] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:01,133] jepsen worker 79 - jepsen.util 1069 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,134] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:01,135] jepsen worker 42 - jepsen.util 402 :invoke :write [14 44] INFO [2022-03-13 17:14:01,135] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:01,156] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,156] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 20] INFO [2022-03-13 17:14:01,167] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,168] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:14:01,177] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,178] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:14:01,187] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,188] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:14:01,245] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,245] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,247] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,247] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,247] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:14:01,247] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:01,274] jepsen worker 68 - jepsen.util 2678 :info :write [22 45] :RequestTimeoutException INFO [2022-03-13 17:14:01,275] jepsen worker 68 - jepsen.util 2768 :invoke :write [22 36] INFO [2022-03-13 17:14:01,290] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,290] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,290] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,291] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:01,292] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,292] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 5] INFO [2022-03-13 17:14:01,292] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,293] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:14:01,301] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,302] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,318] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,319] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:14:01,322] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,323] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:14:01,336] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,336] jepsen worker 52 - jepsen.util 592 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,336] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,336] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,337] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:01,338] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:14:01,339] jepsen worker 52 - jepsen.util 592 :invoke :write [17 30] INFO [2022-03-13 17:14:01,339] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:14:01,391] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,392] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:14:01,393] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,394] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:01,400] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,401] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:14:01,425] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,425] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,426] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:14:01,427] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,433] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,434] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:14:01,446] jepsen worker 46 - jepsen.util 2566 :ok :read [15 48] INFO [2022-03-13 17:14:01,450] jepsen worker 46 - jepsen.util 2566 :invoke :read [15 nil] INFO [2022-03-13 17:14:01,491] jepsen worker 87 - jepsen.util 2607 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,491] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,492] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:14:01,492] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:01,516] jepsen worker 41 - jepsen.util 2471 :info :write [13 12] :RequestTimeoutException INFO [2022-03-13 17:14:01,517] jepsen worker 41 - jepsen.util 2561 :invoke :write [13 49] INFO [2022-03-13 17:14:01,531] jepsen worker 37 - jepsen.util 2017 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,532] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:01,539] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,540] jepsen worker 71 - jepsen.util 521 :invoke :write [23 0] INFO [2022-03-13 17:14:01,561] jepsen worker 34 - jepsen.util 754 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,561] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,561] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,561] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,561] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,562] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:01,562] jepsen worker 34 - jepsen.util 754 :invoke :write [11 28] INFO [2022-03-13 17:14:01,563] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:01,564] jepsen worker 80 - jepsen.util 710 :invoke :write [26 9] INFO [2022-03-13 17:14:01,564] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:01,577] jepsen worker 49 - jepsen.util 2299 :info :write [16 38] :RequestTimeoutException INFO [2022-03-13 17:14:01,577] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:01,582] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,583] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:14:01,602] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,603] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:01,647] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,647] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:01,657] jepsen worker 2 - jepsen.util 2972 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,657] jepsen worker 1 - jepsen.util 2341 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,658] jepsen worker 2 - jepsen.util 2972 :invoke :write [0 27] INFO [2022-03-13 17:14:01,658] jepsen worker 1 - jepsen.util 2341 :invoke :write [0 40] INFO [2022-03-13 17:14:01,660] jepsen worker 81 - jepsen.util 2961 :info :write [27 34] :RequestTimeoutException INFO [2022-03-13 17:14:01,661] jepsen worker 81 - jepsen.util 3051 :invoke :write [27 12] INFO [2022-03-13 17:14:01,681] jepsen worker 47 - jepsen.util 2567 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,682] jepsen worker 47 - jepsen.util 2567 :invoke :read [15 nil] INFO [2022-03-13 17:14:01,683] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,685] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:14:01,697] jepsen worker 46 - jepsen.util 2566 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,698] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:01,699] jepsen worker nemesis - couchbase.util Rebalance check count: 351 INFO [2022-03-13 17:14:01,699] jepsen worker 46 - jepsen.util 2566 :invoke :write [15 26] INFO [2022-03-13 17:14:01,699] jepsen worker nemesis - couchbase.util Rebalance stuck count: 251 INFO [2022-03-13 17:14:01,712] jepsen worker 28 - jepsen.util 2728 :ok :read [9 26] INFO [2022-03-13 17:14:01,712] jepsen worker 28 - jepsen.util 2728 :invoke :write [9 4] INFO [2022-03-13 17:14:01,720] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,720] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,720] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,721] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:14:01,721] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:14:01,722] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 16] INFO [2022-03-13 17:14:01,728] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,729] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 13] INFO [2022-03-13 17:14:01,740] jepsen worker 14 - jepsen.util 374 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,741] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:14:01,747] jepsen worker 83 - jepsen.util 2063 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,747] jepsen worker 83 - jepsen.util 2063 :invoke :read [27 nil] INFO [2022-03-13 17:14:01,786] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,787] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:01,822] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,822] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:01,858] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,859] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:14:01,860] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,861] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 45] INFO [2022-03-13 17:14:01,899] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,900] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:01,901] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,901] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:01,958] jepsen worker 83 - jepsen.util 2063 :ok :read [27 4] INFO [2022-03-13 17:14:01,959] jepsen worker 83 - jepsen.util 2063 :invoke :write [27 43] INFO [2022-03-13 17:14:01,959] jepsen worker 89 - jepsen.util 2429 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,960] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,960] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:01,961] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:01,963] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,963] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:14:01,967] jepsen worker 44 - jepsen.util 584 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,967] jepsen worker 26 - jepsen.util 476 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,968] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:01,969] jepsen worker 44 - jepsen.util 584 :invoke :write [14 18] INFO [2022-03-13 17:14:01,969] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,970] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 16] INFO [2022-03-13 17:14:01,973] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,973] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:01,996] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,996] jepsen worker 3 - jepsen.util 1893 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,996] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,996] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,996] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:01,997] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:01,998] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:01,998] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:14:01,999] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,000] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:14:02,000] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:14:02,001] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,003] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,004] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:14:02,026] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,027] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,064] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,065] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:14:02,116] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,116] jepsen worker 37 - jepsen.util 2017 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,117] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:14:02,117] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:14:02,131] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,131] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:14:02,157] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,158] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,159] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,160] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:14:02,165] jepsen worker 24 - jepsen.util 564 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,165] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:02,167] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,168] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:02,196] jepsen worker 89 - jepsen.util 2429 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,196] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 19] INFO [2022-03-13 17:14:02,197] jepsen worker 66 - jepsen.util 2496 :info :write [22 14] :RequestTimeoutException INFO [2022-03-13 17:14:02,198] jepsen worker 66 - jepsen.util 2586 :invoke :write [22 24] INFO [2022-03-13 17:14:02,211] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,211] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,213] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,213] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,213] jepsen worker 52 - jepsen.util 592 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,214] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,215] jepsen worker 13 - jepsen.util 643 :invoke :write [4 32] INFO [2022-03-13 17:14:02,215] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:02,216] jepsen worker 70 - jepsen.util 700 :invoke :write [23 24] INFO [2022-03-13 17:14:02,217] jepsen worker 52 - jepsen.util 592 :invoke :write [17 11] INFO [2022-03-13 17:14:02,232] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,233] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:02,279] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,280] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:14:02,298] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,299] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:14:02,333] jepsen worker 58 - jepsen.util 2308 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,334] jepsen worker 58 - jepsen.util 2308 :invoke :write [19 25] INFO [2022-03-13 17:14:02,353] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,354] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,355] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,356] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:02,374] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,375] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:14:02,375] jepsen worker 7 - jepsen.util 637 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,376] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,376] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,376] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,376] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,382] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:14:02,383] jepsen worker 7 - jepsen.util 637 :invoke :write [2 18] INFO [2022-03-13 17:14:02,383] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:02,383] jepsen worker 6 - jepsen.util 636 :invoke :write [2 42] INFO [2022-03-13 17:14:02,384] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:02,385] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,385] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:14:02,436] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,436] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,440] jepsen worker 34 - jepsen.util 754 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,441] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:02,441] jepsen worker 89 - jepsen.util 2429 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,442] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,443] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,443] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,443] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,444] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,444] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,444] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,444] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:02,445] jepsen worker 77 - jepsen.util 707 :invoke :write [25 7] INFO [2022-03-13 17:14:02,445] jepsen worker 12 - jepsen.util 732 :invoke :write [4 3] INFO [2022-03-13 17:14:02,446] jepsen worker 61 - jepsen.util 691 :invoke :write [20 2] INFO [2022-03-13 17:14:02,446] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:02,447] jepsen worker 69 - jepsen.util 699 :invoke :write [23 7] INFO [2022-03-13 17:14:02,457] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,457] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:14:02,461] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,461] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,467] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,467] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:14:02,491] jepsen worker 85 - jepsen.util 2425 :ok :read [28 44] INFO [2022-03-13 17:14:02,491] jepsen worker 85 - jepsen.util 2425 :invoke :read [28 nil] INFO [2022-03-13 17:14:02,502] jepsen worker 39 - jepsen.util 2649 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,503] jepsen worker 39 - jepsen.util 2649 :invoke :read [13 nil] INFO [2022-03-13 17:14:02,504] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:02,504] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:02,515] jepsen worker 86 - jepsen.util 2426 :ok :read [28 44] INFO [2022-03-13 17:14:02,515] jepsen worker 86 - jepsen.util 2426 :invoke :read [28 nil] INFO [2022-03-13 17:14:02,539] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,540] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:02,548] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,548] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,548] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,549] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:14:02,566] jepsen worker 16 - jepsen.util 2356 :info :write [5 5] :RequestTimeoutException INFO [2022-03-13 17:14:02,567] jepsen worker 16 - jepsen.util 2446 :invoke :read [5 nil] INFO [2022-03-13 17:14:02,572] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,573] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:02,587] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,588] jepsen worker 21 - jepsen.util 831 :invoke :write [7 29] INFO [2022-03-13 17:14:02,589] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,590] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:14:02,593] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,594] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:14:02,632] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,633] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 40] INFO [2022-03-13 17:14:02,642] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,642] jepsen worker 71 - jepsen.util 521 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,642] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:14:02,643] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:14:02,658] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,658] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:02,662] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,663] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:14:02,664] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,665] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,665] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,665] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:14:02,666] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,666] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,701] jepsen worker 48 - jepsen.util 2478 :info :write [16 11] :RequestTimeoutException INFO [2022-03-13 17:14:02,701] jepsen worker 48 - jepsen.util 2568 :invoke :write [16 45] INFO [2022-03-13 17:14:02,705] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:02,705] jepsen worker nemesis - couchbase.util Rebalance check count: 352 INFO [2022-03-13 17:14:02,705] jepsen worker nemesis - couchbase.util Rebalance stuck count: 252 INFO [2022-03-13 17:14:02,706] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,707] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,712] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,713] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:14:02,719] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,720] jepsen worker 59 - jepsen.util 2579 :info :write [19 36] :RequestTimeoutException INFO [2022-03-13 17:14:02,721] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:14:02,721] jepsen worker 59 - jepsen.util 2669 :invoke :read [19 nil] INFO [2022-03-13 17:14:02,760] jepsen worker 82 - jepsen.util 2602 :info :write [27 20] :RequestTimeoutException INFO [2022-03-13 17:14:02,761] jepsen worker 82 - jepsen.util 2692 :invoke :read [27 nil] INFO [2022-03-13 17:14:02,784] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,784] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,784] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:02,785] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:14:02,785] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:02,785] jepsen worker 84 - jepsen.util 2784 :info :write [28 22] :RequestTimeoutException INFO [2022-03-13 17:14:02,786] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:02,787] jepsen worker 84 - jepsen.util 2874 :invoke :read [28 nil] INFO [2022-03-13 17:14:02,788] jepsen worker 43 - jepsen.util 493 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,788] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:14:02,790] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,791] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:02,792] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,792] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,792] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,792] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,792] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:02,792] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,793] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,793] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:02,796] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:02,796] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,797] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,797] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:14:02,799] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,800] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,803] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,805] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,807] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,809] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:14:02,819] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,820] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,828] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,829] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:14:02,838] jepsen worker 39 - jepsen.util 2649 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,839] jepsen worker 39 - jepsen.util 2649 :invoke :read [13 nil] INFO [2022-03-13 17:14:02,842] jepsen worker 37 - jepsen.util 2017 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,842] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:14:02,847] jepsen worker 85 - jepsen.util 2425 :ok :read [28 44] INFO [2022-03-13 17:14:02,847] jepsen worker 85 - jepsen.util 2425 :invoke :read [28 nil] INFO [2022-03-13 17:14:02,849] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,850] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:02,864] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,864] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,865] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,865] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:14:02,865] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,866] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:14:02,911] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,912] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:02,917] jepsen worker 86 - jepsen.util 2426 :ok :read [28 44] INFO [2022-03-13 17:14:02,918] jepsen worker 86 - jepsen.util 2426 :invoke :write [28 16] INFO [2022-03-13 17:14:02,928] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,928] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,929] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,930] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,933] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,933] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,939] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,940] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,945] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,945] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,946] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:02,946] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,951] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,952] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,955] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,955] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:02,965] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,966] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:02,968] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,968] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:14:02,989] jepsen worker 80 - jepsen.util 710 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,989] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,990] jepsen worker 78 - jepsen.util 528 :invoke :write [26 36] INFO [2022-03-13 17:14:02,990] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:02,996] jepsen worker 60 - jepsen.util 600 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,996] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:02,997] jepsen worker 60 - jepsen.util 600 :invoke :write [20 15] INFO [2022-03-13 17:14:02,997] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:03,019] jepsen worker 82 - jepsen.util 2692 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,020] jepsen worker 82 - jepsen.util 2692 :invoke :read [27 nil] INFO [2022-03-13 17:14:03,020] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,021] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:14:03,054] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,054] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,055] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,056] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:14:03,056] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,057] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:14:03,069] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,070] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:14:03,072] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:03,073] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:03,081] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,081] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,081] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,081] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 15] INFO [2022-03-13 17:14:03,082] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:14:03,091] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:14:03,142] jepsen worker 85 - jepsen.util 2425 :ok :read [28 44] INFO [2022-03-13 17:14:03,143] jepsen worker 85 - jepsen.util 2425 :invoke :read [28 nil] INFO [2022-03-13 17:14:03,151] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,153] jepsen worker 39 - jepsen.util 2649 :ok :read [13 39] INFO [2022-03-13 17:14:03,156] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,604] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,165] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:03,604] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:14:03,185] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,604] jepsen worker 39 - jepsen.util 2649 :invoke :read [13 nil] INFO [2022-03-13 17:14:03,191] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,193] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,193] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,216] jepsen worker 3 - jepsen.util 1893 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,222] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,604] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:14:03,604] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:03,223] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,604] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:03,604] jepsen worker 24 - jepsen.util 564 :invoke :write [8 38] INFO [2022-03-13 17:14:03,226] jepsen worker 84 - jepsen.util 2874 :ok :read [28 44] INFO [2022-03-13 17:14:03,229] jepsen worker 82 - jepsen.util 2692 :ok :read [27 4] INFO [2022-03-13 17:14:03,605] jepsen worker 84 - jepsen.util 2874 :invoke :write [28 46] INFO [2022-03-13 17:14:03,605] jepsen worker 82 - jepsen.util 2692 :invoke :read [27 nil] INFO [2022-03-13 17:14:03,257] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,606] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,606] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:03,606] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,607] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,608] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:03,608] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,609] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,284] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,284] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,610] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:14:03,289] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,610] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:14:03,321] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,338] jepsen worker 19 - jepsen.util 2449 :info :write [6 33] :RequestTimeoutException INFO [2022-03-13 17:14:03,610] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:14:03,610] jepsen worker 19 - jepsen.util 2539 :invoke :read [6 nil] INFO [2022-03-13 17:14:03,611] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,612] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,613] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,396] jepsen worker 70 - jepsen.util 700 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,613] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:03,449] jepsen worker 21 - jepsen.util 831 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,449] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,456] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,616] jepsen worker 14 - jepsen.util 374 :invoke :write [4 21] INFO [2022-03-13 17:14:03,478] jepsen worker 53 - jepsen.util 503 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,616] jepsen worker 22 - jepsen.util 652 :invoke :write [7 31] INFO [2022-03-13 17:14:03,616] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,616] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:03,478] jepsen worker 44 - jepsen.util 584 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,478] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,616] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:03,479] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,479] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,616] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:03,616] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:03,600] jepsen worker 13 - jepsen.util 643 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,600] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,604] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:03,622] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:03,604] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:03,604] jepsen worker 33 - jepsen.util 663 :invoke :write [11 35] INFO [2022-03-13 17:14:03,604] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:03,609] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:14:03,610] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,626] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,626] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,611] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:14:03,615] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:03,616] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:03,622] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:03,629] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:03,632] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,635] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,636] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:14:03,698] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,698] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,704] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,704] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,705] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,705] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:14:03,711] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:03,711] jepsen worker nemesis - couchbase.util Rebalance check count: 353 INFO [2022-03-13 17:14:03,711] jepsen worker nemesis - couchbase.util Rebalance stuck count: 253 INFO [2022-03-13 17:14:03,738] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,738] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,746] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,747] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:03,754] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,755] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,762] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,763] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,764] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,765] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:14:03,784] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,784] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,784] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,785] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:14:03,785] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,786] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,804] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,804] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,804] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,804] jepsen worker 52 - jepsen.util 592 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,805] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:03,806] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:03,806] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:03,807] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:03,807] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,808] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:14:03,824] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,824] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,825] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,825] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:14:03,826] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:14:03,827] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:03,843] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,844] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,843] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,844] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:14:03,845] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,845] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:14:03,852] jepsen worker 78 - jepsen.util 528 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,852] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,853] jepsen worker 70 - jepsen.util 700 :invoke :write [23 49] INFO [2022-03-13 17:14:03,853] jepsen worker 78 - jepsen.util 528 :invoke :write [26 9] INFO [2022-03-13 17:14:03,855] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,855] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,855] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,856] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,859] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,860] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:03,860] jepsen worker 34 - jepsen.util 754 :invoke :write [11 36] INFO [2022-03-13 17:14:03,861] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:14:03,861] jepsen worker 52 - jepsen.util 592 :invoke :write [17 3] INFO [2022-03-13 17:14:03,862] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:03,890] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,891] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:03,900] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,901] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,901] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:03,902] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 20] INFO [2022-03-13 17:14:03,929] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,929] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:14:03,946] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,947] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:03,952] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,954] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:03,956] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,957] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:03,958] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,959] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:03,960] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:14:03,964] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:04,012] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,013] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:04,013] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,013] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:14:04,016] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,016] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,016] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,017] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,018] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,018] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,018] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,019] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,021] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,021] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,022] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,022] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,027] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,027] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,027] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:14:04,028] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 45] INFO [2022-03-13 17:14:04,051] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,053] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:14:04,091] jepsen worker 3 - jepsen.util 1893 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,092] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,132] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,132] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:14:04,142] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,142] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,143] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:14:04,143] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:14:04,152] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,153] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,156] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,156] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:14:04,211] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,212] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,212] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:04,212] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,217] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,218] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:14:04,236] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,236] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,237] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,238] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:14:04,292] jepsen worker 37 - jepsen.util 2017 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,292] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:14:04,294] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,294] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:04,312] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,312] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,313] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,314] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:04,344] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,344] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,345] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,345] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,345] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:14:04,346] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:14:04,354] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,354] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 47] INFO [2022-03-13 17:14:04,359] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,360] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,360] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:04,364] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:14:04,365] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,366] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,375] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,376] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,382] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,383] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,384] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,385] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,394] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,395] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:14:04,438] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,438] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,445] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,446] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,451] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,452] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,468] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,468] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,469] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,470] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,471] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,472] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:04,517] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,517] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,518] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,519] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:14:04,527] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,527] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,528] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,529] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,531] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,531] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,533] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,533] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,542] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,543] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,561] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,561] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,561] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 19] INFO [2022-03-13 17:14:04,562] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,568] jepsen worker 88 - jepsen.util 2068 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,569] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:04,583] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,583] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,584] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:14:04,585] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,586] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,587] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,588] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,588] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,589] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,589] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,590] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,591] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:14:04,605] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,606] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,607] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,607] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:04,633] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,634] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:14:04,636] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,637] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:14:04,673] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,673] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:14:04,681] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,681] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,681] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,682] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,682] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:14:04,683] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:14:04,683] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:14:04,686] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:14:04,718] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:04,718] jepsen worker nemesis - couchbase.util Rebalance check count: 354 INFO [2022-03-13 17:14:04,718] jepsen worker nemesis - couchbase.util Rebalance stuck count: 254 INFO [2022-03-13 17:14:04,734] jepsen worker 37 - jepsen.util 2017 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,735] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 1] INFO [2022-03-13 17:14:04,761] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,762] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,764] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,764] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,765] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,765] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:14:04,773] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,774] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:14:04,818] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,819] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,822] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,823] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,824] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,824] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:04,825] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,826] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:14:04,834] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:04,835] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:14:05,005] jepsen worker 74 - jepsen.util 2774 :info :write [24 43] :SyncWriteAmbiguous INFO [2022-03-13 17:14:05,005] jepsen worker 74 - jepsen.util 2864 :invoke :write [24 4] INFO [2022-03-13 17:14:05,092] jepsen worker 65 - jepsen.util 3035 :ok :read [21 :nil] INFO [2022-03-13 17:14:05,092] jepsen worker 65 - jepsen.util 3035 :invoke :write [21 23] INFO [2022-03-13 17:14:05,246] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,246] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,248] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,248] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,249] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,250] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,260] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,260] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 6] INFO [2022-03-13 17:14:05,293] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,293] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:14:05,315] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,317] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:05,320] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,320] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,321] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:14:05,322] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 22] INFO [2022-03-13 17:14:05,378] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,379] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:14:05,395] jepsen worker 36 - jepsen.util 2196 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,396] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 6] INFO [2022-03-13 17:14:05,427] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,427] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:05,436] jepsen worker 37 - jepsen.util 2017 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,436] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,450] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,451] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,452] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,453] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,454] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,454] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:14:05,455] jepsen worker 14 - jepsen.util 374 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,456] jepsen worker 14 - jepsen.util 374 :invoke :write [4 5] INFO [2022-03-13 17:14:05,459] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,459] jepsen worker 33 - jepsen.util 663 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,460] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:14:05,460] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:05,468] jepsen worker 22 - jepsen.util 652 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,468] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,469] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,469] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:05,471] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,471] jepsen worker 13 - jepsen.util 643 :invoke :write [4 11] INFO [2022-03-13 17:14:05,473] jepsen worker 22 - jepsen.util 652 :invoke :write [7 12] INFO [2022-03-13 17:14:05,474] jepsen worker 80 - jepsen.util 710 :invoke :write [26 19] INFO [2022-03-13 17:14:05,514] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,515] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:14:05,528] jepsen worker 42 - jepsen.util 402 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,529] jepsen worker 77 - jepsen.util 707 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,529] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,529] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,529] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,530] jepsen worker 36 - jepsen.util 2196 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,530] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:05,532] jepsen worker 8 - jepsen.util 728 :invoke :write [2 35] INFO [2022-03-13 17:14:05,532] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:05,533] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,533] jepsen worker 62 - jepsen.util 512 :invoke :write [20 5] INFO [2022-03-13 17:14:05,534] jepsen worker 77 - jepsen.util 707 :invoke :write [25 11] INFO [2022-03-13 17:14:05,540] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,543] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,545] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,545] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:05,549] jepsen worker 17 - jepsen.util 2717 :info :write [5 15] :SyncWriteAmbiguous INFO [2022-03-13 17:14:05,550] jepsen worker 17 - jepsen.util 2807 :invoke :read [5 nil] INFO [2022-03-13 17:14:05,551] jepsen worker 89 - jepsen.util 2429 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,552] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:05,559] jepsen worker 78 - jepsen.util 528 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,560] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:05,562] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,563] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,625] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,626] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:14:05,627] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,627] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,629] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,629] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,635] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,635] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,636] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,636] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:14:05,637] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,637] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,637] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,637] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,638] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:05,639] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 41] INFO [2022-03-13 17:14:05,640] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:14:05,641] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,642] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,642] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:05,648] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,648] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:05,651] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,651] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:14:05,654] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,655] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:05,659] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,660] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:14:05,678] jepsen worker 24 - jepsen.util 564 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,678] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:14:05,709] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,709] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,710] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:14:05,710] jepsen worker 7 - jepsen.util 637 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,711] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:14:05,711] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:05,723] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:05,723] jepsen worker nemesis - couchbase.util Rebalance check count: 355 INFO [2022-03-13 17:14:05,723] jepsen worker nemesis - couchbase.util Rebalance stuck count: 255 INFO [2022-03-13 17:14:05,734] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,734] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:05,746] jepsen worker 72 - jepsen.util 2682 :info :write [24 43] :RequestTimeoutException INFO [2022-03-13 17:14:05,747] jepsen worker 72 - jepsen.util 2772 :invoke :read [24 nil] INFO [2022-03-13 17:14:05,751] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,751] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:05,795] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,796] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:05,815] jepsen worker 72 - jepsen.util 2772 :ok :read [24 24] INFO [2022-03-13 17:14:05,815] jepsen worker 72 - jepsen.util 2772 :invoke :read [24 nil] INFO [2022-03-13 17:14:05,827] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,828] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:05,838] jepsen worker 72 - jepsen.util 2772 :ok :read [24 24] INFO [2022-03-13 17:14:05,839] jepsen worker 72 - jepsen.util 2772 :invoke :write [24 15] INFO [2022-03-13 17:14:05,854] jepsen worker 18 - jepsen.util 2358 :info :write [6 32] :RequestTimeoutException INFO [2022-03-13 17:14:05,854] jepsen worker 18 - jepsen.util 2448 :invoke :write [6 47] INFO [2022-03-13 17:14:05,857] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,857] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:14:05,884] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,885] jepsen worker 9 - jepsen.util 2169 :info :write [3 7] :SyncWriteAmbiguous INFO [2022-03-13 17:14:05,885] jepsen worker 9 - jepsen.util 2259 :invoke :write [3 49] INFO [2022-03-13 17:14:05,886] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:05,887] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,887] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:14:05,896] jepsen worker 34 - jepsen.util 754 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,896] jepsen worker 34 - jepsen.util 754 :invoke :write [11 48] INFO [2022-03-13 17:14:05,923] jepsen worker 61 - jepsen.util 691 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,924] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:05,947] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,948] jepsen worker 37 - jepsen.util 2017 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,948] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:14:05,949] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:14:05,974] jepsen worker 55 - jepsen.util 2485 :info :write [18 34] :RequestTimeoutException INFO [2022-03-13 17:14:05,974] jepsen worker 55 - jepsen.util 2575 :invoke :read [18 nil] INFO [2022-03-13 17:14:05,996] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:05,997] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:14:06,033] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,033] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 9] INFO [2022-03-13 17:14:06,051] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,052] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,054] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,055] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,056] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,056] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,059] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,060] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,064] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,065] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,068] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,070] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:14:06,088] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,088] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,088] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,089] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:06,090] jepsen worker 33 - jepsen.util 663 :invoke :write [11 15] INFO [2022-03-13 17:14:06,091] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:06,100] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,100] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,101] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:06,101] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:06,106] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,107] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:14:06,130] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,131] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:14:06,185] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,185] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:06,193] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,193] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:14:06,210] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,210] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,215] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,215] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,217] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,217] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] WARN [2022-03-13 17:14:06,262] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17082195,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17179169,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9113303,"encode_duration_us":26488,"operation_id":"0x9418","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10009965}],"total_count":1}} INFO [2022-03-13 17:14:06,273] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,273] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,273] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 34] INFO [2022-03-13 17:14:06,274] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:14:06,282] jepsen worker 5 - jepsen.util 2435 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,282] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:06,294] jepsen worker 14 - jepsen.util 374 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,294] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:14:06,305] jepsen worker 22 - jepsen.util 652 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,305] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:06,306] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,307] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,310] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,311] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:06,314] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,314] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:14:06,315] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,315] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,315] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,316] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:14:06,316] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:14:06,317] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:06,333] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,333] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,334] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 10] INFO [2022-03-13 17:14:06,335] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:14:06,340] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:14:06,340] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,340] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,340] jepsen worker 6 - jepsen.util 636 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,341] jepsen worker 43 - jepsen.util 493 :invoke :write [14 14] INFO [2022-03-13 17:14:06,342] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:06,342] jepsen worker 53 - jepsen.util 503 :invoke :write [17 44] INFO [2022-03-13 17:14:06,344] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] WARN [2022-03-13 17:14:06,362] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13115443,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13115443,"total_server_duration_us":1358972,"operation_id":"0x97bd","timeout_ms":30000,"last_server_duration_us":1358972,"total_duration_us":13117791},{"operation_name":"get","last_dispatch_duration_us":11657603,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11657603,"total_server_duration_us":11802,"operation_id":"0x98be","timeout_ms":30000,"last_server_duration_us":11802,"total_duration_us":11659381},{"operation_name":"upsert","last_dispatch_duration_us":8761268,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9753969,"total_server_duration_us":383244,"encode_duration_us":20549,"operation_id":"0x9be4","timeout_ms":10000,"last_server_duration_us":44611,"total_duration_us":10009724},{"operation_name":"upsert","last_dispatch_duration_us":8424987,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9369018,"total_server_duration_us":456861,"encode_duration_us":17889,"operation_id":"0x9bdc","timeout_ms":10000,"last_server_duration_us":42743,"total_duration_us":10009628},{"operation_name":"upsert","last_dispatch_duration_us":8744399,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9722591,"total_server_duration_us":403140,"encode_duration_us":20026,"operation_id":"0x9bdf","timeout_ms":10000,"last_server_duration_us":45838,"total_duration_us":10002684},{"operation_name":"get","last_dispatch_duration_us":9580912,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9580912,"total_server_duration_us":20441,"operation_id":"0x9a08","timeout_ms":30000,"last_server_duration_us":20441,"total_duration_us":9581287},{"operation_name":"upsert","last_dispatch_duration_us":9069135,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9069135,"total_server_duration_us":9053293,"encode_duration_us":18256,"operation_id":"0x9e13","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9072964},{"operation_name":"get","last_dispatch_duration_us":6233557,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6233557,"total_server_duration_us":15480,"operation_id":"0x9c0a","timeout_ms":30000,"last_server_duration_us":15480,"total_duration_us":6235416},{"operation_name":"upsert","last_dispatch_duration_us":2029903,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2029903,"total_server_duration_us":287387,"encode_duration_us":18564,"operation_id":"0xa280","timeout_ms":10000,"last_server_duration_us":287387,"total_duration_us":2038004},{"operation_name":"upsert","last_dispatch_duration_us":1791450,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1791450,"total_server_duration_us":1574360,"encode_duration_us":42973,"operation_id":"0xa265","timeout_ms":10000,"last_server_duration_us":1574360,"total_duration_us":1796490}],"total_count":32}} INFO [2022-03-13 17:14:06,373] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,373] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 19] INFO [2022-03-13 17:14:06,390] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,391] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:06,424] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,425] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,426] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,427] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,438] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,439] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:06,446] jepsen worker 29 - jepsen.util 2819 :info :write [9 2] :RequestTimeoutException INFO [2022-03-13 17:14:06,446] jepsen worker 29 - jepsen.util 2909 :invoke :write [9 31] INFO [2022-03-13 17:14:06,469] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,469] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,471] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 19] INFO [2022-03-13 17:14:06,471] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:14:06,474] jepsen worker 3 - jepsen.util 1893 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,474] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:06,499] jepsen worker 33 - jepsen.util 663 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,500] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,500] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:06,500] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:14:06,516] jepsen worker 52 - jepsen.util 592 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,517] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,517] jepsen worker 52 - jepsen.util 592 :invoke :write [17 45] INFO [2022-03-13 17:14:06,518] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,519] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,519] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:06,559] jepsen worker 89 - jepsen.util 2429 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,560] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:14:06,574] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,574] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,575] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:14:06,575] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:14:06,615] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,615] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:14:06,621] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,622] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:14:06,659] jepsen worker 88 - jepsen.util 2068 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,660] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:14:06,660] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,660] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:06,662] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,662] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:06,700] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,701] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:06,704] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,704] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:06,704] jepsen worker 69 - jepsen.util 699 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,706] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:06,707] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:06,707] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:06,708] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,708] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,709] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:06,709] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,710] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:14:06,710] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:06,712] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,712] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,713] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,714] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,717] jepsen worker 37 - jepsen.util 2017 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,718] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:14:06,724] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,725] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,727] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,728] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:06,730] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:06,730] jepsen worker nemesis - couchbase.util Rebalance check count: 356 INFO [2022-03-13 17:14:06,730] jepsen worker nemesis - couchbase.util Rebalance stuck count: 256 INFO [2022-03-13 17:14:06,732] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,732] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:14:06,738] jepsen worker 70 - jepsen.util 700 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,738] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,739] jepsen worker 22 - jepsen.util 652 :invoke :write [7 28] INFO [2022-03-13 17:14:06,743] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:06,745] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,746] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,746] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,746] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:06,747] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 36] INFO [2022-03-13 17:14:06,747] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:06,786] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,797] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,236] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:06,840] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,855] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,855] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,855] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,236] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:14:06,884] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,900] jepsen worker 57 - jepsen.util 2577 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,906] jepsen worker 3 - jepsen.util 1893 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,236] jepsen worker 57 - jepsen.util 2577 :invoke :write [19 33] INFO [2022-03-13 17:14:07,236] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:14:07,237] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,239] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:06,913] jepsen worker 67 - jepsen.util 2317 :ok :read [22 30] INFO [2022-03-13 17:14:06,930] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:06,957] jepsen worker 10 - jepsen.util 2710 :info :write [3 20] :RequestTimeoutException INFO [2022-03-13 17:14:07,240] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 19] INFO [2022-03-13 17:14:07,056] jepsen worker 59 - jepsen.util 2669 :ok :read [19 44] INFO [2022-03-13 17:14:07,120] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,240] jepsen worker 10 - jepsen.util 2800 :invoke :read [3 nil] INFO [2022-03-13 17:14:07,120] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,240] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:14:07,240] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:14:07,129] jepsen worker 12 - jepsen.util 732 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,129] jepsen worker 62 - jepsen.util 512 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,241] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:07,241] jepsen worker 62 - jepsen.util 512 :invoke :write [20 33] INFO [2022-03-13 17:14:07,198] jepsen worker 30 - jepsen.util 2550 :info :write [10 27] :RequestTimeoutException INFO [2022-03-13 17:14:07,236] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:07,246] jepsen worker 30 - jepsen.util 2640 :invoke :read [10 nil] INFO [2022-03-13 17:14:07,247] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,250] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:07,236] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,236] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:14:07,236] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:14:07,236] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:14:07,239] jepsen worker 67 - jepsen.util 2317 :invoke :write [22 44] INFO [2022-03-13 17:14:07,258] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,261] jepsen worker 59 - jepsen.util 2669 :invoke :read [19 nil] INFO [2022-03-13 17:14:07,262] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:14:07,263] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,264] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,267] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,273] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:14:07,298] jepsen worker 31 - jepsen.util 2731 :info :write [10 8] :RequestTimeoutException INFO [2022-03-13 17:14:07,299] jepsen worker 31 - jepsen.util 2821 :invoke :write [10 47] INFO [2022-03-13 17:14:07,323] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,323] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,324] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:07,324] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,325] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:07,325] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:14:07,339] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,340] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:07,345] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,346] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:07,352] jepsen worker 80 - jepsen.util 710 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,353] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:07,370] jepsen worker 56 - jepsen.util 2756 :info :write [18 28] :RequestTimeoutException INFO [2022-03-13 17:14:07,370] jepsen worker 56 - jepsen.util 2846 :invoke :write [18 33] INFO [2022-03-13 17:14:07,394] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,394] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:14:07,396] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,396] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,398] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,398] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:14:07,418] jepsen worker 59 - jepsen.util 2669 :ok :read [19 44] INFO [2022-03-13 17:14:07,418] jepsen worker 59 - jepsen.util 2669 :invoke :write [19 32] INFO [2022-03-13 17:14:07,442] jepsen worker 37 - jepsen.util 2017 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,443] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,446] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,447] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:07,468] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,469] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:14:07,530] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,531] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,531] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:14:07,531] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,532] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,532] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,532] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:07,533] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,533] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:07,533] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,534] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,535] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:07,540] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,541] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:14:07,543] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,543] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,547] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,548] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,560] jepsen worker 8 - jepsen.util 728 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,560] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,560] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,560] jepsen worker 60 - jepsen.util 600 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,560] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,560] jepsen worker 8 - jepsen.util 728 :invoke :write [2 24] INFO [2022-03-13 17:14:07,561] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,561] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:07,561] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:07,561] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,561] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,562] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:07,563] jepsen worker 44 - jepsen.util 584 :invoke :write [14 26] INFO [2022-03-13 17:14:07,565] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:07,566] jepsen worker 51 - jepsen.util 591 :invoke :write [17 8] INFO [2022-03-13 17:14:07,566] jepsen worker 76 - jepsen.util 706 :invoke :write [25 4] INFO [2022-03-13 17:14:07,567] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:07,567] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 1] INFO [2022-03-13 17:14:07,595] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,596] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,596] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,597] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,598] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,598] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,598] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,599] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,600] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,600] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,601] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:07,601] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:14:07,607] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,607] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,608] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:07,609] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:14:07,646] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,646] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:14:07,653] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,654] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,654] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:07,654] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 34] INFO [2022-03-13 17:14:07,659] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,660] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:14:07,663] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,663] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] WARN [2022-03-13 17:14:07,667] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9044261,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9783031,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9299094,"encode_duration_us":19829,"operation_id":"0x951a","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10009482}],"total_count":1}} INFO [2022-03-13 17:14:07,705] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,705] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:14:07,736] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:07,736] jepsen worker nemesis - couchbase.util Rebalance check count: 357 INFO [2022-03-13 17:14:07,736] jepsen worker nemesis - couchbase.util Rebalance stuck count: 257 INFO [2022-03-13 17:14:07,743] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,743] jepsen worker 13 - jepsen.util 643 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,744] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:07,744] jepsen worker 78 - jepsen.util 528 :invoke :write [26 14] INFO [2022-03-13 17:14:07,746] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,746] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,746] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,747] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,747] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:14:07,748] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,748] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:07,749] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:07,749] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,749] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:14:07,750] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:07,750] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,751] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:07,751] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,751] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:07,752] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,752] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,752] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,752] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:14:07,753] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:07,754] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,754] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,755] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:07,756] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,756] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,757] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,757] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:14:07,757] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:07,759] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,759] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,760] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:07,760] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] WARN [2022-03-13 17:14:07,774] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16328478,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17438871,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9307417,"encode_duration_us":22084,"operation_id":"0x991f","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10010624}],"total_count":1}} INFO [2022-03-13 17:14:07,806] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,806] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,807] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:07,807] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:14:07,808] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,809] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,810] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,811] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:07,833] jepsen worker 37 - jepsen.util 2017 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,834] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,861] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,862] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,863] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,864] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] WARN [2022-03-13 17:14:07,868] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":54639,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9026449,"total_server_duration_us":199169,"encode_duration_us":20220,"operation_id":"0x9ead","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10010406},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":30839,"operation_id":"0x9b90","timeout_ms":10000,"total_duration_us":10007670},{"operation_name":"upsert","last_dispatch_duration_us":113012,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":278067,"total_server_duration_us":62531,"encode_duration_us":19444,"operation_id":"0x9ea3","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10007567},{"operation_name":"upsert","last_dispatch_duration_us":8385014,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17529523,"total_server_duration_us":236558,"encode_duration_us":23625,"operation_id":"0x9913","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10004647},{"operation_name":"upsert","last_dispatch_duration_us":118092,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":569435,"total_server_duration_us":132082,"encode_duration_us":22640,"operation_id":"0x9e75","timeout_ms":10000,"last_server_duration_us":26953,"total_duration_us":10003955},{"operation_name":"upsert","last_dispatch_duration_us":100137,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":159366,"total_server_duration_us":44601,"encode_duration_us":22403,"operation_id":"0x9acd","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10003722},{"operation_name":"upsert","last_dispatch_duration_us":8426805,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8523450,"total_server_duration_us":41440,"encode_duration_us":16980,"operation_id":"0x9c1c","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10003148},{"operation_name":"upsert","last_dispatch_duration_us":9106712,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9194509,"total_server_duration_us":9083449,"encode_duration_us":27965,"operation_id":"0x9e4d","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":9218374},{"operation_name":"upsert","last_dispatch_duration_us":1075362,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1075362,"total_server_duration_us":93498,"encode_duration_us":35521,"operation_id":"0x9b8f","timeout_ms":10000,"last_server_duration_us":93498,"total_duration_us":6249251},{"operation_name":"upsert","last_dispatch_duration_us":4542466,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4542466,"total_server_duration_us":19809,"encode_duration_us":27964,"operation_id":"0xa1af","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":4655221}],"total_count":63}} INFO [2022-03-13 17:14:07,879] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,880] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:14:07,911] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,911] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,914] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,914] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,916] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,917] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:07,918] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,918] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:14:07,926] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,926] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:14:07,943] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,943] jepsen worker 22 - jepsen.util 652 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,944] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:07,944] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:14:07,950] jepsen worker 53 - jepsen.util 503 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,951] jepsen worker 53 - jepsen.util 503 :invoke :write [17 13] INFO [2022-03-13 17:14:07,961] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,961] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,961] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,962] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:07,962] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:07,963] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] WARN [2022-03-13 17:14:07,964] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18251323,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27280284,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9107196,"encode_duration_us":36086,"operation_id":"0x906d","timeout_ms":10000,"last_server_duration_us":9087309,"total_duration_us":10006009},{"operation_name":"upsert","last_dispatch_duration_us":16574616,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18365977,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9431292,"encode_duration_us":19519,"operation_id":"0x937f","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10013983},{"operation_name":"upsert","last_dispatch_duration_us":16380213,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17930690,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9423569,"encode_duration_us":24789,"operation_id":"0x937b","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10007481}],"total_count":3}} WARN [2022-03-13 17:14:07,964] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15803199,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":15803199,"total_server_duration_us":43,"operation_id":"0x9768","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":15806210},{"operation_name":"upsert","last_dispatch_duration_us":9226342,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9226342,"total_server_duration_us":71096,"encode_duration_us":17751,"operation_id":"0x9ab1","timeout_ms":10000,"last_server_duration_us":71096,"total_duration_us":10008629},{"operation_name":"upsert","last_dispatch_duration_us":10563911,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10563911,"total_server_duration_us":46287,"encode_duration_us":16964,"operation_id":"0x9b0b","timeout_ms":10000,"last_server_duration_us":46287,"total_duration_us":10007682},{"operation_name":"upsert","last_dispatch_duration_us":268895,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10142847,"total_server_duration_us":98886,"encode_duration_us":17304,"operation_id":"0x9be7","timeout_ms":10000,"last_server_duration_us":32645,"total_duration_us":10003945},{"operation_name":"upsert","last_dispatch_duration_us":352589,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9997601,"total_server_duration_us":77449,"encode_duration_us":19234,"operation_id":"0x9bca","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10003048},{"operation_name":"upsert","last_dispatch_duration_us":30946,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6453818,"total_server_duration_us":245694,"encode_duration_us":20265,"operation_id":"0x9db0","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10002518},{"operation_name":"upsert","last_dispatch_duration_us":12552101,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":12552101,"total_server_duration_us":38378,"encode_duration_us":17534,"operation_id":"0x998f","timeout_ms":10000,"last_server_duration_us":38378,"total_duration_us":10002355},{"operation_name":"get","last_dispatch_duration_us":9592573,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9592573,"total_server_duration_us":122347,"operation_id":"0x9bfa","timeout_ms":30000,"last_server_duration_us":122347,"total_duration_us":9595125},{"operation_name":"get","last_dispatch_duration_us":9572971,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9572971,"total_server_duration_us":122347,"operation_id":"0x9bf9","timeout_ms":30000,"last_server_duration_us":122347,"total_duration_us":9576151},{"operation_name":"get","last_dispatch_duration_us":9484922,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9484922,"total_server_duration_us":17963,"operation_id":"0x9bf5","timeout_ms":30000,"last_server_duration_us":17963,"total_duration_us":9485311}],"total_count":39}} WARN [2022-03-13 17:14:07,964] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":171468,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9910233,"total_server_duration_us":246497,"encode_duration_us":33539,"operation_id":"0x9b33","timeout_ms":10000,"last_server_duration_us":58999,"total_duration_us":10008206},{"operation_name":"upsert","last_dispatch_duration_us":8874532,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":16691103,"total_server_duration_us":277403,"encode_duration_us":17657,"operation_id":"0x9a17","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10003719},{"operation_name":"upsert","last_dispatch_duration_us":9716937,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9716937,"total_server_duration_us":19262,"encode_duration_us":29854,"operation_id":"0x9eb5","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10002216},{"operation_name":"upsert","last_dispatch_duration_us":9064138,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9393729,"total_server_duration_us":9229399,"encode_duration_us":17378,"operation_id":"0x9927","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9590842},{"operation_name":"upsert","last_dispatch_duration_us":9424189,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9424189,"total_server_duration_us":9049045,"encode_duration_us":16641,"operation_id":"0x9af4","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":9427660},{"operation_name":"get","last_dispatch_duration_us":8595512,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8595512,"total_server_duration_us":49,"operation_id":"0x9b8e","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8595795},{"operation_name":"get","last_dispatch_duration_us":5562164,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5562164,"total_server_duration_us":76,"operation_id":"0x9d4d","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":5562355},{"operation_name":"upsert","last_dispatch_duration_us":1821138,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1821138,"total_server_duration_us":1613941,"encode_duration_us":18153,"operation_id":"0xa25c","timeout_ms":10000,"last_server_duration_us":1613941,"total_duration_us":1833270},{"operation_name":"upsert","last_dispatch_duration_us":987119,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":987119,"total_server_duration_us":68414,"encode_duration_us":35384,"operation_id":"0x9f01","timeout_ms":10000,"last_server_duration_us":68414,"total_duration_us":1010147},{"operation_name":"upsert","last_dispatch_duration_us":951337,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":951337,"total_server_duration_us":19030,"encode_duration_us":18072,"operation_id":"0x9f0d","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":955063}],"total_count":20}} WARN [2022-03-13 17:14:07,969] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17036934,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17036934,"total_server_duration_us":7698,"operation_id":"0x96b8","timeout_ms":30000,"last_server_duration_us":7698,"total_duration_us":17037262},{"operation_name":"get","last_dispatch_duration_us":15769926,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":15769926,"total_server_duration_us":76,"operation_id":"0x9791","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":15770075},{"operation_name":"get","last_dispatch_duration_us":9205120,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9205120,"total_server_duration_us":733,"operation_id":"0x99b7","timeout_ms":30000,"last_server_duration_us":733,"total_duration_us":12333740},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18163,"operation_id":"0x9ab9","timeout_ms":10000,"total_duration_us":10008804},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18604,"operation_id":"0x9e0d","timeout_ms":10000,"total_duration_us":10006346},{"operation_name":"upsert","last_dispatch_duration_us":10130569,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10130569,"total_server_duration_us":33918,"encode_duration_us":18717,"operation_id":"0x9b1d","timeout_ms":10000,"last_server_duration_us":33918,"total_duration_us":10003000},{"operation_name":"get","last_dispatch_duration_us":9134813,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9134813,"total_server_duration_us":40482,"operation_id":"0x9bd1","timeout_ms":30000,"last_server_duration_us":40482,"total_duration_us":9135144},{"operation_name":"upsert","last_dispatch_duration_us":2841628,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2841628,"total_server_duration_us":36525,"encode_duration_us":36483,"operation_id":"0xa28a","timeout_ms":10000,"last_server_duration_us":36525,"total_duration_us":2848601},{"operation_name":"upsert","last_dispatch_duration_us":2622643,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2622643,"total_server_duration_us":163566,"encode_duration_us":24627,"operation_id":"0xa289","timeout_ms":10000,"last_server_duration_us":163566,"total_duration_us":2629430},{"operation_name":"upsert","last_dispatch_duration_us":2241449,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2241449,"total_server_duration_us":172728,"encode_duration_us":24384,"operation_id":"0xa322","timeout_ms":10000,"last_server_duration_us":172728,"total_duration_us":2246333}],"total_count":54}} WARN [2022-03-13 17:14:07,976] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11645521,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11645521,"total_server_duration_us":6384,"operation_id":"0x9d50","timeout_ms":30000,"last_server_duration_us":6384,"total_duration_us":11648187},{"operation_name":"upsert","last_dispatch_duration_us":16328478,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17438871,"total_server_duration_us":9307417,"encode_duration_us":22084,"operation_id":"0x991f","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10010624},{"operation_name":"upsert","last_dispatch_duration_us":32731,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8011328,"total_server_duration_us":279558,"encode_duration_us":23623,"operation_id":"0x9dc3","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10009683},{"operation_name":"upsert","last_dispatch_duration_us":9012306,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":15683276,"total_server_duration_us":67593,"encode_duration_us":17241,"operation_id":"0x9af5","timeout_ms":10000,"last_server_duration_us":48563,"total_duration_us":10008725},{"operation_name":"upsert","last_dispatch_duration_us":45219,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8006151,"total_server_duration_us":338639,"encode_duration_us":22787,"operation_id":"0x9dd3","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10007094},{"operation_name":"upsert","last_dispatch_duration_us":7275575,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8480565,"total_server_duration_us":319013,"encode_duration_us":32533,"operation_id":"0x991a","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10003943},{"operation_name":"get","last_dispatch_duration_us":9080615,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9080615,"total_server_duration_us":92,"operation_id":"0x9f0e","timeout_ms":30000,"last_server_duration_us":92,"total_duration_us":9081012},{"operation_name":"upsert","last_dispatch_duration_us":9053877,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9053877,"total_server_duration_us":9025695,"encode_duration_us":26064,"operation_id":"0x9dcf","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9057808},{"operation_name":"get","last_dispatch_duration_us":5265207,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":5265207,"total_server_duration_us":1115,"operation_id":"0xa049","timeout_ms":30000,"last_server_duration_us":1115,"total_duration_us":5270874},{"operation_name":"upsert","last_dispatch_duration_us":4366962,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4366962,"total_server_duration_us":1951565,"encode_duration_us":21236,"operation_id":"0xa11a","timeout_ms":10000,"last_server_duration_us":1951565,"total_duration_us":4371961}],"total_count":64}} INFO [2022-03-13 17:14:07,980] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:07,981] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:08,001] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,002] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,002] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,002] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:08,003] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:08,004] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:14:08,021] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,021] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,021] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,022] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:14:08,022] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:08,023] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] WARN [2022-03-13 17:14:08,065] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16303056,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16957386,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9408122,"encode_duration_us":19090,"operation_id":"0x97d2","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10008569}],"total_count":1}} WARN [2022-03-13 17:14:08,069] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17785129,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25236625,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9148908,"encode_duration_us":17303,"operation_id":"0x9149","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10001683},{"operation_name":"upsert","last_dispatch_duration_us":17791920,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18114091,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9098090,"encode_duration_us":18655,"operation_id":"0x95d1","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10004031},{"operation_name":"upsert","last_dispatch_duration_us":17718147,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25535300,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9522475,"encode_duration_us":27703,"operation_id":"0x9000","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":10007446}],"total_count":3}} INFO [2022-03-13 17:14:08,075] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,076] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:14:08,094] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,095] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,095] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:08,096] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:08,097] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,098] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,098] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:14:08,100] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:08,110] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,111] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 6] INFO [2022-03-13 17:14:08,116] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,116] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,117] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:14:08,119] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:08,147] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,163] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,163] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,593] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:14:08,163] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,593] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:14:08,593] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:14:08,176] jepsen worker 77 - jepsen.util 707 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,176] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,594] jepsen worker 77 - jepsen.util 707 :invoke :write [25 18] INFO [2022-03-13 17:14:08,594] jepsen worker 61 - jepsen.util 691 :invoke :write [20 46] INFO [2022-03-13 17:14:08,179] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,597] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,179] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,180] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,597] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:08,598] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:08,180] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,598] jepsen worker 12 - jepsen.util 732 :invoke :write [4 13] INFO [2022-03-13 17:14:08,186] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,197] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,598] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:14:08,200] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,598] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:14:08,226] jepsen worker 73 - jepsen.util 2863 :info :write [24 12] :RequestTimeoutException INFO [2022-03-13 17:14:08,598] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:08,598] jepsen worker 73 - jepsen.util 2953 :invoke :write [24 3] INFO [2022-03-13 17:14:08,229] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,290] jepsen worker 40 - jepsen.util 2740 :info :write [13 19] :RequestTimeoutException INFO [2022-03-13 17:14:08,610] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:08,610] jepsen worker 40 - jepsen.util 2830 :invoke :read [13 nil] INFO [2022-03-13 17:14:08,612] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,612] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:14:08,315] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,316] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,346] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,619] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:08,346] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,347] jepsen worker 34 - jepsen.util 754 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,619] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,619] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:08,360] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,619] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,373] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,620] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,406] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:08,406] jepsen worker 43 - jepsen.util 493 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,620] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:08,407] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,620] jepsen worker 43 - jepsen.util 493 :invoke :write [14 41] INFO [2022-03-13 17:14:08,620] jepsen worker 26 - jepsen.util 476 :invoke :write [8 18] INFO [2022-03-13 17:14:08,621] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,622] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,623] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,623] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:14:08,626] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,413] jepsen worker 37 - jepsen.util 2017 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,424] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,575] jepsen worker 52 - jepsen.util 592 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,575] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,630] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,630] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:08,630] jepsen worker 70 - jepsen.util 700 :invoke :write [23 15] INFO [2022-03-13 17:14:08,631] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,633] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,575] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,578] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,585] jepsen worker 51 - jepsen.util 591 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,634] jepsen worker 80 - jepsen.util 710 :invoke :write [26 40] INFO [2022-03-13 17:14:08,634] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,634] jepsen worker 13 - jepsen.util 643 :invoke :write [4 37] INFO [2022-03-13 17:14:08,634] jepsen worker 51 - jepsen.util 591 :invoke :write [17 42] INFO [2022-03-13 17:14:08,635] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,638] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,640] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,640] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,640] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,642] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:14:08,591] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,591] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,593] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:14:08,646] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:14:08,646] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:14:08,619] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:14:08,619] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 44] INFO [2022-03-13 17:14:08,650] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,651] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:14:08,630] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:08,655] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,656] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:08,698] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,699] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,705] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,706] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:14:08,708] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,709] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:08,726] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,727] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:08,728] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,729] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 14] INFO [2022-03-13 17:14:08,741] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:08,741] jepsen worker nemesis - couchbase.util Rebalance check count: 358 INFO [2022-03-13 17:14:08,741] jepsen worker nemesis - couchbase.util Rebalance stuck count: 258 INFO [2022-03-13 17:14:08,769] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,769] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:08,778] jepsen worker 78 - jepsen.util 528 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,778] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,778] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,778] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,779] jepsen worker 78 - jepsen.util 528 :invoke :write [26 16] INFO [2022-03-13 17:14:08,780] jepsen worker 62 - jepsen.util 512 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,780] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,781] jepsen worker 44 - jepsen.util 584 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,782] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,783] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:08,785] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,785] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:08,785] jepsen worker 62 - jepsen.util 512 :invoke :write [20 12] INFO [2022-03-13 17:14:08,786] jepsen worker 34 - jepsen.util 754 :invoke :write [11 13] INFO [2022-03-13 17:14:08,788] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:08,788] jepsen worker 71 - jepsen.util 521 :invoke :write [23 15] INFO [2022-03-13 17:14:08,790] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:08,790] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,792] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,793] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,794] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,794] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,796] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,796] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,797] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,798] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:14:08,801] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,801] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,802] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:08,803] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:14:08,809] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,810] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:14:08,819] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,820] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:14:08,852] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,852] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:08,862] jepsen worker 13 - jepsen.util 643 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,863] jepsen worker 13 - jepsen.util 643 :invoke :write [4 35] INFO [2022-03-13 17:14:08,877] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,877] jepsen worker 24 - jepsen.util 564 :invoke :write [8 13] INFO [2022-03-13 17:14:08,888] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,889] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 16] INFO [2022-03-13 17:14:08,930] jepsen worker 36 - jepsen.util 2196 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,931] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:08,944] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,944] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:08,976] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,976] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:08,979] jepsen worker 27 - jepsen.util 3177 :info :write [9 37] :RequestTimeoutException INFO [2022-03-13 17:14:08,980] jepsen worker 27 - jepsen.util 3267 :invoke :read [9 nil] INFO [2022-03-13 17:14:08,987] jepsen worker 3 - jepsen.util 1893 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:08,988] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,032] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,032] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:09,033] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,034] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:14:09,044] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,044] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,045] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:14:09,045] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:14:09,072] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,072] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,073] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,073] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,073] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,074] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,074] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,074] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,075] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,075] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,076] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,076] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,079] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,079] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,079] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,079] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,079] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:14:09,080] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,081] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,081] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,081] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:14:09,082] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:14:09,082] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,083] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:14:09,119] jepsen worker 89 - jepsen.util 2429 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,119] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,124] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,125] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:14:09,130] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,130] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,143] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,143] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 32] INFO [2022-03-13 17:14:09,188] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,188] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:14:09,195] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,196] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:09,198] jepsen worker 8 - jepsen.util 728 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,199] jepsen worker 76 - jepsen.util 706 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,199] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,199] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,199] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,200] jepsen worker 8 - jepsen.util 728 :invoke :write [2 43] INFO [2022-03-13 17:14:09,200] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:09,201] jepsen worker 60 - jepsen.util 600 :invoke :write [20 4] INFO [2022-03-13 17:14:09,202] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:09,202] jepsen worker 76 - jepsen.util 706 :invoke :write [25 35] INFO [2022-03-13 17:14:09,207] jepsen worker 80 - jepsen.util 710 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,209] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:09,218] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,219] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,227] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,228] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:09,234] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,234] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,235] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,235] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:14:09,236] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,236] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,251] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,251] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,253] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,253] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,258] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,259] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,260] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,261] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,262] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,263] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,264] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,265] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:14:09,268] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,269] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,278] jepsen worker 0 - jepsen.util 2520 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,278] jepsen worker 16 - jepsen.util 2446 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,278] jepsen worker 40 - jepsen.util 2830 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,279] jepsen worker 0 - jepsen.util 2520 :invoke :write [0 30] INFO [2022-03-13 17:14:09,279] jepsen worker 16 - jepsen.util 2446 :invoke :write [5 41] INFO [2022-03-13 17:14:09,280] jepsen worker 40 - jepsen.util 2830 :invoke :write [13 44] INFO [2022-03-13 17:14:09,297] jepsen worker 37 - jepsen.util 2017 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,303] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:14:09,330] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,331] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:14:09,336] jepsen worker 36 - jepsen.util 2196 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,336] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,336] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:14:09,337] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,339] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,339] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:14:09,385] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,385] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,400] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,400] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,400] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,401] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,401] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,401] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:14:09,402] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:09,403] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:14:09,403] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:09,403] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:09,404] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,405] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,405] jepsen worker 14 - jepsen.util 374 :invoke :write [4 34] INFO [2022-03-13 17:14:09,405] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,406] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,407] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:09,407] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:09,408] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,408] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,408] jepsen worker 21 - jepsen.util 831 :invoke :write [7 9] INFO [2022-03-13 17:14:09,409] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:14:09,411] jepsen worker 54 - jepsen.util 2214 :info :write [18 23] :RequestTimeoutException INFO [2022-03-13 17:14:09,411] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,411] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:14:09,412] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:14:09,413] jepsen worker 54 - jepsen.util 2304 :invoke :read [18 nil] INFO [2022-03-13 17:14:09,441] jepsen worker 79 - jepsen.util 1069 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,442] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 32] INFO [2022-03-13 17:14:09,445] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,446] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:14:09,451] jepsen worker 64 - jepsen.util 2584 :info :write [21 24] :RequestTimeoutException INFO [2022-03-13 17:14:09,451] jepsen worker 64 - jepsen.util 2674 :invoke :read [21 nil] INFO [2022-03-13 17:14:09,464] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,465] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,470] jepsen worker 11 - jepsen.util 2531 :info :write [3 45] :RequestTimeoutException INFO [2022-03-13 17:14:09,471] jepsen worker 11 - jepsen.util 2621 :invoke :read [3 nil] INFO [2022-03-13 17:14:09,471] jepsen worker 61 - jepsen.util 691 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,472] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,472] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:09,472] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:09,475] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,475] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,476] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,476] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 13] INFO [2022-03-13 17:14:09,515] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,515] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,516] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 27] INFO [2022-03-13 17:14:09,517] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,523] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,524] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:14:09,532] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,532] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,534] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,534] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,544] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,544] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,546] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,547] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:14:09,547] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,548] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:09,551] jepsen worker 45 - jepsen.util 2925 :info :write [15 24] :RequestTimeoutException INFO [2022-03-13 17:14:09,552] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:09,594] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,595] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:09,624] jepsen worker 70 - jepsen.util 700 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,624] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,624] jepsen worker 70 - jepsen.util 700 :invoke :write [23 25] INFO [2022-03-13 17:14:09,625] jepsen worker 52 - jepsen.util 592 :invoke :write [17 12] INFO [2022-03-13 17:14:09,627] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,627] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,633] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,634] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,637] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,638] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,642] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,643] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,653] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,654] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,657] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,657] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,657] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,658] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,659] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,659] jepsen worker 53 - jepsen.util 503 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,659] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:09,660] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,661] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:09,661] jepsen worker 53 - jepsen.util 503 :invoke :write [17 1] INFO [2022-03-13 17:14:09,661] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,662] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:09,664] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,664] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:09,669] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,670] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:14:09,674] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,674] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,675] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:14:09,675] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,717] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,718] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:14:09,747] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:09,747] jepsen worker nemesis - couchbase.util Rebalance check count: 359 INFO [2022-03-13 17:14:09,747] jepsen worker nemesis - couchbase.util Rebalance stuck count: 259 INFO [2022-03-13 17:14:09,761] jepsen worker 37 - jepsen.util 2017 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,761] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,772] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,773] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,774] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,774] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,778] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,779] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:09,797] jepsen worker 3 - jepsen.util 1893 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,798] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,808] jepsen worker 21 - jepsen.util 831 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,808] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:14:09,854] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,854] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,854] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,855] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,855] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,855] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,857] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,858] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,874] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,875] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:14:09,885] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,885] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,886] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:14:09,886] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:09,904] jepsen worker 71 - jepsen.util 521 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,904] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:14:09,915] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,915] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:09,919] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,919] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,919] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,920] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:14:09,920] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:14:09,920] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 46] INFO [2022-03-13 17:14:09,946] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,947] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,947] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:14:09,947] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:09,952] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,953] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:14:09,955] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,956] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:09,957] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:09,958] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:10,041] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,042] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,042] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,043] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:14:10,043] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,044] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,045] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,045] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:14:10,051] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,051] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,100] jepsen worker 14 - jepsen.util 374 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,100] jepsen worker 78 - jepsen.util 528 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,101] jepsen worker 78 - jepsen.util 528 :invoke :write [26 45] INFO [2022-03-13 17:14:10,102] jepsen worker 14 - jepsen.util 374 :invoke :write [4 37] INFO [2022-03-13 17:14:10,103] jepsen worker 24 - jepsen.util 564 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,104] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:10,117] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,118] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,119] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,120] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:14:10,125] jepsen worker 77 - jepsen.util 707 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,126] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:10,126] jepsen worker 77 - jepsen.util 707 :invoke :write [25 23] INFO [2022-03-13 17:14:10,127] jepsen worker 44 - jepsen.util 584 :invoke :write [14 22] INFO [2022-03-13 17:14:10,139] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,139] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,143] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:10,144] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:14:10,151] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,151] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,152] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,153] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:14:10,164] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,164] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,164] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:10,165] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:14:10,210] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,210] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:10,227] jepsen worker 37 - jepsen.util 2017 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,228] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,231] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,232] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:14:10,232] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,232] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,233] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:14:10,233] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:14:10,300] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,300] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,301] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,302] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,311] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,312] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:14:10,316] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,316] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:10,320] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,320] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,323] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,323] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:10,324] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,324] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:14:10,344] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,344] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,344] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:10,345] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 2] INFO [2022-03-13 17:14:10,355] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,355] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:14:10,356] jepsen worker 14 - jepsen.util 374 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,356] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,357] jepsen worker 14 - jepsen.util 374 :invoke :write [4 21] INFO [2022-03-13 17:14:10,357] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:14:10,420] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,420] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:14:10,424] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,425] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,432] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,432] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,433] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,434] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:10,434] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,435] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,436] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,436] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:14:10,451] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,452] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,455] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,456] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:10,502] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,503] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,505] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,506] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:14:10,531] jepsen worker 37 - jepsen.util 2017 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,531] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,532] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:14:10,532] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,534] jepsen worker 26 - jepsen.util 476 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,535] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,535] jepsen worker 26 - jepsen.util 476 :invoke :write [8 28] INFO [2022-03-13 17:14:10,536] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,539] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,540] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:14:10,543] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,543] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:14:10,554] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,554] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,555] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,555] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:14:10,572] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,573] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,575] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,575] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,577] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,578] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:14:10,600] jepsen worker 34 - jepsen.util 754 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,600] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,601] jepsen worker 51 - jepsen.util 591 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,602] jepsen worker 51 - jepsen.util 591 :invoke :write [17 31] INFO [2022-03-13 17:14:10,621] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,621] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:10,630] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,631] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:10,656] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,657] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:14:10,663] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,663] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,664] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,664] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:10,676] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,677] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:14:10,683] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,684] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,687] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,688] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,689] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,690] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,696] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,696] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,698] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,698] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:10,736] jepsen worker 14 - jepsen.util 374 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,736] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,737] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:14:10,738] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:10,759] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:10,759] jepsen worker nemesis - couchbase.util Rebalance check count: 360 INFO [2022-03-13 17:14:10,759] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,759] jepsen worker nemesis - couchbase.util Rebalance stuck count: 260 INFO [2022-03-13 17:14:10,759] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:14:10,769] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,769] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,769] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:10,770] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,775] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,776] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,776] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,776] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:10,777] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,777] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:14:10,777] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,778] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,778] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:14:10,779] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,779] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,780] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:14:10,787] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,787] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,788] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,789] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,789] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,790] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,791] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,792] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,793] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,795] jepsen worker 33 - jepsen.util 663 :invoke :write [11 16] INFO [2022-03-13 17:14:10,799] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,799] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,799] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,800] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,800] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,802] jepsen worker 13 - jepsen.util 643 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,802] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,803] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,803] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,803] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:14:10,803] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,805] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,806] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:10,806] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,807] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:10,807] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,808] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,808] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:10,810] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,810] jepsen worker 15 - jepsen.util 2715 :info :write [5 3] :RequestTimeoutException INFO [2022-03-13 17:14:10,810] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:10,811] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,812] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:10,813] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:10,815] jepsen worker 15 - jepsen.util 2805 :invoke :write [5 14] INFO [2022-03-13 17:14:10,816] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:10,816] jepsen worker 32 - jepsen.util 2642 :info :write [10 45] :RequestTimeoutException INFO [2022-03-13 17:14:10,817] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:10,818] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:10,819] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,819] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,820] jepsen worker 50 - jepsen.util 2750 :info :write [16 15] :RequestTimeoutException INFO [2022-03-13 17:14:10,820] jepsen worker 32 - jepsen.util 2732 :invoke :read [10 nil] INFO [2022-03-13 17:14:10,821] jepsen worker 21 - jepsen.util 831 :invoke :write [7 26] INFO [2022-03-13 17:14:10,821] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,822] jepsen worker 50 - jepsen.util 2840 :invoke :write [16 30] INFO [2022-03-13 17:14:10,826] jepsen worker 47 - jepsen.util 2567 :ok :read [15 48] INFO [2022-03-13 17:14:10,826] jepsen worker 47 - jepsen.util 2567 :invoke :read [15 nil] INFO [2022-03-13 17:14:10,840] jepsen worker 20 - jepsen.util 3080 :info :write [6 0] :RequestTimeoutException INFO [2022-03-13 17:14:10,841] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:10,877] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,878] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:10,888] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,888] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,888] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,888] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:14:10,888] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,889] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 23] INFO [2022-03-13 17:14:10,889] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,898] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:10,898] jepsen worker 47 - jepsen.util 2567 :ok :read [15 48] INFO [2022-03-13 17:14:10,899] jepsen worker 47 - jepsen.util 2567 :invoke :write [15 19] INFO [2022-03-13 17:14:10,899] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,899] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,900] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,900] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:14:10,901] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:10,902] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 8] INFO [2022-03-13 17:14:10,934] jepsen worker 46 - jepsen.util 2566 :info :write [15 26] :SyncWriteAmbiguous INFO [2022-03-13 17:14:10,935] jepsen worker 46 - jepsen.util 2656 :invoke :read [15 nil] INFO [2022-03-13 17:14:10,945] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,946] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:10,960] jepsen worker 43 - jepsen.util 493 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,960] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,960] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,960] jepsen worker 12 - jepsen.util 732 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,960] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,960] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,961] jepsen worker 43 - jepsen.util 493 :invoke :read [14 nil] INFO [2022-03-13 17:14:10,961] jepsen worker 12 - jepsen.util 732 :invoke :write [4 11] INFO [2022-03-13 17:14:10,962] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:10,963] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:10,963] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:10,963] jepsen worker 69 - jepsen.util 699 :invoke :write [23 44] INFO [2022-03-13 17:14:10,967] jepsen worker 63 - jepsen.util 2583 :info :write [21 41] :RequestTimeoutException INFO [2022-03-13 17:14:10,968] jepsen worker 63 - jepsen.util 2673 :invoke :write [21 35] INFO [2022-03-13 17:14:10,968] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,969] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:10,971] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,971] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,972] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:10,973] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:10,973] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:10,974] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:14:11,005] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,005] jepsen worker 21 - jepsen.util 831 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,005] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,005] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,006] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,007] jepsen worker 14 - jepsen.util 374 :invoke :write [4 34] INFO [2022-03-13 17:14:11,007] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:14:11,009] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:11,010] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,010] jepsen worker 35 - jepsen.util 935 :invoke :write [11 2] INFO [2022-03-13 17:14:11,074] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,074] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,075] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:14:11,075] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:14:11,086] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,086] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,091] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,091] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,093] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,094] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,097] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,097] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,101] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,101] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,101] jepsen worker 46 - jepsen.util 2656 :ok :read [15 48] INFO [2022-03-13 17:14:11,102] jepsen worker 46 - jepsen.util 2656 :invoke :read [15 nil] INFO [2022-03-13 17:14:11,105] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,105] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,110] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,110] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:14:11,113] jepsen worker 3 - jepsen.util 1893 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,114] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,150] jepsen worker 11 - jepsen.util 2621 :ok :read [3 25] INFO [2022-03-13 17:14:11,150] jepsen worker 89 - jepsen.util 2429 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,150] jepsen worker 11 - jepsen.util 2621 :invoke :read [3 nil] INFO [2022-03-13 17:14:11,151] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:11,155] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,155] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:11,158] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,159] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:14:11,164] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,164] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:11,168] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,168] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,168] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:11,169] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,182] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,183] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,184] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,184] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:11,189] jepsen worker 70 - jepsen.util 700 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,190] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:11,195] jepsen worker 46 - jepsen.util 2656 :ok :read [15 48] INFO [2022-03-13 17:14:11,196] jepsen worker 46 - jepsen.util 2656 :invoke :write [15 0] INFO [2022-03-13 17:14:11,202] jepsen worker 14 - jepsen.util 374 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,203] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:14:11,216] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,217] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:14:11,233] jepsen worker 11 - jepsen.util 2621 :ok :read [3 25] INFO [2022-03-13 17:14:11,234] jepsen worker 11 - jepsen.util 2621 :invoke :write [3 24] INFO [2022-03-13 17:14:11,234] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,234] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,234] jepsen worker 35 - jepsen.util 935 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,237] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:11,238] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,243] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:14:11,245] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,245] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:11,266] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,267] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:14:11,268] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,268] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,269] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,270] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,271] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:14:11,271] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,272] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,273] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,274] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,274] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,276] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,277] jepsen worker 14 - jepsen.util 374 :invoke :write [4 35] INFO [2022-03-13 17:14:11,281] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,282] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:14:11,289] jepsen worker 19 - jepsen.util 2539 :ok :read [6 16] INFO [2022-03-13 17:14:11,290] jepsen worker 19 - jepsen.util 2539 :invoke :read [6 nil] INFO [2022-03-13 17:14:11,300] jepsen worker 68 - jepsen.util 2768 :info :write [22 36] :RequestTimeoutException INFO [2022-03-13 17:14:11,301] jepsen worker 68 - jepsen.util 2858 :invoke :write [22 35] INFO [2022-03-13 17:14:11,304] jepsen worker 55 - jepsen.util 2575 :ok :read [18 21] INFO [2022-03-13 17:14:11,305] jepsen worker 55 - jepsen.util 2575 :invoke :write [18 40] INFO [2022-03-13 17:14:11,331] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,331] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,335] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,336] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:11,354] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,354] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:11,383] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,384] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:14:11,392] jepsen worker 62 - jepsen.util 512 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,392] jepsen worker 60 - jepsen.util 600 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,393] jepsen worker 45 - jepsen.util 3015 :ok :read [15 48] INFO [2022-03-13 17:14:11,393] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:14:11,394] jepsen worker 60 - jepsen.util 600 :invoke :write [20 41] INFO [2022-03-13 17:14:11,394] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:11,404] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,404] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:11,411] jepsen worker 19 - jepsen.util 2539 :ok :read [6 16] INFO [2022-03-13 17:14:11,412] jepsen worker 19 - jepsen.util 2539 :invoke :read [6 nil] INFO [2022-03-13 17:14:11,426] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,427] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,428] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,428] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:11,447] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,448] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:11,463] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,464] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:14:11,490] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,491] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,492] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,493] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:14:11,499] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,500] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,500] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,500] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,528] jepsen worker 45 - jepsen.util 3015 :ok :read [15 48] INFO [2022-03-13 17:14:11,529] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:11,538] jepsen worker 19 - jepsen.util 2539 :ok :read [6 16] INFO [2022-03-13 17:14:11,539] jepsen worker 19 - jepsen.util 2539 :invoke :read [6 nil] INFO [2022-03-13 17:14:11,543] jepsen worker 41 - jepsen.util 2561 :info :write [13 49] :RequestTimeoutException INFO [2022-03-13 17:14:11,544] jepsen worker 41 - jepsen.util 2651 :invoke :write [13 9] INFO [2022-03-13 17:14:11,551] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,552] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:11,555] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,556] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:14:11,565] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,565] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,570] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,571] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,573] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,574] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:14:11,588] jepsen worker 33 - jepsen.util 663 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,588] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:14:11,592] jepsen worker 52 - jepsen.util 592 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,593] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:11,613] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,613] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,614] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,614] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,615] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:14:11,616] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:14:11,617] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,618] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:14:11,622] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,623] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,624] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,625] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,626] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,627] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 13] INFO [2022-03-13 17:14:11,634] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,635] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:11,639] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,640] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,641] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,642] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:11,646] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,647] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,673] jepsen worker 45 - jepsen.util 3015 :ok :read [15 48] INFO [2022-03-13 17:14:11,674] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:11,686] jepsen worker 2 - jepsen.util 2972 :info :write [0 27] :RequestTimeoutException INFO [2022-03-13 17:14:11,686] jepsen worker 1 - jepsen.util 2341 :info :write [0 40] :RequestTimeoutException INFO [2022-03-13 17:14:11,686] jepsen worker 81 - jepsen.util 3051 :info :write [27 12] :RequestTimeoutException INFO [2022-03-13 17:14:11,687] jepsen worker 81 - jepsen.util 3141 :invoke :write [27 35] INFO [2022-03-13 17:14:11,688] jepsen worker 2 - jepsen.util 3062 :invoke :read [0 nil] INFO [2022-03-13 17:14:11,688] jepsen worker 1 - jepsen.util 2431 :invoke :write [0 43] INFO [2022-03-13 17:14:11,699] jepsen worker 19 - jepsen.util 2539 :ok :read [6 16] INFO [2022-03-13 17:14:11,700] jepsen worker 19 - jepsen.util 2539 :invoke :write [6 44] INFO [2022-03-13 17:14:11,705] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,705] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:11,743] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,744] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:14:11,746] jepsen worker 28 - jepsen.util 2728 :info :write [9 4] :RequestTimeoutException INFO [2022-03-13 17:14:11,747] jepsen worker 28 - jepsen.util 2818 :invoke :read [9 nil] INFO [2022-03-13 17:14:11,765] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:11,765] jepsen worker nemesis - couchbase.util Rebalance check count: 361 INFO [2022-03-13 17:14:11,765] jepsen worker nemesis - couchbase.util Rebalance stuck count: 261 INFO [2022-03-13 17:14:11,774] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,775] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,776] jepsen worker 45 - jepsen.util 3015 :ok :read [15 48] INFO [2022-03-13 17:14:11,777] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:11,785] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,786] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:11,787] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,788] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:11,793] jepsen worker 8 - jepsen.util 728 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,794] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:11,799] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,800] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,800] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,801] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:14:11,801] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:11,802] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,802] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:14:11,803] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:14:11,802] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,804] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:14:11,859] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,860] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:11,869] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,870] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:14:11,870] jepsen worker 28 - jepsen.util 2818 :ok :read [9 26] INFO [2022-03-13 17:14:11,871] jepsen worker 28 - jepsen.util 2818 :invoke :write [9 20] INFO [2022-03-13 17:14:11,876] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,876] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,877] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,879] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:14:11,884] jepsen worker 76 - jepsen.util 706 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,884] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,886] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:11,886] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:11,887] jepsen worker 45 - jepsen.util 3015 :ok :read [15 48] INFO [2022-03-13 17:14:11,888] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:11,895] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,895] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,896] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:14:11,896] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:11,953] jepsen worker 3 - jepsen.util 1893 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,954] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:11,960] jepsen worker 14 - jepsen.util 374 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,960] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,961] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,961] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:11,961] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:14:11,962] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:11,962] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,963] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,963] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,963] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,964] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:14:11,964] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,964] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:11,965] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,965] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,966] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,966] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:11,966] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:11,967] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,967] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,967] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:14:11,968] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:14:11,980] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,981] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,981] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,981] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,981] jepsen worker 78 - jepsen.util 528 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,981] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,982] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,982] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,985] jepsen worker 79 - jepsen.util 1069 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,985] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:14:11,986] jepsen worker 83 - jepsen.util 2063 :info :write [27 43] :RequestTimeoutException INFO [2022-03-13 17:14:11,986] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:11,987] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:11,988] jepsen worker 34 - jepsen.util 754 :invoke :write [11 32] INFO [2022-03-13 17:14:11,989] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,989] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:11,990] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:11,991] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,991] jepsen worker 13 - jepsen.util 643 :invoke :write [4 31] INFO [2022-03-13 17:14:11,992] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:11,993] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:14:11,994] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:11,994] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:11,995] jepsen worker 83 - jepsen.util 2153 :invoke :read [27 nil] INFO [2022-03-13 17:14:11,996] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,996] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,996] jepsen worker 78 - jepsen.util 528 :invoke :write [26 11] INFO [2022-03-13 17:14:11,997] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:11,998] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:11,998] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:11,999] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:11,999] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,000] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,001] jepsen worker 70 - jepsen.util 700 :invoke :write [23 47] INFO [2022-03-13 17:14:12,001] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:12,002] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:12,003] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,004] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,010] jepsen worker 80 - jepsen.util 710 :invoke :write [26 7] INFO [2022-03-13 17:14:12,011] jepsen worker 52 - jepsen.util 592 :invoke :write [17 35] INFO [2022-03-13 17:14:12,015] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,016] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:14:12,016] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,016] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,019] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,019] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,024] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,025] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:14:12,025] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:14:12,028] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:12,034] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,034] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:12,048] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,048] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:12,075] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,076] jepsen worker 14 - jepsen.util 374 :invoke :write [4 40] INFO [2022-03-13 17:14:12,087] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,088] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,088] jepsen worker 33 - jepsen.util 663 :invoke :write [11 10] INFO [2022-03-13 17:14:12,088] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:14:12,095] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,095] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,096] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:12,096] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:12,144] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,145] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,147] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,148] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,154] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,155] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,155] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:14:12,156] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:12,161] jepsen worker 48 - jepsen.util 2568 :info :write [16 45] :SyncWriteAmbiguous INFO [2022-03-13 17:14:12,162] jepsen worker 48 - jepsen.util 2658 :invoke :write [16 8] INFO [2022-03-13 17:14:12,168] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,169] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:12,195] jepsen worker 86 - jepsen.util 2426 :info :write [28 16] :SyncWriteAmbiguous INFO [2022-03-13 17:14:12,196] jepsen worker 86 - jepsen.util 2516 :invoke :read [28 nil] INFO [2022-03-13 17:14:12,218] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,219] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 6] INFO [2022-03-13 17:14:12,223] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,223] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,223] jepsen worker 77 - jepsen.util 707 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,224] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:14:12,224] jepsen worker 77 - jepsen.util 707 :invoke :write [25 40] INFO [2022-03-13 17:14:12,230] jepsen worker 66 - jepsen.util 2586 :info :write [22 24] :RequestTimeoutException INFO [2022-03-13 17:14:12,231] jepsen worker 66 - jepsen.util 2676 :invoke :read [22 nil] INFO [2022-03-13 17:14:12,236] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:12,239] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,240] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:12,245] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,245] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:14:12,247] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,248] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,248] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:12,248] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:12,254] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,254] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,254] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:14:12,256] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 21] INFO [2022-03-13 17:14:12,292] jepsen worker 58 - jepsen.util 2308 :info :write [19 25] :SyncWriteAmbiguous INFO [2022-03-13 17:14:12,293] jepsen worker 85 - jepsen.util 2425 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,293] jepsen worker 39 - jepsen.util 2649 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,293] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:12,293] jepsen worker 82 - jepsen.util 2692 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,293] jepsen worker 58 - jepsen.util 2398 :invoke :write [19 37] INFO [2022-03-13 17:14:12,294] jepsen worker 2 - jepsen.util 3062 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,294] jepsen worker 66 - jepsen.util 2676 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,294] jepsen worker 82 - jepsen.util 2692 :invoke :write [27 33] INFO [2022-03-13 17:14:12,295] jepsen worker 85 - jepsen.util 2425 :invoke :write [28 23] INFO [2022-03-13 17:14:12,295] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:12,296] jepsen worker 39 - jepsen.util 2649 :invoke :write [13 22] INFO [2022-03-13 17:14:12,297] jepsen worker 2 - jepsen.util 3062 :invoke :read [0 nil] INFO [2022-03-13 17:14:12,297] jepsen worker 66 - jepsen.util 2676 :invoke :read [22 nil] INFO [2022-03-13 17:14:12,299] jepsen worker 66 - jepsen.util 2676 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,299] jepsen worker 2 - jepsen.util 3062 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,299] jepsen worker 66 - jepsen.util 2676 :invoke :read [22 nil] INFO [2022-03-13 17:14:12,300] jepsen worker 2 - jepsen.util 3062 :invoke :write [0 19] INFO [2022-03-13 17:14:12,304] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:12,304] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:12,305] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:12,306] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:12,307] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:12,307] jepsen worker 49 - jepsen.util 2389 :invoke :read [16 nil] INFO [2022-03-13 17:14:12,308] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,308] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,308] jepsen worker 49 - jepsen.util 2389 :ok :read [16 :nil] INFO [2022-03-13 17:14:12,309] jepsen worker 49 - jepsen.util 2389 :invoke :write [16 3] INFO [2022-03-13 17:14:12,312] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,312] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:14:12,314] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,314] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:12,317] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,318] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:14:12,346] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,346] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,347] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:14:12,348] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:14:12,348] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,349] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,385] jepsen worker 36 - jepsen.util 2196 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,386] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:12,389] jepsen worker 86 - jepsen.util 2516 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,390] jepsen worker 17 - jepsen.util 2807 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,390] jepsen worker 86 - jepsen.util 2516 :invoke :write [28 48] INFO [2022-03-13 17:14:12,391] jepsen worker 17 - jepsen.util 2807 :invoke :write [5 10] INFO [2022-03-13 17:14:12,429] jepsen worker 66 - jepsen.util 2676 :ok :read [22 30] INFO [2022-03-13 17:14:12,429] jepsen worker 66 - jepsen.util 2676 :invoke :read [22 nil] INFO [2022-03-13 17:14:12,452] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,453] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:14:12,465] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,465] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,466] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:12,467] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 39] INFO [2022-03-13 17:14:12,472] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,473] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:14:12,526] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,526] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,527] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:12,527] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:12,543] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,543] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:14:12,546] jepsen worker 3 - jepsen.util 1893 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,546] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:14:12,567] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,568] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,579] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,580] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:14:12,611] jepsen worker 37 - jepsen.util 2017 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,611] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,612] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:14:12,612] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:14:12,635] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,635] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:14:12,651] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,651] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,654] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,655] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:12,658] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,658] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:12,661] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,662] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:14:12,701] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,701] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,701] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:12,702] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:12,707] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,707] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,738] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,739] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 40] INFO [2022-03-13 17:14:12,775] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,776] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:12,777] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:12,778] jepsen worker nemesis - couchbase.util Rebalance check count: 362 INFO [2022-03-13 17:14:12,778] jepsen worker nemesis - couchbase.util Rebalance stuck count: 262 INFO [2022-03-13 17:14:12,794] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,794] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 44] INFO [2022-03-13 17:14:12,794] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,796] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:14:12,843] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,844] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:14:12,859] jepsen worker 43 - jepsen.util 493 :ok :read [14 :nil] INFO [2022-03-13 17:14:12,859] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,860] jepsen worker 43 - jepsen.util 493 :invoke :write [14 18] INFO [2022-03-13 17:14:12,860] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:12,878] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,879] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:14:12,889] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,890] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,892] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,892] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,902] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,903] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,904] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,905] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:12,930] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,931] jepsen worker 35 - jepsen.util 935 :invoke :write [11 3] INFO [2022-03-13 17:14:12,933] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,933] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:14:12,956] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,958] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:12,961] jepsen worker 6 - jepsen.util 636 :ok :read [2 5] INFO [2022-03-13 17:14:12,962] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:12,968] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,969] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:12,972] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:12,973] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:12,979] jepsen worker 7 - jepsen.util 637 :ok :read [2 5] INFO [2022-03-13 17:14:12,980] jepsen worker 7 - jepsen.util 637 :invoke :write [2 36] INFO [2022-03-13 17:14:13,030] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,030] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:14:13,053] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,053] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,053] jepsen worker 37 - jepsen.util 2017 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,054] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,055] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:13,055] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:13,071] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,071] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,074] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,075] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,076] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,077] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,078] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,079] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,080] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,080] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,082] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,083] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,092] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,093] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,095] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,096] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,101] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,102] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,102] jepsen worker 88 - jepsen.util 2068 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,103] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,105] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,106] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:13,131] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,132] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,133] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,134] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:14:13,151] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,151] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,203] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,203] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,203] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:13,204] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,206] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,206] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,206] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,207] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,207] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:14:13,207] jepsen worker 66 - jepsen.util 2676 :ok :read [22 30] INFO [2022-03-13 17:14:13,208] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:13,208] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:14:13,211] jepsen worker 66 - jepsen.util 2676 :invoke :read [22 nil] INFO [2022-03-13 17:14:13,211] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,219] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,220] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:14:13,244] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,244] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,245] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:13,245] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:14:13,256] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,257] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:13,259] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,260] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,260] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,260] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:14:13,261] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:14:13,261] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,296] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,297] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:14:13,318] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,318] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,319] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,319] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,358] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,359] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:14:13,385] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,386] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:14:13,427] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,427] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:14:13,431] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,432] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:13,471] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,472] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,472] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,473] jepsen worker 62 - jepsen.util 512 :invoke :write [20 16] INFO [2022-03-13 17:14:13,475] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,475] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:14:13,482] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,482] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,483] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,483] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,484] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 40] INFO [2022-03-13 17:14:13,485] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 20] INFO [2022-03-13 17:14:13,485] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,485] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,486] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,487] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:13,488] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:14:13,502] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:13,503] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,503] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:13,537] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,538] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,540] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,541] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,542] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,542] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,546] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,547] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:13,549] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,550] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:14:13,565] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,566] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,567] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:13,567] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:14:13,596] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,597] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,600] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,600] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:14:13,657] jepsen worker 84 - jepsen.util 2874 :info :write [28 46] :RequestTimeoutException INFO [2022-03-13 17:14:13,657] jepsen worker 84 - jepsen.util 2964 :invoke :read [28 nil] INFO [2022-03-13 17:14:13,661] jepsen worker 66 - jepsen.util 2676 :ok :read [22 30] INFO [2022-03-13 17:14:13,661] jepsen worker 66 - jepsen.util 2676 :invoke :read [22 nil] INFO [2022-03-13 17:14:13,669] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,670] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:14:13,675] jepsen worker 88 - jepsen.util 2068 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,675] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,682] jepsen worker 12 - jepsen.util 732 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,683] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,683] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:13,683] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:14:13,685] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,686] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:13,717] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,717] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:13,718] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:13,719] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:13,723] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,724] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,731] jepsen worker 8 - jepsen.util 728 :ok :read [2 5] INFO [2022-03-13 17:14:13,732] jepsen worker 8 - jepsen.util 728 :invoke :write [2 41] INFO [2022-03-13 17:14:13,736] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,736] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,741] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,741] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,742] jepsen worker 14 - jepsen.util 374 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,743] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:13,743] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,744] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,746] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,747] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,751] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,751] jepsen worker 3 - jepsen.util 1893 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,752] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:14:13,752] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:13,785] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:13,785] jepsen worker nemesis - couchbase.util Rebalance check count: 363 INFO [2022-03-13 17:14:13,785] jepsen worker nemesis - couchbase.util Rebalance stuck count: 263 INFO [2022-03-13 17:14:13,797] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,797] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,798] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,799] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:13,813] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,814] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,814] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,814] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,815] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 37] INFO [2022-03-13 17:14:13,815] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:14:13,816] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:14:13,824] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,825] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,826] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:14:13,827] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,828] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,835] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,839] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,841] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,842] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,844] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,844] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,851] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,852] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,853] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,853] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:13,862] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,863] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:14:13,869] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,869] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:13,872] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,873] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 33] INFO [2022-03-13 17:14:13,884] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,884] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,884] jepsen worker 33 - jepsen.util 663 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,885] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:13,885] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:13,886] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:13,906] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,907] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:13,942] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,943] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:14:13,949] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,950] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:14:13,953] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:13,954] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:14,010] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,011] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:14,026] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,026] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,047] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,048] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:14:14,052] jepsen worker 37 - jepsen.util 2017 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,052] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 42] INFO [2022-03-13 17:14:14,103] jepsen worker 3 - jepsen.util 1893 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,103] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,103] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 1] INFO [2022-03-13 17:14:14,104] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:14,110] jepsen worker 35 - jepsen.util 935 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,110] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,111] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,111] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,111] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:14,112] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:14,112] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:14,113] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:14,114] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,115] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:14,115] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,115] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,116] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,116] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,117] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:14,117] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:14:14,118] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:14:14,119] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:14:14,119] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,120] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,120] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:14:14,121] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 41] INFO [2022-03-13 17:14:14,126] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,127] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:14,127] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,127] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:14:14,129] jepsen worker 70 - jepsen.util 700 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,130] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,130] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,131] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:14:14,131] jepsen worker 43 - jepsen.util 493 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,132] jepsen worker 33 - jepsen.util 663 :invoke :write [11 49] INFO [2022-03-13 17:14:14,132] jepsen worker 43 - jepsen.util 493 :invoke :write [14 4] INFO [2022-03-13 17:14:14,133] jepsen worker 70 - jepsen.util 700 :invoke :write [23 31] INFO [2022-03-13 17:14:14,138] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,141] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:14,184] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,184] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,189] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,189] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,189] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,190] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:14:14,190] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:14,190] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,224] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,225] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,228] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,229] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,231] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,232] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,232] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,233] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:14,248] jepsen worker 37 - jepsen.util 2017 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,248] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,248] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,249] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:14,267] jepsen worker 3 - jepsen.util 1893 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,267] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,267] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,268] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,268] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:14:14,269] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,291] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,292] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,299] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,300] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:14:14,313] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,313] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,313] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:14:14,314] jepsen worker 89 - jepsen.util 2429 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,315] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,319] jepsen worker 24 - jepsen.util 564 :invoke :write [8 5] INFO [2022-03-13 17:14:14,320] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:14:14,320] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:14,324] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,324] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:14:14,325] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,326] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,326] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,327] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:14:14,327] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,328] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:14:14,365] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,366] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,367] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,368] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:14:14,403] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,404] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:14,404] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,405] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,413] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,414] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,434] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,435] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:14:14,504] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,504] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,516] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,516] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,516] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,517] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:14:14,517] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:14:14,519] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:14:14,529] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,529] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:14:14,568] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,568] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,569] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:14:14,569] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,583] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,584] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,585] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,585] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:14,586] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,587] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:14,626] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,627] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 2] INFO [2022-03-13 17:14:14,657] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,657] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,658] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,659] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 3] INFO [2022-03-13 17:14:14,659] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,660] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:14,675] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,676] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:14:14,687] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:14,687] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:14:14,790] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:14,790] jepsen worker nemesis - couchbase.util Rebalance check count: 364 INFO [2022-03-13 17:14:14,790] jepsen worker nemesis - couchbase.util Rebalance stuck count: 264 INFO [2022-03-13 17:14:14,931] jepsen worker 72 - jepsen.util 2772 :info :write [24 15] :SyncWriteAmbiguous INFO [2022-03-13 17:14:14,931] jepsen worker 72 - jepsen.util 2862 :invoke :write [24 35] INFO [2022-03-13 17:14:14,959] jepsen worker 30 - jepsen.util 2640 :ok :read [10 :nil] INFO [2022-03-13 17:14:14,959] jepsen worker 30 - jepsen.util 2640 :invoke :write [10 36] INFO [2022-03-13 17:14:14,991] jepsen worker 54 - jepsen.util 2304 :ok :read [18 21] INFO [2022-03-13 17:14:14,992] jepsen worker 54 - jepsen.util 2304 :invoke :read [18 nil] INFO [2022-03-13 17:14:15,028] jepsen worker 74 - jepsen.util 2864 :info :write [24 4] :RequestTimeoutException INFO [2022-03-13 17:14:15,028] jepsen worker 74 - jepsen.util 2954 :invoke :write [24 21] INFO [2022-03-13 17:14:15,040] jepsen worker 54 - jepsen.util 2304 :ok :read [18 21] INFO [2022-03-13 17:14:15,041] jepsen worker 54 - jepsen.util 2304 :invoke :write [18 2] INFO [2022-03-13 17:14:15,114] jepsen worker 65 - jepsen.util 3035 :info :write [21 23] :RequestTimeoutException INFO [2022-03-13 17:14:15,114] jepsen worker 65 - jepsen.util 3125 :invoke :write [21 31] INFO [2022-03-13 17:14:15,140] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,141] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:14:15,175] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,175] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,175] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,176] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 5] INFO [2022-03-13 17:14:15,179] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:15,179] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 2] INFO [2022-03-13 17:14:15,202] jepsen worker 69 - jepsen.util 699 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,202] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,202] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,204] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:15,204] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,205] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:15,205] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,207] jepsen worker 69 - jepsen.util 699 :invoke :write [23 33] INFO [2022-03-13 17:14:15,207] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:15,207] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,208] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:14:15,208] jepsen worker 76 - jepsen.util 706 :invoke :write [25 49] INFO [2022-03-13 17:14:15,216] jepsen worker 37 - jepsen.util 2017 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,217] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 26] INFO [2022-03-13 17:14:15,221] jepsen worker 33 - jepsen.util 663 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,222] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:14:15,274] jepsen worker 88 - jepsen.util 2068 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,274] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,275] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,275] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:14:15,283] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,284] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,285] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,286] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,290] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,290] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,291] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,292] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,293] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,294] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:14:15,294] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,295] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,296] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,296] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:14:15,300] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,300] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:14:15,357] jepsen worker 3 - jepsen.util 1893 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,357] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:15,374] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,374] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:15,387] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,388] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:15,421] jepsen worker 80 - jepsen.util 710 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,422] jepsen worker 80 - jepsen.util 710 :invoke :write [26 37] INFO [2022-03-13 17:14:15,464] jepsen worker 87 - jepsen.util 2607 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,464] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,465] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,465] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,467] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,467] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:15,488] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,489] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:14:15,508] jepsen worker 26 - jepsen.util 476 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,508] jepsen worker 60 - jepsen.util 600 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,508] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,508] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:15,509] jepsen worker 26 - jepsen.util 476 :invoke :write [8 19] INFO [2022-03-13 17:14:15,509] jepsen worker 6 - jepsen.util 636 :invoke :write [2 6] INFO [2022-03-13 17:14:15,534] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,534] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,535] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,536] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,537] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,538] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:15,549] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,549] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,549] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,549] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,550] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,551] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:14:15,552] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,553] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,553] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,553] jepsen worker 37 - jepsen.util 2017 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,554] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,555] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:15,553] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,555] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:14:15,555] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,556] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,556] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:14:15,557] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,565] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,565] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 49] INFO [2022-03-13 17:14:15,567] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,567] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,567] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:14:15,567] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,568] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,569] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,569] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,574] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 40] INFO [2022-03-13 17:14:15,574] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,575] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,607] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,608] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:15,620] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,620] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:14:15,657] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,657] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:15,681] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,682] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,683] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,683] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,684] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,685] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,686] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,687] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,689] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,690] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:15,704] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,705] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:15,741] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,742] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,742] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:14:15,742] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:14:15,747] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,748] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,749] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,750] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,751] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,752] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:14:15,758] jepsen worker 89 - jepsen.util 2429 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,759] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,767] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,767] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,772] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,772] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:15,773] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,774] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:14:15,795] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:15,795] jepsen worker nemesis - couchbase.util Rebalance check count: 365 INFO [2022-03-13 17:14:15,796] jepsen worker nemesis - couchbase.util Rebalance stuck count: 265 INFO [2022-03-13 17:14:15,830] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,831] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,831] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,831] jepsen worker 37 - jepsen.util 2017 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:15,832] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:15,833] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:15,876] jepsen worker 18 - jepsen.util 2448 :info :write [6 47] :RequestTimeoutException INFO [2022-03-13 17:14:15,877] jepsen worker 18 - jepsen.util 2538 :invoke :read [6 nil] INFO [2022-03-13 17:14:15,916] jepsen worker 9 - jepsen.util 2259 :info :write [3 49] :RequestTimeoutException INFO [2022-03-13 17:14:15,917] jepsen worker 9 - jepsen.util 2349 :invoke :read [3 nil] INFO [2022-03-13 17:14:16,028] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,028] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:16,031] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,032] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:14:16,057] jepsen worker 24 - jepsen.util 564 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,058] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:14:16,237] jepsen worker 32 - jepsen.util 2732 :ok :read [10 :nil] INFO [2022-03-13 17:14:16,238] jepsen worker 32 - jepsen.util 2732 :invoke :write [10 7] INFO [2022-03-13 17:14:16,242] jepsen worker 10 - jepsen.util 2800 :ok :read [3 25] INFO [2022-03-13 17:14:16,243] jepsen worker 10 - jepsen.util 2800 :invoke :read [3 nil] INFO [2022-03-13 17:14:16,250] jepsen worker 27 - jepsen.util 3267 :ok :read [9 26] INFO [2022-03-13 17:14:16,250] jepsen worker 27 - jepsen.util 3267 :invoke :read [9 nil] INFO [2022-03-13 17:14:16,292] jepsen worker 10 - jepsen.util 2800 :ok :read [3 25] INFO [2022-03-13 17:14:16,293] jepsen worker 10 - jepsen.util 2800 :invoke :read [3 nil] INFO [2022-03-13 17:14:16,299] jepsen worker 27 - jepsen.util 3267 :ok :read [9 26] INFO [2022-03-13 17:14:16,300] jepsen worker 27 - jepsen.util 3267 :invoke :write [9 30] INFO [2022-03-13 17:14:16,363] jepsen worker 10 - jepsen.util 2800 :ok :read [3 25] INFO [2022-03-13 17:14:16,364] jepsen worker 10 - jepsen.util 2800 :invoke :read [3 nil] WARN [2022-03-13 17:14:16,375] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9134361,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10076626,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9309036,"encode_duration_us":31200,"operation_id":"0x9ff6","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":10003256},{"operation_name":"upsert","last_dispatch_duration_us":9039088,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10073437,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9347963,"encode_duration_us":21305,"operation_id":"0x9fef","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10007635},{"operation_name":"upsert","last_dispatch_duration_us":16532795,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16532795,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9063917,"encode_duration_us":17554,"operation_id":"0x9f33","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":10007054}],"total_count":3}} WARN [2022-03-13 17:14:16,376] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8330205,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9551562,"total_server_duration_us":334801,"encode_duration_us":20554,"operation_id":"0x9ff9","timeout_ms":10000,"last_server_duration_us":32355,"total_duration_us":10010083},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":21813,"operation_id":"0xa195","timeout_ms":10000,"total_duration_us":10009244},{"operation_name":"upsert","last_dispatch_duration_us":9039088,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10073437,"total_server_duration_us":9347963,"encode_duration_us":21305,"operation_id":"0x9fef","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10007635},{"operation_name":"upsert","last_dispatch_duration_us":16532795,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":16532795,"total_server_duration_us":9063917,"encode_duration_us":17554,"operation_id":"0x9f33","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":10007054},{"operation_name":"upsert","last_dispatch_duration_us":8327859,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9494738,"total_server_duration_us":328942,"encode_duration_us":18661,"operation_id":"0xa017","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10004857},{"operation_name":"upsert","last_dispatch_duration_us":8130530,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8756910,"total_server_duration_us":318876,"encode_duration_us":17382,"operation_id":"0xa0ef","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10004599},{"operation_name":"upsert","last_dispatch_duration_us":9134361,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10076626,"total_server_duration_us":9309036,"encode_duration_us":31200,"operation_id":"0x9ff6","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":10003256},{"operation_name":"get","last_dispatch_duration_us":7652800,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7652800,"total_server_duration_us":10154,"operation_id":"0xa243","timeout_ms":30000,"last_server_duration_us":10154,"total_duration_us":7656050},{"operation_name":"get","last_dispatch_duration_us":5308509,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":5308509,"total_server_duration_us":15907,"operation_id":"0xa370","timeout_ms":30000,"last_server_duration_us":15907,"total_duration_us":5314127},{"operation_name":"get","last_dispatch_duration_us":1817097,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1817097,"total_server_duration_us":43,"operation_id":"0xa551","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":1819739}],"total_count":40}} INFO [2022-03-13 17:14:16,376] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,377] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:16,378] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,378] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,379] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,380] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,382] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,383] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,383] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,383] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,384] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,384] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:14:16,384] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,385] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,386] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,387] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,388] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,389] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,395] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,395] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,429] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,430] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,430] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:14:16,431] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,437] jepsen worker 52 - jepsen.util 592 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,438] jepsen worker 52 - jepsen.util 592 :invoke :write [17 24] INFO [2022-03-13 17:14:16,448] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,448] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,449] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:14:16,450] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:16,463] jepsen worker 10 - jepsen.util 2800 :ok :read [3 25] INFO [2022-03-13 17:14:16,464] jepsen worker 10 - jepsen.util 2800 :invoke :read [3 nil] INFO [2022-03-13 17:14:16,468] jepsen worker 29 - jepsen.util 2909 :info :write [9 31] :RequestTimeoutException INFO [2022-03-13 17:14:16,469] jepsen worker 29 - jepsen.util 2999 :invoke :read [9 nil] INFO [2022-03-13 17:14:16,471] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,471] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,472] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,473] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:14:16,476] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,477] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 39] INFO [2022-03-13 17:14:16,499] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,500] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,503] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,504] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,507] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,508] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,510] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,511] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:16,537] jepsen worker 10 - jepsen.util 2800 :ok :read [3 25] INFO [2022-03-13 17:14:16,537] jepsen worker 10 - jepsen.util 2800 :invoke :write [3 43] INFO [2022-03-13 17:14:16,552] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,552] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,554] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,555] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:14:16,565] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,565] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,565] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,565] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,566] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,567] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:14:16,567] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,567] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,568] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,568] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,568] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,568] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 10] INFO [2022-03-13 17:14:16,569] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:14:16,569] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,570] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,571] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:14:16,589] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,590] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:16,602] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,603] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,607] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,607] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,609] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,610] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:16,648] jepsen worker 7 - jepsen.util 637 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,648] jepsen worker 77 - jepsen.util 707 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,648] jepsen worker 77 - jepsen.util 707 :invoke :write [25 2] INFO [2022-03-13 17:14:16,649] jepsen worker 7 - jepsen.util 637 :invoke :write [2 23] INFO [2022-03-13 17:14:16,666] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,666] jepsen worker 21 - jepsen.util 831 :invoke :write [7 33] INFO [2022-03-13 17:14:16,671] jepsen worker 37 - jepsen.util 2017 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,672] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:14:16,680] jepsen worker 88 - jepsen.util 2068 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,680] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,712] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,712] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:14:16,716] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,717] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 48] INFO [2022-03-13 17:14:16,747] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,747] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,792] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,792] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,801] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:16,801] jepsen worker nemesis - couchbase.util Rebalance check count: 366 INFO [2022-03-13 17:14:16,801] jepsen worker nemesis - couchbase.util Rebalance stuck count: 266 INFO [2022-03-13 17:14:16,809] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,809] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,809] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:14:16,810] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:14:16,852] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,852] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:16,885] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,886] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:14:16,891] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,891] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,892] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,892] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:16,893] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,893] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,893] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,894] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:14:16,895] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,895] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,958] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,958] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,958] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,959] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,959] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,959] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:16,959] jepsen worker 89 - jepsen.util 2429 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,960] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:14:16,961] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 37] INFO [2022-03-13 17:14:16,961] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:14:16,963] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:14:16,963] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:14:16,973] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,974] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,975] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,976] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:16,978] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:16,979] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:14:17,042] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,042] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,046] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,046] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,046] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,047] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,048] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 6] INFO [2022-03-13 17:14:17,048] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:14:17,055] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,056] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:14:17,063] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,065] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 21] INFO [2022-03-13 17:14:17,068] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,069] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:17,071] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,072] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:17,124] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,124] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:17,149] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,149] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:14:17,163] jepsen worker 3 - jepsen.util 1893 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,164] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:17,206] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,206] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,209] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,209] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:17,222] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,223] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:17,226] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,227] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:14:17,242] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,242] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:17,264] jepsen worker 57 - jepsen.util 2577 :info :write [19 33] :RequestTimeoutException INFO [2022-03-13 17:14:17,265] jepsen worker 57 - jepsen.util 2667 :invoke :write [19 19] INFO [2022-03-13 17:14:17,271] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,271] jepsen worker 13 - jepsen.util 643 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,274] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:17,277] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:17,280] jepsen worker 53 - jepsen.util 503 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,280] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,280] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:17,281] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:17,282] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:17,282] jepsen worker 53 - jepsen.util 503 :invoke :write [17 40] INFO [2022-03-13 17:14:17,294] jepsen worker 67 - jepsen.util 2317 :info :write [22 44] :RequestTimeoutException INFO [2022-03-13 17:14:17,296] jepsen worker 67 - jepsen.util 2407 :invoke :read [22 nil] INFO [2022-03-13 17:14:17,311] jepsen worker 37 - jepsen.util 2017 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,311] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:14:17,326] jepsen worker 31 - jepsen.util 2821 :info :write [10 47] :RequestTimeoutException INFO [2022-03-13 17:14:17,326] jepsen worker 31 - jepsen.util 2911 :invoke :write [10 1] INFO [2022-03-13 17:14:17,330] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,330] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:14:17,336] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,336] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:17,339] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,340] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:14:17,362] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,363] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,364] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,364] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,366] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,366] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,374] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,375] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,377] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,378] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,379] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,379] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:17,396] jepsen worker 56 - jepsen.util 2846 :info :write [18 33] :RequestTimeoutException INFO [2022-03-13 17:14:17,397] jepsen worker 56 - jepsen.util 2936 :invoke :write [18 44] INFO [2022-03-13 17:14:17,402] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,402] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,402] jepsen worker 88 - jepsen.util 2068 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,403] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:14:17,404] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:14:17,405] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:17,431] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,432] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:17,439] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,439] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:14:17,443] jepsen worker 59 - jepsen.util 2669 :info :write [19 32] :RequestTimeoutException INFO [2022-03-13 17:14:17,444] jepsen worker 59 - jepsen.util 2759 :invoke :read [19 nil] INFO [2022-03-13 17:14:17,464] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,465] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,465] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:17,466] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:17,476] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,476] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,477] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:17,477] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:17,486] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,487] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:14:17,493] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,493] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:17,513] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,514] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:17,520] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,520] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,522] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,523] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,527] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,527] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,530] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,531] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:14:17,548] jepsen worker 44 - jepsen.util 584 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,548] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,549] jepsen worker 44 - jepsen.util 584 :invoke :write [14 0] INFO [2022-03-13 17:14:17,550] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:14:17,559] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,559] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:14:17,579] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,580] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 4] INFO [2022-03-13 17:14:17,584] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,585] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:14:17,605] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,605] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:17,630] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,630] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,630] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,631] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:17,631] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:14:17,632] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:17,662] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,663] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:14:17,687] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17427363,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17996798,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9169448,"encode_duration_us":22640,"operation_id":"0x9e75","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10003955},{"operation_name":"upsert","last_dispatch_duration_us":17071965,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17071965,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9062854,"encode_duration_us":30839,"operation_id":"0x9b90","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10007670},{"operation_name":"upsert","last_dispatch_duration_us":17461054,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17739122,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9113699,"encode_duration_us":19444,"operation_id":"0x9ea3","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10007567}],"total_count":3}} INFO [2022-03-13 17:14:17,722] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,723] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:17,734] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,735] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:14:17,741] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,742] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,742] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,742] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:17,742] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,742] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:14:17,743] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:17,743] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:14:17,759] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,759] jepsen worker 37 - jepsen.util 2017 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,759] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:14:17,760] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,767] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,768] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 30] INFO [2022-03-13 17:14:17,806] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:17,806] jepsen worker nemesis - couchbase.util Rebalance check count: 367 INFO [2022-03-13 17:14:17,806] jepsen worker nemesis - couchbase.util Rebalance stuck count: 267 INFO [2022-03-13 17:14:17,824] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,824] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:14:17,833] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,833] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,841] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,841] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,843] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,843] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,845] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,845] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,845] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:17,846] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 22] INFO [2022-03-13 17:14:17,847] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,848] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,849] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,849] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:17,872] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,872] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:14:17,888] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10972855,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10972855,"total_server_duration_us":10213,"operation_id":"0x9f35","timeout_ms":30000,"last_server_duration_us":10213,"total_duration_us":10973190},{"operation_name":"upsert","last_dispatch_duration_us":39528,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":39528,"total_server_duration_us":20841,"encode_duration_us":17861,"operation_id":"0xa306","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10003634},{"operation_name":"upsert","last_dispatch_duration_us":8594377,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8627781,"total_server_duration_us":48530,"encode_duration_us":17430,"operation_id":"0xa39c","timeout_ms":10000,"last_server_duration_us":27043,"total_duration_us":10003199},{"operation_name":"upsert","last_dispatch_duration_us":8533268,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8583701,"total_server_duration_us":51837,"encode_duration_us":27141,"operation_id":"0xa106","timeout_ms":10000,"last_server_duration_us":30350,"total_duration_us":10000362},{"operation_name":"upsert","last_dispatch_duration_us":4964969,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":7269722,"total_server_duration_us":354697,"encode_duration_us":36148,"operation_id":"0xa59e","timeout_ms":10000,"last_server_duration_us":265708,"total_duration_us":7383508},{"operation_name":"get","last_dispatch_duration_us":6710745,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":6710745,"total_server_duration_us":27,"operation_id":"0xa1cb","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":6710951},{"operation_name":"get","last_dispatch_duration_us":4579870,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4579870,"total_server_duration_us":38,"operation_id":"0xa758","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":4687053},{"operation_name":"upsert","last_dispatch_duration_us":4365105,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4365105,"total_server_duration_us":202956,"encode_duration_us":19567,"operation_id":"0xa6fe","timeout_ms":10000,"last_server_duration_us":202956,"total_duration_us":4403251},{"operation_name":"upsert","last_dispatch_duration_us":4200518,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":4200518,"total_server_duration_us":1068805,"encode_duration_us":27896,"operation_id":"0xa631","timeout_ms":10000,"last_server_duration_us":1068805,"total_duration_us":4210138},{"operation_name":"upsert","last_dispatch_duration_us":3522837,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3522837,"total_server_duration_us":77290,"encode_duration_us":19068,"operation_id":"0xa668","timeout_ms":10000,"last_server_duration_us":77290,"total_duration_us":4094144}],"total_count":40}} INFO [2022-03-13 17:14:17,917] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,917] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,918] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,919] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:17,920] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,920] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:14:17,921] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,921] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:14:17,974] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,974] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:17,974] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:17,975] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] WARN [2022-03-13 17:14:17,978] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8455878,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9910492,"total_server_duration_us":475366,"encode_duration_us":25981,"operation_id":"0xa0ed","timeout_ms":10000,"last_server_duration_us":96711,"total_duration_us":10005522},{"operation_name":"upsert","last_dispatch_duration_us":8480102,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9726486,"total_server_duration_us":350530,"encode_duration_us":19489,"operation_id":"0xa124","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10005211},{"operation_name":"upsert","last_dispatch_duration_us":17200396,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":17200396,"total_server_duration_us":9050106,"encode_duration_us":18426,"operation_id":"0x9fb4","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10004015},{"operation_name":"upsert","last_dispatch_duration_us":8396472,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9773988,"total_server_duration_us":497057,"encode_duration_us":23242,"operation_id":"0xa0eb","timeout_ms":10000,"last_server_duration_us":32451,"total_duration_us":10003079},{"operation_name":"upsert","last_dispatch_duration_us":9051580,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9389445,"total_server_duration_us":9235973,"encode_duration_us":17272,"operation_id":"0xa1e5","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":9439216},{"operation_name":"get","last_dispatch_duration_us":8978529,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8978529,"total_server_duration_us":7698,"operation_id":"0xa3e5","timeout_ms":30000,"last_server_duration_us":7698,"total_duration_us":8985865},{"operation_name":"get","last_dispatch_duration_us":7252909,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7252909,"total_server_duration_us":38,"operation_id":"0xa4f0","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":7255269},{"operation_name":"get","last_dispatch_duration_us":6838950,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":6838950,"total_server_duration_us":30823,"operation_id":"0xa336","timeout_ms":30000,"last_server_duration_us":30823,"total_duration_us":6839254},{"operation_name":"get","last_dispatch_duration_us":5414857,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":5414857,"total_server_duration_us":32,"operation_id":"0xa614","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":5415163},{"operation_name":"upsert","last_dispatch_duration_us":1492319,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1492319,"total_server_duration_us":1044097,"encode_duration_us":20673,"operation_id":"0xa7c4","timeout_ms":10000,"last_server_duration_us":1044097,"total_duration_us":1502749}],"total_count":26}} WARN [2022-03-13 17:14:17,978] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8380652,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8885212,"total_server_duration_us":215039,"encode_duration_us":23673,"operation_id":"0xa167","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10010051},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":19528,"operation_id":"0xa3fa","timeout_ms":10000,"total_duration_us":10006803},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18956,"operation_id":"0xa368","timeout_ms":10000,"total_duration_us":10002399},{"operation_name":"upsert","last_dispatch_duration_us":9210714,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9919359,"total_server_duration_us":9143637,"encode_duration_us":18289,"operation_id":"0xa19c","timeout_ms":10000,"last_server_duration_us":9068168,"total_duration_us":9938527},{"operation_name":"upsert","last_dispatch_duration_us":9252325,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9252325,"total_server_duration_us":9052231,"encode_duration_us":18748,"operation_id":"0xa20c","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9256504},{"operation_name":"upsert","last_dispatch_duration_us":9101047,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9197085,"total_server_duration_us":9084361,"encode_duration_us":24148,"operation_id":"0xa166","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":9207744},{"operation_name":"get","last_dispatch_duration_us":8688062,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8688062,"total_server_duration_us":100,"operation_id":"0xa23a","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":8688718},{"operation_name":"get","last_dispatch_duration_us":8687859,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8687859,"total_server_duration_us":38,"operation_id":"0xa23c","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":8688274},{"operation_name":"get","last_dispatch_duration_us":8687140,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8687140,"total_server_duration_us":362,"operation_id":"0xa23f","timeout_ms":30000,"last_server_duration_us":362,"total_duration_us":8687707},{"operation_name":"get","last_dispatch_duration_us":8670084,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8670084,"total_server_duration_us":56,"operation_id":"0xa251","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8670472}],"total_count":37}} WARN [2022-03-13 17:14:17,991] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9121542,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10934280,"total_server_duration_us":339837,"encode_duration_us":17422,"operation_id":"0xa161","timeout_ms":10000,"last_server_duration_us":47420,"total_duration_us":10009559},{"operation_name":"upsert","last_dispatch_duration_us":90207,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":5619429,"total_server_duration_us":204731,"encode_duration_us":20913,"operation_id":"0xa36c","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10009241},{"operation_name":"upsert","last_dispatch_duration_us":8901744,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8901744,"total_server_duration_us":22471,"encode_duration_us":18990,"operation_id":"0xa0ec","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10007866},{"operation_name":"upsert","last_dispatch_duration_us":126522,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":3695366,"total_server_duration_us":80847,"encode_duration_us":18160,"operation_id":"0xa401","timeout_ms":10000,"last_server_duration_us":22720,"total_duration_us":10006911},{"operation_name":"upsert","last_dispatch_duration_us":30542,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1815715,"total_server_duration_us":225432,"encode_duration_us":17528,"operation_id":"0xa162","timeout_ms":10000,"last_server_duration_us":18799,"total_duration_us":10006769},{"operation_name":"upsert","last_dispatch_duration_us":83540,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1177986,"total_server_duration_us":143685,"encode_duration_us":21706,"operation_id":"0xa163","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10005903},{"operation_name":"upsert","last_dispatch_duration_us":9040205,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9040205,"total_server_duration_us":26507,"encode_duration_us":46061,"operation_id":"0xa263","timeout_ms":10000,"last_server_duration_us":26507,"total_duration_us":10005323},{"operation_name":"upsert","last_dispatch_duration_us":9506705,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10385873,"total_server_duration_us":9223423,"encode_duration_us":23749,"operation_id":"0xa17d","timeout_ms":10000,"last_server_duration_us":9114986,"total_duration_us":10002855},{"operation_name":"upsert","last_dispatch_duration_us":12519935,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12519935,"total_server_duration_us":20362,"encode_duration_us":16712,"operation_id":"0x9f2f","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10001105},{"operation_name":"get","last_dispatch_duration_us":9114185,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9114185,"total_server_duration_us":14367,"operation_id":"0xa164","timeout_ms":30000,"last_server_duration_us":14367,"total_duration_us":9119373}],"total_count":35}} WARN [2022-03-13 17:14:17,991] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":46272,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":2122094,"total_server_duration_us":545084,"encode_duration_us":28016,"operation_id":"0xa3f4","timeout_ms":10000,"last_server_duration_us":25888,"total_duration_us":10009607},{"operation_name":"upsert","last_dispatch_duration_us":221614,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":2254798,"total_server_duration_us":606540,"encode_duration_us":18392,"operation_id":"0xa3ea","timeout_ms":10000,"last_server_duration_us":64603,"total_duration_us":10008690},{"operation_name":"upsert","last_dispatch_duration_us":35451,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7037394,"total_server_duration_us":541704,"encode_duration_us":17883,"operation_id":"0xa144","timeout_ms":10000,"last_server_duration_us":24497,"total_duration_us":10008060},{"operation_name":"upsert","last_dispatch_duration_us":124738,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":2125076,"total_server_duration_us":573480,"encode_duration_us":19066,"operation_id":"0xa405","timeout_ms":10000,"last_server_duration_us":68547,"total_duration_us":10005576},{"operation_name":"upsert","last_dispatch_duration_us":8361109,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8736286,"total_server_duration_us":258602,"encode_duration_us":17041,"operation_id":"0xa307","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10003866},{"operation_name":"upsert","last_dispatch_duration_us":9066324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9066324,"total_server_duration_us":9053293,"encode_duration_us":19181,"operation_id":"0xa367","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9071395},{"operation_name":"get","last_dispatch_duration_us":7711667,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7711667,"total_server_duration_us":43,"operation_id":"0xa3e7","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":7711949},{"operation_name":"upsert","last_dispatch_duration_us":4239912,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6499471,"total_server_duration_us":731756,"encode_duration_us":21513,"operation_id":"0xa56f","timeout_ms":10000,"last_server_duration_us":587887,"total_duration_us":7596485},{"operation_name":"get","last_dispatch_duration_us":5558312,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":5558312,"total_server_duration_us":8723,"operation_id":"0xa528","timeout_ms":30000,"last_server_duration_us":8723,"total_duration_us":5560113},{"operation_name":"upsert","last_dispatch_duration_us":2578525,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4770365,"total_server_duration_us":281303,"encode_duration_us":19192,"operation_id":"0xa5d2","timeout_ms":10000,"last_server_duration_us":150315,"total_duration_us":4952183}],"total_count":43}} WARN [2022-03-13 17:14:18,078] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17200396,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17200396,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9050106,"encode_duration_us":18426,"operation_id":"0x9fb4","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10004015}],"total_count":1}} WARN [2022-03-13 17:14:18,092] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9506705,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10385873,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9223423,"encode_duration_us":23749,"operation_id":"0xa17d","timeout_ms":10000,"last_server_duration_us":9114986,"total_duration_us":10002855},{"operation_name":"upsert","last_dispatch_duration_us":18267379,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18267379,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9069231,"encode_duration_us":18163,"operation_id":"0x9ab9","timeout_ms":10000,"last_server_duration_us":9069231,"total_duration_us":10008804}],"total_count":2}} INFO [2022-03-13 17:14:18,290] jepsen worker 37 - jepsen.util 2017 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,290] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,291] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,292] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:14:18,303] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,304] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:18,341] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,342] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,345] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,345] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,345] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 12] INFO [2022-03-13 17:14:18,346] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,350] jepsen worker 3 - jepsen.util 1893 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,351] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,413] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,413] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:18,423] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,423] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,424] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:14:18,424] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:14:18,437] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,437] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,437] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,437] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,438] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:14:18,440] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,441] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 19] INFO [2022-03-13 17:14:18,442] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:14:18,447] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,448] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:14:18,492] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,493] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,496] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,496] jepsen worker 89 - jepsen.util 2429 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,497] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,497] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:14:18,498] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,498] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,499] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,500] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:14:18,558] jepsen worker 3 - jepsen.util 1893 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,559] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,570] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,571] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,578] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,579] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,580] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,580] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:14:18,620] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,621] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,637] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,638] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,639] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,639] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,640] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,641] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,642] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,643] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:14:18,675] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,676] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,682] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,682] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,682] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,683] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:14:18,690] jepsen worker 73 - jepsen.util 2953 :info :write [24 3] :RequestTimeoutException INFO [2022-03-13 17:14:18,691] jepsen worker 73 - jepsen.util 3043 :invoke :read [24 nil] INFO [2022-03-13 17:14:18,695] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,696] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:14:18,725] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,725] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,726] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:14:18,726] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,734] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,735] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,736] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,736] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,737] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,738] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,741] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,741] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:14:18,755] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,756] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,759] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,760] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,761] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,762] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:18,779] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,779] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,780] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,781] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,781] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,781] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,781] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:18,782] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,782] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:14:18,783] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,783] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,783] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,784] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,784] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,784] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:14:18,785] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,785] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:14:18,796] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,797] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,798] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,799] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,800] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,809] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,810] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:14:18,812] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:18,812] jepsen worker nemesis - couchbase.util Rebalance check count: 368 INFO [2022-03-13 17:14:18,812] jepsen worker nemesis - couchbase.util Rebalance stuck count: 268 INFO [2022-03-13 17:14:18,812] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,813] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,815] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,816] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:14:18,856] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,856] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,857] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:14:18,857] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:14:18,888] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,889] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:14:18,920] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,921] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:18,940] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,940] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 21] INFO [2022-03-13 17:14:18,950] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,951] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,952] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,953] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,953] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,954] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:14:18,954] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,955] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:18,956] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:18,956] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:19,029] jepsen worker 16 - jepsen.util 2446 :info :write [5 41] :SyncWriteAmbiguous INFO [2022-03-13 17:14:19,030] jepsen worker 16 - jepsen.util 2536 :invoke :write [5 17] INFO [2022-03-13 17:14:19,030] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,030] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] INFO [2022-03-13 17:14:19,034] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,035] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,038] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,038] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:14:19,138] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,138] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,139] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:14:19,139] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:14:19,142] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,142] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:19,149] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,150] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:14:19,196] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,196] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,197] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:19,198] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,198] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,198] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,201] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,201] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,204] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,204] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:14:19,265] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,265] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,266] jepsen worker 37 - jepsen.util 2017 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,266] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,266] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 28] INFO [2022-03-13 17:14:19,267] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,269] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,269] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,269] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:14:19,270] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,275] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,276] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:14:19,307] jepsen worker 40 - jepsen.util 2830 :info :write [13 44] :RequestTimeoutException INFO [2022-03-13 17:14:19,308] jepsen worker 0 - jepsen.util 2520 :info :write [0 30] :RequestTimeoutException INFO [2022-03-13 17:14:19,308] jepsen worker 40 - jepsen.util 2920 :invoke :read [13 nil] INFO [2022-03-13 17:14:19,309] jepsen worker 0 - jepsen.util 2610 :invoke :read [0 nil] INFO [2022-03-13 17:14:19,313] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,313] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,314] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,314] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:19,315] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,315] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:14:19,346] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,346] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:14:19,365] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,365] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,371] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,371] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:14:19,407] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,407] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,407] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,408] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:14:19,408] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 2] INFO [2022-03-13 17:14:19,409] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,412] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,413] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,426] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,427] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,428] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,428] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:14:19,490] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,490] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,491] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,491] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:14:19,496] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,497] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,500] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,500] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,502] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,502] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:14:19,516] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,516] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:19,518] jepsen worker 37 - jepsen.util 2017 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,518] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:14:19,576] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,577] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:19,578] jepsen worker 87 - jepsen.util 2607 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,578] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:14:19,590] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,591] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,591] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,592] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:14:19,595] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,595] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:14:19,653] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,653] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:14:19,671] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,671] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,672] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:14:19,675] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,687] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,688] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:14:19,722] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,723] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:14:19,748] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,748] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:14:19,764] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,764] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:19,792] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,793] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:19,796] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,796] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:19,799] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,800] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:19,804] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,804] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:19,805] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,806] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:14:19,817] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:19,817] jepsen worker nemesis - couchbase.util Rebalance check count: 369 INFO [2022-03-13 17:14:19,817] jepsen worker nemesis - couchbase.util Rebalance stuck count: 269 INFO [2022-03-13 17:14:19,827] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,827] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,828] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,829] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:14:19,839] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,839] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 33] INFO [2022-03-13 17:14:19,847] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,848] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,851] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,851] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,852] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,853] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:19,922] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,923] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,923] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,924] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,925] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,925] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,926] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,927] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,928] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,928] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,945] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,945] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,947] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,947] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,948] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:19,948] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:19,992] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,992] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 7] INFO [2022-03-13 17:14:19,999] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:19,999] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,000] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,001] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:20,002] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,007] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:20,007] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,007] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,009] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:14:20,009] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:14:20,060] jepsen worker 47 - jepsen.util 2567 :info :write [15 19] :SyncWriteAmbiguous INFO [2022-03-13 17:14:20,062] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,062] jepsen worker 47 - jepsen.util 2657 :invoke :write [15 24] INFO [2022-03-13 17:14:20,064] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:14:20,066] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,067] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:14:20,107] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,107] jepsen worker 37 - jepsen.util 2017 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,108] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,108] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,110] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,110] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 34] INFO [2022-03-13 17:14:20,111] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,112] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:14:20,139] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,139] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:14:20,143] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,144] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,182] jepsen worker 89 - jepsen.util 2429 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,183] jepsen worker 21 - jepsen.util 831 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,183] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,183] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,183] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,184] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:14:20,185] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:20,186] jepsen worker 64 - jepsen.util 2674 :ok :read [21 :nil] INFO [2022-03-13 17:14:20,186] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:20,187] jepsen worker 64 - jepsen.util 2674 :invoke :read [21 nil] INFO [2022-03-13 17:14:20,188] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,188] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,188] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,188] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:14:20,188] jepsen worker 78 - jepsen.util 528 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,188] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,189] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:14:20,190] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:14:20,191] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:20,192] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:20,192] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:20,193] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,193] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:14:20,193] jepsen worker 24 - jepsen.util 564 :invoke :write [8 5] INFO [2022-03-13 17:14:20,227] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,228] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,237] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:20,238] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:20,241] jepsen worker 37 - jepsen.util 2017 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,242] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 10] INFO [2022-03-13 17:14:20,253] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,253] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:20,273] jepsen worker 18 - jepsen.util 2538 :ok :read [6 16] INFO [2022-03-13 17:14:20,274] jepsen worker 18 - jepsen.util 2538 :invoke :write [6 13] INFO [2022-03-13 17:14:20,290] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,290] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,348] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,348] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,348] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,349] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,349] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,349] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:14:20,350] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,350] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,351] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:14:20,351] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,352] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 48] INFO [2022-03-13 17:14:20,354] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,355] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,359] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:14:20,372] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,372] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,374] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,374] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,376] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,376] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,377] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,378] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:14:20,378] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,379] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,382] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,382] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,383] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,383] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:20,384] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,384] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:20,385] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:20,387] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:20,391] jepsen worker 8 - jepsen.util 728 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,392] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:20,418] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,419] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:20,441] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,442] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:14:20,485] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,486] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,489] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,490] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,491] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,491] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:20,500] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,500] jepsen worker 37 - jepsen.util 2017 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,501] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,502] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,509] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,509] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,509] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:14:20,510] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:14:20,541] jepsen worker 63 - jepsen.util 2673 :info :write [21 35] :SyncWriteAmbiguous INFO [2022-03-13 17:14:20,542] jepsen worker 63 - jepsen.util 2763 :invoke :read [21 nil] INFO [2022-03-13 17:14:20,570] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,570] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:14:20,575] jepsen worker 3 - jepsen.util 1893 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,575] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:14:20,584] jepsen worker 24 - jepsen.util 564 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,585] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:14:20,587] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,587] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 21] INFO [2022-03-13 17:14:20,593] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,593] jepsen worker 51 - jepsen.util 591 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,593] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,593] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:20,594] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:20,594] jepsen worker 61 - jepsen.util 691 :invoke :write [20 4] INFO [2022-03-13 17:14:20,649] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,650] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,658] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,659] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:20,687] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,688] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:14:20,706] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,707] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:14:20,711] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,711] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,712] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:14:20,712] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,760] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,760] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,761] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,762] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:20,763] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,763] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,764] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,764] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,764] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,765] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,788] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,789] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:14:20,803] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,804] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:14:20,818] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,818] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,818] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,818] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,819] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,819] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,820] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:14:20,823] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:20,823] jepsen worker nemesis - couchbase.util Rebalance check count: 370 INFO [2022-03-13 17:14:20,823] jepsen worker nemesis - couchbase.util Rebalance stuck count: 270 INFO [2022-03-13 17:14:20,822] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,823] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,824] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:14:20,826] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,826] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,826] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,828] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:14:20,829] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,829] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:14:20,832] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,832] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,834] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,834] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 31] INFO [2022-03-13 17:14:20,836] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,836] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:20,837] jepsen worker 15 - jepsen.util 2805 :info :write [5 14] :RequestTimeoutException INFO [2022-03-13 17:14:20,838] jepsen worker 15 - jepsen.util 2895 :invoke :write [5 12] INFO [2022-03-13 17:14:20,854] jepsen worker 50 - jepsen.util 2840 :info :write [16 30] :RequestTimeoutException INFO [2022-03-13 17:14:20,855] jepsen worker 50 - jepsen.util 2930 :invoke :write [16 37] INFO [2022-03-13 17:14:20,867] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,867] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:20,915] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,916] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:14:20,918] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,918] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,920] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,920] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:20,922] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,922] jepsen worker 89 - jepsen.util 2429 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,922] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:14:20,922] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:20,926] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,927] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:14:20,944] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,945] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:14:20,993] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,993] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,993] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:20,994] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:20,995] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:14:20,995] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:14:21,015] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,016] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,025] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,025] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,027] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:21,027] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:21,028] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,029] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,033] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,033] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,034] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,034] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,038] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,039] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,040] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,041] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,042] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,042] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,046] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,047] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,047] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,048] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:14:21,048] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,049] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,050] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,051] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:14:21,090] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,090] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,091] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,091] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,092] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,092] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:14:21,092] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,093] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:14:21,115] jepsen worker 3 - jepsen.util 1893 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,116] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 46] INFO [2022-03-13 17:14:21,129] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,130] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:14:21,192] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,192] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:14:21,195] jepsen worker 70 - jepsen.util 700 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,196] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:14:21,200] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,201] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:14:21,203] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,203] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,204] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,204] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,204] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,204] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:14:21,205] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:21,206] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:14:21,206] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,207] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,208] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,209] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,226] jepsen worker 46 - jepsen.util 2656 :info :write [15 0] :RequestTimeoutException INFO [2022-03-13 17:14:21,229] jepsen worker 46 - jepsen.util 2746 :invoke :write [15 49] INFO [2022-03-13 17:14:21,249] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,249] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,250] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,251] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:14:21,256] jepsen worker 11 - jepsen.util 2621 :info :write [3 24] :RequestTimeoutException INFO [2022-03-13 17:14:21,256] jepsen worker 11 - jepsen.util 2711 :invoke :write [3 15] INFO [2022-03-13 17:14:21,258] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,259] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:14:21,270] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,271] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,272] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:14:21,274] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,323] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,324] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:14:21,328] jepsen worker 68 - jepsen.util 2858 :info :write [22 35] :RequestTimeoutException INFO [2022-03-13 17:14:21,328] jepsen worker 68 - jepsen.util 2948 :invoke :write [22 25] INFO [2022-03-13 17:14:21,330] jepsen worker 55 - jepsen.util 2575 :info :write [18 40] :RequestTimeoutException INFO [2022-03-13 17:14:21,331] jepsen worker 55 - jepsen.util 2665 :invoke :read [18 nil] INFO [2022-03-13 17:14:21,350] jepsen worker 3 - jepsen.util 1893 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,350] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:14:21,352] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,352] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,359] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,359] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,362] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,363] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,374] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,375] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:21,394] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,395] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:14:21,417] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,417] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,418] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:14:21,418] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,463] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,464] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,464] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 5] INFO [2022-03-13 17:14:21,465] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:21,526] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,527] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:14:21,530] jepsen worker 86 - jepsen.util 2516 :info :write [28 48] :SyncWriteAmbiguous INFO [2022-03-13 17:14:21,531] jepsen worker 86 - jepsen.util 2606 :invoke :write [28 32] INFO [2022-03-13 17:14:21,545] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,546] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:14:21,570] jepsen worker 41 - jepsen.util 2651 :info :write [13 9] :RequestTimeoutException INFO [2022-03-13 17:14:21,570] jepsen worker 41 - jepsen.util 2741 :invoke :write [13 12] INFO [2022-03-13 17:14:21,573] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,574] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:21,584] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,585] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:21,595] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,595] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:14:21,613] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,613] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,613] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:21,614] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:14:21,615] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,616] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:14:21,630] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,631] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,632] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,632] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,634] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,634] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:14:21,646] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,646] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,647] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 16] INFO [2022-03-13 17:14:21,647] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,694] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,697] jepsen worker 49 - jepsen.util 2389 :info :write [16 3] :SyncWriteAmbiguous INFO [2022-03-13 17:14:21,698] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,698] jepsen worker 49 - jepsen.util 2479 :invoke :write [16 46] INFO [2022-03-13 17:14:21,700] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:21,700] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:14:21,703] jepsen worker 37 - jepsen.util 2017 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,704] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:14:21,714] jepsen worker 81 - jepsen.util 3141 :info :write [27 35] :RequestTimeoutException INFO [2022-03-13 17:14:21,714] jepsen worker 81 - jepsen.util 3231 :invoke :read [27 nil] INFO [2022-03-13 17:14:21,716] jepsen worker 1 - jepsen.util 2431 :info :write [0 43] :RequestTimeoutException INFO [2022-03-13 17:14:21,716] jepsen worker 1 - jepsen.util 2521 :invoke :write [0 13] INFO [2022-03-13 17:14:21,722] jepsen worker 19 - jepsen.util 2539 :info :write [6 44] :RequestTimeoutException INFO [2022-03-13 17:14:21,727] jepsen worker 19 - jepsen.util 2629 :invoke :read [6 nil] INFO [2022-03-13 17:14:21,733] jepsen worker 66 - jepsen.util 2676 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,734] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,734] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,734] jepsen worker 66 - jepsen.util 2676 :invoke :write [22 40] INFO [2022-03-13 17:14:21,735] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:14:21,735] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:21,756] jepsen worker 81 - jepsen.util 3231 :ok :read [27 4] INFO [2022-03-13 17:14:21,756] jepsen worker 81 - jepsen.util 3231 :invoke :read [27 nil] INFO [2022-03-13 17:14:21,775] jepsen worker 3 - jepsen.util 1893 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,775] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,775] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,776] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:14:21,777] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:14:21,777] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:21,790] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,791] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:21,793] jepsen worker 81 - jepsen.util 3231 :ok :read [27 4] INFO [2022-03-13 17:14:21,793] jepsen worker 81 - jepsen.util 3231 :invoke :read [27 nil] INFO [2022-03-13 17:14:21,816] jepsen worker 62 - jepsen.util 512 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,816] jepsen worker 69 - jepsen.util 699 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,816] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:21,816] jepsen worker 69 - jepsen.util 699 :invoke :write [23 12] INFO [2022-03-13 17:14:21,820] jepsen worker 34 - jepsen.util 754 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,821] jepsen worker 34 - jepsen.util 754 :invoke :write [11 49] INFO [2022-03-13 17:14:21,827] jepsen worker 81 - jepsen.util 3231 :ok :read [27 4] INFO [2022-03-13 17:14:21,828] jepsen worker 81 - jepsen.util 3231 :invoke :read [27 nil] INFO [2022-03-13 17:14:21,843] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,843] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,844] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:21,844] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,846] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,847] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,864] jepsen worker 81 - jepsen.util 3231 :ok :read [27 4] INFO [2022-03-13 17:14:21,865] jepsen worker 81 - jepsen.util 3231 :invoke :read [27 nil] INFO [2022-03-13 17:14:21,878] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,878] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:21,892] jepsen worker 81 - jepsen.util 3231 :ok :read [27 4] INFO [2022-03-13 17:14:21,893] jepsen worker 81 - jepsen.util 3231 :invoke :write [27 2] INFO [2022-03-13 17:14:21,902] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,902] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 28] INFO [2022-03-13 17:14:21,911] jepsen worker 28 - jepsen.util 2818 :info :write [9 20] :RequestTimeoutException INFO [2022-03-13 17:14:21,911] jepsen worker 28 - jepsen.util 2908 :invoke :read [9 nil] INFO [2022-03-13 17:14:21,925] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:21,925] jepsen worker nemesis - couchbase.util Rebalance check count: 371 INFO [2022-03-13 17:14:21,925] jepsen worker nemesis - couchbase.util Rebalance stuck count: 271 INFO [2022-03-13 17:14:21,934] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,935] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 45] INFO [2022-03-13 17:14:21,964] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,965] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:21,966] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,967] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:14:21,976] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,976] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,978] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,979] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 1] INFO [2022-03-13 17:14:21,982] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,982] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,985] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,985] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:21,993] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:21,994] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:14:22,011] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,011] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,011] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,012] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,012] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,013] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,043] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,044] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,045] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,046] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,047] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,048] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,053] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,054] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:14:22,075] jepsen worker 89 - jepsen.util 2429 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,076] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 9] INFO [2022-03-13 17:14:22,081] jepsen worker 0 - jepsen.util 2610 :ok :read [0 41] INFO [2022-03-13 17:14:22,082] jepsen worker 0 - jepsen.util 2610 :invoke :read [0 nil] INFO [2022-03-13 17:14:22,091] jepsen worker 3 - jepsen.util 1893 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,091] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,091] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,092] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,092] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,092] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,093] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:14:22,093] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,093] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:14:22,093] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,094] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,095] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,095] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,095] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,096] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,096] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,096] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:14:22,097] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:14:22,120] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,121] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,122] jepsen worker 37 - jepsen.util 2017 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,123] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,128] jepsen worker 0 - jepsen.util 2610 :ok :read [0 41] INFO [2022-03-13 17:14:22,129] jepsen worker 0 - jepsen.util 2610 :invoke :write [0 33] INFO [2022-03-13 17:14:22,130] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,131] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:14:22,146] jepsen worker 45 - jepsen.util 3015 :ok :read [15 48] INFO [2022-03-13 17:14:22,147] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:22,190] jepsen worker 48 - jepsen.util 2658 :info :write [16 8] :RequestTimeoutException INFO [2022-03-13 17:14:22,191] jepsen worker 48 - jepsen.util 2748 :invoke :write [16 23] INFO [2022-03-13 17:14:22,194] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,194] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:22,198] jepsen worker 59 - jepsen.util 2759 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,198] jepsen worker 59 - jepsen.util 2759 :invoke :read [19 nil] INFO [2022-03-13 17:14:22,199] jepsen worker 59 - jepsen.util 2759 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,200] jepsen worker 59 - jepsen.util 2759 :invoke :read [19 nil] INFO [2022-03-13 17:14:22,201] jepsen worker 59 - jepsen.util 2759 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,202] jepsen worker 59 - jepsen.util 2759 :invoke :read [19 nil] INFO [2022-03-13 17:14:22,203] jepsen worker 59 - jepsen.util 2759 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,204] jepsen worker 59 - jepsen.util 2759 :invoke :read [19 nil] INFO [2022-03-13 17:14:22,205] jepsen worker 59 - jepsen.util 2759 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,206] jepsen worker 59 - jepsen.util 2759 :invoke :read [19 nil] INFO [2022-03-13 17:14:22,208] jepsen worker 59 - jepsen.util 2759 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,209] jepsen worker 59 - jepsen.util 2759 :invoke :write [19 25] INFO [2022-03-13 17:14:22,219] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,219] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,232] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,232] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,233] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:22,233] jepsen worker 21 - jepsen.util 831 :invoke :write [7 49] INFO [2022-03-13 17:14:22,234] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,235] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:14:22,244] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,244] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,244] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,246] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:22,246] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:22,247] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:22,255] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,256] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,263] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,264] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:22,264] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,264] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,265] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,266] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:14:22,316] jepsen worker 89 - jepsen.util 2429 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,316] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,316] jepsen worker 58 - jepsen.util 2398 :info :write [19 37] :RequestTimeoutException INFO [2022-03-13 17:14:22,317] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,318] jepsen worker 58 - jepsen.util 2488 :invoke :read [19 nil] INFO [2022-03-13 17:14:22,318] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,319] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,320] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,324] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,325] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,326] jepsen worker 82 - jepsen.util 2692 :info :write [27 33] :RequestTimeoutException INFO [2022-03-13 17:14:22,326] jepsen worker 2 - jepsen.util 3062 :info :write [0 19] :RequestTimeoutException INFO [2022-03-13 17:14:22,326] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,327] jepsen worker 85 - jepsen.util 2425 :info :write [28 23] :RequestTimeoutException INFO [2022-03-13 17:14:22,327] jepsen worker 82 - jepsen.util 2782 :invoke :write [27 38] INFO [2022-03-13 17:14:22,328] jepsen worker 2 - jepsen.util 3152 :invoke :write [0 10] INFO [2022-03-13 17:14:22,328] jepsen worker 85 - jepsen.util 2515 :invoke :read [28 nil] INFO [2022-03-13 17:14:22,329] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,330] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,331] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 26] INFO [2022-03-13 17:14:22,335] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,336] jepsen worker 39 - jepsen.util 2649 :info :write [13 22] :RequestTimeoutException INFO [2022-03-13 17:14:22,336] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,337] jepsen worker 39 - jepsen.util 2739 :invoke :write [13 24] INFO [2022-03-13 17:14:22,342] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,343] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:22,402] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,403] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 27] INFO [2022-03-13 17:14:22,404] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,404] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,405] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 44] INFO [2022-03-13 17:14:22,406] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,409] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,410] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,410] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:14:22,411] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:14:22,419] jepsen worker 17 - jepsen.util 2807 :info :write [5 10] :RequestTimeoutException INFO [2022-03-13 17:14:22,420] jepsen worker 17 - jepsen.util 2897 :invoke :read [5 nil] INFO [2022-03-13 17:14:22,469] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,471] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:14:22,474] jepsen worker 89 - jepsen.util 2429 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,475] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,477] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,477] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,479] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,479] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,480] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,481] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,494] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,495] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:14:22,521] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,521] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,521] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,521] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,522] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:14:22,523] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:14:22,550] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,550] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,555] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,556] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,557] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,557] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,559] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,559] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,560] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,561] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,574] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,575] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:14:22,604] jepsen worker 88 - jepsen.util 2068 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,605] jepsen worker 85 - jepsen.util 2515 :ok :read [28 44] INFO [2022-03-13 17:14:22,605] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,606] jepsen worker 85 - jepsen.util 2515 :invoke :read [28 nil] INFO [2022-03-13 17:14:22,606] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,606] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,607] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,607] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,607] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 6] INFO [2022-03-13 17:14:22,610] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,610] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:22,611] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,612] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,612] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:14:22,657] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,657] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,658] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:14:22,659] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:14:22,664] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,665] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,681] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,681] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,681] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,681] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,683] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:22,684] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:14:22,684] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,684] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:22,685] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:22,686] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:14:22,687] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,687] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,688] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:14:22,688] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:14:22,690] jepsen worker 37 - jepsen.util 2017 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,691] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,694] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,695] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:22,708] jepsen worker 85 - jepsen.util 2515 :ok :read [28 44] INFO [2022-03-13 17:14:22,709] jepsen worker 85 - jepsen.util 2515 :invoke :read [28 nil] INFO [2022-03-13 17:14:22,729] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,729] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 23] INFO [2022-03-13 17:14:22,742] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,742] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,745] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,745] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,747] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,748] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,751] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,753] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:22,763] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,763] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,764] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,764] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,805] jepsen worker 85 - jepsen.util 2515 :ok :read [28 44] INFO [2022-03-13 17:14:22,805] jepsen worker 85 - jepsen.util 2515 :invoke :write [28 17] INFO [2022-03-13 17:14:22,810] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,811] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,816] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,817] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,817] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,817] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,818] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,819] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,819] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,820] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,820] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,820] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,821] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 15] INFO [2022-03-13 17:14:22,826] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,826] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:22,829] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,834] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,835] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:14:22,859] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,859] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,859] jepsen worker 3 - jepsen.util 1893 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,859] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,860] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 30] INFO [2022-03-13 17:14:22,860] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,861] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,861] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,863] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,863] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,863] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,863] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,864] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,864] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,865] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,865] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,866] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,866] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,866] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,867] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,867] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,867] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,868] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,868] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,868] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,869] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,871] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,871] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,871] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,872] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,873] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:22,873] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,873] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,874] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,874] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,875] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,875] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:14:22,875] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,875] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:22,876] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:14:22,876] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,877] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:14:22,890] jepsen worker 89 - jepsen.util 2429 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,891] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 45] INFO [2022-03-13 17:14:22,933] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:22,933] jepsen worker nemesis - couchbase.util Rebalance check count: 372 INFO [2022-03-13 17:14:22,933] jepsen worker nemesis - couchbase.util Rebalance stuck count: 272 INFO [2022-03-13 17:14:22,934] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,935] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,940] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,941] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:22,944] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:22,945] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:14:23,006] jepsen worker 37 - jepsen.util 2017 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,007] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 20] INFO [2022-03-13 17:14:23,015] jepsen worker 3 - jepsen.util 1893 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,016] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:23,019] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,020] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,021] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,022] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,023] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,023] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:23,072] jepsen worker 80 - jepsen.util 710 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,072] jepsen worker 21 - jepsen.util 831 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,073] jepsen worker 80 - jepsen.util 710 :invoke :write [26 38] INFO [2022-03-13 17:14:23,073] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:23,095] jepsen worker 89 - jepsen.util 2429 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,095] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:14:23,143] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,143] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:23,152] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,153] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,154] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,155] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:14:23,190] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,190] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:14:23,243] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,244] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 34] INFO [2022-03-13 17:14:23,280] jepsen worker 26 - jepsen.util 476 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,280] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,280] jepsen worker 76 - jepsen.util 706 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,281] jepsen worker 26 - jepsen.util 476 :invoke :write [8 9] INFO [2022-03-13 17:14:23,281] jepsen worker 76 - jepsen.util 706 :invoke :write [25 29] INFO [2022-03-13 17:14:23,282] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:23,286] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,287] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,292] jepsen worker 37 - jepsen.util 2017 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,295] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,333] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,333] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:23,339] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,339] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,339] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,340] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 35] INFO [2022-03-13 17:14:23,343] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:14:23,343] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:14:23,358] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,358] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 35] INFO [2022-03-13 17:14:23,365] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,365] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,365] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,366] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,367] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,367] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,367] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:14:23,368] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:14:23,403] jepsen worker 88 - jepsen.util 2068 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,404] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:14:23,448] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,448] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,449] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:23,449] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 22] INFO [2022-03-13 17:14:23,481] jepsen worker 3 - jepsen.util 1893 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,481] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 49] INFO [2022-03-13 17:14:23,487] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,488] jepsen worker 33 - jepsen.util 663 :invoke :write [11 29] INFO [2022-03-13 17:14:23,492] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,493] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:23,536] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,537] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,541] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,542] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,543] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,544] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:14:23,553] jepsen worker 89 - jepsen.util 2429 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,554] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 33] INFO [2022-03-13 17:14:23,555] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,555] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:14:23,612] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,613] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,617] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,617] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:23,633] jepsen worker 37 - jepsen.util 2017 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,633] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,646] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,646] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:23,691] jepsen worker 6 - jepsen.util 636 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,692] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:14:23,697] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,697] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,697] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,697] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,699] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:23,699] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:14:23,700] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:14:23,700] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,700] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:23,701] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:23,715] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,716] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:23,730] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,731] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:14:23,734] jepsen worker 89 - jepsen.util 2429 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,735] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 20] INFO [2022-03-13 17:14:23,786] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,786] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,790] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,791] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,792] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,792] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,794] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,795] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:14:23,800] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,801] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:14:23,802] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,803] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,803] jepsen worker 3 - jepsen.util 1893 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,804] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:14:23,813] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,813] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 35] INFO [2022-03-13 17:14:23,877] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,877] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:23,881] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,881] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,881] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:23,882] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,882] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:14:23,882] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:23,900] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,900] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,900] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:23,901] jepsen worker 52 - jepsen.util 592 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,901] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,901] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:23,902] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:23,903] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:23,904] jepsen worker 89 - jepsen.util 2429 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,905] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:23,906] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,907] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 34] INFO [2022-03-13 17:14:23,924] jepsen worker 84 - jepsen.util 2964 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,925] jepsen worker 84 - jepsen.util 2964 :invoke :read [28 nil] INFO [2022-03-13 17:14:23,938] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:23,938] jepsen worker nemesis - couchbase.util Rebalance check count: 373 INFO [2022-03-13 17:14:23,938] jepsen worker nemesis - couchbase.util Rebalance stuck count: 273 INFO [2022-03-13 17:14:23,954] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,955] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:23,964] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,964] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 34] INFO [2022-03-13 17:14:23,969] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:23,969] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:24,041] jepsen worker 30 - jepsen.util 2640 :info :write [10 36] :SyncWriteAmbiguous INFO [2022-03-13 17:14:24,042] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,042] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,043] jepsen worker 30 - jepsen.util 2730 :invoke :read [10 nil] INFO [2022-03-13 17:14:24,046] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,047] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:14:24,051] jepsen worker 37 - jepsen.util 2017 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,051] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,052] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:14:24,052] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,066] jepsen worker 84 - jepsen.util 2964 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,067] jepsen worker 84 - jepsen.util 2964 :invoke :read [28 nil] INFO [2022-03-13 17:14:24,074] jepsen worker 67 - jepsen.util 2407 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,075] jepsen worker 40 - jepsen.util 2920 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,076] jepsen worker 40 - jepsen.util 2920 :invoke :read [13 nil] INFO [2022-03-13 17:14:24,077] jepsen worker 67 - jepsen.util 2407 :invoke :write [22 3] INFO [2022-03-13 17:14:24,083] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,084] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 39] INFO [2022-03-13 17:14:24,099] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,099] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,099] jepsen worker 89 - jepsen.util 2429 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,100] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 11] INFO [2022-03-13 17:14:24,108] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,108] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] INFO [2022-03-13 17:14:24,115] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,116] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:14:24,150] jepsen worker 58 - jepsen.util 2488 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,151] jepsen worker 58 - jepsen.util 2488 :invoke :read [19 nil] INFO [2022-03-13 17:14:24,152] jepsen worker 40 - jepsen.util 2920 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,153] jepsen worker 40 - jepsen.util 2920 :invoke :read [13 nil] INFO [2022-03-13 17:14:24,154] jepsen worker 43 - jepsen.util 493 :info :write [14 4] :RequestTimeoutException INFO [2022-03-13 17:14:24,155] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:24,177] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,177] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,178] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,179] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,182] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,182] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,182] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,183] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 18] INFO [2022-03-13 17:14:24,184] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,184] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:14:24,196] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,197] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,202] jepsen worker 40 - jepsen.util 2920 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,202] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,202] jepsen worker 58 - jepsen.util 2488 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,203] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:24,203] jepsen worker 58 - jepsen.util 2488 :invoke :write [19 30] INFO [2022-03-13 17:14:24,205] jepsen worker 40 - jepsen.util 2920 :invoke :read [13 nil] INFO [2022-03-13 17:14:24,216] jepsen worker 84 - jepsen.util 2964 :ok :read [28 44] INFO [2022-03-13 17:14:24,217] jepsen worker 84 - jepsen.util 2964 :invoke :write [28 26] INFO [2022-03-13 17:14:24,283] jepsen worker 89 - jepsen.util 2429 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,283] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,284] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,285] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,286] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,286] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 0] INFO [2022-03-13 17:14:24,303] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,304] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,305] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,305] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:24,307] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,308] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:14:24,361] jepsen worker 88 - jepsen.util 2068 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,361] jepsen worker 37 - jepsen.util 2017 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,362] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,362] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,392] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,392] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,402] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,403] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,404] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,405] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,406] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,407] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:14:24,424] jepsen worker 3 - jepsen.util 1893 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,424] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,425] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:14:24,425] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 13] INFO [2022-03-13 17:14:24,428] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,428] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:14:24,429] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,430] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,496] jepsen worker 77 - jepsen.util 707 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,497] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:24,501] jepsen worker 89 - jepsen.util 2429 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,501] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,502] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,502] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,509] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,509] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,511] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,512] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:24,519] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,520] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,524] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,524] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,530] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,530] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,531] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:24,531] jepsen worker 78 - jepsen.util 528 :invoke :write [26 18] INFO [2022-03-13 17:14:24,535] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,536] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,541] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,541] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,545] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,545] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,552] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,552] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,553] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:14:24,553] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 47] INFO [2022-03-13 17:14:24,579] jepsen worker 40 - jepsen.util 2920 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,580] jepsen worker 40 - jepsen.util 2920 :invoke :write [13 2] INFO [2022-03-13 17:14:24,614] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,615] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,620] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,621] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:24,622] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,623] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,622] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,623] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,623] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 9] INFO [2022-03-13 17:14:24,624] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 13] INFO [2022-03-13 17:14:24,625] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,625] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,626] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,627] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:14:24,645] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,645] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,696] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,696] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:24,698] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,699] jepsen worker 22 - jepsen.util 652 :invoke :write [7 0] INFO [2022-03-13 17:14:24,714] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,715] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:24,745] jepsen worker 3 - jepsen.util 1893 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,746] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,748] jepsen worker 37 - jepsen.util 2017 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,749] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,800] jepsen worker 89 - jepsen.util 2429 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,801] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,803] jepsen worker 88 - jepsen.util 2068 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,803] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,804] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,804] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:14:24,807] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,808] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,812] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,812] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 10] INFO [2022-03-13 17:14:24,865] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,867] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:24,870] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,871] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,870] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,872] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:24,887] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,887] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,888] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,888] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 11] INFO [2022-03-13 17:14:24,895] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,895] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,895] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,895] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,896] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:14:24,896] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,897] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:14:24,897] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,897] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,898] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:14:24,899] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:24,900] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,901] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:14:24,905] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:24,905] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,906] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:24,906] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,907] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:14:24,925] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:24,926] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:24,944] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:24,944] jepsen worker nemesis - couchbase.util Rebalance check count: 374 INFO [2022-03-13 17:14:24,945] jepsen worker nemesis - couchbase.util Rebalance stuck count: 274 INFO [2022-03-13 17:14:24,958] jepsen worker 72 - jepsen.util 2862 :info :write [24 35] :RequestTimeoutException INFO [2022-03-13 17:14:24,959] jepsen worker 72 - jepsen.util 2952 :invoke :write [24 25] INFO [2022-03-13 17:14:24,999] jepsen worker 89 - jepsen.util 2429 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,000] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 2] INFO [2022-03-13 17:14:25,000] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,001] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,005] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,005] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:25,016] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,017] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,050] jepsen worker 74 - jepsen.util 2954 :info :write [24 21] :RequestTimeoutException INFO [2022-03-13 17:14:25,051] jepsen worker 74 - jepsen.util 3044 :invoke :read [24 nil] INFO [2022-03-13 17:14:25,064] jepsen worker 54 - jepsen.util 2304 :info :write [18 2] :RequestTimeoutException INFO [2022-03-13 17:14:25,064] jepsen worker 54 - jepsen.util 2394 :invoke :read [18 nil] INFO [2022-03-13 17:14:25,083] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,083] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:25,093] jepsen worker 37 - jepsen.util 2017 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,094] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 4] INFO [2022-03-13 17:14:25,108] jepsen worker 3 - jepsen.util 1893 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,108] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,109] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:25,109] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:25,116] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,117] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:14:25,144] jepsen worker 65 - jepsen.util 3125 :info :write [21 31] :RequestTimeoutException INFO [2022-03-13 17:14:25,144] jepsen worker 65 - jepsen.util 3215 :invoke :read [21 nil] INFO [2022-03-13 17:14:25,174] jepsen worker 34 - jepsen.util 754 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,174] jepsen worker 34 - jepsen.util 754 :invoke :write [11 8] INFO [2022-03-13 17:14:25,177] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,177] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,182] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,183] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:14:25,191] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,191] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:14:25,216] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,216] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:25,236] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,236] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:25,237] jepsen worker 89 - jepsen.util 2429 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,238] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,240] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,240] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,245] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,245] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:14:25,293] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,293] jepsen worker 37 - jepsen.util 2017 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,294] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:25,295] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,309] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,309] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,309] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,310] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,311] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:14:25,311] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,312] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 9] INFO [2022-03-13 17:14:25,312] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:14:25,313] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,313] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,315] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,315] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:14:25,315] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,316] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:25,316] jepsen worker 7 - jepsen.util 637 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,319] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:25,319] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:25,320] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 11] INFO [2022-03-13 17:14:25,367] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,368] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:25,415] jepsen worker 27 - jepsen.util 3267 :info :write [9 30] :SyncWriteAmbiguous INFO [2022-03-13 17:14:25,415] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,415] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,416] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,417] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:14:25,418] jepsen worker 27 - jepsen.util 3357 :invoke :read [9 nil] INFO [2022-03-13 17:14:25,426] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,427] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:14:25,439] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,439] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,475] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,475] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,476] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,476] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,479] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,479] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:14:25,500] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,500] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,504] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,505] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:14:25,516] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,517] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:14:25,518] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,519] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:14:25,523] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,523] jepsen worker 22 - jepsen.util 652 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,524] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:14:25,525] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:14:25,537] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,537] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,537] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,538] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:25,538] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,538] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:14:25,540] jepsen worker 14 - jepsen.util 374 :invoke :write [4 41] INFO [2022-03-13 17:14:25,543] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:25,545] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,545] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,548] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,548] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,548] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,549] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:25,549] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:25,550] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:25,551] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,552] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,552] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:25,552] jepsen worker 33 - jepsen.util 663 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,552] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:25,553] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:25,553] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,554] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:14:25,554] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,555] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:25,556] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,557] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:14:25,571] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,571] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,572] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 38] INFO [2022-03-13 17:14:25,572] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:25,574] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,575] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:14:25,644] jepsen worker 3 - jepsen.util 1893 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,644] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 18] INFO [2022-03-13 17:14:25,645] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,646] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 29] INFO [2022-03-13 17:14:25,664] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,664] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:14:25,720] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,720] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] INFO [2022-03-13 17:14:25,728] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,728] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:14:25,732] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,733] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,734] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,736] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:25,744] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,745] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,746] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 1] INFO [2022-03-13 17:14:25,746] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,812] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,813] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:14:25,836] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,837] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:25,857] jepsen worker 37 - jepsen.util 2017 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,858] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:25,911] jepsen worker 89 - jepsen.util 2429 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,911] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 4] INFO [2022-03-13 17:14:25,916] jepsen worker 3 - jepsen.util 1893 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,916] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:25,916] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,917] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:25,917] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,918] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:25,932] jepsen worker 80 - jepsen.util 710 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,932] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,932] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,933] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:25,934] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:25,934] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:25,938] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,939] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:14:25,940] jepsen worker 14 - jepsen.util 374 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,941] jepsen worker 14 - jepsen.util 374 :invoke :write [4 39] INFO [2022-03-13 17:14:25,950] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:25,950] jepsen worker nemesis - couchbase.util Rebalance check count: 375 INFO [2022-03-13 17:14:25,950] jepsen worker nemesis - couchbase.util Rebalance stuck count: 275 INFO [2022-03-13 17:14:25,979] jepsen worker 88 - jepsen.util 2068 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:25,979] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:14:26,021] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,021] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,022] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,022] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:14:26,025] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,026] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,027] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,027] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,028] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:14:26,028] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:26,061] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,061] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,061] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,062] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:14:26,062] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,063] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,115] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,115] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,115] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,116] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,117] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:14:26,117] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,131] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,131] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,134] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,135] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:14:26,137] jepsen worker 89 - jepsen.util 2429 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,138] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,141] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,142] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 18] INFO [2022-03-13 17:14:26,165] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,165] jepsen worker 44 - jepsen.util 584 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,165] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,166] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,166] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,165] jepsen worker 53 - jepsen.util 503 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,166] jepsen worker 44 - jepsen.util 584 :invoke :write [14 30] INFO [2022-03-13 17:14:26,167] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,168] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,168] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,169] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:26,169] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:26,170] jepsen worker 51 - jepsen.util 591 :invoke :write [17 27] INFO [2022-03-13 17:14:26,171] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,183] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,183] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,184] jepsen worker 75 - jepsen.util 435 :invoke :write [25 20] INFO [2022-03-13 17:14:26,184] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:14:26,260] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,260] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,261] jepsen worker 32 - jepsen.util 2732 :info :write [10 7] :RequestTimeoutException INFO [2022-03-13 17:14:26,261] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:14:26,262] jepsen worker 32 - jepsen.util 2822 :invoke :read [10 nil] INFO [2022-03-13 17:14:26,262] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,275] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,275] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,275] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,275] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,275] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,276] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:26,276] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 11] INFO [2022-03-13 17:14:26,276] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,277] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,277] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,278] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:14:26,279] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:14:26,285] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,286] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:14:26,293] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,294] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,295] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,296] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:26,330] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,331] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,348] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,349] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:26,354] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,355] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:26,356] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,357] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:26,358] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,359] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:26,378] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,379] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:14:26,383] jepsen worker 89 - jepsen.util 2429 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,384] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 46] INFO [2022-03-13 17:14:26,394] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,395] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,396] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,397] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,408] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,409] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,415] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,415] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] WARN [2022-03-13 17:14:26,460] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21689448,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21689448,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9077736,"encode_duration_us":21813,"operation_id":"0xa195","timeout_ms":10000,"last_server_duration_us":9077736,"total_duration_us":10009244}],"total_count":1}} WARN [2022-03-13 17:14:26,460] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10265953,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10265953,"total_server_duration_us":7645,"operation_id":"0xa7bf","timeout_ms":30000,"last_server_duration_us":7645,"total_duration_us":10266360},{"operation_name":"get","last_dispatch_duration_us":10235910,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10235910,"total_server_duration_us":8176,"operation_id":"0xa6bf","timeout_ms":30000,"last_server_duration_us":8176,"total_duration_us":10240425},{"operation_name":"upsert","last_dispatch_duration_us":10222976,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10222976,"total_server_duration_us":61012,"encode_duration_us":30855,"operation_id":"0xa6c3","timeout_ms":10000,"last_server_duration_us":61012,"total_duration_us":10008737},{"operation_name":"upsert","last_dispatch_duration_us":10013762,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10429870,"total_server_duration_us":125491,"encode_duration_us":17716,"operation_id":"0xa65f","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10006748},{"operation_name":"upsert","last_dispatch_duration_us":10000415,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13088151,"total_server_duration_us":182258,"encode_duration_us":87742,"operation_id":"0xa4c2","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10003712},{"operation_name":"upsert","last_dispatch_duration_us":93985,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":93985,"total_server_duration_us":23138,"encode_duration_us":19352,"operation_id":"0xa6a8","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10000959},{"operation_name":"upsert","last_dispatch_duration_us":9152099,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9528783,"total_server_duration_us":9106831,"encode_duration_us":21535,"operation_id":"0xa630","timeout_ms":10000,"last_server_duration_us":9081990,"total_duration_us":9549281},{"operation_name":"upsert","last_dispatch_duration_us":3488764,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3488764,"total_server_duration_us":2654248,"encode_duration_us":23220,"operation_id":"0xa91c","timeout_ms":10000,"last_server_duration_us":2654248,"total_duration_us":3492548},{"operation_name":"get","last_dispatch_duration_us":3111828,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3111828,"total_server_duration_us":56,"operation_id":"0xa93d","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":3112937},{"operation_name":"get","last_dispatch_duration_us":3109785,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3109785,"total_server_duration_us":32,"operation_id":"0xa93e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3110142}],"total_count":37}} INFO [2022-03-13 17:14:26,478] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,478] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,478] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,478] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,479] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,479] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:14:26,480] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,481] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:14:26,481] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,481] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,482] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,483] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:14:26,483] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,484] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 16] INFO [2022-03-13 17:14:26,547] jepsen worker 89 - jepsen.util 2429 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,547] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,549] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,550] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 6] INFO [2022-03-13 17:14:26,555] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,556] jepsen worker 24 - jepsen.util 564 :invoke :write [8 28] INFO [2022-03-13 17:14:26,570] jepsen worker 10 - jepsen.util 2800 :info :write [3 43] :RequestTimeoutException INFO [2022-03-13 17:14:26,571] jepsen worker 10 - jepsen.util 2890 :invoke :read [3 nil] INFO [2022-03-13 17:14:26,578] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,579] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:14:26,605] jepsen worker 3 - jepsen.util 1893 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,605] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:14:26,646] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,647] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:14:26,660] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,661] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:14:26,694] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,695] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,727] jepsen worker 37 - jepsen.util 2017 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,728] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,751] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,752] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:14:26,753] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,754] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,755] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:14:26,755] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,757] jepsen worker 78 - jepsen.util 528 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,757] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,758] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:26,758] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:14:26,765] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,766] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:26,784] jepsen worker 89 - jepsen.util 2429 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,785] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:14:26,791] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,792] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] INFO [2022-03-13 17:14:26,813] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,813] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:14:26,825] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,825] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,825] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 27] INFO [2022-03-13 17:14:26,826] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:14:26,828] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,828] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,869] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,869] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,870] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,871] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:26,871] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,872] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,873] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:26,873] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 43] INFO [2022-03-13 17:14:26,881] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,881] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,882] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:26,882] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:14:26,882] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,883] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:14:26,941] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,942] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:26,958] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:26,958] jepsen worker nemesis - couchbase.util Rebalance check count: 376 INFO [2022-03-13 17:14:26,958] jepsen worker nemesis - couchbase.util Rebalance stuck count: 276 INFO [2022-03-13 17:14:26,961] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,961] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,962] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 38] INFO [2022-03-13 17:14:26,962] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:14:26,975] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,976] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:26,993] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,995] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:14:26,997] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,997] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,998] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:26,998] jepsen worker 61 - jepsen.util 691 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:26,998] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:26,999] jepsen worker 61 - jepsen.util 691 :invoke :write [20 30] INFO [2022-03-13 17:14:27,007] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,007] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,009] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,010] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:14:27,028] jepsen worker 37 - jepsen.util 2017 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,029] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:14:27,078] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,079] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:27,091] jepsen worker 3 - jepsen.util 1893 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,091] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,092] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,092] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,108] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,109] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:14:27,139] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,139] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:27,159] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,159] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:14:27,186] jepsen worker 89 - jepsen.util 2429 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,187] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,194] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,194] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 14] INFO [2022-03-13 17:14:27,197] jepsen worker 26 - jepsen.util 476 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,197] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,197] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:27,197] jepsen worker 69 - jepsen.util 699 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,198] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:27,199] jepsen worker 43 - jepsen.util 583 :invoke :write [14 24] INFO [2022-03-13 17:14:27,199] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:27,199] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:27,199] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,200] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:27,201] jepsen worker 75 - jepsen.util 435 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,201] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:14:27,203] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,204] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:27,205] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,205] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:27,207] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,207] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:27,215] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,215] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,215] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,216] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,216] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:14:27,217] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:14:27,269] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,270] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,277] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,277] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,278] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,278] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,279] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,279] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,280] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,281] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:14:27,288] jepsen worker 57 - jepsen.util 2667 :info :write [19 19] :RequestTimeoutException INFO [2022-03-13 17:14:27,289] jepsen worker 57 - jepsen.util 2757 :invoke :read [19 nil] INFO [2022-03-13 17:14:27,289] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,290] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 8] INFO [2022-03-13 17:14:27,290] jepsen worker 57 - jepsen.util 2757 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,292] jepsen worker 57 - jepsen.util 2757 :invoke :write [19 46] INFO [2022-03-13 17:14:27,330] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,331] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:27,340] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,340] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:14:27,354] jepsen worker 31 - jepsen.util 2911 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:14:27,355] jepsen worker 31 - jepsen.util 3001 :invoke :read [10 nil] INFO [2022-03-13 17:14:27,386] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,387] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,388] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,389] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,391] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,391] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,391] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,391] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:14:27,392] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,392] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:14:27,394] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,395] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:14:27,400] jepsen worker 14 - jepsen.util 374 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,400] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,401] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,401] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:27,402] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,402] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:27,402] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,402] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:27,402] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,403] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:14:27,404] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:27,405] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:27,405] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:14:27,406] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:27,426] jepsen worker 56 - jepsen.util 2936 :info :write [18 44] :RequestTimeoutException INFO [2022-03-13 17:14:27,427] jepsen worker 56 - jepsen.util 3026 :invoke :read [18 nil] INFO [2022-03-13 17:14:27,443] jepsen worker 76 - jepsen.util 706 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,443] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,443] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:27,444] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:27,446] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,446] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:27,446] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,447] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,451] jepsen worker 34 - jepsen.util 754 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,451] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,452] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,452] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,452] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:27,453] jepsen worker 80 - jepsen.util 710 :invoke :write [26 4] INFO [2022-03-13 17:14:27,454] jepsen worker 52 - jepsen.util 592 :invoke :write [17 16] INFO [2022-03-13 17:14:27,454] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,455] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,455] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,456] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 24] INFO [2022-03-13 17:14:27,457] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:27,524] jepsen worker 89 - jepsen.util 2429 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,525] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 24] INFO [2022-03-13 17:14:27,526] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,527] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:14:27,536] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,536] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,611] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,611] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,612] jepsen worker 37 - jepsen.util 2017 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,612] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,613] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,613] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,613] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,613] jepsen worker 78 - jepsen.util 528 :invoke :write [26 16] INFO [2022-03-13 17:14:27,614] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,618] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,619] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:14:27,620] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:27,622] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,622] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,623] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:27,623] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,623] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:27,624] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:14:27,624] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,625] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,625] jepsen worker 33 - jepsen.util 663 :invoke :write [11 45] INFO [2022-03-13 17:14:27,626] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:27,627] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:27,628] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,628] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,628] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,629] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:14:27,629] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,630] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:27,631] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:14:27,631] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,632] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,654] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,655] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:27,667] jepsen worker 24 - jepsen.util 564 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,668] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:14:27,704] jepsen worker 89 - jepsen.util 2429 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,704] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,705] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:14:27,706] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 25] INFO [2022-03-13 17:14:27,714] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,715] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:14:27,727] jepsen worker 3 - jepsen.util 1893 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,727] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,727] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,727] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,727] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,728] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,728] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,729] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,729] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,729] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,730] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,730] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,732] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,732] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,732] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,732] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,732] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:14:27,733] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 3] INFO [2022-03-13 17:14:27,733] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:14:27,734] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,735] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,735] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:14:27,772] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,773] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:14:27,794] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,795] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,824] jepsen worker 79 - jepsen.util 1069 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,824] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,824] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 0] INFO [2022-03-13 17:14:27,825] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:27,826] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,827] jepsen worker 62 - jepsen.util 512 :invoke :write [20 11] INFO [2022-03-13 17:14:27,860] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,861] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] WARN [2022-03-13 17:14:27,863] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17827226,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17866754,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9384382,"encode_duration_us":17861,"operation_id":"0xa306","timeout_ms":10000,"last_server_duration_us":9363541,"total_duration_us":10003634}],"total_count":1}} INFO [2022-03-13 17:14:27,866] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,867] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:14:27,877] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,878] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,888] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,888] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,889] jepsen worker 8 - jepsen.util 728 :invoke :write [2 27] INFO [2022-03-13 17:14:27,893] jepsen worker 26 - jepsen.util 476 :invoke :write [8 46] INFO [2022-03-13 17:14:27,894] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,894] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,894] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:27,895] jepsen worker 23 - jepsen.util 563 :invoke :write [7 15] INFO [2022-03-13 17:14:27,911] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,912] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:27,934] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,936] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] WARN [2022-03-13 17:14:27,962] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":6149042,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":6149042,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":383701,"encode_duration_us":18389,"operation_id":"0xa7ff","timeout_ms":10000,"last_server_duration_us":383701,"total_duration_us":10002530}],"total_count":1}} WARN [2022-03-13 17:14:27,963] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8002974,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9624436,"total_server_duration_us":526845,"encode_duration_us":18164,"operation_id":"0xa6f7","timeout_ms":10000,"last_server_duration_us":66694,"total_duration_us":10008849},{"operation_name":"upsert","last_dispatch_duration_us":9054926,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11167306,"total_server_duration_us":9428101,"encode_duration_us":17723,"operation_id":"0xa686","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10007085},{"operation_name":"upsert","last_dispatch_duration_us":8065740,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9500482,"total_server_duration_us":469576,"encode_duration_us":21200,"operation_id":"0xa71a","timeout_ms":10000,"last_server_duration_us":60506,"total_duration_us":10006386},{"operation_name":"upsert","last_dispatch_duration_us":8914443,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8914443,"total_server_duration_us":22223,"encode_duration_us":26689,"operation_id":"0xa8ff","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10005733},{"operation_name":"upsert","last_dispatch_duration_us":8948032,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9199755,"total_server_duration_us":111384,"encode_duration_us":16841,"operation_id":"0xa8d0","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10005026},{"operation_name":"upsert","last_dispatch_duration_us":62539,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":1418075,"total_server_duration_us":96173,"encode_duration_us":17748,"operation_id":"0xa85a","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":10003353},{"operation_name":"upsert","last_dispatch_duration_us":9090948,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9090948,"total_server_duration_us":9030999,"encode_duration_us":18218,"operation_id":"0xa8d2","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":9094801},{"operation_name":"get","last_dispatch_duration_us":6777119,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":6777119,"total_server_duration_us":22971,"operation_id":"0xa94f","timeout_ms":30000,"last_server_duration_us":22971,"total_duration_us":6777552},{"operation_name":"get","last_dispatch_duration_us":4765116,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4765116,"total_server_duration_us":32,"operation_id":"0xaa2e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":4765360},{"operation_name":"upsert","last_dispatch_duration_us":3281023,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3281023,"total_server_duration_us":344976,"encode_duration_us":28210,"operation_id":"0xa975","timeout_ms":10000,"last_server_duration_us":344976,"total_duration_us":3287588}],"total_count":16}} INFO [2022-03-13 17:14:27,964] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:27,964] jepsen worker nemesis - couchbase.util Rebalance check count: 377 INFO [2022-03-13 17:14:27,964] jepsen worker nemesis - couchbase.util Rebalance stuck count: 277 INFO [2022-03-13 17:14:27,967] jepsen worker 37 - jepsen.util 2017 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,967] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException WARN [2022-03-13 17:14:27,967] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10733576,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10733576,"total_server_duration_us":88840,"operation_id":"0xa530","timeout_ms":30000,"last_server_duration_us":88840,"total_duration_us":10733808},{"operation_name":"upsert","last_dispatch_duration_us":343831,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10119369,"total_server_duration_us":267083,"encode_duration_us":21369,"operation_id":"0xa70e","timeout_ms":10000,"last_server_duration_us":34512,"total_duration_us":10009899},{"operation_name":"upsert","last_dispatch_duration_us":443483,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10262406,"total_server_duration_us":246095,"encode_duration_us":18214,"operation_id":"0xa70f","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10007856},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":20980,"operation_id":"0xa95d","timeout_ms":10000,"total_duration_us":10006846},{"operation_name":"upsert","last_dispatch_duration_us":413044,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9738380,"total_server_duration_us":194727,"encode_duration_us":26077,"operation_id":"0xa710","timeout_ms":10000,"last_server_duration_us":34612,"total_duration_us":10005253},{"operation_name":"upsert","last_dispatch_duration_us":315940,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9941315,"total_server_duration_us":275191,"encode_duration_us":34657,"operation_id":"0xa712","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10005117},{"operation_name":"get","last_dispatch_duration_us":9376496,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9376496,"total_server_duration_us":16848,"operation_id":"0xa61b","timeout_ms":30000,"last_server_duration_us":16848,"total_duration_us":9379405},{"operation_name":"upsert","last_dispatch_duration_us":9360438,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9360438,"total_server_duration_us":9062854,"encode_duration_us":18989,"operation_id":"0xa711","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":9365767},{"operation_name":"get","last_dispatch_duration_us":8066653,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8066653,"total_server_duration_us":421,"operation_id":"0xa7c0","timeout_ms":30000,"last_server_duration_us":421,"total_duration_us":8071700},{"operation_name":"get","last_dispatch_duration_us":4378816,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":4378816,"total_server_duration_us":49,"operation_id":"0xa8c9","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":4380895}],"total_count":37}} INFO [2022-03-13 17:14:27,967] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:14:27,967] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":14180003,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14180003,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9026756,"encode_duration_us":18956,"operation_id":"0xa368","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10002399}],"total_count":1}} INFO [2022-03-13 17:14:27,968] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:27,968] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 8] INFO [2022-03-13 17:14:27,969] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:27,973] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,973] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,974] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:14:27,974] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 19] INFO [2022-03-13 17:14:27,986] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:27,987] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:28,026] jepsen worker 3 - jepsen.util 1893 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,026] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:28,035] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,036] jepsen worker 14 - jepsen.util 374 :invoke :write [4 20] INFO [2022-03-13 17:14:28,044] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,045] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:28,047] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,049] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] WARN [2022-03-13 17:14:28,063] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8525136,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9705003,"total_server_duration_us":248176,"encode_duration_us":17158,"operation_id":"0xa51a","timeout_ms":10000,"last_server_duration_us":18799,"total_duration_us":10009911},{"operation_name":"upsert","last_dispatch_duration_us":11061965,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11061965,"total_server_duration_us":33328,"encode_duration_us":16782,"operation_id":"0xa857","timeout_ms":10000,"last_server_duration_us":33328,"total_duration_us":10009436},{"operation_name":"upsert","last_dispatch_duration_us":8568132,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9693865,"total_server_duration_us":192635,"encode_duration_us":19843,"operation_id":"0xa51c","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10008308},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":17474,"operation_id":"0xa94d","timeout_ms":10000,"total_duration_us":10005128},{"operation_name":"upsert","last_dispatch_duration_us":7821880,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":7821880,"total_server_duration_us":19185,"encode_duration_us":21018,"operation_id":"0xa65e","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10004429},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":20372,"operation_id":"0xa619","timeout_ms":10000,"total_duration_us":10001547},{"operation_name":"upsert","last_dispatch_duration_us":7559867,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9787178,"total_server_duration_us":338602,"encode_duration_us":19242,"operation_id":"0xa5e7","timeout_ms":10000,"last_server_duration_us":203378,"total_duration_us":9970515},{"operation_name":"upsert","last_dispatch_duration_us":9266454,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9702321,"total_server_duration_us":9119842,"encode_duration_us":17647,"operation_id":"0xa51b","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":9729750},{"operation_name":"upsert","last_dispatch_duration_us":8583642,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":8583642,"total_server_duration_us":396248,"encode_duration_us":24220,"operation_id":"0xa981","timeout_ms":10000,"last_server_duration_us":396248,"total_duration_us":8591501},{"operation_name":"upsert","last_dispatch_duration_us":8044949,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":8044949,"total_server_duration_us":618768,"encode_duration_us":20145,"operation_id":"0xa864","timeout_ms":10000,"last_server_duration_us":618768,"total_duration_us":8050933}],"total_count":23}} WARN [2022-03-13 17:14:28,064] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9054926,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11167306,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9428101,"encode_duration_us":17723,"operation_id":"0xa686","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10007085}],"total_count":1}} INFO [2022-03-13 17:14:28,072] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,072] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,073] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:28,073] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:14:28,078] jepsen worker 37 - jepsen.util 2017 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,079] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,093] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,093] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,094] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:28,095] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:28,140] jepsen worker 83 - jepsen.util 2153 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,140] jepsen worker 89 - jepsen.util 2429 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,141] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:28,142] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,142] jepsen worker 83 - jepsen.util 2153 :invoke :write [27 37] INFO [2022-03-13 17:14:28,143] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,144] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,144] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:28,146] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,146] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 32] WARN [2022-03-13 17:14:28,163] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10342792,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10342792,"total_server_duration_us":24325,"encode_duration_us":17386,"operation_id":"0xa6a6","timeout_ms":10000,"last_server_duration_us":24325,"total_duration_us":10009846},{"operation_name":"upsert","last_dispatch_duration_us":8563609,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9010674,"total_server_duration_us":162752,"encode_duration_us":22460,"operation_id":"0xa64d","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10007179},{"operation_name":"upsert","last_dispatch_duration_us":131338,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":366783,"total_server_duration_us":101798,"encode_duration_us":17616,"operation_id":"0xa652","timeout_ms":10000,"last_server_duration_us":34512,"total_duration_us":10003724},{"operation_name":"upsert","last_dispatch_duration_us":146591,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9778737,"total_server_duration_us":50115,"encode_duration_us":19053,"operation_id":"0xa70c","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10002663},{"operation_name":"upsert","last_dispatch_duration_us":7620816,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":8697765,"total_server_duration_us":776112,"encode_duration_us":30656,"operation_id":"0xa6e0","timeout_ms":10000,"last_server_duration_us":599221,"total_duration_us":9801512},{"operation_name":"upsert","last_dispatch_duration_us":9134043,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9134043,"total_server_duration_us":9054355,"encode_duration_us":19809,"operation_id":"0xa623","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9138923},{"operation_name":"upsert","last_dispatch_duration_us":7168879,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7848956,"total_server_duration_us":2775788,"encode_duration_us":19171,"operation_id":"0xa6df","timeout_ms":10000,"last_server_duration_us":2639139,"total_duration_us":8172277},{"operation_name":"upsert","last_dispatch_duration_us":7612423,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7612423,"total_server_duration_us":805388,"encode_duration_us":32194,"operation_id":"0xa876","timeout_ms":10000,"last_server_duration_us":805388,"total_duration_us":7617082},{"operation_name":"upsert","last_dispatch_duration_us":7399532,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7399532,"total_server_duration_us":808427,"encode_duration_us":20639,"operation_id":"0xa8e1","timeout_ms":10000,"last_server_duration_us":808427,"total_duration_us":7441773},{"operation_name":"upsert","last_dispatch_duration_us":7014830,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7014830,"total_server_duration_us":387308,"encode_duration_us":43020,"operation_id":"0xa805","timeout_ms":10000,"last_server_duration_us":387308,"total_duration_us":7018948}],"total_count":25}} WARN [2022-03-13 17:14:28,163] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":50043,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":316750,"total_server_duration_us":147157,"encode_duration_us":17827,"operation_id":"0xa85c","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10010830},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":16797,"operation_id":"0xa953","timeout_ms":10000,"total_duration_us":10009859},{"operation_name":"upsert","last_dispatch_duration_us":8991548,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9558506,"total_server_duration_us":310859,"encode_duration_us":22016,"operation_id":"0xa61c","timeout_ms":10000,"last_server_duration_us":23983,"total_duration_us":10008585},{"operation_name":"upsert","last_dispatch_duration_us":145070,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":670048,"total_server_duration_us":364651,"encode_duration_us":19434,"operation_id":"0xa6a7","timeout_ms":10000,"last_server_duration_us":58376,"total_duration_us":10006361},{"operation_name":"upsert","last_dispatch_duration_us":8501183,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8924112,"total_server_duration_us":203425,"encode_duration_us":17405,"operation_id":"0xa85b","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10004773},{"operation_name":"upsert","last_dispatch_duration_us":6149042,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6149042,"total_server_duration_us":383701,"encode_duration_us":18389,"operation_id":"0xa7ff","timeout_ms":10000,"last_server_duration_us":383701,"total_duration_us":10002530},{"operation_name":"upsert","last_dispatch_duration_us":9114988,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9114988,"total_server_duration_us":9088372,"encode_duration_us":19589,"operation_id":"0xa719","timeout_ms":10000,"last_server_duration_us":9088372,"total_duration_us":9118529},{"operation_name":"upsert","last_dispatch_duration_us":9059377,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9059377,"total_server_duration_us":9046921,"encode_duration_us":17433,"operation_id":"0xa858","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9063388},{"operation_name":"upsert","last_dispatch_duration_us":8546889,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":8546889,"total_server_duration_us":160495,"encode_duration_us":19715,"operation_id":"0xa950","timeout_ms":10000,"last_server_duration_us":160495,"total_duration_us":8862573},{"operation_name":"upsert","last_dispatch_duration_us":8617456,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":8617456,"total_server_duration_us":759278,"encode_duration_us":26391,"operation_id":"0xa919","timeout_ms":10000,"last_server_duration_us":759278,"total_duration_us":8639823}],"total_count":29}} WARN [2022-03-13 17:14:28,163] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17997431,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19175418,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9192730,"encode_duration_us":21706,"operation_id":"0xa163","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005903},{"operation_name":"upsert","last_dispatch_duration_us":9104276,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12799643,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9128830,"encode_duration_us":18160,"operation_id":"0xa401","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10006911},{"operation_name":"upsert","last_dispatch_duration_us":17961655,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19777371,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9284036,"encode_duration_us":17528,"operation_id":"0xa162","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10006769}],"total_count":3}} INFO [2022-03-13 17:14:28,184] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,184] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,184] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,185] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:28,185] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 47] INFO [2022-03-13 17:14:28,185] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:28,191] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,192] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,193] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,194] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,196] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,196] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,201] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,201] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,204] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,205] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,207] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,211] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,246] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,246] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,247] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:28,247] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:14:28,263] jepsen worker 33 - jepsen.util 663 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,264] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:28,270] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,270] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,270] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,271] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:28,272] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:14:28,272] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:28,275] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,275] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,276] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,276] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,278] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,279] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,279] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:28,279] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:14:28,299] jepsen worker 44 - jepsen.util 584 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,299] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:28,303] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,303] jepsen worker 43 - jepsen.util 583 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,304] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:28,304] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:28,304] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:28,305] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:28,359] jepsen worker 3 - jepsen.util 1893 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,359] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,359] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,360] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:14:28,360] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:14:28,360] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:28,383] jepsen worker 89 - jepsen.util 2429 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,383] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:28,388] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,388] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:28,398] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,399] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 25] INFO [2022-03-13 17:14:28,401] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,402] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,404] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,405] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,456] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,456] jepsen worker 24 - jepsen.util 564 :invoke :write [8 8] INFO [2022-03-13 17:14:28,459] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,460] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:28,472] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,472] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,472] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,473] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,473] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:28,473] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:28,474] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:28,475] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 32] INFO [2022-03-13 17:14:28,500] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,500] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,500] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,501] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:28,502] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:28,503] jepsen worker 13 - jepsen.util 643 :invoke :write [4 34] INFO [2022-03-13 17:14:28,517] jepsen worker 37 - jepsen.util 2017 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,518] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 7] INFO [2022-03-13 17:14:28,615] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,615] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 40] INFO [2022-03-13 17:14:28,637] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,638] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:28,651] jepsen worker 89 - jepsen.util 2429 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,652] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:28,659] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,659] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 44] INFO [2022-03-13 17:14:28,683] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,683] jepsen worker 14 - jepsen.util 374 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,684] jepsen worker 21 - jepsen.util 831 :invoke :write [7 29] INFO [2022-03-13 17:14:28,685] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:28,712] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,712] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:28,712] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,713] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,713] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:14:28,713] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:14:28,776] jepsen worker 3 - jepsen.util 1893 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,776] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:28,784] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,784] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:14:28,815] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,815] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,817] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,818] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:14:28,870] jepsen worker 88 - jepsen.util 2068 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,871] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:14:28,872] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,872] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:28,874] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,875] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:14:28,888] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,888] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,888] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,888] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:28,889] jepsen worker 33 - jepsen.util 663 :invoke :write [11 8] INFO [2022-03-13 17:14:28,889] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:28,890] jepsen worker 89 - jepsen.util 2429 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,891] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 16] INFO [2022-03-13 17:14:28,897] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,898] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,903] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,903] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,908] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,909] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,913] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,913] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,918] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,919] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,920] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,920] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,921] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,922] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,934] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,935] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:14:28,943] jepsen worker 37 - jepsen.util 2017 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,944] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:28,945] jepsen worker 80 - jepsen.util 710 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,945] jepsen worker 24 - jepsen.util 564 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,946] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:28,946] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,946] jepsen worker 24 - jepsen.util 564 :invoke :write [8 14] INFO [2022-03-13 17:14:28,946] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,946] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:14:28,955] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:14:28,963] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,963] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,963] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,964] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:28,965] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:28,968] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:14:28,979] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,979] jepsen worker 51 - jepsen.util 591 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,979] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,979] jepsen worker 33 - jepsen.util 663 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,979] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:14:28,980] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:28,981] jepsen worker 6 - jepsen.util 636 :invoke :write [2 24] INFO [2022-03-13 17:14:28,982] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:28,983] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:28,985] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:28,985] jepsen worker nemesis - couchbase.util Rebalance check count: 378 INFO [2022-03-13 17:14:28,985] jepsen worker nemesis - couchbase.util Rebalance stuck count: 278 INFO [2022-03-13 17:14:28,988] jepsen worker 51 - jepsen.util 591 :invoke :write [17 3] INFO [2022-03-13 17:14:29,026] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,027] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,034] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,034] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,036] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,037] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,038] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,038] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,038] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,038] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,039] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:14:29,039] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,041] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,042] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,043] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,044] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,048] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,049] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,049] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,049] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:29,049] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,050] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 17] INFO [2022-03-13 17:14:29,057] jepsen worker 16 - jepsen.util 2536 :info :write [5 17] :RequestTimeoutException INFO [2022-03-13 17:14:29,058] jepsen worker 16 - jepsen.util 2626 :invoke :write [5 38] INFO [2022-03-13 17:14:29,074] jepsen worker 24 - jepsen.util 564 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,075] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:14:29,088] jepsen worker 89 - jepsen.util 2429 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,088] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,089] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,089] jepsen worker 21 - jepsen.util 831 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,089] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:14:29,089] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,090] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:29,090] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,090] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:29,091] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:14:29,106] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,107] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:14:29,120] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,120] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,120] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,121] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,121] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:14:29,122] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:14:29,156] jepsen worker 62 - jepsen.util 512 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,157] jepsen worker 62 - jepsen.util 512 :invoke :write [20 33] INFO [2022-03-13 17:14:29,165] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,165] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,166] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,167] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,170] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,171] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:29,175] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,175] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,176] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,176] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:14:29,177] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,178] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,180] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,180] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,180] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,181] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:29,181] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:29,182] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:29,187] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,188] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:14:29,189] jepsen worker 3 - jepsen.util 1893 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,189] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,190] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,191] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:14:29,198] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,198] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,207] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,207] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:29,252] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,252] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:14:29,292] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,292] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,292] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,292] jepsen worker 61 - jepsen.util 691 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,292] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,293] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,295] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:29,296] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:29,296] jepsen worker 60 - jepsen.util 600 :invoke :write [20 35] INFO [2022-03-13 17:14:29,299] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:29,300] jepsen worker 69 - jepsen.util 699 :invoke :write [23 13] INFO [2022-03-13 17:14:29,300] jepsen worker 12 - jepsen.util 732 :invoke :write [4 31] INFO [2022-03-13 17:14:29,307] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,311] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,312] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:14:29,313] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 27] INFO [2022-03-13 17:14:29,317] jepsen worker 9 - jepsen.util 2349 :ok :read [3 25] INFO [2022-03-13 17:14:29,318] jepsen worker 9 - jepsen.util 2349 :invoke :read [3 nil] INFO [2022-03-13 17:14:29,328] jepsen worker 63 - jepsen.util 2763 :ok :read [21 :nil] INFO [2022-03-13 17:14:29,329] jepsen worker 63 - jepsen.util 2763 :invoke :write [21 47] INFO [2022-03-13 17:14:29,349] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,349] jepsen worker 64 - jepsen.util 2674 :ok :read [21 :nil] INFO [2022-03-13 17:14:29,349] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:14:29,350] jepsen worker 64 - jepsen.util 2674 :invoke :write [21 48] INFO [2022-03-13 17:14:29,351] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,351] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,352] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,352] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,359] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,359] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,359] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:14:29,360] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:29,361] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,362] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,362] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,363] jepsen worker 14 - jepsen.util 374 :invoke :write [4 8] INFO [2022-03-13 17:14:29,363] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:14:29,369] jepsen worker 21 - jepsen.util 831 :invoke :write [7 21] INFO [2022-03-13 17:14:29,387] jepsen worker 79 - jepsen.util 1069 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,387] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:29,395] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,395] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,412] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,413] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:29,443] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:29,443] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:29,446] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,447] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:14:29,448] jepsen worker 10 - jepsen.util 2890 :ok :read [3 25] INFO [2022-03-13 17:14:29,448] jepsen worker 10 - jepsen.util 2890 :invoke :read [3 nil] INFO [2022-03-13 17:14:29,483] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,483] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:29,492] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,493] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:14:29,518] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,518] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,518] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,519] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,519] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:14:29,519] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,522] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,522] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:14:29,528] jepsen worker 23 - jepsen.util 563 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,528] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:29,528] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:29,529] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:29,534] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,534] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,534] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:14:29,534] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,557] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,558] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:14:29,592] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:29,592] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:29,594] jepsen worker 14 - jepsen.util 374 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,595] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:29,600] jepsen worker 89 - jepsen.util 2429 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,602] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,602] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,603] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 36] INFO [2022-03-13 17:14:29,604] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,605] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,606] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,607] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,609] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,609] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,612] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,612] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,613] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 1] INFO [2022-03-13 17:14:29,613] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,614] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,615] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,615] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:14:29,625] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:14:29,626] jepsen worker 10 - jepsen.util 2890 :ok :read [3 25] INFO [2022-03-13 17:14:29,626] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,627] jepsen worker 10 - jepsen.util 2890 :invoke :read [3 nil] INFO [2022-03-13 17:14:29,628] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,693] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,693] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,694] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,695] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,695] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:29,696] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:29,702] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,702] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,703] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,703] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,711] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,711] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,712] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:14:29,712] jepsen worker 24 - jepsen.util 564 :invoke :write [8 4] INFO [2022-03-13 17:14:29,722] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:29,723] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:29,751] jepsen worker 10 - jepsen.util 2890 :ok :read [3 25] INFO [2022-03-13 17:14:29,752] jepsen worker 10 - jepsen.util 2890 :invoke :read [3 nil] INFO [2022-03-13 17:14:29,759] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,760] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:29,779] jepsen worker 52 - jepsen.util 592 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,780] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:29,787] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,787] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:14:29,808] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,808] jepsen worker 37 - jepsen.util 2017 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,809] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,809] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 23] INFO [2022-03-13 17:14:29,818] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,818] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:29,822] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,822] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:14:29,858] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:29,858] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:29,870] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,871] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:14:29,882] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,882] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,882] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,882] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,883] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:14:29,883] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,887] jepsen worker 10 - jepsen.util 2890 :ok :read [3 25] INFO [2022-03-13 17:14:29,887] jepsen worker 10 - jepsen.util 2890 :invoke :write [3 42] INFO [2022-03-13 17:14:29,901] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,901] jepsen worker 89 - jepsen.util 2429 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,901] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,902] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,908] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,909] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,910] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,911] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,912] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,913] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 5] INFO [2022-03-13 17:14:29,918] jepsen worker 21 - jepsen.util 831 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,918] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:29,928] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,929] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:29,951] jepsen worker 51 - jepsen.util 591 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,952] jepsen worker 26 - jepsen.util 476 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,952] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,952] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:29,952] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:29,953] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:29,953] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,953] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,953] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:29,954] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:29,968] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,968] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,968] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,969] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:29,969] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:14:29,970] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:29,983] jepsen worker 37 - jepsen.util 2017 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,983] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:29,984] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 14] INFO [2022-03-13 17:14:29,985] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:14:29,997] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:29,997] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:30,004] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:30,004] jepsen worker nemesis - couchbase.util Rebalance check count: 379 INFO [2022-03-13 17:14:30,004] jepsen worker nemesis - couchbase.util Rebalance stuck count: 279 INFO [2022-03-13 17:14:30,044] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,044] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,044] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,044] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,045] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:30,046] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,046] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 43] INFO [2022-03-13 17:14:30,047] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,056] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,057] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,062] jepsen worker 89 - jepsen.util 2429 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,063] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 3] INFO [2022-03-13 17:14:30,064] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,064] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,067] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,068] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:30,086] jepsen worker 47 - jepsen.util 2657 :info :write [15 24] :RequestTimeoutException INFO [2022-03-13 17:14:30,087] jepsen worker 47 - jepsen.util 2747 :invoke :write [15 35] INFO [2022-03-13 17:14:30,116] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,117] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:30,124] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,125] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:30,142] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,142] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,144] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,145] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,154] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,155] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,162] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,162] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:30,172] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,172] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,173] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,173] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,173] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,174] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,174] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:14:30,175] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:30,185] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,185] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,185] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,186] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:30,187] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:30,187] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,187] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:14:30,188] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:14:30,188] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,189] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,189] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:30,190] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:30,191] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,191] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,198] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,199] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:14:30,200] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:14:30,200] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:30,202] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,203] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:14:30,208] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:30,209] jepsen worker 20 - jepsen.util 3170 :invoke :read [6 nil] INFO [2022-03-13 17:14:30,225] jepsen worker 8 - jepsen.util 728 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,225] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:30,225] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:30,225] jepsen worker 60 - jepsen.util 600 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,225] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,225] jepsen worker 8 - jepsen.util 728 :invoke :write [2 45] INFO [2022-03-13 17:14:30,226] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,226] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:30,226] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:30,227] jepsen worker 43 - jepsen.util 583 :invoke :write [14 0] INFO [2022-03-13 17:14:30,228] jepsen worker 53 - jepsen.util 503 :invoke :write [17 28] INFO [2022-03-13 17:14:30,229] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:30,237] jepsen worker 24 - jepsen.util 564 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,237] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:30,264] jepsen worker 37 - jepsen.util 2017 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,264] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,264] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,265] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,266] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,266] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:14:30,269] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,269] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:14:30,278] jepsen worker 3 - jepsen.util 1893 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,279] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,279] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,279] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 2] INFO [2022-03-13 17:14:30,280] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,285] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 8] INFO [2022-03-13 17:14:30,296] jepsen worker 18 - jepsen.util 2538 :info :write [6 13] :RequestTimeoutException INFO [2022-03-13 17:14:30,297] jepsen worker 18 - jepsen.util 2628 :invoke :read [6 nil] INFO [2022-03-13 17:14:30,313] jepsen worker 89 - jepsen.util 2429 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,313] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,314] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,314] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 15] INFO [2022-03-13 17:14:30,321] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,321] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:14:30,332] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,332] jepsen worker 78 - jepsen.util 528 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,332] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,333] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:30,333] jepsen worker 78 - jepsen.util 528 :invoke :write [26 0] INFO [2022-03-13 17:14:30,334] jepsen worker 34 - jepsen.util 754 :invoke :write [11 16] INFO [2022-03-13 17:14:30,365] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,366] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:14:30,371] jepsen worker 6 - jepsen.util 636 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,372] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:30,379] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,379] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,380] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:14:30,380] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 2] INFO [2022-03-13 17:14:30,382] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,382] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,383] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:14:30,383] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,392] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,392] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,392] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,392] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,395] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:14:30,396] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:30,397] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:14:30,398] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,398] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:30,399] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:30,399] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,400] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,400] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:30,402] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:14:30,402] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,403] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:14:30,410] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,411] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:30,450] jepsen worker 3 - jepsen.util 1893 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,450] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,455] jepsen worker 20 - jepsen.util 3170 :ok :read [6 16] INFO [2022-03-13 17:14:30,456] jepsen worker 20 - jepsen.util 3170 :invoke :write [6 14] INFO [2022-03-13 17:14:30,469] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,469] jepsen worker 18 - jepsen.util 2628 :ok :read [6 16] INFO [2022-03-13 17:14:30,469] jepsen worker 18 - jepsen.util 2628 :invoke :read [6 nil] INFO [2022-03-13 17:14:30,470] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,471] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,472] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,477] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,477] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:14:30,481] jepsen worker 89 - jepsen.util 2429 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,482] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 42] INFO [2022-03-13 17:14:30,532] jepsen worker 69 - jepsen.util 699 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,533] jepsen worker 69 - jepsen.util 699 :invoke :write [23 3] INFO [2022-03-13 17:14:30,533] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,535] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:14:30,536] jepsen worker 18 - jepsen.util 2628 :ok :read [6 16] INFO [2022-03-13 17:14:30,536] jepsen worker 18 - jepsen.util 2628 :invoke :read [6 nil] INFO [2022-03-13 17:14:30,553] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,554] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:14:30,555] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,555] jepsen worker 88 - jepsen.util 2068 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,556] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:14:30,557] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,562] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,564] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:30,588] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,588] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:30,591] jepsen worker 18 - jepsen.util 2628 :ok :read [6 16] INFO [2022-03-13 17:14:30,592] jepsen worker 18 - jepsen.util 2628 :invoke :read [6 nil] INFO [2022-03-13 17:14:30,626] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,626] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,627] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 0] INFO [2022-03-13 17:14:30,627] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:30,628] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,629] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:14:30,631] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,632] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,633] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,634] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,638] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,639] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,644] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,644] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,646] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,647] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,649] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,653] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,690] jepsen worker 18 - jepsen.util 2628 :ok :read [6 16] INFO [2022-03-13 17:14:30,690] jepsen worker 18 - jepsen.util 2628 :invoke :write [6 32] INFO [2022-03-13 17:14:30,699] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,700] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,700] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,700] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:30,705] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,705] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,706] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:14:30,706] jepsen worker 37 - jepsen.util 2017 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,707] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 3] INFO [2022-03-13 17:14:30,708] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:30,720] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,720] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,721] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:14:30,722] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:14:30,747] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,747] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:14:30,776] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,777] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:30,788] jepsen worker 3 - jepsen.util 1893 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,789] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,795] jepsen worker 89 - jepsen.util 2429 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,796] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:30,797] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,798] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:30,805] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,805] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:30,807] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,808] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:14:30,839] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,839] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:30,861] jepsen worker 15 - jepsen.util 2895 :info :write [5 12] :RequestTimeoutException INFO [2022-03-13 17:14:30,861] jepsen worker 15 - jepsen.util 2985 :invoke :read [5 nil] INFO [2022-03-13 17:14:30,888] jepsen worker 50 - jepsen.util 2930 :info :write [16 37] :RequestTimeoutException INFO [2022-03-13 17:14:30,888] jepsen worker 50 - jepsen.util 3020 :invoke :read [16 nil] INFO [2022-03-13 17:14:30,899] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,899] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:14:30,901] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,901] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:30,905] jepsen worker 73 - jepsen.util 3043 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,906] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,907] jepsen worker 73 - jepsen.util 3043 :invoke :read [24 nil] INFO [2022-03-13 17:14:30,908] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:14:30,912] jepsen worker 31 - jepsen.util 3001 :ok :read [10 :nil] INFO [2022-03-13 17:14:30,913] jepsen worker 73 - jepsen.util 3043 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,914] jepsen worker 73 - jepsen.util 3043 :invoke :read [24 nil] INFO [2022-03-13 17:14:30,915] jepsen worker 31 - jepsen.util 3001 :invoke :read [10 nil] INFO [2022-03-13 17:14:30,916] jepsen worker 73 - jepsen.util 3043 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,916] jepsen worker 31 - jepsen.util 3001 :ok :read [10 :nil] INFO [2022-03-13 17:14:30,918] jepsen worker 31 - jepsen.util 3001 :invoke :read [10 nil] INFO [2022-03-13 17:14:30,918] jepsen worker 45 - jepsen.util 3015 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,918] jepsen worker 54 - jepsen.util 2394 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,919] jepsen worker 73 - jepsen.util 3043 :invoke :write [24 14] INFO [2022-03-13 17:14:30,922] jepsen worker 31 - jepsen.util 3001 :ok :read [10 :nil] INFO [2022-03-13 17:14:30,923] jepsen worker 31 - jepsen.util 3001 :invoke :write [10 41] INFO [2022-03-13 17:14:30,923] jepsen worker 29 - jepsen.util 2999 :ok :read [9 26] INFO [2022-03-13 17:14:30,926] jepsen worker 54 - jepsen.util 2394 :invoke :write [18 19] INFO [2022-03-13 17:14:30,926] jepsen worker 29 - jepsen.util 2999 :invoke :read [9 nil] INFO [2022-03-13 17:14:30,928] jepsen worker 28 - jepsen.util 2908 :ok :read [9 26] INFO [2022-03-13 17:14:30,929] jepsen worker 28 - jepsen.util 2908 :invoke :read [9 nil] INFO [2022-03-13 17:14:30,935] jepsen worker 45 - jepsen.util 3015 :invoke :read [15 nil] INFO [2022-03-13 17:14:30,950] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,950] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,951] jepsen worker 33 - jepsen.util 663 :invoke :write [11 14] INFO [2022-03-13 17:14:30,951] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:30,967] jepsen worker 62 - jepsen.util 512 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,967] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,967] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,967] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,968] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:30,968] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:30,968] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,968] jepsen worker 7 - jepsen.util 637 :invoke :write [2 42] INFO [2022-03-13 17:14:30,969] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:30,969] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:30,970] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:30,971] jepsen worker 29 - jepsen.util 2999 :ok :read [9 26] INFO [2022-03-13 17:14:30,972] jepsen worker 29 - jepsen.util 2999 :invoke :write [9 47] INFO [2022-03-13 17:14:30,978] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:14:30,984] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:30,988] jepsen worker 1 - jepsen.util 2521 :info :write [0 13] :SyncWriteAmbiguous INFO [2022-03-13 17:14:30,989] jepsen worker 1 - jepsen.util 2611 :invoke :write [0 11] INFO [2022-03-13 17:14:30,989] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:31,006] jepsen worker 37 - jepsen.util 2017 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,006] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,007] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:31,007] jepsen worker 45 - jepsen.util 3015 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,007] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 6] INFO [2022-03-13 17:14:31,008] jepsen worker 45 - jepsen.util 3015 :invoke :write [15 5] INFO [2022-03-13 17:14:31,012] jepsen worker 15 - jepsen.util 2985 :ok :read [5 43] INFO [2022-03-13 17:14:31,013] jepsen worker 15 - jepsen.util 2985 :invoke :write [5 33] INFO [2022-03-13 17:14:31,018] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:31,018] jepsen worker nemesis - couchbase.util Rebalance check count: 380 INFO [2022-03-13 17:14:31,018] jepsen worker nemesis - couchbase.util Rebalance stuck count: 280 INFO [2022-03-13 17:14:31,033] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,033] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,034] jepsen worker 71 - jepsen.util 521 :invoke :write [23 24] INFO [2022-03-13 17:14:31,034] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,035] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:31,035] jepsen worker 28 - jepsen.util 2908 :ok :read [9 26] INFO [2022-03-13 17:14:31,035] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:31,036] jepsen worker 28 - jepsen.util 2908 :invoke :write [9 42] INFO [2022-03-13 17:14:31,043] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,043] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,043] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:31,043] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,044] jepsen worker 21 - jepsen.util 831 :invoke :write [7 44] INFO [2022-03-13 17:14:31,053] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:31,064] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,064] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,065] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:31,065] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:31,072] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,072] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 43] INFO [2022-03-13 17:14:31,074] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,075] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:31,150] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,150] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,152] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,152] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,152] jepsen worker 12 - jepsen.util 732 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,153] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,153] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,153] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:31,153] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:31,154] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,154] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,154] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:31,155] jepsen worker 60 - jepsen.util 600 :invoke :write [20 25] INFO [2022-03-13 17:14:31,156] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:31,157] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,157] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:31,158] jepsen worker 44 - jepsen.util 584 :invoke :write [14 30] INFO [2022-03-13 17:14:31,160] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:14:31,160] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:31,161] jepsen worker 61 - jepsen.util 691 :invoke :write [20 37] INFO [2022-03-13 17:14:31,162] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,162] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,163] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:14:31,163] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:31,167] jepsen worker 33 - jepsen.util 663 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,168] jepsen worker 33 - jepsen.util 663 :invoke :write [11 10] INFO [2022-03-13 17:14:31,172] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,172] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,172] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,172] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,173] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:31,173] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:31,174] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 9] INFO [2022-03-13 17:14:31,174] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,175] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:31,175] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,176] jepsen worker 37 - jepsen.util 2017 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,176] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,176] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 40] INFO [2022-03-13 17:14:31,176] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,177] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:14:31,178] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:14:31,178] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:14:31,179] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 32] INFO [2022-03-13 17:14:31,198] jepsen worker 71 - jepsen.util 521 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,198] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,200] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:31,201] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:14:31,236] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,236] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:14:31,249] jepsen worker 89 - jepsen.util 2429 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,250] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 47] INFO [2022-03-13 17:14:31,250] jepsen worker 46 - jepsen.util 2746 :info :write [15 49] :RequestTimeoutException INFO [2022-03-13 17:14:31,251] jepsen worker 46 - jepsen.util 2836 :invoke :write [15 31] INFO [2022-03-13 17:14:31,276] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,277] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:31,284] jepsen worker 11 - jepsen.util 2711 :info :write [3 15] :RequestTimeoutException INFO [2022-03-13 17:14:31,284] jepsen worker 11 - jepsen.util 2801 :invoke :write [3 20] INFO [2022-03-13 17:14:31,330] jepsen worker 3 - jepsen.util 1893 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,331] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 42] INFO [2022-03-13 17:14:31,350] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,350] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,351] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:14:31,351] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,353] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,354] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,356] jepsen worker 68 - jepsen.util 2948 :info :write [22 25] :RequestTimeoutException INFO [2022-03-13 17:14:31,356] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,357] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:31,358] jepsen worker 68 - jepsen.util 3038 :invoke :write [22 21] INFO [2022-03-13 17:14:31,363] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,364] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:14:31,404] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,404] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,405] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:31,406] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:31,411] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,411] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:14:31,453] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,454] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:14:31,474] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,475] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:14:31,484] jepsen worker 37 - jepsen.util 2017 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,485] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 23] INFO [2022-03-13 17:14:31,534] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,535] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:31,561] jepsen worker 86 - jepsen.util 2606 :info :write [28 32] :RequestTimeoutException INFO [2022-03-13 17:14:31,561] jepsen worker 86 - jepsen.util 2696 :invoke :read [28 nil] INFO [2022-03-13 17:14:31,566] jepsen worker 33 - jepsen.util 663 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,566] jepsen worker 89 - jepsen.util 2429 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,567] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:31,567] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:31,570] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,571] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 17] INFO [2022-03-13 17:14:31,591] jepsen worker 88 - jepsen.util 2068 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,592] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:31,598] jepsen worker 41 - jepsen.util 2741 :info :write [13 12] :RequestTimeoutException INFO [2022-03-13 17:14:31,598] jepsen worker 41 - jepsen.util 2831 :invoke :read [13 nil] INFO [2022-03-13 17:14:31,603] jepsen worker 86 - jepsen.util 2696 :ok :read [28 44] INFO [2022-03-13 17:14:31,603] jepsen worker 86 - jepsen.util 2696 :invoke :read [28 nil] INFO [2022-03-13 17:14:31,618] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,619] jepsen worker 21 - jepsen.util 831 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,620] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:31,627] jepsen worker 21 - jepsen.util 831 :invoke :write [7 40] INFO [2022-03-13 17:14:31,631] jepsen worker 86 - jepsen.util 2696 :ok :read [28 44] INFO [2022-03-13 17:14:31,632] jepsen worker 86 - jepsen.util 2696 :invoke :read [28 nil] INFO [2022-03-13 17:14:31,642] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,643] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:14:31,694] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,695] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:31,696] jepsen worker 3 - jepsen.util 1893 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,697] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 14] INFO [2022-03-13 17:14:31,720] jepsen worker 49 - jepsen.util 2479 :info :write [16 46] :RequestTimeoutException INFO [2022-03-13 17:14:31,721] jepsen worker 49 - jepsen.util 2569 :invoke :write [16 33] INFO [2022-03-13 17:14:31,741] jepsen worker 37 - jepsen.util 2017 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,742] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,762] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,763] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:14:31,771] jepsen worker 53 - jepsen.util 503 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,772] jepsen worker 69 - jepsen.util 699 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,772] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,772] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,772] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,772] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,772] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,773] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:31,773] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:14:31,774] jepsen worker 69 - jepsen.util 699 :invoke :write [23 32] INFO [2022-03-13 17:14:31,775] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,776] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,776] jepsen worker 66 - jepsen.util 2676 :info :write [22 40] :RequestTimeoutException INFO [2022-03-13 17:14:31,779] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:31,780] jepsen worker 51 - jepsen.util 591 :invoke :write [17 43] INFO [2022-03-13 17:14:31,780] jepsen worker 66 - jepsen.util 2766 :invoke :write [22 38] INFO [2022-03-13 17:14:31,783] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,783] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:31,783] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:31,784] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:31,784] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,784] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:14:31,790] jepsen worker 89 - jepsen.util 2429 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,791] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:14:31,797] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,798] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,800] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,801] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,802] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,802] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,803] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,804] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:31,805] jepsen worker 13 - jepsen.util 643 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,806] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,806] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,806] jepsen worker 13 - jepsen.util 643 :invoke :write [4 31] INFO [2022-03-13 17:14:31,807] jepsen worker 70 - jepsen.util 700 :invoke :write [23 16] INFO [2022-03-13 17:14:31,808] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:31,808] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,809] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:31,839] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,839] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,840] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:14:31,840] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:31,887] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,887] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,887] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,888] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:31,889] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,889] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,889] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,889] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,890] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:31,890] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:31,892] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,892] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,892] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 49] INFO [2022-03-13 17:14:31,893] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:31,915] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,915] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:31,923] jepsen worker 81 - jepsen.util 3231 :info :write [27 2] :RequestTimeoutException INFO [2022-03-13 17:14:31,924] jepsen worker 3 - jepsen.util 1893 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,924] jepsen worker 81 - jepsen.util 3321 :invoke :read [27 nil] INFO [2022-03-13 17:14:31,925] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:31,952] jepsen worker 89 - jepsen.util 2429 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,953] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 31] INFO [2022-03-13 17:14:31,983] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,983] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,983] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:14:31,984] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:14:31,992] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,992] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,992] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,993] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:31,993] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:31,994] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:31,995] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:31,995] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:14:32,002] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,002] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,003] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,003] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:32,004] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:14:32,007] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:14:32,024] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:32,024] jepsen worker nemesis - couchbase.util Rebalance check count: 381 INFO [2022-03-13 17:14:32,024] jepsen worker nemesis - couchbase.util Rebalance stuck count: 281 INFO [2022-03-13 17:14:32,043] jepsen worker 37 - jepsen.util 2017 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,044] jepsen worker 37 - jepsen.util 2017 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,063] jepsen worker 21 - jepsen.util 831 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,063] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:32,065] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,065] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,065] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:32,065] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:32,072] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,072] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,074] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,075] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,079] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,079] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,081] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,081] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,082] jepsen worker 3 - jepsen.util 1893 :invoke :read [1 nil] INFO [2022-03-13 17:14:32,082] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:14:32,083] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,083] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,084] jepsen worker 33 - jepsen.util 663 :invoke :write [11 15] INFO [2022-03-13 17:14:32,085] jepsen worker 71 - jepsen.util 521 :invoke :write [23 47] INFO [2022-03-13 17:14:32,086] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,087] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:14:32,164] jepsen worker 0 - jepsen.util 2610 :info :write [0 33] :RequestTimeoutException INFO [2022-03-13 17:14:32,164] jepsen worker 0 - jepsen.util 2700 :invoke :read [0 nil] INFO [2022-03-13 17:14:32,172] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,173] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,174] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,174] jepsen worker 37 - jepsen.util 2017 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,175] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 24] INFO [2022-03-13 17:14:32,179] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:14:32,180] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,183] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:14:32,200] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,203] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 8] INFO [2022-03-13 17:14:32,204] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,205] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:32,210] jepsen worker 60 - jepsen.util 600 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,210] jepsen worker 8 - jepsen.util 728 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,211] jepsen worker 60 - jepsen.util 600 :invoke :write [20 19] INFO [2022-03-13 17:14:32,211] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:32,217] jepsen worker 48 - jepsen.util 2748 :info :write [16 23] :RequestTimeoutException INFO [2022-03-13 17:14:32,218] jepsen worker 48 - jepsen.util 2838 :invoke :read [16 nil] INFO [2022-03-13 17:14:32,225] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,225] jepsen worker 89 - jepsen.util 2429 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,225] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,225] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,225] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,226] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:32,227] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:32,227] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:32,227] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,228] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:32,228] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:32,228] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:32,228] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,229] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,229] jepsen worker 89 - jepsen.util 2429 :invoke :read [29 nil] INFO [2022-03-13 17:14:32,229] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,229] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:32,230] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:14:32,230] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,230] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,230] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:32,230] jepsen worker 89 - jepsen.util 2429 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,231] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:14:32,231] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 30] INFO [2022-03-13 17:14:32,236] jepsen worker 59 - jepsen.util 2759 :info :write [19 25] :RequestTimeoutException INFO [2022-03-13 17:14:32,237] jepsen worker 59 - jepsen.util 2849 :invoke :read [19 nil] INFO [2022-03-13 17:14:32,245] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,245] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:32,245] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,246] jepsen worker 21 - jepsen.util 831 :invoke :write [7 36] INFO [2022-03-13 17:14:32,250] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,251] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:32,260] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,260] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,268] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,269] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:14:32,289] jepsen worker 3 - jepsen.util 1893 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,289] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,289] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:14:32,290] jepsen worker 3 - jepsen.util 1893 :invoke :write [1 29] INFO [2022-03-13 17:14:32,294] jepsen worker 37 - jepsen.util 2017 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,295] jepsen worker 37 - jepsen.util 2017 :invoke :write [12 29] INFO [2022-03-13 17:14:32,351] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,352] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,353] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,354] jepsen worker 82 - jepsen.util 2782 :info :write [27 38] :RequestTimeoutException INFO [2022-03-13 17:14:32,354] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:14:32,355] jepsen worker 82 - jepsen.util 2872 :invoke :write [27 1] INFO [2022-03-13 17:14:32,356] jepsen worker 2 - jepsen.util 3152 :info :write [0 10] :RequestTimeoutException INFO [2022-03-13 17:14:32,357] jepsen worker 2 - jepsen.util 3242 :invoke :write [0 18] INFO [2022-03-13 17:14:32,370] jepsen worker 39 - jepsen.util 2739 :info :write [13 24] :RequestTimeoutException INFO [2022-03-13 17:14:32,371] jepsen worker 39 - jepsen.util 2829 :invoke :write [13 15] INFO [2022-03-13 17:14:32,384] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,385] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:32,400] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,401] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:32,417] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,417] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:14:32,428] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,428] jepsen worker 43 - jepsen.util 583 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,429] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:32,429] jepsen worker 43 - jepsen.util 583 :invoke :write [14 46] INFO [2022-03-13 17:14:32,430] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,430] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:32,431] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,431] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:14:32,433] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 11] INFO [2022-03-13 17:14:32,434] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,434] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,435] jepsen worker 75 - jepsen.util 435 :invoke :write [25 21] INFO [2022-03-13 17:14:32,435] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:32,442] jepsen worker 26 - jepsen.util 476 :invoke :write [8 22] INFO [2022-03-13 17:14:32,445] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,446] jepsen worker 89 - jepsen.util 2429 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,447] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,448] jepsen worker 89 - jepsen.util 2429 :invoke :write [29 36] INFO [2022-03-13 17:14:32,465] jepsen worker 88 - jepsen.util 2068 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,466] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 47] INFO [2022-03-13 17:14:32,477] jepsen worker 71 - jepsen.util 521 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,478] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:32,485] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,486] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:32,494] jepsen worker 44 - jepsen.util 584 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,495] jepsen worker 44 - jepsen.util 584 :invoke :write [14 5] INFO [2022-03-13 17:14:32,632] jepsen worker 38 - jepsen.util 2468 :ok :read [12 20] INFO [2022-03-13 17:14:32,632] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,637] jepsen worker 33 - jepsen.util 663 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,637] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,637] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:32,638] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:14:32,669] jepsen worker 75 - jepsen.util 435 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,670] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,670] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:32,670] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:32,673] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,674] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:14:32,701] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,701] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,701] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,702] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:32,702] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:32,703] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:32,753] jepsen worker 38 - jepsen.util 2468 :ok :read [12 20] INFO [2022-03-13 17:14:32,753] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:32,791] jepsen worker 5 - jepsen.util 2435 :ok :read [1 42] INFO [2022-03-13 17:14:32,791] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:32,826] jepsen worker 85 - jepsen.util 2515 :info :write [28 17] :RequestTimeoutException INFO [2022-03-13 17:14:32,827] jepsen worker 85 - jepsen.util 2605 :invoke :write [28 0] INFO [2022-03-13 17:14:32,844] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,844] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,844] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,844] jepsen worker 7 - jepsen.util 637 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,844] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,846] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:32,847] jepsen worker 62 - jepsen.util 512 :invoke :write [20 20] INFO [2022-03-13 17:14:32,847] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:32,848] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:32,848] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:32,868] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,869] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,870] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:14:32,871] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:14:32,875] jepsen worker 61 - jepsen.util 691 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,875] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,875] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,876] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:32,876] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,877] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:14:32,877] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:32,878] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:32,903] jepsen worker 4 - jepsen.util 2254 :ok :read [1 42] INFO [2022-03-13 17:14:32,904] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:32,915] jepsen worker 38 - jepsen.util 2468 :ok :read [12 20] INFO [2022-03-13 17:14:32,915] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:14:32,966] jepsen worker 88 - jepsen.util 2068 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:32,966] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:14:33,006] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,007] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,008] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,008] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:14:33,031] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:33,031] jepsen worker nemesis - couchbase.util Rebalance check count: 382 INFO [2022-03-13 17:14:33,031] jepsen worker nemesis - couchbase.util Rebalance stuck count: 282 INFO [2022-03-13 17:14:33,063] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,064] jepsen worker 34 - jepsen.util 754 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,064] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:14:33,065] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:33,069] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,069] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,069] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,069] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:14:33,069] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,070] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:14:33,071] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:33,074] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:33,075] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,075] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,076] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:14:33,076] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:33,078] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,078] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:33,079] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,080] jepsen worker 33 - jepsen.util 663 :invoke :write [11 42] INFO [2022-03-13 17:14:33,097] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,097] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,099] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,099] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:14:33,114] jepsen worker 21 - jepsen.util 831 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,114] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,114] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:14:33,115] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:33,121] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,121] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:33,124] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,125] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:33,132] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,132] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:14:33,141] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,142] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:33,165] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,166] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,168] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,169] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,170] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,171] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:14:33,188] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,189] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:14:33,192] jepsen worker 78 - jepsen.util 528 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,192] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:33,195] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,195] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,196] jepsen worker 52 - jepsen.util 592 :invoke :write [17 39] INFO [2022-03-13 17:14:33,196] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,196] jepsen worker 80 - jepsen.util 710 :invoke :write [26 12] INFO [2022-03-13 17:14:33,196] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,198] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,198] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,201] jepsen worker 69 - jepsen.util 699 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,201] jepsen worker 43 - jepsen.util 583 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,202] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:33,202] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:33,205] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,206] jepsen worker 24 - jepsen.util 564 :invoke :write [8 15] INFO [2022-03-13 17:14:33,206] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,207] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,209] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,210] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:33,239] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,240] jepsen worker 32 - jepsen.util 2822 :ok :read [10 :nil] INFO [2022-03-13 17:14:33,240] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:14:33,240] jepsen worker 32 - jepsen.util 2822 :invoke :read [10 nil] INFO [2022-03-13 17:14:33,283] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,284] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,284] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:33,288] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:33,298] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,298] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:33,299] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,309] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:33,312] jepsen worker 32 - jepsen.util 2822 :ok :read [10 :nil] INFO [2022-03-13 17:14:33,313] jepsen worker 32 - jepsen.util 2822 :invoke :read [10 nil] INFO [2022-03-13 17:14:33,315] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,315] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:14:33,350] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,351] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:14:33,370] jepsen worker 32 - jepsen.util 2822 :ok :read [10 :nil] INFO [2022-03-13 17:14:33,371] jepsen worker 32 - jepsen.util 2822 :invoke :read [10 nil] INFO [2022-03-13 17:14:33,381] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,382] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,383] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,383] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,385] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,385] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,386] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,387] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:33,403] jepsen worker 32 - jepsen.util 2822 :ok :read [10 :nil] INFO [2022-03-13 17:14:33,403] jepsen worker 32 - jepsen.util 2822 :invoke :write [10 39] INFO [2022-03-13 17:14:33,417] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,418] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:14:33,446] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,447] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:14:33,456] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,456] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:33,459] jepsen worker 70 - jepsen.util 700 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,460] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,460] jepsen worker 70 - jepsen.util 700 :invoke :write [23 17] INFO [2022-03-13 17:14:33,467] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,468] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:14:33,470] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:33,475] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,476] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,476] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:33,477] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:14:33,488] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,488] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,489] jepsen worker 23 - jepsen.util 563 :invoke :write [7 49] INFO [2022-03-13 17:14:33,490] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:33,491] jepsen worker 33 - jepsen.util 663 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,492] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:33,523] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,523] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:33,556] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,557] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,559] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,560] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,564] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,565] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,570] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,571] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:14:33,634] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,635] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,636] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,637] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:33,664] jepsen worker 24 - jepsen.util 564 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,664] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,664] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,664] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,665] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:33,665] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,665] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:14:33,666] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:33,666] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:33,667] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,668] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,669] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:14:33,697] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,697] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,699] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,699] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:14:33,748] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,749] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,750] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,751] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:14:33,760] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,760] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,760] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,760] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:33,761] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:14:33,762] jepsen worker 71 - jepsen.util 521 :invoke :write [23 24] INFO [2022-03-13 17:14:33,769] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,769] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,769] jepsen worker 13 - jepsen.util 643 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,770] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:33,771] jepsen worker 78 - jepsen.util 528 :invoke :write [26 9] INFO [2022-03-13 17:14:33,771] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:33,780] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,780] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:33,819] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,819] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,819] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:14:33,819] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,820] jepsen worker 14 - jepsen.util 374 :invoke :write [4 30] INFO [2022-03-13 17:14:33,820] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:14:33,826] jepsen worker 26 - jepsen.util 476 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,827] jepsen worker 26 - jepsen.util 476 :invoke :write [8 43] INFO [2022-03-13 17:14:33,836] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,837] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:14:33,881] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,882] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,883] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,884] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,885] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,885] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,886] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,887] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,888] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,889] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:14:33,892] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,892] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,893] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:14:33,893] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:14:33,905] jepsen worker 80 - jepsen.util 710 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,906] jepsen worker 80 - jepsen.util 710 :invoke :write [26 28] INFO [2022-03-13 17:14:33,913] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,914] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:33,917] jepsen worker 51 - jepsen.util 591 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,918] jepsen worker 51 - jepsen.util 591 :invoke :write [17 38] INFO [2022-03-13 17:14:33,922] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,923] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:14:33,953] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,954] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,956] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,957] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:14:33,977] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,977] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:14:33,988] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,989] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,991] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,991] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:33,993] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:33,994] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:14:34,016] jepsen worker 71 - jepsen.util 521 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,017] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:14:34,023] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,024] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:34,036] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:34,036] jepsen worker nemesis - couchbase.util Rebalance check count: 383 INFO [2022-03-13 17:14:34,036] jepsen worker nemesis - couchbase.util Rebalance stuck count: 283 INFO [2022-03-13 17:14:34,083] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,083] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,085] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,085] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,089] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,089] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:14:34,100] jepsen worker 67 - jepsen.util 2407 :info :write [22 3] :RequestTimeoutException INFO [2022-03-13 17:14:34,101] jepsen worker 67 - jepsen.util 2497 :invoke :write [22 48] INFO [2022-03-13 17:14:34,130] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,130] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,132] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,132] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,133] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,134] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,135] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,135] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,136] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,137] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,138] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,138] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,160] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,160] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,160] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,160] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:14:34,161] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,161] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,163] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,163] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,165] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,165] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,166] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,167] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:34,192] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,192] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:34,199] jepsen worker 79 - jepsen.util 1069 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,199] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,199] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,199] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,200] jepsen worker 53 - jepsen.util 503 :invoke :write [17 36] INFO [2022-03-13 17:14:34,200] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,200] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:34,200] jepsen worker 7 - jepsen.util 637 :invoke :write [2 41] INFO [2022-03-13 17:14:34,201] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:34,201] jepsen worker 6 - jepsen.util 636 :invoke :write [2 38] INFO [2022-03-13 17:14:34,206] jepsen worker 52 - jepsen.util 592 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,206] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:34,229] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,230] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:34,230] jepsen worker 58 - jepsen.util 2488 :info :write [19 30] :RequestTimeoutException INFO [2022-03-13 17:14:34,231] jepsen worker 58 - jepsen.util 2578 :invoke :read [19 nil] INFO [2022-03-13 17:14:34,241] jepsen worker 84 - jepsen.util 2964 :info :write [28 26] :RequestTimeoutException INFO [2022-03-13 17:14:34,241] jepsen worker 84 - jepsen.util 3054 :invoke :write [28 39] INFO [2022-03-13 17:14:34,250] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,255] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:34,271] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,272] jepsen worker 21 - jepsen.util 831 :invoke :write [7 29] INFO [2022-03-13 17:14:34,276] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,277] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:14:34,284] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,284] jepsen worker 60 - jepsen.util 600 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,285] jepsen worker 60 - jepsen.util 600 :invoke :write [20 39] INFO [2022-03-13 17:14:34,286] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:34,323] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,324] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:34,355] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,356] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:14:34,390] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,391] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,392] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,393] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,394] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,394] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:34,416] jepsen worker 14 - jepsen.util 374 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,416] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:34,417] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,417] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,417] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:34,418] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:14:34,420] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,420] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:34,425] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,425] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:34,457] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,457] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,459] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,459] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:34,488] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,489] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:34,498] jepsen worker 70 - jepsen.util 700 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,498] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,498] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,498] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,499] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:14:34,499] jepsen worker 13 - jepsen.util 643 :invoke :write [4 44] INFO [2022-03-13 17:14:34,500] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:34,501] jepsen worker 70 - jepsen.util 700 :invoke :write [23 48] INFO [2022-03-13 17:14:34,519] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,522] jepsen worker 30 - jepsen.util 2730 :ok :read [10 :nil] INFO [2022-03-13 17:14:34,522] jepsen worker 27 - jepsen.util 3357 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,522] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,523] jepsen worker 27 - jepsen.util 3357 :invoke :write [9 47] INFO [2022-03-13 17:14:34,523] jepsen worker 30 - jepsen.util 2730 :invoke :read [10 nil] INFO [2022-03-13 17:14:34,524] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,525] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:34,552] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,552] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:14:34,588] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,589] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,590] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,591] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:34,610] jepsen worker 40 - jepsen.util 2920 :info :write [13 2] :RequestTimeoutException INFO [2022-03-13 17:14:34,610] jepsen worker 40 - jepsen.util 3010 :invoke :read [13 nil] INFO [2022-03-13 17:14:34,619] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,620] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:34,624] jepsen worker 62 - jepsen.util 512 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,624] jepsen worker 44 - jepsen.util 584 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,624] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,624] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,625] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,624] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:34,625] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:34,625] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:34,626] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:34,627] jepsen worker 43 - jepsen.util 583 :invoke :write [14 2] INFO [2022-03-13 17:14:34,628] jepsen worker 42 - jepsen.util 402 :invoke :write [14 49] INFO [2022-03-13 17:14:34,629] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:34,634] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:34,634] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:34,639] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,640] jepsen worker 33 - jepsen.util 663 :invoke :write [11 20] INFO [2022-03-13 17:14:34,646] jepsen worker 21 - jepsen.util 831 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,646] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,646] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,649] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:34,650] jepsen worker 30 - jepsen.util 2730 :ok :read [10 :nil] INFO [2022-03-13 17:14:34,651] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:34,652] jepsen worker 30 - jepsen.util 2730 :invoke :write [10 25] INFO [2022-03-13 17:14:34,653] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:14:34,660] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,661] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:34,666] jepsen worker 40 - jepsen.util 3010 :ok :read [13 39] INFO [2022-03-13 17:14:34,667] jepsen worker 40 - jepsen.util 3010 :invoke :read [13 nil] INFO [2022-03-13 17:14:34,697] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,698] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,700] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,700] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,707] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,708] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,709] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,710] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:14:34,714] jepsen worker 40 - jepsen.util 3010 :ok :read [13 39] INFO [2022-03-13 17:14:34,715] jepsen worker 40 - jepsen.util 3010 :invoke :write [13 49] INFO [2022-03-13 17:14:34,733] jepsen worker 78 - jepsen.util 528 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,734] jepsen worker 78 - jepsen.util 528 :invoke :write [26 28] INFO [2022-03-13 17:14:34,757] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,758] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,759] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,760] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:34,822] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,824] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,825] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,826] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,827] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,828] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:34,840] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,840] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:14:34,847] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,847] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,847] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,848] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:34,849] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:14:34,849] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:14:34,852] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,853] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,855] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,856] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:14:34,906] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,907] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:14:34,908] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,908] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,909] jepsen worker 75 - jepsen.util 435 :invoke :write [25 11] INFO [2022-03-13 17:14:34,909] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:34,940] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,941] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:34,974] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,975] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:34,976] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:34,977] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:34,986] jepsen worker 72 - jepsen.util 2952 :info :write [24 25] :RequestTimeoutException INFO [2022-03-13 17:14:34,987] jepsen worker 72 - jepsen.util 3042 :invoke :read [24 nil] INFO [2022-03-13 17:14:35,031] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,031] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:14:35,044] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,044] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,045] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:35,046] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:14:35,055] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:35,055] jepsen worker nemesis - couchbase.util Rebalance check count: 384 INFO [2022-03-13 17:14:35,055] jepsen worker nemesis - couchbase.util Rebalance stuck count: 284 INFO [2022-03-13 17:14:35,056] jepsen worker 26 - jepsen.util 476 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,056] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,057] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,057] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,057] jepsen worker 12 - jepsen.util 732 :invoke :write [4 4] INFO [2022-03-13 17:14:35,058] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,058] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,058] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:35,058] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,058] jepsen worker 80 - jepsen.util 710 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,059] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:35,062] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:35,062] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:35,062] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:35,063] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 21] INFO [2022-03-13 17:14:35,066] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:35,078] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,078] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,088] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,088] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,089] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:14:35,090] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:35,098] jepsen worker 65 - jepsen.util 3215 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,098] jepsen worker 56 - jepsen.util 3026 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,098] jepsen worker 65 - jepsen.util 3215 :invoke :read [21 nil] INFO [2022-03-13 17:14:35,099] jepsen worker 56 - jepsen.util 3026 :invoke :write [18 43] INFO [2022-03-13 17:14:35,101] jepsen worker 65 - jepsen.util 3215 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,102] jepsen worker 65 - jepsen.util 3215 :invoke :read [21 nil] INFO [2022-03-13 17:14:35,103] jepsen worker 65 - jepsen.util 3215 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,104] jepsen worker 65 - jepsen.util 3215 :invoke :write [21 49] INFO [2022-03-13 17:14:35,121] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,121] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:14:35,124] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,125] jepsen worker 14 - jepsen.util 374 :invoke :write [4 24] INFO [2022-03-13 17:14:35,129] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,130] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,131] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,132] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,136] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,136] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,144] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,145] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:14:35,171] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,171] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,172] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,173] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,174] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,174] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,176] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,176] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,177] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,178] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,179] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,180] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,205] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,205] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,205] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:14:35,206] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,207] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,208] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:35,276] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,276] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,276] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:14:35,277] jepsen worker 22 - jepsen.util 652 :invoke :write [7 32] INFO [2022-03-13 17:14:35,283] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,284] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,284] jepsen worker 33 - jepsen.util 663 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,285] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:14:35,285] jepsen worker 33 - jepsen.util 663 :invoke :write [11 32] INFO [2022-03-13 17:14:35,286] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:14:35,340] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,343] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:14:35,370] jepsen worker 14 - jepsen.util 374 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,371] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:35,373] jepsen worker 53 - jepsen.util 503 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,374] jepsen worker 53 - jepsen.util 503 :invoke :write [17 22] INFO [2022-03-13 17:14:35,382] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,383] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,384] jepsen worker 13 - jepsen.util 643 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,384] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,384] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:35,385] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,386] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,387] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:14:35,407] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,407] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,408] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:35,408] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:35,433] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,434] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,435] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,436] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,437] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,438] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,439] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,439] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,440] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,441] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,443] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,444] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:14:35,495] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,495] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:35,507] jepsen worker 23 - jepsen.util 563 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,508] jepsen worker 23 - jepsen.util 563 :invoke :write [7 32] INFO [2022-03-13 17:14:35,532] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,533] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:35,536] jepsen worker 70 - jepsen.util 700 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,536] jepsen worker 70 - jepsen.util 700 :invoke :write [23 28] INFO [2022-03-13 17:14:35,572] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,573] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,580] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,580] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:35,589] jepsen worker 22 - jepsen.util 652 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,589] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:14:35,601] jepsen worker 75 - jepsen.util 435 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,601] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:14:35,612] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,613] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,616] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,616] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:35,644] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,644] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:35,691] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,692] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,693] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,694] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,695] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,696] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,697] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,698] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,699] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,699] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:35,783] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,784] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:35,786] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,787] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:14:35,888] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,889] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:35,951] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,952] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:35,956] jepsen worker 6 - jepsen.util 636 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,956] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,956] jepsen worker 6 - jepsen.util 636 :invoke :write [2 7] INFO [2022-03-13 17:14:35,966] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:35,969] jepsen worker 78 - jepsen.util 528 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,969] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,969] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,969] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,969] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,970] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:35,971] jepsen worker 13 - jepsen.util 643 :invoke :write [4 45] INFO [2022-03-13 17:14:35,972] jepsen worker 52 - jepsen.util 592 :invoke :write [17 18] INFO [2022-03-13 17:14:35,972] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:35,973] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:35,981] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:35,982] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:14:36,023] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,023] jepsen worker 33 - jepsen.util 663 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,023] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,024] jepsen worker 51 - jepsen.util 591 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,025] jepsen worker 33 - jepsen.util 663 :invoke :write [11 26] INFO [2022-03-13 17:14:36,025] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:14:36,027] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:36,028] jepsen worker 51 - jepsen.util 591 :invoke :write [17 33] INFO [2022-03-13 17:14:36,044] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,044] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,046] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,047] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,048] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,049] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:36,061] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:36,061] jepsen worker nemesis - couchbase.util Rebalance check count: 385 INFO [2022-03-13 17:14:36,062] jepsen worker nemesis - couchbase.util Rebalance stuck count: 285 INFO [2022-03-13 17:14:36,075] jepsen worker 70 - jepsen.util 700 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,076] jepsen worker 70 - jepsen.util 700 :invoke :write [23 45] INFO [2022-03-13 17:14:36,078] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,078] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,078] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,079] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:36,080] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,081] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:36,081] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,082] jepsen worker 34 - jepsen.util 754 :invoke :write [11 4] INFO [2022-03-13 17:14:36,083] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,085] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:36,121] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,122] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:36,149] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,149] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,151] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,151] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,152] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,153] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,154] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,154] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,156] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,156] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,156] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,156] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,157] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:14:36,157] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:36,158] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:14:36,158] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,159] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:36,159] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:14:36,161] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,161] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:14:36,163] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,164] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:36,165] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,165] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:14:36,170] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,171] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:36,172] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,173] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:36,174] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,175] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:14:36,178] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,180] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,182] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,184] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,184] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,184] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:36,187] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,187] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:36,215] jepsen worker 33 - jepsen.util 663 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,215] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:36,243] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,243] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,243] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,244] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,245] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:36,245] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:14:36,246] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:36,246] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,246] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:36,247] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,247] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:14:36,247] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,248] jepsen worker 24 - jepsen.util 564 :invoke :write [8 39] INFO [2022-03-13 17:14:36,250] jepsen worker 33 - jepsen.util 663 :invoke :write [11 33] INFO [2022-03-13 17:14:36,260] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,260] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,262] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,262] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:36,262] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,263] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,271] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,272] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,275] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,275] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:36,336] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,336] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,337] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,338] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,339] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,339] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,341] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,341] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:14:36,370] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,371] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,372] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,373] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,374] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,375] jepsen worker 35 - jepsen.util 935 :invoke :write [11 22] INFO [2022-03-13 17:14:36,375] jepsen worker 13 - jepsen.util 643 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,375] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,376] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:36,377] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:36,379] jepsen worker 7 - jepsen.util 637 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,380] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,380] jepsen worker 60 - jepsen.util 600 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,380] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:36,380] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,380] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,381] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:36,381] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,381] jepsen worker 61 - jepsen.util 691 :invoke :write [20 28] INFO [2022-03-13 17:14:36,381] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,382] jepsen worker 7 - jepsen.util 637 :invoke :write [2 30] INFO [2022-03-13 17:14:36,382] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:36,383] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:36,384] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:36,384] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] WARN [2022-03-13 17:14:36,386] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19052199,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19146185,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9075369,"encode_duration_us":19352,"operation_id":"0xa6a8","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10000959},{"operation_name":"upsert","last_dispatch_duration_us":9268404,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11265015,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9127781,"encode_duration_us":29435,"operation_id":"0xabcf","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10003040},{"operation_name":"upsert","last_dispatch_duration_us":9125800,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12153405,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9147569,"encode_duration_us":19608,"operation_id":"0xab55","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008894}],"total_count":3}} WARN [2022-03-13 17:14:36,386] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14432728,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":14432728,"total_server_duration_us":18953,"operation_id":"0xa8e4","timeout_ms":30000,"last_server_duration_us":18953,"total_duration_us":14434757},{"operation_name":"get","last_dispatch_duration_us":12213081,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":12213081,"total_server_duration_us":27,"operation_id":"0xa9f1","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":12213354},{"operation_name":"upsert","last_dispatch_duration_us":9125800,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12153405,"total_server_duration_us":9147569,"encode_duration_us":19608,"operation_id":"0xab55","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008894},{"operation_name":"upsert","last_dispatch_duration_us":9002668,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12899874,"total_server_duration_us":240462,"encode_duration_us":18318,"operation_id":"0xab05","timeout_ms":10000,"last_server_duration_us":33132,"total_duration_us":10003523},{"operation_name":"upsert","last_dispatch_duration_us":9268404,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11265015,"total_server_duration_us":9127781,"encode_duration_us":29435,"operation_id":"0xabcf","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10003040},{"operation_name":"upsert","last_dispatch_duration_us":77322,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":528053,"total_server_duration_us":76888,"encode_duration_us":20651,"operation_id":"0xaccd","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10002397},{"operation_name":"get","last_dispatch_duration_us":8997195,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8997195,"total_server_duration_us":19,"operation_id":"0xab8e","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":8999438},{"operation_name":"get","last_dispatch_duration_us":8763873,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8763873,"total_server_duration_us":32,"operation_id":"0xabad","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8764188},{"operation_name":"get","last_dispatch_duration_us":5846787,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":5846787,"total_server_duration_us":23,"operation_id":"0xad38","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":5846994},{"operation_name":"get","last_dispatch_duration_us":3556527,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":3556527,"total_server_duration_us":19,"operation_id":"0xae4c","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":3556747}],"total_count":47}} INFO [2022-03-13 17:14:36,386] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:36,416] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,416] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:36,432] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,433] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,434] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,435] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,436] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,436] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:14:36,484] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,485] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,486] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,487] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:14:36,518] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,518] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,519] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,520] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,521] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,522] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:14:36,553] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,553] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,554] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,555] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,556] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,556] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:14:36,579] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,580] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:14:36,583] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,584] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:14:36,596] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,596] jepsen worker 22 - jepsen.util 652 :invoke :write [7 31] INFO [2022-03-13 17:14:36,632] jepsen worker 52 - jepsen.util 592 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,632] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,632] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,633] jepsen worker 80 - jepsen.util 710 :invoke :write [26 29] INFO [2022-03-13 17:14:36,634] jepsen worker 78 - jepsen.util 528 :invoke :write [26 13] INFO [2022-03-13 17:14:36,634] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:36,646] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,647] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:14:36,664] jepsen worker 24 - jepsen.util 564 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,665] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:36,715] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,716] jepsen worker 21 - jepsen.util 831 :invoke :write [7 21] INFO [2022-03-13 17:14:36,725] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,726] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:36,794] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,795] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:36,803] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,804] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:14:36,824] jepsen worker 43 - jepsen.util 583 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,824] jepsen worker 12 - jepsen.util 732 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,825] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,827] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,828] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:36,828] jepsen worker 43 - jepsen.util 583 :invoke :write [14 6] INFO [2022-03-13 17:14:36,833] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,833] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:36,854] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,855] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:14:36,856] jepsen worker 22 - jepsen.util 652 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,856] jepsen worker 70 - jepsen.util 700 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,856] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:36,857] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:36,858] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,858] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:14:36,910] jepsen worker 33 - jepsen.util 663 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,910] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,910] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,911] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:36,911] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:36,912] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:36,913] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,913] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:14:36,919] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,920] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:14:36,924] jepsen worker 42 - jepsen.util 402 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,925] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:36,958] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,959] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,960] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,961] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,962] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,962] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,964] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,964] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,965] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,965] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:36,990] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,991] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:36,993] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:36,994] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:37,026] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,027] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,028] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,029] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,030] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,031] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:14:37,042] jepsen worker 34 - jepsen.util 754 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,042] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,042] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,043] jepsen worker 34 - jepsen.util 754 :invoke :write [11 39] INFO [2022-03-13 17:14:37,043] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:37,044] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:37,050] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,050] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,050] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,051] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:14:37,052] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:37,052] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:37,053] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,054] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:14:37,067] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:37,068] jepsen worker nemesis - couchbase.util Rebalance check count: 386 INFO [2022-03-13 17:14:37,068] jepsen worker nemesis - couchbase.util Rebalance stuck count: 286 INFO [2022-03-13 17:14:37,073] jepsen worker 35 - jepsen.util 935 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,074] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:37,092] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,093] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:14:37,100] jepsen worker 23 - jepsen.util 563 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,100] jepsen worker 23 - jepsen.util 563 :invoke :write [7 25] INFO [2022-03-13 17:14:37,126] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,127] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:37,164] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,164] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,166] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,166] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,168] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,168] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:14:37,179] jepsen worker 80 - jepsen.util 710 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,180] jepsen worker 80 - jepsen.util 710 :invoke :write [26 34] INFO [2022-03-13 17:14:37,203] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,204] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,206] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,206] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:14:37,236] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,236] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:14:37,259] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,259] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:37,261] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,261] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:14:37,264] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,264] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,264] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,264] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:37,265] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:14:37,265] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:37,272] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,273] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,275] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,275] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:14:37,311] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,311] jepsen worker 78 - jepsen.util 528 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,311] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,312] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,312] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:37,313] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:37,314] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:37,314] jepsen worker 70 - jepsen.util 700 :invoke :write [23 12] INFO [2022-03-13 17:14:37,318] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,319] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:14:37,326] jepsen worker 57 - jepsen.util 2757 :info :write [19 46] :RequestTimeoutException INFO [2022-03-13 17:14:37,327] jepsen worker 57 - jepsen.util 2847 :invoke :write [19 45] INFO [2022-03-13 17:14:37,349] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,350] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:14:37,379] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,380] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,381] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,382] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,384] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,385] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:37,408] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,409] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,411] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,411] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,413] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,414] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,415] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,416] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,418] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,418] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,420] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,421] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,432] jepsen worker 17 - jepsen.util 2897 :ok :read [5 43] INFO [2022-03-13 17:14:37,433] jepsen worker 17 - jepsen.util 2897 :invoke :write [5 15] INFO [2022-03-13 17:14:37,445] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,445] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,445] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,446] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:37,447] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,448] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:14:37,469] jepsen worker 79 - jepsen.util 1069 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,470] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 12] INFO [2022-03-13 17:14:37,472] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,472] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,472] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,473] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:37,474] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,474] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,474] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:14:37,475] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,475] jepsen worker 21 - jepsen.util 831 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,475] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:37,476] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:14:37,477] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:37,478] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:37,478] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:37,479] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,480] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,480] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:14:37,480] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,482] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:14:37,487] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:37,495] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,496] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:37,497] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,497] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:37,527] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,527] jepsen worker 34 - jepsen.util 754 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,528] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:37,528] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:37,529] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,530] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:14:37,565] jepsen worker 51 - jepsen.util 591 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,566] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,566] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:37,566] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,566] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,566] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,567] jepsen worker 44 - jepsen.util 584 :invoke :write [14 43] INFO [2022-03-13 17:14:37,569] jepsen worker 60 - jepsen.util 600 :invoke :write [20 32] INFO [2022-03-13 17:14:37,570] jepsen worker 51 - jepsen.util 591 :invoke :write [17 45] INFO [2022-03-13 17:14:37,571] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:37,572] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:37,572] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,575] jepsen worker 76 - jepsen.util 706 :invoke :write [25 0] INFO [2022-03-13 17:14:37,575] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,575] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:14:37,576] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 6] INFO [2022-03-13 17:14:37,581] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,581] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,581] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,582] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:14:37,596] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:37,596] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:14:37,596] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,596] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,597] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:37,598] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:14:37,678] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,679] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,680] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,681] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,700] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,700] jepsen worker 80 - jepsen.util 710 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,700] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,700] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,700] jepsen worker 13 - jepsen.util 643 :invoke :write [4 28] INFO [2022-03-13 17:14:37,701] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:37,701] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:37,702] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:37,728] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,729] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,729] jepsen worker 36 - jepsen.util 2196 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,730] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,731] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,731] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,732] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:37,732] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:37,734] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,734] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:37,760] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,760] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:14:37,827] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,827] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,828] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:14:37,828] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] WARN [2022-03-13 17:14:37,885] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17292851,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17292851,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9032060,"encode_duration_us":20372,"operation_id":"0xa619","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10001547},{"operation_name":"upsert","last_dispatch_duration_us":19984478,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":28612260,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9114572,"encode_duration_us":17430,"operation_id":"0xa39c","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10003199},{"operation_name":"upsert","last_dispatch_duration_us":18159388,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18159388,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9024634,"encode_duration_us":17409,"operation_id":"0xaa1c","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009519},{"operation_name":"upsert","last_dispatch_duration_us":18177091,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25998971,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9051245,"encode_duration_us":21018,"operation_id":"0xa65e","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10004429}],"total_count":4}} WARN [2022-03-13 17:14:37,888] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17892215,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20017291,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9621463,"encode_duration_us":19066,"operation_id":"0xa405","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10005576},{"operation_name":"upsert","last_dispatch_duration_us":17920915,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18590964,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9409448,"encode_duration_us":19434,"operation_id":"0xa6a7","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10006361},{"operation_name":"upsert","last_dispatch_duration_us":17540455,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17857205,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9197263,"encode_duration_us":17827,"operation_id":"0xa85c","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10010830},{"operation_name":"upsert","last_dispatch_duration_us":15821537,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15821537,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9050106,"encode_duration_us":16797,"operation_id":"0xa953","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10009859}],"total_count":4}} INFO [2022-03-13 17:14:37,890] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,890] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:37,916] jepsen worker 53 - jepsen.util 503 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,916] jepsen worker 61 - jepsen.util 691 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,917] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:37,918] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:37,919] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,919] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,920] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:37,920] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:37,922] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,923] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:37,924] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,925] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:14:37,938] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,938] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,938] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:37,939] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:14:37,950] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,950] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:37,970] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:37,971] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:37,985] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException WARN [2022-03-13 17:14:37,986] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16143743,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":16143743,"total_server_duration_us":7698,"operation_id":"0xa6d7","timeout_ms":30000,"last_server_duration_us":7698,"total_duration_us":16144162},{"operation_name":"get","last_dispatch_duration_us":8807888,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8807888,"total_server_duration_us":76,"operation_id":"0xabd1","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":14997528},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":21338,"operation_id":"0xab06","timeout_ms":10000,"total_duration_us":10011014},{"operation_name":"upsert","last_dispatch_duration_us":18159388,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18159388,"total_server_duration_us":9024634,"encode_duration_us":17409,"operation_id":"0xaa1c","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009519},{"operation_name":"upsert","last_dispatch_duration_us":8499957,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8629274,"total_server_duration_us":40171,"encode_duration_us":19822,"operation_id":"0xab5e","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10006752},{"operation_name":"upsert","last_dispatch_duration_us":8498642,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8556485,"total_server_duration_us":38526,"encode_duration_us":19973,"operation_id":"0xabb5","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10006211},{"operation_name":"get","last_dispatch_duration_us":9952067,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9952067,"total_server_duration_us":38,"operation_id":"0xad3e","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9952377},{"operation_name":"get","last_dispatch_duration_us":7670258,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7670258,"total_server_duration_us":43,"operation_id":"0xae5a","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":7670452},{"operation_name":"upsert","last_dispatch_duration_us":2137893,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2137893,"total_server_duration_us":97793,"encode_duration_us":100399,"operation_id":"0xadc4","timeout_ms":10000,"last_server_duration_us":97793,"total_duration_us":2708204},{"operation_name":"upsert","last_dispatch_duration_us":2340410,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2340410,"total_server_duration_us":62540,"encode_duration_us":26586,"operation_id":"0xad74","timeout_ms":10000,"last_server_duration_us":62540,"total_duration_us":2347699}],"total_count":72}} INFO [2022-03-13 17:14:37,986] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] WARN [2022-03-13 17:14:37,988] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11883611,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11883611,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9042674,"encode_duration_us":20980,"operation_id":"0xa95d","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10006846}],"total_count":1}} WARN [2022-03-13 17:14:37,988] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10477631,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10477631,"total_server_duration_us":62,"operation_id":"0xacb5","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":10477914},{"operation_name":"upsert","last_dispatch_duration_us":8984897,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8984897,"total_server_duration_us":21405,"encode_duration_us":18716,"operation_id":"0xad04","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10010097},{"operation_name":"upsert","last_dispatch_duration_us":135401,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9886555,"total_server_duration_us":211246,"encode_duration_us":17518,"operation_id":"0xaccc","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10009174},{"operation_name":"upsert","last_dispatch_duration_us":112090,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9938076,"total_server_duration_us":214453,"encode_duration_us":16659,"operation_id":"0xacbb","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10006132},{"operation_name":"upsert","last_dispatch_duration_us":13270969,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13270969,"total_server_duration_us":28493,"encode_duration_us":16966,"operation_id":"0xab2e","timeout_ms":10000,"last_server_duration_us":28493,"total_duration_us":10003782},{"operation_name":"upsert","last_dispatch_duration_us":9237098,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11764620,"total_server_duration_us":9133697,"encode_duration_us":18776,"operation_id":"0xab71","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10001773},{"operation_name":"get","last_dispatch_duration_us":9102854,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9102854,"total_server_duration_us":76,"operation_id":"0xad55","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":9103152},{"operation_name":"upsert","last_dispatch_duration_us":1954917,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1954917,"total_server_duration_us":196453,"encode_duration_us":28892,"operation_id":"0xb007","timeout_ms":10000,"last_server_duration_us":196453,"total_duration_us":1975259},{"operation_name":"upsert","last_dispatch_duration_us":1431331,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1431331,"total_server_duration_us":171350,"encode_duration_us":28492,"operation_id":"0xae5b","timeout_ms":10000,"last_server_duration_us":171350,"total_duration_us":1436947},{"operation_name":"upsert","last_dispatch_duration_us":1288527,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1288527,"total_server_duration_us":351353,"encode_duration_us":28789,"operation_id":"0xaf2f","timeout_ms":10000,"last_server_duration_us":351353,"total_duration_us":1293644}],"total_count":13}} INFO [2022-03-13 17:14:38,009] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,010] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,011] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:38,012] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:38,013] jepsen worker 70 - jepsen.util 700 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,014] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,014] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,014] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,014] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,015] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:38,016] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:38,017] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:38,018] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:38,018] jepsen worker 70 - jepsen.util 700 :invoke :write [23 10] INFO [2022-03-13 17:14:38,020] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,020] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:38,053] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,053] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:38,058] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,058] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,060] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,060] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:38,073] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:38,073] jepsen worker nemesis - couchbase.util Rebalance check count: 387 INFO [2022-03-13 17:14:38,073] jepsen worker nemesis - couchbase.util Rebalance stuck count: 287 WARN [2022-03-13 17:14:38,088] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7191939,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8696667,"total_server_duration_us":134658,"encode_duration_us":16967,"operation_id":"0xab37","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10009681},{"operation_name":"upsert","last_dispatch_duration_us":7448493,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8714868,"total_server_duration_us":158187,"encode_duration_us":31706,"operation_id":"0xab79","timeout_ms":10000,"last_server_duration_us":36119,"total_duration_us":10009297},{"operation_name":"upsert","last_dispatch_duration_us":33726,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5684362,"total_server_duration_us":315464,"encode_duration_us":17734,"operation_id":"0xad32","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10008793},{"operation_name":"upsert","last_dispatch_duration_us":30738,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1103418,"total_server_duration_us":163057,"encode_duration_us":20123,"operation_id":"0xabcc","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10007701},{"operation_name":"upsert","last_dispatch_duration_us":200145,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10031115,"total_server_duration_us":159283,"encode_duration_us":17847,"operation_id":"0xaa8c","timeout_ms":10000,"last_server_duration_us":32258,"total_duration_us":10004992},{"operation_name":"upsert","last_dispatch_duration_us":197305,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9924297,"total_server_duration_us":175725,"encode_duration_us":18615,"operation_id":"0xaaa6","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10003218},{"operation_name":"upsert","last_dispatch_duration_us":9235954,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9235954,"total_server_duration_us":9040551,"encode_duration_us":27877,"operation_id":"0xab76","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9239789},{"operation_name":"get","last_dispatch_duration_us":9186276,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9186276,"total_server_duration_us":32065,"operation_id":"0xaaa2","timeout_ms":30000,"last_server_duration_us":32065,"total_duration_us":9188759},{"operation_name":"get","last_dispatch_duration_us":9161296,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9161296,"total_server_duration_us":1046,"operation_id":"0xaa97","timeout_ms":30000,"last_server_duration_us":1046,"total_duration_us":9161485},{"operation_name":"get","last_dispatch_duration_us":2858178,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":2858178,"total_server_duration_us":43,"operation_id":"0xadf7","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":2860523}],"total_count":42}} WARN [2022-03-13 17:14:38,089] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9237098,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11764620,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9133697,"encode_duration_us":18776,"operation_id":"0xab71","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10001773},{"operation_name":"upsert","last_dispatch_duration_us":17951710,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19369786,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9146279,"encode_duration_us":17748,"operation_id":"0xa85a","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10003353}],"total_count":2}} INFO [2022-03-13 17:14:38,096] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,097] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,099] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,099] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,101] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,101] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,102] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,103] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,104] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,105] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,107] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,107] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:38,116] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,116] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,117] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:38,117] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:14:38,120] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,121] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:38,121] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,122] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,122] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:14:38,124] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:38,128] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,128] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,130] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:14:38,130] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,132] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:38,133] jepsen worker 22 - jepsen.util 652 :invoke :write [7 7] INFO [2022-03-13 17:14:38,139] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,140] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:38,142] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,143] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:38,149] jepsen worker 86 - jepsen.util 2696 :ok :read [28 44] INFO [2022-03-13 17:14:38,150] jepsen worker 86 - jepsen.util 2696 :invoke :write [28 30] WARN [2022-03-13 17:14:38,176] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13377432,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":13377432,"total_server_duration_us":9977,"operation_id":"0xa8ca","timeout_ms":30000,"last_server_duration_us":9977,"total_duration_us":13379469},{"operation_name":"upsert","last_dispatch_duration_us":76829,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":750103,"total_server_duration_us":449669,"encode_duration_us":25008,"operation_id":"0xae4a","timeout_ms":10000,"last_server_duration_us":27764,"total_duration_us":10008497},{"operation_name":"upsert","last_dispatch_duration_us":40187,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1656574,"total_server_duration_us":682864,"encode_duration_us":20553,"operation_id":"0xabb8","timeout_ms":10000,"last_server_duration_us":27132,"total_duration_us":10006806},{"operation_name":"upsert","last_dispatch_duration_us":35840,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1448125,"total_server_duration_us":660918,"encode_duration_us":18601,"operation_id":"0xac29","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10001548},{"operation_name":"get","last_dispatch_duration_us":8784682,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8784682,"total_server_duration_us":517,"operation_id":"0xaad3","timeout_ms":30000,"last_server_duration_us":517,"total_duration_us":8785288},{"operation_name":"upsert","last_dispatch_duration_us":3278700,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3278700,"total_server_duration_us":38690,"encode_duration_us":17403,"operation_id":"0xaf02","timeout_ms":10000,"last_server_duration_us":38690,"total_duration_us":3284603},{"operation_name":"get","last_dispatch_duration_us":2858784,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2858784,"total_server_duration_us":49,"operation_id":"0xaf38","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":2859314},{"operation_name":"upsert","last_dispatch_duration_us":2827794,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2827794,"total_server_duration_us":52420,"encode_duration_us":19809,"operation_id":"0xb01a","timeout_ms":10000,"last_server_duration_us":52420,"total_duration_us":2837595},{"operation_name":"upsert","last_dispatch_duration_us":2705077,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2705077,"total_server_duration_us":350820,"encode_duration_us":17880,"operation_id":"0xb019","timeout_ms":10000,"last_server_duration_us":350820,"total_duration_us":2711452},{"operation_name":"upsert","last_dispatch_duration_us":2664513,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2664513,"total_server_duration_us":106947,"encode_duration_us":35911,"operation_id":"0xae92","timeout_ms":10000,"last_server_duration_us":106947,"total_duration_us":2681911}],"total_count":51}} WARN [2022-03-13 17:14:38,176] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17619947,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17986730,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9135981,"encode_duration_us":17616,"operation_id":"0xa652","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10003724}],"total_count":1}} INFO [2022-03-13 17:14:38,178] jepsen worker 83 - jepsen.util 2153 :info :write [27 37] :RequestTimeoutException INFO [2022-03-13 17:14:38,179] jepsen worker 83 - jepsen.util 2243 :invoke :write [27 24] INFO [2022-03-13 17:14:38,181] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,182] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,183] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,184] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] WARN [2022-03-13 17:14:38,189] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":164124,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":521437,"total_server_duration_us":233658,"encode_duration_us":21808,"operation_id":"0xab8f","timeout_ms":10000,"last_server_duration_us":58128,"total_duration_us":10007610},{"operation_name":"upsert","last_dispatch_duration_us":11925395,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11925395,"total_server_duration_us":24497,"encode_duration_us":19244,"operation_id":"0xab32","timeout_ms":10000,"last_server_duration_us":24497,"total_duration_us":10007266},{"operation_name":"upsert","last_dispatch_duration_us":8155732,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8421400,"total_server_duration_us":99671,"encode_duration_us":18755,"operation_id":"0xabcb","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10007047},{"operation_name":"upsert","last_dispatch_duration_us":8173380,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8577929,"total_server_duration_us":154984,"encode_duration_us":27476,"operation_id":"0xabaf","timeout_ms":10000,"last_server_duration_us":26953,"total_duration_us":10006900},{"operation_name":"get","last_dispatch_duration_us":6976622,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6976622,"total_server_duration_us":38,"operation_id":"0xadc2","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":6976968},{"operation_name":"upsert","last_dispatch_duration_us":2337699,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2337699,"total_server_duration_us":138798,"encode_duration_us":18742,"operation_id":"0xb2e5","timeout_ms":10000,"last_server_duration_us":138798,"total_duration_us":2521888},{"operation_name":"upsert","last_dispatch_duration_us":2308858,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2308858,"total_server_duration_us":283237,"encode_duration_us":23034,"operation_id":"0xb2c5","timeout_ms":10000,"last_server_duration_us":283237,"total_duration_us":2382521},{"operation_name":"upsert","last_dispatch_duration_us":2284077,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2284077,"total_server_duration_us":185342,"encode_duration_us":33967,"operation_id":"0xaeb8","timeout_ms":10000,"last_server_duration_us":185342,"total_duration_us":2301191},{"operation_name":"upsert","last_dispatch_duration_us":2263570,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2263570,"total_server_duration_us":35613,"encode_duration_us":22130,"operation_id":"0xb295","timeout_ms":10000,"last_server_duration_us":35613,"total_duration_us":2273689},{"operation_name":"upsert","last_dispatch_duration_us":2159468,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2159468,"total_server_duration_us":186971,"encode_duration_us":24045,"operation_id":"0xb00c","timeout_ms":10000,"last_server_duration_us":186971,"total_duration_us":2176063}],"total_count":58}} INFO [2022-03-13 17:14:38,193] jepsen worker 48 - jepsen.util 2838 :ok :read [16 :nil] INFO [2022-03-13 17:14:38,194] jepsen worker 48 - jepsen.util 2838 :invoke :read [16 nil] INFO [2022-03-13 17:14:38,208] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,208] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,208] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:38,209] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:38,215] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,216] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,217] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,218] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,219] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,220] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:14:38,245] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,246] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,247] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,247] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,248] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,249] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:38,276] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,277] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,279] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,280] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,281] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,282] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,283] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,284] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,286] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,286] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,287] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,288] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:14:38,311] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,312] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 21] INFO [2022-03-13 17:14:38,323] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,324] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,358] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:38,328] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,359] jepsen worker 21 - jepsen.util 831 :invoke :write [7 21] INFO [2022-03-13 17:14:38,359] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:38,328] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,359] jepsen worker 34 - jepsen.util 754 :invoke :write [11 40] INFO [2022-03-13 17:14:38,328] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,329] jepsen worker 13 - jepsen.util 643 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,360] jepsen worker 80 - jepsen.util 710 :invoke :write [26 17] INFO [2022-03-13 17:14:38,360] jepsen worker 13 - jepsen.util 643 :invoke :write [4 11] INFO [2022-03-13 17:14:38,329] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,339] jepsen worker 6 - jepsen.util 636 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,363] jepsen worker 52 - jepsen.util 592 :invoke :write [17 25] INFO [2022-03-13 17:14:38,363] jepsen worker 6 - jepsen.util 636 :invoke :write [2 12] INFO [2022-03-13 17:14:38,339] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,366] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:14:38,340] jepsen worker 23 - jepsen.util 563 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,342] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,370] jepsen worker 23 - jepsen.util 563 :invoke :write [7 10] INFO [2022-03-13 17:14:38,370] jepsen worker 69 - jepsen.util 699 :invoke :write [23 17] INFO [2022-03-13 17:14:38,374] jepsen worker 19 - jepsen.util 2629 :ok :read [6 16] INFO [2022-03-13 17:14:38,342] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,355] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,355] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,376] jepsen worker 77 - jepsen.util 707 :invoke :write [25 32] INFO [2022-03-13 17:14:38,376] jepsen worker 14 - jepsen.util 374 :invoke :write [4 7] INFO [2022-03-13 17:14:38,376] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,381] jepsen worker 55 - jepsen.util 2665 :ok :read [18 21] INFO [2022-03-13 17:14:38,383] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,389] jepsen worker 55 - jepsen.util 2665 :invoke :write [18 8] INFO [2022-03-13 17:14:38,401] jepsen worker 19 - jepsen.util 2629 :invoke :read [6 nil] INFO [2022-03-13 17:14:38,402] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:38,417] jepsen worker 74 - jepsen.util 3044 :ok :read [24 24] INFO [2022-03-13 17:14:38,417] jepsen worker 74 - jepsen.util 3044 :invoke :write [24 3] INFO [2022-03-13 17:14:38,430] jepsen worker 9 - jepsen.util 2349 :ok :read [3 25] INFO [2022-03-13 17:14:38,430] jepsen worker 9 - jepsen.util 2349 :invoke :read [3 nil] INFO [2022-03-13 17:14:38,437] jepsen worker 36 - jepsen.util 2196 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,438] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 32] INFO [2022-03-13 17:14:38,469] jepsen worker 19 - jepsen.util 2629 :ok :read [6 16] INFO [2022-03-13 17:14:38,470] jepsen worker 19 - jepsen.util 2629 :invoke :write [6 1] INFO [2022-03-13 17:14:38,472] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,472] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,474] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,474] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:38,514] jepsen worker 36 - jepsen.util 2196 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,514] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,516] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,517] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:14:38,549] jepsen worker 9 - jepsen.util 2349 :ok :read [3 25] INFO [2022-03-13 17:14:38,550] jepsen worker 9 - jepsen.util 2349 :invoke :write [3 13] INFO [2022-03-13 17:14:38,551] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,551] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,552] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:14:38,552] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,552] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,552] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,552] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,552] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,552] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:38,554] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:14:38,554] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:38,555] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:38,556] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:38,556] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:38,556] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,557] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,557] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:14:38,558] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:14:38,559] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,559] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:38,560] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,561] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:38,562] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,563] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:14:38,589] jepsen worker 21 - jepsen.util 831 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,590] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:38,595] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,596] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:14:38,619] jepsen worker 14 - jepsen.util 374 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,620] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,620] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:14:38,621] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:38,636] jepsen worker 22 - jepsen.util 652 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,637] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:14:38,650] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,652] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:38,654] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,655] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,655] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,655] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:38,655] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:38,656] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:14:38,671] jepsen worker 7 - jepsen.util 637 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,671] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:38,698] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,698] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,700] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,701] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,702] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,703] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,704] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,705] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,705] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,706] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,711] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,712] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:14:38,727] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,727] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,727] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:38,734] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:38,744] jepsen worker 44 - jepsen.util 584 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,744] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:38,803] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,804] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:14:38,827] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,828] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,829] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,830] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:14:38,856] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,857] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,858] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,858] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:38,885] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,885] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,887] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,888] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,889] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,890] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,891] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,892] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:14:38,916] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,916] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:14:38,927] jepsen worker 70 - jepsen.util 700 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,927] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,927] jepsen worker 70 - jepsen.util 700 :invoke :write [23 42] INFO [2022-03-13 17:14:38,928] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:38,945] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,946] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:14:38,951] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,951] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,951] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,952] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:38,953] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:14:38,953] jepsen worker 14 - jepsen.util 374 :invoke :write [4 6] INFO [2022-03-13 17:14:38,974] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,975] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:38,976] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:38,977] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:14:39,014] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,015] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,016] jepsen worker 43 - jepsen.util 583 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,016] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,016] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,016] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,016] jepsen worker 43 - jepsen.util 583 :invoke :write [14 20] INFO [2022-03-13 17:14:39,016] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:39,017] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:39,017] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,018] jepsen worker 26 - jepsen.util 476 :invoke :write [8 47] INFO [2022-03-13 17:14:39,018] jepsen worker 42 - jepsen.util 402 :invoke :write [14 17] INFO [2022-03-13 17:14:39,020] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,020] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:14:39,024] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,025] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,026] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,027] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:14:39,058] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,058] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,059] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,060] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:39,079] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:39,079] jepsen worker nemesis - couchbase.util Rebalance check count: 388 INFO [2022-03-13 17:14:39,079] jepsen worker nemesis - couchbase.util Rebalance stuck count: 288 INFO [2022-03-13 17:14:39,080] jepsen worker 16 - jepsen.util 2626 :info :write [5 38] :RequestTimeoutException INFO [2022-03-13 17:14:39,081] jepsen worker 16 - jepsen.util 2716 :invoke :read [5 nil] INFO [2022-03-13 17:14:39,084] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,085] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,086] jepsen worker 21 - jepsen.util 831 :invoke :write [7 16] INFO [2022-03-13 17:14:39,086] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:14:39,096] jepsen worker 80 - jepsen.util 710 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,097] jepsen worker 80 - jepsen.util 710 :invoke :write [26 14] INFO [2022-03-13 17:14:39,103] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,104] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,104] jepsen worker 76 - jepsen.util 706 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,106] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:39,113] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,114] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,137] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,138] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:39,152] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,152] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:14:39,154] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,154] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,155] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,156] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,158] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,159] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,160] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,160] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:14:39,169] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,170] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,171] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,172] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:14:39,181] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,181] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:39,213] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,214] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,215] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,215] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,215] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:14:39,216] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,217] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,217] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,218] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,219] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:39,240] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,240] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:14:39,305] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,306] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,307] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,307] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,309] jepsen worker 10 - jepsen.util 2890 :info :write [3 42] :SyncWriteAmbiguous INFO [2022-03-13 17:14:39,309] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,310] jepsen worker 10 - jepsen.util 2980 :invoke :read [3 nil] INFO [2022-03-13 17:14:39,310] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,317] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,317] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,318] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,319] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,353] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,353] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,354] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,354] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,358] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,358] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,359] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,359] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:14:39,360] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,360] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,361] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,362] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:14:39,366] jepsen worker 63 - jepsen.util 2763 :info :write [21 47] :RequestTimeoutException INFO [2022-03-13 17:14:39,367] jepsen worker 63 - jepsen.util 2853 :invoke :write [21 46] INFO [2022-03-13 17:14:39,379] jepsen worker 52 - jepsen.util 592 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,380] jepsen worker 52 - jepsen.util 592 :invoke :write [17 19] INFO [2022-03-13 17:14:39,396] jepsen worker 64 - jepsen.util 2674 :info :write [21 48] :RequestTimeoutException INFO [2022-03-13 17:14:39,397] jepsen worker 64 - jepsen.util 2764 :invoke :write [21 1] INFO [2022-03-13 17:14:39,419] jepsen worker 13 - jepsen.util 643 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,420] jepsen worker 13 - jepsen.util 643 :invoke :write [4 6] INFO [2022-03-13 17:14:39,433] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,434] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,447] jepsen worker 14 - jepsen.util 374 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,448] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:39,453] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,454] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,454] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:14:39,455] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:39,457] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,458] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:39,464] jepsen worker 79 - jepsen.util 1069 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,464] jepsen worker 60 - jepsen.util 600 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,464] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,466] jepsen worker 53 - jepsen.util 503 :invoke :write [17 9] INFO [2022-03-13 17:14:39,474] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:39,475] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:39,550] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,551] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,587] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,588] jepsen worker 21 - jepsen.util 831 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,588] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,588] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,591] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,592] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,593] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,593] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,593] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,594] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,594] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:39,595] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,596] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,597] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,598] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,599] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,600] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,601] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,602] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,602] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,603] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,604] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:14:39,604] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,605] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:14:39,652] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,653] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:14:39,696] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,697] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,698] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,699] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:14:39,745] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,745] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,747] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,748] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:14:39,785] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,786] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:39,792] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,792] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,793] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,793] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,793] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:14:39,794] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:39,795] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,795] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,795] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,796] jepsen worker 34 - jepsen.util 754 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,797] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,797] jepsen worker 14 - jepsen.util 374 :invoke :write [4 23] INFO [2022-03-13 17:14:39,797] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,798] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,798] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:39,798] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:14:39,803] jepsen worker 34 - jepsen.util 754 :invoke :write [11 27] INFO [2022-03-13 17:14:39,804] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:14:39,812] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,812] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:39,819] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,826] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:39,827] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,828] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:14:39,850] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,850] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,852] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,854] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:14:39,855] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:14:39,857] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:14:39,861] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,862] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:14:39,895] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,895] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:14:39,903] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,903] jepsen worker 6 - jepsen.util 636 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,903] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,903] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,903] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,903] jepsen worker 51 - jepsen.util 591 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,904] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:39,905] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:39,905] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:39,906] jepsen worker 6 - jepsen.util 636 :invoke :write [2 21] INFO [2022-03-13 17:14:39,907] jepsen worker 12 - jepsen.util 732 :invoke :write [4 38] INFO [2022-03-13 17:14:39,907] jepsen worker 21 - jepsen.util 831 :invoke :write [7 48] INFO [2022-03-13 17:14:39,910] jepsen worker 70 - jepsen.util 700 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,911] jepsen worker 70 - jepsen.util 700 :invoke :write [23 25] INFO [2022-03-13 17:14:39,933] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,933] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,937] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,938] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,940] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,941] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:14:39,964] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,964] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:39,966] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,966] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:14:39,996] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:39,997] jepsen worker 71 - jepsen.util 521 :invoke :write [23 1] INFO [2022-03-13 17:14:40,008] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,008] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,009] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,010] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,012] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,013] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:40,054] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,054] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,056] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,056] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,058] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,058] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,059] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,060] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,061] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,062] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:14:40,063] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,064] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,068] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,068] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,070] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,071] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,076] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,077] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,078] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,079] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,082] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,083] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,084] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,084] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:14:40,085] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:40,085] jepsen worker nemesis - couchbase.util Rebalance check count: 389 INFO [2022-03-13 17:14:40,085] jepsen worker nemesis - couchbase.util Rebalance stuck count: 289 INFO [2022-03-13 17:14:40,087] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,087] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,087] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,088] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,089] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,090] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:14:40,091] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,092] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:14:40,096] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,097] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,099] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,100] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,101] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,102] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,103] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,104] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:14:40,114] jepsen worker 47 - jepsen.util 2747 :info :write [15 35] :RequestTimeoutException INFO [2022-03-13 17:14:40,115] jepsen worker 47 - jepsen.util 2837 :invoke :read [15 nil] INFO [2022-03-13 17:14:40,132] jepsen worker 54 - jepsen.util 2394 :info :write [18 19] :SyncWriteAmbiguous INFO [2022-03-13 17:14:40,132] jepsen worker 20 - jepsen.util 3170 :info :write [6 14] :SyncWriteAmbiguous INFO [2022-03-13 17:14:40,133] jepsen worker 54 - jepsen.util 2484 :invoke :write [18 45] INFO [2022-03-13 17:14:40,133] jepsen worker 20 - jepsen.util 3260 :invoke :read [6 nil] INFO [2022-03-13 17:14:40,163] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,163] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,164] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,164] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,164] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:40,165] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:40,166] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:14:40,166] jepsen worker 8 - jepsen.util 728 :invoke :write [2 7] INFO [2022-03-13 17:14:40,172] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,172] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:40,199] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,200] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,201] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,202] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:40,231] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,232] jepsen worker 80 - jepsen.util 710 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,232] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:14:40,233] jepsen worker 80 - jepsen.util 710 :invoke :write [26 12] INFO [2022-03-13 17:14:40,261] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,263] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:40,318] jepsen worker 14 - jepsen.util 374 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,318] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,318] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:40,319] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:14:40,323] jepsen worker 23 - jepsen.util 563 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,324] jepsen worker 23 - jepsen.util 563 :invoke :write [7 40] INFO [2022-03-13 17:14:40,325] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,325] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:40,326] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,327] jepsen worker 33 - jepsen.util 663 :invoke :write [11 2] INFO [2022-03-13 17:14:40,356] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,357] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:40,368] jepsen worker 20 - jepsen.util 3260 :ok :read [6 16] INFO [2022-03-13 17:14:40,369] jepsen worker 20 - jepsen.util 3260 :invoke :write [6 6] INFO [2022-03-13 17:14:40,386] jepsen worker 21 - jepsen.util 831 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,387] jepsen worker 21 - jepsen.util 831 :invoke :write [7 36] INFO [2022-03-13 17:14:40,393] jepsen worker 71 - jepsen.util 521 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,394] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:40,404] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,404] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:14:40,453] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,453] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,454] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:40,455] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:40,458] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,458] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:14:40,481] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,481] jepsen worker 33 - jepsen.util 663 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,481] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,481] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,482] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:14:40,487] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:14:40,491] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:14:40,495] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:40,496] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,497] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:14:40,512] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,512] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,513] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:40,514] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:14:40,533] jepsen worker 52 - jepsen.util 592 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,534] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:40,538] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,539] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:14:40,560] jepsen worker 43 - jepsen.util 583 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,561] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:40,586] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,587] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,590] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,590] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:40,617] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,618] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:14:40,619] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,620] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,621] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,622] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:14:40,626] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,627] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:40,640] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,640] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:40,668] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,669] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:14:40,713] jepsen worker 77 - jepsen.util 707 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,714] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:40,714] jepsen worker 77 - jepsen.util 707 :invoke :write [25 0] INFO [2022-03-13 17:14:40,715] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:40,715] jepsen worker 18 - jepsen.util 2628 :info :write [6 32] :RequestTimeoutException INFO [2022-03-13 17:14:40,716] jepsen worker 18 - jepsen.util 2718 :invoke :read [6 nil] INFO [2022-03-13 17:14:40,730] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,731] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,734] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,734] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,739] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,740] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:14:40,749] jepsen worker 13 - jepsen.util 643 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,750] jepsen worker 13 - jepsen.util 643 :invoke :write [4 6] INFO [2022-03-13 17:14:40,751] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,752] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:40,779] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,779] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,779] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,780] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:40,781] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,782] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:14:40,804] jepsen worker 21 - jepsen.util 831 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,804] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,804] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:40,805] jepsen worker 14 - jepsen.util 374 :invoke :write [4 29] INFO [2022-03-13 17:14:40,829] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,830] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,831] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,832] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,833] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,833] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:14:40,865] jepsen worker 26 - jepsen.util 476 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,866] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:40,870] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,871] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:14:40,875] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,875] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,877] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,877] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 14] INFO [2022-03-13 17:14:40,880] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,881] jepsen worker 22 - jepsen.util 652 :invoke :write [7 37] INFO [2022-03-13 17:14:40,910] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,911] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:14:40,942] jepsen worker 36 - jepsen.util 2196 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,943] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:14:40,950] jepsen worker 73 - jepsen.util 3043 :info :write [24 14] :RequestTimeoutException INFO [2022-03-13 17:14:40,951] jepsen worker 73 - jepsen.util 3133 :invoke :read [24 nil] INFO [2022-03-13 17:14:40,960] jepsen worker 31 - jepsen.util 3001 :info :write [10 41] :RequestTimeoutException INFO [2022-03-13 17:14:40,961] jepsen worker 31 - jepsen.util 3091 :invoke :write [10 45] INFO [2022-03-13 17:14:40,976] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,976] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,976] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,976] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,976] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:40,977] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:40,978] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,978] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:14:40,979] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,980] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:40,983] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,984] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:14:40,987] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,987] jepsen worker 69 - jepsen.util 699 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,988] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,988] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,987] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:40,989] jepsen worker 51 - jepsen.util 591 :invoke :write [17 0] INFO [2022-03-13 17:14:40,990] jepsen worker 61 - jepsen.util 691 :invoke :write [20 33] INFO [2022-03-13 17:14:40,991] jepsen worker 69 - jepsen.util 699 :invoke :write [23 27] INFO [2022-03-13 17:14:40,993] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:40,994] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:41,000] jepsen worker 34 - jepsen.util 754 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,000] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:41,018] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,018] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,021] jepsen worker 29 - jepsen.util 2999 :info :write [9 47] :RequestTimeoutException INFO [2022-03-13 17:14:41,021] jepsen worker 29 - jepsen.util 3089 :invoke :read [9 nil] INFO [2022-03-13 17:14:41,023] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,027] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,029] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,029] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,030] jepsen worker 45 - jepsen.util 3015 :info :write [15 5] :RequestTimeoutException INFO [2022-03-13 17:14:41,031] jepsen worker 45 - jepsen.util 3105 :invoke :write [15 19] INFO [2022-03-13 17:14:41,031] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,032] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,034] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,034] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:41,036] jepsen worker 15 - jepsen.util 2985 :info :write [5 33] :RequestTimeoutException INFO [2022-03-13 17:14:41,037] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,037] jepsen worker 15 - jepsen.util 3075 :invoke :write [5 41] INFO [2022-03-13 17:14:41,037] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:41,038] jepsen worker 1 - jepsen.util 2611 :info :write [0 11] :RequestTimeoutException INFO [2022-03-13 17:14:41,038] jepsen worker 1 - jepsen.util 2701 :invoke :read [0 nil] INFO [2022-03-13 17:14:41,040] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,040] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,040] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:14:41,041] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:41,061] jepsen worker 28 - jepsen.util 2908 :info :write [9 42] :RequestTimeoutException INFO [2022-03-13 17:14:41,062] jepsen worker 28 - jepsen.util 2998 :invoke :write [9 11] INFO [2022-03-13 17:14:41,063] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,063] jepsen worker 14 - jepsen.util 374 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,063] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,064] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,065] jepsen worker 35 - jepsen.util 935 :invoke :write [11 22] INFO [2022-03-13 17:14:41,066] jepsen worker 14 - jepsen.util 374 :invoke :write [4 13] INFO [2022-03-13 17:14:41,079] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:14:41,079] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:14:41,091] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:41,091] jepsen worker nemesis - couchbase.util Rebalance check count: 390 INFO [2022-03-13 17:14:41,091] jepsen worker nemesis - couchbase.util Rebalance stuck count: 290 INFO [2022-03-13 17:14:41,095] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,096] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:14:41,100] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,100] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,100] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,102] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:14:41,102] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:41,103] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:41,136] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,137] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,138] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,139] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:14:41,157] jepsen worker 70 - jepsen.util 700 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,158] jepsen worker 70 - jepsen.util 700 :invoke :write [23 6] INFO [2022-03-13 17:14:41,167] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,168] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:14:41,172] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,173] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:41,214] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,214] jepsen worker 12 - jepsen.util 732 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,214] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 38] INFO [2022-03-13 17:14:41,215] jepsen worker 42 - jepsen.util 402 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,215] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:41,215] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:14:41,217] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,217] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,218] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,218] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,218] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,218] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,219] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:14:41,220] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:14:41,220] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:41,221] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:14:41,221] jepsen worker 22 - jepsen.util 652 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,222] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:41,224] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,224] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:14:41,263] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,264] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:41,290] jepsen worker 46 - jepsen.util 2836 :info :write [15 31] :RequestTimeoutException INFO [2022-03-13 17:14:41,291] jepsen worker 46 - jepsen.util 2926 :invoke :read [15 nil] INFO [2022-03-13 17:14:41,300] jepsen worker 14 - jepsen.util 374 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,300] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:14:41,304] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,305] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,306] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,307] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,308] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,309] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,310] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,311] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:14:41,314] jepsen worker 11 - jepsen.util 2801 :info :write [3 20] :RequestTimeoutException INFO [2022-03-13 17:14:41,315] jepsen worker 11 - jepsen.util 2891 :invoke :write [3 43] INFO [2022-03-13 17:14:41,328] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,328] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,329] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:14:41,330] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:14:41,342] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,342] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,344] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,344] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,345] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,346] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,347] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,347] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,348] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,349] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,349] jepsen worker 80 - jepsen.util 710 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,349] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,350] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,350] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:41,351] jepsen worker 78 - jepsen.util 528 :invoke :write [26 34] INFO [2022-03-13 17:14:41,351] jepsen worker 52 - jepsen.util 592 :invoke :write [17 44] INFO [2022-03-13 17:14:41,374] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,374] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,375] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,376] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:14:41,377] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,377] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:14:41,389] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,389] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:14:41,396] jepsen worker 68 - jepsen.util 3038 :info :write [22 21] :RequestTimeoutException INFO [2022-03-13 17:14:41,397] jepsen worker 68 - jepsen.util 3128 :invoke :write [22 1] INFO [2022-03-13 17:14:41,405] jepsen worker 35 - jepsen.util 935 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,406] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:41,431] jepsen worker 23 - jepsen.util 563 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,431] jepsen worker 53 - jepsen.util 503 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,432] jepsen worker 23 - jepsen.util 563 :invoke :write [7 36] INFO [2022-03-13 17:14:41,433] jepsen worker 53 - jepsen.util 503 :invoke :write [17 22] INFO [2022-03-13 17:14:41,434] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:41,434] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:41,448] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,449] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,457] jepsen worker 13 - jepsen.util 643 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,458] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:41,474] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,474] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,475] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:14:41,475] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:41,483] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,483] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,484] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:14:41,484] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:41,514] jepsen worker 37 - jepsen.util 2017 :info :write [12 29] :SyncWriteAmbiguous INFO [2022-03-13 17:14:41,515] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,516] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:41,516] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,517] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,517] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:14:41,520] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,521] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:41,548] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,548] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,548] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:41,549] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:41,550] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,551] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:14:41,554] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,554] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,554] jepsen worker 6 - jepsen.util 636 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,555] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:41,555] jepsen worker 7 - jepsen.util 637 :invoke :write [2 9] INFO [2022-03-13 17:14:41,556] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:41,577] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,577] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,578] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,579] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:41,580] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:41,580] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:14:41,594] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,595] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:41,619] jepsen worker 77 - jepsen.util 707 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,619] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,619] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,620] jepsen worker 60 - jepsen.util 600 :invoke :write [20 39] INFO [2022-03-13 17:14:41,620] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:14:41,621] jepsen worker 76 - jepsen.util 706 :invoke :write [25 5] INFO [2022-03-13 17:14:41,630] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,631] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:41,633] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,633] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:41,634] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,635] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:41,635] jepsen worker 70 - jepsen.util 700 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,635] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,637] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,637] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,639] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:14:41,640] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:41,641] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 48] INFO [2022-03-13 17:14:41,644] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,644] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:14:41,650] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,670] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,671] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,671] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,671] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:41,672] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:14:41,672] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:14:41,673] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,673] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,675] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:41,675] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,677] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,678] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,679] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,680] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,684] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,685] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:41,688] jepsen worker 3 - jepsen.util 1893 :info :write [1 29] :SyncWriteAmbiguous INFO [2022-03-13 17:14:41,690] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:14:41,719] jepsen worker 8 - jepsen.util 728 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,719] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:41,719] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,720] jepsen worker 43 - jepsen.util 583 :invoke :write [14 39] INFO [2022-03-13 17:14:41,721] jepsen worker 26 - jepsen.util 476 :invoke :write [8 43] INFO [2022-03-13 17:14:41,721] jepsen worker 8 - jepsen.util 728 :invoke :write [2 15] INFO [2022-03-13 17:14:41,738] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,738] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,738] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,739] jepsen worker 37 - jepsen.util 2107 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,739] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:14:41,740] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:14:41,740] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:41,741] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,748] jepsen worker 49 - jepsen.util 2569 :info :write [16 33] :RequestTimeoutException INFO [2022-03-13 17:14:41,748] jepsen worker 49 - jepsen.util 2659 :invoke :write [16 1] INFO [2022-03-13 17:14:41,756] jepsen worker 78 - jepsen.util 528 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,756] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:41,759] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,760] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:41,767] jepsen worker 89 - jepsen.util 2429 :info :write [29 36] :SyncWriteAmbiguous INFO [2022-03-13 17:14:41,768] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,768] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:14:41,769] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:41,783] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,784] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:14:41,788] jepsen worker 51 - jepsen.util 591 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,788] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:41,790] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,791] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:14:41,810] jepsen worker 66 - jepsen.util 2766 :info :write [22 38] :RequestTimeoutException INFO [2022-03-13 17:14:41,811] jepsen worker 66 - jepsen.util 2856 :invoke :read [22 nil] INFO [2022-03-13 17:14:41,871] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,871] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,872] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,873] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:14:41,874] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,874] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,874] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:41,875] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,875] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,876] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,876] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:14:41,876] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,876] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:41,877] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:41,878] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,878] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,878] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:14:41,880] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:14:41,902] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,902] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:41,934] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,935] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 46] INFO [2022-03-13 17:14:41,976] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,977] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:41,994] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:41,995] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:14:42,046] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,047] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:14:42,078] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,078] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,079] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,080] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,084] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,085] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:14:42,098] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:42,098] jepsen worker nemesis - couchbase.util Rebalance check count: 391 INFO [2022-03-13 17:14:42,098] jepsen worker nemesis - couchbase.util Rebalance stuck count: 291 INFO [2022-03-13 17:14:42,110] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,110] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,110] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:14:42,111] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:14:42,125] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,125] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,126] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,127] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,164] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,165] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,167] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,168] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:14:42,171] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,172] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:14:42,195] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,195] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,196] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:42,196] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 18] INFO [2022-03-13 17:14:42,197] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,198] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:14:42,227] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,228] jepsen worker 14 - jepsen.util 374 :invoke :write [4 24] INFO [2022-03-13 17:14:42,247] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,249] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:14:42,255] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,255] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:42,263] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,264] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:14:42,266] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,266] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:42,303] jepsen worker 52 - jepsen.util 592 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,303] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,303] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,304] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,304] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:42,304] jepsen worker 13 - jepsen.util 643 :invoke :write [4 40] INFO [2022-03-13 17:14:42,305] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:42,305] jepsen worker 34 - jepsen.util 754 :invoke :write [11 38] INFO [2022-03-13 17:14:42,306] jepsen worker 3 - jepsen.util 1983 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,307] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,308] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,309] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:14:42,312] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,312] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,330] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,331] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:42,332] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,332] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,332] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,333] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:42,334] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:14:42,337] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,337] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,337] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,337] jepsen worker 89 - jepsen.util 2519 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,337] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:14:42,337] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 45] INFO [2022-03-13 17:14:42,338] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,338] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 37] INFO [2022-03-13 17:14:42,339] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:14:42,339] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,340] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,341] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,342] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:14:42,348] jepsen worker 79 - jepsen.util 1069 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,349] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 6] INFO [2022-03-13 17:14:42,356] jepsen worker 69 - jepsen.util 699 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,357] jepsen worker 69 - jepsen.util 699 :invoke :write [23 2] INFO [2022-03-13 17:14:42,378] jepsen worker 82 - jepsen.util 2872 :info :write [27 1] :RequestTimeoutException INFO [2022-03-13 17:14:42,379] jepsen worker 82 - jepsen.util 2962 :invoke :read [27 nil] INFO [2022-03-13 17:14:42,379] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,380] jepsen worker 2 - jepsen.util 3242 :info :write [0 18] :RequestTimeoutException INFO [2022-03-13 17:14:42,381] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:14:42,381] jepsen worker 2 - jepsen.util 3332 :invoke :read [0 nil] INFO [2022-03-13 17:14:42,395] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,395] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:42,404] jepsen worker 39 - jepsen.util 2829 :info :write [13 15] :RequestTimeoutException INFO [2022-03-13 17:14:42,404] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,404] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,404] jepsen worker 39 - jepsen.util 2919 :invoke :write [13 43] INFO [2022-03-13 17:14:42,405] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:14:42,405] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:42,407] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,407] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:42,446] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,447] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 1] INFO [2022-03-13 17:14:42,455] jepsen worker 89 - jepsen.util 2519 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,455] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 47] INFO [2022-03-13 17:14:42,482] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,482] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,482] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:14:42,483] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:42,495] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,495] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,495] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,495] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 49] INFO [2022-03-13 17:14:42,496] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,496] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:42,497] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,498] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,502] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,502] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,502] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:14:42,503] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:42,504] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,504] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:14:42,515] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,516] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,520] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,521] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,525] jepsen worker 32 - jepsen.util 2822 :info :write [10 39] :SyncWriteAmbiguous INFO [2022-03-13 17:14:42,526] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,527] jepsen worker 32 - jepsen.util 2912 :invoke :read [10 nil] INFO [2022-03-13 17:14:42,528] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,529] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,530] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:14:42,533] jepsen worker 61 - jepsen.util 691 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,533] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:42,533] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,533] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,534] jepsen worker 61 - jepsen.util 691 :invoke :write [20 47] INFO [2022-03-13 17:14:42,535] jepsen worker 44 - jepsen.util 584 :invoke :write [14 31] INFO [2022-03-13 17:14:42,546] jepsen worker 12 - jepsen.util 732 :invoke :write [4 37] INFO [2022-03-13 17:14:42,547] jepsen worker 42 - jepsen.util 402 :invoke :write [14 22] INFO [2022-03-13 17:14:42,562] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,562] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,563] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:14:42,563] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:42,569] jepsen worker 14 - jepsen.util 374 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,569] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,569] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,570] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:42,570] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:42,571] jepsen worker 35 - jepsen.util 935 :invoke :write [11 22] INFO [2022-03-13 17:14:42,590] jepsen worker 3 - jepsen.util 1983 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,590] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,592] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,593] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,597] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,598] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:14:42,617] jepsen worker 88 - jepsen.util 2068 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,617] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:42,624] jepsen worker 37 - jepsen.util 2107 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,624] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,625] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,626] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:14:42,635] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,635] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,635] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,636] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:42,636] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:14:42,637] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:42,651] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,651] jepsen worker 53 - jepsen.util 503 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,651] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,651] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,651] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,651] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,652] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:42,653] jepsen worker 25 - jepsen.util 565 :invoke :write [8 15] INFO [2022-03-13 17:14:42,654] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:14:42,655] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:42,655] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:42,655] jepsen worker 21 - jepsen.util 831 :invoke :write [7 44] INFO [2022-03-13 17:14:42,656] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,656] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:14:42,706] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,707] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:14:42,712] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,713] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:14:42,718] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,718] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,718] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,718] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:42,719] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:14:42,719] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:42,721] jepsen worker 23 - jepsen.util 563 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,722] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:42,742] jepsen worker 35 - jepsen.util 935 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,743] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:14:42,749] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,749] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:42,754] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,755] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,756] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,756] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:14:42,783] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,783] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,784] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:42,785] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:14:42,803] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,803] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,803] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,804] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:42,804] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:42,805] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:14:42,819] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,819] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 19] INFO [2022-03-13 17:14:42,829] jepsen worker 50 - jepsen.util 3020 :ok :read [16 :nil] INFO [2022-03-13 17:14:42,830] jepsen worker 50 - jepsen.util 3020 :invoke :read [16 nil] INFO [2022-03-13 17:14:42,836] jepsen worker 25 - jepsen.util 565 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,836] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:14:42,847] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,847] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,848] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:14:42,848] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:14:42,856] jepsen worker 85 - jepsen.util 2605 :info :write [28 0] :RequestTimeoutException INFO [2022-03-13 17:14:42,856] jepsen worker 85 - jepsen.util 2695 :invoke :write [28 22] INFO [2022-03-13 17:14:42,860] jepsen worker 7 - jepsen.util 637 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,860] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:42,862] jepsen worker 41 - jepsen.util 2831 :ok :read [13 39] INFO [2022-03-13 17:14:42,863] jepsen worker 41 - jepsen.util 2831 :invoke :write [13 33] INFO [2022-03-13 17:14:42,868] jepsen worker 89 - jepsen.util 2519 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,871] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,871] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,873] jepsen worker 59 - jepsen.util 2849 :ok :read [19 25] INFO [2022-03-13 17:14:42,873] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,873] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:14:42,874] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:14:42,878] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,883] jepsen worker 58 - jepsen.util 2578 :ok :read [19 25] INFO [2022-03-13 17:14:42,887] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,888] jepsen worker 58 - jepsen.util 2578 :invoke :write [19 16] INFO [2022-03-13 17:14:42,888] jepsen worker 59 - jepsen.util 2849 :invoke :read [19 nil] INFO [2022-03-13 17:14:42,891] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,892] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,893] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 37] INFO [2022-03-13 17:14:42,895] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:14:42,901] jepsen worker 21 - jepsen.util 831 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,901] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:42,902] jepsen worker 50 - jepsen.util 3020 :ok :read [16 :nil] INFO [2022-03-13 17:14:42,903] jepsen worker 50 - jepsen.util 3020 :invoke :read [16 nil] INFO [2022-03-13 17:14:42,908] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,908] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,908] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:14:42,909] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:14:42,910] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,911] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:42,919] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,920] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,921] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,921] jepsen worker 60 - jepsen.util 600 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,922] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:42,923] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:42,923] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,924] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:42,925] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,926] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:14:42,935] jepsen worker 16 - jepsen.util 2716 :ok :read [5 43] INFO [2022-03-13 17:14:42,936] jepsen worker 16 - jepsen.util 2716 :invoke :write [5 44] INFO [2022-03-13 17:14:42,960] jepsen worker 50 - jepsen.util 3020 :ok :read [16 :nil] INFO [2022-03-13 17:14:42,961] jepsen worker 50 - jepsen.util 3020 :invoke :write [16 19] INFO [2022-03-13 17:14:42,966] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,967] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:42,970] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,970] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,971] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:42,971] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:14:42,975] jepsen worker 59 - jepsen.util 2849 :ok :read [19 25] INFO [2022-03-13 17:14:42,976] jepsen worker 59 - jepsen.util 2849 :invoke :write [19 31] INFO [2022-03-13 17:14:42,980] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,981] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:14:42,991] jepsen worker 3 - jepsen.util 1983 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:42,991] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:42,999] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,000] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:14:43,015] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,015] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:43,029] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,030] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,030] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,030] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:43,030] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,031] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:43,031] jepsen worker 52 - jepsen.util 592 :invoke :write [17 21] INFO [2022-03-13 17:14:43,031] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:43,034] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,034] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,035] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:43,036] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:43,038] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,038] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:43,086] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,086] jepsen worker 43 - jepsen.util 583 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,086] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,087] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,087] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:43,088] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:43,088] jepsen worker 77 - jepsen.util 707 :invoke :write [25 6] INFO [2022-03-13 17:14:43,091] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:14:43,104] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:43,104] jepsen worker nemesis - couchbase.util Rebalance check count: 392 INFO [2022-03-13 17:14:43,104] jepsen worker nemesis - couchbase.util Rebalance stuck count: 292 INFO [2022-03-13 17:14:43,108] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,108] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,109] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:14:43,110] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:14:43,125] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,125] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,126] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:43,127] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:14:43,137] jepsen worker 2 - jepsen.util 3332 :ok :read [0 41] INFO [2022-03-13 17:14:43,138] jepsen worker 2 - jepsen.util 3332 :invoke :read [0 nil] INFO [2022-03-13 17:14:43,145] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,145] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,145] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,146] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,147] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 46] INFO [2022-03-13 17:14:43,147] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:14:43,148] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 28] INFO [2022-03-13 17:14:43,156] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:43,166] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,166] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:14:43,200] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,200] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,201] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,201] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:14:43,202] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:43,202] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:14:43,208] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,208] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,226] jepsen worker 37 - jepsen.util 2107 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,227] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:43,241] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,241] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,242] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,242] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:14:43,243] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:43,244] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:43,245] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,245] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,247] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,247] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:14:43,263] jepsen worker 26 - jepsen.util 476 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,263] jepsen worker 76 - jepsen.util 706 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,264] jepsen worker 26 - jepsen.util 476 :invoke :write [8 3] INFO [2022-03-13 17:14:43,264] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:43,267] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,267] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,268] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:14:43,268] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:43,303] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,304] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:14:43,314] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,315] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:14:43,330] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,331] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,331] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,331] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:43,331] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:43,332] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,332] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:14:43,332] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:14:43,363] jepsen worker 89 - jepsen.util 2519 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,363] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,363] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,364] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 34] INFO [2022-03-13 17:14:43,364] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:14:43,369] jepsen worker 8 - jepsen.util 728 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,369] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:43,374] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:43,374] jepsen worker 2 - jepsen.util 3332 :ok :read [0 41] INFO [2022-03-13 17:14:43,375] jepsen worker 2 - jepsen.util 3332 :invoke :read [0 nil] INFO [2022-03-13 17:14:43,376] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,376] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 37] INFO [2022-03-13 17:14:43,380] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,380] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,380] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,381] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:43,381] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:43,382] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:43,390] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,390] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,391] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:43,392] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:14:43,411] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,412] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,422] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,423] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,425] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,426] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:14:43,436] jepsen worker 36 - jepsen.util 2196 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,436] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:43,448] jepsen worker 69 - jepsen.util 699 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,449] jepsen worker 69 - jepsen.util 699 :invoke :write [23 19] INFO [2022-03-13 17:14:43,449] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,449] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,449] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,449] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,449] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,449] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,450] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:14:43,451] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:43,451] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:14:43,454] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:14:43,454] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:14:43,454] jepsen worker 71 - jepsen.util 521 :invoke :write [23 27] INFO [2022-03-13 17:14:43,456] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,457] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:14:43,494] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,495] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,498] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,499] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,499] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,499] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 28] INFO [2022-03-13 17:14:43,499] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:14:43,500] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:43,533] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,534] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:14:43,552] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,553] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:43,571] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,571] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,573] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,574] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,576] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,577] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:14:43,578] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,578] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,578] jepsen worker 42 - jepsen.util 402 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,578] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,578] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,578] jepsen worker 6 - jepsen.util 636 :invoke :write [2 44] INFO [2022-03-13 17:14:43,584] jepsen worker 62 - jepsen.util 512 :invoke :write [20 41] INFO [2022-03-13 17:14:43,585] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:43,586] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:43,586] jepsen worker 42 - jepsen.util 402 :invoke :write [14 43] INFO [2022-03-13 17:14:43,588] jepsen worker 2 - jepsen.util 3332 :ok :read [0 41] INFO [2022-03-13 17:14:43,589] jepsen worker 2 - jepsen.util 3332 :invoke :read [0 nil] INFO [2022-03-13 17:14:43,598] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,598] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,601] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:14:43,601] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:43,615] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,615] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:43,620] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,620] jepsen worker 89 - jepsen.util 2519 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,620] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,621] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:43,621] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:14:43,622] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 47] INFO [2022-03-13 17:14:43,636] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,636] jepsen worker 33 - jepsen.util 663 :invoke :write [11 45] INFO [2022-03-13 17:14:43,649] jepsen worker 44 - jepsen.util 584 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,650] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:43,666] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,666] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,667] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:43,668] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:14:43,672] jepsen worker 3 - jepsen.util 1983 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,674] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 28] INFO [2022-03-13 17:14:43,677] jepsen worker 71 - jepsen.util 521 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,678] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:43,695] jepsen worker 37 - jepsen.util 2107 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,695] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:43,708] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,709] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:14:43,736] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,737] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:14:43,742] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,742] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,743] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:14:43,743] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:14:43,750] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,750] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:14:43,782] jepsen worker 2 - jepsen.util 3332 :ok :read [0 41] INFO [2022-03-13 17:14:43,783] jepsen worker 2 - jepsen.util 3332 :invoke :write [0 27] INFO [2022-03-13 17:14:43,795] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,796] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:14:43,822] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,822] jepsen worker 33 - jepsen.util 663 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,822] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:43,823] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:14:43,836] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:43,836] jepsen worker 61 - jepsen.util 691 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,836] jepsen worker 79 - jepsen.util 1069 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,837] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 7] INFO [2022-03-13 17:14:43,837] jepsen worker 61 - jepsen.util 691 :invoke :write [20 15] INFO [2022-03-13 17:14:43,838] jepsen worker 43 - jepsen.util 583 :invoke :write [14 21] INFO [2022-03-13 17:14:43,839] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,840] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:14:43,844] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,845] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,845] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:14:43,846] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:43,854] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,854] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,855] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:14:43,855] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:14:43,864] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,864] jepsen worker 89 - jepsen.util 2519 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,864] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,865] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:43,865] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:43,866] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:14:43,879] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,879] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 28] INFO [2022-03-13 17:14:43,910] jepsen worker 3 - jepsen.util 1983 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,911] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:43,913] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,914] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:14:43,941] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,942] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:14:43,947] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,947] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:43,970] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,970] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,970] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,971] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:14:43,971] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:43,972] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:43,973] jepsen worker 14 - jepsen.util 374 :invoke :write [4 39] INFO [2022-03-13 17:14:43,975] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:44,011] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,012] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,019] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,019] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,021] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,022] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,024] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,025] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:14:44,033] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,034] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,043] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,044] jepsen worker 26 - jepsen.util 476 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,044] jepsen worker 12 - jepsen.util 732 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,044] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,044] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,044] jepsen worker 51 - jepsen.util 591 :invoke :write [17 27] INFO [2022-03-13 17:14:44,045] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,045] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:44,046] jepsen worker 26 - jepsen.util 476 :invoke :write [8 28] INFO [2022-03-13 17:14:44,046] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:44,047] jepsen worker 7 - jepsen.util 637 :invoke :write [2 24] INFO [2022-03-13 17:14:44,047] jepsen worker 53 - jepsen.util 503 :invoke :write [17 40] INFO [2022-03-13 17:14:44,050] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,051] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:44,053] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,054] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:44,094] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,094] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,095] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:44,095] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,110] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:44,110] jepsen worker nemesis - couchbase.util Rebalance check count: 393 INFO [2022-03-13 17:14:44,110] jepsen worker nemesis - couchbase.util Rebalance stuck count: 293 INFO [2022-03-13 17:14:44,115] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,117] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:14:44,126] jepsen worker 67 - jepsen.util 2497 :info :write [22 48] :RequestTimeoutException INFO [2022-03-13 17:14:44,127] jepsen worker 67 - jepsen.util 2587 :invoke :read [22 nil] INFO [2022-03-13 17:14:44,130] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,131] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,138] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,139] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,140] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,140] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:14:44,141] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,142] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,142] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:14:44,147] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:44,176] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,176] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,179] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,180] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:14:44,180] jepsen worker 37 - jepsen.util 2107 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,181] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,202] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,202] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,203] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:14:44,204] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:14:44,212] jepsen worker 6 - jepsen.util 636 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,212] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:44,223] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,224] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,227] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,227] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 9] INFO [2022-03-13 17:14:44,231] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,231] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,231] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,232] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,233] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,233] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,234] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,235] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,235] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,236] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,236] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,237] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,241] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,241] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,241] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:14:44,246] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,247] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,247] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,248] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 43] INFO [2022-03-13 17:14:44,249] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,264] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,264] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,265] jepsen worker 75 - jepsen.util 435 :invoke :write [25 8] INFO [2022-03-13 17:14:44,265] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,265] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:44,266] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:14:44,268] jepsen worker 84 - jepsen.util 3054 :info :write [28 39] :RequestTimeoutException INFO [2022-03-13 17:14:44,269] jepsen worker 84 - jepsen.util 3144 :invoke :write [28 37] INFO [2022-03-13 17:14:44,285] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,285] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,285] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,286] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:44,286] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,287] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:14:44,293] jepsen worker 13 - jepsen.util 643 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,293] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,294] jepsen worker 14 - jepsen.util 374 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,294] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,295] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:44,296] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,296] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,297] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:44,298] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,298] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:44,298] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 48] INFO [2022-03-13 17:14:44,299] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:14:44,299] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,300] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,300] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,300] jepsen worker 77 - jepsen.util 707 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,300] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:44,301] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,301] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:14:44,302] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,302] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:44,303] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:14:44,325] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,325] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:14:44,333] jepsen worker 65 - jepsen.util 3215 :info :write [21 49] :SyncWriteAmbiguous INFO [2022-03-13 17:14:44,334] jepsen worker 65 - jepsen.util 3305 :invoke :write [21 16] INFO [2022-03-13 17:14:44,363] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,363] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,364] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:44,365] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:14:44,369] jepsen worker 89 - jepsen.util 2519 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,370] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,373] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,373] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,374] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:14:44,374] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,386] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,387] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:14:44,421] jepsen worker 42 - jepsen.util 402 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,423] jepsen worker 42 - jepsen.util 402 :invoke :write [14 4] INFO [2022-03-13 17:14:44,442] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,442] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,442] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,443] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,447] jepsen worker 5 - jepsen.util 2435 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,448] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,449] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,450] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:14:44,487] jepsen worker 75 - jepsen.util 435 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,487] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:44,488] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,489] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:44,490] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,490] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:14:44,496] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,496] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,497] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:44,497] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:44,504] jepsen worker 69 - jepsen.util 699 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,505] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,505] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:44,506] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,506] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:44,506] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,506] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,508] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:44,508] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,509] jepsen worker 69 - jepsen.util 699 :invoke :write [23 19] INFO [2022-03-13 17:14:44,510] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:14:44,510] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,511] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,512] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,515] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,516] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,517] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,518] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,520] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,520] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,520] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,520] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,521] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,522] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,522] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:14:44,523] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:14:44,525] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,526] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:14:44,539] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,539] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:44,544] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,544] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,545] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:14:44,546] jepsen worker 24 - jepsen.util 564 :invoke :write [8 4] INFO [2022-03-13 17:14:44,550] jepsen worker 27 - jepsen.util 3357 :info :write [9 47] :RequestTimeoutException INFO [2022-03-13 17:14:44,551] jepsen worker 37 - jepsen.util 2107 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,552] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,552] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,552] jepsen worker 27 - jepsen.util 3447 :invoke :read [9 nil] INFO [2022-03-13 17:14:44,552] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,553] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 14] INFO [2022-03-13 17:14:44,554] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,554] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,554] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 21] INFO [2022-03-13 17:14:44,555] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:14:44,556] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,572] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,572] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:14:44,618] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,618] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,619] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:14:44,619] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,619] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,619] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:14:44,620] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:44,620] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:14:44,621] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,621] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:44,623] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,624] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:14:44,639] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,640] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:44,642] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,642] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:14:44,651] jepsen worker 62 - jepsen.util 512 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,652] jepsen worker 61 - jepsen.util 691 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,652] jepsen worker 62 - jepsen.util 512 :invoke :write [20 19] INFO [2022-03-13 17:14:44,652] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,653] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:44,653] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:44,662] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,662] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,663] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:44,663] jepsen worker 33 - jepsen.util 663 :invoke :write [11 41] INFO [2022-03-13 17:14:44,677] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,678] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:14:44,690] jepsen worker 30 - jepsen.util 2730 :info :write [10 25] :RequestTimeoutException INFO [2022-03-13 17:14:44,690] jepsen worker 30 - jepsen.util 2820 :invoke :read [10 nil] INFO [2022-03-13 17:14:44,708] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,709] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:44,750] jepsen worker 40 - jepsen.util 3010 :info :write [13 49] :RequestTimeoutException INFO [2022-03-13 17:14:44,750] jepsen worker 40 - jepsen.util 3100 :invoke :write [13 38] INFO [2022-03-13 17:14:44,754] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,754] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,788] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,789] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,789] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,789] jepsen worker 79 - jepsen.util 1069 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,789] jepsen worker 51 - jepsen.util 591 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,789] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:44,789] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:44,790] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 43] INFO [2022-03-13 17:14:44,790] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:44,791] jepsen worker 77 - jepsen.util 707 :invoke :write [25 36] INFO [2022-03-13 17:14:44,797] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,798] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:14:44,803] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,803] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,804] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,805] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,812] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,812] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:14:44,861] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,862] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:44,863] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,864] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,864] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,864] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,865] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,865] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:14:44,867] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,868] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:44,869] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,869] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:14:44,870] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,870] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:14:44,889] jepsen worker 24 - jepsen.util 564 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,889] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,890] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:14:44,890] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:14:44,896] jepsen worker 43 - jepsen.util 583 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,896] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,897] jepsen worker 89 - jepsen.util 2519 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,897] jepsen worker 43 - jepsen.util 583 :invoke :write [14 22] INFO [2022-03-13 17:14:44,898] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,898] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 1] INFO [2022-03-13 17:14:44,899] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:44,900] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:44,917] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,918] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:44,920] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,920] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:14:44,947] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:44,947] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,947] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,949] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:14:44,949] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:44,950] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:44,956] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,957] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:14:44,988] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,989] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:44,989] jepsen worker 33 - jepsen.util 663 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:44,990] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:45,006] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,007] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,016] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,017] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:14:45,045] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,045] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,045] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,045] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,046] jepsen worker 14 - jepsen.util 374 :invoke :write [4 30] INFO [2022-03-13 17:14:45,047] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:14:45,047] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:14:45,047] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:14:45,054] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,054] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:45,073] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,075] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:45,083] jepsen worker 37 - jepsen.util 2107 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,084] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,084] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 39] INFO [2022-03-13 17:14:45,085] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:14:45,092] jepsen worker 26 - jepsen.util 476 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,093] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:45,108] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,109] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,113] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,114] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:14:45,116] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:45,117] jepsen worker nemesis - couchbase.util Rebalance check count: 394 INFO [2022-03-13 17:14:45,117] jepsen worker nemesis - couchbase.util Rebalance stuck count: 294 INFO [2022-03-13 17:14:45,123] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,124] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,125] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,126] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:14:45,128] jepsen worker 56 - jepsen.util 3026 :info :write [18 43] :RequestTimeoutException INFO [2022-03-13 17:14:45,128] jepsen worker 56 - jepsen.util 3116 :invoke :read [18 nil] INFO [2022-03-13 17:14:45,144] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,144] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:14:45,158] jepsen worker 14 - jepsen.util 374 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,159] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:45,164] jepsen worker 69 - jepsen.util 699 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,164] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,164] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,164] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,164] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,165] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:45,165] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:45,167] jepsen worker 51 - jepsen.util 591 :invoke :write [17 6] INFO [2022-03-13 17:14:45,167] jepsen worker 61 - jepsen.util 691 :invoke :write [20 47] INFO [2022-03-13 17:14:45,167] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:45,171] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,172] jepsen worker 89 - jepsen.util 2519 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,172] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:45,173] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 48] INFO [2022-03-13 17:14:45,200] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,200] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:45,219] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,220] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,221] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,222] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:14:45,254] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,254] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,254] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:45,255] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 16] INFO [2022-03-13 17:14:45,274] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,274] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,274] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,275] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,274] jepsen worker 34 - jepsen.util 754 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,275] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:14:45,276] jepsen worker 78 - jepsen.util 528 :invoke :write [26 45] INFO [2022-03-13 17:14:45,276] jepsen worker 70 - jepsen.util 700 :invoke :write [23 6] INFO [2022-03-13 17:14:45,277] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:45,280] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,280] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,281] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:14:45,282] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:14:45,283] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:45,289] jepsen worker 7 - jepsen.util 637 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,290] jepsen worker 7 - jepsen.util 637 :invoke :write [2 10] INFO [2022-03-13 17:14:45,315] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,315] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:45,339] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,340] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:45,365] jepsen worker 77 - jepsen.util 707 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,366] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:45,366] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,366] jepsen worker 77 - jepsen.util 707 :invoke :write [25 49] INFO [2022-03-13 17:14:45,367] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:45,368] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:14:45,389] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,390] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:14:45,403] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,403] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,403] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,404] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:14:45,413] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,413] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,414] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:14:45,414] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:14:45,473] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,474] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,479] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,480] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:14:45,488] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,489] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 14] INFO [2022-03-13 17:14:45,517] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,517] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,518] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:45,518] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:45,539] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,539] jepsen worker 88 - jepsen.util 2068 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,539] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:14:45,540] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:14:45,544] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,545] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:14:45,554] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,554] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,567] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,568] jepsen worker 37 - jepsen.util 2107 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,569] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,569] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:45,582] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,583] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:45,583] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,583] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,588] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,589] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:45,617] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,618] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:45,623] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,624] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:45,631] jepsen worker 53 - jepsen.util 503 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,631] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,632] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,632] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,632] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,632] jepsen worker 53 - jepsen.util 503 :invoke :write [17 47] INFO [2022-03-13 17:14:45,632] jepsen worker 69 - jepsen.util 699 :invoke :write [23 22] INFO [2022-03-13 17:14:45,633] jepsen worker 60 - jepsen.util 600 :invoke :write [20 21] INFO [2022-03-13 17:14:45,634] jepsen worker 76 - jepsen.util 706 :invoke :write [25 44] INFO [2022-03-13 17:14:45,634] jepsen worker 12 - jepsen.util 732 :invoke :write [4 24] INFO [2022-03-13 17:14:45,646] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,647] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:14:45,659] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,660] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:14:45,669] jepsen worker 89 - jepsen.util 2519 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,669] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:45,714] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,714] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,726] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,727] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:14:45,736] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,736] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,737] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:14:45,737] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:45,740] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,741] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:45,743] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,744] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:14:45,744] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,744] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,744] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,744] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,745] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:14:45,745] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:45,746] jepsen worker 21 - jepsen.util 831 :invoke :write [7 29] INFO [2022-03-13 17:14:45,748] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:45,748] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,749] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,749] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:45,750] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:14:45,752] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,753] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:14:45,754] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,755] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:45,806] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,806] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,806] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,807] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 34] INFO [2022-03-13 17:14:45,808] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:14:45,808] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:14:45,815] jepsen worker 36 - jepsen.util 2196 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,816] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:45,818] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,818] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:45,819] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,819] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:14:45,819] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,820] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:45,821] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,821] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 13] INFO [2022-03-13 17:14:45,825] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,825] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:14:45,852] jepsen worker 62 - jepsen.util 512 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,853] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:45,866] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,867] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:45,884] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,884] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:45,886] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,887] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:14:45,900] jepsen worker 51 - jepsen.util 591 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,900] jepsen worker 51 - jepsen.util 591 :invoke :write [17 25] INFO [2022-03-13 17:14:45,908] jepsen worker 87 - jepsen.util 2607 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,908] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:14:45,920] jepsen worker 21 - jepsen.util 831 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,921] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:45,924] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,925] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:45,926] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,927] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:14:45,951] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,951] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,952] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:14:45,953] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 46] INFO [2022-03-13 17:14:45,963] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,963] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,964] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,964] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:45,965] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:14:45,966] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:45,981] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:45,983] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:14:46,006] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,006] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:46,007] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,008] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:14:46,031] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,031] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,034] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,035] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] INFO [2022-03-13 17:14:46,058] jepsen worker 79 - jepsen.util 1069 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,059] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,059] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:46,060] jepsen worker 8 - jepsen.util 728 :invoke :write [2 17] INFO [2022-03-13 17:14:46,064] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,064] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:14:46,067] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,067] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,068] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:14:46,068] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,068] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:46,069] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:46,087] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,088] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,096] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:46,096] jepsen worker 61 - jepsen.util 691 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,097] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:46,098] jepsen worker 44 - jepsen.util 584 :invoke :write [14 11] INFO [2022-03-13 17:14:46,116] jepsen worker 89 - jepsen.util 2519 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,116] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,116] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 11] INFO [2022-03-13 17:14:46,117] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:46,119] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,120] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:14:46,122] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:46,123] jepsen worker nemesis - couchbase.util Rebalance check count: 395 INFO [2022-03-13 17:14:46,123] jepsen worker nemesis - couchbase.util Rebalance stuck count: 295 INFO [2022-03-13 17:14:46,152] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,153] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:14:46,166] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,167] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,167] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:46,167] jepsen worker 14 - jepsen.util 374 :invoke :write [4 37] INFO [2022-03-13 17:14:46,171] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,171] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:14:46,200] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,200] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,201] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:14:46,201] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:46,216] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,217] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,219] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,220] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 5] INFO [2022-03-13 17:14:46,220] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,220] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,221] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,221] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,228] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,229] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:14:46,253] jepsen worker 37 - jepsen.util 2107 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,253] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,254] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:46,254] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:46,284] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,284] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,284] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,284] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:46,285] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:46,285] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:14:46,288] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,288] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,289] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,289] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,290] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,290] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,290] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,290] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,291] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:14:46,292] jepsen worker 43 - jepsen.util 583 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,292] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,292] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,292] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,292] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,292] jepsen worker 43 - jepsen.util 583 :invoke :write [14 23] INFO [2022-03-13 17:14:46,293] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:46,293] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,293] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:14:46,294] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,294] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:14:46,295] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,295] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,296] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 17] INFO [2022-03-13 17:14:46,297] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:14:46,298] jepsen worker 89 - jepsen.util 2519 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,299] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 18] INFO [2022-03-13 17:14:46,305] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,306] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:14:46,370] jepsen worker 3 - jepsen.util 1983 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,371] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:14:46,379] jepsen worker 14 - jepsen.util 374 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,380] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,380] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:14:46,381] jepsen worker 21 - jepsen.util 831 :invoke :write [7 9] INFO [2022-03-13 17:14:46,388] jepsen worker 77 - jepsen.util 707 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,389] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,389] jepsen worker 77 - jepsen.util 707 :invoke :write [25 46] INFO [2022-03-13 17:14:46,390] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:46,394] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,394] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,394] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,396] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:14:46,396] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:14:46,397] jepsen worker 71 - jepsen.util 521 :invoke :write [23 15] WARN [2022-03-13 17:14:46,401] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18022722,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18550775,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9122747,"encode_duration_us":20651,"operation_id":"0xaccd","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10002397}],"total_count":1}} WARN [2022-03-13 17:14:46,401] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11939338,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11939338,"total_server_duration_us":32,"operation_id":"0xb078","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11939695},{"operation_name":"get","last_dispatch_duration_us":11243322,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11243322,"total_server_duration_us":23,"operation_id":"0xb0e8","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":11245167},{"operation_name":"get","last_dispatch_duration_us":10615529,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10615529,"total_server_duration_us":23,"operation_id":"0xb148","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":10616840},{"operation_name":"upsert","last_dispatch_duration_us":131533,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9674699,"total_server_duration_us":193196,"encode_duration_us":20542,"operation_id":"0xb083","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10010763},{"operation_name":"upsert","last_dispatch_duration_us":134576,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9712172,"total_server_duration_us":236448,"encode_duration_us":30639,"operation_id":"0xb086","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10005491},{"operation_name":"upsert","last_dispatch_duration_us":135427,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9666249,"total_server_duration_us":200587,"encode_duration_us":43956,"operation_id":"0xb08f","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10004473},{"operation_name":"upsert","last_dispatch_duration_us":151457,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9735590,"total_server_duration_us":191036,"encode_duration_us":21632,"operation_id":"0xb099","timeout_ms":10000,"last_server_duration_us":24928,"total_duration_us":10002703},{"operation_name":"upsert","last_dispatch_duration_us":169127,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":742264,"total_server_duration_us":93703,"encode_duration_us":19636,"operation_id":"0xb091","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10002167},{"operation_name":"upsert","last_dispatch_duration_us":9173659,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9173659,"total_server_duration_us":9041612,"encode_duration_us":18370,"operation_id":"0xb084","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9185747},{"operation_name":"get","last_dispatch_duration_us":8633972,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8633972,"total_server_duration_us":32,"operation_id":"0xb24e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8635334}],"total_count":25}} INFO [2022-03-13 17:14:46,402] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,403] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,459] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,460] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:46,465] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,466] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:14:46,470] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,471] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:46,474] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,475] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:46,485] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,486] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:14:46,544] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,545] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:46,573] jepsen worker 89 - jepsen.util 2519 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,573] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,574] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,574] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,575] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:46,575] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,576] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:14:46,576] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:14:46,577] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,577] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,578] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,579] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,580] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,580] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,585] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,585] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:14:46,605] jepsen worker 7 - jepsen.util 637 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,605] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,606] jepsen worker 7 - jepsen.util 637 :invoke :write [2 39] INFO [2022-03-13 17:14:46,607] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:46,608] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,609] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:14:46,647] jepsen worker 37 - jepsen.util 2107 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,648] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 25] INFO [2022-03-13 17:14:46,659] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,660] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:46,670] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,670] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,671] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,671] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,682] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,683] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,684] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,685] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:14:46,720] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,720] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:14:46,723] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,723] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,723] jepsen worker 21 - jepsen.util 831 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,723] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,724] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:14:46,725] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:46,725] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:46,726] jepsen worker 21 - jepsen.util 831 :invoke :write [7 35] INFO [2022-03-13 17:14:46,726] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,727] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:46,727] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,728] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,728] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:14:46,730] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:46,739] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,740] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:14:46,748] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,750] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:14:46,766] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,766] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,766] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,767] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:14:46,767] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 37] INFO [2022-03-13 17:14:46,768] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,769] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,770] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,770] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,771] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:14:46,771] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,772] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:14:46,799] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,799] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:14:46,830] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,830] jepsen worker 71 - jepsen.util 521 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,832] jepsen worker 42 - jepsen.util 402 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,832] jepsen worker 76 - jepsen.util 706 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,832] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:14:46,833] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:46,833] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:46,834] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:46,838] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,839] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,842] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,843] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,844] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,844] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:46,847] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,848] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:14:46,858] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,859] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:14:46,876] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,876] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:14:46,891] jepsen worker 37 - jepsen.util 2107 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,892] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 37] INFO [2022-03-13 17:14:46,915] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,917] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:14:46,950] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,951] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 31] INFO [2022-03-13 17:14:46,958] jepsen worker 88 - jepsen.util 2068 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,958] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:46,992] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:46,992] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:14:47,058] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,058] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,066] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,066] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:14:47,094] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,095] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,095] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:47,096] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,128] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:47,128] jepsen worker nemesis - couchbase.util Rebalance check count: 396 INFO [2022-03-13 17:14:47,128] jepsen worker nemesis - couchbase.util Rebalance stuck count: 296 INFO [2022-03-13 17:14:47,143] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,144] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,148] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,148] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,148] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,148] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,149] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:14:47,149] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,150] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,151] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,151] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,152] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,152] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,152] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,153] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:14:47,153] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,154] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,154] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:14:47,156] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,156] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:14:47,221] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,221] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,227] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,227] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,229] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,230] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:14:47,237] jepsen worker 89 - jepsen.util 2519 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,237] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,237] jepsen worker 37 - jepsen.util 2107 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,238] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,238] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:47,238] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,240] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,240] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 12] INFO [2022-03-13 17:14:47,253] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,254] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:14:47,262] jepsen worker 48 - jepsen.util 2838 :ok :read [16 :nil] INFO [2022-03-13 17:14:47,263] jepsen worker 48 - jepsen.util 2838 :invoke :write [16 17] INFO [2022-03-13 17:14:47,305] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,306] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:14:47,321] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,321] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,321] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,321] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:14:47,322] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 29] INFO [2022-03-13 17:14:47,322] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:47,356] jepsen worker 57 - jepsen.util 2847 :info :write [19 45] :RequestTimeoutException INFO [2022-03-13 17:14:47,357] jepsen worker 57 - jepsen.util 2937 :invoke :write [19 44] INFO [2022-03-13 17:14:47,370] jepsen worker 82 - jepsen.util 2962 :ok :read [27 4] INFO [2022-03-13 17:14:47,371] jepsen worker 82 - jepsen.util 2962 :invoke :write [27 30] INFO [2022-03-13 17:14:47,384] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,385] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,391] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,391] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:14:47,408] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,409] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:47,415] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,416] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,458] jepsen worker 17 - jepsen.util 2897 :info :write [5 15] :RequestTimeoutException INFO [2022-03-13 17:14:47,458] jepsen worker 17 - jepsen.util 2987 :invoke :write [5 37] INFO [2022-03-13 17:14:47,463] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,464] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:14:47,487] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,487] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,533] jepsen worker 89 - jepsen.util 2519 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,534] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,539] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,539] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:14:47,559] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,559] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,559] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,559] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,560] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,560] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,562] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,562] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,562] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,563] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,563] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,564] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:14:47,564] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,565] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:14:47,565] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,565] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,567] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,568] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:14:47,581] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,582] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,602] jepsen worker 74 - jepsen.util 3044 :info :write [24 3] :SyncWriteAmbiguous INFO [2022-03-13 17:14:47,602] jepsen worker 74 - jepsen.util 3134 :invoke :read [24 nil] INFO [2022-03-13 17:14:47,637] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,637] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:14:47,662] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,663] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:14:47,695] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,695] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,696] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:47,697] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:14:47,740] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,741] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,743] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,743] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:14:47,754] jepsen worker 37 - jepsen.util 2107 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,754] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,755] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,756] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,758] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,759] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:14:47,769] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,769] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,770] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,771] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:14:47,772] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,773] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,782] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,784] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 0] WARN [2022-03-13 17:14:47,809] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17817961,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17817961,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9036305,"encode_duration_us":21338,"operation_id":"0xab06","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10011014}],"total_count":1}} INFO [2022-03-13 17:14:47,812] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,813] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,815] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,815] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:14:47,845] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,845] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:14:47,850] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,851] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,880] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,880] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:47,903] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,904] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] WARN [2022-03-13 17:14:47,910] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17181928,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17703365,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9289075,"encode_duration_us":21808,"operation_id":"0xab8f","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10007610}],"total_count":1}} INFO [2022-03-13 17:14:47,912] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,912] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:47,918] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,919] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:14:47,919] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,919] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,919] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,920] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,921] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,921] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:14:47,923] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:14:47,927] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:14:47,958] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,958] jepsen worker 53 - jepsen.util 503 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,958] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,959] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:47,960] jepsen worker 62 - jepsen.util 512 :invoke :write [20 32] INFO [2022-03-13 17:14:47,961] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,961] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:47,962] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:14:47,963] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,963] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,963] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,964] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:47,966] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,967] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:47,997] jepsen worker 21 - jepsen.util 831 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:47,998] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] WARN [2022-03-13 17:14:48,002] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16461404,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17564823,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9215288,"encode_duration_us":20123,"operation_id":"0xabcc","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10007701}],"total_count":1}} WARN [2022-03-13 17:14:48,009] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":50481,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":363639,"total_server_duration_us":206012,"encode_duration_us":18650,"operation_id":"0xb2ca","timeout_ms":10000,"last_server_duration_us":37653,"total_duration_us":10010022},{"operation_name":"upsert","last_dispatch_duration_us":15423784,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":15423784,"total_server_duration_us":23813,"encode_duration_us":38468,"operation_id":"0xb092","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10009626},{"operation_name":"upsert","encode_duration_us":18618,"operation_id":"0xb0f6","timeout_ms":10000,"total_duration_us":10007351},{"operation_name":"upsert","last_dispatch_duration_us":9111385,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18283335,"total_server_duration_us":121967,"encode_duration_us":29595,"operation_id":"0xaecd","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10005581},{"operation_name":"upsert","last_dispatch_duration_us":9053719,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9053719,"total_server_duration_us":24154,"encode_duration_us":17098,"operation_id":"0xb155","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10004702},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":21682,"operation_id":"0xb250","timeout_ms":10000,"total_duration_us":10004341},{"operation_name":"upsert","last_dispatch_duration_us":9104558,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9104558,"total_server_duration_us":22637,"encode_duration_us":20968,"operation_id":"0xb3ec","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10002890},{"operation_name":"upsert","last_dispatch_duration_us":9294985,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9294985,"total_server_duration_us":9245337,"encode_duration_us":18687,"operation_id":"0xb164","timeout_ms":10000,"last_server_duration_us":9245337,"total_duration_us":9298833},{"operation_name":"upsert","last_dispatch_duration_us":9195401,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9195401,"total_server_duration_us":9074546,"encode_duration_us":25730,"operation_id":"0xb2cb","timeout_ms":10000,"last_server_duration_us":9074546,"total_duration_us":9201196},{"operation_name":"upsert","last_dispatch_duration_us":1995684,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1995684,"total_server_duration_us":113628,"encode_duration_us":55189,"operation_id":"0xb489","timeout_ms":10000,"last_server_duration_us":113628,"total_duration_us":2561577}],"total_count":70}} WARN [2022-03-13 17:14:48,015] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8624714,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":12866931,"total_server_duration_us":321077,"encode_duration_us":28040,"operation_id":"0xb0ba","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10009997},{"operation_name":"upsert","last_dispatch_duration_us":9053809,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10079140,"total_server_duration_us":9308976,"encode_duration_us":18430,"operation_id":"0xb27b","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10007796},{"operation_name":"upsert","last_dispatch_duration_us":55902,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":427691,"total_server_duration_us":240491,"encode_duration_us":27198,"operation_id":"0xb2a0","timeout_ms":10000,"last_server_duration_us":42525,"total_duration_us":10007140},{"operation_name":"upsert","last_dispatch_duration_us":29380,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":224332,"total_server_duration_us":71057,"encode_duration_us":16842,"operation_id":"0xb156","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10005637},{"operation_name":"upsert","last_dispatch_duration_us":45324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":966000,"total_server_duration_us":291013,"encode_duration_us":31202,"operation_id":"0xb29a","timeout_ms":10000,"last_server_duration_us":28768,"total_duration_us":10005388},{"operation_name":"upsert","last_dispatch_duration_us":7254917,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":12598351,"total_server_duration_us":260868,"encode_duration_us":17519,"operation_id":"0xaf50","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10003951},{"operation_name":"upsert","last_dispatch_duration_us":7196938,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9907704,"total_server_duration_us":280108,"encode_duration_us":27002,"operation_id":"0xb10a","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10001894},{"operation_name":"upsert","last_dispatch_duration_us":8936564,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9462167,"total_server_duration_us":149715,"encode_duration_us":24284,"operation_id":"0xb153","timeout_ms":10000,"last_server_duration_us":73821,"total_duration_us":9476109},{"operation_name":"upsert","last_dispatch_duration_us":9375573,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9375573,"total_server_duration_us":9160840,"encode_duration_us":17024,"operation_id":"0xb151","timeout_ms":10000,"last_server_duration_us":9160840,"total_duration_us":9379377},{"operation_name":"upsert","last_dispatch_duration_us":8982743,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":8982743,"total_server_duration_us":51592,"encode_duration_us":20571,"operation_id":"0xb1a6","timeout_ms":10000,"last_server_duration_us":51592,"total_duration_us":8988851}],"total_count":18}} INFO [2022-03-13 17:14:48,020] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,021] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:14:48,026] jepsen worker 69 - jepsen.util 699 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,027] jepsen worker 69 - jepsen.util 699 :invoke :write [23 44] INFO [2022-03-13 17:14:48,038] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,038] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,038] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,040] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,040] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:14:48,041] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,045] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,046] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,052] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,052] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,052] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:14:48,053] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,054] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,055] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,056] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,057] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:14:48,067] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,067] jepsen worker 37 - jepsen.util 2107 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,068] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:48,069] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:48,098] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,098] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,099] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,099] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:14:48,100] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:14:48,101] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 28] WARN [2022-03-13 17:14:48,102] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17709,"operation_id":"0xb3e0","timeout_ms":10000,"total_duration_us":10010567},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":27711,"operation_id":"0xb0c2","timeout_ms":10000,"total_duration_us":10009298},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":19786,"operation_id":"0xb199","timeout_ms":10000,"total_duration_us":10008397},{"operation_name":"upsert","last_dispatch_duration_us":7340875,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8732585,"total_server_duration_us":387369,"encode_duration_us":38406,"operation_id":"0xaf7c","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10007436},{"operation_name":"upsert","last_dispatch_duration_us":34011,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":860991,"total_server_duration_us":339848,"encode_duration_us":18766,"operation_id":"0xb063","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10005663},{"operation_name":"upsert","last_dispatch_duration_us":9207013,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9569374,"total_server_duration_us":9161120,"encode_duration_us":21407,"operation_id":"0xb02e","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":9652928},{"operation_name":"upsert","last_dispatch_duration_us":9257322,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9387554,"total_server_duration_us":9076183,"encode_duration_us":19235,"operation_id":"0xafd7","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9401233},{"operation_name":"upsert","last_dispatch_duration_us":9195883,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":9195883,"total_server_duration_us":9052231,"encode_duration_us":17461,"operation_id":"0xb152","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9199667},{"operation_name":"get","last_dispatch_duration_us":8720872,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":8720872,"total_server_duration_us":56,"operation_id":"0xb194","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8721062},{"operation_name":"upsert","last_dispatch_duration_us":850644,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":850644,"total_server_duration_us":481182,"encode_duration_us":17728,"operation_id":"0xb63c","timeout_ms":10000,"last_server_duration_us":481182,"total_duration_us":855551}],"total_count":22}} INFO [2022-03-13 17:14:48,112] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,113] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] WARN [2022-03-13 17:14:48,115] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9053809,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10079140,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9308976,"encode_duration_us":18430,"operation_id":"0xb27b","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10007796},{"operation_name":"upsert","last_dispatch_duration_us":9165483,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18150381,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9051343,"encode_duration_us":18716,"operation_id":"0xad04","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10010097}],"total_count":2}} INFO [2022-03-13 17:14:48,116] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,118] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:48,131] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,131] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:14:48,136] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:48,136] jepsen worker nemesis - couchbase.util Rebalance check count: 397 INFO [2022-03-13 17:14:48,136] jepsen worker nemesis - couchbase.util Rebalance stuck count: 297 INFO [2022-03-13 17:14:48,156] jepsen worker 8 - jepsen.util 728 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,156] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,158] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,158] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:48,163] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,163] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:48,177] jepsen worker 86 - jepsen.util 2696 :info :write [28 30] :RequestTimeoutException INFO [2022-03-13 17:14:48,178] jepsen worker 86 - jepsen.util 2786 :invoke :write [28 47] INFO [2022-03-13 17:14:48,178] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,179] jepsen worker 14 - jepsen.util 374 :invoke :write [4 11] INFO [2022-03-13 17:14:48,181] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,181] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,194] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,194] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] WARN [2022-03-13 17:14:48,195] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17014161,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":17014161,"total_server_duration_us":5860,"operation_id":"0xab35","timeout_ms":30000,"last_server_duration_us":5860,"total_duration_us":17016247},{"operation_name":"get","last_dispatch_duration_us":16627440,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16627440,"total_server_duration_us":12055,"operation_id":"0xab74","timeout_ms":30000,"last_server_duration_us":12055,"total_duration_us":16629670},{"operation_name":"get","last_dispatch_duration_us":13339794,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":13339794,"total_server_duration_us":9919,"operation_id":"0xad37","timeout_ms":30000,"last_server_duration_us":9919,"total_duration_us":13342214},{"operation_name":"upsert","last_dispatch_duration_us":153033,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9294658,"total_server_duration_us":72618,"encode_duration_us":28325,"operation_id":"0xaf76","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10008389},{"operation_name":"upsert","last_dispatch_duration_us":13099237,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":13099237,"total_server_duration_us":21324,"encode_duration_us":17927,"operation_id":"0xb23a","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10006152},{"operation_name":"upsert","last_dispatch_duration_us":15501901,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":15883865,"total_server_duration_us":9216893,"encode_duration_us":21607,"operation_id":"0xb093","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10001310},{"operation_name":"upsert","last_dispatch_duration_us":9150503,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9150503,"total_server_duration_us":9056479,"encode_duration_us":24289,"operation_id":"0xb48b","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":9157176},{"operation_name":"get","last_dispatch_duration_us":9080027,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9080027,"total_server_duration_us":9919,"operation_id":"0xaf6f","timeout_ms":30000,"last_server_duration_us":9919,"total_duration_us":9082870},{"operation_name":"get","last_dispatch_duration_us":9067791,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9067791,"total_server_duration_us":43,"operation_id":"0xb467","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9068212},{"operation_name":"get","last_dispatch_duration_us":6492456,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":6492456,"total_server_duration_us":76,"operation_id":"0xb0f0","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":6495922}],"total_count":46}} WARN [2022-03-13 17:14:48,195] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15501901,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15883865,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9216893,"encode_duration_us":21607,"operation_id":"0xb093","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10001310},{"operation_name":"upsert","last_dispatch_duration_us":18019627,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27030301,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9189508,"encode_duration_us":22460,"operation_id":"0xa64d","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10007179},{"operation_name":"upsert","last_dispatch_duration_us":7399708,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9056282,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":8073029,"encode_duration_us":20553,"operation_id":"0xabb8","timeout_ms":10000,"last_server_duration_us":7390165,"total_duration_us":10006806},{"operation_name":"upsert","last_dispatch_duration_us":15448947,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16199050,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9499775,"encode_duration_us":25008,"operation_id":"0xae4a","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10008497}],"total_count":4}} INFO [2022-03-13 17:14:48,206] jepsen worker 83 - jepsen.util 2243 :info :write [27 24] :RequestTimeoutException INFO [2022-03-13 17:14:48,207] jepsen worker 83 - jepsen.util 2333 :invoke :read [27 nil] WARN [2022-03-13 17:14:48,210] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":83986,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":2197649,"total_server_duration_us":86907,"encode_duration_us":19085,"operation_id":"0xb0bc","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10009496},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":24087,"operation_id":"0xb159","timeout_ms":10000,"total_duration_us":10008032},{"operation_name":"upsert","last_dispatch_duration_us":9005132,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":12411265,"total_server_duration_us":125484,"encode_duration_us":18118,"operation_id":"0xaff0","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10007468},{"operation_name":"upsert","last_dispatch_duration_us":9097158,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9097158,"total_server_duration_us":9046921,"encode_duration_us":17797,"operation_id":"0xb1d7","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9101219},{"operation_name":"get","last_dispatch_duration_us":1852621,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1852621,"total_server_duration_us":43,"operation_id":"0xb2bc","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3956787},{"operation_name":"get","last_dispatch_duration_us":1842674,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1842674,"total_server_duration_us":23,"operation_id":"0xb2c1","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":3949396},{"operation_name":"upsert","last_dispatch_duration_us":1295122,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1295122,"total_server_duration_us":43727,"encode_duration_us":21308,"operation_id":"0xb839","timeout_ms":10000,"last_server_duration_us":43727,"total_duration_us":2387239},{"operation_name":"upsert","last_dispatch_duration_us":2250934,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2250934,"total_server_duration_us":90332,"encode_duration_us":18298,"operation_id":"0xb32a","timeout_ms":10000,"last_server_duration_us":90332,"total_duration_us":2364255},{"operation_name":"upsert","last_dispatch_duration_us":2277781,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2277781,"total_server_duration_us":1078241,"encode_duration_us":19104,"operation_id":"0xb90b","timeout_ms":10000,"last_server_duration_us":1078241,"total_duration_us":2306481},{"operation_name":"upsert","last_dispatch_duration_us":2232310,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2232310,"total_server_duration_us":281293,"encode_duration_us":20055,"operation_id":"0xb36d","timeout_ms":10000,"last_server_duration_us":281293,"total_duration_us":2268238}],"total_count":66}} INFO [2022-03-13 17:14:48,214] jepsen worker 51 - jepsen.util 591 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,215] jepsen worker 51 - jepsen.util 591 :invoke :write [17 10] INFO [2022-03-13 17:14:48,231] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,232] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,245] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,246] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:14:48,266] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,267] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:48,275] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,275] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:14:48,292] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,293] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,294] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,295] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,301] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,301] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:14:48,332] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,332] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,332] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,332] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,347] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,332] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:48,347] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,342] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,342] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,347] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:48,347] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:48,342] jepsen worker 37 - jepsen.util 2107 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,347] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:48,348] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:48,349] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,349] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,349] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,349] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,350] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,350] jepsen worker 71 - jepsen.util 521 :invoke :write [23 13] INFO [2022-03-13 17:14:48,351] jepsen worker 33 - jepsen.util 663 :invoke :write [11 24] INFO [2022-03-13 17:14:48,352] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:48,353] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,353] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,354] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,354] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,354] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,356] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,356] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:48,356] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:48,357] jepsen worker 44 - jepsen.util 584 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,357] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 10] INFO [2022-03-13 17:14:48,357] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,357] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,358] jepsen worker 44 - jepsen.util 584 :invoke :write [14 46] INFO [2022-03-13 17:14:48,359] jepsen worker 61 - jepsen.util 691 :invoke :write [20 21] INFO [2022-03-13 17:14:48,362] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:48,363] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,364] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:14:48,366] jepsen worker 43 - jepsen.util 583 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,366] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,367] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,367] jepsen worker 43 - jepsen.util 583 :invoke :write [14 14] INFO [2022-03-13 17:14:48,372] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,373] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,373] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,375] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,376] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,377] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,377] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,377] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,377] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:14:48,380] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,383] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,416] jepsen worker 55 - jepsen.util 2665 :info :write [18 8] :RequestTimeoutException INFO [2022-03-13 17:14:48,417] jepsen worker 55 - jepsen.util 2755 :invoke :write [18 30] INFO [2022-03-13 17:14:48,428] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,429] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 36] INFO [2022-03-13 17:14:48,438] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,439] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,444] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,445] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:14:48,446] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,447] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,448] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,449] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,451] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,452] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,455] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,456] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 18] INFO [2022-03-13 17:14:48,458] jepsen worker 33 - jepsen.util 663 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,459] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,462] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,462] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,462] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,463] jepsen worker 14 - jepsen.util 374 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,463] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,463] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,463] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,463] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:14:48,464] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:14:48,465] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,467] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,467] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,468] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,468] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:14:48,469] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:14:48,471] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,472] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:14:48,472] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:14:48,479] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,480] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:14:48,496] jepsen worker 19 - jepsen.util 2629 :info :write [6 1] :RequestTimeoutException INFO [2022-03-13 17:14:48,497] jepsen worker 19 - jepsen.util 2719 :invoke :read [6 nil] INFO [2022-03-13 17:14:48,513] jepsen worker 77 - jepsen.util 707 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,513] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,513] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,516] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:48,516] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:48,517] jepsen worker 6 - jepsen.util 636 :invoke :write [2 39] INFO [2022-03-13 17:14:48,523] jepsen worker 71 - jepsen.util 521 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,524] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:48,579] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,580] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,580] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,581] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,581] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,581] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,582] jepsen worker 21 - jepsen.util 831 :invoke :write [7 16] INFO [2022-03-13 17:14:48,583] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:14:48,583] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:14:48,584] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:48,586] jepsen worker 9 - jepsen.util 2349 :info :write [3 13] :RequestTimeoutException INFO [2022-03-13 17:14:48,586] jepsen worker 9 - jepsen.util 2439 :invoke :read [3 nil] INFO [2022-03-13 17:14:48,616] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:48,616] jepsen worker 7 - jepsen.util 637 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,616] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,616] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,617] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:48,618] jepsen worker 42 - jepsen.util 402 :invoke :write [14 43] INFO [2022-03-13 17:14:48,618] jepsen worker 53 - jepsen.util 503 :invoke :write [17 34] INFO [2022-03-13 17:14:48,619] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:48,641] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,642] jepsen worker 75 - jepsen.util 435 :invoke :write [25 32] INFO [2022-03-13 17:14:48,661] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,661] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:48,679] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,680] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,680] jepsen worker 12 - jepsen.util 732 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,681] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:48,687] jepsen worker 3 - jepsen.util 1983 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,688] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,689] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,689] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] INFO [2022-03-13 17:14:48,704] jepsen worker 37 - jepsen.util 2107 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,705] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 10] INFO [2022-03-13 17:14:48,740] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,741] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,745] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,746] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,746] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,746] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,747] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,747] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,748] jepsen worker 33 - jepsen.util 663 :invoke :write [11 21] INFO [2022-03-13 17:14:48,748] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,749] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:14:48,749] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,750] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,751] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,788] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,789] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,795] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,796] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:14:48,797] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,797] jepsen worker 89 - jepsen.util 2519 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,798] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:14:48,798] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,799] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,799] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,799] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,800] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,800] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:14:48,801] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,801] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:14:48,801] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,802] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,803] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,803] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,803] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:14:48,805] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 33] INFO [2022-03-13 17:14:48,805] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,808] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,809] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:48,810] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,810] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:14:48,829] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,829] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,829] jepsen worker 62 - jepsen.util 512 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,830] jepsen worker 75 - jepsen.util 435 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,830] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,831] jepsen worker 26 - jepsen.util 476 :invoke :write [8 22] INFO [2022-03-13 17:14:48,831] jepsen worker 8 - jepsen.util 728 :invoke :write [2 40] INFO [2022-03-13 17:14:48,832] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:48,832] jepsen worker 62 - jepsen.util 512 :invoke :write [20 6] INFO [2022-03-13 17:14:48,834] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:14:48,842] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,843] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:48,848] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,849] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,878] jepsen worker 81 - jepsen.util 3321 :ok :read [27 4] INFO [2022-03-13 17:14:48,879] jepsen worker 81 - jepsen.util 3321 :invoke :write [27 26] INFO [2022-03-13 17:14:48,885] jepsen worker 60 - jepsen.util 600 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,885] jepsen worker 60 - jepsen.util 600 :invoke :write [20 41] INFO [2022-03-13 17:14:48,887] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,888] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:14:48,915] jepsen worker 37 - jepsen.util 2107 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,916] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:48,919] jepsen worker 33 - jepsen.util 663 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,919] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,933] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,933] jepsen worker 21 - jepsen.util 831 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,933] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,933] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,933] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:48,934] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:14:48,934] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,935] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,936] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,936] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,936] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:48,937] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:48,938] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,939] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,939] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,939] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:48,940] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:48,940] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,940] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:14:48,941] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:14:48,941] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,941] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,942] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,942] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:14:48,942] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:48,943] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:48,946] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,946] jepsen worker 14 - jepsen.util 374 :invoke :write [4 6] INFO [2022-03-13 17:14:48,984] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:48,985] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:14:49,006] jepsen worker 83 - jepsen.util 2333 :ok :read [27 4] INFO [2022-03-13 17:14:49,007] jepsen worker 83 - jepsen.util 2333 :invoke :read [27 nil] INFO [2022-03-13 17:14:49,022] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,023] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,024] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,025] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 17] INFO [2022-03-13 17:14:49,036] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,036] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,037] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:14:49,037] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:49,040] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,041] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:14:49,042] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,042] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,046] jepsen worker 69 - jepsen.util 699 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,046] jepsen worker 43 - jepsen.util 583 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,049] jepsen worker 69 - jepsen.util 699 :invoke :write [23 42] INFO [2022-03-13 17:14:49,051] jepsen worker 43 - jepsen.util 583 :invoke :write [14 4] INFO [2022-03-13 17:14:49,104] jepsen worker 83 - jepsen.util 2333 :ok :read [27 4] INFO [2022-03-13 17:14:49,105] jepsen worker 83 - jepsen.util 2333 :invoke :read [27 nil] INFO [2022-03-13 17:14:49,107] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,107] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:14:49,135] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,135] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,135] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:14:49,135] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,136] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,136] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:14:49,139] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,140] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,141] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,142] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:49,142] jepsen worker nemesis - couchbase.util Rebalance check count: 398 INFO [2022-03-13 17:14:49,142] jepsen worker nemesis - couchbase.util Rebalance stuck count: 298 INFO [2022-03-13 17:14:49,142] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:14:49,156] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,159] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:49,159] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,160] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,160] jepsen worker 89 - jepsen.util 2519 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,161] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:49,166] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,166] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:49,167] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,167] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,167] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,168] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:49,168] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,169] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:49,169] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:49,170] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:14:49,180] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,182] jepsen worker 6 - jepsen.util 636 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,182] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,182] jepsen worker 51 - jepsen.util 591 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,182] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,182] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:49,183] jepsen worker 6 - jepsen.util 636 :invoke :write [2 43] INFO [2022-03-13 17:14:49,184] jepsen worker 7 - jepsen.util 637 :invoke :write [2 27] INFO [2022-03-13 17:14:49,184] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:49,185] jepsen worker 51 - jepsen.util 591 :invoke :write [17 27] INFO [2022-03-13 17:14:49,189] jepsen worker 0 - jepsen.util 2700 :ok :read [0 41] INFO [2022-03-13 17:14:49,189] jepsen worker 0 - jepsen.util 2700 :invoke :write [0 33] INFO [2022-03-13 17:14:49,211] jepsen worker 83 - jepsen.util 2333 :ok :read [27 4] INFO [2022-03-13 17:14:49,213] jepsen worker 3 - jepsen.util 1983 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,213] jepsen worker 83 - jepsen.util 2333 :invoke :write [27 39] INFO [2022-03-13 17:14:49,215] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:14:49,232] jepsen worker 47 - jepsen.util 2837 :ok :read [15 48] INFO [2022-03-13 17:14:49,233] jepsen worker 47 - jepsen.util 2837 :invoke :read [15 nil] INFO [2022-03-13 17:14:49,244] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,244] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:14:49,247] jepsen worker 29 - jepsen.util 3089 :ok :read [9 26] INFO [2022-03-13 17:14:49,247] jepsen worker 29 - jepsen.util 3089 :invoke :write [9 1] INFO [2022-03-13 17:14:49,256] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,256] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,256] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,257] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,257] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 14] INFO [2022-03-13 17:14:49,258] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:14:49,271] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,271] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,272] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:49,272] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:49,274] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,275] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,275] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 31] INFO [2022-03-13 17:14:49,282] jepsen worker 75 - jepsen.util 435 :invoke :write [25 10] INFO [2022-03-13 17:14:49,289] jepsen worker 56 - jepsen.util 3116 :ok :read [18 21] INFO [2022-03-13 17:14:49,289] jepsen worker 56 - jepsen.util 3116 :invoke :read [18 nil] INFO [2022-03-13 17:14:49,294] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,295] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,334] jepsen worker 47 - jepsen.util 2837 :ok :read [15 48] INFO [2022-03-13 17:14:49,334] jepsen worker 47 - jepsen.util 2837 :invoke :read [15 nil] INFO [2022-03-13 17:14:49,359] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,360] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:14:49,362] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,363] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,363] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,363] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:14:49,367] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:14:49,367] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:14:49,373] jepsen worker 42 - jepsen.util 402 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,374] jepsen worker 42 - jepsen.util 402 :invoke :write [14 22] INFO [2022-03-13 17:14:49,380] jepsen worker 14 - jepsen.util 374 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,381] jepsen worker 14 - jepsen.util 374 :invoke :write [4 33] INFO [2022-03-13 17:14:49,390] jepsen worker 63 - jepsen.util 2853 :info :write [21 46] :RequestTimeoutException INFO [2022-03-13 17:14:49,391] jepsen worker 63 - jepsen.util 2943 :invoke :read [21 nil] INFO [2022-03-13 17:14:49,396] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,397] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:14:49,406] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,407] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,411] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,412] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:14:49,423] jepsen worker 64 - jepsen.util 2764 :info :write [21 1] :RequestTimeoutException INFO [2022-03-13 17:14:49,424] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,424] jepsen worker 44 - jepsen.util 584 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,424] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,424] jepsen worker 64 - jepsen.util 2854 :invoke :write [21 2] INFO [2022-03-13 17:14:49,425] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:49,425] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:14:49,427] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:49,428] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,429] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,429] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,431] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,460] jepsen worker 56 - jepsen.util 3116 :ok :read [18 21] INFO [2022-03-13 17:14:49,461] jepsen worker 56 - jepsen.util 3116 :invoke :write [18 26] INFO [2022-03-13 17:14:49,463] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,463] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,463] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:14:49,465] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:49,498] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,498] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,506] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,507] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,511] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,512] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:14:49,524] jepsen worker 47 - jepsen.util 2837 :ok :read [15 48] INFO [2022-03-13 17:14:49,524] jepsen worker 47 - jepsen.util 2837 :invoke :read [15 nil] INFO [2022-03-13 17:14:49,529] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,529] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,530] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,530] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,530] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,531] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:14:49,532] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,532] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,533] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,535] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:14:49,535] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,536] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:14:49,568] jepsen worker 37 - jepsen.util 2107 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,569] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,569] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 9] INFO [2022-03-13 17:14:49,569] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:49,631] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,631] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:14:49,633] jepsen worker 47 - jepsen.util 2837 :ok :read [15 48] INFO [2022-03-13 17:14:49,633] jepsen worker 47 - jepsen.util 2837 :invoke :write [15 14] INFO [2022-03-13 17:14:49,667] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,667] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:14:49,695] jepsen worker 9 - jepsen.util 2439 :ok :read [3 25] INFO [2022-03-13 17:14:49,695] jepsen worker 9 - jepsen.util 2439 :invoke :write [3 31] INFO [2022-03-13 17:14:49,701] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,701] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:14:49,754] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,754] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:14:49,787] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,788] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:49,790] jepsen worker 89 - jepsen.util 2519 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,790] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,792] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,792] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:49,794] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,795] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,796] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,797] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,798] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,799] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,800] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,801] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:14:49,811] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,811] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,812] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,812] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:14:49,820] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,820] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 15] INFO [2022-03-13 17:14:49,828] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,829] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:49,885] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,886] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:49,888] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,888] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,888] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,888] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:14:49,889] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:14:49,890] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:14:49,892] jepsen worker 37 - jepsen.util 2107 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,892] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:49,899] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,900] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:14:49,935] jepsen worker 75 - jepsen.util 435 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,935] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:14:49,976] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,977] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:14:49,979] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,980] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,980] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:14:49,983] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:14:49,984] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,984] jepsen worker 53 - jepsen.util 503 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:49,985] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:14:49,985] jepsen worker 53 - jepsen.util 503 :invoke :write [17 5] INFO [2022-03-13 17:14:50,005] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,005] jepsen worker 52 - jepsen.util 592 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,005] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,005] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,007] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:50,008] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:50,009] jepsen worker 52 - jepsen.util 592 :invoke :write [17 28] INFO [2022-03-13 17:14:50,009] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:50,027] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,027] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:14:50,048] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,049] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:14:50,056] jepsen worker 61 - jepsen.util 691 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,057] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,058] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:50,059] jepsen worker 61 - jepsen.util 691 :invoke :write [20 2] INFO [2022-03-13 17:14:50,060] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,061] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:14:50,074] jepsen worker 14 - jepsen.util 374 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,074] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,074] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,076] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,077] jepsen worker 14 - jepsen.util 374 :invoke :write [4 9] INFO [2022-03-13 17:14:50,078] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:50,082] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,083] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 10] INFO [2022-03-13 17:14:50,121] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,122] jepsen worker 24 - jepsen.util 564 :invoke :write [8 5] INFO [2022-03-13 17:14:50,149] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:50,150] jepsen worker nemesis - couchbase.util Rebalance check count: 399 INFO [2022-03-13 17:14:50,150] jepsen worker nemesis - couchbase.util Rebalance stuck count: 299 INFO [2022-03-13 17:14:50,155] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,156] jepsen worker 54 - jepsen.util 2484 :info :write [18 45] :RequestTimeoutException INFO [2022-03-13 17:14:50,157] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,157] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:14:50,158] jepsen worker 54 - jepsen.util 2574 :invoke :write [18 37] INFO [2022-03-13 17:14:50,159] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:14:50,179] jepsen worker 89 - jepsen.util 2519 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,181] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 35] INFO [2022-03-13 17:14:50,187] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,188] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:50,215] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,216] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:50,233] jepsen worker 8 - jepsen.util 728 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,234] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:14:50,237] jepsen worker 3 - jepsen.util 1983 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,239] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 36] INFO [2022-03-13 17:14:50,243] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,244] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,244] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,245] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,260] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,260] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,261] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:50,261] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:50,282] jepsen worker 60 - jepsen.util 600 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,282] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,283] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:50,283] jepsen worker 60 - jepsen.util 600 :invoke :write [20 19] INFO [2022-03-13 17:14:50,312] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,313] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:50,333] jepsen worker 14 - jepsen.util 374 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,333] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,333] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,333] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,333] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:50,334] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:14:50,335] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,335] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:14:50,336] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:50,337] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,337] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:50,338] jepsen worker 14 - jepsen.util 374 :invoke :write [4 19] INFO [2022-03-13 17:14:50,338] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,339] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:14:50,339] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,340] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,340] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:14:50,343] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,346] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,347] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:14:50,356] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,356] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,390] jepsen worker 20 - jepsen.util 3260 :info :write [6 6] :RequestTimeoutException INFO [2022-03-13 17:14:50,391] jepsen worker 20 - jepsen.util 3350 :invoke :read [6 nil] INFO [2022-03-13 17:14:50,396] jepsen worker 26 - jepsen.util 476 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,396] jepsen worker 69 - jepsen.util 699 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,397] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:50,398] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:50,417] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,417] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:14:50,446] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,446] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,447] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 20] INFO [2022-03-13 17:14:50,447] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:14:50,460] jepsen worker 3 - jepsen.util 1983 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,460] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 19] INFO [2022-03-13 17:14:50,479] jepsen worker 43 - jepsen.util 583 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,479] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,480] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 11] INFO [2022-03-13 17:14:50,481] jepsen worker 43 - jepsen.util 583 :invoke :write [14 22] INFO [2022-03-13 17:14:50,483] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,483] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,484] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 24] INFO [2022-03-13 17:14:50,485] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,494] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,494] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:14:50,501] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,503] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,519] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,519] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:14:50,521] jepsen worker 51 - jepsen.util 591 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,521] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:50,521] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,522] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:50,523] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:50,523] jepsen worker 51 - jepsen.util 591 :invoke :write [17 35] INFO [2022-03-13 17:14:50,526] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,526] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,526] jepsen worker 24 - jepsen.util 564 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,526] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:50,527] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,527] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:14:50,528] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:50,529] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:14:50,584] jepsen worker 87 - jepsen.util 2607 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,585] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:14:50,586] jepsen worker 3 - jepsen.util 1983 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,586] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,590] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,591] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 19] INFO [2022-03-13 17:14:50,591] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,592] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:14:50,592] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,592] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,592] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:50,593] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:50,599] jepsen worker 89 - jepsen.util 2519 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,599] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,600] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,600] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,600] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,600] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:50,602] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,602] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,602] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,603] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,604] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,604] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,604] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 9] INFO [2022-03-13 17:14:50,605] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:14:50,614] jepsen worker 7 - jepsen.util 637 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,615] jepsen worker 7 - jepsen.util 637 :invoke :write [2 14] INFO [2022-03-13 17:14:50,636] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,636] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,669] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,669] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:14:50,701] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,702] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,707] jepsen worker 37 - jepsen.util 2107 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,708] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,708] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 45] INFO [2022-03-13 17:14:50,708] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,709] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,710] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,711] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,712] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:50,712] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,713] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:14:50,722] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,722] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,722] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,723] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:50,724] jepsen worker 33 - jepsen.util 663 :invoke :write [11 1] INFO [2022-03-13 17:14:50,724] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:14:50,730] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,731] jepsen worker 77 - jepsen.util 707 :invoke :write [25 19] INFO [2022-03-13 17:14:50,737] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,738] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,739] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,740] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,740] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,741] jepsen worker 12 - jepsen.util 732 :invoke :write [4 11] INFO [2022-03-13 17:14:50,778] jepsen worker 89 - jepsen.util 2519 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,779] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,786] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,786] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,787] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:14:50,787] jepsen worker 24 - jepsen.util 564 :invoke :write [8 28] INFO [2022-03-13 17:14:50,802] jepsen worker 6 - jepsen.util 636 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,802] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:50,821] jepsen worker 3 - jepsen.util 1983 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,821] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,827] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,827] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,829] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,829] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,833] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,834] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:14:50,849] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,849] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,849] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,849] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,850] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:14:50,850] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,851] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,851] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,851] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,852] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:14:50,853] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,853] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,855] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,855] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:14:50,870] jepsen worker 14 - jepsen.util 374 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,871] jepsen worker 14 - jepsen.util 374 :invoke :write [4 35] INFO [2022-03-13 17:14:50,880] jepsen worker 33 - jepsen.util 663 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,880] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:50,884] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,885] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,891] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,892] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,892] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,893] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:50,897] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,897] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:14:50,911] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,911] jepsen worker 61 - jepsen.util 691 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,912] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:50,912] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:50,934] jepsen worker 37 - jepsen.util 2107 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,934] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,935] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,935] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:14:50,939] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,939] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,939] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 32] INFO [2022-03-13 17:14:50,940] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,941] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,941] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,941] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:14:50,942] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:50,944] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,944] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:14:50,965] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,966] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:50,984] jepsen worker 31 - jepsen.util 3091 :info :write [10 45] :RequestTimeoutException INFO [2022-03-13 17:14:50,985] jepsen worker 31 - jepsen.util 3181 :invoke :read [10 nil] INFO [2022-03-13 17:14:50,985] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,986] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,989] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,990] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:50,995] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:50,995] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:14:51,022] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,023] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,047] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,047] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:51,049] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,050] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,052] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,052] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,053] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,053] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,053] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:14:51,056] jepsen worker 24 - jepsen.util 564 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,056] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,056] jepsen worker 45 - jepsen.util 3105 :info :write [15 19] :RequestTimeoutException INFO [2022-03-13 17:14:51,058] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:51,059] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:51,060] jepsen worker 15 - jepsen.util 3075 :info :write [5 41] :RequestTimeoutException INFO [2022-03-13 17:14:51,060] jepsen worker 45 - jepsen.util 3195 :invoke :read [15 nil] INFO [2022-03-13 17:14:51,061] jepsen worker 15 - jepsen.util 3165 :invoke :read [5 nil] INFO [2022-03-13 17:14:51,062] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,065] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,066] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:14:51,090] jepsen worker 28 - jepsen.util 2998 :info :write [9 11] :RequestTimeoutException INFO [2022-03-13 17:14:51,091] jepsen worker 28 - jepsen.util 3088 :invoke :read [9 nil] INFO [2022-03-13 17:14:51,131] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,131] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:14:51,153] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,153] jepsen worker 89 - jepsen.util 2519 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,153] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 31] INFO [2022-03-13 17:14:51,154] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,156] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:51,156] jepsen worker nemesis - couchbase.util Rebalance check count: 400 INFO [2022-03-13 17:14:51,157] jepsen worker nemesis - couchbase.util Rebalance stuck count: 300 INFO [2022-03-13 17:14:51,157] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,158] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:14:51,186] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,186] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,186] jepsen worker 53 - jepsen.util 503 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,187] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,187] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,188] jepsen worker 60 - jepsen.util 600 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,188] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,189] jepsen worker 8 - jepsen.util 728 :invoke :write [2 9] INFO [2022-03-13 17:14:51,191] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:51,191] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:51,192] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:51,192] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,192] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,193] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:14:51,194] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,195] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,195] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:14:51,196] jepsen worker 60 - jepsen.util 600 :invoke :write [20 3] INFO [2022-03-13 17:14:51,197] jepsen worker 23 - jepsen.util 563 :invoke :write [7 15] INFO [2022-03-13 17:14:51,199] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:51,200] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:51,202] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,239] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,240] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,240] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,240] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,240] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,240] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,240] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,241] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,242] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,243] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:51,243] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,243] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:14:51,244] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:51,245] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:51,246] jepsen worker 71 - jepsen.util 521 :invoke :write [23 47] INFO [2022-03-13 17:14:51,246] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:51,247] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,247] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:51,248] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,249] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:14:51,259] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,259] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,259] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:14:51,260] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:14:51,278] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,278] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:14:51,300] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,301] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,301] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,301] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,317] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,318] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:14:51,329] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,329] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:51,332] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,332] jepsen worker 14 - jepsen.util 374 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,332] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,333] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:14:51,335] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:51,336] jepsen worker 11 - jepsen.util 2891 :info :write [3 43] :RequestTimeoutException INFO [2022-03-13 17:14:51,337] jepsen worker 11 - jepsen.util 2981 :invoke :write [3 45] INFO [2022-03-13 17:14:51,337] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:51,357] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,358] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,359] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,360] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:14:51,365] jepsen worker 62 - jepsen.util 512 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,365] jepsen worker 62 - jepsen.util 512 :invoke :write [20 2] INFO [2022-03-13 17:14:51,385] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,386] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,392] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,392] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,392] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,393] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:14:51,394] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:14:51,394] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,396] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,396] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:14:51,419] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,419] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:14:51,426] jepsen worker 68 - jepsen.util 3128 :info :write [22 1] :RequestTimeoutException INFO [2022-03-13 17:14:51,426] jepsen worker 71 - jepsen.util 521 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,427] jepsen worker 68 - jepsen.util 3218 :invoke :read [22 nil] INFO [2022-03-13 17:14:51,427] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,482] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,483] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,483] jepsen worker 70 - jepsen.util 700 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,484] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,491] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,491] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,495] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,495] jepsen worker 51 - jepsen.util 591 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,496] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,496] jepsen worker 51 - jepsen.util 591 :invoke :write [17 25] INFO [2022-03-13 17:14:51,498] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,498] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:14:51,504] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,504] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,504] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,504] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:14:51,505] jepsen worker 35 - jepsen.util 935 :invoke :write [11 17] INFO [2022-03-13 17:14:51,509] jepsen worker 14 - jepsen.util 374 :invoke :write [4 37] INFO [2022-03-13 17:14:51,520] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,521] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,525] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,526] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,536] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,537] jepsen worker 24 - jepsen.util 564 :invoke :write [8 46] INFO [2022-03-13 17:14:51,574] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,575] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:14:51,591] jepsen worker 72 - jepsen.util 3042 :ok :read [24 24] INFO [2022-03-13 17:14:51,591] jepsen worker 72 - jepsen.util 3042 :invoke :read [24 nil] INFO [2022-03-13 17:14:51,603] jepsen worker 89 - jepsen.util 2519 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,604] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,604] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,604] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,605] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,605] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:14:51,625] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,625] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,626] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,627] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,634] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,635] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,635] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,636] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,660] jepsen worker 42 - jepsen.util 402 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,660] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:51,662] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,662] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,662] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,662] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,663] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:14:51,664] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,664] jepsen worker 46 - jepsen.util 2926 :ok :read [15 48] INFO [2022-03-13 17:14:51,665] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:14:51,665] jepsen worker 46 - jepsen.util 2926 :invoke :write [15 8] INFO [2022-03-13 17:14:51,665] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,675] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,676] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,676] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 17] INFO [2022-03-13 17:14:51,678] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:51,679] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,680] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,680] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:14:51,681] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:14:51,695] jepsen worker 24 - jepsen.util 564 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,695] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,695] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,695] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:51,696] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,697] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,697] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,697] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:14:51,698] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:51,699] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:14:51,702] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 44] INFO [2022-03-13 17:14:51,702] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,704] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,705] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 24] INFO [2022-03-13 17:14:51,749] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,749] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:14:51,770] jepsen worker 49 - jepsen.util 2659 :info :write [16 1] :RequestTimeoutException INFO [2022-03-13 17:14:51,771] jepsen worker 49 - jepsen.util 2749 :invoke :write [16 47] INFO [2022-03-13 17:14:51,772] jepsen worker 77 - jepsen.util 707 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,772] jepsen worker 77 - jepsen.util 707 :invoke :write [25 37] INFO [2022-03-13 17:14:51,776] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,776] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,777] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,777] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:51,777] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:51,777] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:51,799] jepsen worker 78 - jepsen.util 528 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,799] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,800] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,800] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,800] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,800] jepsen worker 78 - jepsen.util 528 :invoke :write [26 3] INFO [2022-03-13 17:14:51,802] jepsen worker 34 - jepsen.util 754 :invoke :write [11 10] INFO [2022-03-13 17:14:51,802] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:51,803] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:51,803] jepsen worker 13 - jepsen.util 643 :invoke :write [4 28] INFO [2022-03-13 17:14:51,803] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,804] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:51,805] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,805] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:51,806] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,807] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:14:51,838] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,839] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:51,839] jepsen worker 52 - jepsen.util 592 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,839] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,840] jepsen worker 52 - jepsen.util 592 :invoke :write [17 46] INFO [2022-03-13 17:14:51,840] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:51,841] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,842] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:51,847] jepsen worker 3 - jepsen.util 1983 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,848] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:14:51,855] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,855] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:14:51,866] jepsen worker 35 - jepsen.util 935 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,867] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:51,905] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,905] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,905] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,906] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:14:51,906] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:14:51,907] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:14:51,909] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,909] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,910] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,910] jepsen worker 33 - jepsen.util 663 :invoke :write [11 36] INFO [2022-03-13 17:14:51,910] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,910] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:14:51,914] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,914] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:51,918] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:14:51,918] jepsen worker 43 - jepsen.util 583 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,922] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:14:51,929] jepsen worker 37 - jepsen.util 2107 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,929] jepsen worker 43 - jepsen.util 583 :invoke :write [14 22] INFO [2022-03-13 17:14:51,930] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 10] INFO [2022-03-13 17:14:51,961] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:51,962] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:14:52,007] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,007] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:14:52,012] jepsen worker 89 - jepsen.util 2519 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,012] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,035] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,035] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:14:52,040] jepsen worker 14 - jepsen.util 374 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,040] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:52,045] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,045] jepsen worker 78 - jepsen.util 528 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,045] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:14:52,046] jepsen worker 78 - jepsen.util 528 :invoke :write [26 20] INFO [2022-03-13 17:14:52,047] jepsen worker 12 - jepsen.util 732 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,047] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,047] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,047] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:52,047] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,047] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:52,048] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:52,049] jepsen worker 44 - jepsen.util 584 :invoke :write [14 35] INFO [2022-03-13 17:14:52,049] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:52,049] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:52,068] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,068] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,085] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,086] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,093] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,093] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:14:52,105] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,105] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:52,123] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,124] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:14:52,141] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,142] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:14:52,158] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,159] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,161] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,161] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:14:52,163] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:52,163] jepsen worker nemesis - couchbase.util Rebalance check count: 401 INFO [2022-03-13 17:14:52,163] jepsen worker nemesis - couchbase.util Rebalance stuck count: 301 INFO [2022-03-13 17:14:52,187] jepsen worker 34 - jepsen.util 754 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,188] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:52,191] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,191] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,192] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:14:52,193] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:52,194] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,194] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,194] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,194] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:14:52,195] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:52,196] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:52,196] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,197] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:52,197] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,198] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,198] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:14:52,198] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:52,199] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,202] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:14:52,209] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,209] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,210] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,210] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:14:52,222] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,222] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,223] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,223] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:52,223] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 28] INFO [2022-03-13 17:14:52,224] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:14:52,243] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,244] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,250] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,251] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:14:52,264] jepsen worker 79 - jepsen.util 1069 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,264] jepsen worker 60 - jepsen.util 600 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,265] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:52,266] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 3] INFO [2022-03-13 17:14:52,291] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,292] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,292] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,293] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:14:52,302] jepsen worker 37 - jepsen.util 2107 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,302] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 20] INFO [2022-03-13 17:14:52,328] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,328] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:52,331] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,332] jepsen worker 22 - jepsen.util 652 :invoke :write [7 12] INFO [2022-03-13 17:14:52,340] jepsen worker 33 - jepsen.util 663 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,340] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:52,347] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,348] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:14:52,361] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,361] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,361] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,362] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,387] jepsen worker 13 - jepsen.util 643 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,387] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,387] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,389] jepsen worker 13 - jepsen.util 643 :invoke :write [4 26] INFO [2022-03-13 17:14:52,389] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:52,390] jepsen worker 80 - jepsen.util 710 :invoke :write [26 1] INFO [2022-03-13 17:14:52,398] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,399] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:52,425] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,425] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,428] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,428] jepsen worker 39 - jepsen.util 2919 :info :write [13 43] :RequestTimeoutException INFO [2022-03-13 17:14:52,429] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:14:52,429] jepsen worker 39 - jepsen.util 3009 :invoke :read [13 nil] INFO [2022-03-13 17:14:52,471] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,472] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,472] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,472] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,472] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:52,472] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,473] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:14:52,473] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,474] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,474] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:52,474] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,475] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:14:52,475] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,475] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:14:52,499] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,500] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:52,503] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,506] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,507] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:14:52,507] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:52,530] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,530] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,530] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,530] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,532] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:52,532] jepsen worker 24 - jepsen.util 564 :invoke :write [8 46] INFO [2022-03-13 17:14:52,533] jepsen worker 33 - jepsen.util 663 :invoke :write [11 7] INFO [2022-03-13 17:14:52,533] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,533] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:52,534] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:14:52,568] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,569] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:14:52,588] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,588] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:14:52,625] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,625] jepsen worker 52 - jepsen.util 592 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,626] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:52,627] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:52,631] jepsen worker 23 - jepsen.util 563 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,631] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,631] jepsen worker 7 - jepsen.util 637 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,631] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,632] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:14:52,633] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:52,633] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:52,634] jepsen worker 26 - jepsen.util 476 :invoke :write [8 49] INFO [2022-03-13 17:14:52,639] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,640] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:14:52,662] jepsen worker 89 - jepsen.util 2519 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,662] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,662] jepsen worker 37 - jepsen.util 2107 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,663] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 17] INFO [2022-03-13 17:14:52,663] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:52,664] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:14:52,665] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,667] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:14:52,709] jepsen worker 8 - jepsen.util 728 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,709] jepsen worker 8 - jepsen.util 728 :invoke :write [2 16] INFO [2022-03-13 17:14:52,715] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,715] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:14:52,723] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,723] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,723] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:52,724] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:52,725] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,726] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 21] INFO [2022-03-13 17:14:52,726] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,727] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:14:52,761] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,762] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:14:52,780] jepsen worker 22 - jepsen.util 652 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,781] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,781] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:52,781] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,782] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,783] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:14:52,793] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,793] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,794] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,794] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:14:52,795] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:52,795] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:52,799] jepsen worker 24 - jepsen.util 564 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,800] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:52,830] jepsen worker 51 - jepsen.util 591 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,830] jepsen worker 51 - jepsen.util 591 :invoke :write [17 32] INFO [2022-03-13 17:14:52,838] jepsen worker 89 - jepsen.util 2519 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,839] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 22] INFO [2022-03-13 17:14:52,855] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,856] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,864] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,864] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:14:52,867] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,868] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,883] jepsen worker 85 - jepsen.util 2695 :info :write [28 22] :RequestTimeoutException INFO [2022-03-13 17:14:52,958] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,959] jepsen worker 41 - jepsen.util 2831 :info :write [13 33] :RequestTimeoutException INFO [2022-03-13 17:14:52,960] jepsen worker 58 - jepsen.util 2578 :info :write [19 16] :RequestTimeoutException INFO [2022-03-13 17:14:52,960] jepsen worker 85 - jepsen.util 2785 :invoke :write [28 42] INFO [2022-03-13 17:14:52,961] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,962] jepsen worker 41 - jepsen.util 2921 :invoke :write [13 49] INFO [2022-03-13 17:14:52,962] jepsen worker 58 - jepsen.util 2668 :invoke :write [19 2] INFO [2022-03-13 17:14:52,965] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,966] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,967] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,967] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:14:52,967] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,968] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:52,969] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,969] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:14:52,970] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,970] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,971] jepsen worker 16 - jepsen.util 2716 :info :write [5 44] :RequestTimeoutException INFO [2022-03-13 17:14:52,971] jepsen worker 78 - jepsen.util 528 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,972] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,972] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,973] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,973] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,975] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:52,977] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:52,978] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,978] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:52,980] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:52,981] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:52,982] jepsen worker 16 - jepsen.util 2806 :invoke :read [5 nil] INFO [2022-03-13 17:14:52,983] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:14:52,984] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,986] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,986] jepsen worker 14 - jepsen.util 374 :invoke :write [4 24] INFO [2022-03-13 17:14:52,990] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,991] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,992] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,993] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:52,994] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:14:52,995] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:52,995] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,001] jepsen worker 50 - jepsen.util 3020 :info :write [16 19] :RequestTimeoutException INFO [2022-03-13 17:14:53,002] jepsen worker 50 - jepsen.util 3110 :invoke :write [16 41] INFO [2022-03-13 17:14:53,008] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,008] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,008] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,009] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:14:53,010] jepsen worker 59 - jepsen.util 2849 :info :write [19 31] :RequestTimeoutException INFO [2022-03-13 17:14:53,011] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:14:53,011] jepsen worker 59 - jepsen.util 2939 :invoke :read [19 nil] INFO [2022-03-13 17:14:53,012] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:14:53,014] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,015] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:14:53,067] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,067] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:53,070] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,071] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:14:53,095] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,096] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:53,103] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,104] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:14:53,148] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,148] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:53,154] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,154] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:53,155] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:53,155] jepsen worker 62 - jepsen.util 512 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,156] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,156] jepsen worker 6 - jepsen.util 636 :invoke :write [2 8] INFO [2022-03-13 17:14:53,157] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:53,158] jepsen worker 42 - jepsen.util 402 :invoke :write [14 28] INFO [2022-03-13 17:14:53,171] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,171] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:53,173] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:53,173] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,173] jepsen worker nemesis - couchbase.util Rebalance check count: 402 INFO [2022-03-13 17:14:53,173] jepsen worker nemesis - couchbase.util Rebalance stuck count: 302 INFO [2022-03-13 17:14:53,174] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:14:53,183] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,184] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:14:53,226] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,226] jepsen worker 14 - jepsen.util 374 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,227] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:53,227] jepsen worker 89 - jepsen.util 2519 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,227] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:53,227] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,233] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,234] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:53,236] jepsen worker 77 - jepsen.util 707 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,236] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,236] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,236] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,236] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,237] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:53,238] jepsen worker 76 - jepsen.util 706 :invoke :write [25 40] INFO [2022-03-13 17:14:53,239] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:14:53,239] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:53,239] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:53,243] jepsen worker 33 - jepsen.util 663 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,243] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,243] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,244] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,245] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:53,246] jepsen worker 24 - jepsen.util 564 :invoke :write [8 28] INFO [2022-03-13 17:14:53,247] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:14:53,247] jepsen worker 71 - jepsen.util 521 :invoke :write [23 34] INFO [2022-03-13 17:14:53,248] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,251] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:14:53,269] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,270] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,288] jepsen worker 37 - jepsen.util 2107 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,288] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,289] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 11] INFO [2022-03-13 17:14:53,289] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:14:53,347] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,347] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,347] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,354] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:14:53,354] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:14:53,354] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 1] INFO [2022-03-13 17:14:53,360] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,361] jepsen worker 80 - jepsen.util 710 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,361] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,361] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:53,362] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:53,363] jepsen worker 80 - jepsen.util 710 :invoke :write [26 33] INFO [2022-03-13 17:14:53,365] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,365] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:53,367] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,368] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:14:53,374] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,375] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:53,377] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,378] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:14:53,386] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,387] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:14:53,397] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,397] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:14:53,418] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,423] jepsen worker 10 - jepsen.util 2980 :ok :read [3 25] INFO [2022-03-13 17:14:53,423] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:53,423] jepsen worker 10 - jepsen.util 2980 :invoke :write [3 3] INFO [2022-03-13 17:14:53,442] jepsen worker 63 - jepsen.util 2943 :ok :read [21 :nil] INFO [2022-03-13 17:14:53,443] jepsen worker 63 - jepsen.util 2943 :invoke :read [21 nil] INFO [2022-03-13 17:14:53,449] jepsen worker 27 - jepsen.util 3447 :ok :read [9 26] INFO [2022-03-13 17:14:53,449] jepsen worker 63 - jepsen.util 2943 :ok :read [21 :nil] INFO [2022-03-13 17:14:53,450] jepsen worker 27 - jepsen.util 3447 :invoke :read [9 nil] INFO [2022-03-13 17:14:53,450] jepsen worker 63 - jepsen.util 2943 :invoke :read [21 nil] INFO [2022-03-13 17:14:53,454] jepsen worker 18 - jepsen.util 2718 :ok :read [6 16] INFO [2022-03-13 17:14:53,455] jepsen worker 18 - jepsen.util 2718 :invoke :write [6 46] INFO [2022-03-13 17:14:53,460] jepsen worker 74 - jepsen.util 3134 :ok :read [24 24] INFO [2022-03-13 17:14:53,461] jepsen worker 74 - jepsen.util 3134 :invoke :read [24 nil] INFO [2022-03-13 17:14:53,486] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,487] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:14:53,496] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,496] jepsen worker 37 - jepsen.util 2107 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,497] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,497] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:53,498] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,498] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:14:53,499] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,499] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:53,511] jepsen worker 27 - jepsen.util 3447 :ok :read [9 26] INFO [2022-03-13 17:14:53,512] jepsen worker 27 - jepsen.util 3447 :invoke :write [9 2] INFO [2022-03-13 17:14:53,528] jepsen worker 63 - jepsen.util 2943 :ok :read [21 :nil] INFO [2022-03-13 17:14:53,529] jepsen worker 63 - jepsen.util 2943 :invoke :write [21 1] INFO [2022-03-13 17:14:53,540] jepsen worker 89 - jepsen.util 2519 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,541] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,541] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,542] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:53,544] jepsen worker 24 - jepsen.util 564 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,544] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:53,545] jepsen worker 74 - jepsen.util 3134 :ok :read [24 24] INFO [2022-03-13 17:14:53,546] jepsen worker 74 - jepsen.util 3134 :invoke :read [24 nil] INFO [2022-03-13 17:14:53,558] jepsen worker 43 - jepsen.util 583 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,558] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:53,558] jepsen worker 44 - jepsen.util 584 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,559] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:53,559] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,560] jepsen worker 60 - jepsen.util 600 :invoke :write [20 2] INFO [2022-03-13 17:14:53,560] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,561] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:53,593] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,593] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:14:53,601] jepsen worker 13 - jepsen.util 643 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,601] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,602] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,602] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,602] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,603] jepsen worker 78 - jepsen.util 528 :invoke :write [26 11] INFO [2022-03-13 17:14:53,604] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:53,604] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:53,605] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:53,605] jepsen worker 52 - jepsen.util 592 :invoke :write [17 34] INFO [2022-03-13 17:14:53,610] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,611] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:53,622] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,622] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,623] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:53,624] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:14:53,639] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,640] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:53,640] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,641] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,644] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,645] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:14:53,659] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,659] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:53,661] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,662] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:14:53,705] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,705] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,705] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:14:53,706] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:14:53,711] jepsen worker 74 - jepsen.util 3134 :ok :read [24 24] INFO [2022-03-13 17:14:53,711] jepsen worker 74 - jepsen.util 3134 :invoke :read [24 nil] INFO [2022-03-13 17:14:53,716] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,716] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,717] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,717] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:53,719] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,719] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:14:53,720] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,721] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:14:53,746] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,747] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,759] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,759] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,761] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:53,761] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,761] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:53,762] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:53,791] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,791] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,791] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,792] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:53,792] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,792] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:14:53,793] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:53,793] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:53,795] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,796] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,796] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,796] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:53,797] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:14:53,797] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:14:53,798] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,798] jepsen worker 25 - jepsen.util 565 :invoke :write [8 15] INFO [2022-03-13 17:14:53,810] jepsen worker 2 - jepsen.util 3332 :info :write [0 27] :RequestTimeoutException INFO [2022-03-13 17:14:53,811] jepsen worker 2 - jepsen.util 3422 :invoke :write [0 16] INFO [2022-03-13 17:14:53,814] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,815] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:14:53,833] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,833] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,833] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:53,834] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 36] INFO [2022-03-13 17:14:53,842] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,843] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:53,846] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,847] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:14:53,856] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,857] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:14:53,865] jepsen worker 74 - jepsen.util 3134 :ok :read [24 24] INFO [2022-03-13 17:14:53,866] jepsen worker 74 - jepsen.util 3134 :invoke :write [24 46] INFO [2022-03-13 17:14:53,894] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,894] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,894] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,895] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:14:53,934] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,934] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:14:53,946] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,947] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:14:53,974] jepsen worker 71 - jepsen.util 521 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,975] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:53,984] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,984] jepsen worker 80 - jepsen.util 710 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,984] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,984] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,985] jepsen worker 70 - jepsen.util 700 :invoke :write [23 25] INFO [2022-03-13 17:14:53,986] jepsen worker 80 - jepsen.util 710 :invoke :write [26 34] INFO [2022-03-13 17:14:53,990] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,990] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:53,990] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:53,991] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:53,992] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:53,993] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,002] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,003] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:54,003] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,004] jepsen worker 79 - jepsen.util 1069 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,004] jepsen worker 51 - jepsen.util 591 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,004] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,004] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,005] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,005] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:54,005] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,005] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,006] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:14:54,007] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:54,009] jepsen worker 7 - jepsen.util 637 :invoke :write [2 15] INFO [2022-03-13 17:14:54,009] jepsen worker 51 - jepsen.util 591 :invoke :write [17 33] INFO [2022-03-13 17:14:54,010] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:54,011] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:54,011] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:54,035] jepsen worker 89 - jepsen.util 2519 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,036] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,059] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,059] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:54,075] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,076] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:14:54,080] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,080] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,080] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,080] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:14:54,081] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:14:54,081] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:14:54,127] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,127] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:54,156] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,156] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,157] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,158] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,159] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,159] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,160] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,161] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,178] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:54,178] jepsen worker nemesis - couchbase.util Rebalance check count: 403 INFO [2022-03-13 17:14:54,178] jepsen worker nemesis - couchbase.util Rebalance stuck count: 303 INFO [2022-03-13 17:14:54,179] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,180] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:54,185] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,186] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:14:54,188] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,189] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,189] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,189] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,189] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:14:54,191] jepsen worker 71 - jepsen.util 521 :invoke :write [23 13] INFO [2022-03-13 17:14:54,192] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:54,193] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,194] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:14:54,194] jepsen worker 33 - jepsen.util 663 :invoke :write [11 42] INFO [2022-03-13 17:14:54,204] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,205] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,215] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,216] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,218] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,218] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,220] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,220] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:14:54,223] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,224] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:14:54,245] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,247] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,248] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,248] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,249] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:14:54,249] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,255] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,256] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,296] jepsen worker 84 - jepsen.util 3144 :info :write [28 37] :RequestTimeoutException INFO [2022-03-13 17:14:54,297] jepsen worker 84 - jepsen.util 3234 :invoke :read [28 nil] INFO [2022-03-13 17:14:54,327] jepsen worker 84 - jepsen.util 3234 :ok :read [28 44] INFO [2022-03-13 17:14:54,327] jepsen worker 84 - jepsen.util 3234 :invoke :write [28 22] INFO [2022-03-13 17:14:54,330] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,331] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,335] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,335] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,335] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,336] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,336] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,337] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,337] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,338] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,338] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,338] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 32] INFO [2022-03-13 17:14:54,339] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:54,339] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 27] INFO [2022-03-13 17:14:54,343] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,343] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,344] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,344] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:14:54,345] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,346] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,346] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,347] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,347] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,347] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,347] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,348] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,349] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:14:54,349] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,350] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,350] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,351] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,351] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,352] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:14:54,353] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:14:54,354] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,359] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,373] jepsen worker 65 - jepsen.util 3305 :info :write [21 16] :RequestTimeoutException INFO [2022-03-13 17:14:54,374] jepsen worker 65 - jepsen.util 3395 :invoke :read [21 nil] INFO [2022-03-13 17:14:54,381] jepsen worker 25 - jepsen.util 565 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,381] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:14:54,403] jepsen worker 73 - jepsen.util 3133 :ok :read [24 24] INFO [2022-03-13 17:14:54,404] jepsen worker 73 - jepsen.util 3133 :invoke :read [24 nil] INFO [2022-03-13 17:14:54,420] jepsen worker 32 - jepsen.util 2912 :ok :read [10 :nil] INFO [2022-03-13 17:14:54,420] jepsen worker 32 - jepsen.util 2912 :invoke :write [10 19] INFO [2022-03-13 17:14:54,423] jepsen worker 30 - jepsen.util 2820 :ok :read [10 :nil] INFO [2022-03-13 17:14:54,424] jepsen worker 30 - jepsen.util 2820 :invoke :read [10 nil] INFO [2022-03-13 17:14:54,432] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,432] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:14:54,440] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,441] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,441] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,441] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:14:54,442] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,443] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,444] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,445] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,447] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,447] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:14:54,495] jepsen worker 28 - jepsen.util 3088 :ok :read [9 26] INFO [2022-03-13 17:14:54,496] jepsen worker 28 - jepsen.util 3088 :invoke :write [9 38] INFO [2022-03-13 17:14:54,499] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,500] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:14:54,525] jepsen worker 30 - jepsen.util 2820 :ok :read [10 :nil] INFO [2022-03-13 17:14:54,525] jepsen worker 30 - jepsen.util 2820 :invoke :read [10 nil] INFO [2022-03-13 17:14:54,539] jepsen worker 73 - jepsen.util 3133 :ok :read [24 24] INFO [2022-03-13 17:14:54,539] jepsen worker 73 - jepsen.util 3133 :invoke :write [24 0] INFO [2022-03-13 17:14:54,556] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,557] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,558] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,558] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,567] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,568] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,568] jepsen worker 26 - jepsen.util 476 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,569] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:54,575] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,576] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,578] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,579] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,580] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,580] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,581] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,582] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 21] INFO [2022-03-13 17:14:54,620] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,621] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,664] jepsen worker 89 - jepsen.util 2519 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,664] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,664] jepsen worker 37 - jepsen.util 2107 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,665] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:14:54,665] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 36] INFO [2022-03-13 17:14:54,666] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:14:54,689] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,689] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,691] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:14:54,691] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:54,696] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,697] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:54,709] jepsen worker 78 - jepsen.util 528 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,709] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:54,727] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,728] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:14:54,753] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,753] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,754] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:54,755] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:14:54,760] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,760] jepsen worker 8 - jepsen.util 728 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,761] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:54,761] jepsen worker 8 - jepsen.util 728 :invoke :write [2 27] INFO [2022-03-13 17:14:54,773] jepsen worker 40 - jepsen.util 3100 :info :write [13 38] :RequestTimeoutException INFO [2022-03-13 17:14:54,774] jepsen worker 40 - jepsen.util 3190 :invoke :read [13 nil] INFO [2022-03-13 17:14:54,790] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,790] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,790] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,791] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,795] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,795] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:54,795] jepsen worker 76 - jepsen.util 706 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,795] jepsen worker 3 - jepsen.util 1983 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,796] jepsen worker 44 - jepsen.util 584 :invoke :write [14 20] INFO [2022-03-13 17:14:54,797] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:54,797] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 23] INFO [2022-03-13 17:14:54,798] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:54,841] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,841] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,842] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:54,842] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:14:54,867] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,867] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,868] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,867] jepsen worker 71 - jepsen.util 521 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,869] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:14:54,869] jepsen worker 71 - jepsen.util 521 :invoke :write [23 35] INFO [2022-03-13 17:14:54,870] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:14:54,878] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:14:54,886] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,887] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:54,891] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,892] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:54,893] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,893] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,893] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,894] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,894] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,895] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,895] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:14:54,896] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,897] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,897] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,898] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:14:54,898] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,900] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,901] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:54,902] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,902] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,903] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:14:54,903] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:14:54,911] jepsen worker 42 - jepsen.util 402 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,912] jepsen worker 42 - jepsen.util 402 :invoke :write [14 3] INFO [2022-03-13 17:14:54,986] jepsen worker 3 - jepsen.util 1983 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,987] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,995] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,996] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:54,997] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:54,998] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,004] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,005] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:14:55,016] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,016] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:55,020] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,020] jepsen worker 24 - jepsen.util 564 :invoke :write [8 34] INFO [2022-03-13 17:14:55,037] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,037] jepsen worker 52 - jepsen.util 592 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,038] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,038] jepsen worker 89 - jepsen.util 2519 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,039] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,039] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,039] jepsen worker 34 - jepsen.util 754 :invoke :write [11 44] INFO [2022-03-13 17:14:55,040] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:55,041] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 31] INFO [2022-03-13 17:14:55,041] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:14:55,043] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,045] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,045] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:55,046] jepsen worker 22 - jepsen.util 652 :invoke :write [7 45] INFO [2022-03-13 17:14:55,047] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,048] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:14:55,088] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,090] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:55,093] jepsen worker 33 - jepsen.util 663 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,093] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,094] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:55,095] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:55,114] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,115] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:14:55,116] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,116] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,117] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:55,118] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:14:55,119] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,119] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:14:55,126] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,126] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 20] INFO [2022-03-13 17:14:55,176] jepsen worker 80 - jepsen.util 710 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,176] jepsen worker 80 - jepsen.util 710 :invoke :write [26 40] INFO [2022-03-13 17:14:55,185] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:55,185] jepsen worker nemesis - couchbase.util Rebalance check count: 404 INFO [2022-03-13 17:14:55,185] jepsen worker nemesis - couchbase.util Rebalance stuck count: 304 INFO [2022-03-13 17:14:55,194] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,194] jepsen worker 60 - jepsen.util 600 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,195] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,194] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,195] jepsen worker 60 - jepsen.util 600 :invoke :write [20 34] INFO [2022-03-13 17:14:55,196] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,196] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:14:55,196] jepsen worker 61 - jepsen.util 691 :invoke :write [20 16] INFO [2022-03-13 17:14:55,215] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,215] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,215] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:55,216] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,227] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,228] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,232] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,233] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,235] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,236] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:14:55,260] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,260] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,261] jepsen worker 70 - jepsen.util 700 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,262] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,262] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,262] jepsen worker 70 - jepsen.util 700 :invoke :write [23 7] INFO [2022-03-13 17:14:55,262] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,263] jepsen worker 71 - jepsen.util 521 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,263] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:55,264] jepsen worker 78 - jepsen.util 528 :invoke :write [26 24] INFO [2022-03-13 17:14:55,264] jepsen worker 52 - jepsen.util 592 :invoke :write [17 14] INFO [2022-03-13 17:14:55,265] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,283] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,283] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,283] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,284] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,285] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,285] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,285] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,286] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,286] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,287] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:14:55,287] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,287] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,288] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,288] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:14:55,289] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,289] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,290] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,291] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:14:55,327] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,327] jepsen worker 37 - jepsen.util 2107 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,328] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 5] INFO [2022-03-13 17:14:55,330] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:55,330] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,331] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,332] jepsen worker 22 - jepsen.util 652 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,333] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:14:55,336] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,336] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:14:55,352] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,352] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,352] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,353] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:55,354] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:14:55,354] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:14:55,358] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,359] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,395] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,395] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:14:55,410] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,410] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,413] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,415] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,430] jepsen worker 89 - jepsen.util 2519 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,431] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 37] INFO [2022-03-13 17:14:55,468] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,468] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,468] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,468] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:55,471] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,471] jepsen worker 24 - jepsen.util 564 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,471] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,471] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,472] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:14:55,474] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:55,474] jepsen worker 51 - jepsen.util 591 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,474] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,475] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,475] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:55,475] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,476] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,476] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,477] jepsen worker 51 - jepsen.util 591 :invoke :write [17 26] INFO [2022-03-13 17:14:55,478] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:14:55,479] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:55,480] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:14:55,480] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,483] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:14:55,483] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:55,484] jepsen worker 77 - jepsen.util 707 :invoke :write [25 37] INFO [2022-03-13 17:14:55,485] jepsen worker 76 - jepsen.util 706 :invoke :write [25 30] INFO [2022-03-13 17:14:55,487] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,487] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,487] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:55,488] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,490] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,490] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 24] INFO [2022-03-13 17:14:55,492] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:55,492] jepsen worker 53 - jepsen.util 503 :invoke :write [17 10] INFO [2022-03-13 17:14:55,493] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:14:55,493] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:14:55,494] jepsen worker 6 - jepsen.util 636 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,500] jepsen worker 6 - jepsen.util 636 :invoke :write [2 0] INFO [2022-03-13 17:14:55,502] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,502] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,503] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,505] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,506] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:14:55,506] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,514] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,514] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:14:55,519] jepsen worker 34 - jepsen.util 754 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,520] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:55,527] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,528] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:14:55,539] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,539] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,540] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 34] INFO [2022-03-13 17:14:55,540] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:14:55,605] jepsen worker 37 - jepsen.util 2107 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,606] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 16] INFO [2022-03-13 17:14:55,606] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,606] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:14:55,620] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,623] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,623] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,623] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,626] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,626] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,676] jepsen worker 88 - jepsen.util 2068 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,676] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,677] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:14:55,677] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:55,687] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,687] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,688] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:14:55,688] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:14:55,699] jepsen worker 7 - jepsen.util 637 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,700] jepsen worker 7 - jepsen.util 637 :invoke :write [2 30] INFO [2022-03-13 17:14:55,702] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,702] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,703] jepsen worker 62 - jepsen.util 512 :invoke :write [20 38] INFO [2022-03-13 17:14:55,704] jepsen worker 26 - jepsen.util 476 :invoke :write [8 28] INFO [2022-03-13 17:14:55,716] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,716] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,727] jepsen worker 89 - jepsen.util 2519 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,728] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 24] INFO [2022-03-13 17:14:55,729] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,729] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,756] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,757] jepsen worker 71 - jepsen.util 521 :invoke :write [23 35] INFO [2022-03-13 17:14:55,762] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,762] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,762] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,762] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,763] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:14:55,763] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:14:55,764] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:14:55,764] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,765] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,766] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,767] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,768] jepsen worker 44 - jepsen.util 584 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,768] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,768] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,768] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,769] jepsen worker 69 - jepsen.util 699 :invoke :write [23 19] INFO [2022-03-13 17:14:55,770] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,770] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:55,770] jepsen worker 12 - jepsen.util 732 :invoke :write [4 22] INFO [2022-03-13 17:14:55,775] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,777] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,778] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,779] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,780] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,780] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,781] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:14:55,781] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,782] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:14:55,789] jepsen worker 80 - jepsen.util 710 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,789] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,789] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:55,790] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,790] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:55,791] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,792] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,793] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,830] jepsen worker 1 - jepsen.util 2701 :ok :read [0 41] INFO [2022-03-13 17:14:55,830] jepsen worker 1 - jepsen.util 2701 :invoke :read [0 nil] INFO [2022-03-13 17:14:55,832] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,832] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:14:55,856] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,857] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,859] jepsen worker 15 - jepsen.util 3165 :ok :read [5 43] INFO [2022-03-13 17:14:55,860] jepsen worker 15 - jepsen.util 3165 :invoke :write [5 4] INFO [2022-03-13 17:14:55,886] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,887] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,887] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,888] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 27] INFO [2022-03-13 17:14:55,904] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,904] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:55,935] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,935] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,936] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:14:55,937] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:55,938] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,938] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 10] INFO [2022-03-13 17:14:55,948] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,948] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,949] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:14:55,950] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:55,964] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,964] jepsen worker 37 - jepsen.util 2107 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,964] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:55,965] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:55,985] jepsen worker 8 - jepsen.util 728 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,985] jepsen worker 70 - jepsen.util 700 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,985] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,986] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:55,986] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,986] jepsen worker 8 - jepsen.util 728 :invoke :write [2 10] INFO [2022-03-13 17:14:55,987] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,987] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:55,997] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:55,998] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:14:56,036] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,036] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:56,037] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,037] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:56,067] jepsen worker 89 - jepsen.util 2519 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,067] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,068] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:14:56,069] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,076] jepsen worker 60 - jepsen.util 600 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,076] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:14:56,105] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,105] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,105] jepsen worker 88 - jepsen.util 2068 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,105] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,105] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,106] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,106] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,107] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:14:56,110] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 3] INFO [2022-03-13 17:14:56,111] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:14:56,111] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,114] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:56,133] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,136] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,174] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,175] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,175] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:14:56,175] jepsen worker 78 - jepsen.util 528 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,176] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,176] jepsen worker 3 - jepsen.util 1983 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,177] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,177] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,178] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:14:56,178] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:14:56,178] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,179] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,180] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,180] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:14:56,183] jepsen worker 71 - jepsen.util 521 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,183] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,183] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,183] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,184] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:14:56,184] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:14:56,185] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:56,185] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:56,191] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,191] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,191] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:56,191] jepsen worker nemesis - couchbase.util Rebalance check count: 405 INFO [2022-03-13 17:14:56,191] jepsen worker nemesis - couchbase.util Rebalance stuck count: 305 INFO [2022-03-13 17:14:56,198] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:56,198] jepsen worker 33 - jepsen.util 663 :invoke :write [11 34] INFO [2022-03-13 17:14:56,199] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,204] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,204] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,205] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,206] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:14:56,206] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,207] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:14:56,210] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,222] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,223] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,225] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,225] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,234] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,234] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,294] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,295] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:56,299] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,300] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:14:56,305] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,306] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:14:56,320] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,321] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:14:56,385] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,385] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,385] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,385] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,386] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:14:56,386] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:14:56,390] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,390] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,390] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,391] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,395] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,395] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,395] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 6] INFO [2022-03-13 17:14:56,396] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,398] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,398] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,449] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,449] jepsen worker 37 - jepsen.util 2107 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,449] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,450] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,455] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,456] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,457] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:14:56,457] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:14:56,489] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,489] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:14:56,526] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,527] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,527] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 31] INFO [2022-03-13 17:14:56,528] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,537] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,538] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,548] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,549] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] WARN [2022-03-13 17:14:56,563] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17398046,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18140310,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9154432,"encode_duration_us":19636,"operation_id":"0xb091","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10002167},{"operation_name":"upsert","last_dispatch_duration_us":9081749,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10597590,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9282789,"encode_duration_us":32205,"operation_id":"0xb753","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006818},{"operation_name":"upsert","last_dispatch_duration_us":9039894,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12418303,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9350798,"encode_duration_us":19949,"operation_id":"0xb60c","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10002341},{"operation_name":"upsert","last_dispatch_duration_us":9064222,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10567291,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9317104,"encode_duration_us":25439,"operation_id":"0xb754","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10008256}],"total_count":4}} WARN [2022-03-13 17:14:56,563] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8229117,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9805421,"total_server_duration_us":373901,"encode_duration_us":21060,"operation_id":"0xb73b","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10074240},{"operation_name":"upsert","last_dispatch_duration_us":42525,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1705163,"total_server_duration_us":321963,"encode_duration_us":24060,"operation_id":"0xb744","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10046983},{"operation_name":"upsert","last_dispatch_duration_us":32118,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":103734,"total_server_duration_us":61521,"encode_duration_us":25268,"operation_id":"0xb830","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10012653},{"operation_name":"upsert","last_dispatch_duration_us":9064222,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10567291,"total_server_duration_us":9317104,"encode_duration_us":25439,"operation_id":"0xb754","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10008256},{"operation_name":"upsert","last_dispatch_duration_us":9081749,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10597590,"total_server_duration_us":9282789,"encode_duration_us":32205,"operation_id":"0xb753","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006818},{"operation_name":"upsert","last_dispatch_duration_us":8444969,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9633696,"total_server_duration_us":276224,"encode_duration_us":29188,"operation_id":"0xb74e","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10005111},{"operation_name":"upsert","last_dispatch_duration_us":8225787,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9354518,"total_server_duration_us":302002,"encode_duration_us":22103,"operation_id":"0xb7dd","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10004877},{"operation_name":"upsert","last_dispatch_duration_us":9039894,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12418303,"total_server_duration_us":9350798,"encode_duration_us":19949,"operation_id":"0xb60c","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10002341},{"operation_name":"upsert","last_dispatch_duration_us":173740,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":468508,"total_server_duration_us":60468,"encode_duration_us":18409,"operation_id":"0xb5a0","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10002259},{"operation_name":"upsert","last_dispatch_duration_us":1394732,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1394732,"total_server_duration_us":50535,"encode_duration_us":16955,"operation_id":"0xb9b3","timeout_ms":10000,"last_server_duration_us":50535,"total_duration_us":1420842}],"total_count":35}} INFO [2022-03-13 17:14:56,580] jepsen worker 36 - jepsen.util 2196 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,580] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,580] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,580] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,581] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,582] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 18] INFO [2022-03-13 17:14:56,585] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,585] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,588] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,588] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,589] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,590] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:14:56,595] jepsen worker 52 - jepsen.util 592 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,595] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,595] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,596] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:56,597] jepsen worker 34 - jepsen.util 754 :invoke :write [11 45] INFO [2022-03-13 17:14:56,598] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,598] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,598] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,598] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,598] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,598] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:56,599] jepsen worker 13 - jepsen.util 643 :invoke :write [4 19] INFO [2022-03-13 17:14:56,600] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,601] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,601] jepsen worker 52 - jepsen.util 592 :invoke :write [17 37] INFO [2022-03-13 17:14:56,601] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:56,608] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,608] jepsen worker 42 - jepsen.util 402 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,608] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:56,608] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,609] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,608] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,609] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,610] jepsen worker 39 - jepsen.util 3009 :ok :read [13 39] INFO [2022-03-13 17:14:56,610] jepsen worker 70 - jepsen.util 700 :invoke :write [23 44] INFO [2022-03-13 17:14:56,612] jepsen worker 42 - jepsen.util 402 :invoke :write [14 7] INFO [2022-03-13 17:14:56,613] jepsen worker 43 - jepsen.util 583 :invoke :write [14 37] INFO [2022-03-13 17:14:56,614] jepsen worker 39 - jepsen.util 3009 :invoke :read [13 nil] INFO [2022-03-13 17:14:56,614] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,618] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:56,619] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,620] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,621] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,621] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,621] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,622] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,623] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:14:56,623] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,624] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,625] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,641] jepsen worker 39 - jepsen.util 3009 :ok :read [13 39] INFO [2022-03-13 17:14:56,642] jepsen worker 39 - jepsen.util 3009 :invoke :read [13 nil] INFO [2022-03-13 17:14:56,643] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,643] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,643] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,644] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:56,645] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:56,645] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:56,649] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,649] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,664] jepsen worker 61 - jepsen.util 691 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,665] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:56,667] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,667] jepsen worker 37 - jepsen.util 2107 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,668] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,668] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 28] INFO [2022-03-13 17:14:56,674] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,675] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,677] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,677] jepsen worker 39 - jepsen.util 3009 :ok :read [13 39] INFO [2022-03-13 17:14:56,678] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,678] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,678] jepsen worker 39 - jepsen.util 3009 :invoke :write [13 16] INFO [2022-03-13 17:14:56,679] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:14:56,685] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,685] jepsen worker 59 - jepsen.util 2939 :ok :read [19 25] INFO [2022-03-13 17:14:56,685] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:14:56,686] jepsen worker 59 - jepsen.util 2939 :invoke :read [19 nil] INFO [2022-03-13 17:14:56,716] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,716] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:14:56,717] jepsen worker 59 - jepsen.util 2939 :ok :read [19 25] INFO [2022-03-13 17:14:56,718] jepsen worker 59 - jepsen.util 2939 :invoke :read [19 nil] INFO [2022-03-13 17:14:56,730] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,731] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,737] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,738] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:14:56,748] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,749] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,750] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,750] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,751] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 8] INFO [2022-03-13 17:14:56,751] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,753] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,754] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,760] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,760] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,761] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:14:56,761] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,763] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,764] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:14:56,779] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,779] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,780] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,780] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:56,780] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,781] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:14:56,782] jepsen worker 14 - jepsen.util 374 :invoke :write [4 9] INFO [2022-03-13 17:14:56,783] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:56,790] jepsen worker 59 - jepsen.util 2939 :ok :read [19 25] INFO [2022-03-13 17:14:56,791] jepsen worker 59 - jepsen.util 2939 :invoke :write [19 43] INFO [2022-03-13 17:14:56,791] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,792] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,794] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:56,795] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:14:56,798] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,799] jepsen worker 21 - jepsen.util 831 :invoke :write [7 28] INFO [2022-03-13 17:14:56,801] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,801] jepsen worker 89 - jepsen.util 2519 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,802] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:14:56,802] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:14:56,811] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,811] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,816] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,816] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,819] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,820] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:14:56,849] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,850] jepsen worker 71 - jepsen.util 521 :invoke :write [23 24] INFO [2022-03-13 17:14:56,864] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,864] jepsen worker 52 - jepsen.util 592 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,864] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,865] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,865] jepsen worker 52 - jepsen.util 592 :invoke :write [17 18] INFO [2022-03-13 17:14:56,865] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:56,875] jepsen worker 88 - jepsen.util 2068 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,876] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:56,877] jepsen worker 79 - jepsen.util 1069 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,877] jepsen worker 51 - jepsen.util 591 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,878] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 8] INFO [2022-03-13 17:14:56,878] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:56,887] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,887] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,893] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,894] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,895] jepsen worker 37 - jepsen.util 2107 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,896] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,902] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,903] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:14:56,911] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,912] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:56,943] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,943] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:56,954] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,954] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:56,985] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:56,986] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:14:57,003] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,003] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,003] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,004] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:14:57,004] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:57,005] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:14:57,005] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,010] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:14:57,012] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,014] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:57,016] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,016] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:57,017] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,018] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 20] INFO [2022-03-13 17:14:57,048] jepsen worker 13 - jepsen.util 643 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,048] jepsen worker 6 - jepsen.util 636 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,049] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:57,049] jepsen worker 14 - jepsen.util 374 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,049] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:57,050] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:57,051] jepsen worker 33 - jepsen.util 663 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,052] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:57,060] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,061] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:14:57,093] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,093] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:14:57,096] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,097] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:14:57,172] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,172] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,172] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:57,173] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:14:57,188] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,188] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:57,197] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:57,197] jepsen worker nemesis - couchbase.util Rebalance check count: 406 INFO [2022-03-13 17:14:57,197] jepsen worker nemesis - couchbase.util Rebalance stuck count: 306 INFO [2022-03-13 17:14:57,226] jepsen worker 34 - jepsen.util 754 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,226] jepsen worker 34 - jepsen.util 754 :invoke :write [11 1] INFO [2022-03-13 17:14:57,267] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,267] jepsen worker 21 - jepsen.util 831 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,267] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,268] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:14:57,269] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:14:57,270] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:14:57,281] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,281] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:14:57,286] jepsen worker 48 - jepsen.util 2838 :info :write [16 17] :RequestTimeoutException INFO [2022-03-13 17:14:57,287] jepsen worker 48 - jepsen.util 2928 :invoke :read [16 nil] INFO [2022-03-13 17:14:57,309] jepsen worker 26 - jepsen.util 476 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,310] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:57,322] jepsen worker 70 - jepsen.util 700 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,322] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,323] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,323] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:14:57,323] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:14:57,323] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:57,396] jepsen worker 57 - jepsen.util 2937 :info :write [19 44] :RequestTimeoutException INFO [2022-03-13 17:14:57,396] jepsen worker 57 - jepsen.util 3027 :invoke :read [19 nil] INFO [2022-03-13 17:14:57,407] jepsen worker 82 - jepsen.util 2962 :info :write [27 30] :RequestTimeoutException INFO [2022-03-13 17:14:57,407] jepsen worker 82 - jepsen.util 3052 :invoke :read [27 nil] INFO [2022-03-13 17:14:57,428] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,428] jepsen worker 22 - jepsen.util 652 :invoke :write [7 28] INFO [2022-03-13 17:14:57,435] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,436] jepsen worker 76 - jepsen.util 706 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,436] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:14:57,437] jepsen worker 76 - jepsen.util 706 :invoke :write [25 11] INFO [2022-03-13 17:14:57,437] jepsen worker 44 - jepsen.util 584 :invoke :write [14 26] INFO [2022-03-13 17:14:57,439] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:57,440] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,440] jepsen worker 71 - jepsen.util 521 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,441] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:14:57,441] jepsen worker 33 - jepsen.util 663 :invoke :write [11 40] INFO [2022-03-13 17:14:57,446] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,447] jepsen worker 24 - jepsen.util 564 :invoke :write [8 39] INFO [2022-03-13 17:14:57,481] jepsen worker 17 - jepsen.util 2987 :info :write [5 37] :RequestTimeoutException INFO [2022-03-13 17:14:57,481] jepsen worker 17 - jepsen.util 3077 :invoke :read [5 nil] INFO [2022-03-13 17:14:57,522] jepsen worker 52 - jepsen.util 592 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,522] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,522] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:57,523] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:14:57,525] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,526] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:57,576] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,577] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:57,580] jepsen worker 7 - jepsen.util 637 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,580] jepsen worker 7 - jepsen.util 637 :invoke :write [2 39] INFO [2022-03-13 17:14:57,620] jepsen worker 33 - jepsen.util 663 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,621] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:57,625] jepsen worker 22 - jepsen.util 652 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,626] jepsen worker 22 - jepsen.util 652 :invoke :write [7 28] INFO [2022-03-13 17:14:57,687] jepsen worker 12 - jepsen.util 732 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,688] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:14:57,727] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,728] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:57,822] jepsen worker 69 - jepsen.util 699 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,822] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,822] jepsen worker 62 - jepsen.util 512 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,822] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,824] jepsen worker 51 - jepsen.util 591 :invoke :write [17 19] INFO [2022-03-13 17:14:57,825] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:57,826] jepsen worker 69 - jepsen.util 699 :invoke :write [23 16] INFO [2022-03-13 17:14:57,826] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:57,836] jepsen worker 24 - jepsen.util 564 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,836] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:57,864] jepsen worker 42 - jepsen.util 402 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,865] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] WARN [2022-03-13 17:14:57,865] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18163802,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18163802,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9052231,"encode_duration_us":21682,"operation_id":"0xb250","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10004341},{"operation_name":"upsert","last_dispatch_duration_us":17862457,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18226097,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9260367,"encode_duration_us":18650,"operation_id":"0xb2ca","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10010022}],"total_count":2}} INFO [2022-03-13 17:14:57,948] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,948] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,948] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,948] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,949] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:14:57,951] jepsen worker 77 - jepsen.util 707 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,951] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,951] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,952] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:57,953] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:57,953] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,954] jepsen worker 77 - jepsen.util 707 :invoke :write [25 0] INFO [2022-03-13 17:14:57,955] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,955] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:14:57,955] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,955] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:14:57,957] jepsen worker 60 - jepsen.util 600 :invoke :write [20 31] INFO [2022-03-13 17:14:57,958] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,958] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:14:57,959] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:14:57,960] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:57,960] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,961] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,961] jepsen worker 33 - jepsen.util 663 :invoke :write [11 6] INFO [2022-03-13 17:14:57,962] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:57,962] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:14:57,970] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:57,971] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:14:58,045] jepsen worker 43 - jepsen.util 583 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,046] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:58,051] jepsen worker 34 - jepsen.util 754 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,052] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,052] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,052] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,052] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,052] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,052] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:14:58,053] jepsen worker 52 - jepsen.util 592 :invoke :write [17 33] INFO [2022-03-13 17:14:58,055] jepsen worker 13 - jepsen.util 643 :invoke :write [4 13] INFO [2022-03-13 17:14:58,055] jepsen worker 80 - jepsen.util 710 :invoke :write [26 15] INFO [2022-03-13 17:14:58,056] jepsen worker 78 - jepsen.util 528 :invoke :write [26 10] INFO [2022-03-13 17:14:58,057] jepsen worker 70 - jepsen.util 700 :invoke :write [23 17] WARN [2022-03-13 17:14:58,066] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13434675,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13434675,"total_server_duration_us":62,"operation_id":"0xb5f3","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":13437102},{"operation_name":"get","last_dispatch_duration_us":11891910,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11891910,"total_server_duration_us":38,"operation_id":"0xb6ef","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":11892251},{"operation_name":"upsert","last_dispatch_duration_us":10175601,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10175601,"total_server_duration_us":20362,"encode_duration_us":18932,"operation_id":"0xb690","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10001485},{"operation_name":"upsert","last_dispatch_duration_us":13326467,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13326467,"total_server_duration_us":29601,"encode_duration_us":26095,"operation_id":"0xb60e","timeout_ms":10000,"last_server_duration_us":29601,"total_duration_us":10001125},{"operation_name":"upsert","last_dispatch_duration_us":14974076,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":14974076,"total_server_duration_us":20921,"encode_duration_us":21578,"operation_id":"0xb515","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10000545},{"operation_name":"get","last_dispatch_duration_us":9729635,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9729635,"total_server_duration_us":27,"operation_id":"0xb884","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9732120},{"operation_name":"upsert","last_dispatch_duration_us":5157025,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5157025,"total_server_duration_us":67617,"encode_duration_us":17496,"operation_id":"0xb947","timeout_ms":10000,"last_server_duration_us":67617,"total_duration_us":5160837},{"operation_name":"get","last_dispatch_duration_us":3384975,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":3384975,"total_server_duration_us":43,"operation_id":"0xbc39","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3387299},{"operation_name":"upsert","last_dispatch_duration_us":760088,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":760088,"total_server_duration_us":193753,"encode_duration_us":36299,"operation_id":"0xbd9d","timeout_ms":10000,"last_server_duration_us":193753,"total_duration_us":764485},{"operation_name":"upsert","last_dispatch_duration_us":676984,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":676984,"total_server_duration_us":100596,"encode_duration_us":18585,"operation_id":"0xbe24","timeout_ms":10000,"last_server_duration_us":100596,"total_duration_us":680800}],"total_count":12}} WARN [2022-03-13 17:14:58,066] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18033598,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26454999,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9151902,"encode_duration_us":18755,"operation_id":"0xabcb","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10007047},{"operation_name":"upsert","last_dispatch_duration_us":18007576,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20205226,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9109420,"encode_duration_us":19085,"operation_id":"0xb0bc","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10009496}],"total_count":2}} INFO [2022-03-13 17:14:58,072] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,073] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:14:58,101] jepsen worker 44 - jepsen.util 584 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,102] jepsen worker 44 - jepsen.util 584 :invoke :write [14 44] WARN [2022-03-13 17:14:58,165] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16999267,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":16999267,"total_server_duration_us":62,"operation_id":"0xb135","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":17002136},{"operation_name":"upsert","last_dispatch_duration_us":91733,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8270315,"total_server_duration_us":89046,"encode_duration_us":16927,"operation_id":"0xb62d","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10003870},{"operation_name":"get","last_dispatch_duration_us":9092002,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9092002,"total_server_duration_us":8778,"operation_id":"0xb58a","timeout_ms":30000,"last_server_duration_us":8778,"total_duration_us":9095059},{"operation_name":"get","last_dispatch_duration_us":8200074,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8200074,"total_server_duration_us":12698,"operation_id":"0xb601","timeout_ms":30000,"last_server_duration_us":12698,"total_duration_us":8203574},{"operation_name":"get","last_dispatch_duration_us":4134746,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":4134746,"total_server_duration_us":49,"operation_id":"0xb8ba","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":4137752},{"operation_name":"upsert","last_dispatch_duration_us":1177792,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1177792,"total_server_duration_us":1050888,"encode_duration_us":17190,"operation_id":"0xb9af","timeout_ms":10000,"last_server_duration_us":1050888,"total_duration_us":1181435},{"operation_name":"upsert","last_dispatch_duration_us":1162869,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1162869,"total_server_duration_us":51828,"encode_duration_us":19425,"operation_id":"0xb9ca","timeout_ms":10000,"last_server_duration_us":51828,"total_duration_us":1166521},{"operation_name":"upsert","last_dispatch_duration_us":1107184,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1107184,"total_server_duration_us":26241,"encode_duration_us":17300,"operation_id":"0xb9c8","timeout_ms":10000,"last_server_duration_us":26241,"total_duration_us":1111008},{"operation_name":"get","last_dispatch_duration_us":1085869,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1085869,"total_server_duration_us":43,"operation_id":"0xb9c3","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":1086042},{"operation_name":"upsert","last_dispatch_duration_us":947272,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":947272,"total_server_duration_us":105504,"encode_duration_us":29446,"operation_id":"0xbf76","timeout_ms":10000,"last_server_duration_us":105504,"total_duration_us":951315}],"total_count":29}} WARN [2022-03-13 17:14:58,165] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16451785,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17312777,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9375092,"encode_duration_us":18766,"operation_id":"0xb063","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10005663},{"operation_name":"upsert","last_dispatch_duration_us":17727352,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17727352,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9045859,"encode_duration_us":27711,"operation_id":"0xb0c2","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10009298}],"total_count":2}} WARN [2022-03-13 17:14:58,166] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17523195,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18489195,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9340058,"encode_duration_us":31202,"operation_id":"0xb29a","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005388},{"operation_name":"upsert","last_dispatch_duration_us":16282696,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16710387,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9283165,"encode_duration_us":27198,"operation_id":"0xb2a0","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10007140},{"operation_name":"upsert","last_dispatch_duration_us":16191483,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16415815,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9113731,"encode_duration_us":16842,"operation_id":"0xb156","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10005637}],"total_count":3}} WARN [2022-03-13 17:14:58,166] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9093694,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9093694,"total_server_duration_us":147353,"operation_id":"0xb608","timeout_ms":30000,"last_server_duration_us":147353,"total_duration_us":14774401},{"operation_name":"get","last_dispatch_duration_us":14091969,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":14091969,"total_server_duration_us":32,"operation_id":"0xb50b","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":14093917},{"operation_name":"get","last_dispatch_duration_us":12703798,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12703798,"total_server_duration_us":15,"operation_id":"0xb5d8","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":12706857},{"operation_name":"upsert","last_dispatch_duration_us":9038970,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9038970,"total_server_duration_us":33819,"encode_duration_us":19383,"operation_id":"0xb6d6","timeout_ms":10000,"last_server_duration_us":33819,"total_duration_us":10003231},{"operation_name":"get","last_dispatch_duration_us":8878696,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8878696,"total_server_duration_us":32,"operation_id":"0xb862","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8881130},{"operation_name":"get","last_dispatch_duration_us":5836398,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5836398,"total_server_duration_us":32,"operation_id":"0xba19","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":5838662},{"operation_name":"get","last_dispatch_duration_us":4781178,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":4781178,"total_server_duration_us":31681,"operation_id":"0xbc34","timeout_ms":30000,"last_server_duration_us":31681,"total_duration_us":4783300},{"operation_name":"get","last_dispatch_duration_us":4049692,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":4049692,"total_server_duration_us":7282,"operation_id":"0xbb31","timeout_ms":30000,"last_server_duration_us":7282,"total_duration_us":4050074},{"operation_name":"get","last_dispatch_duration_us":1344034,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1344034,"total_server_duration_us":32,"operation_id":"0xb9c4","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3447892},{"operation_name":"upsert","last_dispatch_duration_us":2115390,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2115390,"total_server_duration_us":53734,"encode_duration_us":37536,"operation_id":"0xb910","timeout_ms":10000,"last_server_duration_us":53734,"total_duration_us":3213838}],"total_count":87}} WARN [2022-03-13 17:14:58,166] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16937363,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16937363,"total_server_duration_us":27,"operation_id":"0xb11c","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":16939770},{"operation_name":"get","last_dispatch_duration_us":16582840,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16582840,"total_server_duration_us":49,"operation_id":"0xb2b8","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":16585167},{"operation_name":"get","last_dispatch_duration_us":10355549,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10355549,"total_server_duration_us":69,"operation_id":"0xb626","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":10357567},{"operation_name":"upsert","last_dispatch_duration_us":33376,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7003510,"total_server_duration_us":321134,"encode_duration_us":17909,"operation_id":"0xb739","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10008439},{"operation_name":"upsert","last_dispatch_duration_us":12183359,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":12183359,"total_server_duration_us":20921,"encode_duration_us":18136,"operation_id":"0xb517","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10007212},{"operation_name":"upsert","last_dispatch_duration_us":10654665,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10654665,"total_server_duration_us":21082,"encode_duration_us":17697,"operation_id":"0xb5f8","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10004710},{"operation_name":"upsert","last_dispatch_duration_us":40877,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":5152581,"total_server_duration_us":327337,"encode_duration_us":19165,"operation_id":"0xb888","timeout_ms":10000,"last_server_duration_us":26596,"total_duration_us":10003069},{"operation_name":"upsert","last_dispatch_duration_us":907699,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":907699,"total_server_duration_us":112147,"encode_duration_us":22002,"operation_id":"0xbaea","timeout_ms":10000,"last_server_duration_us":112147,"total_duration_us":2510097},{"operation_name":"upsert","last_dispatch_duration_us":2198899,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2198899,"total_server_duration_us":72181,"encode_duration_us":23699,"operation_id":"0xbd80","timeout_ms":10000,"last_server_duration_us":72181,"total_duration_us":2305688},{"operation_name":"upsert","last_dispatch_duration_us":1171437,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1171437,"total_server_duration_us":127155,"encode_duration_us":23761,"operation_id":"0xbb32","timeout_ms":10000,"last_server_duration_us":127155,"total_duration_us":2260929}],"total_count":71}} INFO [2022-03-13 17:14:58,200] jepsen worker 86 - jepsen.util 2786 :info :write [28 47] :RequestTimeoutException INFO [2022-03-13 17:14:58,201] jepsen worker 86 - jepsen.util 2876 :invoke :read [28 nil] INFO [2022-03-13 17:14:58,203] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:58,203] jepsen worker nemesis - couchbase.util Rebalance check count: 407 INFO [2022-03-13 17:14:58,203] jepsen worker nemesis - couchbase.util Rebalance stuck count: 307 INFO [2022-03-13 17:14:58,217] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,217] jepsen worker 34 - jepsen.util 754 :invoke :write [11 18] INFO [2022-03-13 17:14:58,220] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,221] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:58,235] jepsen worker 33 - jepsen.util 663 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,236] jepsen worker 33 - jepsen.util 663 :invoke :write [11 25] INFO [2022-03-13 17:14:58,245] jepsen worker 66 - jepsen.util 2856 :ok :read [22 30] INFO [2022-03-13 17:14:58,247] jepsen worker 66 - jepsen.util 2856 :invoke :write [22 3] INFO [2022-03-13 17:14:58,249] jepsen worker 67 - jepsen.util 2587 :ok :read [22 30] INFO [2022-03-13 17:14:58,250] jepsen worker 67 - jepsen.util 2587 :invoke :read [22 nil] INFO [2022-03-13 17:14:58,255] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,255] jepsen worker 79 - jepsen.util 1069 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,256] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:58,256] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] WARN [2022-03-13 17:14:58,270] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9038281,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9444373,"total_server_duration_us":9089213,"encode_duration_us":26692,"operation_id":"0xb9ee","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008727},{"operation_name":"upsert","last_dispatch_duration_us":8997482,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9164796,"total_server_duration_us":49504,"encode_duration_us":17774,"operation_id":"0xb494","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10008149},{"operation_name":"upsert","last_dispatch_duration_us":8348456,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9456040,"total_server_duration_us":218584,"encode_duration_us":30876,"operation_id":"0xb9ed","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10007562},{"operation_name":"upsert","last_dispatch_duration_us":9020777,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9157653,"total_server_duration_us":42821,"encode_duration_us":28879,"operation_id":"0xb4ad","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10006739},{"operation_name":"upsert","last_dispatch_duration_us":7943685,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":14944224,"total_server_duration_us":272793,"encode_duration_us":22654,"operation_id":"0xb63d","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10005990},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18103,"operation_id":"0xb60a","timeout_ms":10000,"total_duration_us":10005881},{"operation_name":"upsert","last_dispatch_duration_us":116210,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9886171,"total_server_duration_us":139491,"encode_duration_us":20167,"operation_id":"0xb465","timeout_ms":10000,"last_server_duration_us":26418,"total_duration_us":10005829},{"operation_name":"upsert","last_dispatch_duration_us":178868,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9768631,"total_server_duration_us":106950,"encode_duration_us":21183,"operation_id":"0xb468","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10005316},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":17876,"operation_id":"0xb58f","timeout_ms":10000,"total_duration_us":10005023},{"operation_name":"upsert","last_dispatch_duration_us":51164,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":701529,"total_server_duration_us":139404,"encode_duration_us":17559,"operation_id":"0xb9e7","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10004887}],"total_count":61}} INFO [2022-03-13 17:14:58,294] jepsen worker 68 - jepsen.util 3218 :ok :read [22 30] INFO [2022-03-13 17:14:58,294] jepsen worker 68 - jepsen.util 3218 :invoke :write [22 17] INFO [2022-03-13 17:14:58,300] jepsen worker 16 - jepsen.util 2806 :ok :read [5 43] INFO [2022-03-13 17:14:58,301] jepsen worker 16 - jepsen.util 2806 :invoke :write [5 23] INFO [2022-03-13 17:14:58,334] jepsen worker 40 - jepsen.util 3190 :ok :read [13 39] INFO [2022-03-13 17:14:58,335] jepsen worker 40 - jepsen.util 3190 :invoke :read [13 nil] INFO [2022-03-13 17:14:58,339] jepsen worker 67 - jepsen.util 2587 :ok :read [22 30] INFO [2022-03-13 17:14:58,340] jepsen worker 67 - jepsen.util 2587 :invoke :write [22 12] INFO [2022-03-13 17:14:58,349] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,362] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:14:58,363] jepsen worker 76 - jepsen.util 706 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,363] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,364] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,369] jepsen worker 12 - jepsen.util 732 :invoke :write [4 36] INFO [2022-03-13 17:14:58,370] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:14:58,370] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] WARN [2022-03-13 17:14:58,371] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18015854,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27310513,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9101495,"encode_duration_us":28325,"operation_id":"0xaf76","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10008389},{"operation_name":"upsert","last_dispatch_duration_us":9038281,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9444373,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9089213,"encode_duration_us":26692,"operation_id":"0xb9ee","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008727}],"total_count":2}} INFO [2022-03-13 17:14:58,383] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,384] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:58,413] jepsen worker 29 - jepsen.util 3089 :info :write [9 1] :SyncWriteAmbiguous INFO [2022-03-13 17:14:58,413] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,413] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,414] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,415] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:58,415] jepsen worker 29 - jepsen.util 3179 :invoke :read [9 nil] INFO [2022-03-13 17:14:58,415] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:58,416] jepsen worker 21 - jepsen.util 831 :invoke :write [7 33] INFO [2022-03-13 17:14:58,418] jepsen worker 45 - jepsen.util 3195 :ok :read [15 48] INFO [2022-03-13 17:14:58,419] jepsen worker 45 - jepsen.util 3195 :invoke :read [15 nil] INFO [2022-03-13 17:14:58,446] jepsen worker 55 - jepsen.util 2755 :info :write [18 30] :RequestTimeoutException INFO [2022-03-13 17:14:58,448] jepsen worker 55 - jepsen.util 2845 :invoke :write [18 35] INFO [2022-03-13 17:14:58,455] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,456] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:14:58,487] jepsen worker 45 - jepsen.util 3195 :ok :read [15 48] INFO [2022-03-13 17:14:58,489] jepsen worker 45 - jepsen.util 3195 :invoke :write [15 15] INFO [2022-03-13 17:14:58,499] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,499] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,499] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,500] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:14:58,501] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:58,501] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:14:58,504] jepsen worker 40 - jepsen.util 3190 :ok :read [13 39] INFO [2022-03-13 17:14:58,505] jepsen worker 40 - jepsen.util 3190 :invoke :write [13 2] INFO [2022-03-13 17:14:58,547] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,548] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:14:58,582] jepsen worker 20 - jepsen.util 3350 :ok :read [6 16] INFO [2022-03-13 17:14:58,583] jepsen worker 20 - jepsen.util 3350 :invoke :write [6 46] INFO [2022-03-13 17:14:58,616] jepsen worker 69 - jepsen.util 699 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,616] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:58,616] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,617] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:58,616] jepsen worker 7 - jepsen.util 637 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,617] jepsen worker 62 - jepsen.util 512 :invoke :write [20 36] INFO [2022-03-13 17:14:58,618] jepsen worker 7 - jepsen.util 637 :invoke :write [2 48] INFO [2022-03-13 17:14:58,619] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:58,620] jepsen worker 69 - jepsen.util 699 :invoke :write [23 16] INFO [2022-03-13 17:14:58,623] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:58,634] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,634] jepsen worker 21 - jepsen.util 831 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,634] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:14:58,635] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:14:58,697] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,697] jepsen worker 24 - jepsen.util 564 :invoke :write [8 5] INFO [2022-03-13 17:14:58,756] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,757] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:58,769] jepsen worker 8 - jepsen.util 728 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,769] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,770] jepsen worker 8 - jepsen.util 728 :invoke :write [2 13] INFO [2022-03-13 17:14:58,770] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:14:58,778] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,779] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:58,785] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,786] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:58,807] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,807] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,807] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:14:58,808] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:14:58,865] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,865] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:58,871] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,871] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,871] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:14:58,872] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:14:58,878] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,878] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:14:58,904] jepsen worker 81 - jepsen.util 3321 :info :write [27 26] :RequestTimeoutException INFO [2022-03-13 17:14:58,905] jepsen worker 81 - jepsen.util 3411 :invoke :write [27 7] INFO [2022-03-13 17:14:58,906] jepsen worker 33 - jepsen.util 663 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,907] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:58,920] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,920] jepsen worker 51 - jepsen.util 591 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,921] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:14:58,921] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:14:58,940] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,940] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:14:58,955] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,956] jepsen worker 14 - jepsen.util 374 :invoke :write [4 30] INFO [2022-03-13 17:14:58,961] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,962] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:14:58,970] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,971] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:58,980] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:14:58,981] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:14:59,013] jepsen worker 22 - jepsen.util 652 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,013] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:59,048] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,048] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,049] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,049] jepsen worker 37 - jepsen.util 2107 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,049] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,049] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:14:59,049] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:59,050] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:14:59,050] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,051] jepsen worker 35 - jepsen.util 935 :invoke :write [11 17] INFO [2022-03-13 17:14:59,051] jepsen worker 25 - jepsen.util 565 :invoke :write [8 3] INFO [2022-03-13 17:14:59,051] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:14:59,061] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,062] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:59,103] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,103] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:14:59,137] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,138] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,138] jepsen worker 53 - jepsen.util 503 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,138] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:14:59,139] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:14:59,139] jepsen worker 53 - jepsen.util 503 :invoke :write [17 10] INFO [2022-03-13 17:14:59,140] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:14:59,140] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:14:59,140] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:14:59,141] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:14:59,149] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,150] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,164] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,165] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:14:59,180] jepsen worker 77 - jepsen.util 707 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,180] jepsen worker 14 - jepsen.util 374 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,181] jepsen worker 77 - jepsen.util 707 :invoke :write [25 42] INFO [2022-03-13 17:14:59,184] jepsen worker 80 - jepsen.util 710 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,185] jepsen worker 80 - jepsen.util 710 :invoke :write [26 28] INFO [2022-03-13 17:14:59,188] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,189] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:14:59,195] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:14:59,204] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,204] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,204] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,204] jepsen worker 24 - jepsen.util 564 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,205] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,205] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:14:59,207] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:59,208] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:59,208] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:59,209] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:59,209] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:14:59,209] jepsen worker nemesis - couchbase.util Rebalance check count: 408 INFO [2022-03-13 17:14:59,209] jepsen worker nemesis - couchbase.util Rebalance stuck count: 308 INFO [2022-03-13 17:14:59,216] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,216] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,216] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,216] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,217] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:14:59,219] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:14:59,220] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:14:59,221] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,221] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,221] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,221] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:59,222] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,222] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:14:59,222] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,223] jepsen worker 35 - jepsen.util 935 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,223] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:14:59,225] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:59,225] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,225] jepsen worker 33 - jepsen.util 663 :invoke :write [11 18] INFO [2022-03-13 17:14:59,226] jepsen worker 0 - jepsen.util 2700 :info :write [0 33] :RequestTimeoutException INFO [2022-03-13 17:14:59,226] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,227] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,227] jepsen worker 0 - jepsen.util 2790 :invoke :write [0 41] INFO [2022-03-13 17:14:59,229] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,234] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,235] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,239] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,240] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,241] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,241] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,242] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,242] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,244] jepsen worker 83 - jepsen.util 2333 :info :write [27 39] :RequestTimeoutException INFO [2022-03-13 17:14:59,244] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,244] jepsen worker 83 - jepsen.util 2423 :invoke :write [27 7] INFO [2022-03-13 17:14:59,250] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,250] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,251] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:14:59,251] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,252] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,253] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,253] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:14:59,253] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,254] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:14:59,276] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,277] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 30] INFO [2022-03-13 17:14:59,289] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,290] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,347] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,347] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:14:59,368] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,369] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,372] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,373] jepsen worker 13 - jepsen.util 643 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,373] jepsen worker 21 - jepsen.util 831 :invoke :write [7 47] INFO [2022-03-13 17:14:59,375] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:14:59,383] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,384] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,384] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:14:59,385] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:14:59,391] jepsen worker 62 - jepsen.util 512 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,391] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:14:59,395] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,396] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:14:59,447] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,447] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,447] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,448] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,448] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,449] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:14:59,449] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,449] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:14:59,450] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,450] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,451] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,451] jepsen worker 25 - jepsen.util 565 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,452] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,452] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:14:59,452] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,453] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,453] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,454] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:14:59,454] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,455] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,460] jepsen worker 64 - jepsen.util 2854 :info :write [21 2] :RequestTimeoutException INFO [2022-03-13 17:14:59,460] jepsen worker 64 - jepsen.util 2944 :invoke :read [21 nil] INFO [2022-03-13 17:14:59,464] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,464] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,464] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,465] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:14:59,465] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,466] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,467] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,467] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:14:59,481] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,482] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,486] jepsen worker 56 - jepsen.util 3116 :info :write [18 26] :RequestTimeoutException INFO [2022-03-13 17:14:59,486] jepsen worker 56 - jepsen.util 3206 :invoke :write [18 34] INFO [2022-03-13 17:14:59,491] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,491] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:14:59,494] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,495] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:59,562] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,562] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,562] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:14:59,563] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 37] INFO [2022-03-13 17:14:59,564] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,565] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,578] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,578] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,590] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,590] jepsen worker 60 - jepsen.util 600 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,591] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:14:59,591] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,591] jepsen worker 60 - jepsen.util 600 :invoke :write [20 2] INFO [2022-03-13 17:14:59,591] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,595] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:59,595] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,596] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:14:59,596] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:14:59,615] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,615] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:14:59,651] jepsen worker 89 - jepsen.util 2519 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,651] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,652] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,652] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:14:59,657] jepsen worker 47 - jepsen.util 2837 :info :write [15 14] :RequestTimeoutException INFO [2022-03-13 17:14:59,658] jepsen worker 47 - jepsen.util 2927 :invoke :read [15 nil] INFO [2022-03-13 17:14:59,659] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,660] jepsen worker 70 - jepsen.util 700 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,660] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,660] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 26] INFO [2022-03-13 17:14:59,661] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:59,661] jepsen worker 70 - jepsen.util 700 :invoke :write [23 23] INFO [2022-03-13 17:14:59,671] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,672] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:59,674] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,675] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,679] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,680] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:14:59,682] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,683] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:14:59,706] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,706] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,706] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:14:59,707] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:14:59,715] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,716] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:14:59,718] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,718] jepsen worker 22 - jepsen.util 652 :invoke :write [7 31] INFO [2022-03-13 17:14:59,720] jepsen worker 9 - jepsen.util 2439 :info :write [3 31] :RequestTimeoutException INFO [2022-03-13 17:14:59,721] jepsen worker 9 - jepsen.util 2529 :invoke :read [3 nil] INFO [2022-03-13 17:14:59,723] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,723] jepsen worker 21 - jepsen.util 831 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,724] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,726] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:59,727] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:14:59,728] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:14:59,730] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,731] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:14:59,743] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,744] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:59,755] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,755] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:14:59,760] jepsen worker 9 - jepsen.util 2529 :ok :read [3 25] INFO [2022-03-13 17:14:59,761] jepsen worker 9 - jepsen.util 2529 :invoke :write [3 22] INFO [2022-03-13 17:14:59,787] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,787] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,787] jepsen worker 44 - jepsen.util 584 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,787] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,787] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,787] jepsen worker 7 - jepsen.util 637 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,791] jepsen worker 6 - jepsen.util 636 :invoke :write [2 2] INFO [2022-03-13 17:14:59,792] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:14:59,793] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:14:59,793] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:14:59,794] jepsen worker 7 - jepsen.util 637 :invoke :write [2 21] INFO [2022-03-13 17:14:59,794] jepsen worker 76 - jepsen.util 706 :invoke :write [25 24] INFO [2022-03-13 17:14:59,799] jepsen worker 78 - jepsen.util 528 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,800] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:14:59,841] jepsen worker 33 - jepsen.util 663 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,841] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,842] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,842] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:14:59,843] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 34] INFO [2022-03-13 17:14:59,843] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:14:59,852] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,852] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,853] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:14:59,853] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:14:59,871] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,872] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:14:59,889] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,890] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:14:59,904] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,904] jepsen worker 37 - jepsen.util 2107 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,905] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 15] INFO [2022-03-13 17:14:59,907] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:59,913] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,913] jepsen worker 71 - jepsen.util 521 :invoke :write [23 42] INFO [2022-03-13 17:14:59,922] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,922] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:14:59,954] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,955] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:14:59,978] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,978] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,979] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:14:59,979] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:14:59,980] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:14:59,980] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,036] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,036] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,036] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,036] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,037] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,037] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,037] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:15:00,038] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:00,038] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:15:00,039] jepsen worker 33 - jepsen.util 663 :invoke :write [11 9] INFO [2022-03-13 17:15:00,039] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:15:00,040] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:00,043] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,044] jepsen worker 14 - jepsen.util 374 :invoke :write [4 24] INFO [2022-03-13 17:15:00,069] jepsen worker 88 - jepsen.util 2068 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,070] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,072] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,073] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:15:00,120] jepsen worker 12 - jepsen.util 732 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,120] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,120] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:00,120] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,120] jepsen worker 8 - jepsen.util 728 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,120] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,120] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:00,120] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,121] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,121] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:15:00,122] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 1] INFO [2022-03-13 17:15:00,123] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:15:00,124] jepsen worker 43 - jepsen.util 583 :invoke :write [14 26] INFO [2022-03-13 17:15:00,126] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:00,126] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,127] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:00,128] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:00,128] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,128] jepsen worker 89 - jepsen.util 2519 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,129] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,130] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,130] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,132] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:15:00,134] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:15:00,136] jepsen worker 34 - jepsen.util 754 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,136] jepsen worker 22 - jepsen.util 652 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,137] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:15:00,138] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:15:00,151] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,151] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 28] INFO [2022-03-13 17:15:00,169] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,170] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,171] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:15:00,171] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:00,184] jepsen worker 54 - jepsen.util 2574 :info :write [18 37] :RequestTimeoutException INFO [2022-03-13 17:15:00,184] jepsen worker 54 - jepsen.util 2664 :invoke :read [18 nil] INFO [2022-03-13 17:15:00,190] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,191] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:15:00,208] jepsen worker 37 - jepsen.util 2107 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,208] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,208] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 34] INFO [2022-03-13 17:15:00,211] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:15:00,211] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,212] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,213] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,214] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:15:00,214] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:00,214] jepsen worker nemesis - couchbase.util Rebalance check count: 409 INFO [2022-03-13 17:15:00,214] jepsen worker nemesis - couchbase.util Rebalance stuck count: 309 INFO [2022-03-13 17:15:00,218] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,218] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 41] INFO [2022-03-13 17:15:00,237] jepsen worker 3 - jepsen.util 1983 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,238] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:15:00,251] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,252] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:15:00,260] jepsen worker 71 - jepsen.util 521 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,260] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,261] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:00,261] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:15:00,313] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,313] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:15:00,314] jepsen worker 37 - jepsen.util 2107 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,315] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,334] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,334] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:15:00,374] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,375] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:15:00,409] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,410] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,424] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,424] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:00,428] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,429] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:15:00,444] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,444] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,444] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,445] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,450] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,450] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,450] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:15:00,451] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,452] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,453] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 4] INFO [2022-03-13 17:15:00,479] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,480] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:15:00,487] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,488] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:15:00,505] jepsen worker 89 - jepsen.util 2519 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,505] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,534] jepsen worker 53 - jepsen.util 503 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,534] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,535] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:00,535] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:00,535] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,536] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:15:00,540] jepsen worker 52 - jepsen.util 592 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,541] jepsen worker 52 - jepsen.util 592 :invoke :write [17 22] INFO [2022-03-13 17:15:00,550] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,550] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:00,550] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,550] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,551] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:15:00,552] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,552] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,553] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:00,554] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,555] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 1] INFO [2022-03-13 17:15:00,594] jepsen worker 69 - jepsen.util 699 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,594] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,594] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,595] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:00,595] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:15:00,596] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:00,617] jepsen worker 14 - jepsen.util 374 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,618] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:00,624] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,624] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,624] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,625] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 39] INFO [2022-03-13 17:15:00,627] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,628] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:15:00,630] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,631] jepsen worker 33 - jepsen.util 663 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,631] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,631] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:15:00,631] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,631] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:15:00,632] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:00,638] jepsen worker 33 - jepsen.util 663 :invoke :write [11 39] INFO [2022-03-13 17:15:00,639] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,640] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,683] jepsen worker 6 - jepsen.util 636 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,683] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:00,686] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,687] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:00,688] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,688] jepsen worker 80 - jepsen.util 710 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,688] jepsen worker 13 - jepsen.util 643 :invoke :write [4 16] INFO [2022-03-13 17:15:00,689] jepsen worker 80 - jepsen.util 710 :invoke :write [26 14] INFO [2022-03-13 17:15:00,690] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,690] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,691] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:15:00,691] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:00,713] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,713] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,714] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,714] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,723] jepsen worker 3 - jepsen.util 1983 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,724] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:00,726] jepsen worker 11 - jepsen.util 2981 :info :write [3 45] :SyncWriteAmbiguous INFO [2022-03-13 17:15:00,726] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,727] jepsen worker 11 - jepsen.util 3071 :invoke :write [3 9] INFO [2022-03-13 17:15:00,728] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:15:00,742] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,743] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:15:00,748] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,752] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:00,755] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,756] jepsen worker 72 - jepsen.util 3042 :ok :read [24 24] INFO [2022-03-13 17:15:00,756] jepsen worker 22 - jepsen.util 652 :invoke :write [7 45] INFO [2022-03-13 17:15:00,759] jepsen worker 72 - jepsen.util 3042 :invoke :write [24 12] INFO [2022-03-13 17:15:00,766] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,766] jepsen worker 77 - jepsen.util 707 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,767] jepsen worker 21 - jepsen.util 831 :invoke :write [7 33] INFO [2022-03-13 17:15:00,767] jepsen worker 77 - jepsen.util 707 :invoke :write [25 11] INFO [2022-03-13 17:15:00,775] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,776] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:15:00,795] jepsen worker 65 - jepsen.util 3395 :ok :read [21 :nil] INFO [2022-03-13 17:15:00,797] jepsen worker 65 - jepsen.util 3395 :invoke :read [21 nil] INFO [2022-03-13 17:15:00,798] jepsen worker 64 - jepsen.util 2944 :ok :read [21 :nil] INFO [2022-03-13 17:15:00,799] jepsen worker 64 - jepsen.util 2944 :invoke :write [21 0] INFO [2022-03-13 17:15:00,828] jepsen worker 65 - jepsen.util 3395 :ok :read [21 :nil] INFO [2022-03-13 17:15:00,829] jepsen worker 65 - jepsen.util 3395 :invoke :write [21 21] INFO [2022-03-13 17:15:00,837] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,837] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,837] jepsen worker 70 - jepsen.util 700 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,838] jepsen worker 34 - jepsen.util 754 :invoke :write [11 19] INFO [2022-03-13 17:15:00,839] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:15:00,839] jepsen worker 78 - jepsen.util 528 :invoke :write [26 49] INFO [2022-03-13 17:15:00,841] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,841] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,841] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,841] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:15:00,844] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,844] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:00,845] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,846] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:15:00,846] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,846] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,848] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:15:00,848] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:00,850] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,851] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:15:00,851] jepsen worker 37 - jepsen.util 2107 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,852] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 24] INFO [2022-03-13 17:15:00,863] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,863] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:15:00,899] jepsen worker 7 - jepsen.util 637 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,900] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:15:00,903] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,903] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,903] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,903] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:00,904] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:15:00,904] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:00,925] jepsen worker 89 - jepsen.util 2519 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,925] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,926] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,926] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:15:00,927] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:15:00,927] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:00,930] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,931] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:15:00,966] jepsen worker 33 - jepsen.util 663 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,966] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,966] jepsen worker 33 - jepsen.util 663 :invoke :write [11 48] INFO [2022-03-13 17:15:00,967] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:15:00,968] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:00,968] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,968] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:00,969] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:00,969] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,970] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,970] jepsen worker 60 - jepsen.util 600 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,970] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:00,970] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:00,971] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:00,990] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,991] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:00,993] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:00,993] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:15:01,008] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,008] jepsen worker 21 - jepsen.util 831 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,008] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,009] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:15:01,009] jepsen worker 21 - jepsen.util 831 :invoke :write [7 9] INFO [2022-03-13 17:15:01,010] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,023] jepsen worker 52 - jepsen.util 592 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,023] jepsen worker 22 - jepsen.util 652 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,023] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,024] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:01,024] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:01,025] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:15:01,026] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,026] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:15:01,029] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,030] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:15:01,077] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,077] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:15:01,101] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,102] jepsen worker 24 - jepsen.util 564 :invoke :write [8 18] INFO [2022-03-13 17:15:01,107] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,108] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,137] jepsen worker 37 - jepsen.util 2107 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,138] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:01,163] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,163] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,164] jepsen worker 14 - jepsen.util 374 :invoke :write [4 22] INFO [2022-03-13 17:15:01,165] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:15:01,171] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,172] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:15:01,172] jepsen worker 80 - jepsen.util 710 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,173] jepsen worker 80 - jepsen.util 710 :invoke :write [26 49] INFO [2022-03-13 17:15:01,185] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,187] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,204] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,204] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:01,220] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:01,220] jepsen worker nemesis - couchbase.util Rebalance check count: 410 INFO [2022-03-13 17:15:01,220] jepsen worker nemesis - couchbase.util Rebalance stuck count: 310 INFO [2022-03-13 17:15:01,241] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,242] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,244] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,245] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:15:01,246] jepsen worker 43 - jepsen.util 583 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,246] jepsen worker 76 - jepsen.util 706 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,247] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,247] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,247] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,247] jepsen worker 76 - jepsen.util 706 :invoke :write [25 4] INFO [2022-03-13 17:15:01,248] jepsen worker 12 - jepsen.util 732 :invoke :write [4 38] INFO [2022-03-13 17:15:01,249] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:15:01,250] jepsen worker 43 - jepsen.util 583 :invoke :write [14 22] INFO [2022-03-13 17:15:01,251] jepsen worker 23 - jepsen.util 563 :invoke :write [7 38] INFO [2022-03-13 17:15:01,254] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,256] jepsen worker 69 - jepsen.util 699 :invoke :write [23 36] INFO [2022-03-13 17:15:01,261] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,261] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,261] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,262] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:15:01,263] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,264] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:15:01,271] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,272] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:15:01,287] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,287] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,299] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,299] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,300] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,301] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:15:01,302] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,303] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:15:01,307] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,308] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:15:01,310] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,310] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:01,356] jepsen worker 13 - jepsen.util 643 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,357] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,357] jepsen worker 70 - jepsen.util 700 :invoke :write [23 11] INFO [2022-03-13 17:15:01,358] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:01,375] jepsen worker 79 - jepsen.util 1069 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,376] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 29] INFO [2022-03-13 17:15:01,381] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,381] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,381] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,382] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,382] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:15:01,383] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 2] INFO [2022-03-13 17:15:01,383] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:15:01,390] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 38] INFO [2022-03-13 17:15:01,397] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,397] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,410] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,410] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,415] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,416] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,422] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,423] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:15:01,433] jepsen worker 33 - jepsen.util 663 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,434] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,444] jepsen worker 21 - jepsen.util 831 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,445] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:01,465] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,466] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,467] jepsen worker 77 - jepsen.util 707 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,467] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:01,467] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,467] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,467] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,468] jepsen worker 44 - jepsen.util 584 :invoke :write [14 44] INFO [2022-03-13 17:15:01,470] jepsen worker 51 - jepsen.util 591 :invoke :write [17 25] INFO [2022-03-13 17:15:01,470] jepsen worker 77 - jepsen.util 707 :invoke :write [25 25] INFO [2022-03-13 17:15:01,472] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:01,472] jepsen worker 61 - jepsen.util 691 :invoke :write [20 11] INFO [2022-03-13 17:15:01,496] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,497] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,501] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,501] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:15:01,504] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,504] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:15:01,547] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,547] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,547] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,548] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,548] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:15:01,549] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,549] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,550] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,550] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:15:01,551] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,552] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,552] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:15:01,557] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,558] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:15:01,582] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,583] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:15:01,584] jepsen worker 34 - jepsen.util 754 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,584] jepsen worker 34 - jepsen.util 754 :invoke :write [11 48] INFO [2022-03-13 17:15:01,592] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,592] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:01,633] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,633] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,636] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,637] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,638] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,638] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:15:01,640] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,641] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,657] jepsen worker 89 - jepsen.util 2519 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,657] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,694] jepsen worker 46 - jepsen.util 2926 :info :write [15 8] :RequestTimeoutException INFO [2022-03-13 17:15:01,695] jepsen worker 46 - jepsen.util 3016 :invoke :read [15 nil] INFO [2022-03-13 17:15:01,705] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,706] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,707] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,708] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,709] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,709] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,713] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,713] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,714] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,715] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:15:01,759] jepsen worker 37 - jepsen.util 2107 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,759] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:15:01,782] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,783] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 18] INFO [2022-03-13 17:15:01,796] jepsen worker 49 - jepsen.util 2749 :info :write [16 47] :RequestTimeoutException INFO [2022-03-13 17:15:01,797] jepsen worker 49 - jepsen.util 2839 :invoke :write [16 36] INFO [2022-03-13 17:15:01,801] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,801] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,802] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,802] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:15:01,805] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,806] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,809] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,810] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,810] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,811] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:15:01,811] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,812] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:15:01,836] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,837] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:01,838] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,838] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:01,839] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,839] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:01,839] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,840] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 28] INFO [2022-03-13 17:15:01,843] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,844] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:01,844] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,845] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:15:01,846] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,847] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:01,848] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,849] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:15:01,855] jepsen worker 14 - jepsen.util 374 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,856] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:01,868] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,869] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:15:01,884] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,884] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,902] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,903] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:15:01,912] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,913] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,913] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,913] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,914] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,914] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,914] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:01,914] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,914] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:01,914] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,915] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:15:01,915] jepsen worker 26 - jepsen.util 476 :invoke :write [8 31] INFO [2022-03-13 17:15:01,916] jepsen worker 53 - jepsen.util 503 :invoke :write [17 15] INFO [2022-03-13 17:15:01,917] jepsen worker 8 - jepsen.util 728 :invoke :write [2 31] INFO [2022-03-13 17:15:01,917] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:01,921] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:15:01,937] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,937] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,938] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:15:01,939] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,939] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,941] jepsen worker 3 - jepsen.util 1983 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,943] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,943] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:01,944] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,944] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:01,944] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,945] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,945] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,945] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:01,945] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,945] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,946] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:01,946] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,946] jepsen worker 24 - jepsen.util 564 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,947] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,947] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,947] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,947] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:01,947] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,948] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:15:01,948] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,948] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:15:01,948] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,949] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:15:01,950] jepsen worker 75 - jepsen.util 435 :invoke :write [25 11] INFO [2022-03-13 17:15:01,951] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:15:01,951] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:01,952] jepsen worker 14 - jepsen.util 374 :invoke :write [4 30] INFO [2022-03-13 17:15:01,952] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,953] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:01,954] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,955] jepsen worker 24 - jepsen.util 564 :invoke :write [8 29] INFO [2022-03-13 17:15:01,958] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,959] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,961] jepsen worker 78 - jepsen.util 528 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,961] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,963] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,963] jepsen worker 78 - jepsen.util 528 :invoke :write [26 32] INFO [2022-03-13 17:15:01,971] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:01,972] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,973] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:01,975] jepsen worker 76 - jepsen.util 706 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,976] jepsen worker 76 - jepsen.util 706 :invoke :write [25 7] INFO [2022-03-13 17:15:01,995] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:01,996] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:15:02,014] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,014] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,016] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,016] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:15:02,041] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,041] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,042] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 33] INFO [2022-03-13 17:15:02,042] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:02,081] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,082] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:15:02,093] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,094] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,098] jepsen worker 75 - jepsen.util 435 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,099] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,099] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:02,100] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,100] jepsen worker 89 - jepsen.util 2519 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,100] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 21] INFO [2022-03-13 17:15:02,101] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 19] INFO [2022-03-13 17:15:02,101] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:02,111] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,111] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:15:02,120] jepsen worker 43 - jepsen.util 583 :fail :write [14 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,121] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:02,176] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,176] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,177] jepsen worker 12 - jepsen.util 732 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,177] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,177] jepsen worker 12 - jepsen.util 732 :invoke :write [4 45] INFO [2022-03-13 17:15:02,178] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,179] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,180] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,181] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,182] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:15:02,192] jepsen worker 80 - jepsen.util 710 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,192] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,193] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:15:02,194] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:02,198] jepsen worker 14 - jepsen.util 374 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,198] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:15:02,215] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,216] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:15:02,227] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:02,227] jepsen worker nemesis - couchbase.util Rebalance check count: 411 INFO [2022-03-13 17:15:02,227] jepsen worker nemesis - couchbase.util Rebalance stuck count: 311 INFO [2022-03-13 17:15:02,230] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,231] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:15:02,240] jepsen worker 3 - jepsen.util 1983 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,241] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 13] INFO [2022-03-13 17:15:02,244] jepsen worker 24 - jepsen.util 564 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,244] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,244] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:02,245] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:02,304] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,304] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:15:02,305] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,305] jepsen worker 37 - jepsen.util 2107 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,306] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 27] INFO [2022-03-13 17:15:02,306] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:15:02,313] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,313] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,314] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:02,314] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,318] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,319] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:15:02,378] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,379] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:02,383] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,383] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,384] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,384] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,384] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:15:02,385] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:02,385] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:02,386] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:02,386] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,386] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:02,387] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,387] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,387] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,387] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,388] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,388] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,388] jepsen worker 70 - jepsen.util 700 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,389] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,389] jepsen worker 33 - jepsen.util 663 :invoke :write [11 28] INFO [2022-03-13 17:15:02,391] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:15:02,391] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:15:02,393] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:02,394] jepsen worker 3 - jepsen.util 1983 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,394] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:15:02,398] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:15:02,398] jepsen worker 89 - jepsen.util 2519 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,399] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:02,402] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:02,403] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,405] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:15:02,406] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,406] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,407] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:15:02,417] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:15:02,440] jepsen worker 23 - jepsen.util 563 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,440] jepsen worker 79 - jepsen.util 1069 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,440] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,440] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,441] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,441] jepsen worker 23 - jepsen.util 563 :invoke :write [7 45] INFO [2022-03-13 17:15:02,441] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:02,442] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:02,442] jepsen worker 6 - jepsen.util 636 :invoke :write [2 41] INFO [2022-03-13 17:15:02,443] jepsen worker 7 - jepsen.util 637 :invoke :write [2 12] INFO [2022-03-13 17:15:02,459] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,460] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,463] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,463] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,468] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,469] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:02,475] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,475] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:15:02,489] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,489] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,490] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:15:02,490] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:02,504] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,505] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:15:02,512] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,513] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:15:02,516] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,517] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:02,518] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,519] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:15:02,523] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,530] jepsen worker 71 - jepsen.util 521 :invoke :write [23 40] INFO [2022-03-13 17:15:02,535] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,536] jepsen worker 14 - jepsen.util 374 :invoke :write [4 37] INFO [2022-03-13 17:15:02,543] jepsen worker 37 - jepsen.util 2107 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,543] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,544] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 18] INFO [2022-03-13 17:15:02,545] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:02,561] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,562] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:15:02,583] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,583] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:15:02,603] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,604] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:15:02,617] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,618] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:15:02,661] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,661] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,661] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:15:02,662] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 21] INFO [2022-03-13 17:15:02,675] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,676] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:15:02,684] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,685] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:02,695] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,696] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:02,730] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,731] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 13] INFO [2022-03-13 17:15:02,771] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,772] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:15:02,775] jepsen worker 37 - jepsen.util 2107 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,776] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:02,980] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,980] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:15:02,988] jepsen worker 85 - jepsen.util 2785 :info :write [28 42] :RequestTimeoutException INFO [2022-03-13 17:15:02,989] jepsen worker 85 - jepsen.util 2875 :invoke :read [28 nil] INFO [2022-03-13 17:15:02,992] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,992] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:02,993] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:02,993] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 26] INFO [2022-03-13 17:15:03,000] jepsen worker 41 - jepsen.util 2921 :info :write [13 49] :RequestTimeoutException INFO [2022-03-13 17:15:03,001] jepsen worker 41 - jepsen.util 3011 :invoke :read [13 nil] INFO [2022-03-13 17:15:03,004] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,004] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,004] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,005] jepsen worker 13 - jepsen.util 643 :invoke :write [4 20] INFO [2022-03-13 17:15:03,006] jepsen worker 80 - jepsen.util 710 :invoke :write [26 4] INFO [2022-03-13 17:15:03,006] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:15:03,023] jepsen worker 58 - jepsen.util 2668 :info :write [19 2] :RequestTimeoutException INFO [2022-03-13 17:15:03,024] jepsen worker 58 - jepsen.util 2758 :invoke :write [19 25] INFO [2022-03-13 17:15:03,036] jepsen worker 50 - jepsen.util 3110 :info :write [16 41] :RequestTimeoutException INFO [2022-03-13 17:15:03,037] jepsen worker 50 - jepsen.util 3200 :invoke :read [16 nil] INFO [2022-03-13 17:15:03,077] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,077] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,097] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,098] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 9] INFO [2022-03-13 17:15:03,114] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,114] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,115] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:03,115] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:15:03,143] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,143] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,144] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 37] INFO [2022-03-13 17:15:03,144] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:03,192] jepsen worker 3 - jepsen.util 1983 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,192] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:15:03,202] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:03,202] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,203] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,203] jepsen worker 43 - jepsen.util 583 :invoke :write [14 48] INFO [2022-03-13 17:15:03,204] jepsen worker 62 - jepsen.util 512 :invoke :write [20 29] INFO [2022-03-13 17:15:03,204] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 34] INFO [2022-03-13 17:15:03,233] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:03,234] jepsen worker nemesis - couchbase.util Rebalance check count: 412 INFO [2022-03-13 17:15:03,234] jepsen worker nemesis - couchbase.util Rebalance stuck count: 312 INFO [2022-03-13 17:15:03,242] jepsen worker 5 - jepsen.util 2435 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,242] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:15:03,274] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,275] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:15:03,277] jepsen worker 89 - jepsen.util 2519 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,277] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,278] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,278] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:15:03,283] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,284] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:15:03,305] jepsen worker 6 - jepsen.util 636 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,306] jepsen worker 6 - jepsen.util 636 :invoke :write [2 17] INFO [2022-03-13 17:15:03,337] jepsen worker 12 - jepsen.util 732 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,337] jepsen worker 12 - jepsen.util 732 :invoke :write [4 4] INFO [2022-03-13 17:15:03,361] jepsen worker 37 - jepsen.util 2107 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,361] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,362] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:03,363] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 46] INFO [2022-03-13 17:15:03,395] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,396] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,398] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,398] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,404] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,405] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:15:03,448] jepsen worker 10 - jepsen.util 2980 :info :write [3 3] :RequestTimeoutException INFO [2022-03-13 17:15:03,449] jepsen worker 10 - jepsen.util 3070 :invoke :read [3 nil] INFO [2022-03-13 17:15:03,460] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,460] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:15:03,466] jepsen worker 88 - jepsen.util 2068 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,467] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:03,473] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,474] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:15:03,478] jepsen worker 18 - jepsen.util 2718 :info :write [6 46] :RequestTimeoutException INFO [2022-03-13 17:15:03,479] jepsen worker 18 - jepsen.util 2808 :invoke :write [6 12] INFO [2022-03-13 17:15:03,479] jepsen worker 14 - jepsen.util 374 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,480] jepsen worker 14 - jepsen.util 374 :invoke :write [4 16] INFO [2022-03-13 17:15:03,498] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,499] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,499] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 18] INFO [2022-03-13 17:15:03,500] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:03,537] jepsen worker 32 - jepsen.util 2912 :info :write [10 19] :SyncWriteAmbiguous INFO [2022-03-13 17:15:03,538] jepsen worker 27 - jepsen.util 3447 :info :write [9 2] :RequestTimeoutException INFO [2022-03-13 17:15:03,538] jepsen worker 32 - jepsen.util 3002 :invoke :read [10 nil] INFO [2022-03-13 17:15:03,538] jepsen worker 27 - jepsen.util 3537 :invoke :read [9 nil] INFO [2022-03-13 17:15:03,543] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,544] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:15:03,547] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,547] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,548] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,548] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:15:03,558] jepsen worker 63 - jepsen.util 2943 :info :write [21 1] :RequestTimeoutException INFO [2022-03-13 17:15:03,559] jepsen worker 63 - jepsen.util 3033 :invoke :write [21 32] INFO [2022-03-13 17:15:03,567] jepsen worker 7 - jepsen.util 637 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,568] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:15:03,580] jepsen worker 30 - jepsen.util 2820 :ok :read [10 :nil] INFO [2022-03-13 17:15:03,581] jepsen worker 30 - jepsen.util 2820 :invoke :write [10 35] INFO [2022-03-13 17:15:03,601] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,601] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,602] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,602] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:15:03,618] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,619] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,619] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:03,620] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:03,645] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,645] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,647] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,648] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,649] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,649] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:15:03,672] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,672] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,672] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:03,673] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:15:03,682] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,683] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,685] jepsen worker 71 - jepsen.util 521 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,685] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:15:03,685] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,686] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,688] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,688] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,689] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,690] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,693] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,694] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:15:03,698] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,699] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,699] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,699] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,729] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,729] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,734] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,735] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:03,735] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,736] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:15:03,737] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,738] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:03,745] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,745] jepsen worker 23 - jepsen.util 563 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,746] jepsen worker 60 - jepsen.util 600 :invoke :write [20 3] INFO [2022-03-13 17:15:03,746] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:03,749] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,749] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:15:03,754] jepsen worker 89 - jepsen.util 2519 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,755] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,757] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,758] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:15:03,768] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,769] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:03,784] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,784] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,784] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,785] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,786] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,786] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,787] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 31] INFO [2022-03-13 17:15:03,787] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,788] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:15:03,788] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:03,790] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,791] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:03,809] jepsen worker 37 - jepsen.util 2107 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,809] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,810] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,811] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:15:03,811] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,811] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,812] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 24] INFO [2022-03-13 17:15:03,812] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:03,813] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,813] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,814] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:15:03,814] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:15:03,838] jepsen worker 2 - jepsen.util 3422 :info :write [0 16] :RequestTimeoutException INFO [2022-03-13 17:15:03,839] jepsen worker 2 - jepsen.util 3512 :invoke :write [0 15] INFO [2022-03-13 17:15:03,855] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,856] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:15:03,887] jepsen worker 74 - jepsen.util 3134 :info :write [24 46] :RequestTimeoutException INFO [2022-03-13 17:15:03,888] jepsen worker 74 - jepsen.util 3224 :invoke :write [24 27] INFO [2022-03-13 17:15:03,923] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,924] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:03,974] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,975] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,977] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,977] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:15:03,981] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,981] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:03,994] jepsen worker 14 - jepsen.util 374 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,994] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:03,994] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:15:03,995] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:15:04,003] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,003] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:04,005] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,005] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:04,006] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,007] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:04,008] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,008] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:15:04,045] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,045] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:04,054] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,055] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:15:04,064] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,065] jepsen worker 42 - jepsen.util 402 :invoke :write [14 12] INFO [2022-03-13 17:15:04,098] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,098] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,099] jepsen worker 21 - jepsen.util 831 :invoke :write [7 44] INFO [2022-03-13 17:15:04,103] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:15:04,143] jepsen worker 87 - jepsen.util 2607 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,144] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,185] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,186] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,208] jepsen worker 69 - jepsen.util 699 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,209] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:04,211] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,212] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:04,213] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,214] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:04,215] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,216] jepsen worker 22 - jepsen.util 652 :invoke :write [7 32] INFO [2022-03-13 17:15:04,232] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,232] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,232] jepsen worker 37 - jepsen.util 2107 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,233] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 8] INFO [2022-03-13 17:15:04,233] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 29] INFO [2022-03-13 17:15:04,239] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:04,239] jepsen worker nemesis - couchbase.util Rebalance check count: 413 INFO [2022-03-13 17:15:04,239] jepsen worker nemesis - couchbase.util Rebalance stuck count: 313 INFO [2022-03-13 17:15:04,241] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,243] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,244] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,248] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,249] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,250] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,251] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,259] jepsen worker 33 - jepsen.util 663 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,259] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:15:04,274] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,275] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:04,298] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,298] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,299] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,299] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,299] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,299] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:15:04,299] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,300] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,301] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,301] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:15:04,301] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,302] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:15:04,303] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:15:04,305] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,306] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,306] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:15:04,307] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,308] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,309] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,310] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:15:04,350] jepsen worker 22 - jepsen.util 652 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,350] jepsen worker 84 - jepsen.util 3234 :info :write [28 22] :RequestTimeoutException INFO [2022-03-13 17:15:04,351] jepsen worker 84 - jepsen.util 3324 :invoke :write [28 31] INFO [2022-03-13 17:15:04,353] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:15:04,405] jepsen worker 37 - jepsen.util 2107 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,406] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,405] jepsen worker 89 - jepsen.util 2519 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,407] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:15:04,407] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 21] INFO [2022-03-13 17:15:04,408] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 14] INFO [2022-03-13 17:15:04,415] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,416] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:04,431] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,432] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:15:04,434] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,435] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,440] jepsen worker 53 - jepsen.util 503 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,440] jepsen worker 44 - jepsen.util 584 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,440] jepsen worker 53 - jepsen.util 503 :invoke :write [17 49] INFO [2022-03-13 17:15:04,450] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:04,453] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,454] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,458] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,459] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:15:04,473] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,473] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,520] jepsen worker 28 - jepsen.util 3088 :info :write [9 38] :RequestTimeoutException INFO [2022-03-13 17:15:04,520] jepsen worker 28 - jepsen.util 3178 :invoke :read [9 nil] INFO [2022-03-13 17:15:04,523] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,523] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,530] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,531] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:15:04,531] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,532] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,560] jepsen worker 73 - jepsen.util 3133 :info :write [24 0] :RequestTimeoutException INFO [2022-03-13 17:15:04,561] jepsen worker 73 - jepsen.util 3223 :invoke :write [24 22] INFO [2022-03-13 17:15:04,580] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,580] jepsen worker 22 - jepsen.util 652 :invoke :write [7 0] INFO [2022-03-13 17:15:04,603] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,603] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,603] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,604] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:15:04,605] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,606] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,606] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,606] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:15:04,607] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,607] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 40] INFO [2022-03-13 17:15:04,614] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,615] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,617] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,618] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:15:04,649] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,649] jepsen worker 37 - jepsen.util 2107 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,649] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,652] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,652] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,653] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 7] INFO [2022-03-13 17:15:04,658] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,658] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,658] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:15:04,659] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:15:04,715] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,716] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,717] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,717] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,721] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,721] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,722] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,723] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,724] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,724] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,728] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,729] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,764] jepsen worker 38 - jepsen.util 2468 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,764] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:15:04,775] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,775] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,776] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:15:04,776] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,778] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,778] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,781] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,781] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,782] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,782] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,784] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,784] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,785] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,786] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,787] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,787] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,788] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,789] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,790] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,790] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:15:04,811] jepsen worker 88 - jepsen.util 2068 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,811] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,842] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,843] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 23] INFO [2022-03-13 17:15:04,880] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,880] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,880] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,881] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:15:04,881] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:15:04,882] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,884] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,884] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,886] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,886] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:04,887] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,888] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:15:04,894] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,895] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:15:04,927] jepsen worker 37 - jepsen.util 2107 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,928] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:04,953] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,954] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:04,957] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:04,957] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:15:04,999] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,000] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:15:05,024] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,025] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:15:05,032] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,033] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:15:05,081] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,083] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:05,084] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,085] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:15:05,103] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,103] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,124] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,124] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,124] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 9] INFO [2022-03-13 17:15:05,125] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:05,131] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,132] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:15:05,135] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,135] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:05,136] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,137] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:05,138] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,139] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:05,140] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,140] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:15:05,171] jepsen worker 8 - jepsen.util 728 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,172] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:15:05,178] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,179] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,180] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:15:05,180] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:05,190] jepsen worker 17 - jepsen.util 3077 :ok :read [5 43] INFO [2022-03-13 17:15:05,191] jepsen worker 17 - jepsen.util 3077 :invoke :write [5 26] INFO [2022-03-13 17:15:05,200] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,200] jepsen worker 22 - jepsen.util 652 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,200] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,201] jepsen worker 1 - jepsen.util 2701 :ok :read [0 41] INFO [2022-03-13 17:15:05,201] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:15:05,202] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,202] jepsen worker 1 - jepsen.util 2701 :invoke :read [0 nil] INFO [2022-03-13 17:15:05,203] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:15:05,204] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,205] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:05,208] jepsen worker 89 - jepsen.util 2519 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,209] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,216] jepsen worker 82 - jepsen.util 3052 :ok :read [27 4] INFO [2022-03-13 17:15:05,216] jepsen worker 82 - jepsen.util 3052 :invoke :read [27 nil] INFO [2022-03-13 17:15:05,232] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,233] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:15:05,245] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:05,245] jepsen worker nemesis - couchbase.util Rebalance check count: 414 INFO [2022-03-13 17:15:05,245] jepsen worker nemesis - couchbase.util Rebalance stuck count: 314 INFO [2022-03-13 17:15:05,261] jepsen worker 51 - jepsen.util 591 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,262] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:15:05,285] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,285] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,285] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,287] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:15:05,287] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,288] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:15:05,288] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,289] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,290] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,290] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,291] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,292] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,293] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,294] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,295] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,296] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:15:05,298] jepsen worker 33 - jepsen.util 663 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,299] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,299] jepsen worker 33 - jepsen.util 663 :invoke :write [11 17] INFO [2022-03-13 17:15:05,299] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,299] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:05,300] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:15:05,309] jepsen worker 21 - jepsen.util 831 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,309] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,310] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:05,310] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:05,314] jepsen worker 34 - jepsen.util 754 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,315] jepsen worker 34 - jepsen.util 754 :invoke :write [11 41] INFO [2022-03-13 17:15:05,315] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,315] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,316] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:15:05,317] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:05,383] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,383] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,384] jepsen worker 24 - jepsen.util 564 :invoke :write [8 2] INFO [2022-03-13 17:15:05,384] jepsen worker 43 - jepsen.util 583 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,386] jepsen worker 71 - jepsen.util 521 :invoke :write [23 10] INFO [2022-03-13 17:15:05,387] jepsen worker 43 - jepsen.util 583 :invoke :write [14 15] INFO [2022-03-13 17:15:05,403] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,404] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:05,471] jepsen worker 76 - jepsen.util 706 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,472] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:05,475] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,475] jepsen worker 78 - jepsen.util 528 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,476] jepsen worker 70 - jepsen.util 700 :invoke :read [23 nil] INFO [2022-03-13 17:15:05,476] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:15:05,531] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,532] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:15:05,545] jepsen worker 57 - jepsen.util 3027 :ok :read [19 16] INFO [2022-03-13 17:15:05,545] jepsen worker 57 - jepsen.util 3027 :invoke :write [19 15] INFO [2022-03-13 17:15:05,587] jepsen worker 41 - jepsen.util 3011 :ok :read [13 39] INFO [2022-03-13 17:15:05,588] jepsen worker 41 - jepsen.util 3011 :invoke :read [13 nil] INFO [2022-03-13 17:15:05,659] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,660] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:05,663] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,663] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,663] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,664] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,664] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:05,664] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:15:05,665] jepsen worker 14 - jepsen.util 374 :invoke :write [4 20] INFO [2022-03-13 17:15:05,665] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:15:05,667] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,667] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:05,668] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,669] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:15:05,676] jepsen worker 13 - jepsen.util 643 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,676] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:05,705] jepsen worker 41 - jepsen.util 3011 :ok :read [13 39] INFO [2022-03-13 17:15:05,706] jepsen worker 41 - jepsen.util 3011 :invoke :write [13 12] INFO [2022-03-13 17:15:05,824] jepsen worker 80 - jepsen.util 710 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,824] jepsen worker 80 - jepsen.util 710 :invoke :write [26 2] INFO [2022-03-13 17:15:05,826] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,827] jepsen worker 52 - jepsen.util 592 :invoke :write [17 15] INFO [2022-03-13 17:15:05,829] jepsen worker 33 - jepsen.util 663 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,830] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:05,835] jepsen worker 26 - jepsen.util 476 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,836] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:05,856] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,857] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:05,858] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,859] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:05,860] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,861] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:15:05,886] jepsen worker 34 - jepsen.util 754 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,886] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,886] jepsen worker 70 - jepsen.util 700 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,887] jepsen worker 78 - jepsen.util 528 :invoke :write [26 15] INFO [2022-03-13 17:15:05,887] jepsen worker 70 - jepsen.util 700 :invoke :write [23 4] INFO [2022-03-13 17:15:05,888] jepsen worker 34 - jepsen.util 754 :invoke :write [11 30] INFO [2022-03-13 17:15:05,888] jepsen worker 15 - jepsen.util 3165 :info :write [5 4] :RequestTimeoutException INFO [2022-03-13 17:15:05,889] jepsen worker 15 - jepsen.util 3255 :invoke :read [5 nil] INFO [2022-03-13 17:15:05,890] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,891] jepsen worker 13 - jepsen.util 643 :invoke :write [4 31] INFO [2022-03-13 17:15:05,933] jepsen worker 14 - jepsen.util 374 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:05,935] jepsen worker 14 - jepsen.util 374 :invoke :write [4 11] INFO [2022-03-13 17:15:05,955] jepsen worker 31 - jepsen.util 3181 :ok :read [10 :nil] INFO [2022-03-13 17:15:05,955] jepsen worker 31 - jepsen.util 3181 :invoke :write [10 2] INFO [2022-03-13 17:15:05,967] jepsen worker 19 - jepsen.util 2719 :ok :read [6 16] INFO [2022-03-13 17:15:05,968] jepsen worker 19 - jepsen.util 2719 :invoke :write [6 12] INFO [2022-03-13 17:15:05,984] jepsen worker 47 - jepsen.util 2927 :ok :read [15 48] INFO [2022-03-13 17:15:05,984] jepsen worker 47 - jepsen.util 2927 :invoke :write [15 17] INFO [2022-03-13 17:15:06,001] jepsen worker 80 - jepsen.util 710 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,001] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,001] jepsen worker 12 - jepsen.util 732 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,002] jepsen worker 80 - jepsen.util 710 :invoke :write [26 21] INFO [2022-03-13 17:15:06,002] jepsen worker 12 - jepsen.util 732 :invoke :write [4 48] INFO [2022-03-13 17:15:06,003] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:15:06,007] jepsen worker 24 - jepsen.util 564 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,008] jepsen worker 24 - jepsen.util 564 :invoke :write [8 36] INFO [2022-03-13 17:15:06,028] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,029] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:06,030] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,030] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,031] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:06,031] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,032] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,033] jepsen worker 22 - jepsen.util 652 :invoke :write [7 8] INFO [2022-03-13 17:15:06,035] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,036] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:15:06,037] jepsen worker 37 - jepsen.util 2107 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,038] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:15:06,048] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,050] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,052] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,053] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,059] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,060] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,061] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,062] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,070] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,071] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:15:06,074] jepsen worker 52 - jepsen.util 592 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,074] jepsen worker 52 - jepsen.util 592 :invoke :write [17 10] INFO [2022-03-13 17:15:06,090] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,091] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 37] INFO [2022-03-13 17:15:06,160] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,161] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:15:06,178] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,179] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,181] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,181] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:15:06,183] jepsen worker 34 - jepsen.util 754 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,184] jepsen worker 34 - jepsen.util 754 :invoke :write [11 8] INFO [2022-03-13 17:15:06,186] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,186] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:06,195] jepsen worker 71 - jepsen.util 521 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,197] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,199] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,199] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,199] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:06,200] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:06,201] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:15:06,212] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,212] jepsen worker 28 - jepsen.util 3178 :ok :read [9 26] INFO [2022-03-13 17:15:06,213] jepsen worker 28 - jepsen.util 3178 :invoke :read [9 nil] INFO [2022-03-13 17:15:06,228] jepsen worker 22 - jepsen.util 652 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,229] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:15:06,230] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,230] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:06,237] jepsen worker 32 - jepsen.util 3002 :ok :read [10 :nil] INFO [2022-03-13 17:15:06,237] jepsen worker 32 - jepsen.util 3002 :invoke :write [10 5] INFO [2022-03-13 17:15:06,252] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:06,252] jepsen worker nemesis - couchbase.util Rebalance check count: 415 INFO [2022-03-13 17:15:06,252] jepsen worker nemesis - couchbase.util Rebalance stuck count: 315 INFO [2022-03-13 17:15:06,279] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,280] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:15:06,297] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,298] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:15:06,345] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,346] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:15:06,354] jepsen worker 28 - jepsen.util 3178 :ok :read [9 26] INFO [2022-03-13 17:15:06,354] jepsen worker 28 - jepsen.util 3178 :invoke :write [9 40] INFO [2022-03-13 17:15:06,365] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,369] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:15:06,372] jepsen worker 24 - jepsen.util 564 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,372] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:15:06,378] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,378] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,379] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:06,380] jepsen worker 33 - jepsen.util 663 :invoke :write [11 37] INFO [2022-03-13 17:15:06,404] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,404] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,405] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:06,405] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,410] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,411] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,411] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:15:06,412] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,416] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,416] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,417] jepsen worker 88 - jepsen.util 2068 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,417] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,417] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,417] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,418] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,419] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,420] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,420] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 26] INFO [2022-03-13 17:15:06,423] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,424] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,426] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,428] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:15:06,445] jepsen worker 70 - jepsen.util 700 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,445] jepsen worker 79 - jepsen.util 1069 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,445] jepsen worker 60 - jepsen.util 600 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,445] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:06,445] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,447] jepsen worker 70 - jepsen.util 700 :invoke :write [23 1] INFO [2022-03-13 17:15:06,448] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:06,449] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:06,449] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:06,451] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:06,455] jepsen worker 54 - jepsen.util 2664 :ok :read [18 21] INFO [2022-03-13 17:15:06,456] jepsen worker 54 - jepsen.util 2664 :invoke :write [18 29] INFO [2022-03-13 17:15:06,483] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,484] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,484] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:15:06,484] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:15:06,513] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,514] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,518] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,518] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,520] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,521] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,521] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,521] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,521] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:15:06,522] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:15:06,523] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,524] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:15:06,524] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,525] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,526] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,526] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:15:06,534] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,535] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:15:06,538] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,539] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] WARN [2022-03-13 17:15:06,578] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17480707,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17949215,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9099957,"encode_duration_us":18409,"operation_id":"0xb5a0","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10002259},{"operation_name":"upsert","last_dispatch_duration_us":9091095,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17830185,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9208294,"encode_duration_us":17525,"operation_id":"0xb9f2","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10004193},{"operation_name":"upsert","last_dispatch_duration_us":12220315,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13925479,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":4473585,"encode_duration_us":24060,"operation_id":"0xb744","timeout_ms":10000,"last_server_duration_us":4151622,"total_duration_us":10046983},{"operation_name":"upsert","last_dispatch_duration_us":17440186,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27114886,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9239055,"encode_duration_us":20542,"operation_id":"0xb083","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10010763}],"total_count":4}} WARN [2022-03-13 17:15:06,578] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":54725,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7685507,"total_server_duration_us":336691,"encode_duration_us":16895,"operation_id":"0xbb6d","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10007546},{"operation_name":"upsert","last_dispatch_duration_us":9091095,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":17830185,"total_server_duration_us":9208294,"encode_duration_us":17525,"operation_id":"0xb9f2","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10004193},{"operation_name":"get","last_dispatch_duration_us":8132112,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8132112,"total_server_duration_us":27,"operation_id":"0xc01a","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":8133110},{"operation_name":"get","last_dispatch_duration_us":7333372,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7333372,"total_server_duration_us":92,"operation_id":"0xbc33","timeout_ms":30000,"last_server_duration_us":92,"total_duration_us":7335843},{"operation_name":"get","last_dispatch_duration_us":12801,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12801,"total_server_duration_us":6191,"operation_id":"0xbd07","timeout_ms":30000,"last_server_duration_us":6191,"total_duration_us":4164529},{"operation_name":"get","last_dispatch_duration_us":14947,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":14947,"total_server_duration_us":7541,"operation_id":"0xbd6d","timeout_ms":30000,"last_server_duration_us":7541,"total_duration_us":3655043},{"operation_name":"get","last_dispatch_duration_us":2570556,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2570556,"total_server_duration_us":49,"operation_id":"0xc331","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":2572632},{"operation_name":"upsert","last_dispatch_duration_us":335132,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1950424,"total_server_duration_us":351793,"encode_duration_us":18180,"operation_id":"0xc004","timeout_ms":10000,"last_server_duration_us":63053,"total_duration_us":2012239},{"operation_name":"upsert","last_dispatch_duration_us":325833,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1771889,"total_server_duration_us":373386,"encode_duration_us":17296,"operation_id":"0xc003","timeout_ms":10000,"last_server_duration_us":60885,"total_duration_us":1839621},{"operation_name":"upsert","last_dispatch_duration_us":716217,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1319598,"total_server_duration_us":350574,"encode_duration_us":17656,"operation_id":"0xc2d6","timeout_ms":10000,"last_server_duration_us":164337,"total_duration_us":1348967}],"total_count":33}} INFO [2022-03-13 17:15:06,581] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,581] jepsen worker 24 - jepsen.util 564 :invoke :write [8 13] INFO [2022-03-13 17:15:06,617] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,618] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 24] INFO [2022-03-13 17:15:06,628] jepsen worker 61 - jepsen.util 691 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,628] jepsen worker 42 - jepsen.util 402 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,629] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:06,629] jepsen worker 42 - jepsen.util 402 :invoke :write [14 18] INFO [2022-03-13 17:15:06,634] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,635] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:15:06,655] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,656] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:15:06,676] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,676] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,677] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:15:06,678] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 8] INFO [2022-03-13 17:15:06,721] jepsen worker 39 - jepsen.util 3009 :info :write [13 16] :RequestTimeoutException INFO [2022-03-13 17:15:06,721] jepsen worker 39 - jepsen.util 3099 :invoke :read [13 nil] INFO [2022-03-13 17:15:06,722] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,722] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:15:06,730] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,731] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:15:06,739] jepsen worker 33 - jepsen.util 663 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,740] jepsen worker 14 - jepsen.util 374 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,740] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:06,741] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,741] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:15:06,742] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:15:06,744] jepsen worker 13 - jepsen.util 643 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,744] jepsen worker 13 - jepsen.util 643 :invoke :write [4 46] INFO [2022-03-13 17:15:06,758] jepsen worker 89 - jepsen.util 2519 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,758] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,759] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,759] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,794] jepsen worker 53 - jepsen.util 503 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,794] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:06,816] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,817] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,818] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,819] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,820] jepsen worker 59 - jepsen.util 2939 :info :write [19 43] :RequestTimeoutException INFO [2022-03-13 17:15:06,820] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,820] jepsen worker 59 - jepsen.util 3029 :invoke :read [19 nil] INFO [2022-03-13 17:15:06,821] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,821] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,822] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,825] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,826] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,827] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,828] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:15:06,869] jepsen worker 36 - jepsen.util 2196 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,869] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,870] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:15:06,870] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 44] INFO [2022-03-13 17:15:06,879] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,879] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 17] INFO [2022-03-13 17:15:06,899] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,899] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,899] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,900] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,901] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:15:06,902] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,902] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,903] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,903] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:15:06,903] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,905] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,905] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,906] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,907] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:06,908] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,908] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,908] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:15:06,911] jepsen worker 77 - jepsen.util 707 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,912] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,912] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,913] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:15:06,914] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:06,914] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:15:06,915] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:15:06,915] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,916] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:15:06,928] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,929] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:06,943] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,944] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:06,945] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:06,946] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:15:07,007] jepsen worker 3 - jepsen.util 1983 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,007] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,008] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,008] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,011] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,011] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:15:07,039] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,039] jepsen worker 21 - jepsen.util 831 :invoke :write [7 1] INFO [2022-03-13 17:15:07,093] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,094] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:07,095] jepsen worker 37 - jepsen.util 2107 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,096] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 48] INFO [2022-03-13 17:15:07,097] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,097] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:15:07,100] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,101] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:15:07,104] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,105] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,134] jepsen worker 78 - jepsen.util 528 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,134] jepsen worker 78 - jepsen.util 528 :invoke :read [26 nil] INFO [2022-03-13 17:15:07,139] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,140] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:15:07,141] jepsen worker 48 - jepsen.util 2928 :ok :read [16 :nil] INFO [2022-03-13 17:15:07,142] jepsen worker 48 - jepsen.util 2928 :invoke :write [16 47] INFO [2022-03-13 17:15:07,145] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,145] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:15:07,158] jepsen worker 24 - jepsen.util 564 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,159] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,159] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:07,160] jepsen worker 33 - jepsen.util 663 :invoke :write [11 23] INFO [2022-03-13 17:15:07,185] jepsen worker 89 - jepsen.util 2519 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,186] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,193] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,194] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,195] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,196] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,197] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,198] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:15:07,200] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,200] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,200] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,201] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:15:07,202] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,203] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:15:07,205] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,206] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,207] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:15:07,207] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:15:07,230] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,230] jepsen worker 71 - jepsen.util 521 :invoke :write [23 3] INFO [2022-03-13 17:15:07,233] jepsen worker 43 - jepsen.util 583 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,233] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,233] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,233] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:07,234] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:07,234] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,234] jepsen worker 7 - jepsen.util 637 :invoke :write [2 20] INFO [2022-03-13 17:15:07,235] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:15:07,259] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:07,259] jepsen worker nemesis - couchbase.util Rebalance check count: 416 INFO [2022-03-13 17:15:07,259] jepsen worker nemesis - couchbase.util Rebalance stuck count: 316 INFO [2022-03-13 17:15:07,293] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,293] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,298] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,299] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:15:07,303] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,303] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,325] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,325] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,326] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,326] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:07,349] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,350] jepsen worker 35 - jepsen.util 935 :invoke :write [11 3] INFO [2022-03-13 17:15:07,351] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,351] jepsen worker 12 - jepsen.util 732 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,351] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,352] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,352] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:07,352] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:15:07,353] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:07,354] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:07,355] jepsen worker 69 - jepsen.util 699 :invoke :write [23 35] INFO [2022-03-13 17:15:07,355] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,355] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,355] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,356] jepsen worker 61 - jepsen.util 691 :invoke :write [20 43] INFO [2022-03-13 17:15:07,357] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:07,358] jepsen worker 51 - jepsen.util 591 :invoke :write [17 9] INFO [2022-03-13 17:15:07,364] jepsen worker 60 - jepsen.util 600 :invoke :write [20 9] INFO [2022-03-13 17:15:07,366] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,367] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,378] jepsen worker 80 - jepsen.util 710 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,379] jepsen worker 80 - jepsen.util 710 :invoke :read [26 nil] INFO [2022-03-13 17:15:07,398] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,398] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:07,400] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,401] jepsen worker 14 - jepsen.util 374 :invoke :write [4 14] INFO [2022-03-13 17:15:07,403] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,403] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,404] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 18] INFO [2022-03-13 17:15:07,404] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:15:07,437] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,437] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,437] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,438] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,438] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,439] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:15:07,448] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,448] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,448] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,449] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:15:07,449] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,450] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 1] INFO [2022-03-13 17:15:07,451] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,451] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,453] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,454] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,457] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,457] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,459] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,459] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 10] INFO [2022-03-13 17:15:07,480] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,480] jepsen worker 37 - jepsen.util 2107 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,480] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,480] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,480] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:15:07,481] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:07,482] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:07,482] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:15:07,484] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,485] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:15:07,492] jepsen worker 21 - jepsen.util 831 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,492] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,492] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:07,493] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:07,500] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,501] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:15:07,508] jepsen worker 66 - jepsen.util 2856 :info :write [22 3] :SyncWriteAmbiguous INFO [2022-03-13 17:15:07,509] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:07,537] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,538] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 2] INFO [2022-03-13 17:15:07,543] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,544] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,546] jepsen worker 29 - jepsen.util 3179 :ok :read [9 26] INFO [2022-03-13 17:15:07,547] jepsen worker 29 - jepsen.util 3179 :invoke :write [9 11] INFO [2022-03-13 17:15:07,551] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,552] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,553] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,554] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:15:07,600] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,600] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:07,624] jepsen worker 3 - jepsen.util 1983 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,624] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:15:07,627] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,628] jepsen worker 33 - jepsen.util 663 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,628] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:07,629] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,629] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:07,630] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,705] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,705] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:07,733] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:07,734] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:07,735] jepsen worker 88 - jepsen.util 2068 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,735] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:15:07,756] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,757] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:15:07,770] jepsen worker 89 - jepsen.util 2519 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,771] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:15:07,773] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,774] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:15:07,811] jepsen worker 87 - jepsen.util 2607 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,811] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,812] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,812] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,836] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,836] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,837] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,838] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:07,839] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,840] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:07,855] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,855] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,855] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:07,856] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 7] INFO [2022-03-13 17:15:07,857] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,857] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:15:07,880] jepsen worker 52 - jepsen.util 592 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,881] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] WARN [2022-03-13 17:15:07,882] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9044709,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10561470,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9343103,"encode_duration_us":22363,"operation_id":"0xbdc3","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10006791},{"operation_name":"upsert","last_dispatch_duration_us":11067858,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11067858,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9022513,"encode_duration_us":18743,"operation_id":"0xbdfb","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10007481}],"total_count":2}} INFO [2022-03-13 17:15:07,934] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,935] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,941] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,941] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:15:07,946] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,946] jepsen worker 35 - jepsen.util 935 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,946] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,947] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,947] jepsen worker 62 - jepsen.util 512 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,947] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,947] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,949] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,949] jepsen worker 71 - jepsen.util 521 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,950] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,951] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:07,951] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,951] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 38] INFO [2022-03-13 17:15:07,953] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:15:07,954] jepsen worker 77 - jepsen.util 707 :invoke :write [25 40] INFO [2022-03-13 17:15:07,957] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:15:07,957] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,958] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:07,959] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:07,959] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,960] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:15:07,960] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:07,961] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:15:07,962] jepsen worker 62 - jepsen.util 512 :invoke :write [20 17] WARN [2022-03-13 17:15:07,989] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9046209,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14198791,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9362581,"encode_duration_us":19165,"operation_id":"0xb888","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10003069}],"total_count":1}} INFO [2022-03-13 17:15:08,035] jepsen worker 34 - jepsen.util 754 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,036] jepsen worker 34 - jepsen.util 754 :invoke :read [11 nil] INFO [2022-03-13 17:15:08,040] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,041] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:15:08,042] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,043] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,043] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:08,043] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,044] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:08,044] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:08,044] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,045] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:08,052] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,052] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,052] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,052] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,053] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:08,054] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:08,055] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:08,055] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,055] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:15:08,056] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,056] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,066] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:08,066] jepsen worker 42 - jepsen.util 402 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,066] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,067] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:08,067] jepsen worker 33 - jepsen.util 663 :invoke :write [11 0] INFO [2022-03-13 17:15:08,068] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:08,068] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,068] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:08,068] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,069] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:08,069] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:15:08,070] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,071] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:15:08,077] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,077] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,078] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,079] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,079] jepsen worker 14 - jepsen.util 374 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,079] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,079] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:15:08,080] jepsen worker 21 - jepsen.util 831 :invoke :write [7 24] INFO [2022-03-13 17:15:08,081] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:08,082] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] WARN [2022-03-13 17:15:08,090] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9057753,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9057753,"total_server_duration_us":20203,"encode_duration_us":17112,"operation_id":"0xbf47","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10010325},{"operation_name":"upsert","last_dispatch_duration_us":9090950,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11977579,"total_server_duration_us":42898,"encode_duration_us":17978,"operation_id":"0xbd62","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10009142},{"operation_name":"upsert","last_dispatch_duration_us":11067858,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11067858,"total_server_duration_us":9022513,"encode_duration_us":18743,"operation_id":"0xbdfb","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10007481},{"operation_name":"upsert","last_dispatch_duration_us":9044709,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10561470,"total_server_duration_us":9343103,"encode_duration_us":22363,"operation_id":"0xbdc3","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10006791},{"operation_name":"upsert","last_dispatch_duration_us":8739422,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10208161,"total_server_duration_us":352474,"encode_duration_us":17332,"operation_id":"0xbda6","timeout_ms":10000,"last_server_duration_us":28584,"total_duration_us":10004632},{"operation_name":"upsert","last_dispatch_duration_us":8564750,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10160905,"total_server_duration_us":490933,"encode_duration_us":20966,"operation_id":"0xbdb8","timeout_ms":10000,"last_server_duration_us":26153,"total_duration_us":10004321},{"operation_name":"upsert","last_dispatch_duration_us":8735512,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10204260,"total_server_duration_us":355997,"encode_duration_us":20674,"operation_id":"0xbda2","timeout_ms":10000,"last_server_duration_us":25800,"total_duration_us":10003643},{"operation_name":"upsert","last_dispatch_duration_us":8436235,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9871849,"total_server_duration_us":365161,"encode_duration_us":20412,"operation_id":"0xbe02","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10000586},{"operation_name":"get","last_dispatch_duration_us":9350349,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9350349,"total_server_duration_us":271897,"operation_id":"0xbf44","timeout_ms":30000,"last_server_duration_us":271897,"total_duration_us":9352711},{"operation_name":"get","last_dispatch_duration_us":7787067,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":7787067,"total_server_duration_us":11865,"operation_id":"0xc01b","timeout_ms":30000,"last_server_duration_us":11865,"total_duration_us":7790026}],"total_count":66}} WARN [2022-03-13 17:15:08,090] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":41041,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":544437,"total_server_duration_us":335633,"encode_duration_us":27017,"operation_id":"0xbd67","timeout_ms":10000,"last_server_duration_us":29788,"total_duration_us":10010433},{"operation_name":"upsert","last_dispatch_duration_us":9068428,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":14175518,"total_server_duration_us":85893,"encode_duration_us":30066,"operation_id":"0xbb3c","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10004551},{"operation_name":"upsert","last_dispatch_duration_us":34488,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":477400,"total_server_duration_us":298060,"encode_duration_us":18196,"operation_id":"0xbe39","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10004385},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":18995,"operation_id":"0xbe5d","timeout_ms":10000,"total_duration_us":10003732},{"operation_name":"upsert","last_dispatch_duration_us":9056137,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9056137,"total_server_duration_us":22306,"encode_duration_us":17588,"operation_id":"0xbe67","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10002814},{"operation_name":"upsert","last_dispatch_duration_us":9072757,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13379836,"total_server_duration_us":9401923,"encode_duration_us":19860,"operation_id":"0xba5e","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10001877},{"operation_name":"upsert","last_dispatch_duration_us":9087528,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9087528,"total_server_duration_us":9049045,"encode_duration_us":23011,"operation_id":"0xbe4c","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":9091974},{"operation_name":"get","last_dispatch_duration_us":9054104,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9054104,"total_server_duration_us":231,"operation_id":"0xbe5f","timeout_ms":30000,"last_server_duration_us":231,"total_duration_us":9054327},{"operation_name":"get","last_dispatch_duration_us":7688112,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7688112,"total_server_duration_us":274053,"operation_id":"0xc029","timeout_ms":30000,"last_server_duration_us":274053,"total_duration_us":7692396},{"operation_name":"upsert","last_dispatch_duration_us":82296,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1701795,"total_server_duration_us":557598,"encode_duration_us":21450,"operation_id":"0xc001","timeout_ms":10000,"last_server_duration_us":56157,"total_duration_us":1851751}],"total_count":24}} WARN [2022-03-13 17:15:08,090] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16408724,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":16408724,"total_server_duration_us":7028,"operation_id":"0xb69b","timeout_ms":30000,"last_server_duration_us":7028,"total_duration_us":16411387},{"operation_name":"get","last_dispatch_duration_us":14101748,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14101748,"total_server_duration_us":43,"operation_id":"0xb804","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":14103779},{"operation_name":"upsert","last_dispatch_duration_us":8824201,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9971819,"total_server_duration_us":111710,"encode_duration_us":17356,"operation_id":"0xba84","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10011203},{"operation_name":"upsert","last_dispatch_duration_us":8967372,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":17961121,"total_server_duration_us":91134,"encode_duration_us":17935,"operation_id":"0xbb3f","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10006359},{"operation_name":"upsert","last_dispatch_duration_us":17847856,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":17847856,"total_server_duration_us":9047983,"encode_duration_us":19397,"operation_id":"0xbb6b","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10003919},{"operation_name":"upsert","last_dispatch_duration_us":8951278,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18297158,"total_server_duration_us":130234,"encode_duration_us":33562,"operation_id":"0xbb1d","timeout_ms":10000,"last_server_duration_us":39319,"total_duration_us":10002437},{"operation_name":"upsert","last_dispatch_duration_us":9140450,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9205630,"total_server_duration_us":9047898,"encode_duration_us":30587,"operation_id":"0xc075","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9228738},{"operation_name":"upsert","last_dispatch_duration_us":9131719,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9131719,"total_server_duration_us":9059667,"encode_duration_us":26053,"operation_id":"0xbb24","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":9137194},{"operation_name":"get","last_dispatch_duration_us":9110712,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9110712,"total_server_duration_us":27,"operation_id":"0xc081","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9112957},{"operation_name":"get","last_dispatch_duration_us":8170755,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8170755,"total_server_duration_us":32,"operation_id":"0xbbb5","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8174078}],"total_count":40}} WARN [2022-03-13 17:15:08,091] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17847856,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17847856,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9047983,"encode_duration_us":19397,"operation_id":"0xbb6b","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10003919},{"operation_name":"upsert","last_dispatch_duration_us":26559897,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":35274765,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9200861,"encode_duration_us":31706,"operation_id":"0xab79","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10009297}],"total_count":2}} INFO [2022-03-13 17:15:08,096] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,096] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:15:08,142] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,142] jepsen worker 37 - jepsen.util 2107 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,142] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,143] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 11] INFO [2022-03-13 17:15:08,148] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,149] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,155] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,156] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:15:08,175] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,176] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] WARN [2022-03-13 17:15:08,190] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":50683,"operation_id":"0xbd6f","timeout_ms":10000,"total_duration_us":10009806},{"operation_name":"upsert","last_dispatch_duration_us":36384,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9627464,"total_server_duration_us":189200,"encode_duration_us":19253,"operation_id":"0xbca1","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10009093},{"operation_name":"upsert","last_dispatch_duration_us":9070397,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10562940,"total_server_duration_us":41789,"encode_duration_us":18412,"operation_id":"0xbb9b","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10006602},{"operation_name":"upsert","last_dispatch_duration_us":35035,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":1121457,"total_server_duration_us":302755,"encode_duration_us":17977,"operation_id":"0xbaec","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10005807},{"operation_name":"upsert","last_dispatch_duration_us":8272030,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9335647,"total_server_duration_us":337770,"encode_duration_us":27890,"operation_id":"0xbb1f","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10002209},{"operation_name":"upsert","last_dispatch_duration_us":9362400,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9362400,"total_server_duration_us":9040551,"encode_duration_us":20210,"operation_id":"0xbc5d","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9367294},{"operation_name":"get","last_dispatch_duration_us":9140800,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9140800,"total_server_duration_us":32,"operation_id":"0xbc8d","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9143035},{"operation_name":"get","last_dispatch_duration_us":6419919,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6419919,"total_server_duration_us":812,"operation_id":"0xbe45","timeout_ms":30000,"last_server_duration_us":812,"total_duration_us":6420275},{"operation_name":"upsert","last_dispatch_duration_us":1579064,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1579064,"total_server_duration_us":442536,"encode_duration_us":29991,"operation_id":"0xc342","timeout_ms":10000,"last_server_duration_us":442536,"total_duration_us":4712993},{"operation_name":"upsert","last_dispatch_duration_us":2599712,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3849484,"total_server_duration_us":315440,"encode_duration_us":23171,"operation_id":"0xc2a1","timeout_ms":10000,"last_server_duration_us":292302,"total_duration_us":3896027}],"total_count":76}} WARN [2022-03-13 17:15:08,190] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9072757,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13379836,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9401923,"encode_duration_us":19860,"operation_id":"0xba5e","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10001877}],"total_count":1}} INFO [2022-03-13 17:15:08,231] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,232] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,233] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,233] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,234] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,235] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:15:08,240] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,241] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,263] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,263] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,264] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:15:08,264] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:08,264] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 45] INFO [2022-03-13 17:15:08,264] jepsen worker nemesis - couchbase.util Rebalance check count: 417 INFO [2022-03-13 17:15:08,264] jepsen worker nemesis - couchbase.util Rebalance stuck count: 317 INFO [2022-03-13 17:15:08,291] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,292] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,294] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:15:08,294] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9225015,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9225015,"total_server_duration_us":32,"operation_id":"0xba99","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":17455174},{"operation_name":"get","last_dispatch_duration_us":8778382,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8778382,"total_server_duration_us":15,"operation_id":"0xbc2a","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":14968989},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":17784,"operation_id":"0xbcb4","timeout_ms":10000,"total_duration_us":10007313},{"operation_name":"upsert","last_dispatch_duration_us":8941993,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8941993,"total_server_duration_us":18723,"encode_duration_us":27827,"operation_id":"0xbd72","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10005659},{"operation_name":"get","last_dispatch_duration_us":8764863,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8764863,"total_server_duration_us":43,"operation_id":"0xc011","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9853268},{"operation_name":"get","last_dispatch_duration_us":117532,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":117532,"total_server_duration_us":32,"operation_id":"0xc142","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6309559},{"operation_name":"upsert","last_dispatch_duration_us":2574018,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3694686,"total_server_duration_us":214242,"encode_duration_us":17215,"operation_id":"0xc26d","timeout_ms":10000,"last_server_duration_us":70556,"total_duration_us":3711965},{"operation_name":"upsert","last_dispatch_duration_us":2465362,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3473275,"total_server_duration_us":143424,"encode_duration_us":19009,"operation_id":"0xc2b5","timeout_ms":10000,"last_server_duration_us":101224,"total_duration_us":3492407},{"operation_name":"get","last_dispatch_duration_us":3232293,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3232293,"total_server_duration_us":2471082,"operation_id":"0xc2b1","timeout_ms":30000,"last_server_duration_us":2471082,"total_duration_us":3232611},{"operation_name":"upsert","last_dispatch_duration_us":2790742,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2790742,"total_server_duration_us":131339,"encode_duration_us":19560,"operation_id":"0xc336","timeout_ms":10000,"last_server_duration_us":131339,"total_duration_us":2797598}],"total_count":44}} INFO [2022-03-13 17:15:08,294] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,295] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,296] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,297] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,297] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,298] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,299] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,303] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,303] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,309] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,310] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,325] jepsen worker 68 - jepsen.util 3218 :info :write [22 17] :RequestTimeoutException INFO [2022-03-13 17:15:08,326] jepsen worker 68 - jepsen.util 3308 :invoke :read [22 nil] INFO [2022-03-13 17:15:08,329] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,331] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,336] jepsen worker 16 - jepsen.util 2806 :info :write [5 23] :RequestTimeoutException INFO [2022-03-13 17:15:08,337] jepsen worker 16 - jepsen.util 2896 :invoke :write [5 14] INFO [2022-03-13 17:15:08,376] jepsen worker 67 - jepsen.util 2587 :info :write [22 12] :RequestTimeoutException INFO [2022-03-13 17:15:08,376] jepsen worker 67 - jepsen.util 2677 :invoke :read [22 nil] INFO [2022-03-13 17:15:08,379] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,380] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,380] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,380] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:15:08,381] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,381] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:15:08,384] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,384] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,384] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,384] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,385] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:15:08,385] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,386] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,387] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,387] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,388] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,388] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:15:08,388] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:15:08,391] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,392] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] WARN [2022-03-13 17:15:08,394] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16957378,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17658907,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9181016,"encode_duration_us":17559,"operation_id":"0xb9e7","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10004887},{"operation_name":"upsert","last_dispatch_duration_us":25628123,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25628123,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9054355,"encode_duration_us":17876,"operation_id":"0xb58f","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10005023}],"total_count":2}} INFO [2022-03-13 17:15:08,398] jepsen worker 89 - jepsen.util 2519 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,399] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,428] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,428] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,428] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,428] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 43] INFO [2022-03-13 17:15:08,429] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:15:08,429] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,476] jepsen worker 55 - jepsen.util 2845 :info :write [18 35] :RequestTimeoutException INFO [2022-03-13 17:15:08,477] jepsen worker 55 - jepsen.util 2935 :invoke :write [18 39] INFO [2022-03-13 17:15:08,479] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,479] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,480] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,481] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:08,492] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,492] jepsen worker 37 - jepsen.util 2107 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,494] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,494] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,498] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,498] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,499] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,500] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,501] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,501] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:15:08,506] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,506] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 25] INFO [2022-03-13 17:15:08,508] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,509] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,511] jepsen worker 45 - jepsen.util 3195 :info :write [15 15] :RequestTimeoutException INFO [2022-03-13 17:15:08,511] jepsen worker 45 - jepsen.util 3285 :invoke :read [15 nil] INFO [2022-03-13 17:15:08,536] jepsen worker 40 - jepsen.util 3190 :info :write [13 2] :RequestTimeoutException INFO [2022-03-13 17:15:08,537] jepsen worker 40 - jepsen.util 3280 :invoke :write [13 5] INFO [2022-03-13 17:15:08,543] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,543] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,543] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:15:08,554] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:15:08,564] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,565] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,594] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,595] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:15:08,606] jepsen worker 20 - jepsen.util 3350 :info :write [6 46] :RequestTimeoutException INFO [2022-03-13 17:15:08,607] jepsen worker 20 - jepsen.util 3440 :invoke :write [6 30] INFO [2022-03-13 17:15:08,651] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,651] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,652] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:15:08,653] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:15:08,654] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,655] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,657] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,658] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,659] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,660] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:15:08,665] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,666] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:15:08,692] jepsen worker 89 - jepsen.util 2519 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,692] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,730] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,731] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,741] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,742] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:15:08,764] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,765] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:15:08,792] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,793] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:15:08,850] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,851] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,851] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,851] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,859] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,860] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,860] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:15:08,861] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:15:08,861] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,862] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:15:08,931] jepsen worker 81 - jepsen.util 3411 :info :write [27 7] :RequestTimeoutException INFO [2022-03-13 17:15:08,932] jepsen worker 81 - jepsen.util 3501 :invoke :write [27 34] INFO [2022-03-13 17:15:08,939] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,939] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,939] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,939] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 49] INFO [2022-03-13 17:15:08,940] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,941] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:15:08,955] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,956] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:08,957] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,958] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:15:08,962] jepsen worker 37 - jepsen.util 2107 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,962] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:08,989] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:08,990] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:15:09,044] jepsen worker 88 - jepsen.util 2068 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,044] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,056] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,057] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,065] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,066] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,067] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,067] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:15:09,098] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,098] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,099] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:15:09,099] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:15:09,131] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,131] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,131] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,134] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,161] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,161] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,175] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,175] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 9] INFO [2022-03-13 17:15:09,201] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,201] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,202] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,202] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:15:09,231] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,231] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,231] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,232] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,233] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,233] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,233] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:15:09,234] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,234] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:15:09,236] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,237] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,237] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,238] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,239] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,239] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,240] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,241] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,241] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,243] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,243] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,245] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,245] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:15:09,256] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,256] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,257] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,257] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,258] jepsen worker 0 - jepsen.util 2790 :info :write [0 41] :RequestTimeoutException INFO [2022-03-13 17:15:09,259] jepsen worker 0 - jepsen.util 2880 :invoke :write [0 22] INFO [2022-03-13 17:15:09,270] jepsen worker 83 - jepsen.util 2423 :info :write [27 7] :RequestTimeoutException INFO [2022-03-13 17:15:09,270] jepsen worker 83 - jepsen.util 2513 :invoke :write [27 14] INFO [2022-03-13 17:15:09,271] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:09,272] jepsen worker nemesis - couchbase.util Rebalance check count: 418 INFO [2022-03-13 17:15:09,272] jepsen worker nemesis - couchbase.util Rebalance stuck count: 318 INFO [2022-03-13 17:15:09,286] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,287] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,288] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,289] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,290] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,291] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:15:09,336] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,337] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 29] INFO [2022-03-13 17:15:09,348] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,348] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:15:09,351] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,352] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:09,353] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,353] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,353] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:09,353] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:09,353] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,354] jepsen worker 44 - jepsen.util 584 :invoke :write [14 30] INFO [2022-03-13 17:15:09,354] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:09,355] jepsen worker 76 - jepsen.util 706 :invoke :write [25 31] INFO [2022-03-13 17:15:09,359] jepsen worker 3 - jepsen.util 1983 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,359] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 23] INFO [2022-03-13 17:15:09,369] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,369] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,370] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,370] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,412] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,413] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,414] jepsen worker 33 - jepsen.util 663 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,414] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:09,422] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,422] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,423] jepsen worker 14 - jepsen.util 374 :invoke :write [4 32] INFO [2022-03-13 17:15:09,423] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:15:09,425] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,425] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,425] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,425] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,426] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:15:09,427] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,428] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 47] INFO [2022-03-13 17:15:09,428] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,438] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,438] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,439] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,440] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,441] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:15:09,446] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:15:09,447] jepsen worker 70 - jepsen.util 700 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,448] jepsen worker 70 - jepsen.util 700 :invoke :write [23 42] INFO [2022-03-13 17:15:09,458] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,459] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,471] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,476] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:15:09,484] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,484] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:09,506] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,506] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:15:09,510] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,511] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:15:09,516] jepsen worker 56 - jepsen.util 3206 :info :write [18 34] :RequestTimeoutException INFO [2022-03-13 17:15:09,517] jepsen worker 56 - jepsen.util 3296 :invoke :read [18 nil] INFO [2022-03-13 17:15:09,532] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,532] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,535] jepsen worker 3 - jepsen.util 1983 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,537] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,541] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,542] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:15:09,587] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,587] jepsen worker 87 - jepsen.util 2607 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,587] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,588] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,591] jepsen worker 6 - jepsen.util 636 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,591] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,591] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:09,591] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,592] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:09,592] jepsen worker 43 - jepsen.util 583 :invoke :write [14 47] INFO [2022-03-13 17:15:09,593] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:09,593] jepsen worker 6 - jepsen.util 636 :invoke :write [2 14] INFO [2022-03-13 17:15:09,607] jepsen worker 21 - jepsen.util 831 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,607] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:15:09,612] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,613] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:09,615] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,615] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,615] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,615] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,615] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:09,616] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,616] jepsen worker 89 - jepsen.util 2519 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,617] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,617] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:15:09,618] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:09,619] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,619] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:09,620] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,620] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,620] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:09,621] jepsen worker 33 - jepsen.util 663 :invoke :write [11 3] INFO [2022-03-13 17:15:09,624] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,624] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,625] jepsen worker 24 - jepsen.util 564 :invoke :write [8 13] INFO [2022-03-13 17:15:09,625] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:09,627] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,628] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:09,647] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,648] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,649] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,651] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:15:09,655] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,655] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,656] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,656] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,657] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,657] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:15:09,658] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,658] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,663] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,664] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 25] INFO [2022-03-13 17:15:09,698] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,698] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,698] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,699] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,699] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:15:09,702] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,703] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,703] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:15:09,704] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,704] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,705] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,708] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,708] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,709] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,711] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,712] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:15:09,713] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,713] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:09,714] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,714] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,714] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,715] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,715] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:15:09,716] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,717] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,718] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:15:09,726] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,727] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,736] jepsen worker 61 - jepsen.util 691 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,737] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:09,755] jepsen worker 14 - jepsen.util 374 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,756] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:09,762] jepsen worker 9 - jepsen.util 2529 :info :write [3 22] :SyncWriteAmbiguous INFO [2022-03-13 17:15:09,763] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,763] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,763] jepsen worker 9 - jepsen.util 2619 :invoke :write [3 27] INFO [2022-03-13 17:15:09,763] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,763] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,764] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:15:09,764] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:15:09,809] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,810] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:09,830] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,830] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:15:09,833] jepsen worker 89 - jepsen.util 2519 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,833] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,855] jepsen worker 72 - jepsen.util 3042 :info :write [24 12] :SyncWriteAmbiguous INFO [2022-03-13 17:15:09,856] jepsen worker 72 - jepsen.util 3132 :invoke :write [24 9] INFO [2022-03-13 17:15:09,857] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,858] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:09,859] jepsen worker 33 - jepsen.util 663 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,860] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:09,872] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,872] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:09,874] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,875] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:15:09,892] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,892] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,893] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,893] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:09,938] jepsen worker 46 - jepsen.util 3016 :ok :read [15 48] INFO [2022-03-13 17:15:09,939] jepsen worker 46 - jepsen.util 3016 :invoke :write [15 32] INFO [2022-03-13 17:15:09,952] jepsen worker 7 - jepsen.util 637 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,953] jepsen worker 7 - jepsen.util 637 :invoke :write [2 38] INFO [2022-03-13 17:15:09,956] jepsen worker 24 - jepsen.util 564 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,957] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:09,972] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,972] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,972] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,972] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,973] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:09,973] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:09,974] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:15:09,974] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,974] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:15:09,975] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,975] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:09,975] jepsen worker 14 - jepsen.util 374 :invoke :write [4 16] INFO [2022-03-13 17:15:09,976] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:09,976] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:15:09,993] jepsen worker 10 - jepsen.util 3070 :ok :read [3 25] INFO [2022-03-13 17:15:09,994] jepsen worker 10 - jepsen.util 3070 :invoke :read [3 nil] INFO [2022-03-13 17:15:10,003] jepsen worker 69 - jepsen.util 699 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,004] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:10,089] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,090] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:10,109] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,110] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:15:10,173] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,173] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,173] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,173] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,174] jepsen worker 33 - jepsen.util 663 :invoke :write [11 4] INFO [2022-03-13 17:15:10,175] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:15:10,176] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:15:10,177] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:10,204] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,205] jepsen worker 24 - jepsen.util 564 :invoke :write [8 30] INFO [2022-03-13 17:15:10,207] jepsen worker 79 - jepsen.util 1069 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,207] jepsen worker 51 - jepsen.util 591 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,208] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:15:10,209] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:10,221] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,222] jepsen worker 21 - jepsen.util 831 :invoke :write [7 15] INFO [2022-03-13 17:15:10,236] jepsen worker 33 - jepsen.util 663 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,237] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:15:10,277] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:10,277] jepsen worker nemesis - couchbase.util Rebalance check count: 419 INFO [2022-03-13 17:15:10,277] jepsen worker nemesis - couchbase.util Rebalance stuck count: 319 INFO [2022-03-13 17:15:10,312] jepsen worker 86 - jepsen.util 2876 :ok :read [28 44] INFO [2022-03-13 17:15:10,313] jepsen worker 86 - jepsen.util 2876 :invoke :read [28 nil] INFO [2022-03-13 17:15:10,321] jepsen worker 59 - jepsen.util 3029 :ok :read [19 16] INFO [2022-03-13 17:15:10,322] jepsen worker 59 - jepsen.util 3029 :invoke :read [19 nil] INFO [2022-03-13 17:15:10,340] jepsen worker 86 - jepsen.util 2876 :ok :read [28 44] INFO [2022-03-13 17:15:10,340] jepsen worker 86 - jepsen.util 2876 :invoke :read [28 nil] INFO [2022-03-13 17:15:10,349] jepsen worker 59 - jepsen.util 3029 :ok :read [19 16] INFO [2022-03-13 17:15:10,350] jepsen worker 59 - jepsen.util 3029 :invoke :read [19 nil] INFO [2022-03-13 17:15:10,366] jepsen worker 86 - jepsen.util 2876 :ok :read [28 44] INFO [2022-03-13 17:15:10,367] jepsen worker 86 - jepsen.util 2876 :invoke :write [28 25] INFO [2022-03-13 17:15:10,371] jepsen worker 59 - jepsen.util 3029 :ok :read [19 16] INFO [2022-03-13 17:15:10,371] jepsen worker 59 - jepsen.util 3029 :invoke :read [19 nil] INFO [2022-03-13 17:15:10,378] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,379] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:15:10,397] jepsen worker 59 - jepsen.util 3029 :ok :read [19 16] INFO [2022-03-13 17:15:10,398] jepsen worker 59 - jepsen.util 3029 :invoke :read [19 nil] INFO [2022-03-13 17:15:10,445] jepsen worker 59 - jepsen.util 3029 :ok :read [19 16] INFO [2022-03-13 17:15:10,445] jepsen worker 59 - jepsen.util 3029 :invoke :read [19 nil] INFO [2022-03-13 17:15:10,464] jepsen worker 62 - jepsen.util 512 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,464] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:10,464] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,464] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:10,464] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,465] jepsen worker 75 - jepsen.util 435 :invoke :write [25 27] INFO [2022-03-13 17:15:10,466] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:10,467] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:10,499] jepsen worker 59 - jepsen.util 3029 :ok :read [19 16] INFO [2022-03-13 17:15:10,499] jepsen worker 59 - jepsen.util 3029 :invoke :read [19 nil] INFO [2022-03-13 17:15:10,515] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,515] jepsen worker 60 - jepsen.util 600 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,516] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,516] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:10,517] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:10,517] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:10,521] jepsen worker 14 - jepsen.util 374 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,521] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:10,537] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,538] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:10,540] jepsen worker 59 - jepsen.util 3029 :ok :read [19 16] INFO [2022-03-13 17:15:10,540] jepsen worker 59 - jepsen.util 3029 :invoke :write [19 39] INFO [2022-03-13 17:15:10,585] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,586] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:15:10,602] jepsen worker 43 - jepsen.util 583 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,602] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:10,635] jepsen worker 24 - jepsen.util 564 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,635] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:15:10,725] jepsen worker 77 - jepsen.util 707 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,726] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:15:10,735] jepsen worker 21 - jepsen.util 831 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,736] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:10,750] jepsen worker 11 - jepsen.util 3071 :info :write [3 9] :RequestTimeoutException INFO [2022-03-13 17:15:10,751] jepsen worker 11 - jepsen.util 3161 :invoke :read [3 nil] INFO [2022-03-13 17:15:10,768] jepsen worker 6 - jepsen.util 636 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,768] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:10,823] jepsen worker 64 - jepsen.util 2944 :info :write [21 0] :RequestTimeoutException INFO [2022-03-13 17:15:10,824] jepsen worker 64 - jepsen.util 3034 :invoke :write [21 10] INFO [2022-03-13 17:15:10,854] jepsen worker 65 - jepsen.util 3395 :info :write [21 21] :RequestTimeoutException INFO [2022-03-13 17:15:10,855] jepsen worker 44 - jepsen.util 584 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,856] jepsen worker 65 - jepsen.util 3485 :invoke :write [21 4] INFO [2022-03-13 17:15:10,857] jepsen worker 44 - jepsen.util 584 :invoke :write [14 37] INFO [2022-03-13 17:15:10,879] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,879] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,880] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:15:10,881] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:15:10,889] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,890] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:10,921] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,921] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:10,922] jepsen worker 7 - jepsen.util 637 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:10,922] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,921] jepsen worker 76 - jepsen.util 706 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,922] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,924] jepsen worker 26 - jepsen.util 476 :invoke :write [8 17] INFO [2022-03-13 17:15:10,924] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,924] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,925] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 46] INFO [2022-03-13 17:15:10,926] jepsen worker 61 - jepsen.util 691 :invoke :write [20 49] INFO [2022-03-13 17:15:10,928] jepsen worker 51 - jepsen.util 591 :invoke :write [17 6] INFO [2022-03-13 17:15:10,929] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] INFO [2022-03-13 17:15:10,930] jepsen worker 76 - jepsen.util 706 :invoke :write [25 6] INFO [2022-03-13 17:15:10,930] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:15:10,930] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:10,931] jepsen worker 69 - jepsen.util 699 :invoke :write [23 14] INFO [2022-03-13 17:15:10,934] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:10,935] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,935] jepsen worker 7 - jepsen.util 637 :invoke :write [2 29] INFO [2022-03-13 17:15:10,935] jepsen worker 43 - jepsen.util 583 :invoke :write [14 45] INFO [2022-03-13 17:15:10,936] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:15:10,937] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:10,938] jepsen worker 8 - jepsen.util 728 :invoke :write [2 29] INFO [2022-03-13 17:15:10,939] jepsen worker 60 - jepsen.util 600 :invoke :write [20 45] INFO [2022-03-13 17:15:10,944] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:10,946] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:15:10,955] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,956] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:15:10,958] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,958] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,958] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:10,959] jepsen worker 6 - jepsen.util 636 :invoke :write [2 2] INFO [2022-03-13 17:15:10,960] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,960] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,960] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:10,961] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:10,970] jepsen worker 44 - jepsen.util 584 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,970] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,971] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:10,971] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,971] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:10,974] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:15:10,994] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,994] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,995] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:10,995] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:10,999] jepsen worker 75 - jepsen.util 435 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:10,999] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,000] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:11,001] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:11,009] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,009] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,009] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,009] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,010] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 28] INFO [2022-03-13 17:15:11,010] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:15:11,010] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,011] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 30] INFO [2022-03-13 17:15:11,012] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,012] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,015] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,016] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,016] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,019] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:15:11,019] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,020] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:15:11,032] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:11,032] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,032] jepsen worker 26 - jepsen.util 476 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,033] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:11,034] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:11,035] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:11,052] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,054] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:11,059] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,060] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:11,061] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,062] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:15:11,085] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,085] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,087] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:15:11,088] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:15:11,122] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,122] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:15:11,123] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,123] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:11,127] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,127] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,127] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,127] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,128] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:11,129] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:11,129] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,130] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,130] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:15:11,130] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,131] jepsen worker 33 - jepsen.util 663 :invoke :write [11 30] INFO [2022-03-13 17:15:11,131] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:11,132] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:15:11,132] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,133] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:15:11,133] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:15:11,135] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,136] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:15:11,152] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,153] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,154] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,154] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,155] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,156] jepsen worker 14 - jepsen.util 374 :invoke :write [4 13] INFO [2022-03-13 17:15:11,157] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:15:11,158] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:15:11,201] jepsen worker 89 - jepsen.util 2519 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,202] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:15:11,222] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,223] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:15:11,247] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,247] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:15:11,255] jepsen worker 76 - jepsen.util 706 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,256] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:11,283] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:11,283] jepsen worker nemesis - couchbase.util Rebalance check count: 420 INFO [2022-03-13 17:15:11,283] jepsen worker nemesis - couchbase.util Rebalance stuck count: 320 INFO [2022-03-13 17:15:11,284] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,284] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:15:11,293] jepsen worker 8 - jepsen.util 728 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,293] jepsen worker 8 - jepsen.util 728 :invoke :write [2 27] INFO [2022-03-13 17:15:11,306] jepsen worker 37 - jepsen.util 2107 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,306] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,320] jepsen worker 33 - jepsen.util 663 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,321] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:11,321] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,322] jepsen worker 35 - jepsen.util 935 :invoke :write [11 2] INFO [2022-03-13 17:15:11,345] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,345] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,346] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:15:11,346] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:15:11,358] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,359] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,365] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,366] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:15:11,391] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,392] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,392] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:15:11,392] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,428] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,428] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,428] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,429] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,430] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,430] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,431] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,431] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:11,431] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,432] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,432] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:15:11,433] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,439] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,439] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,440] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,441] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 18] INFO [2022-03-13 17:15:11,451] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,452] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:11,483] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,484] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:15:11,519] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,519] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,519] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,520] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,520] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,521] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,521] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,522] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,522] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,522] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:15:11,526] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,527] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,528] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,529] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 10] INFO [2022-03-13 17:15:11,531] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,532] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:15:11,552] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,553] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:11,554] jepsen worker 60 - jepsen.util 600 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,554] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:11,555] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,555] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:11,555] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,556] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:11,556] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:11,557] jepsen worker 77 - jepsen.util 707 :invoke :write [25 29] INFO [2022-03-13 17:15:11,577] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,578] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:11,581] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,582] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:15:11,587] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,587] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,588] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,588] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:15:11,589] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,589] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:15:11,597] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,598] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:11,604] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,604] jepsen worker 43 - jepsen.util 583 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,605] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,605] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:11,606] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:11,607] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:11,662] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,663] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:15:11,673] jepsen worker 89 - jepsen.util 2519 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,673] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,678] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,678] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,679] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:15:11,679] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:11,692] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,692] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:11,727] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,727] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:15:11,761] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,762] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:15:11,775] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,775] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,776] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:11,776] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:15:11,826] jepsen worker 49 - jepsen.util 2839 :info :write [16 36] :RequestTimeoutException INFO [2022-03-13 17:15:11,826] jepsen worker 49 - jepsen.util 2929 :invoke :write [16 10] INFO [2022-03-13 17:15:11,833] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,833] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,834] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,834] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 44] INFO [2022-03-13 17:15:11,835] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,835] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:15:11,855] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,855] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,856] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:15:11,856] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 2] INFO [2022-03-13 17:15:11,910] jepsen worker 37 - jepsen.util 2107 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,911] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:11,914] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,915] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:15:11,917] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,918] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,918] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:15:11,918] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:11,920] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:11,921] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:15:12,008] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,009] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:12,011] jepsen worker 87 - jepsen.util 2607 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,011] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:15:12,024] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,024] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,025] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:15:12,027] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:12,066] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,067] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:12,070] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,070] jepsen worker 89 - jepsen.util 2519 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,070] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,071] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 26] INFO [2022-03-13 17:15:12,071] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:15:12,072] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:15:12,131] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,132] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:12,143] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,144] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 35] INFO [2022-03-13 17:15:12,166] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,167] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:12,170] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,170] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:15:12,178] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,179] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:15:12,216] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,217] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:15:12,288] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:12,288] jepsen worker nemesis - couchbase.util Rebalance check count: 421 INFO [2022-03-13 17:15:12,288] jepsen worker nemesis - couchbase.util Rebalance stuck count: 321 INFO [2022-03-13 17:15:12,467] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,467] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,467] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,467] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,468] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] INFO [2022-03-13 17:15:12,468] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:15:12,469] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:12,470] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:12,475] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,475] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,476] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:12,476] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:12,477] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,477] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:15:12,479] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,479] jepsen worker 75 - jepsen.util 435 :invoke :write [25 43] INFO [2022-03-13 17:15:12,525] jepsen worker 69 - jepsen.util 699 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,526] jepsen worker 69 - jepsen.util 699 :invoke :write [23 39] INFO [2022-03-13 17:15:12,529] jepsen worker 14 - jepsen.util 374 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,530] jepsen worker 14 - jepsen.util 374 :invoke :write [4 6] INFO [2022-03-13 17:15:12,587] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,587] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:15:12,600] jepsen worker 35 - jepsen.util 935 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,600] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,600] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,600] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:12,601] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:15:12,601] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,604] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,604] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:12,605] jepsen worker 62 - jepsen.util 512 :invoke :write [20 2] INFO [2022-03-13 17:15:12,605] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:12,675] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,675] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:15:12,683] jepsen worker 14 - jepsen.util 374 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,683] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,683] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,684] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:12,684] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,685] jepsen worker 14 - jepsen.util 374 :invoke :write [4 7] INFO [2022-03-13 17:15:12,712] jepsen worker 51 - jepsen.util 591 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,713] jepsen worker 51 - jepsen.util 591 :invoke :read [17 nil] INFO [2022-03-13 17:15:12,757] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,757] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,757] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,757] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:12,758] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:15:12,759] jepsen worker 21 - jepsen.util 831 :invoke :write [7 22] INFO [2022-03-13 17:15:12,771] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,771] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:12,779] jepsen worker 7 - jepsen.util 637 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,780] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:15:12,805] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,805] jepsen worker 14 - jepsen.util 374 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,806] jepsen worker 14 - jepsen.util 374 :invoke :write [4 13] INFO [2022-03-13 17:15:12,806] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:15:12,839] jepsen worker 75 - jepsen.util 435 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,840] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:12,886] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,887] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:12,910] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,911] jepsen worker 23 - jepsen.util 563 :invoke :write [7 16] INFO [2022-03-13 17:15:12,930] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,931] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,941] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,941] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,941] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,941] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,942] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,943] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:15:12,943] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:12,944] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:12,947] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,947] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,947] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,948] jepsen worker 33 - jepsen.util 663 :invoke :write [11 1] INFO [2022-03-13 17:15:12,948] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,949] jepsen worker 71 - jepsen.util 521 :invoke :write [23 40] INFO [2022-03-13 17:15:12,949] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,950] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,951] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,951] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,952] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,952] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,953] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,954] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,955] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,955] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:12,956] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:12,957] jepsen worker 24 - jepsen.util 564 :invoke :write [8 8] INFO [2022-03-13 17:15:13,000] jepsen worker 6 - jepsen.util 636 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,001] jepsen worker 6 - jepsen.util 636 :invoke :write [2 35] INFO [2022-03-13 17:15:13,023] jepsen worker 21 - jepsen.util 831 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,024] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:15:13,040] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,040] jepsen worker 89 - jepsen.util 2519 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,041] jepsen worker 75 - jepsen.util 435 :invoke :write [25 49] INFO [2022-03-13 17:15:13,041] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:13,047] jepsen worker 58 - jepsen.util 2758 :info :write [19 25] :RequestTimeoutException INFO [2022-03-13 17:15:13,048] jepsen worker 58 - jepsen.util 2848 :invoke :write [19 6] INFO [2022-03-13 17:15:13,055] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,055] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:13,056] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,057] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:15:13,081] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,081] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,081] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:13,082] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:15:13,104] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,105] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 37] INFO [2022-03-13 17:15:13,116] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,117] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:13,128] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,128] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:13,128] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,128] jepsen worker 61 - jepsen.util 691 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,129] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,131] jepsen worker 76 - jepsen.util 706 :invoke :write [25 49] INFO [2022-03-13 17:15:13,132] jepsen worker 71 - jepsen.util 521 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,133] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:15:13,134] jepsen worker 61 - jepsen.util 691 :invoke :write [20 27] INFO [2022-03-13 17:15:13,136] jepsen worker 44 - jepsen.util 584 :invoke :write [14 18] INFO [2022-03-13 17:15:13,136] jepsen worker 60 - jepsen.util 600 :invoke :write [20 16] INFO [2022-03-13 17:15:13,137] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:15:13,145] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,147] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:15:13,157] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,158] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:13,159] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,160] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:13,194] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,194] jepsen worker 14 - jepsen.util 374 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,195] jepsen worker 14 - jepsen.util 374 :invoke :write [4 49] INFO [2022-03-13 17:15:13,195] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:15:13,199] jepsen worker 79 - jepsen.util 1069 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,200] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 38] INFO [2022-03-13 17:15:13,206] jepsen worker 36 - jepsen.util 2196 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,207] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:15:13,208] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,210] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:13,220] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,221] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:15:13,236] jepsen worker 33 - jepsen.util 663 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,236] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:13,249] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,249] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,249] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:15:13,250] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:13,251] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,251] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,251] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,251] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,251] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,253] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:13,253] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:15:13,254] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:15:13,254] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:13,255] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:15:13,257] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,257] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:13,258] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,259] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:15:13,289] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,290] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:13,294] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:13,294] jepsen worker nemesis - couchbase.util Rebalance check count: 422 INFO [2022-03-13 17:15:13,294] jepsen worker nemesis - couchbase.util Rebalance stuck count: 322 INFO [2022-03-13 17:15:13,303] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,303] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:15:13,317] jepsen worker 14 - jepsen.util 374 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,318] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:13,331] jepsen worker 77 - jepsen.util 707 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,332] jepsen worker 77 - jepsen.util 707 :invoke :write [25 45] INFO [2022-03-13 17:15:13,335] jepsen worker 24 - jepsen.util 564 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,336] jepsen worker 24 - jepsen.util 564 :invoke :write [8 8] INFO [2022-03-13 17:15:13,340] jepsen worker 37 - jepsen.util 2107 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,341] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,341] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:13,342] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:13,350] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,351] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:15:13,380] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,380] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,380] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,380] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:15:13,919] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:15:13,386] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,919] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:13,401] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,401] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,920] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:13,920] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:15:13,405] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,405] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:13,922] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:13,405] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,406] jepsen worker 8 - jepsen.util 728 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,922] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:13,406] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:13,922] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:13,408] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,414] jepsen worker 75 - jepsen.util 435 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,922] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:15:13,922] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:15:13,461] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,463] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,923] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 33] INFO [2022-03-13 17:15:13,923] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:15:13,482] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,500] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,500] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,930] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:15:13,502] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,930] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:15:13,502] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,506] jepsen worker 18 - jepsen.util 2808 :info :write [6 12] :RequestTimeoutException INFO [2022-03-13 17:15:13,930] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:15:13,934] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,935] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,506] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,512] jepsen worker 51 - jepsen.util 591 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,935] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:15:13,512] jepsen worker 69 - jepsen.util 699 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,935] jepsen worker 51 - jepsen.util 591 :invoke :write [17 5] INFO [2022-03-13 17:15:13,539] jepsen worker 24 - jepsen.util 564 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,588] jepsen worker 63 - jepsen.util 3033 :info :write [21 32] :RequestTimeoutException INFO [2022-03-13 17:15:13,935] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:13,610] jepsen worker 30 - jepsen.util 2820 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:15:13,935] jepsen worker 63 - jepsen.util 3123 :invoke :read [21 nil] INFO [2022-03-13 17:15:13,644] jepsen worker 62 - jepsen.util 512 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,936] jepsen worker 30 - jepsen.util 2910 :invoke :read [10 nil] INFO [2022-03-13 17:15:13,936] jepsen worker 62 - jepsen.util 512 :invoke :write [20 38] INFO [2022-03-13 17:15:13,694] jepsen worker 23 - jepsen.util 563 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,812] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,936] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:13,936] jepsen worker 7 - jepsen.util 637 :invoke :write [2 26] INFO [2022-03-13 17:15:13,936] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:15:13,870] jepsen worker 2 - jepsen.util 3512 :info :write [0 15] :RequestTimeoutException INFO [2022-03-13 17:15:13,942] jepsen worker 2 - jepsen.util 3602 :invoke :write [0 28] INFO [2022-03-13 17:15:13,945] jepsen worker 76 - jepsen.util 706 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,913] jepsen worker 74 - jepsen.util 3224 :info :write [24 27] :RequestTimeoutException INFO [2022-03-13 17:15:13,919] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:13,922] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:13,946] jepsen worker 74 - jepsen.util 3314 :invoke :read [24 nil] INFO [2022-03-13 17:15:13,922] jepsen worker 42 - jepsen.util 402 :invoke :write [14 39] INFO [2022-03-13 17:15:13,930] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:15:13,930] jepsen worker 18 - jepsen.util 2898 :invoke :write [6 7] INFO [2022-03-13 17:15:13,930] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:13,955] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,955] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,960] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:13,960] jepsen worker 50 - jepsen.util 3200 :ok :read [16 :nil] INFO [2022-03-13 17:15:13,966] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,977] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:13,978] jepsen worker 50 - jepsen.util 3200 :invoke :read [16 nil] INFO [2022-03-13 17:15:13,979] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:13,979] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:15:13,979] jepsen worker 33 - jepsen.util 663 :invoke :write [11 27] INFO [2022-03-13 17:15:13,980] jepsen worker 6 - jepsen.util 636 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:13,984] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:15:13,986] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:14,038] jepsen worker 74 - jepsen.util 3314 :ok :read [24 24] INFO [2022-03-13 17:15:14,038] jepsen worker 74 - jepsen.util 3314 :invoke :read [24 nil] INFO [2022-03-13 17:15:14,053] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,053] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,054] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:15:14,055] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 43] INFO [2022-03-13 17:15:14,067] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,068] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:15:14,071] jepsen worker 39 - jepsen.util 3099 :ok :read [13 39] INFO [2022-03-13 17:15:14,072] jepsen worker 39 - jepsen.util 3099 :invoke :write [13 39] INFO [2022-03-13 17:15:14,075] jepsen worker 74 - jepsen.util 3314 :ok :read [24 24] INFO [2022-03-13 17:15:14,076] jepsen worker 74 - jepsen.util 3314 :invoke :read [24 nil] INFO [2022-03-13 17:15:14,078] jepsen worker 50 - jepsen.util 3200 :ok :read [16 :nil] INFO [2022-03-13 17:15:14,079] jepsen worker 50 - jepsen.util 3200 :invoke :read [16 nil] INFO [2022-03-13 17:15:14,089] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,090] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:15:14,105] jepsen worker 74 - jepsen.util 3314 :ok :read [24 24] INFO [2022-03-13 17:15:14,106] jepsen worker 74 - jepsen.util 3314 :invoke :write [24 46] INFO [2022-03-13 17:15:14,114] jepsen worker 50 - jepsen.util 3200 :ok :read [16 :nil] INFO [2022-03-13 17:15:14,114] jepsen worker 50 - jepsen.util 3200 :invoke :write [16 17] INFO [2022-03-13 17:15:14,118] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,118] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,126] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,127] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,128] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:14,128] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:15:14,138] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,139] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:15:14,189] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,190] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,195] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,196] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,196] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:15:14,197] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:14,209] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,209] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:14,230] jepsen worker 89 - jepsen.util 2519 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,231] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 14] INFO [2022-03-13 17:15:14,240] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,241] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,247] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,248] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:15:14,267] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,267] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,268] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,268] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,269] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,270] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,271] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,271] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,271] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:15:14,273] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,273] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:15:14,274] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,275] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:14,275] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:15:14,275] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,276] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:15:14,292] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,294] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,299] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,299] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,300] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:14,301] jepsen worker nemesis - couchbase.util Rebalance check count: 423 INFO [2022-03-13 17:15:14,301] jepsen worker nemesis - couchbase.util Rebalance stuck count: 323 INFO [2022-03-13 17:15:14,302] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,303] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:15:14,339] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,339] jepsen worker 1 - jepsen.util 2701 :ok :read [0 41] INFO [2022-03-13 17:15:14,340] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:14,340] jepsen worker 1 - jepsen.util 2701 :invoke :read [0 nil] INFO [2022-03-13 17:15:14,344] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,345] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:14,348] jepsen worker 79 - jepsen.util 1069 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,348] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,348] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,348] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,348] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:14,351] jepsen worker 8 - jepsen.util 728 :invoke :write [2 12] INFO [2022-03-13 17:15:14,351] jepsen worker 82 - jepsen.util 3052 :ok :read [27 4] INFO [2022-03-13 17:15:14,352] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 11] INFO [2022-03-13 17:15:14,353] jepsen worker 26 - jepsen.util 476 :invoke :write [8 22] INFO [2022-03-13 17:15:14,354] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:14,354] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:14,356] jepsen worker 82 - jepsen.util 3052 :invoke :write [27 44] INFO [2022-03-13 17:15:14,363] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,364] jepsen worker 75 - jepsen.util 435 :invoke :write [25 37] INFO [2022-03-13 17:15:14,370] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,371] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,373] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,374] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:15:14,384] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,385] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:15:14,386] jepsen worker 84 - jepsen.util 3324 :info :write [28 31] :RequestTimeoutException INFO [2022-03-13 17:15:14,389] jepsen worker 84 - jepsen.util 3414 :invoke :write [28 49] INFO [2022-03-13 17:15:14,391] jepsen worker 15 - jepsen.util 3255 :ok :read [5 43] INFO [2022-03-13 17:15:14,392] jepsen worker 15 - jepsen.util 3255 :invoke :read [5 nil] INFO [2022-03-13 17:15:14,395] jepsen worker 67 - jepsen.util 2677 :ok :read [22 30] INFO [2022-03-13 17:15:14,395] jepsen worker 67 - jepsen.util 2677 :invoke :read [22 nil] INFO [2022-03-13 17:15:14,406] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,406] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,407] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:14,408] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,440] jepsen worker 37 - jepsen.util 2107 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,441] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 31] INFO [2022-03-13 17:15:14,450] jepsen worker 13 - jepsen.util 643 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,450] jepsen worker 78 - jepsen.util 528 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,451] jepsen worker 34 - jepsen.util 754 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,451] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,451] jepsen worker 80 - jepsen.util 710 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,451] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:14,452] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:14,453] jepsen worker 78 - jepsen.util 528 :invoke :write [26 40] INFO [2022-03-13 17:15:14,453] jepsen worker 34 - jepsen.util 754 :invoke :write [11 7] INFO [2022-03-13 17:15:14,455] jepsen worker 80 - jepsen.util 710 :invoke :write [26 2] INFO [2022-03-13 17:15:14,467] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,468] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:15:14,477] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,478] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:14,484] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,484] jepsen worker 44 - jepsen.util 584 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,484] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,484] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,484] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,484] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:14,485] jepsen worker 44 - jepsen.util 584 :invoke :write [14 26] INFO [2022-03-13 17:15:14,486] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,487] jepsen worker 14 - jepsen.util 374 :invoke :write [4 34] INFO [2022-03-13 17:15:14,490] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:15:14,490] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:15:14,491] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:14,493] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,494] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:15:14,495] jepsen worker 33 - jepsen.util 663 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,495] jepsen worker 33 - jepsen.util 663 :invoke :write [11 22] INFO [2022-03-13 17:15:14,513] jepsen worker 42 - jepsen.util 402 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,514] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] INFO [2022-03-13 17:15:14,545] jepsen worker 89 - jepsen.util 2519 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,546] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,554] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,554] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,554] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,555] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,555] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,556] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,557] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,557] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,558] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,559] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:15:14,580] jepsen worker 73 - jepsen.util 3223 :info :write [24 22] :RequestTimeoutException INFO [2022-03-13 17:15:14,580] jepsen worker 73 - jepsen.util 3313 :invoke :write [24 42] INFO [2022-03-13 17:15:14,588] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,588] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:14,601] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,602] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:14,636] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,637] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:15:14,764] jepsen worker 14 - jepsen.util 374 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,764] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:14,772] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,772] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,772] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:14,773] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:14,780] jepsen worker 77 - jepsen.util 707 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,780] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:15:14,795] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,796] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:15:14,824] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,825] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:14,923] jepsen worker 75 - jepsen.util 435 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,924] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:14,924] jepsen worker 7 - jepsen.util 637 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,924] jepsen worker 7 - jepsen.util 637 :invoke :write [2 46] INFO [2022-03-13 17:15:14,935] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,936] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,936] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:14,937] jepsen worker 14 - jepsen.util 374 :invoke :write [4 31] INFO [2022-03-13 17:15:14,938] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,938] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 1] INFO [2022-03-13 17:15:14,961] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,961] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,961] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,964] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:15:14,965] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,966] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:15:14,967] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,968] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,969] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,970] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:15:14,975] jepsen worker 37 - jepsen.util 2107 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,975] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,975] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,976] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,976] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 39] INFO [2022-03-13 17:15:14,979] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:14,980] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,980] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:14,980] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,982] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,982] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,982] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:14,983] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:14,983] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,984] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,984] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,985] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:15:14,985] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:14,986] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,987] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:14,988] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,989] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 32] INFO [2022-03-13 17:15:14,989] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,991] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:14,994] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:14,995] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:15:15,012] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,013] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:15:15,065] jepsen worker 31 - jepsen.util 3181 :info :write [10 2] :SyncWriteAmbiguous INFO [2022-03-13 17:15:15,066] jepsen worker 31 - jepsen.util 3271 :invoke :read [10 nil] INFO [2022-03-13 17:15:15,070] jepsen worker 3 - jepsen.util 1983 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,071] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,072] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,073] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,075] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,076] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,077] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,078] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:15,079] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,080] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:15:15,083] jepsen worker 37 - jepsen.util 2107 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,084] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 15] INFO [2022-03-13 17:15:15,084] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,085] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,085] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:15:15,085] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,085] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:15,086] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:15,144] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,145] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:15:15,160] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,161] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,164] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,164] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:15:15,209] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,210] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,211] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,211] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,212] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,213] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,214] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,214] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,216] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,217] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,218] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,218] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:15:15,220] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,220] jepsen worker 14 - jepsen.util 374 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,220] jepsen worker 17 - jepsen.util 3077 :info :write [5 26] :RequestTimeoutException INFO [2022-03-13 17:15:15,220] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,220] jepsen worker 17 - jepsen.util 3167 :invoke :read [5 nil] INFO [2022-03-13 17:15:15,221] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:15,222] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:15,222] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:15,222] jepsen worker 60 - jepsen.util 600 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,223] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,223] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,223] jepsen worker 60 - jepsen.util 600 :invoke :write [20 35] INFO [2022-03-13 17:15:15,224] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:15,224] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:15,231] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,231] jepsen worker 33 - jepsen.util 663 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,232] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,232] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,233] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:15:15,235] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:15:15,236] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:15:15,236] jepsen worker 24 - jepsen.util 564 :invoke :write [8 3] INFO [2022-03-13 17:15:15,253] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,253] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,253] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,253] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,255] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:15:15,256] jepsen worker 21 - jepsen.util 831 :invoke :write [7 48] INFO [2022-03-13 17:15:15,256] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:15,257] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:15,258] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,258] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,258] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:15:15,265] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:15:15,266] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,267] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,275] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,276] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:15,287] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,287] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,289] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,290] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:15:15,303] jepsen worker 37 - jepsen.util 2107 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,304] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 14] INFO [2022-03-13 17:15:15,307] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:15,307] jepsen worker nemesis - couchbase.util Rebalance check count: 424 INFO [2022-03-13 17:15:15,307] jepsen worker nemesis - couchbase.util Rebalance stuck count: 324 INFO [2022-03-13 17:15:15,346] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,346] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,346] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,347] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:15:15,347] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,348] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 47] INFO [2022-03-13 17:15:15,355] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,356] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,360] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,360] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:15:15,363] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,363] jepsen worker 36 - jepsen.util 2196 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,364] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,365] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:15:15,368] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,368] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,369] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,369] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:15,370] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,370] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 41] INFO [2022-03-13 17:15:15,419] jepsen worker 21 - jepsen.util 831 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,419] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:15,437] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,438] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:15,440] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,440] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:15:15,456] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,457] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,538] jepsen worker 63 - jepsen.util 3123 :ok :read [21 :nil] INFO [2022-03-13 17:15:15,538] jepsen worker 63 - jepsen.util 3123 :invoke :read [21 nil] INFO [2022-03-13 17:15:15,541] jepsen worker 63 - jepsen.util 3123 :ok :read [21 :nil] INFO [2022-03-13 17:15:15,541] jepsen worker 63 - jepsen.util 3123 :invoke :read [21 nil] INFO [2022-03-13 17:15:15,543] jepsen worker 63 - jepsen.util 3123 :ok :read [21 :nil] INFO [2022-03-13 17:15:15,544] jepsen worker 63 - jepsen.util 3123 :invoke :read [21 nil] INFO [2022-03-13 17:15:15,546] jepsen worker 63 - jepsen.util 3123 :ok :read [21 :nil] INFO [2022-03-13 17:15:15,546] jepsen worker 63 - jepsen.util 3123 :invoke :read [21 nil] INFO [2022-03-13 17:15:15,551] jepsen worker 24 - jepsen.util 564 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,552] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:15:15,559] jepsen worker 51 - jepsen.util 591 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,559] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,560] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,560] jepsen worker 12 - jepsen.util 732 :invoke :write [4 16] INFO [2022-03-13 17:15:15,561] jepsen worker 76 - jepsen.util 706 :invoke :write [25 8] INFO [2022-03-13 17:15:15,561] jepsen worker 51 - jepsen.util 591 :invoke :write [17 26] INFO [2022-03-13 17:15:15,570] jepsen worker 57 - jepsen.util 3027 :info :write [19 15] :RequestTimeoutException INFO [2022-03-13 17:15:15,571] jepsen worker 57 - jepsen.util 3117 :invoke :write [19 48] INFO [2022-03-13 17:15:15,579] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,580] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,582] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,583] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:15:15,587] jepsen worker 63 - jepsen.util 3123 :ok :read [21 :nil] INFO [2022-03-13 17:15:15,589] jepsen worker 63 - jepsen.util 3123 :invoke :read [21 nil] INFO [2022-03-13 17:15:15,591] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,592] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:15:15,593] jepsen worker 63 - jepsen.util 3123 :ok :read [21 :nil] INFO [2022-03-13 17:15:15,593] jepsen worker 63 - jepsen.util 3123 :invoke :write [21 8] INFO [2022-03-13 17:15:15,616] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,617] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:15:15,727] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,728] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:15,730] jepsen worker 41 - jepsen.util 3011 :info :write [13 12] :RequestTimeoutException INFO [2022-03-13 17:15:15,731] jepsen worker 41 - jepsen.util 3101 :invoke :write [13 29] INFO [2022-03-13 17:15:15,731] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,732] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:15:15,735] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,737] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:15:15,749] jepsen worker 62 - jepsen.util 512 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,749] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,749] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:15,749] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:15,750] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:15,750] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:15,789] jepsen worker 88 - jepsen.util 2068 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,790] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 47] INFO [2022-03-13 17:15:15,794] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,795] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:15:15,799] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,800] jepsen worker 35 - jepsen.util 935 :invoke :write [11 3] INFO [2022-03-13 17:15:15,840] jepsen worker 61 - jepsen.util 691 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,840] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:15,881] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,881] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,884] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,884] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,885] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,886] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:15:15,896] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,896] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,896] jepsen worker 37 - jepsen.util 2107 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,896] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:15,898] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:15:15,898] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:15,899] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,900] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 33] INFO [2022-03-13 17:15:15,924] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,924] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:15:15,928] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,929] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,929] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,929] jepsen worker 14 - jepsen.util 374 :invoke :write [4 33] INFO [2022-03-13 17:15:15,930] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:15:15,931] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:15,934] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,935] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,935] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:15,936] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:15:15,947] jepsen worker 8 - jepsen.util 728 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,948] jepsen worker 8 - jepsen.util 728 :invoke :write [2 23] INFO [2022-03-13 17:15:15,991] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:15,992] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:15,996] jepsen worker 19 - jepsen.util 2719 :info :write [6 12] :RequestTimeoutException INFO [2022-03-13 17:15:15,997] jepsen worker 19 - jepsen.util 2809 :invoke :write [6 13] INFO [2022-03-13 17:15:16,009] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,010] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:15:16,016] jepsen worker 47 - jepsen.util 2927 :info :write [15 17] :RequestTimeoutException INFO [2022-03-13 17:15:16,016] jepsen worker 47 - jepsen.util 3017 :invoke :read [15 nil] INFO [2022-03-13 17:15:16,035] jepsen worker 89 - jepsen.util 2519 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,036] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 12] INFO [2022-03-13 17:15:16,037] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,038] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:16,041] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,042] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:15:16,080] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,080] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:16,111] jepsen worker 35 - jepsen.util 935 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,111] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,112] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:16,112] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:16,121] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,122] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:16,149] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,150] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:16,165] jepsen worker 85 - jepsen.util 2875 :ok :read [28 44] INFO [2022-03-13 17:15:16,165] jepsen worker 85 - jepsen.util 2875 :invoke :write [28 47] INFO [2022-03-13 17:15:16,183] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,184] jepsen worker 33 - jepsen.util 663 :invoke :write [11 5] INFO [2022-03-13 17:15:16,198] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,198] jepsen worker 37 - jepsen.util 2107 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,199] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:15:16,200] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 26] INFO [2022-03-13 17:15:16,200] jepsen worker 88 - jepsen.util 2068 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,201] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:15:16,202] jepsen worker 44 - jepsen.util 584 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,203] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,203] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,205] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,205] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,208] jepsen worker 69 - jepsen.util 699 :invoke :write [23 0] INFO [2022-03-13 17:15:16,209] jepsen worker 23 - jepsen.util 563 :invoke :write [7 8] INFO [2022-03-13 17:15:16,209] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,210] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,216] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:16,217] jepsen worker 44 - jepsen.util 584 :invoke :write [14 30] INFO [2022-03-13 17:15:16,217] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:15:16,219] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:15:16,224] jepsen worker 77 - jepsen.util 707 :invoke :write [25 29] INFO [2022-03-13 17:15:16,226] jepsen worker 89 - jepsen.util 2519 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,227] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,253] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,253] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,254] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,254] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:15:16,255] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,255] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,256] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,256] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,257] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,258] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,259] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,259] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,260] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,261] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,266] jepsen worker 32 - jepsen.util 3002 :info :write [10 5] :RequestTimeoutException INFO [2022-03-13 17:15:16,266] jepsen worker 32 - jepsen.util 3092 :invoke :write [10 39] INFO [2022-03-13 17:15:16,312] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:16,312] jepsen worker nemesis - couchbase.util Rebalance check count: 425 INFO [2022-03-13 17:15:16,312] jepsen worker nemesis - couchbase.util Rebalance stuck count: 325 INFO [2022-03-13 17:15:16,323] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,323] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,324] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,324] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:15:16,325] jepsen worker 71 - jepsen.util 521 :invoke :write [23 0] INFO [2022-03-13 17:15:16,325] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:16,341] jepsen worker 79 - jepsen.util 1069 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,342] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:16,348] jepsen worker 14 - jepsen.util 374 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,348] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,348] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,348] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,349] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:16,350] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,350] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:15:16,350] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:15:16,351] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:16,352] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,353] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:15:16,357] jepsen worker 33 - jepsen.util 663 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,358] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,359] jepsen worker 33 - jepsen.util 663 :invoke :write [11 38] INFO [2022-03-13 17:15:16,359] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:16,366] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:16,386] jepsen worker 28 - jepsen.util 3178 :info :write [9 40] :RequestTimeoutException INFO [2022-03-13 17:15:16,387] jepsen worker 28 - jepsen.util 3268 :invoke :read [9 nil] INFO [2022-03-13 17:15:16,402] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,402] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,403] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:15:16,403] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 8] INFO [2022-03-13 17:15:16,404] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,405] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:16,417] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,418] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 9] INFO [2022-03-13 17:15:16,453] jepsen worker 37 - jepsen.util 2107 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,453] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 21] INFO [2022-03-13 17:15:16,477] jepsen worker 54 - jepsen.util 2664 :info :write [18 29] :RequestTimeoutException INFO [2022-03-13 17:15:16,478] jepsen worker 54 - jepsen.util 2754 :invoke :read [18 nil] INFO [2022-03-13 17:15:16,538] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,539] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:16,562] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,563] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:16,584] jepsen worker 60 - jepsen.util 600 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,585] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] WARN [2022-03-13 17:15:16,592] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":32837,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7854155,"total_server_duration_us":310320,"encode_duration_us":17499,"operation_id":"0xc0c7","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10007816},{"operation_name":"upsert","last_dispatch_duration_us":30268,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1522864,"total_server_duration_us":399777,"encode_duration_us":17339,"operation_id":"0xc478","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10007175},{"operation_name":"upsert","last_dispatch_duration_us":162060,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9650796,"total_server_duration_us":226911,"encode_duration_us":35792,"operation_id":"0xbfd1","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10006049},{"operation_name":"upsert","last_dispatch_duration_us":28990,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1486668,"total_server_duration_us":409197,"encode_duration_us":18458,"operation_id":"0xc48e","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10005789},{"operation_name":"upsert","last_dispatch_duration_us":33999,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":482727,"total_server_duration_us":287708,"encode_duration_us":17794,"operation_id":"0xc08d","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10003433},{"operation_name":"upsert","last_dispatch_duration_us":128951,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9736984,"total_server_duration_us":188824,"encode_duration_us":25881,"operation_id":"0xbfeb","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10002733},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":17541,"operation_id":"0xc3f7","timeout_ms":10000,"total_duration_us":10000523},{"operation_name":"upsert","last_dispatch_duration_us":9057500,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9379341,"total_server_duration_us":9251366,"encode_duration_us":23777,"operation_id":"0xc156","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9976105},{"operation_name":"upsert","last_dispatch_duration_us":1485173,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1485173,"total_server_duration_us":82259,"encode_duration_us":34412,"operation_id":"0xc5dd","timeout_ms":10000,"last_server_duration_us":82259,"total_duration_us":1491643},{"operation_name":"upsert","last_dispatch_duration_us":1432423,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1432423,"total_server_duration_us":1317051,"encode_duration_us":29912,"operation_id":"0xc5c1","timeout_ms":10000,"last_server_duration_us":1317051,"total_duration_us":1438318}],"total_count":37}} INFO [2022-03-13 17:15:16,607] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,607] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,607] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:15:16,608] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:15:16,663] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,664] jepsen worker 89 - jepsen.util 2519 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,664] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 3] INFO [2022-03-13 17:15:16,664] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:16,700] jepsen worker 37 - jepsen.util 2107 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,700] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,700] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 36] INFO [2022-03-13 17:15:16,701] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:16,719] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,720] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,721] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,721] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,721] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:15:16,722] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:15:16,722] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:16,724] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 22] INFO [2022-03-13 17:15:16,726] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,727] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:16,745] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,746] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:15:16,763] jepsen worker 71 - jepsen.util 521 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,764] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,764] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:16,765] jepsen worker 24 - jepsen.util 564 :invoke :write [8 14] INFO [2022-03-13 17:15:16,771] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,772] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:16,789] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:16,790] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:16,826] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,827] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:16,849] jepsen worker 89 - jepsen.util 2519 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,850] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:16,910] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:16,911] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:16,940] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:16,941] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:16,967] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:16,967] jepsen worker 25 - jepsen.util 565 :invoke :write [8 10] INFO [2022-03-13 17:15:16,991] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:16,992] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:17,027] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,027] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:15:17,036] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:17,037] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:17,113] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:17,113] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:17,144] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:17,144] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:17,160] jepsen worker 33 - jepsen.util 663 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,160] jepsen worker 33 - jepsen.util 663 :invoke :write [11 47] INFO [2022-03-13 17:15:17,160] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,161] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:15:17,168] jepsen worker 76 - jepsen.util 706 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,168] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:17,172] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:17,172] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,173] jepsen worker 66 - jepsen.util 2946 :invoke :read [22 nil] INFO [2022-03-13 17:15:17,173] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:17,176] jepsen worker 48 - jepsen.util 2928 :info :write [16 47] :RequestTimeoutException INFO [2022-03-13 17:15:17,177] jepsen worker 48 - jepsen.util 3018 :invoke :read [16 nil] INFO [2022-03-13 17:15:17,202] jepsen worker 66 - jepsen.util 2946 :ok :read [22 30] INFO [2022-03-13 17:15:17,203] jepsen worker 66 - jepsen.util 2946 :invoke :write [22 21] INFO [2022-03-13 17:15:17,212] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,212] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,212] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:17,213] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:17,318] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:17,318] jepsen worker nemesis - couchbase.util Rebalance check count: 426 INFO [2022-03-13 17:15:17,318] jepsen worker nemesis - couchbase.util Rebalance stuck count: 326 INFO [2022-03-13 17:15:17,364] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,364] jepsen worker 24 - jepsen.util 564 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,365] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:15:17,365] jepsen worker 7 - jepsen.util 637 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,365] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,365] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:15:17,365] jepsen worker 7 - jepsen.util 637 :invoke :write [2 30] INFO [2022-03-13 17:15:17,366] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:17,390] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,391] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:15:17,576] jepsen worker 29 - jepsen.util 3179 :info :write [9 11] :RequestTimeoutException INFO [2022-03-13 17:15:17,576] jepsen worker 29 - jepsen.util 3269 :invoke :read [9 nil] INFO [2022-03-13 17:15:17,776] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,777] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:15:17,779] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,779] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,779] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:17,780] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:17,781] jepsen worker 25 - jepsen.util 565 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,781] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:17,792] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,792] jepsen worker 51 - jepsen.util 591 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,792] jepsen worker 51 - jepsen.util 591 :invoke :write [17 34] INFO [2022-03-13 17:15:17,793] jepsen worker 61 - jepsen.util 691 :invoke :write [20 28] INFO [2022-03-13 17:15:17,808] jepsen worker 33 - jepsen.util 663 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,810] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] WARN [2022-03-13 17:15:17,904] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":14974373,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14974373,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9029938,"encode_duration_us":26193,"operation_id":"0xc0f7","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10003691},{"operation_name":"upsert","last_dispatch_duration_us":11903147,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11903147,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9039489,"encode_duration_us":17721,"operation_id":"0xc36f","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10009952}],"total_count":2}} INFO [2022-03-13 17:15:17,979] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:17,980] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] WARN [2022-03-13 17:15:18,105] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11903147,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11903147,"total_server_duration_us":9039489,"encode_duration_us":17721,"operation_id":"0xc36f","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10009952},{"operation_name":"upsert","last_dispatch_duration_us":14974373,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":14974373,"total_server_duration_us":9029938,"encode_duration_us":26193,"operation_id":"0xc0f7","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10003691},{"operation_name":"upsert","last_dispatch_duration_us":11218693,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":11218693,"total_server_duration_us":19966,"encode_duration_us":19999,"operation_id":"0xc338","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10002644},{"operation_name":"upsert","last_dispatch_duration_us":8982216,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9022335,"total_server_duration_us":48477,"encode_duration_us":20186,"operation_id":"0xc4db","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10000991},{"operation_name":"get","last_dispatch_duration_us":9104655,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9104655,"total_server_duration_us":69,"operation_id":"0xc445","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":9118058},{"operation_name":"get","last_dispatch_duration_us":9112628,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9112628,"total_server_duration_us":32,"operation_id":"0xc449","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9116214},{"operation_name":"get","last_dispatch_duration_us":8465392,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8465392,"total_server_duration_us":126,"operation_id":"0xc49c","timeout_ms":30000,"last_server_duration_us":126,"total_duration_us":8473663},{"operation_name":"get","last_dispatch_duration_us":5991077,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5991077,"total_server_duration_us":32,"operation_id":"0xc5f4","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":5994745},{"operation_name":"upsert","last_dispatch_duration_us":3090268,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3090268,"total_server_duration_us":47079,"encode_duration_us":22329,"operation_id":"0xc564","timeout_ms":10000,"last_server_duration_us":47079,"total_duration_us":3128378},{"operation_name":"upsert","last_dispatch_duration_us":2814349,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2814349,"total_server_duration_us":74648,"encode_duration_us":19084,"operation_id":"0xc560","timeout_ms":10000,"last_server_duration_us":74648,"total_duration_us":2829815}],"total_count":61}} WARN [2022-03-13 17:15:18,105] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10922010,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10922010,"total_server_duration_us":49,"operation_id":"0xc337","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":10922342},{"operation_name":"upsert","last_dispatch_duration_us":10134865,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10134865,"total_server_duration_us":26953,"encode_duration_us":19025,"operation_id":"0xc3aa","timeout_ms":10000,"last_server_duration_us":26953,"total_duration_us":10010272},{"operation_name":"upsert","last_dispatch_duration_us":30998,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9643763,"total_server_duration_us":209406,"encode_duration_us":18904,"operation_id":"0xc376","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10009577},{"operation_name":"upsert","last_dispatch_duration_us":35968,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9428419,"total_server_duration_us":228041,"encode_duration_us":19039,"operation_id":"0xc448","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10009158},{"operation_name":"upsert","last_dispatch_duration_us":11876165,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11876165,"total_server_duration_us":20203,"encode_duration_us":19259,"operation_id":"0xc1e4","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10002558},{"operation_name":"upsert","last_dispatch_duration_us":14688138,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":14688138,"total_server_duration_us":31969,"encode_duration_us":22881,"operation_id":"0xc0f8","timeout_ms":10000,"last_server_duration_us":31969,"total_duration_us":10001633},{"operation_name":"get","last_dispatch_duration_us":7330843,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7330843,"total_server_duration_us":27,"operation_id":"0xc50e","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":7333022},{"operation_name":"upsert","last_dispatch_duration_us":4737799,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4737799,"total_server_duration_us":46626,"encode_duration_us":24202,"operation_id":"0xc674","timeout_ms":10000,"last_server_duration_us":46626,"total_duration_us":4742150},{"operation_name":"upsert","last_dispatch_duration_us":1802935,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1802935,"total_server_duration_us":1311442,"encode_duration_us":38372,"operation_id":"0xc587","timeout_ms":10000,"last_server_duration_us":1311442,"total_duration_us":1808490},{"operation_name":"get","last_dispatch_duration_us":1389946,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1389946,"total_server_duration_us":38,"operation_id":"0xc8e2","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":1390140}],"total_count":17}} WARN [2022-03-13 17:15:18,107] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17529120,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18650578,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9332693,"encode_duration_us":17977,"operation_id":"0xbaec","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10005807},{"operation_name":"upsert","last_dispatch_duration_us":17264390,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17264390,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9034183,"encode_duration_us":50683,"operation_id":"0xbd6f","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10009806}],"total_count":2}} WARN [2022-03-13 17:15:18,109] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":30898,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1063015,"total_server_duration_us":267484,"encode_duration_us":17973,"operation_id":"0xc591","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10010463},{"operation_name":"upsert","last_dispatch_duration_us":17964488,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18256984,"total_server_duration_us":9085885,"encode_duration_us":27711,"operation_id":"0xc079","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007725},{"operation_name":"upsert","last_dispatch_duration_us":40517,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5138304,"total_server_duration_us":282058,"encode_duration_us":19325,"operation_id":"0xc359","timeout_ms":10000,"last_server_duration_us":28401,"total_duration_us":10007065},{"operation_name":"upsert","last_dispatch_duration_us":33224,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9002578,"total_server_duration_us":244626,"encode_duration_us":21824,"operation_id":"0xc11d","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10006966},{"operation_name":"upsert","last_dispatch_duration_us":8787473,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8787473,"total_server_duration_us":22306,"encode_duration_us":21882,"operation_id":"0xc28a","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10006061},{"operation_name":"upsert","last_dispatch_duration_us":43865,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9795830,"total_server_duration_us":193798,"encode_duration_us":17772,"operation_id":"0xc0a4","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10005275},{"operation_name":"upsert","last_dispatch_duration_us":9075549,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18161892,"total_server_duration_us":65240,"encode_duration_us":26036,"operation_id":"0xc096","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10004863},{"operation_name":"upsert","last_dispatch_duration_us":8949879,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9337583,"total_server_duration_us":88214,"encode_duration_us":26833,"operation_id":"0xc076","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10003711},{"operation_name":"upsert","last_dispatch_duration_us":8804755,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8804755,"total_server_duration_us":19887,"encode_duration_us":31278,"operation_id":"0xc3d4","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10002695},{"operation_name":"upsert","last_dispatch_duration_us":17938943,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18212376,"total_server_duration_us":9070812,"encode_duration_us":32354,"operation_id":"0xc077","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10002013}],"total_count":35}} WARN [2022-03-13 17:15:18,109] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17938943,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18212376,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9070812,"encode_duration_us":32354,"operation_id":"0xc077","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10002013},{"operation_name":"upsert","last_dispatch_duration_us":17964488,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18256984,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9085885,"encode_duration_us":27711,"operation_id":"0xc079","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007725}],"total_count":2}} INFO [2022-03-13 17:15:18,203] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,203] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] WARN [2022-03-13 17:15:18,205] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18074449,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18074449,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9043735,"encode_duration_us":18995,"operation_id":"0xbe5d","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10003732},{"operation_name":"upsert","last_dispatch_duration_us":17164747,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17709184,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9385739,"encode_duration_us":27017,"operation_id":"0xbd67","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10010433},{"operation_name":"upsert","last_dispatch_duration_us":16916975,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17394375,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9343919,"encode_duration_us":18196,"operation_id":"0xbe39","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10004385}],"total_count":3}} WARN [2022-03-13 17:15:18,207] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12093933,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12093933,"total_server_duration_us":6143,"operation_id":"0xc06e","timeout_ms":30000,"last_server_duration_us":6143,"total_duration_us":12095928},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":18218,"operation_id":"0xc3ac","timeout_ms":10000,"total_duration_us":10006468},{"operation_name":"upsert","last_dispatch_duration_us":8133579,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8754090,"total_server_duration_us":202241,"encode_duration_us":18052,"operation_id":"0xc1f7","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10006447},{"operation_name":"upsert","last_dispatch_duration_us":8239855,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8824852,"total_server_duration_us":185583,"encode_duration_us":20528,"operation_id":"0xc1f3","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10003438},{"operation_name":"upsert","last_dispatch_duration_us":9060315,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9060315,"total_server_duration_us":9047983,"encode_duration_us":28081,"operation_id":"0xc1eb","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9066362},{"operation_name":"get","last_dispatch_duration_us":8189529,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8189529,"total_server_duration_us":27,"operation_id":"0xc279","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":8229538},{"operation_name":"get","last_dispatch_duration_us":6516116,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6516116,"total_server_duration_us":27,"operation_id":"0xc34f","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":6518457},{"operation_name":"upsert","last_dispatch_duration_us":2623009,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2623009,"total_server_duration_us":155181,"encode_duration_us":20634,"operation_id":"0xc348","timeout_ms":10000,"last_server_duration_us":155181,"total_duration_us":6263591},{"operation_name":"get","last_dispatch_duration_us":3481487,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":3481487,"total_server_duration_us":23,"operation_id":"0xc520","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":3482854},{"operation_name":"upsert","last_dispatch_duration_us":2691168,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2691168,"total_server_duration_us":129065,"encode_duration_us":21754,"operation_id":"0xc554","timeout_ms":10000,"last_server_duration_us":129065,"total_duration_us":2695346}],"total_count":58}} WARN [2022-03-13 17:15:18,309] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13159635,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":13159635,"total_server_duration_us":43,"operation_id":"0xc32f","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":13161986},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":22478,"operation_id":"0xc545","timeout_ms":10000,"total_duration_us":10010713},{"operation_name":"upsert","last_dispatch_duration_us":32521,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9254244,"total_server_duration_us":201870,"encode_duration_us":21148,"operation_id":"0xc4a4","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10009652},{"operation_name":"upsert","last_dispatch_duration_us":156457,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":454142,"total_server_duration_us":62177,"encode_duration_us":18683,"operation_id":"0xc4c2","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10009291},{"operation_name":"upsert","last_dispatch_duration_us":148253,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":547144,"total_server_duration_us":124150,"encode_duration_us":20888,"operation_id":"0xc4a3","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10007102},{"operation_name":"upsert","last_dispatch_duration_us":8509869,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16555839,"total_server_duration_us":183414,"encode_duration_us":20772,"operation_id":"0xc084","timeout_ms":10000,"last_server_duration_us":33622,"total_duration_us":10006442},{"operation_name":"upsert","last_dispatch_duration_us":151725,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":298757,"total_server_duration_us":43651,"encode_duration_us":27710,"operation_id":"0xc4c7","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10003541},{"operation_name":"upsert","last_dispatch_duration_us":9079104,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9079104,"total_server_duration_us":9059667,"encode_duration_us":23690,"operation_id":"0xc4a2","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":9084183},{"operation_name":"upsert","last_dispatch_duration_us":7507410,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7507410,"total_server_duration_us":104546,"encode_duration_us":18739,"operation_id":"0xc515","timeout_ms":10000,"last_server_duration_us":104546,"total_duration_us":7686567},{"operation_name":"get","last_dispatch_duration_us":7254363,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7254363,"total_server_duration_us":38,"operation_id":"0xc541","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":7315360}],"total_count":15}} INFO [2022-03-13 17:15:18,310] jepsen worker 37 - jepsen.util 2107 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,310] jepsen worker 5 - jepsen.util 2435 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,311] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,311] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 46] INFO [2022-03-13 17:15:18,312] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,312] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,313] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:15:18,313] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:15:18,326] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:18,326] jepsen worker nemesis - couchbase.util Rebalance check count: 427 INFO [2022-03-13 17:15:18,326] jepsen worker nemesis - couchbase.util Rebalance stuck count: 327 INFO [2022-03-13 17:15:18,361] jepsen worker 16 - jepsen.util 2896 :info :write [5 14] :RequestTimeoutException INFO [2022-03-13 17:15:18,362] jepsen worker 16 - jepsen.util 2986 :invoke :read [5 nil] INFO [2022-03-13 17:15:18,366] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,366] jepsen worker 36 - jepsen.util 2196 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,367] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,367] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,368] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,368] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,368] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,368] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:15:18,369] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 48] INFO [2022-03-13 17:15:18,370] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,371] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,372] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,373] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,374] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,379] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,379] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,448] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,379] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:15:18,448] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] WARN [2022-03-13 17:15:18,409] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":27189403,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":36374196,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9127638,"encode_duration_us":24016,"operation_id":"0xb485","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10002583},{"operation_name":"upsert","last_dispatch_duration_us":27190651,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":36355448,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9093239,"encode_duration_us":17774,"operation_id":"0xb494","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10008149},{"operation_name":"upsert","last_dispatch_duration_us":24346768,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24346768,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9049045,"encode_duration_us":17784,"operation_id":"0xbcb4","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10007313}],"total_count":3}} INFO [2022-03-13 17:15:18,418] jepsen worker 37 - jepsen.util 2107 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,448] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,448] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,500] jepsen worker 55 - jepsen.util 2935 :info :write [18 39] :RequestTimeoutException INFO [2022-03-13 17:15:18,500] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,501] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:18,501] jepsen worker 55 - jepsen.util 3025 :invoke :read [18 nil] INFO [2022-03-13 17:15:18,516] jepsen worker 89 - jepsen.util 2519 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,516] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,517] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,517] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,517] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,518] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,519] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,519] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,519] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:15:18,520] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 9] INFO [2022-03-13 17:15:18,521] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,521] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,523] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,524] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:15:18,549] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,549] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,550] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:15:18,550] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 34] INFO [2022-03-13 17:15:18,566] jepsen worker 40 - jepsen.util 3280 :info :write [13 5] :RequestTimeoutException INFO [2022-03-13 17:15:18,567] jepsen worker 40 - jepsen.util 3370 :invoke :read [13 nil] INFO [2022-03-13 17:15:18,572] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,572] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,573] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:18,573] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 1] INFO [2022-03-13 17:15:18,577] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,577] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:15:18,592] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,592] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,593] jepsen worker 75 - jepsen.util 435 :invoke :write [25 36] INFO [2022-03-13 17:15:18,593] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:18,595] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,595] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,595] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,595] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,596] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:15:18,596] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:18,597] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:15:18,597] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:15:18,598] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,599] jepsen worker 26 - jepsen.util 476 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,599] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:18,599] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,599] jepsen worker 43 - jepsen.util 583 :invoke :write [14 21] INFO [2022-03-13 17:15:18,600] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:18,606] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:18,606] jepsen worker 53 - jepsen.util 503 :invoke :write [17 27] INFO [2022-03-13 17:15:18,610] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,611] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,611] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,611] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:18,612] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:15:18,619] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:15:18,637] jepsen worker 20 - jepsen.util 3440 :info :write [6 30] :RequestTimeoutException INFO [2022-03-13 17:15:18,637] jepsen worker 12 - jepsen.util 732 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,637] jepsen worker 20 - jepsen.util 3530 :invoke :read [6 nil] INFO [2022-03-13 17:15:18,638] jepsen worker 12 - jepsen.util 732 :invoke :write [4 39] INFO [2022-03-13 17:15:18,652] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,653] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,690] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,690] jepsen worker 37 - jepsen.util 2107 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,690] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,690] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,690] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,691] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:15:18,691] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,692] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,739] jepsen worker 89 - jepsen.util 2519 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,739] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,739] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,739] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,740] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:18,740] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:15:18,741] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,741] jepsen worker 3 - jepsen.util 1983 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,742] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 6] INFO [2022-03-13 17:15:18,743] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:15:18,744] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,745] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,745] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,746] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,808] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,808] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,809] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 1] INFO [2022-03-13 17:15:18,810] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:15:18,820] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,820] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:18,824] jepsen worker 9 - jepsen.util 2619 :info :write [3 27] :SyncWriteAmbiguous INFO [2022-03-13 17:15:18,825] jepsen worker 9 - jepsen.util 2709 :invoke :read [3 nil] INFO [2022-03-13 17:15:18,828] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,829] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:15:18,831] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,832] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,832] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,832] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:18,833] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,833] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,834] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:15:18,834] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:15:18,835] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,836] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:15:18,867] jepsen worker 27 - jepsen.util 3537 :ok :read [9 26] INFO [2022-03-13 17:15:18,867] jepsen worker 27 - jepsen.util 3537 :invoke :read [9 nil] INFO [2022-03-13 17:15:18,886] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:18,887] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:15:18,904] jepsen worker 30 - jepsen.util 2910 :ok :read [10 :nil] INFO [2022-03-13 17:15:18,905] jepsen worker 30 - jepsen.util 2910 :invoke :write [10 25] INFO [2022-03-13 17:15:18,906] jepsen worker 31 - jepsen.util 3271 :ok :read [10 :nil] INFO [2022-03-13 17:15:18,909] jepsen worker 31 - jepsen.util 3271 :invoke :write [10 29] INFO [2022-03-13 17:15:18,926] jepsen worker 11 - jepsen.util 3161 :ok :read [3 25] INFO [2022-03-13 17:15:18,926] jepsen worker 11 - jepsen.util 3161 :invoke :write [3 47] INFO [2022-03-13 17:15:18,960] jepsen worker 81 - jepsen.util 3501 :info :write [27 34] :RequestTimeoutException INFO [2022-03-13 17:15:18,961] jepsen worker 81 - jepsen.util 3591 :invoke :read [27 nil] INFO [2022-03-13 17:15:18,965] jepsen worker 55 - jepsen.util 3025 :ok :read [18 21] INFO [2022-03-13 17:15:18,965] jepsen worker 55 - jepsen.util 3025 :invoke :write [18 1] INFO [2022-03-13 17:15:18,970] jepsen worker 29 - jepsen.util 3269 :ok :read [9 26] INFO [2022-03-13 17:15:18,971] jepsen worker 29 - jepsen.util 3269 :invoke :write [9 45] INFO [2022-03-13 17:15:19,004] jepsen worker 27 - jepsen.util 3537 :ok :read [9 26] INFO [2022-03-13 17:15:19,005] jepsen worker 27 - jepsen.util 3537 :invoke :write [9 6] INFO [2022-03-13 17:15:19,058] jepsen worker 45 - jepsen.util 3285 :ok :read [15 48] INFO [2022-03-13 17:15:19,059] jepsen worker 45 - jepsen.util 3285 :invoke :read [15 nil] INFO [2022-03-13 17:15:19,078] jepsen worker 56 - jepsen.util 3296 :ok :read [18 21] INFO [2022-03-13 17:15:19,079] jepsen worker 56 - jepsen.util 3296 :invoke :read [18 nil] INFO [2022-03-13 17:15:19,153] jepsen worker 10 - jepsen.util 3070 :ok :read [3 25] INFO [2022-03-13 17:15:19,154] jepsen worker 10 - jepsen.util 3070 :invoke :read [3 nil] INFO [2022-03-13 17:15:19,287] jepsen worker 0 - jepsen.util 2880 :info :write [0 22] :RequestTimeoutException INFO [2022-03-13 17:15:19,288] jepsen worker 0 - jepsen.util 2970 :invoke :read [0 nil] INFO [2022-03-13 17:15:19,290] jepsen worker 83 - jepsen.util 2513 :info :write [27 14] :RequestTimeoutException INFO [2022-03-13 17:15:19,291] jepsen worker 83 - jepsen.util 2603 :invoke :read [27 nil] INFO [2022-03-13 17:15:19,333] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:19,333] jepsen worker nemesis - couchbase.util Rebalance check count: 428 INFO [2022-03-13 17:15:19,333] jepsen worker nemesis - couchbase.util Rebalance stuck count: 328 INFO [2022-03-13 17:15:19,476] jepsen worker 70 - jepsen.util 700 :info :write [23 42] :RequestTimeoutException INFO [2022-03-13 17:15:19,476] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:19,645] jepsen worker 59 - jepsen.util 3029 :info :write [19 39] :SyncWriteAmbiguous INFO [2022-03-13 17:15:19,646] jepsen worker 59 - jepsen.util 3119 :invoke :write [19 14] INFO [2022-03-13 17:15:19,751] jepsen worker 48 - jepsen.util 3018 :ok :read [16 :nil] INFO [2022-03-13 17:15:19,752] jepsen worker 48 - jepsen.util 3018 :invoke :read [16 nil] INFO [2022-03-13 17:15:19,779] jepsen worker 48 - jepsen.util 3018 :ok :read [16 :nil] INFO [2022-03-13 17:15:19,780] jepsen worker 48 - jepsen.util 3018 :invoke :write [16 13] INFO [2022-03-13 17:15:19,876] jepsen worker 72 - jepsen.util 3132 :info :write [24 9] :RequestTimeoutException INFO [2022-03-13 17:15:19,876] jepsen worker 72 - jepsen.util 3222 :invoke :read [24 nil] INFO [2022-03-13 17:15:19,963] jepsen worker 46 - jepsen.util 3016 :info :write [15 32] :RequestTimeoutException INFO [2022-03-13 17:15:19,964] jepsen worker 46 - jepsen.util 3106 :invoke :write [15 37] INFO [2022-03-13 17:15:20,338] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:20,338] jepsen worker nemesis - couchbase.util Rebalance check count: 429 INFO [2022-03-13 17:15:20,338] jepsen worker nemesis - couchbase.util Rebalance stuck count: 329 INFO [2022-03-13 17:15:20,394] jepsen worker 86 - jepsen.util 2876 :info :write [28 25] :RequestTimeoutException INFO [2022-03-13 17:15:20,394] jepsen worker 86 - jepsen.util 2966 :invoke :read [28 nil] INFO [2022-03-13 17:15:20,848] jepsen worker 64 - jepsen.util 3034 :info :write [21 10] :RequestTimeoutException INFO [2022-03-13 17:15:20,848] jepsen worker 64 - jepsen.util 3124 :invoke :read [21 nil] INFO [2022-03-13 17:15:20,880] jepsen worker 65 - jepsen.util 3485 :info :write [21 4] :RequestTimeoutException INFO [2022-03-13 17:15:20,881] jepsen worker 65 - jepsen.util 3575 :invoke :write [21 32] INFO [2022-03-13 17:15:21,343] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:21,343] jepsen worker nemesis - couchbase.util Rebalance check count: 430 INFO [2022-03-13 17:15:21,343] jepsen worker nemesis - couchbase.util Rebalance stuck count: 330 INFO [2022-03-13 17:15:21,854] jepsen worker 49 - jepsen.util 2929 :info :write [16 10] :RequestTimeoutException INFO [2022-03-13 17:15:21,854] jepsen worker 49 - jepsen.util 3019 :invoke :write [16 48] INFO [2022-03-13 17:15:22,015] jepsen worker 8 - jepsen.util 728 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,015] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,015] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:22,015] jepsen worker 8 - jepsen.util 728 :invoke :write [2 24] INFO [2022-03-13 17:15:22,015] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,016] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:22,016] jepsen worker 42 - jepsen.util 402 :invoke :write [14 43] INFO [2022-03-13 17:15:22,019] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,019] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,020] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,020] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:15:22,020] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:22,021] jepsen worker 33 - jepsen.util 663 :invoke :write [11 46] INFO [2022-03-13 17:15:22,021] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:22,086] jepsen worker 23 - jepsen.util 563 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,087] jepsen worker 23 - jepsen.util 563 :invoke :write [7 25] INFO [2022-03-13 17:15:22,109] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,110] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:22,151] jepsen worker 7 - jepsen.util 637 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,152] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,152] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:15:22,152] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:22,152] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,153] jepsen worker 62 - jepsen.util 512 :invoke :write [20 10] INFO [2022-03-13 17:15:22,155] jepsen worker 75 - jepsen.util 435 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,155] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:15:22,157] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,157] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:15:22,265] jepsen worker 44 - jepsen.util 584 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,265] jepsen worker 44 - jepsen.util 584 :invoke :write [14 32] INFO [2022-03-13 17:15:22,350] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:22,351] jepsen worker nemesis - couchbase.util Rebalance check count: 431 INFO [2022-03-13 17:15:22,351] jepsen worker nemesis - couchbase.util Rebalance stuck count: 331 INFO [2022-03-13 17:15:22,439] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,440] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:22,446] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,446] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:22,490] jepsen worker 53 - jepsen.util 503 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,491] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:22,636] jepsen worker 33 - jepsen.util 663 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,636] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:22,640] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,640] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:22,706] jepsen worker 69 - jepsen.util 699 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,707] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,707] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:22,708] jepsen worker 69 - jepsen.util 699 :invoke :write [23 23] INFO [2022-03-13 17:15:22,709] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,709] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,709] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,711] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:22,712] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:15:22,713] jepsen worker 21 - jepsen.util 831 :invoke :write [7 24] INFO [2022-03-13 17:15:22,848] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,848] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,848] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:22,849] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,849] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:22,849] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:22,850] jepsen worker 13 - jepsen.util 643 :invoke :write [4 43] INFO [2022-03-13 17:15:22,850] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:23,050] jepsen worker 43 - jepsen.util 583 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,050] jepsen worker 43 - jepsen.util 583 :invoke :write [14 20] INFO [2022-03-13 17:15:23,051] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,052] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,052] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,052] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:23,053] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:23,053] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:23,075] jepsen worker 58 - jepsen.util 2848 :info :write [19 6] :RequestTimeoutException INFO [2022-03-13 17:15:23,076] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,077] jepsen worker 58 - jepsen.util 2938 :invoke :read [19 nil] INFO [2022-03-13 17:15:23,077] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:15:23,102] jepsen worker 58 - jepsen.util 2938 :ok :read [19 16] INFO [2022-03-13 17:15:23,102] jepsen worker 58 - jepsen.util 2938 :invoke :write [19 35] INFO [2022-03-13 17:15:23,209] jepsen worker 39 - jepsen.util 3099 :info :write [13 39] :SyncWriteAmbiguous INFO [2022-03-13 17:15:23,210] jepsen worker 39 - jepsen.util 3189 :invoke :write [13 18] INFO [2022-03-13 17:15:23,252] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,252] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,252] jepsen worker 21 - jepsen.util 831 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,253] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:23,254] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:23,254] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:15:23,283] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,283] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,283] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:23,284] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,284] jepsen worker 14 - jepsen.util 374 :invoke :write [4 37] INFO [2022-03-13 17:15:23,285] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:23,356] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:23,356] jepsen worker nemesis - couchbase.util Rebalance check count: 432 INFO [2022-03-13 17:15:23,356] jepsen worker nemesis - couchbase.util Rebalance stuck count: 332 INFO [2022-03-13 17:15:23,456] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,456] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,456] jepsen worker 1 - jepsen.util 2701 :ok :read [0 41] INFO [2022-03-13 17:15:23,456] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,235] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:24,235] jepsen worker 1 - jepsen.util 2701 :invoke :read [0 nil] INFO [2022-03-13 17:15:24,235] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:23,457] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:15:23,463] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,463] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,236] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:15:24,236] jepsen worker 24 - jepsen.util 564 :invoke :write [8 38] INFO [2022-03-13 17:15:23,464] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,237] jepsen worker 33 - jepsen.util 663 :invoke :read [11 nil] INFO [2022-03-13 17:15:23,464] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,473] jepsen worker 77 - jepsen.util 707 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,238] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:24,238] jepsen worker 77 - jepsen.util 707 :invoke :write [25 39] INFO [2022-03-13 17:15:23,488] jepsen worker 14 - jepsen.util 374 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,513] jepsen worker 67 - jepsen.util 2677 :ok :read [22 30] INFO [2022-03-13 17:15:24,238] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:24,238] jepsen worker 67 - jepsen.util 2677 :invoke :read [22 nil] INFO [2022-03-13 17:15:23,519] jepsen worker 15 - jepsen.util 3255 :ok :read [5 43] INFO [2022-03-13 17:15:23,662] jepsen worker 8 - jepsen.util 728 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,244] jepsen worker 15 - jepsen.util 3255 :invoke :write [5 26] INFO [2022-03-13 17:15:24,244] jepsen worker 8 - jepsen.util 728 :invoke :write [2 27] INFO [2022-03-13 17:15:23,696] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,696] jepsen worker 12 - jepsen.util 732 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,867] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,867] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,867] jepsen worker 42 - jepsen.util 402 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,253] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:24,253] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:23,901] jepsen worker 23 - jepsen.util 563 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,902] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,926] jepsen worker 62 - jepsen.util 512 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,253] jepsen worker 60 - jepsen.util 600 :invoke :write [20 34] INFO [2022-03-13 17:15:23,926] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,253] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:24,253] jepsen worker 62 - jepsen.util 512 :invoke :write [20 8] INFO [2022-03-13 17:15:23,926] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:23,926] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,254] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:23,994] jepsen worker 18 - jepsen.util 2898 :info :write [6 7] :RequestTimeoutException INFO [2022-03-13 17:15:24,255] jepsen worker 18 - jepsen.util 2988 :invoke :read [6 nil] INFO [2022-03-13 17:15:24,008] jepsen worker 2 - jepsen.util 3602 :info :write [0 28] :RequestTimeoutException INFO [2022-03-13 17:15:24,255] jepsen worker 2 - jepsen.util 3692 :invoke :read [0 nil] INFO [2022-03-13 17:15:24,088] jepsen worker 44 - jepsen.util 584 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,140] jepsen worker 74 - jepsen.util 3314 :info :write [24 46] :RequestTimeoutException INFO [2022-03-13 17:15:24,150] jepsen worker 50 - jepsen.util 3200 :info :write [16 17] :RequestTimeoutException INFO [2022-03-13 17:15:24,259] jepsen worker 74 - jepsen.util 3404 :invoke :write [24 41] INFO [2022-03-13 17:15:24,259] jepsen worker 50 - jepsen.util 3290 :invoke :read [16 nil] INFO [2022-03-13 17:15:24,155] jepsen worker 54 - jepsen.util 2754 :ok :read [18 21] INFO [2022-03-13 17:15:24,253] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:24,262] jepsen worker 54 - jepsen.util 2754 :invoke :write [18 49] INFO [2022-03-13 17:15:24,253] jepsen worker 26 - jepsen.util 476 :invoke :write [8 28] INFO [2022-03-13 17:15:24,253] jepsen worker 7 - jepsen.util 637 :invoke :write [2 40] INFO [2022-03-13 17:15:24,253] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:24,254] jepsen worker 6 - jepsen.util 636 :invoke :write [2 25] INFO [2022-03-13 17:15:24,259] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:24,272] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,272] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,273] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,276] jepsen worker 33 - jepsen.util 663 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,277] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,283] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:15:24,284] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:15:24,285] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:15:24,285] jepsen worker 33 - jepsen.util 663 :invoke :write [11 10] INFO [2022-03-13 17:15:24,286] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:24,320] jepsen worker 43 - jepsen.util 583 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,321] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,321] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:24,321] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,321] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:24,322] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 41] INFO [2022-03-13 17:15:24,323] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:24,323] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:24,343] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,343] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:15:24,362] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:24,362] jepsen worker nemesis - couchbase.util Rebalance check count: 433 INFO [2022-03-13 17:15:24,362] jepsen worker nemesis - couchbase.util Rebalance stuck count: 333 INFO [2022-03-13 17:15:24,388] jepsen worker 82 - jepsen.util 3052 :info :write [27 44] :RequestTimeoutException INFO [2022-03-13 17:15:24,389] jepsen worker 82 - jepsen.util 3142 :invoke :read [27 nil] INFO [2022-03-13 17:15:24,398] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,399] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:24,416] jepsen worker 84 - jepsen.util 3414 :info :write [28 49] :RequestTimeoutException INFO [2022-03-13 17:15:24,417] jepsen worker 84 - jepsen.util 3504 :invoke :write [28 12] INFO [2022-03-13 17:15:24,427] jepsen worker 37 - jepsen.util 2107 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,427] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:24,434] jepsen worker 82 - jepsen.util 3142 :ok :read [27 4] INFO [2022-03-13 17:15:24,434] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,435] jepsen worker 82 - jepsen.util 3142 :invoke :write [27 31] INFO [2022-03-13 17:15:24,435] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:15:24,450] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,451] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:15:24,476] jepsen worker 34 - jepsen.util 754 :info :write [11 7] :RequestTimeoutException INFO [2022-03-13 17:15:24,477] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:24,478] jepsen worker 18 - jepsen.util 2988 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,479] jepsen worker 18 - jepsen.util 2988 :invoke :read [6 nil] INFO [2022-03-13 17:15:24,485] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,486] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:24,486] jepsen worker 80 - jepsen.util 710 :info :write [26 2] :RequestTimeoutException INFO [2022-03-13 17:15:24,487] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:24,496] jepsen worker 78 - jepsen.util 528 :info :write [26 40] :RequestTimeoutException INFO [2022-03-13 17:15:24,496] jepsen worker 78 - jepsen.util 618 :invoke :write [26 13] INFO [2022-03-13 17:15:24,497] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,497] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,503] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:15:24,503] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:24,506] jepsen worker 47 - jepsen.util 3017 :ok :read [15 48] INFO [2022-03-13 17:15:24,506] jepsen worker 47 - jepsen.util 3017 :invoke :read [15 nil] INFO [2022-03-13 17:15:24,520] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,521] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:15:24,525] jepsen worker 69 - jepsen.util 699 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,526] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,526] jepsen worker 69 - jepsen.util 699 :invoke :write [23 47] INFO [2022-03-13 17:15:24,527] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:24,533] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,534] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,534] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:24,535] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:24,544] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,545] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:24,561] jepsen worker 47 - jepsen.util 3017 :ok :read [15 48] INFO [2022-03-13 17:15:24,561] jepsen worker 47 - jepsen.util 3017 :invoke :read [15 nil] INFO [2022-03-13 17:15:24,585] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,585] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:24,594] jepsen worker 47 - jepsen.util 3017 :ok :read [15 48] INFO [2022-03-13 17:15:24,595] jepsen worker 47 - jepsen.util 3017 :invoke :read [15 nil] INFO [2022-03-13 17:15:24,606] jepsen worker 73 - jepsen.util 3313 :info :write [24 42] :RequestTimeoutException INFO [2022-03-13 17:15:24,607] jepsen worker 73 - jepsen.util 3403 :invoke :write [24 8] INFO [2022-03-13 17:15:24,615] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,616] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:15:24,617] jepsen worker 18 - jepsen.util 2988 :ok :read [6 16] INFO [2022-03-13 17:15:24,617] jepsen worker 18 - jepsen.util 2988 :invoke :write [6 19] INFO [2022-03-13 17:15:24,636] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,636] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:24,638] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,638] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,639] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:24,640] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:15:24,641] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,641] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:15:24,646] jepsen worker 28 - jepsen.util 3268 :ok :read [9 26] INFO [2022-03-13 17:15:24,648] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,648] jepsen worker 28 - jepsen.util 3268 :invoke :write [9 39] INFO [2022-03-13 17:15:24,649] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:15:24,654] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,655] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:24,659] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,659] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:15:24,741] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,760] jepsen worker 89 - jepsen.util 2519 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,760] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,761] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,762] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,763] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:24,764] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 32] INFO [2022-03-13 17:15:24,765] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:24,765] jepsen worker 47 - jepsen.util 3017 :ok :read [15 48] INFO [2022-03-13 17:15:24,766] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,767] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:24,768] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,768] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 26] INFO [2022-03-13 17:15:24,768] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,769] jepsen worker 47 - jepsen.util 3017 :invoke :read [15 nil] INFO [2022-03-13 17:15:24,769] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:15:24,769] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:15:24,770] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:15:24,789] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,789] jepsen worker 22 - jepsen.util 652 :invoke :write [7 0] INFO [2022-03-13 17:15:24,853] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,854] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:24,857] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,857] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:15:24,862] jepsen worker 47 - jepsen.util 3017 :ok :read [15 48] INFO [2022-03-13 17:15:24,863] jepsen worker 47 - jepsen.util 3017 :invoke :read [15 nil] INFO [2022-03-13 17:15:24,871] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,871] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:24,895] jepsen worker 24 - jepsen.util 564 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,896] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:24,901] jepsen worker 47 - jepsen.util 3017 :ok :read [15 48] INFO [2022-03-13 17:15:24,902] jepsen worker 47 - jepsen.util 3017 :invoke :read [15 nil] INFO [2022-03-13 17:15:24,917] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,917] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:15:24,945] jepsen worker 89 - jepsen.util 2519 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,945] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 46] INFO [2022-03-13 17:15:24,955] jepsen worker 47 - jepsen.util 3017 :ok :read [15 48] INFO [2022-03-13 17:15:24,956] jepsen worker 47 - jepsen.util 3017 :invoke :write [15 22] INFO [2022-03-13 17:15:24,981] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:24,982] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:15:25,000] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,002] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:15:25,071] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,071] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:15:25,076] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,076] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,078] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,078] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,085] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,085] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,086] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,087] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:15:25,112] jepsen worker 62 - jepsen.util 512 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,112] jepsen worker 62 - jepsen.util 512 :invoke :write [20 20] INFO [2022-03-13 17:15:25,116] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,117] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:25,130] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,130] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,131] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:15:25,131] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,171] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,171] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:25,179] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,180] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:25,216] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,217] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,217] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:25,218] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,218] jepsen worker 68 - jepsen.util 3308 :ok :read [22 30] INFO [2022-03-13 17:15:25,219] jepsen worker 68 - jepsen.util 3308 :invoke :write [22 16] INFO [2022-03-13 17:15:25,279] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,280] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:15:25,281] jepsen worker 16 - jepsen.util 2986 :ok :read [5 43] INFO [2022-03-13 17:15:25,282] jepsen worker 16 - jepsen.util 2986 :invoke :write [5 49] INFO [2022-03-13 17:15:25,283] jepsen worker 86 - jepsen.util 2966 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,284] jepsen worker 86 - jepsen.util 2966 :invoke :read [28 nil] INFO [2022-03-13 17:15:25,290] jepsen worker 37 - jepsen.util 2107 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,291] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:25,298] jepsen worker 0 - jepsen.util 2970 :ok :read [0 41] INFO [2022-03-13 17:15:25,298] jepsen worker 0 - jepsen.util 2970 :invoke :read [0 nil] INFO [2022-03-13 17:15:25,314] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,314] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,314] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,315] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,315] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 1] INFO [2022-03-13 17:15:25,318] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,319] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:25,319] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,324] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,324] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,324] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:15:25,325] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:25,326] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,327] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:15:25,330] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,330] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,330] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,330] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,331] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:25,331] jepsen worker 71 - jepsen.util 521 :invoke :write [23 42] INFO [2022-03-13 17:15:25,332] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:15:25,332] jepsen worker 80 - jepsen.util 800 :invoke :write [26 40] INFO [2022-03-13 17:15:25,338] jepsen worker 22 - jepsen.util 652 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,339] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:15:25,353] jepsen worker 86 - jepsen.util 2966 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,354] jepsen worker 86 - jepsen.util 2966 :invoke :read [28 nil] INFO [2022-03-13 17:15:25,359] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,360] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,362] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,369] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,372] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:25,372] jepsen worker nemesis - couchbase.util Rebalance check count: 434 INFO [2022-03-13 17:15:25,372] jepsen worker nemesis - couchbase.util Rebalance stuck count: 334 INFO [2022-03-13 17:15:25,392] jepsen worker 0 - jepsen.util 2970 :ok :read [0 41] INFO [2022-03-13 17:15:25,392] jepsen worker 0 - jepsen.util 2970 :invoke :write [0 1] INFO [2022-03-13 17:15:25,427] jepsen worker 86 - jepsen.util 2966 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,428] jepsen worker 86 - jepsen.util 2966 :invoke :read [28 nil] INFO [2022-03-13 17:15:25,494] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,495] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,498] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,498] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,499] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,500] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:25,504] jepsen worker 89 - jepsen.util 2519 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,504] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,505] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:15:25,506] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 24] INFO [2022-03-13 17:15:25,521] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,522] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:25,522] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,522] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,523] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:15:25,524] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:15:25,531] jepsen worker 86 - jepsen.util 2966 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,532] jepsen worker 86 - jepsen.util 2966 :invoke :write [28 46] INFO [2022-03-13 17:15:25,541] jepsen worker 88 - jepsen.util 2068 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,542] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 49] INFO [2022-03-13 17:15:25,551] jepsen worker 61 - jepsen.util 691 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,552] jepsen worker 61 - jepsen.util 691 :invoke :write [20 47] INFO [2022-03-13 17:15:25,564] jepsen worker 33 - jepsen.util 663 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:25,565] jepsen worker 33 - jepsen.util 663 :invoke :write [11 31] INFO [2022-03-13 17:15:25,600] jepsen worker 57 - jepsen.util 3117 :info :write [19 48] :RequestTimeoutException INFO [2022-03-13 17:15:25,601] jepsen worker 57 - jepsen.util 3207 :invoke :write [19 22] INFO [2022-03-13 17:15:25,616] jepsen worker 63 - jepsen.util 3123 :info :write [21 8] :RequestTimeoutException INFO [2022-03-13 17:15:25,617] jepsen worker 63 - jepsen.util 3213 :invoke :write [21 48] INFO [2022-03-13 17:15:25,754] jepsen worker 41 - jepsen.util 3101 :info :write [13 29] :RequestTimeoutException INFO [2022-03-13 17:15:25,754] jepsen worker 41 - jepsen.util 3191 :invoke :write [13 16] INFO [2022-03-13 17:15:26,025] jepsen worker 19 - jepsen.util 2809 :info :write [6 13] :RequestTimeoutException INFO [2022-03-13 17:15:26,027] jepsen worker 19 - jepsen.util 2899 :invoke :read [6 nil] INFO [2022-03-13 17:15:26,107] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,107] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,108] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:15:26,109] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:26,117] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,117] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:15:26,129] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,130] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:15:26,160] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,160] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:15:26,178] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,179] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,186] jepsen worker 85 - jepsen.util 2875 :info :write [28 47] :RequestTimeoutException INFO [2022-03-13 17:15:26,187] jepsen worker 85 - jepsen.util 2965 :invoke :read [28 nil] INFO [2022-03-13 17:15:26,190] jepsen worker 88 - jepsen.util 2068 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,191] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,222] jepsen worker 89 - jepsen.util 2519 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,223] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,225] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,225] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,232] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,232] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,232] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,233] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:15:26,237] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,238] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 23] INFO [2022-03-13 17:15:26,260] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,260] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,261] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,262] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,262] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,263] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:15:26,267] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,267] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,268] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,268] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:15:26,269] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,269] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:15:26,290] jepsen worker 32 - jepsen.util 3092 :info :write [10 39] :RequestTimeoutException INFO [2022-03-13 17:15:26,291] jepsen worker 32 - jepsen.util 3182 :invoke :write [10 4] INFO [2022-03-13 17:15:26,326] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,327] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:26,328] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,329] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,343] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,344] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,348] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,348] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,349] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,350] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,351] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,352] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,360] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,360] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:15:26,377] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:26,377] jepsen worker nemesis - couchbase.util Rebalance check count: 435 INFO [2022-03-13 17:15:26,377] jepsen worker nemesis - couchbase.util Rebalance stuck count: 335 INFO [2022-03-13 17:15:26,407] jepsen worker 81 - jepsen.util 3591 :ok :read [27 4] INFO [2022-03-13 17:15:26,408] jepsen worker 81 - jepsen.util 3591 :invoke :write [27 23] INFO [2022-03-13 17:15:26,514] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,514] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,515] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:26,515] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,516] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,516] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,517] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,517] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,517] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 44] INFO [2022-03-13 17:15:26,518] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:26,518] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,519] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,519] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,519] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:15:26,530] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,531] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] WARN [2022-03-13 17:15:26,547] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15308924,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15308924,"total_server_duration_us":43,"operation_id":"0xc364","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":15311310},{"operation_name":"upsert","last_dispatch_duration_us":9066425,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9524506,"total_server_duration_us":9279424,"encode_duration_us":19292,"operation_id":"0xc649","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008618},{"operation_name":"upsert","last_dispatch_duration_us":36579,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":612642,"total_server_duration_us":322305,"encode_duration_us":18555,"operation_id":"0xc5f3","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10004512},{"operation_name":"upsert","last_dispatch_duration_us":209814,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":3591730,"total_server_duration_us":131220,"encode_duration_us":22174,"operation_id":"0xc9f2","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10001095},{"operation_name":"get","last_dispatch_duration_us":8149061,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8149061,"total_server_duration_us":56,"operation_id":"0xc737","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8151264},{"operation_name":"upsert","last_dispatch_duration_us":5844250,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":5844250,"total_server_duration_us":88246,"encode_duration_us":17600,"operation_id":"0xcaec","timeout_ms":10000,"last_server_duration_us":88246,"total_duration_us":5853789},{"operation_name":"upsert","last_dispatch_duration_us":5656875,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":5656875,"total_server_duration_us":5518715,"encode_duration_us":20175,"operation_id":"0xcada","timeout_ms":10000,"last_server_duration_us":5518715,"total_duration_us":5663900},{"operation_name":"get","last_dispatch_duration_us":4967807,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":4967807,"total_server_duration_us":43,"operation_id":"0xc89f","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":4968153},{"operation_name":"upsert","last_dispatch_duration_us":4039832,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":4039832,"total_server_duration_us":38795,"encode_duration_us":31176,"operation_id":"0xcacc","timeout_ms":10000,"last_server_duration_us":38795,"total_duration_us":4063172},{"operation_name":"upsert","last_dispatch_duration_us":3993556,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3993556,"total_server_duration_us":150873,"encode_duration_us":31262,"operation_id":"0xcac9","timeout_ms":10000,"last_server_duration_us":150873,"total_duration_us":4010901}],"total_count":35}} WARN [2022-03-13 17:15:26,548] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17675262,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18157989,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9342063,"encode_duration_us":17794,"operation_id":"0xc08d","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10003433},{"operation_name":"upsert","last_dispatch_duration_us":9066425,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9524506,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9279424,"encode_duration_us":19292,"operation_id":"0xc649","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008618}],"total_count":2}} INFO [2022-03-13 17:15:26,582] jepsen worker 3 - jepsen.util 1983 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,582] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,721] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,721] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,721] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,722] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:26,722] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,723] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,724] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,724] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,724] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:15:26,726] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,728] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,728] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:15:26,730] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,731] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:26,772] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,772] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:15:26,774] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,774] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,774] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:15:26,775] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:15:26,923] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,923] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,925] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,926] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:15:26,961] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,962] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:26,963] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,964] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:15:26,997] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:26,998] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:15:27,129] jepsen worker 89 - jepsen.util 2519 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,129] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 41] INFO [2022-03-13 17:15:27,137] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,137] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:15:27,152] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,152] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,155] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,155] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,156] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,157] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,226] jepsen worker 66 - jepsen.util 2946 :info :write [22 21] :RequestTimeoutException INFO [2022-03-13 17:15:27,226] jepsen worker 66 - jepsen.util 3036 :invoke :write [22 17] INFO [2022-03-13 17:15:27,331] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,332] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,340] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,340] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,340] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:15:27,341] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 18] INFO [2022-03-13 17:15:27,375] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,376] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:15:27,383] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:27,383] jepsen worker nemesis - couchbase.util Rebalance check count: 436 INFO [2022-03-13 17:15:27,383] jepsen worker nemesis - couchbase.util Rebalance stuck count: 336 INFO [2022-03-13 17:15:27,535] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,535] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,536] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:15:27,536] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:15:27,539] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,539] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:27,580] jepsen worker 89 - jepsen.util 2519 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,581] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 1] INFO [2022-03-13 17:15:27,739] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,740] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:27,751] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,751] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,818] jepsen worker 51 - jepsen.util 591 :info :write [17 34] :RequestTimeoutException INFO [2022-03-13 17:15:27,818] jepsen worker 51 - jepsen.util 681 :invoke :read [17 nil] INFO [2022-03-13 17:15:27,943] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,943] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] WARN [2022-03-13 17:15:27,945] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13714201,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13714201,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9032060,"encode_duration_us":16664,"operation_id":"0xc73f","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10006231},{"operation_name":"upsert","last_dispatch_duration_us":14083341,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14083341,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9035244,"encode_duration_us":21382,"operation_id":"0xc669","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006507}],"total_count":2}} INFO [2022-03-13 17:15:27,946] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,947] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:15:27,950] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,951] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,951] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,951] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:27,952] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,953] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,953] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:27,953] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,954] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,954] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:15:27,955] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,955] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,954] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,956] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:27,956] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,957] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,957] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,958] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:27,958] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,959] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,959] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:15:27,959] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:15:27,962] jepsen worker 37 - jepsen.util 2107 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,962] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,963] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:15:27,963] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 37] INFO [2022-03-13 17:15:27,966] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,967] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:27,968] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,969] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:27,971] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,972] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,972] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:27,973] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:27,973] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,974] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:27,975] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:27,976] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:15:28,045] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15106848,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15106848,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9049045,"encode_duration_us":18218,"operation_id":"0xc3ac","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10006468}],"total_count":1}} INFO [2022-03-13 17:15:28,107] jepsen worker 11 - jepsen.util 3161 :info :write [3 47] :SyncWriteAmbiguous INFO [2022-03-13 17:15:28,108] jepsen worker 11 - jepsen.util 3251 :invoke :read [3 nil] WARN [2022-03-13 17:15:28,145] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":14083341,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":14083341,"total_server_duration_us":9035244,"encode_duration_us":21382,"operation_id":"0xc669","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006507},{"operation_name":"upsert","last_dispatch_duration_us":13714201,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13714201,"total_server_duration_us":9032060,"encode_duration_us":16664,"operation_id":"0xc73f","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10006231},{"operation_name":"upsert","last_dispatch_duration_us":9073780,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9073780,"total_server_duration_us":20521,"encode_duration_us":26863,"operation_id":"0xc8e8","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10004066},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":61571,"operation_id":"0xc8b9","timeout_ms":10000,"total_duration_us":10002893},{"operation_name":"upsert","last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","encode_duration_us":23804,"operation_id":"0xca80","timeout_ms":10000,"total_duration_us":10001042},{"operation_name":"get","last_dispatch_duration_us":9103583,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9103583,"total_server_duration_us":24069,"operation_id":"0xc8ea","timeout_ms":30000,"last_server_duration_us":24069,"total_duration_us":9111011},{"operation_name":"get","last_dispatch_duration_us":9100349,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9100349,"total_server_duration_us":10036,"operation_id":"0xc8e1","timeout_ms":30000,"last_server_duration_us":10036,"total_duration_us":9102369},{"operation_name":"get","last_dispatch_duration_us":9099762,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9099762,"total_server_duration_us":69,"operation_id":"0xc8eb","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":9101677},{"operation_name":"get","last_dispatch_duration_us":8236203,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8236203,"total_server_duration_us":8946,"operation_id":"0xc9ff","timeout_ms":30000,"last_server_duration_us":8946,"total_duration_us":8238275},{"operation_name":"upsert","last_dispatch_duration_us":2058356,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2058356,"total_server_duration_us":376530,"encode_duration_us":20885,"operation_id":"0xca7f","timeout_ms":10000,"last_server_duration_us":376530,"total_duration_us":7736727}],"total_count":26}} INFO [2022-03-13 17:15:28,151] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,151] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:15:28,152] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,153] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:28,154] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,155] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:15:28,181] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,182] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 21] INFO [2022-03-13 17:15:28,210] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,211] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:28,214] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,214] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:28,224] jepsen worker 56 - jepsen.util 3296 :ok :read [18 21] INFO [2022-03-13 17:15:28,225] jepsen worker 56 - jepsen.util 3296 :invoke :read [18 nil] INFO [2022-03-13 17:15:28,231] jepsen worker 45 - jepsen.util 3285 :ok :read [15 48] INFO [2022-03-13 17:15:28,231] jepsen worker 9 - jepsen.util 2709 :ok :read [3 25] INFO [2022-03-13 17:15:28,231] jepsen worker 45 - jepsen.util 3285 :invoke :read [15 nil] INFO [2022-03-13 17:15:28,231] jepsen worker 9 - jepsen.util 2709 :invoke :read [3 nil] INFO [2022-03-13 17:15:28,233] jepsen worker 10 - jepsen.util 3070 :ok :read [3 25] INFO [2022-03-13 17:15:28,233] jepsen worker 10 - jepsen.util 3070 :invoke :write [3 49] WARN [2022-03-13 17:15:28,245] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7647067,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7647067,"total_server_duration_us":21895,"encode_duration_us":17556,"operation_id":"0xc9a6","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10010406},{"operation_name":"upsert","last_dispatch_duration_us":55138,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":438970,"total_server_duration_us":262657,"encode_duration_us":22803,"operation_id":"0xc8ca","timeout_ms":10000,"last_server_duration_us":29508,"total_duration_us":10009755},{"operation_name":"upsert","last_dispatch_duration_us":7296209,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8016463,"total_server_duration_us":247931,"encode_duration_us":26461,"operation_id":"0xc8cb","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10008423},{"operation_name":"upsert","last_dispatch_duration_us":8459636,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11052038,"total_server_duration_us":320957,"encode_duration_us":17992,"operation_id":"0xc740","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10005340},{"operation_name":"upsert","last_dispatch_duration_us":9052106,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13905800,"total_server_duration_us":9294948,"encode_duration_us":19491,"operation_id":"0xc60d","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10003088},{"operation_name":"upsert","last_dispatch_duration_us":36506,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5470444,"total_server_duration_us":250589,"encode_duration_us":17235,"operation_id":"0xc66b","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10001796},{"operation_name":"upsert","last_dispatch_duration_us":9101033,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9101033,"total_server_duration_us":9042674,"encode_duration_us":20084,"operation_id":"0xc8c5","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9115259},{"operation_name":"upsert","last_dispatch_duration_us":8773308,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":8773308,"total_server_duration_us":37963,"encode_duration_us":20030,"operation_id":"0xc9b9","timeout_ms":10000,"last_server_duration_us":37963,"total_duration_us":8776713},{"operation_name":"get","last_dispatch_duration_us":7660187,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7660187,"total_server_duration_us":10095,"operation_id":"0xca05","timeout_ms":30000,"last_server_duration_us":10095,"total_duration_us":7662136},{"operation_name":"upsert","last_dispatch_duration_us":5668478,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":5668478,"total_server_duration_us":39424,"encode_duration_us":21903,"operation_id":"0xcae9","timeout_ms":10000,"last_server_duration_us":39424,"total_duration_us":5677278}],"total_count":22}} WARN [2022-03-13 17:15:28,245] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9094551,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9094551,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9064980,"encode_duration_us":22724,"operation_id":"0xc8f1","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10002663},{"operation_name":"upsert","last_dispatch_duration_us":18064664,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27402248,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9137259,"encode_duration_us":26833,"operation_id":"0xc076","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10003711}],"total_count":2}} WARN [2022-03-13 17:15:28,245] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8453176,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17094725,"total_server_duration_us":220651,"encode_duration_us":18320,"operation_id":"0xc620","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10010217},{"operation_name":"upsert","last_dispatch_duration_us":31060,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":3614317,"total_server_duration_us":392565,"encode_duration_us":19215,"operation_id":"0xc9d3","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10008625},{"operation_name":"upsert","last_dispatch_duration_us":8387093,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8606268,"total_server_duration_us":61434,"encode_duration_us":17957,"operation_id":"0xc84d","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10008516},{"operation_name":"upsert","last_dispatch_duration_us":206175,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9606013,"total_server_duration_us":204347,"encode_duration_us":19298,"operation_id":"0xca4f","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10002925},{"operation_name":"upsert","last_dispatch_duration_us":9094551,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9094551,"total_server_duration_us":9064980,"encode_duration_us":22724,"operation_id":"0xc8f1","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10002663},{"operation_name":"upsert","last_dispatch_duration_us":9035930,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9035930,"total_server_duration_us":9023573,"encode_duration_us":19150,"operation_id":"0xc6d6","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9039425},{"operation_name":"get","last_dispatch_duration_us":7429252,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7429252,"total_server_duration_us":13355,"operation_id":"0xcaf0","timeout_ms":30000,"last_server_duration_us":13355,"total_duration_us":7431065},{"operation_name":"upsert","last_dispatch_duration_us":5573353,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":5573353,"total_server_duration_us":35916,"encode_duration_us":19184,"operation_id":"0xcae7","timeout_ms":10000,"last_server_duration_us":35916,"total_duration_us":5586056},{"operation_name":"upsert","last_dispatch_duration_us":4018383,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":4018383,"total_server_duration_us":3401737,"encode_duration_us":16918,"operation_id":"0xca81","timeout_ms":10000,"last_server_duration_us":3401737,"total_duration_us":4021746},{"operation_name":"get","last_dispatch_duration_us":3814728,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":3814728,"total_server_duration_us":32,"operation_id":"0xca82","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3815635}],"total_count":26}} INFO [2022-03-13 17:15:28,246] jepsen worker 72 - jepsen.util 3222 :ok :read [24 24] INFO [2022-03-13 17:15:28,247] jepsen worker 72 - jepsen.util 3222 :invoke :read [24 nil] WARN [2022-03-13 17:15:28,253] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16874971,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":16874971,"total_server_duration_us":43,"operation_id":"0xc5f1","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":16877180},{"operation_name":"upsert","last_dispatch_duration_us":11027568,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":11027568,"total_server_duration_us":169585,"encode_duration_us":34186,"operation_id":"0xc8fe","timeout_ms":10000,"last_server_duration_us":169585,"total_duration_us":10008467},{"operation_name":"upsert","last_dispatch_duration_us":8564232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8600855,"total_server_duration_us":43853,"encode_duration_us":21104,"operation_id":"0xc631","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10007412},{"operation_name":"upsert","last_dispatch_duration_us":8572967,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8710719,"total_server_duration_us":47544,"encode_duration_us":17849,"operation_id":"0xc922","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10007311},{"operation_name":"upsert","last_dispatch_duration_us":12681477,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":12681477,"total_server_duration_us":126463,"encode_duration_us":21085,"operation_id":"0xc682","timeout_ms":10000,"last_server_duration_us":126463,"total_duration_us":10005880},{"operation_name":"upsert","last_dispatch_duration_us":10658318,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10658318,"total_server_duration_us":197078,"encode_duration_us":26957,"operation_id":"0xc8fc","timeout_ms":10000,"last_server_duration_us":197078,"total_duration_us":10003310},{"operation_name":"upsert","last_dispatch_duration_us":265452,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10064824,"total_server_duration_us":190139,"encode_duration_us":17220,"operation_id":"0xc9de","timeout_ms":10000,"last_server_duration_us":35111,"total_duration_us":10002779},{"operation_name":"upsert","last_dispatch_duration_us":8378232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8413279,"total_server_duration_us":47118,"encode_duration_us":17237,"operation_id":"0xc6e5","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10001689},{"operation_name":"upsert","last_dispatch_duration_us":10449605,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10449605,"total_server_duration_us":226308,"encode_duration_us":21203,"operation_id":"0xc8f5","timeout_ms":10000,"last_server_duration_us":226308,"total_duration_us":10001115},{"operation_name":"upsert","last_dispatch_duration_us":63890,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8980822,"total_server_duration_us":122302,"encode_duration_us":20948,"operation_id":"0xc9a9","timeout_ms":10000,"last_server_duration_us":31298,"total_duration_us":10000780}],"total_count":17}} INFO [2022-03-13 17:15:28,253] jepsen worker 56 - jepsen.util 3296 :ok :read [18 21] INFO [2022-03-13 17:15:28,254] jepsen worker 56 - jepsen.util 3296 :invoke :write [18 35] INFO [2022-03-13 17:15:28,264] jepsen worker 45 - jepsen.util 3285 :ok :read [15 48] INFO [2022-03-13 17:15:28,264] jepsen worker 45 - jepsen.util 3285 :invoke :read [15 nil] INFO [2022-03-13 17:15:28,270] jepsen worker 9 - jepsen.util 2709 :ok :read [3 25] INFO [2022-03-13 17:15:28,271] jepsen worker 9 - jepsen.util 2709 :invoke :read [3 nil] INFO [2022-03-13 17:15:28,279] jepsen worker 72 - jepsen.util 3222 :ok :read [24 24] INFO [2022-03-13 17:15:28,280] jepsen worker 72 - jepsen.util 3222 :invoke :write [24 34] INFO [2022-03-13 17:15:28,327] jepsen worker 45 - jepsen.util 3285 :ok :read [15 48] INFO [2022-03-13 17:15:28,328] jepsen worker 45 - jepsen.util 3285 :invoke :write [15 45] INFO [2022-03-13 17:15:28,333] jepsen worker 9 - jepsen.util 2709 :ok :read [3 25] INFO [2022-03-13 17:15:28,334] jepsen worker 9 - jepsen.util 2709 :invoke :read [3 nil] WARN [2022-03-13 17:15:28,345] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9052106,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":13905800,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9294948,"encode_duration_us":19491,"operation_id":"0xc60d","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10003088}],"total_count":1}} WARN [2022-03-13 17:15:28,346] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10527090,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10527090,"total_server_duration_us":56,"operation_id":"0xc615","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":10529014},{"operation_name":"upsert","last_dispatch_duration_us":14178889,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":14178889,"total_server_duration_us":22471,"encode_duration_us":37957,"operation_id":"0xc8b3","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10008462},{"operation_name":"upsert","last_dispatch_duration_us":76987,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8256337,"total_server_duration_us":304714,"encode_duration_us":20309,"operation_id":"0xc705","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10006196},{"operation_name":"upsert","last_dispatch_duration_us":30838,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8498816,"total_server_duration_us":243496,"encode_duration_us":20622,"operation_id":"0xc7df","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10006087},{"operation_name":"upsert","last_dispatch_duration_us":18175069,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":18175069,"total_server_duration_us":9057542,"encode_duration_us":19724,"operation_id":"0xc6ea","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10004214},{"operation_name":"upsert","last_dispatch_duration_us":52445,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":4836809,"total_server_duration_us":326227,"encode_duration_us":17956,"operation_id":"0xc9b4","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10004174},{"operation_name":"get","last_dispatch_duration_us":9544850,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9544850,"total_server_duration_us":49,"operation_id":"0xc693","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9547373},{"operation_name":"get","last_dispatch_duration_us":9139896,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9139896,"total_server_duration_us":9513,"operation_id":"0xc6f0","timeout_ms":30000,"last_server_duration_us":9513,"total_duration_us":9141954},{"operation_name":"upsert","last_dispatch_duration_us":9064131,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9064131,"total_server_duration_us":9052231,"encode_duration_us":32339,"operation_id":"0xc725","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9070205},{"operation_name":"upsert","last_dispatch_duration_us":6044699,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6044699,"total_server_duration_us":3357032,"encode_duration_us":17963,"operation_id":"0xc9d1","timeout_ms":10000,"last_server_duration_us":3357032,"total_duration_us":6048378}],"total_count":44}} WARN [2022-03-13 17:15:28,354] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10449605,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10449605,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":226308,"encode_duration_us":21203,"operation_id":"0xc8f5","timeout_ms":10000,"last_server_duration_us":226308,"total_duration_us":10001115},{"operation_name":"upsert","last_dispatch_duration_us":17538084,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17836841,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9083140,"encode_duration_us":27710,"operation_id":"0xc4c7","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10003541},{"operation_name":"upsert","last_dispatch_duration_us":10658318,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10658318,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":197078,"encode_duration_us":26957,"operation_id":"0xc8fc","timeout_ms":10000,"last_server_duration_us":197078,"total_duration_us":10003310},{"operation_name":"upsert","last_dispatch_duration_us":11027568,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11027568,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":169585,"encode_duration_us":34186,"operation_id":"0xc8fe","timeout_ms":10000,"last_server_duration_us":169585,"total_duration_us":10008467},{"operation_name":"upsert","last_dispatch_duration_us":17510456,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18057600,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9170009,"encode_duration_us":20888,"operation_id":"0xc4a3","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10007102},{"operation_name":"upsert","last_dispatch_duration_us":12681477,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12681477,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":126463,"encode_duration_us":21085,"operation_id":"0xc682","timeout_ms":10000,"last_server_duration_us":126463,"total_duration_us":10005880},{"operation_name":"upsert","last_dispatch_duration_us":18023864,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18023864,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9036305,"encode_duration_us":22478,"operation_id":"0xc545","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10010713},{"operation_name":"upsert","last_dispatch_duration_us":17538747,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17992890,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9103789,"encode_duration_us":18683,"operation_id":"0xc4c2","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10009291}],"total_count":8}} INFO [2022-03-13 17:15:28,355] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,355] jepsen worker 89 - jepsen.util 2519 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,356] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:28,356] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 34] INFO [2022-03-13 17:15:28,389] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:28,389] jepsen worker nemesis - couchbase.util Rebalance check count: 437 INFO [2022-03-13 17:15:28,389] jepsen worker nemesis - couchbase.util Rebalance stuck count: 337 INFO [2022-03-13 17:15:28,431] jepsen worker 9 - jepsen.util 2709 :ok :read [3 25] INFO [2022-03-13 17:15:28,432] jepsen worker 9 - jepsen.util 2709 :invoke :read [3 nil] INFO [2022-03-13 17:15:28,492] jepsen worker 19 - jepsen.util 2899 :ok :read [6 16] INFO [2022-03-13 17:15:28,492] jepsen worker 19 - jepsen.util 2899 :invoke :read [6 nil] INFO [2022-03-13 17:15:28,652] jepsen worker 9 - jepsen.util 2709 :ok :read [3 25] INFO [2022-03-13 17:15:28,681] jepsen worker 64 - jepsen.util 3124 :ok :read [21 :nil] INFO [2022-03-13 17:15:28,774] jepsen worker 2 - jepsen.util 3692 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:28,778] jepsen worker 83 - jepsen.util 2603 :ok :read [27 4] INFO [2022-03-13 17:15:29,600] jepsen worker 9 - jepsen.util 2709 :invoke :write [3 23] INFO [2022-03-13 17:15:29,601] jepsen worker 64 - jepsen.util 3124 :invoke :write [21 19] INFO [2022-03-13 17:15:29,601] jepsen worker 2 - jepsen.util 3692 :invoke :read [0 nil] INFO [2022-03-13 17:15:29,601] jepsen worker 83 - jepsen.util 2603 :invoke :read [27 nil] INFO [2022-03-13 17:15:28,931] jepsen worker 30 - jepsen.util 2910 :info :write [10 25] :RequestTimeoutException INFO [2022-03-13 17:15:28,940] jepsen worker 31 - jepsen.util 3271 :info :write [10 29] :RequestTimeoutException INFO [2022-03-13 17:15:29,602] jepsen worker 30 - jepsen.util 3000 :invoke :read [10 nil] INFO [2022-03-13 17:15:29,602] jepsen worker 31 - jepsen.util 3361 :invoke :write [10 42] INFO [2022-03-13 17:15:28,996] jepsen worker 29 - jepsen.util 3269 :info :write [9 45] :RequestTimeoutException INFO [2022-03-13 17:15:29,602] jepsen worker 29 - jepsen.util 3359 :invoke :write [9 11] INFO [2022-03-13 17:15:29,000] jepsen worker 55 - jepsen.util 3025 :info :write [18 1] :RequestTimeoutException INFO [2022-03-13 17:15:29,031] jepsen worker 27 - jepsen.util 3537 :info :write [9 6] :RequestTimeoutException INFO [2022-03-13 17:15:29,603] jepsen worker 55 - jepsen.util 3115 :invoke :write [18 40] INFO [2022-03-13 17:15:29,603] jepsen worker 27 - jepsen.util 3627 :invoke :read [9 nil] INFO [2022-03-13 17:15:29,394] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:29,603] jepsen worker nemesis - couchbase.util Rebalance check count: 438 INFO [2022-03-13 17:15:29,603] jepsen worker nemesis - couchbase.util Rebalance stuck count: 338 INFO [2022-03-13 17:15:29,605] jepsen worker 30 - jepsen.util 3000 :ok :read [10 :nil] INFO [2022-03-13 17:15:29,606] jepsen worker 30 - jepsen.util 3000 :invoke :write [10 14] INFO [2022-03-13 17:15:29,610] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,611] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:15:29,645] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,646] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:15:29,649] jepsen worker 19 - jepsen.util 2899 :ok :read [6 16] INFO [2022-03-13 17:15:29,649] jepsen worker 19 - jepsen.util 2899 :invoke :read [6 nil] INFO [2022-03-13 17:15:29,670] jepsen worker 59 - jepsen.util 3119 :info :write [19 14] :RequestTimeoutException INFO [2022-03-13 17:15:29,671] jepsen worker 59 - jepsen.util 3209 :invoke :read [19 nil] INFO [2022-03-13 17:15:29,690] jepsen worker 37 - jepsen.util 2107 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,691] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 1] INFO [2022-03-13 17:15:29,712] jepsen worker 3 - jepsen.util 1983 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,713] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:15:29,721] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,721] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,722] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:29,722] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:15:29,743] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,744] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:29,747] jepsen worker 19 - jepsen.util 2899 :ok :read [6 16] INFO [2022-03-13 17:15:29,748] jepsen worker 19 - jepsen.util 2899 :invoke :write [6 10] INFO [2022-03-13 17:15:29,780] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,781] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:29,791] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,791] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:29,792] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,793] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:15:29,803] jepsen worker 48 - jepsen.util 3018 :info :write [16 13] :RequestTimeoutException INFO [2022-03-13 17:15:29,805] jepsen worker 48 - jepsen.util 3108 :invoke :write [16 27] INFO [2022-03-13 17:15:29,811] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,811] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,812] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:15:29,813] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:29,833] jepsen worker 2 - jepsen.util 3692 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,834] jepsen worker 2 - jepsen.util 3692 :invoke :write [0 23] INFO [2022-03-13 17:15:29,835] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,835] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:29,844] jepsen worker 83 - jepsen.util 2603 :ok :read [27 4] INFO [2022-03-13 17:15:29,846] jepsen worker 83 - jepsen.util 2603 :invoke :write [27 28] INFO [2022-03-13 17:15:29,870] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,870] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,870] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:15:29,871] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:15:29,885] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:29,886] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:29,986] jepsen worker 46 - jepsen.util 3106 :info :write [15 37] :RequestTimeoutException INFO [2022-03-13 17:15:29,987] jepsen worker 46 - jepsen.util 3196 :invoke :write [15 18] INFO [2022-03-13 17:15:30,045] jepsen worker 89 - jepsen.util 2519 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,046] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:30,047] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,047] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:30,051] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,052] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:30,054] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,054] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,055] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 48] INFO [2022-03-13 17:15:30,056] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:30,057] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,058] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 0] INFO [2022-03-13 17:15:30,098] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,098] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:15:30,247] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,247] jepsen worker 37 - jepsen.util 2107 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,248] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 29] INFO [2022-03-13 17:15:30,248] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:30,250] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,251] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:30,252] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,252] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:30,254] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,254] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:15:30,267] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,267] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:15:30,283] jepsen worker 89 - jepsen.util 2519 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,284] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 25] INFO [2022-03-13 17:15:30,471] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,471] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:30,472] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,473] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:15:30,480] jepsen worker 36 - jepsen.util 2196 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,480] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:15:30,608] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:30,608] jepsen worker nemesis - couchbase.util Rebalance check count: 439 INFO [2022-03-13 17:15:30,608] jepsen worker nemesis - couchbase.util Rebalance stuck count: 339 INFO [2022-03-13 17:15:30,663] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,663] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,663] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:15:30,664] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:15:30,690] jepsen worker 37 - jepsen.util 2107 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,691] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:30,867] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,868] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:15:30,869] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:30,869] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:15:30,908] jepsen worker 65 - jepsen.util 3575 :info :write [21 32] :RequestTimeoutException INFO [2022-03-13 17:15:30,909] jepsen worker 65 - jepsen.util 3665 :invoke :write [21 35] INFO [2022-03-13 17:15:31,007] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,007] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,007] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,008] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:31,008] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:31,009] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,049] jepsen worker 8 - jepsen.util 728 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,049] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:31,065] jepsen worker 80 - jepsen.util 800 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,066] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:31,096] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,096] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,096] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:15:31,097] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,100] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,100] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,101] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:15:31,102] jepsen worker 13 - jepsen.util 643 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,103] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,103] jepsen worker 13 - jepsen.util 643 :invoke :write [4 22] INFO [2022-03-13 17:15:31,110] jepsen worker 34 - jepsen.util 844 :invoke :write [11 42] INFO [2022-03-13 17:15:31,114] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:31,115] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,116] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:15:31,119] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,121] jepsen worker 52 - jepsen.util 592 :invoke :write [17 24] INFO [2022-03-13 17:15:31,144] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,145] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:31,224] jepsen worker 71 - jepsen.util 521 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,225] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:15:31,231] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,232] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,234] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,234] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,234] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,234] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,234] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:31,234] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,235] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,235] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,236] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,236] jepsen worker 14 - jepsen.util 374 :invoke :write [4 9] INFO [2022-03-13 17:15:31,236] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:15:31,237] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,238] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,239] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,240] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,240] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:15:31,248] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,249] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:31,250] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,250] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,251] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,252] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:31,253] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,253] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,254] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,254] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:31,255] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,256] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:15:31,266] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,266] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:31,277] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,277] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,277] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:15:31,278] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:31,279] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,280] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:31,281] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,282] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:31,283] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,284] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:15:31,289] jepsen worker 33 - jepsen.util 663 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,289] jepsen worker 51 - jepsen.util 681 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,290] jepsen worker 51 - jepsen.util 681 :invoke :read [17 nil] INFO [2022-03-13 17:15:31,291] jepsen worker 89 - jepsen.util 2519 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,291] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,291] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,291] jepsen worker 33 - jepsen.util 663 :invoke :write [11 43] INFO [2022-03-13 17:15:31,291] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,292] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,293] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 32] INFO [2022-03-13 17:15:31,298] jepsen worker 7 - jepsen.util 637 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,299] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:15:31,299] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:31,300] jepsen worker 7 - jepsen.util 637 :invoke :write [2 32] INFO [2022-03-13 17:15:31,307] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,308] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:15:31,361] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,362] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:31,431] jepsen worker 14 - jepsen.util 374 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,432] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:31,443] jepsen worker 34 - jepsen.util 844 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,444] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,468] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,468] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,469] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:31,469] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:31,478] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,479] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:15:31,482] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,484] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:31,485] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,486] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:15:31,503] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,503] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,511] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,512] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:31,522] jepsen worker 77 - jepsen.util 707 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,523] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:15:31,523] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,523] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,523] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:31,524] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:31,524] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:31,524] jepsen worker 44 - jepsen.util 584 :invoke :write [14 38] INFO [2022-03-13 17:15:31,525] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,526] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:31,528] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,528] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:15:31,536] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,537] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:15:31,578] jepsen worker 26 - jepsen.util 476 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,579] jepsen worker 26 - jepsen.util 476 :invoke :write [8 11] INFO [2022-03-13 17:15:31,614] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:31,614] jepsen worker nemesis - couchbase.util Rebalance check count: 440 INFO [2022-03-13 17:15:31,614] jepsen worker nemesis - couchbase.util Rebalance stuck count: 340 INFO [2022-03-13 17:15:31,626] jepsen worker 13 - jepsen.util 643 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,626] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:31,631] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,632] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,632] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,632] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,632] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,632] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:31,633] jepsen worker 21 - jepsen.util 831 :invoke :write [7 28] INFO [2022-03-13 17:15:31,634] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,634] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:31,635] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,635] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,636] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:31,636] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,637] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,637] jepsen worker 80 - jepsen.util 800 :invoke :write [26 2] INFO [2022-03-13 17:15:31,638] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,639] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:15:31,639] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,641] jepsen worker 25 - jepsen.util 565 :ok :read [8 :nil] INFO [2022-03-13 17:15:31,642] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:15:31,661] jepsen worker 51 - jepsen.util 681 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,661] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,662] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,663] jepsen worker 51 - jepsen.util 681 :invoke :read [17 nil] INFO [2022-03-13 17:15:31,681] jepsen worker 35 - jepsen.util 935 :ok :read [11 7] INFO [2022-03-13 17:15:31,682] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:31,880] jepsen worker 49 - jepsen.util 3019 :info :write [16 48] :RequestTimeoutException INFO [2022-03-13 17:15:31,880] jepsen worker 49 - jepsen.util 3109 :invoke :write [16 36] INFO [2022-03-13 17:15:31,887] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,887] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,887] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,887] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:15:31,888] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:15:31,889] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:31,912] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:31,912] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,912] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:31,913] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:31,914] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:31,914] jepsen worker 53 - jepsen.util 503 :invoke :write [17 38] INFO [2022-03-13 17:15:31,975] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,976] jepsen worker 51 - jepsen.util 681 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:31,976] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:31,977] jepsen worker 51 - jepsen.util 681 :invoke :read [17 nil] INFO [2022-03-13 17:15:32,065] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,066] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:32,078] jepsen worker 34 - jepsen.util 844 :ok :read [11 7] INFO [2022-03-13 17:15:32,078] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:32,091] jepsen worker 89 - jepsen.util 2519 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,091] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,091] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,092] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,093] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,093] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:15:32,107] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,108] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:32,114] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,115] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:32,116] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,116] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:15:32,295] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,295] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,295] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,296] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,296] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,297] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,297] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,297] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,297] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,298] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:15:32,323] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,323] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,323] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,324] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,324] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,325] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,326] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,326] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,327] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,328] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,329] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,329] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,329] jepsen worker 52 - jepsen.util 592 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,329] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,330] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:32,331] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,331] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,332] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:15:32,333] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:32,333] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,348] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,349] jepsen worker 79 - jepsen.util 1069 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,349] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:15:32,353] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:32,476] jepsen worker 17 - jepsen.util 3167 :ok :read [5 43] INFO [2022-03-13 17:15:32,476] jepsen worker 17 - jepsen.util 3167 :invoke :read [5 nil] INFO [2022-03-13 17:15:32,487] jepsen worker 40 - jepsen.util 3370 :ok :read [13 39] INFO [2022-03-13 17:15:32,488] jepsen worker 40 - jepsen.util 3370 :invoke :read [13 nil] INFO [2022-03-13 17:15:32,499] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,499] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,520] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,520] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,520] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,521] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,522] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:15:32,523] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:32,523] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,523] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:15:32,532] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,533] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:15:32,540] jepsen worker 80 - jepsen.util 800 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,541] jepsen worker 80 - jepsen.util 800 :invoke :write [26 5] INFO [2022-03-13 17:15:32,553] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,554] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,557] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:32,558] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:32,567] jepsen worker 34 - jepsen.util 844 :ok :read [11 7] INFO [2022-03-13 17:15:32,568] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:32,620] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:32,620] jepsen worker nemesis - couchbase.util Rebalance check count: 441 INFO [2022-03-13 17:15:32,620] jepsen worker nemesis - couchbase.util Rebalance stuck count: 341 INFO [2022-03-13 17:15:32,704] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,705] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:15:32,711] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,711] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 36] INFO [2022-03-13 17:15:32,744] jepsen worker 61 - jepsen.util 691 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,744] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,745] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,745] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:32,746] jepsen worker 76 - jepsen.util 706 :invoke :write [25 19] INFO [2022-03-13 17:15:32,746] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:32,749] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,750] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,750] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,751] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,751] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,752] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:15:32,752] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,753] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,766] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,766] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,767] jepsen worker 52 - jepsen.util 592 :invoke :write [17 15] INFO [2022-03-13 17:15:32,768] jepsen worker 13 - jepsen.util 643 :invoke :write [4 1] INFO [2022-03-13 17:15:32,773] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,773] jepsen worker 62 - jepsen.util 512 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,773] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:32,774] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:32,821] jepsen worker 33 - jepsen.util 663 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,822] jepsen worker 34 - jepsen.util 844 :ok :read [11 7] INFO [2022-03-13 17:15:32,822] jepsen worker 33 - jepsen.util 663 :invoke :write [11 12] INFO [2022-03-13 17:15:32,823] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:32,915] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,915] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:15:32,917] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,917] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,917] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,917] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:32,918] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,918] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:32,922] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,922] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:32,924] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,925] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:15:32,926] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,926] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,927] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:15:32,927] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 0] INFO [2022-03-13 17:15:32,955] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,955] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:32,970] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:32,971] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:15:33,000] jepsen worker 35 - jepsen.util 935 :ok :read [11 7] INFO [2022-03-13 17:15:33,001] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:33,002] jepsen worker 52 - jepsen.util 592 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,003] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:33,119] jepsen worker 3 - jepsen.util 1983 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,119] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:15:33,121] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,121] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,122] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:33,123] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:33,123] jepsen worker 58 - jepsen.util 2938 :info :write [19 35] :RequestTimeoutException INFO [2022-03-13 17:15:33,124] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,124] jepsen worker 58 - jepsen.util 3028 :invoke :read [19 nil] INFO [2022-03-13 17:15:33,124] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,124] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 29] INFO [2022-03-13 17:15:33,125] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:33,127] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,127] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:15:33,164] jepsen worker 69 - jepsen.util 699 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,164] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,164] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,165] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,165] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:33,166] jepsen worker 70 - jepsen.util 790 :invoke :write [23 8] INFO [2022-03-13 17:15:33,167] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:15:33,167] jepsen worker 69 - jepsen.util 699 :invoke :write [23 31] INFO [2022-03-13 17:15:33,181] jepsen worker 13 - jepsen.util 643 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,182] jepsen worker 13 - jepsen.util 643 :invoke :write [4 47] INFO [2022-03-13 17:15:33,201] jepsen worker 20 - jepsen.util 3530 :ok :read [6 16] INFO [2022-03-13 17:15:33,202] jepsen worker 20 - jepsen.util 3530 :invoke :write [6 48] INFO [2022-03-13 17:15:33,203] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,204] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:15:33,217] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,218] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:33,236] jepsen worker 34 - jepsen.util 844 :ok :read [11 7] INFO [2022-03-13 17:15:33,236] jepsen worker 39 - jepsen.util 3189 :info :write [13 18] :RequestTimeoutException INFO [2022-03-13 17:15:33,236] jepsen worker 34 - jepsen.util 844 :invoke :write [11 35] INFO [2022-03-13 17:15:33,237] jepsen worker 39 - jepsen.util 3279 :invoke :write [13 27] INFO [2022-03-13 17:15:33,323] jepsen worker 89 - jepsen.util 2519 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,323] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:33,325] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,326] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:15:33,363] jepsen worker 21 - jepsen.util 831 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,364] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:15:33,365] jepsen worker 7 - jepsen.util 637 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,366] jepsen worker 7 - jepsen.util 637 :invoke :write [2 16] INFO [2022-03-13 17:15:33,412] jepsen worker 13 - jepsen.util 643 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,412] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,413] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:33,413] jepsen worker 13 - jepsen.util 643 :invoke :write [4 21] INFO [2022-03-13 17:15:33,505] jepsen worker 27 - jepsen.util 3627 :ok :read [9 26] INFO [2022-03-13 17:15:33,505] jepsen worker 27 - jepsen.util 3627 :invoke :write [9 30] INFO [2022-03-13 17:15:33,527] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,528] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:15:33,555] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,555] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:15:33,568] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,569] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:15:33,575] jepsen worker 44 - jepsen.util 584 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,576] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:33,624] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:33,624] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:33,624] jepsen worker 26 - jepsen.util 476 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,625] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:33,625] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:33,625] jepsen worker nemesis - couchbase.util Rebalance check count: 442 INFO [2022-03-13 17:15:33,625] jepsen worker nemesis - couchbase.util Rebalance stuck count: 342 INFO [2022-03-13 17:15:33,625] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:15:33,627] jepsen worker 26 - jepsen.util 476 :invoke :write [8 29] INFO [2022-03-13 17:15:33,732] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,732] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:15:33,739] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,739] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,740] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:33,740] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:33,746] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,746] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,747] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:15:33,747] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:15:33,788] jepsen worker 80 - jepsen.util 800 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,788] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,789] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,789] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:33,790] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:33,790] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:33,794] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,795] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:33,796] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,797] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 36] INFO [2022-03-13 17:15:33,835] jepsen worker 34 - jepsen.util 844 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,836] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,836] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,836] jepsen worker 52 - jepsen.util 592 :invoke :write [17 39] INFO [2022-03-13 17:15:33,837] jepsen worker 22 - jepsen.util 652 :invoke :write [7 3] INFO [2022-03-13 17:15:33,837] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:33,921] jepsen worker 28 - jepsen.util 3268 :info :write [9 39] :SyncWriteAmbiguous INFO [2022-03-13 17:15:33,922] jepsen worker 28 - jepsen.util 3358 :invoke :write [9 9] INFO [2022-03-13 17:15:33,935] jepsen worker 37 - jepsen.util 2107 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,935] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,936] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:33,937] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:15:33,938] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,938] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,939] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:15:33,939] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 45] INFO [2022-03-13 17:15:33,992] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,992] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:33,997] jepsen worker 60 - jepsen.util 600 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:33,998] jepsen worker 60 - jepsen.util 600 :invoke :write [20 31] INFO [2022-03-13 17:15:34,023] jepsen worker 6 - jepsen.util 636 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,024] jepsen worker 6 - jepsen.util 636 :invoke :write [2 21] INFO [2022-03-13 17:15:34,193] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,194] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:15:34,200] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,200] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,201] jepsen worker 13 - jepsen.util 643 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,201] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:34,202] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:34,202] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:34,203] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,203] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,203] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,203] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,204] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,205] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:15:34,205] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:34,206] jepsen worker 80 - jepsen.util 800 :invoke :write [26 37] INFO [2022-03-13 17:15:34,207] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:34,208] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:34,290] jepsen worker 54 - jepsen.util 2754 :info :write [18 49] :RequestTimeoutException INFO [2022-03-13 17:15:34,290] jepsen worker 54 - jepsen.util 2844 :invoke :write [18 12] INFO [2022-03-13 17:15:34,298] jepsen worker 74 - jepsen.util 3404 :info :write [24 41] :RequestTimeoutException INFO [2022-03-13 17:15:34,298] jepsen worker 15 - jepsen.util 3255 :info :write [5 26] :RequestTimeoutException INFO [2022-03-13 17:15:34,299] jepsen worker 15 - jepsen.util 3345 :invoke :read [5 nil] INFO [2022-03-13 17:15:34,299] jepsen worker 74 - jepsen.util 3494 :invoke :read [24 nil] INFO [2022-03-13 17:15:34,343] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,343] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,344] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:34,345] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:34,345] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,346] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,346] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:15:34,347] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:34,348] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,348] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,349] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:15:34,349] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:15:34,378] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,379] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:34,413] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,413] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,413] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,414] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:34,414] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:34,415] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:34,415] jepsen worker 78 - jepsen.util 618 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,416] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,416] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,416] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:34,416] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:34,417] jepsen worker 61 - jepsen.util 691 :invoke :write [20 25] INFO [2022-03-13 17:15:34,447] jepsen worker 84 - jepsen.util 3504 :info :write [28 12] :RequestTimeoutException INFO [2022-03-13 17:15:34,447] jepsen worker 84 - jepsen.util 3594 :invoke :write [28 40] INFO [2022-03-13 17:15:34,453] jepsen worker 82 - jepsen.util 3142 :info :write [27 31] :RequestTimeoutException INFO [2022-03-13 17:15:34,454] jepsen worker 82 - jepsen.util 3232 :invoke :read [27 nil] INFO [2022-03-13 17:15:34,546] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,546] jepsen worker 89 - jepsen.util 2519 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,547] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:34,547] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:15:34,550] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,550] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:34,611] jepsen worker 22 - jepsen.util 652 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,611] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,612] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,611] jepsen worker 52 - jepsen.util 592 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,613] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:15:34,613] jepsen worker 34 - jepsen.util 844 :invoke :write [11 40] INFO [2022-03-13 17:15:34,614] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:34,614] jepsen worker 13 - jepsen.util 643 :invoke :write [4 24] INFO [2022-03-13 17:15:34,615] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,615] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,616] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,616] jepsen worker 8 - jepsen.util 728 :invoke :write [2 2] INFO [2022-03-13 17:15:34,616] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:34,617] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:34,618] jepsen worker 53 - jepsen.util 503 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,618] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,618] jepsen worker 53 - jepsen.util 503 :invoke :write [17 7] INFO [2022-03-13 17:15:34,618] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 41] INFO [2022-03-13 17:15:34,630] jepsen worker 73 - jepsen.util 3403 :info :write [24 8] :RequestTimeoutException INFO [2022-03-13 17:15:34,631] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:34,631] jepsen worker nemesis - couchbase.util Rebalance check count: 443 INFO [2022-03-13 17:15:34,631] jepsen worker nemesis - couchbase.util Rebalance stuck count: 343 INFO [2022-03-13 17:15:34,632] jepsen worker 73 - jepsen.util 3493 :invoke :read [24 nil] INFO [2022-03-13 17:15:34,647] jepsen worker 18 - jepsen.util 2988 :info :write [6 19] :RequestTimeoutException INFO [2022-03-13 17:15:34,648] jepsen worker 18 - jepsen.util 3078 :invoke :write [6 46] INFO [2022-03-13 17:15:34,752] jepsen worker 3 - jepsen.util 1983 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,752] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:34,753] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,754] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:15:34,766] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,766] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:34,768] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,768] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:15:34,816] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,816] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:34,824] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,825] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:34,860] jepsen worker 76 - jepsen.util 706 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,860] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:34,860] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,861] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:15:34,958] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,958] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:15:34,986] jepsen worker 47 - jepsen.util 3017 :info :write [15 22] :RequestTimeoutException INFO [2022-03-13 17:15:34,987] jepsen worker 47 - jepsen.util 3107 :invoke :write [15 18] INFO [2022-03-13 17:15:34,998] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:34,998] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,000] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,001] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,002] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,003] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:15:35,009] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,009] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,009] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:35,010] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 47] INFO [2022-03-13 17:15:35,036] jepsen worker 80 - jepsen.util 800 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,036] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,036] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,036] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,036] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,036] jepsen worker 80 - jepsen.util 800 :invoke :write [26 19] INFO [2022-03-13 17:15:35,037] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:35,038] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:15:35,038] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:35,039] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:35,040] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,040] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:35,045] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,045] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,046] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,046] jepsen worker 35 - jepsen.util 935 :invoke :write [11 48] INFO [2022-03-13 17:15:35,047] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:35,047] jepsen worker 34 - jepsen.util 844 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,047] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:15:35,047] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,048] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:35,048] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,049] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:35,049] jepsen worker 21 - jepsen.util 831 :invoke :write [7 9] INFO [2022-03-13 17:15:35,104] jepsen worker 68 - jepsen.util 3308 :info :write [22 16] :SyncWriteAmbiguous INFO [2022-03-13 17:15:35,104] jepsen worker 68 - jepsen.util 3398 :invoke :read [22 nil] INFO [2022-03-13 17:15:35,136] jepsen worker 68 - jepsen.util 3398 :ok :read [22 30] INFO [2022-03-13 17:15:35,137] jepsen worker 68 - jepsen.util 3398 :invoke :write [22 38] INFO [2022-03-13 17:15:35,223] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,224] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:35,225] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,225] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:15:35,316] jepsen worker 16 - jepsen.util 2986 :info :write [5 49] :RequestTimeoutException INFO [2022-03-13 17:15:35,317] jepsen worker 16 - jepsen.util 3076 :invoke :write [5 16] INFO [2022-03-13 17:15:35,367] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,367] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,367] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,367] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,368] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,368] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:35,378] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,379] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:35,380] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,380] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:15:35,386] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,386] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:15:35,416] jepsen worker 0 - jepsen.util 2970 :info :write [0 1] :RequestTimeoutException INFO [2022-03-13 17:15:35,417] jepsen worker 0 - jepsen.util 3060 :invoke :write [0 38] INFO [2022-03-13 17:15:35,428] jepsen worker 7 - jepsen.util 637 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,428] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:35,429] jepsen worker 7 - jepsen.util 637 :invoke :write [2 33] INFO [2022-03-13 17:15:35,429] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:35,435] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,435] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,436] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,436] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:15:35,566] jepsen worker 86 - jepsen.util 2966 :info :write [28 46] :RequestTimeoutException INFO [2022-03-13 17:15:35,566] jepsen worker 86 - jepsen.util 3056 :invoke :read [28 nil] INFO [2022-03-13 17:15:35,582] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,582] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:15:35,624] jepsen worker 57 - jepsen.util 3207 :info :write [19 22] :RequestTimeoutException INFO [2022-03-13 17:15:35,624] jepsen worker 57 - jepsen.util 3297 :invoke :write [19 25] INFO [2022-03-13 17:15:35,632] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,632] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:35,635] jepsen worker 63 - jepsen.util 3213 :info :write [21 48] :RequestTimeoutException INFO [2022-03-13 17:15:35,636] jepsen worker 63 - jepsen.util 3303 :invoke :read [21 nil] INFO [2022-03-13 17:15:35,637] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:35,637] jepsen worker nemesis - couchbase.util Rebalance check count: 444 INFO [2022-03-13 17:15:35,637] jepsen worker nemesis - couchbase.util Rebalance stuck count: 344 INFO [2022-03-13 17:15:35,644] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,645] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:35,775] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,776] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,776] jepsen worker 41 - jepsen.util 3191 :info :write [13 16] :RequestTimeoutException INFO [2022-03-13 17:15:35,777] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,777] jepsen worker 41 - jepsen.util 3281 :invoke :write [13 15] INFO [2022-03-13 17:15:35,778] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,778] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,778] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,779] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:15:35,779] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:35,779] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,780] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:15:35,804] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,804] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:15:35,995] jepsen worker 37 - jepsen.util 2107 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,996] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:15:35,998] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:35,999] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:15:36,099] jepsen worker 13 - jepsen.util 643 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,099] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,099] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,100] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:36,100] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:36,101] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:36,187] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,187] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,187] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:15:36,188] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:36,212] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,212] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:36,214] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,215] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:15:36,244] jepsen worker 80 - jepsen.util 800 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,244] jepsen worker 69 - jepsen.util 699 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,245] jepsen worker 80 - jepsen.util 800 :invoke :write [26 37] INFO [2022-03-13 17:15:36,245] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:36,248] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,248] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,248] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,248] jepsen worker 13 - jepsen.util 643 :invoke :write [4 17] INFO [2022-03-13 17:15:36,249] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:36,250] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:36,320] jepsen worker 32 - jepsen.util 3182 :info :write [10 4] :RequestTimeoutException INFO [2022-03-13 17:15:36,320] jepsen worker 32 - jepsen.util 3272 :invoke :write [10 17] INFO [2022-03-13 17:15:36,436] jepsen worker 81 - jepsen.util 3591 :info :write [27 23] :RequestTimeoutException INFO [2022-03-13 17:15:36,437] jepsen worker 81 - jepsen.util 3681 :invoke :read [27 nil] INFO [2022-03-13 17:15:36,455] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,455] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,456] jepsen worker 52 - jepsen.util 592 :invoke :write [17 24] INFO [2022-03-13 17:15:36,457] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] WARN [2022-03-13 17:15:36,560] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":45448,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9693643,"total_server_duration_us":274234,"encode_duration_us":21712,"operation_id":"0xcaee","timeout_ms":10000,"last_server_duration_us":27854,"total_duration_us":10008728},{"operation_name":"upsert","last_dispatch_duration_us":193819,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":798565,"total_server_duration_us":134157,"encode_duration_us":18064,"operation_id":"0xcaed","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10006523},{"operation_name":"upsert","last_dispatch_duration_us":29263,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9803277,"total_server_duration_us":346679,"encode_duration_us":19492,"operation_id":"0xcaf3","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10005967},{"operation_name":"upsert","last_dispatch_duration_us":131987,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9759534,"total_server_duration_us":248734,"encode_duration_us":32304,"operation_id":"0xcaf2","timeout_ms":10000,"last_server_duration_us":26330,"total_duration_us":10001637},{"operation_name":"get","last_dispatch_duration_us":9468766,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9468766,"total_server_duration_us":28493,"operation_id":"0xcaf8","timeout_ms":30000,"last_server_duration_us":28493,"total_duration_us":9470745},{"operation_name":"upsert","last_dispatch_duration_us":9147493,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9147493,"total_server_duration_us":9047983,"encode_duration_us":28129,"operation_id":"0xcaef","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9151333},{"operation_name":"get","last_dispatch_duration_us":120810,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":120810,"total_server_duration_us":3647,"operation_id":"0xcb48","timeout_ms":30000,"last_server_duration_us":3647,"total_duration_us":7831886},{"operation_name":"upsert","last_dispatch_duration_us":6477161,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":6477161,"total_server_duration_us":5629329,"encode_duration_us":21940,"operation_id":"0xcc24","timeout_ms":10000,"last_server_duration_us":5629329,"total_duration_us":6481561},{"operation_name":"get","last_dispatch_duration_us":5889623,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":5889623,"total_server_duration_us":43,"operation_id":"0xcc6b","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5889843},{"operation_name":"upsert","last_dispatch_duration_us":5856621,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":5856621,"total_server_duration_us":146801,"encode_duration_us":32624,"operation_id":"0xcc84","timeout_ms":10000,"last_server_duration_us":146801,"total_duration_us":5866416}],"total_count":46}} WARN [2022-03-13 17:15:36,561] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16554752,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17167395,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9372411,"encode_duration_us":18555,"operation_id":"0xc5f3","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10004512},{"operation_name":"upsert","last_dispatch_duration_us":16409209,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17932074,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9441389,"encode_duration_us":17339,"operation_id":"0xc478","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10007175}],"total_count":2}} INFO [2022-03-13 17:15:36,591] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,593] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,593] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:36,595] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:15:36,595] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,595] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,596] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:36,597] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:15:36,608] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,608] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,609] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:15:36,614] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:36,627] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,629] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:36,630] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,631] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:36,632] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,633] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:15:36,642] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:36,642] jepsen worker nemesis - couchbase.util Rebalance check count: 445 INFO [2022-03-13 17:15:36,642] jepsen worker nemesis - couchbase.util Rebalance stuck count: 345 INFO [2022-03-13 17:15:36,651] jepsen worker 58 - jepsen.util 3028 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,652] jepsen worker 58 - jepsen.util 3028 :invoke :read [19 nil] INFO [2022-03-13 17:15:36,656] jepsen worker 26 - jepsen.util 476 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,656] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:36,712] jepsen worker 58 - jepsen.util 3028 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,712] jepsen worker 58 - jepsen.util 3028 :invoke :read [19 nil] INFO [2022-03-13 17:15:36,741] jepsen worker 58 - jepsen.util 3028 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,742] jepsen worker 58 - jepsen.util 3028 :invoke :read [19 nil] INFO [2022-03-13 17:15:36,772] jepsen worker 58 - jepsen.util 3028 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,773] jepsen worker 58 - jepsen.util 3028 :invoke :read [19 nil] INFO [2022-03-13 17:15:36,798] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,798] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:36,799] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,800] jepsen worker 58 - jepsen.util 3028 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,800] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:15:36,801] jepsen worker 58 - jepsen.util 3028 :invoke :read [19 nil] INFO [2022-03-13 17:15:36,829] jepsen worker 58 - jepsen.util 3028 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,829] jepsen worker 58 - jepsen.util 3028 :invoke :read [19 nil] INFO [2022-03-13 17:15:36,839] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,839] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:36,840] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,841] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:15:36,860] jepsen worker 58 - jepsen.util 3028 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,861] jepsen worker 58 - jepsen.util 3028 :invoke :write [19 34] INFO [2022-03-13 17:15:36,864] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,865] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:36,866] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,867] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:15:36,881] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,881] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,881] jepsen worker 51 - jepsen.util 681 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,882] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:36,884] jepsen worker 51 - jepsen.util 681 :invoke :write [17 11] INFO [2022-03-13 17:15:36,884] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:36,889] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,889] jepsen worker 13 - jepsen.util 643 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:36,890] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:36,891] jepsen worker 13 - jepsen.util 643 :invoke :write [4 18] INFO [2022-03-13 17:15:37,002] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,002] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:37,003] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,003] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:15:37,067] jepsen worker 70 - jepsen.util 790 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,068] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:37,069] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:37,069] jepsen worker 44 - jepsen.util 584 :invoke :write [14 9] INFO [2022-03-13 17:15:37,223] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,223] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 0] INFO [2022-03-13 17:15:37,225] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,225] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:15:37,238] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,238] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,238] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 3] INFO [2022-03-13 17:15:37,239] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:37,254] jepsen worker 66 - jepsen.util 3036 :info :write [22 17] :RequestTimeoutException INFO [2022-03-13 17:15:37,255] jepsen worker 66 - jepsen.util 3126 :invoke :read [22 nil] INFO [2022-03-13 17:15:37,272] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,272] jepsen worker 42 - jepsen.util 402 :invoke :write [14 18] INFO [2022-03-13 17:15:37,274] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,274] jepsen worker 52 - jepsen.util 592 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,275] jepsen worker 52 - jepsen.util 592 :invoke :write [17 42] INFO [2022-03-13 17:15:37,275] jepsen worker 34 - jepsen.util 844 :invoke :write [11 18] INFO [2022-03-13 17:15:37,277] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,277] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:37,305] jepsen worker 10 - jepsen.util 3070 :info :write [3 49] :SyncWriteAmbiguous INFO [2022-03-13 17:15:37,306] jepsen worker 10 - jepsen.util 3160 :invoke :read [3 nil] INFO [2022-03-13 17:15:37,322] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,323] jepsen worker 77 - jepsen.util 707 :invoke :write [25 36] INFO [2022-03-13 17:15:37,365] jepsen worker 10 - jepsen.util 3160 :ok :read [3 25] INFO [2022-03-13 17:15:37,365] jepsen worker 10 - jepsen.util 3160 :invoke :read [3 nil] INFO [2022-03-13 17:15:37,392] jepsen worker 10 - jepsen.util 3160 :ok :read [3 25] INFO [2022-03-13 17:15:37,393] jepsen worker 10 - jepsen.util 3160 :invoke :write [3 34] INFO [2022-03-13 17:15:37,409] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,409] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:37,415] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,416] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:37,417] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,417] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:37,418] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,419] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:15:37,428] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,428] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 43] INFO [2022-03-13 17:15:37,434] jepsen worker 56 - jepsen.util 3296 :info :write [18 35] :SyncWriteAmbiguous INFO [2022-03-13 17:15:37,435] jepsen worker 56 - jepsen.util 3386 :invoke :read [18 nil] INFO [2022-03-13 17:15:37,465] jepsen worker 45 - jepsen.util 3285 :info :write [15 45] :SyncWriteAmbiguous INFO [2022-03-13 17:15:37,466] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,466] jepsen worker 45 - jepsen.util 3375 :invoke :write [15 18] INFO [2022-03-13 17:15:37,467] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,467] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:37,468] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:15:37,476] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,477] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:37,634] jepsen worker 74 - jepsen.util 3494 :ok :read [24 24] INFO [2022-03-13 17:15:37,635] jepsen worker 74 - jepsen.util 3494 :invoke :read [24 nil] INFO [2022-03-13 17:15:37,647] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:37,648] jepsen worker nemesis - couchbase.util Rebalance check count: 446 INFO [2022-03-13 17:15:37,648] jepsen worker nemesis - couchbase.util Rebalance stuck count: 346 INFO [2022-03-13 17:15:37,680] jepsen worker 6 - jepsen.util 636 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,680] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,681] jepsen worker 6 - jepsen.util 636 :invoke :write [2 1] INFO [2022-03-13 17:15:37,681] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:37,691] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,692] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:37,704] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,704] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:15:37,722] jepsen worker 60 - jepsen.util 600 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,722] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,722] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,722] jepsen worker 60 - jepsen.util 600 :invoke :write [20 23] INFO [2022-03-13 17:15:37,724] jepsen worker 78 - jepsen.util 618 :invoke :write [26 3] INFO [2022-03-13 17:15:37,725] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:15:37,775] jepsen worker 74 - jepsen.util 3494 :ok :read [24 24] INFO [2022-03-13 17:15:37,775] jepsen worker 74 - jepsen.util 3494 :invoke :read [24 nil] INFO [2022-03-13 17:15:37,823] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,823] jepsen worker 88 - jepsen.util 2068 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,824] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:37,824] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,825] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:37,825] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:37,834] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,835] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:15:37,836] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,837] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:15:37,858] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,859] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,859] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,859] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:37,860] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:15:37,863] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 8] INFO [2022-03-13 17:15:37,884] jepsen worker 13 - jepsen.util 643 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:37,884] jepsen worker 13 - jepsen.util 643 :invoke :write [4 8] INFO [2022-03-13 17:15:37,923] jepsen worker 74 - jepsen.util 3494 :ok :read [24 24] INFO [2022-03-13 17:15:37,924] jepsen worker 74 - jepsen.util 3494 :invoke :read [24 nil] WARN [2022-03-13 17:15:37,958] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7647701,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":7647701,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":70825,"encode_duration_us":23804,"operation_id":"0xca80","timeout_ms":10000,"last_server_duration_us":70825,"total_duration_us":10001042},{"operation_name":"upsert","last_dispatch_duration_us":18431981,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18431981,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9044797,"encode_duration_us":61571,"operation_id":"0xc8b9","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10002893}],"total_count":2}} INFO [2022-03-13 17:15:38,018] jepsen worker 74 - jepsen.util 3494 :ok :read [24 24] INFO [2022-03-13 17:15:38,019] jepsen worker 74 - jepsen.util 3494 :invoke :read [24 nil] INFO [2022-03-13 17:15:38,028] jepsen worker 37 - jepsen.util 2107 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,029] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:38,096] jepsen worker 21 - jepsen.util 831 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,096] jepsen worker 21 - jepsen.util 831 :invoke :write [7 33] INFO [2022-03-13 17:15:38,110] jepsen worker 74 - jepsen.util 3494 :ok :read [24 24] INFO [2022-03-13 17:15:38,111] jepsen worker 74 - jepsen.util 3494 :invoke :write [24 12] WARN [2022-03-13 17:15:38,160] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9277272,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12093008,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9149781,"encode_duration_us":24936,"operation_id":"0xcc0d","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10003963},{"operation_name":"upsert","last_dispatch_duration_us":9046027,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12660344,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9426748,"encode_duration_us":19215,"operation_id":"0xc9d3","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10008625}],"total_count":2}} WARN [2022-03-13 17:15:38,160] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9068700,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13905510,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9379520,"encode_duration_us":17956,"operation_id":"0xc9b4","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10004174},{"operation_name":"upsert","last_dispatch_duration_us":18175069,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18175069,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9057542,"encode_duration_us":19724,"operation_id":"0xc6ea","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10004214}],"total_count":2}} WARN [2022-03-13 17:15:38,160] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8941755,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10119669,"total_server_duration_us":125162,"encode_duration_us":17461,"operation_id":"0xccd9","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10010446},{"operation_name":"upsert","last_dispatch_duration_us":8437516,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9240793,"total_server_duration_us":460511,"encode_duration_us":17003,"operation_id":"0xcaf1","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10007379},{"operation_name":"upsert","last_dispatch_duration_us":8881370,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":13322602,"total_server_duration_us":181244,"encode_duration_us":17727,"operation_id":"0xcbb7","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10007306},{"operation_name":"upsert","last_dispatch_duration_us":9277272,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":12093008,"total_server_duration_us":9149781,"encode_duration_us":24936,"operation_id":"0xcc0d","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10003963},{"operation_name":"upsert","last_dispatch_duration_us":8655764,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9233592,"total_server_duration_us":372806,"encode_duration_us":19413,"operation_id":"0xcb22","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10002029},{"operation_name":"upsert","last_dispatch_duration_us":45740,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8051470,"total_server_duration_us":126771,"encode_duration_us":16937,"operation_id":"0xccb5","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10001118},{"operation_name":"upsert","last_dispatch_duration_us":6775688,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":6775688,"total_server_duration_us":5521297,"encode_duration_us":26006,"operation_id":"0xcc03","timeout_ms":10000,"last_server_duration_us":5521297,"total_duration_us":6784412},{"operation_name":"get","last_dispatch_duration_us":6199209,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":6199209,"total_server_duration_us":32,"operation_id":"0xcc5f","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6199389},{"operation_name":"upsert","last_dispatch_duration_us":5865715,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":5865715,"total_server_duration_us":97793,"encode_duration_us":18051,"operation_id":"0xcc7f","timeout_ms":10000,"last_server_duration_us":97793,"total_duration_us":5874007},{"operation_name":"upsert","last_dispatch_duration_us":5701191,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":5701191,"total_server_duration_us":35312,"encode_duration_us":18973,"operation_id":"0xccad","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":5704663}],"total_count":27}} INFO [2022-03-13 17:15:38,231] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,232] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:15:38,241] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,242] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:38,244] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,244] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:38,249] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,250] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:38,257] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,258] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:15:38,259] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17237189,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":17237189,"total_server_duration_us":5813,"operation_id":"0xc961","timeout_ms":30000,"last_server_duration_us":5813,"total_duration_us":17239194},{"operation_name":"get","last_dispatch_duration_us":13904175,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":13904175,"total_server_duration_us":11552,"operation_id":"0xcaae","timeout_ms":30000,"last_server_duration_us":11552,"total_duration_us":13906321},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":45967,"operation_id":"0xcbfe","timeout_ms":10000,"total_duration_us":10007680},{"operation_name":"upsert","last_dispatch_duration_us":60128,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9403542,"total_server_duration_us":287202,"encode_duration_us":33933,"operation_id":"0xcc00","timeout_ms":10000,"last_server_duration_us":23983,"total_duration_us":10004156},{"operation_name":"upsert","last_dispatch_duration_us":8409458,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12692889,"total_server_duration_us":262030,"encode_duration_us":22760,"operation_id":"0xcb49","timeout_ms":10000,"last_server_duration_us":48909,"total_duration_us":10003872},{"operation_name":"upsert","last_dispatch_duration_us":2284127,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9885301,"total_server_duration_us":500566,"encode_duration_us":17632,"operation_id":"0xcc23","timeout_ms":10000,"last_server_duration_us":371326,"total_duration_us":9900746},{"operation_name":"upsert","last_dispatch_duration_us":2264925,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9690836,"total_server_duration_us":467273,"encode_duration_us":31909,"operation_id":"0xcbf9","timeout_ms":10000,"last_server_duration_us":356436,"total_duration_us":9711379},{"operation_name":"upsert","last_dispatch_duration_us":9125692,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9125692,"total_server_duration_us":9039489,"encode_duration_us":121011,"operation_id":"0xcc4b","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":9150530},{"operation_name":"upsert","last_dispatch_duration_us":1890525,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1890525,"total_server_duration_us":342070,"encode_duration_us":24684,"operation_id":"0xcc30","timeout_ms":10000,"last_server_duration_us":342070,"total_duration_us":8612377},{"operation_name":"get","last_dispatch_duration_us":1476280,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1476280,"total_server_duration_us":23,"operation_id":"0xcc25","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":8217219}],"total_count":40}} WARN [2022-03-13 17:15:38,260] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14535990,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":14535990,"total_server_duration_us":11118,"operation_id":"0xcacb","timeout_ms":30000,"last_server_duration_us":11118,"total_duration_us":14538021},{"operation_name":"upsert","last_dispatch_duration_us":6957626,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":6957626,"total_server_duration_us":21977,"encode_duration_us":19852,"operation_id":"0xccd1","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10008154},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":16648,"operation_id":"0xcb6e","timeout_ms":10000,"total_duration_us":10008121},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18271,"operation_id":"0xcb18","timeout_ms":10000,"total_duration_us":10005128},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":16862,"operation_id":"0xcc40","timeout_ms":10000,"total_duration_us":10005014},{"operation_name":"upsert","last_dispatch_duration_us":8910511,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8910511,"total_server_duration_us":20441,"encode_duration_us":25344,"operation_id":"0xcbfc","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10001851},{"operation_name":"upsert","last_dispatch_duration_us":5719927,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5719927,"total_server_duration_us":86769,"encode_duration_us":33008,"operation_id":"0xcc87","timeout_ms":10000,"last_server_duration_us":86769,"total_duration_us":5727045},{"operation_name":"upsert","last_dispatch_duration_us":1716345,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1716345,"total_server_duration_us":48105,"encode_duration_us":30391,"operation_id":"0xcd5e","timeout_ms":10000,"last_server_duration_us":48105,"total_duration_us":1730396},{"operation_name":"upsert","last_dispatch_duration_us":1655618,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1655618,"total_server_duration_us":1274306,"encode_duration_us":20911,"operation_id":"0xcd5d","timeout_ms":10000,"last_server_duration_us":1274306,"total_duration_us":1663795},{"operation_name":"upsert","last_dispatch_duration_us":1615786,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1615786,"total_server_duration_us":395968,"encode_duration_us":18014,"operation_id":"0xcfba","timeout_ms":10000,"last_server_duration_us":395968,"total_duration_us":1619646}],"total_count":37}} WARN [2022-03-13 17:15:38,260] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16392787,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21863231,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9281588,"encode_duration_us":17235,"operation_id":"0xc66b","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10001796},{"operation_name":"upsert","last_dispatch_duration_us":17446108,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17885079,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9308516,"encode_duration_us":22803,"operation_id":"0xc8ca","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10009755}],"total_count":2}} WARN [2022-03-13 17:15:38,260] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":114535,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7003848,"total_server_duration_us":262374,"encode_duration_us":17566,"operation_id":"0xcd1b","timeout_ms":10000,"last_server_duration_us":32258,"total_duration_us":10009521},{"operation_name":"upsert","last_dispatch_duration_us":8510402,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9424846,"total_server_duration_us":369843,"encode_duration_us":17560,"operation_id":"0xcb20","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10005445},{"operation_name":"upsert","last_dispatch_duration_us":67220,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8497220,"total_server_duration_us":271854,"encode_duration_us":18265,"operation_id":"0xccb4","timeout_ms":10000,"last_server_duration_us":46739,"total_duration_us":10003689},{"operation_name":"upsert","last_dispatch_duration_us":8634234,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9098364,"total_server_duration_us":245757,"encode_duration_us":18010,"operation_id":"0xcbb6","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10002529},{"operation_name":"upsert","last_dispatch_duration_us":30660,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9410531,"total_server_duration_us":221103,"encode_duration_us":17438,"operation_id":"0xcc0e","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10000448},{"operation_name":"upsert","last_dispatch_duration_us":2094668,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":9689300,"total_server_duration_us":45060,"encode_duration_us":23145,"operation_id":"0xcbfd","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":9733831},{"operation_name":"get","last_dispatch_duration_us":9378595,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9378595,"total_server_duration_us":10095,"operation_id":"0xcae2","timeout_ms":30000,"last_server_duration_us":10095,"total_duration_us":9381334},{"operation_name":"upsert","last_dispatch_duration_us":9081430,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9147400,"total_server_duration_us":9069725,"encode_duration_us":18027,"operation_id":"0xcd6e","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9159476},{"operation_name":"get","last_dispatch_duration_us":9133676,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9133676,"total_server_duration_us":19,"operation_id":"0xcaf4","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":9135624},{"operation_name":"get","last_dispatch_duration_us":9119238,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9119238,"total_server_duration_us":8778,"operation_id":"0xcaf5","timeout_ms":30000,"last_server_duration_us":8778,"total_duration_us":9121368}],"total_count":54}} WARN [2022-03-13 17:15:38,266] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7162983,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7717206,"total_server_duration_us":283129,"encode_duration_us":19159,"operation_id":"0xcc63","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10010102},{"operation_name":"upsert","last_dispatch_duration_us":29363,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":647181,"total_server_duration_us":323176,"encode_duration_us":19290,"operation_id":"0xcc42","timeout_ms":10000,"last_server_duration_us":18799,"total_duration_us":10009342},{"operation_name":"upsert","last_dispatch_duration_us":217283,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1905720,"total_server_duration_us":228310,"encode_duration_us":25909,"operation_id":"0xcc78","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10007470},{"operation_name":"upsert","last_dispatch_duration_us":7275402,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9272186,"total_server_duration_us":224748,"encode_duration_us":26434,"operation_id":"0xcca4","timeout_ms":10000,"last_server_duration_us":28036,"total_duration_us":10007041},{"operation_name":"upsert","last_dispatch_duration_us":7283964,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9195019,"total_server_duration_us":249857,"encode_duration_us":17131,"operation_id":"0xcc88","timeout_ms":10000,"last_server_duration_us":35111,"total_duration_us":10006393},{"operation_name":"upsert","last_dispatch_duration_us":325994,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1741952,"total_server_duration_us":198175,"encode_duration_us":19283,"operation_id":"0xccb6","timeout_ms":10000,"last_server_duration_us":47079,"total_duration_us":10001994},{"operation_name":"upsert","last_dispatch_duration_us":9173452,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9790213,"total_server_duration_us":9138879,"encode_duration_us":17024,"operation_id":"0xcc73","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9866355},{"operation_name":"upsert","last_dispatch_duration_us":8224717,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":8224717,"total_server_duration_us":5444066,"encode_duration_us":21420,"operation_id":"0xcbaf","timeout_ms":10000,"last_server_duration_us":5444066,"total_duration_us":8228288},{"operation_name":"get","last_dispatch_duration_us":6621991,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6621991,"total_server_duration_us":27,"operation_id":"0xcc11","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":6622635},{"operation_name":"get","last_dispatch_duration_us":5577289,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":5577289,"total_server_duration_us":19,"operation_id":"0xcc99","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":5577430}],"total_count":21}} INFO [2022-03-13 17:15:38,267] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,267] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:15:38,292] jepsen worker 34 - jepsen.util 844 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,293] jepsen worker 34 - jepsen.util 844 :invoke :write [11 11] INFO [2022-03-13 17:15:38,303] jepsen worker 72 - jepsen.util 3222 :info :write [24 34] :RequestTimeoutException INFO [2022-03-13 17:15:38,304] jepsen worker 72 - jepsen.util 3312 :invoke :write [24 41] INFO [2022-03-13 17:15:38,435] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,435] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] WARN [2022-03-13 17:15:38,466] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9049494,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18030316,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9160730,"encode_duration_us":20948,"operation_id":"0xc9a9","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10000780}],"total_count":1}} INFO [2022-03-13 17:15:38,470] jepsen worker 88 - jepsen.util 2068 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,471] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:38,503] jepsen worker 8 - jepsen.util 728 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,515] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,639] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,728] jepsen worker 8 - jepsen.util 728 :invoke :write [2 2] INFO [2022-03-13 17:15:38,639] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,639] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,729] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:15:38,729] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:38,640] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,729] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:38,652] jepsen worker 89 - jepsen.util 2519 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,653] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:38,729] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 13] INFO [2022-03-13 17:15:38,729] jepsen worker nemesis - couchbase.util Rebalance check count: 447 INFO [2022-03-13 17:15:38,729] jepsen worker nemesis - couchbase.util Rebalance stuck count: 347 INFO [2022-03-13 17:15:38,654] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,655] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,730] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:15:38,676] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,730] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 35] INFO [2022-03-13 17:15:38,730] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:38,700] jepsen worker 52 - jepsen.util 592 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,734] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:38,716] jepsen worker 35 - jepsen.util 935 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,734] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:38,716] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,728] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:38,735] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:38,729] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 21] INFO [2022-03-13 17:15:38,758] jepsen worker 61 - jepsen.util 691 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,760] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:38,843] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,843] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,844] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:15:38,844] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:15:38,909] jepsen worker 80 - jepsen.util 800 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,909] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,909] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:38,910] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:38,911] jepsen worker 13 - jepsen.util 643 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,911] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,912] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:38,912] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:38,913] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,913] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:38,915] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,915] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:38,919] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,920] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:38,922] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,923] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:15:38,961] jepsen worker 21 - jepsen.util 831 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,961] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:38,961] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,961] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,962] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:15:38,963] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:38,964] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,964] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:38,964] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:38,965] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:38,994] jepsen worker 48 - jepsen.util 3108 :info :write [16 27] :SyncWriteAmbiguous INFO [2022-03-13 17:15:38,995] jepsen worker 48 - jepsen.util 3198 :invoke :read [16 nil] INFO [2022-03-13 17:15:39,047] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,048] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:39,049] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,049] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:39,079] jepsen worker 66 - jepsen.util 3126 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,079] jepsen worker 66 - jepsen.util 3126 :invoke :read [22 nil] INFO [2022-03-13 17:15:39,098] jepsen worker 86 - jepsen.util 3056 :ok :read [28 44] INFO [2022-03-13 17:15:39,099] jepsen worker 86 - jepsen.util 3056 :invoke :read [28 nil] INFO [2022-03-13 17:15:39,108] jepsen worker 53 - jepsen.util 503 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,108] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:39,111] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:39,112] jepsen worker 43 - jepsen.util 583 :invoke :write [14 8] INFO [2022-03-13 17:15:39,116] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,116] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,116] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,117] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:15:39,117] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,117] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:39,118] jepsen worker 80 - jepsen.util 800 :invoke :write [26 3] INFO [2022-03-13 17:15:39,119] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:15:39,134] jepsen worker 86 - jepsen.util 3056 :ok :read [28 44] INFO [2022-03-13 17:15:39,134] jepsen worker 86 - jepsen.util 3056 :invoke :read [28 nil] INFO [2022-03-13 17:15:39,135] jepsen worker 66 - jepsen.util 3126 :ok :read [22 30] INFO [2022-03-13 17:15:39,137] jepsen worker 66 - jepsen.util 3126 :invoke :write [22 30] INFO [2022-03-13 17:15:39,171] jepsen worker 86 - jepsen.util 3056 :ok :read [28 44] INFO [2022-03-13 17:15:39,172] jepsen worker 86 - jepsen.util 3056 :invoke :read [28 nil] INFO [2022-03-13 17:15:39,228] jepsen worker 86 - jepsen.util 3056 :ok :read [28 44] INFO [2022-03-13 17:15:39,229] jepsen worker 86 - jepsen.util 3056 :invoke :write [28 49] INFO [2022-03-13 17:15:39,251] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,251] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,252] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:15:39,252] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:39,313] jepsen worker 34 - jepsen.util 844 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,313] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,313] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,314] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:39,315] jepsen worker 52 - jepsen.util 592 :invoke :write [17 41] INFO [2022-03-13 17:15:39,315] jepsen worker 13 - jepsen.util 643 :invoke :write [4 48] INFO [2022-03-13 17:15:39,319] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,320] jepsen worker 34 - jepsen.util 844 :invoke :write [11 24] INFO [2022-03-13 17:15:39,357] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,357] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,358] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:15:39,358] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:39,367] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,367] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:15:39,519] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,519] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,520] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,687] jepsen worker 76 - jepsen.util 706 :invoke :write [25 13] INFO [2022-03-13 17:15:39,687] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:15:39,520] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,687] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:39,687] jepsen worker 70 - jepsen.util 790 :invoke :write [23 4] INFO [2022-03-13 17:15:39,634] jepsen worker 55 - jepsen.util 3115 :info :write [18 40] :RequestTimeoutException INFO [2022-03-13 17:15:39,688] jepsen worker 55 - jepsen.util 3205 :invoke :read [18 nil] INFO [2022-03-13 17:15:39,640] jepsen worker 64 - jepsen.util 3124 :info :write [21 19] :RequestTimeoutException INFO [2022-03-13 17:15:39,690] jepsen worker 64 - jepsen.util 3214 :invoke :read [21 nil] INFO [2022-03-13 17:15:39,643] jepsen worker 9 - jepsen.util 2709 :info :write [3 23] :RequestTimeoutException INFO [2022-03-13 17:15:39,648] jepsen worker 31 - jepsen.util 3361 :info :write [10 42] :RequestTimeoutException INFO [2022-03-13 17:15:39,691] jepsen worker 9 - jepsen.util 2799 :invoke :write [3 14] INFO [2022-03-13 17:15:39,691] jepsen worker 31 - jepsen.util 3451 :invoke :write [10 3] INFO [2022-03-13 17:15:39,649] jepsen worker 29 - jepsen.util 3359 :info :write [9 11] :RequestTimeoutException INFO [2022-03-13 17:15:39,706] jepsen worker 29 - jepsen.util 3449 :invoke :write [9 45] INFO [2022-03-13 17:15:39,650] jepsen worker 30 - jepsen.util 3000 :info :write [10 14] :RequestTimeoutException INFO [2022-03-13 17:15:39,707] jepsen worker 30 - jepsen.util 3090 :invoke :read [10 nil] INFO [2022-03-13 17:15:39,721] jepsen worker 80 - jepsen.util 800 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,721] jepsen worker 80 - jepsen.util 800 :invoke :write [26 20] INFO [2022-03-13 17:15:39,735] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:39,735] jepsen worker nemesis - couchbase.util Rebalance check count: 448 INFO [2022-03-13 17:15:39,735] jepsen worker nemesis - couchbase.util Rebalance stuck count: 348 INFO [2022-03-13 17:15:39,757] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,757] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:15:39,770] jepsen worker 19 - jepsen.util 2899 :info :write [6 10] :RequestTimeoutException INFO [2022-03-13 17:15:39,771] jepsen worker 19 - jepsen.util 2989 :invoke :read [6 nil] INFO [2022-03-13 17:15:39,860] jepsen worker 2 - jepsen.util 3692 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:15:39,861] jepsen worker 2 - jepsen.util 3782 :invoke :read [0 nil] INFO [2022-03-13 17:15:39,870] jepsen worker 83 - jepsen.util 2603 :info :write [27 28] :RequestTimeoutException INFO [2022-03-13 17:15:39,870] jepsen worker 83 - jepsen.util 2693 :invoke :write [27 39] INFO [2022-03-13 17:15:39,926] jepsen worker 13 - jepsen.util 643 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,927] jepsen worker 13 - jepsen.util 643 :invoke :write [4 29] INFO [2022-03-13 17:15:39,928] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,928] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,929] jepsen worker 7 - jepsen.util 637 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,929] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:39,929] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:39,930] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:15:39,931] jepsen worker 7 - jepsen.util 637 :invoke :write [2 35] INFO [2022-03-13 17:15:39,932] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:40,008] jepsen worker 46 - jepsen.util 3196 :info :write [15 18] :RequestTimeoutException INFO [2022-03-13 17:15:40,009] jepsen worker 46 - jepsen.util 3286 :invoke :write [15 35] INFO [2022-03-13 17:15:40,009] jepsen worker 52 - jepsen.util 592 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,010] jepsen worker 52 - jepsen.util 592 :invoke :write [17 39] INFO [2022-03-13 17:15:40,014] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,015] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:15:40,342] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,343] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:15:40,346] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,346] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:15:40,355] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,358] jepsen worker 44 - jepsen.util 584 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,358] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:15:40,363] jepsen worker 44 - jepsen.util 584 :invoke :write [14 2] INFO [2022-03-13 17:15:40,398] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,399] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:40,402] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,403] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:15:40,450] jepsen worker 3 - jepsen.util 1983 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,450] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:40,458] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,459] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:40,459] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,460] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:15:40,492] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,492] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:15:40,553] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,553] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:15:40,740] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:40,740] jepsen worker nemesis - couchbase.util Rebalance check count: 449 INFO [2022-03-13 17:15:40,740] jepsen worker nemesis - couchbase.util Rebalance stuck count: 349 INFO [2022-03-13 17:15:40,749] jepsen worker 80 - jepsen.util 800 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,749] jepsen worker 80 - jepsen.util 800 :invoke :write [26 30] INFO [2022-03-13 17:15:40,751] jepsen worker 79 - jepsen.util 1069 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,752] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:40,754] jepsen worker 34 - jepsen.util 844 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,755] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:40,767] jepsen worker 89 - jepsen.util 2519 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,767] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,767] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:40,770] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:15:40,792] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,793] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:40,808] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,809] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:15:40,811] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,811] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:40,930] jepsen worker 65 - jepsen.util 3665 :info :write [21 35] :RequestTimeoutException INFO [2022-03-13 17:15:40,931] jepsen worker 65 - jepsen.util 3755 :invoke :write [21 25] INFO [2022-03-13 17:15:40,958] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:40,959] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:15:41,156] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,157] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:41,175] jepsen worker 77 - jepsen.util 707 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,176] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:15:41,194] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,194] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:41,360] jepsen worker 13 - jepsen.util 643 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,361] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:41,367] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,368] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:41,375] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,376] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:15:41,377] jepsen worker 37 - jepsen.util 2107 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,377] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,379] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 0] INFO [2022-03-13 17:15:41,379] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:15:41,564] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,564] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:15:41,579] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,580] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:41,746] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:41,746] jepsen worker nemesis - couchbase.util Rebalance check count: 450 INFO [2022-03-13 17:15:41,746] jepsen worker nemesis - couchbase.util Rebalance stuck count: 350 INFO [2022-03-13 17:15:41,904] jepsen worker 49 - jepsen.util 3109 :info :write [16 36] :RequestTimeoutException INFO [2022-03-13 17:15:41,904] jepsen worker 49 - jepsen.util 3199 :invoke :read [16 nil] INFO [2022-03-13 17:15:41,960] jepsen worker 1 - jepsen.util 2701 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,960] jepsen worker 85 - jepsen.util 2965 :fail :read [28 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,960] jepsen worker 17 - jepsen.util 3167 :fail :read [5 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,961] jepsen worker 85 - jepsen.util 2965 :invoke :write [28 37] INFO [2022-03-13 17:15:41,961] jepsen worker 40 - jepsen.util 3370 :fail :read [13 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:41,962] jepsen worker 1 - jepsen.util 2701 :invoke :read [0 nil] INFO [2022-03-13 17:15:41,963] jepsen worker 40 - jepsen.util 3370 :invoke :write [13 3] INFO [2022-03-13 17:15:41,964] jepsen worker 17 - jepsen.util 3167 :invoke :read [5 nil] INFO [2022-03-13 17:15:41,969] jepsen worker 67 - jepsen.util 2677 :ok :read [22 30] INFO [2022-03-13 17:15:41,970] jepsen worker 67 - jepsen.util 2677 :invoke :write [22 32] INFO [2022-03-13 17:15:42,050] jepsen worker 2 - jepsen.util 3782 :ok :read [0 41] INFO [2022-03-13 17:15:42,050] jepsen worker 2 - jepsen.util 3782 :invoke :read [0 nil] INFO [2022-03-13 17:15:42,061] jepsen worker 1 - jepsen.util 2701 :ok :read [0 41] INFO [2022-03-13 17:15:42,061] jepsen worker 1 - jepsen.util 2701 :invoke :write [0 37] INFO [2022-03-13 17:15:42,095] jepsen worker 17 - jepsen.util 3167 :ok :read [5 43] INFO [2022-03-13 17:15:42,095] jepsen worker 17 - jepsen.util 3167 :invoke :read [5 nil] INFO [2022-03-13 17:15:42,192] jepsen worker 2 - jepsen.util 3782 :ok :read [0 41] INFO [2022-03-13 17:15:42,193] jepsen worker 2 - jepsen.util 3782 :invoke :read [0 nil] INFO [2022-03-13 17:15:42,313] jepsen worker 17 - jepsen.util 3167 :ok :read [5 43] INFO [2022-03-13 17:15:42,313] jepsen worker 17 - jepsen.util 3167 :invoke :write [5 19] INFO [2022-03-13 17:15:42,320] jepsen worker 2 - jepsen.util 3782 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,321] jepsen worker 2 - jepsen.util 3782 :invoke :read [0 nil] INFO [2022-03-13 17:15:42,373] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,374] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:42,375] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,375] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,376] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:42,376] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:42,377] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,377] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:15:42,385] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,385] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,385] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,385] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:42,386] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:42,386] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:42,399] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,399] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:15:42,399] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,400] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:42,400] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,401] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:42,402] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,402] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:42,403] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,404] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:15:42,437] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,438] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:15:42,448] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,449] jepsen worker 63 - jepsen.util 3303 :ok :read [21 :nil] INFO [2022-03-13 17:15:42,450] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:42,451] jepsen worker 63 - jepsen.util 3303 :invoke :write [21 2] INFO [2022-03-13 17:15:42,452] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,455] jepsen worker 2 - jepsen.util 3782 :ok :read [0 41] INFO [2022-03-13 17:15:42,457] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:42,458] jepsen worker 2 - jepsen.util 3782 :invoke :read [0 nil] INFO [2022-03-13 17:15:42,459] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,460] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:42,462] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,464] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:15:42,484] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,487] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:15:42,578] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,579] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:15:42,715] jepsen worker 2 - jepsen.util 3782 :ok :read [0 41] INFO [2022-03-13 17:15:42,715] jepsen worker 2 - jepsen.util 3782 :invoke :write [0 23] INFO [2022-03-13 17:15:42,751] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:42,751] jepsen worker nemesis - couchbase.util Rebalance check count: 451 INFO [2022-03-13 17:15:42,751] jepsen worker nemesis - couchbase.util Rebalance stuck count: 351 INFO [2022-03-13 17:15:42,799] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,800] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:42,815] jepsen worker 37 - jepsen.util 2107 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,815] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:42,815] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 14] INFO [2022-03-13 17:15:42,816] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 41] INFO [2022-03-13 17:15:42,846] jepsen worker 33 - jepsen.util 663 :info :write [11 12] :RequestTimeoutException INFO [2022-03-13 17:15:42,847] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:42,905] jepsen worker 11 - jepsen.util 3251 :ok :read [3 25] INFO [2022-03-13 17:15:42,906] jepsen worker 11 - jepsen.util 3251 :invoke :write [3 15] INFO [2022-03-13 17:15:42,923] jepsen worker 30 - jepsen.util 3090 :ok :read [10 :nil] INFO [2022-03-13 17:15:42,923] jepsen worker 30 - jepsen.util 3090 :invoke :read [10 nil] INFO [2022-03-13 17:15:42,924] jepsen worker 30 - jepsen.util 3090 :ok :read [10 :nil] INFO [2022-03-13 17:15:42,925] jepsen worker 30 - jepsen.util 3090 :invoke :write [10 1] INFO [2022-03-13 17:15:42,977] jepsen worker 27 - jepsen.util 3627 :info :write [9 30] :SyncWriteAmbiguous INFO [2022-03-13 17:15:42,977] jepsen worker 27 - jepsen.util 3717 :invoke :read [9 nil] INFO [2022-03-13 17:15:43,190] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,190] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,190] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:43,191] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:15:43,192] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,193] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 9] INFO [2022-03-13 17:15:43,240] jepsen worker 20 - jepsen.util 3530 :info :write [6 48] :RequestTimeoutException INFO [2022-03-13 17:15:43,241] jepsen worker 20 - jepsen.util 3620 :invoke :write [6 20] INFO [2022-03-13 17:15:43,258] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,259] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:15:43,262] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,265] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:43,266] jepsen worker 39 - jepsen.util 3279 :info :write [13 27] :RequestTimeoutException INFO [2022-03-13 17:15:43,267] jepsen worker 39 - jepsen.util 3369 :invoke :write [13 44] INFO [2022-03-13 17:15:43,391] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,391] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:43,392] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,393] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,393] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:15:43,394] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:15:43,395] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,396] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:15:43,756] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:43,756] jepsen worker nemesis - couchbase.util Rebalance check count: 452 INFO [2022-03-13 17:15:43,817] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,850] jepsen worker nemesis - couchbase.util Rebalance stuck count: 352 INFO [2022-03-13 17:15:43,845] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,850] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:15:43,847] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,847] jepsen worker 80 - jepsen.util 800 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,848] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,852] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:43,852] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:43,848] jepsen worker 42 - jepsen.util 402 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,850] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:15:43,852] jepsen worker 42 - jepsen.util 402 :invoke :write [14 40] INFO [2022-03-13 17:15:43,852] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:43,898] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,898] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,899] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,899] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,899] jepsen worker 21 - jepsen.util 831 :invoke :write [7 23] INFO [2022-03-13 17:15:43,901] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,901] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,901] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:43,902] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:15:43,903] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:43,904] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:43,904] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:15:43,904] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:15:43,905] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:15:43,950] jepsen worker 28 - jepsen.util 3358 :info :write [9 9] :RequestTimeoutException INFO [2022-03-13 17:15:43,951] jepsen worker 28 - jepsen.util 3448 :invoke :read [9 nil] INFO [2022-03-13 17:15:44,059] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,060] jepsen worker 33 - jepsen.util 753 :invoke :write [11 41] INFO [2022-03-13 17:15:44,076] jepsen worker 52 - jepsen.util 592 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,077] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,077] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:44,078] jepsen worker 13 - jepsen.util 643 :invoke :write [4 1] INFO [2022-03-13 17:15:44,078] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,079] jepsen worker 34 - jepsen.util 844 :invoke :write [11 19] INFO [2022-03-13 17:15:44,080] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,081] jepsen worker 12 - jepsen.util 732 :invoke :write [4 39] INFO [2022-03-13 17:15:44,092] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,092] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:44,095] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,096] jepsen worker 21 - jepsen.util 831 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,096] jepsen worker 28 - jepsen.util 3448 :ok :read [9 26] INFO [2022-03-13 17:15:44,096] jepsen worker 52 - jepsen.util 592 :invoke :write [17 36] INFO [2022-03-13 17:15:44,098] jepsen worker 28 - jepsen.util 3448 :invoke :read [9 nil] INFO [2022-03-13 17:15:44,098] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:44,263] jepsen worker 28 - jepsen.util 3448 :ok :read [9 26] INFO [2022-03-13 17:15:44,264] jepsen worker 28 - jepsen.util 3448 :invoke :write [9 22] INFO [2022-03-13 17:15:44,310] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,310] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:44,320] jepsen worker 68 - jepsen.util 3398 :info :write [22 38] :SyncWriteAmbiguous INFO [2022-03-13 17:15:44,321] jepsen worker 54 - jepsen.util 2844 :info :write [18 12] :RequestTimeoutException INFO [2022-03-13 17:15:44,321] jepsen worker 68 - jepsen.util 3488 :invoke :read [22 nil] INFO [2022-03-13 17:15:44,322] jepsen worker 54 - jepsen.util 2934 :invoke :write [18 5] INFO [2022-03-13 17:15:44,326] jepsen worker 59 - jepsen.util 3209 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,327] jepsen worker 59 - jepsen.util 3209 :invoke :write [19 2] INFO [2022-03-13 17:15:44,459] jepsen worker 48 - jepsen.util 3198 :ok :read [16 :nil] INFO [2022-03-13 17:15:44,459] jepsen worker 49 - jepsen.util 3199 :ok :read [16 :nil] INFO [2022-03-13 17:15:44,460] jepsen worker 48 - jepsen.util 3198 :invoke :write [16 22] INFO [2022-03-13 17:15:44,460] jepsen worker 49 - jepsen.util 3199 :invoke :read [16 nil] INFO [2022-03-13 17:15:44,463] jepsen worker 82 - jepsen.util 3232 :ok :read [27 4] INFO [2022-03-13 17:15:44,464] jepsen worker 82 - jepsen.util 3232 :invoke :read [27 nil] INFO [2022-03-13 17:15:44,474] jepsen worker 81 - jepsen.util 3681 :ok :read [27 4] INFO [2022-03-13 17:15:44,475] jepsen worker 81 - jepsen.util 3681 :invoke :read [27 nil] INFO [2022-03-13 17:15:44,476] jepsen worker 84 - jepsen.util 3594 :info :write [28 40] :RequestTimeoutException INFO [2022-03-13 17:15:44,477] jepsen worker 84 - jepsen.util 3684 :invoke :read [28 nil] INFO [2022-03-13 17:15:44,478] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,478] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:15:44,484] jepsen worker 6 - jepsen.util 636 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,484] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:44,513] jepsen worker 68 - jepsen.util 3488 :ok :read [22 30] INFO [2022-03-13 17:15:44,514] jepsen worker 68 - jepsen.util 3488 :invoke :write [22 27] INFO [2022-03-13 17:15:44,518] jepsen worker 34 - jepsen.util 844 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,518] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:44,524] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,524] jepsen worker 33 - jepsen.util 753 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,524] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:44,525] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:44,525] jepsen worker 49 - jepsen.util 3199 :ok :read [16 :nil] INFO [2022-03-13 17:15:44,526] jepsen worker 49 - jepsen.util 3199 :invoke :read [16 nil] INFO [2022-03-13 17:15:44,551] jepsen worker 82 - jepsen.util 3232 :ok :read [27 4] INFO [2022-03-13 17:15:44,552] jepsen worker 82 - jepsen.util 3232 :invoke :write [27 24] INFO [2022-03-13 17:15:44,586] jepsen worker 81 - jepsen.util 3681 :ok :read [27 4] INFO [2022-03-13 17:15:44,587] jepsen worker 81 - jepsen.util 3681 :invoke :write [27 19] INFO [2022-03-13 17:15:44,602] jepsen worker 49 - jepsen.util 3199 :ok :read [16 :nil] INFO [2022-03-13 17:15:44,603] jepsen worker 49 - jepsen.util 3199 :invoke :write [16 41] INFO [2022-03-13 17:15:44,667] jepsen worker 18 - jepsen.util 3078 :info :write [6 46] :RequestTimeoutException INFO [2022-03-13 17:15:44,668] jepsen worker 18 - jepsen.util 3168 :invoke :write [6 6] INFO [2022-03-13 17:15:44,699] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,700] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:15:44,856] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:44,856] jepsen worker nemesis - couchbase.util Rebalance check count: 453 INFO [2022-03-13 17:15:44,856] jepsen worker nemesis - couchbase.util Rebalance stuck count: 353 INFO [2022-03-13 17:15:44,903] jepsen worker 13 - jepsen.util 643 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,903] jepsen worker 60 - jepsen.util 600 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,904] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:44,904] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:44,904] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:44,905] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:15:45,010] jepsen worker 47 - jepsen.util 3107 :info :write [15 18] :RequestTimeoutException INFO [2022-03-13 17:15:45,011] jepsen worker 47 - jepsen.util 3197 :invoke :write [15 36] INFO [2022-03-13 17:15:45,100] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,101] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,101] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,101] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,102] jepsen worker 80 - jepsen.util 800 :invoke :write [26 45] INFO [2022-03-13 17:15:45,103] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:45,103] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:15:45,104] jepsen worker 33 - jepsen.util 753 :invoke :write [11 33] INFO [2022-03-13 17:15:45,304] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,305] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:45,306] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,307] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:45,308] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,309] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:45,310] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,311] jepsen worker 8 - jepsen.util 728 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,311] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:15:45,311] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,312] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:45,315] jepsen worker 8 - jepsen.util 728 :invoke :write [2 20] INFO [2022-03-13 17:15:45,336] jepsen worker 16 - jepsen.util 3076 :info :write [5 16] :RequestTimeoutException INFO [2022-03-13 17:15:45,336] jepsen worker 16 - jepsen.util 3166 :invoke :read [5 nil] INFO [2022-03-13 17:15:45,362] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,362] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:15:45,446] jepsen worker 0 - jepsen.util 3060 :info :write [0 38] :RequestTimeoutException INFO [2022-03-13 17:15:45,447] jepsen worker 0 - jepsen.util 3150 :invoke :write [0 32] INFO [2022-03-13 17:15:45,504] jepsen worker 78 - jepsen.util 618 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,504] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,504] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:45,505] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:45,523] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,523] jepsen worker 52 - jepsen.util 592 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,524] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,525] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:45,525] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:45,526] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,526] jepsen worker 52 - jepsen.util 592 :invoke :write [17 9] INFO [2022-03-13 17:15:45,526] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,527] jepsen worker 13 - jepsen.util 643 :invoke :write [4 19] INFO [2022-03-13 17:15:45,527] jepsen worker 34 - jepsen.util 844 :invoke :write [11 18] INFO [2022-03-13 17:15:45,530] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,530] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,531] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:15:45,545] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:15:45,648] jepsen worker 57 - jepsen.util 3297 :info :write [19 25] :RequestTimeoutException INFO [2022-03-13 17:15:45,648] jepsen worker 57 - jepsen.util 3387 :invoke :read [19 nil] INFO [2022-03-13 17:15:45,728] jepsen worker 80 - jepsen.util 800 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,728] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:45,731] jepsen worker 43 - jepsen.util 583 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,732] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,733] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:45,734] jepsen worker 43 - jepsen.util 583 :invoke :write [14 14] INFO [2022-03-13 17:15:45,804] jepsen worker 41 - jepsen.util 3281 :info :write [13 15] :RequestTimeoutException INFO [2022-03-13 17:15:45,804] jepsen worker 41 - jepsen.util 3371 :invoke :read [13 nil] INFO [2022-03-13 17:15:45,861] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:45,862] jepsen worker nemesis - couchbase.util Rebalance check count: 454 INFO [2022-03-13 17:15:45,862] jepsen worker nemesis - couchbase.util Rebalance stuck count: 354 INFO [2022-03-13 17:15:45,912] jepsen worker 33 - jepsen.util 753 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,912] jepsen worker 33 - jepsen.util 753 :invoke :write [11 41] INFO [2022-03-13 17:15:45,920] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,920] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:45,922] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,923] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:15:45,965] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:45,966] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:46,123] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:46,124] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:46,141] jepsen worker 52 - jepsen.util 592 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:46,142] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:46,155] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:46,156] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:15:46,332] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:46,332] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:46,346] jepsen worker 32 - jepsen.util 3272 :info :write [10 17] :RequestTimeoutException INFO [2022-03-13 17:15:46,347] jepsen worker 73 - jepsen.util 3493 :ok :read [24 24] INFO [2022-03-13 17:15:46,347] jepsen worker 32 - jepsen.util 3362 :invoke :write [10 11] INFO [2022-03-13 17:15:46,347] jepsen worker 73 - jepsen.util 3493 :invoke :read [24 nil] INFO [2022-03-13 17:15:46,480] jepsen worker 10 - jepsen.util 3160 :info :write [3 34] :SyncWriteAmbiguous INFO [2022-03-13 17:15:46,482] jepsen worker 10 - jepsen.util 3250 :invoke :read [3 nil] WARN [2022-03-13 17:15:46,583] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11697888,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11697888,"total_server_duration_us":5767,"operation_id":"0xcf8f","timeout_ms":30000,"last_server_duration_us":5767,"total_duration_us":11700118},{"operation_name":"upsert","last_dispatch_duration_us":6877150,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7531084,"total_server_duration_us":316819,"encode_duration_us":17518,"operation_id":"0xcdd9","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10008544},{"operation_name":"upsert","last_dispatch_duration_us":12367288,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":12367288,"total_server_duration_us":37241,"encode_duration_us":19054,"operation_id":"0xcf48","timeout_ms":10000,"last_server_duration_us":37241,"total_duration_us":10008254},{"operation_name":"upsert","last_dispatch_duration_us":97403,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7637461,"total_server_duration_us":375276,"encode_duration_us":17219,"operation_id":"0xcdda","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10006479},{"operation_name":"upsert","last_dispatch_duration_us":6266015,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6964807,"total_server_duration_us":263139,"encode_duration_us":33231,"operation_id":"0xcdbd","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10005894},{"operation_name":"upsert","last_dispatch_duration_us":8926552,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15973601,"total_server_duration_us":271598,"encode_duration_us":17435,"operation_id":"0xcdcb","timeout_ms":10000,"last_server_duration_us":27132,"total_duration_us":10004101},{"operation_name":"upsert","last_dispatch_duration_us":15261283,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16006788,"total_server_duration_us":9265713,"encode_duration_us":41769,"operation_id":"0xcdc2","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10002085},{"operation_name":"upsert","last_dispatch_duration_us":9117174,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9153966,"total_server_duration_us":9063595,"encode_duration_us":19958,"operation_id":"0xcdd6","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9167418},{"operation_name":"upsert","last_dispatch_duration_us":3630074,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3630074,"total_server_duration_us":602237,"encode_duration_us":30027,"operation_id":"0xcfb6","timeout_ms":10000,"last_server_duration_us":602237,"total_duration_us":3638761},{"operation_name":"get","last_dispatch_duration_us":3508509,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":3508509,"total_server_duration_us":7282,"operation_id":"0xcfd9","timeout_ms":30000,"last_server_duration_us":7282,"total_duration_us":3510567}],"total_count":56}} WARN [2022-03-13 17:15:46,583] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15261283,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16006788,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9265713,"encode_duration_us":41769,"operation_id":"0xcdc2","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10002085},{"operation_name":"upsert","last_dispatch_duration_us":17347745,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18146310,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9157730,"encode_duration_us":18064,"operation_id":"0xcaed","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10006523}],"total_count":2}} INFO [2022-03-13 17:15:46,867] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:46,867] jepsen worker nemesis - couchbase.util Rebalance check count: 455 INFO [2022-03-13 17:15:46,867] jepsen worker nemesis - couchbase.util Rebalance stuck count: 355 INFO [2022-03-13 17:15:46,872] jepsen worker 45 - jepsen.util 3375 :info :write [15 18] :SyncWriteAmbiguous INFO [2022-03-13 17:15:46,873] jepsen worker 45 - jepsen.util 3465 :invoke :write [15 13] INFO [2022-03-13 17:15:46,886] jepsen worker 58 - jepsen.util 3028 :info :write [19 34] :RequestTimeoutException INFO [2022-03-13 17:15:46,887] jepsen worker 58 - jepsen.util 3118 :invoke :read [19 nil] INFO [2022-03-13 17:15:46,916] jepsen worker 51 - jepsen.util 681 :info :write [17 11] :RequestTimeoutException INFO [2022-03-13 17:15:46,917] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:15:47,076] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,076] jepsen worker 7 - jepsen.util 637 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,076] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,076] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 41] INFO [2022-03-13 17:15:47,077] jepsen worker 7 - jepsen.util 637 :invoke :write [2 37] INFO [2022-03-13 17:15:47,077] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:47,115] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,115] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,116] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:47,117] jepsen worker 80 - jepsen.util 800 :invoke :write [26 2] INFO [2022-03-13 17:15:47,137] jepsen worker 13 - jepsen.util 643 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,138] jepsen worker 13 - jepsen.util 643 :invoke :write [4 23] INFO [2022-03-13 17:15:47,213] jepsen worker 10 - jepsen.util 3250 :ok :read [3 25] INFO [2022-03-13 17:15:47,213] jepsen worker 10 - jepsen.util 3250 :invoke :read [3 nil] INFO [2022-03-13 17:15:47,263] jepsen worker 70 - jepsen.util 790 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,264] jepsen worker 70 - jepsen.util 790 :invoke :write [23 46] INFO [2022-03-13 17:15:47,269] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,269] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,270] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:15:47,270] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:47,276] jepsen worker 33 - jepsen.util 753 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,276] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,277] jepsen worker 33 - jepsen.util 753 :invoke :write [11 13] INFO [2022-03-13 17:15:47,278] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:47,288] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,289] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:15:47,291] jepsen worker 36 - jepsen.util 2196 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,291] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,293] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:47,296] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 38] INFO [2022-03-13 17:15:47,296] jepsen worker 3 - jepsen.util 1983 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,297] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:47,299] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,300] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:47,300] jepsen worker 10 - jepsen.util 3250 :ok :read [3 25] INFO [2022-03-13 17:15:47,301] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,301] jepsen worker 10 - jepsen.util 3250 :invoke :read [3 nil] INFO [2022-03-13 17:15:47,302] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:47,302] jepsen worker 37 - jepsen.util 2107 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,303] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 20] INFO [2022-03-13 17:15:47,303] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,304] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:47,305] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,306] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:15:47,311] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,312] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:15:47,331] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,332] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,332] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:47,333] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,333] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:47,333] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:15:47,334] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,334] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,335] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:15:47,335] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:15:47,349] jepsen worker 10 - jepsen.util 3250 :ok :read [3 25] INFO [2022-03-13 17:15:47,349] jepsen worker 10 - jepsen.util 3250 :invoke :read [3 nil] INFO [2022-03-13 17:15:47,392] jepsen worker 10 - jepsen.util 3250 :ok :read [3 25] INFO [2022-03-13 17:15:47,392] jepsen worker 10 - jepsen.util 3250 :invoke :write [3 31] INFO [2022-03-13 17:15:47,668] jepsen worker 80 - jepsen.util 800 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,668] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,668] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,668] jepsen worker 80 - jepsen.util 800 :invoke :write [26 26] INFO [2022-03-13 17:15:47,669] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:15:47,669] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:15:47,687] jepsen worker 42 - jepsen.util 402 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,687] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,687] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,688] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:47,688] jepsen worker 6 - jepsen.util 636 :invoke :write [2 12] INFO [2022-03-13 17:15:47,689] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:47,699] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,699] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,699] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,700] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:47,700] jepsen worker 33 - jepsen.util 753 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,700] jepsen worker 14 - jepsen.util 374 :invoke :write [4 33] INFO [2022-03-13 17:15:47,702] jepsen worker 33 - jepsen.util 753 :invoke :write [11 45] INFO [2022-03-13 17:15:47,702] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:15:47,763] jepsen worker 34 - jepsen.util 844 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,763] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,764] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:47,764] jepsen worker 52 - jepsen.util 592 :invoke :write [17 28] INFO [2022-03-13 17:15:47,872] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:47,872] jepsen worker nemesis - couchbase.util Rebalance check count: 456 INFO [2022-03-13 17:15:47,872] jepsen worker nemesis - couchbase.util Rebalance stuck count: 356 INFO [2022-03-13 17:15:47,892] jepsen worker 12 - jepsen.util 732 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,893] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,894] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:15:47,895] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:47,895] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,896] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:47,900] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,901] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:47,916] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,916] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:47,942] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,942] jepsen worker 8 - jepsen.util 728 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,943] jepsen worker 8 - jepsen.util 728 :invoke :write [2 31] INFO [2022-03-13 17:15:47,943] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:47,960] jepsen worker 80 - jepsen.util 800 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,960] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:47,969] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,969] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] WARN [2022-03-13 17:15:47,974] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18078221,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27152001,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9090815,"encode_duration_us":26863,"operation_id":"0xc8e8","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":10004066}],"total_count":1}} INFO [2022-03-13 17:15:47,977] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:47,978] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:48,103] jepsen worker 89 - jepsen.util 2519 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,104] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:48,134] jepsen worker 74 - jepsen.util 3494 :info :write [24 12] :RequestTimeoutException INFO [2022-03-13 17:15:48,135] jepsen worker 74 - jepsen.util 3584 :invoke :read [24 nil] INFO [2022-03-13 17:15:48,147] jepsen worker 76 - jepsen.util 706 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,147] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,147] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,147] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,147] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,148] jepsen worker 61 - jepsen.util 691 :invoke :write [20 43] INFO [2022-03-13 17:15:48,148] jepsen worker 76 - jepsen.util 706 :invoke :write [25 18] INFO [2022-03-13 17:15:48,149] jepsen worker 69 - jepsen.util 699 :invoke :write [23 3] INFO [2022-03-13 17:15:48,150] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:48,150] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:48,152] jepsen worker 13 - jepsen.util 643 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,153] jepsen worker 13 - jepsen.util 643 :invoke :write [4 8] INFO [2022-03-13 17:15:48,154] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,154] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:48,155] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,155] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,155] jepsen worker 34 - jepsen.util 844 :invoke :write [11 20] INFO [2022-03-13 17:15:48,156] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:48,156] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,157] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:15:48,157] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,158] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,159] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:48,159] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] WARN [2022-03-13 17:15:48,173] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":120223,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9363843,"total_server_duration_us":90073,"encode_duration_us":27247,"operation_id":"0xcf05","timeout_ms":10000,"last_server_duration_us":38066,"total_duration_us":10010417},{"operation_name":"upsert","last_dispatch_duration_us":12636836,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":12636836,"total_server_duration_us":22141,"encode_duration_us":36398,"operation_id":"0xcdc0","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10010075},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":19922,"operation_id":"0xcfa7","timeout_ms":10000,"total_duration_us":10002387},{"operation_name":"upsert","last_dispatch_duration_us":11386992,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11386992,"total_server_duration_us":19730,"encode_duration_us":18260,"operation_id":"0xce19","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10001736},{"operation_name":"get","last_dispatch_duration_us":115087,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":115087,"total_server_duration_us":3533,"operation_id":"0xcfe8","timeout_ms":30000,"last_server_duration_us":3533,"total_duration_us":6812528},{"operation_name":"upsert","last_dispatch_duration_us":4033587,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":4033587,"total_server_duration_us":562885,"encode_duration_us":26591,"operation_id":"0xd227","timeout_ms":10000,"last_server_duration_us":562885,"total_duration_us":4037847},{"operation_name":"upsert","last_dispatch_duration_us":3863407,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":3863407,"total_server_duration_us":3431211,"encode_duration_us":25230,"operation_id":"0xd1fd","timeout_ms":10000,"last_server_duration_us":3431211,"total_duration_us":3868221},{"operation_name":"upsert","last_dispatch_duration_us":3458183,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":3458183,"total_server_duration_us":29975,"encode_duration_us":17735,"operation_id":"0xd225","timeout_ms":10000,"last_server_duration_us":29975,"total_duration_us":3462166},{"operation_name":"upsert","last_dispatch_duration_us":2890195,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2890195,"total_server_duration_us":354828,"encode_duration_us":19274,"operation_id":"0xd198","timeout_ms":10000,"last_server_duration_us":354828,"total_duration_us":2893664},{"operation_name":"upsert","last_dispatch_duration_us":1859850,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":1859850,"total_server_duration_us":1170525,"encode_duration_us":16806,"operation_id":"0xd14d","timeout_ms":10000,"last_server_duration_us":1170525,"total_duration_us":1863460}],"total_count":29}} WARN [2022-03-13 17:15:48,174] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9057167,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17108638,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9167322,"encode_duration_us":16937,"operation_id":"0xccb5","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10001118},{"operation_name":"upsert","last_dispatch_duration_us":9054686,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11719103,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9222027,"encode_duration_us":22196,"operation_id":"0xcf69","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10005966},{"operation_name":"upsert","last_dispatch_duration_us":14424230,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":14424230,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":194167,"encode_duration_us":17559,"operation_id":"0xceea","timeout_ms":10000,"last_server_duration_us":194167,"total_duration_us":10005827}],"total_count":3}} WARN [2022-03-13 17:15:48,174] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17723767,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":17723767,"total_server_duration_us":410675,"operation_id":"0xcbe2","timeout_ms":30000,"last_server_duration_us":410675,"total_duration_us":17724081},{"operation_name":"get","last_dispatch_duration_us":17714239,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":17714239,"total_server_duration_us":397091,"operation_id":"0xcbe8","timeout_ms":30000,"last_server_duration_us":397091,"total_duration_us":17716621},{"operation_name":"get","last_dispatch_duration_us":15772417,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":15772417,"total_server_duration_us":43,"operation_id":"0xccc4","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":15772627},{"operation_name":"upsert","last_dispatch_duration_us":8697132,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9560048,"total_server_duration_us":321748,"encode_duration_us":17693,"operation_id":"0xcf5a","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10011745},{"operation_name":"upsert","last_dispatch_duration_us":8629806,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13743914,"total_server_duration_us":329720,"encode_duration_us":35618,"operation_id":"0xcdbf","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10009380},{"operation_name":"upsert","last_dispatch_duration_us":6362571,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":6362571,"total_server_duration_us":35815,"encode_duration_us":16788,"operation_id":"0xcfe9","timeout_ms":10000,"last_server_duration_us":35815,"total_duration_us":10005880},{"operation_name":"upsert","last_dispatch_duration_us":8569439,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13425444,"total_server_duration_us":343437,"encode_duration_us":18189,"operation_id":"0xcdde","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10002145},{"operation_name":"upsert","last_dispatch_duration_us":31487,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":696181,"total_server_duration_us":298240,"encode_duration_us":17170,"operation_id":"0xcf95","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10001546},{"operation_name":"get","last_dispatch_duration_us":9483671,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9483671,"total_server_duration_us":126,"operation_id":"0xceac","timeout_ms":30000,"last_server_duration_us":126,"total_duration_us":9483930},{"operation_name":"get","last_dispatch_duration_us":9472294,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9472294,"total_server_duration_us":56,"operation_id":"0xceb3","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9472514}],"total_count":32}} INFO [2022-03-13 17:15:48,175] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,175] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:48,217] jepsen worker 52 - jepsen.util 592 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,217] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,219] jepsen worker 52 - jepsen.util 592 :invoke :write [17 1] INFO [2022-03-13 17:15:48,220] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] WARN [2022-03-13 17:15:48,274] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17622240,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17622240,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9034183,"encode_duration_us":16862,"operation_id":"0xcc40","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10005014},{"operation_name":"upsert","last_dispatch_duration_us":21671241,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21671241,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9043735,"encode_duration_us":18271,"operation_id":"0xcb18","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10005128}],"total_count":2}} WARN [2022-03-13 17:15:48,274] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10385583,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10385583,"total_server_duration_us":43,"operation_id":"0xd043","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":10386006},{"operation_name":"get","last_dispatch_duration_us":10384247,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":10384247,"total_server_duration_us":15,"operation_id":"0xd045","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":10384451},{"operation_name":"upsert","last_dispatch_duration_us":34602,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8271849,"total_server_duration_us":378506,"encode_duration_us":17161,"operation_id":"0xd006","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10007995},{"operation_name":"upsert","last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","encode_duration_us":25165,"operation_id":"0xd049","timeout_ms":10000,"total_duration_us":10007057},{"operation_name":"upsert","last_dispatch_duration_us":9054686,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11719103,"total_server_duration_us":9222027,"encode_duration_us":22196,"operation_id":"0xcf69","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10005966},{"operation_name":"upsert","last_dispatch_duration_us":14424230,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":14424230,"total_server_duration_us":194167,"encode_duration_us":17559,"operation_id":"0xceea","timeout_ms":10000,"last_server_duration_us":194167,"total_duration_us":10005827},{"operation_name":"upsert","last_dispatch_duration_us":8917211,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9457745,"total_server_duration_us":203073,"encode_duration_us":22753,"operation_id":"0xd044","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10001375},{"operation_name":"upsert","last_dispatch_duration_us":52138,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1905313,"total_server_duration_us":221371,"encode_duration_us":16928,"operation_id":"0xcfbb","timeout_ms":10000,"last_server_duration_us":24497,"total_duration_us":10001338},{"operation_name":"upsert","last_dispatch_duration_us":9045998,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9350035,"total_server_duration_us":9156169,"encode_duration_us":18987,"operation_id":"0xcf1a","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":9442938},{"operation_name":"upsert","last_dispatch_duration_us":8025505,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":8025505,"total_server_duration_us":28036,"encode_duration_us":19236,"operation_id":"0xceab","timeout_ms":10000,"last_server_duration_us":28036,"total_duration_us":8029314}],"total_count":16}} WARN [2022-03-13 17:15:48,274] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8362013,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16980930,"total_server_duration_us":230893,"encode_duration_us":36141,"operation_id":"0xcdbe","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10006040},{"operation_name":"upsert","last_dispatch_duration_us":72224,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8246484,"total_server_duration_us":368398,"encode_duration_us":17418,"operation_id":"0xcd6f","timeout_ms":10000,"last_server_duration_us":26241,"total_duration_us":10005666},{"operation_name":"upsert","last_dispatch_duration_us":8586165,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8894149,"total_server_duration_us":98037,"encode_duration_us":17322,"operation_id":"0xce8b","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10005297},{"operation_name":"upsert","last_dispatch_duration_us":8338801,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16922085,"total_server_duration_us":284166,"encode_duration_us":25238,"operation_id":"0xcdbc","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10005243},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":21506,"operation_id":"0xcff4","timeout_ms":10000,"total_duration_us":10004206},{"operation_name":"upsert","last_dispatch_duration_us":55833,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8980284,"total_server_duration_us":232241,"encode_duration_us":20656,"operation_id":"0xd09f","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10001777},{"operation_name":"upsert","last_dispatch_duration_us":9102242,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9354087,"total_server_duration_us":9080259,"encode_duration_us":25518,"operation_id":"0xd069","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":9377815},{"operation_name":"upsert","last_dispatch_duration_us":9115103,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9146602,"total_server_duration_us":9055131,"encode_duration_us":17400,"operation_id":"0xcfb8","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":9161311},{"operation_name":"upsert","last_dispatch_duration_us":7106598,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":7106598,"total_server_duration_us":1098518,"encode_duration_us":17891,"operation_id":"0xd138","timeout_ms":10000,"last_server_duration_us":1098518,"total_duration_us":7126432},{"operation_name":"upsert","last_dispatch_duration_us":6778570,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6778570,"total_server_duration_us":380936,"encode_duration_us":20264,"operation_id":"0xd086","timeout_ms":10000,"last_server_duration_us":380936,"total_duration_us":6782240}],"total_count":40}} WARN [2022-03-13 17:15:48,278] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14780446,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":14780446,"total_server_duration_us":6432,"operation_id":"0xcd60","timeout_ms":30000,"last_server_duration_us":6432,"total_duration_us":14782418},{"operation_name":"get","last_dispatch_duration_us":14653980,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":14653980,"total_server_duration_us":56,"operation_id":"0xcdc5","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":14654715},{"operation_name":"upsert","last_dispatch_duration_us":11094769,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":11094769,"total_server_duration_us":21568,"encode_duration_us":17637,"operation_id":"0xcf06","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10010926},{"operation_name":"upsert","last_dispatch_duration_us":169391,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9693553,"total_server_duration_us":143588,"encode_duration_us":18009,"operation_id":"0xcfc3","timeout_ms":10000,"last_server_duration_us":33721,"total_duration_us":10010517},{"operation_name":"get","last_dispatch_duration_us":8908362,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8908362,"total_server_duration_us":6977,"operation_id":"0xcf68","timeout_ms":30000,"last_server_duration_us":6977,"total_duration_us":9992431},{"operation_name":"get","last_dispatch_duration_us":8017792,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8017792,"total_server_duration_us":38,"operation_id":"0xd007","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":8020605},{"operation_name":"get","last_dispatch_duration_us":5462874,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5462874,"total_server_duration_us":32,"operation_id":"0xd0eb","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":5463233},{"operation_name":"upsert","last_dispatch_duration_us":4041871,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4041871,"total_server_duration_us":2496206,"encode_duration_us":17898,"operation_id":"0xd13c","timeout_ms":10000,"last_server_duration_us":2496206,"total_duration_us":4047962},{"operation_name":"get","last_dispatch_duration_us":3321121,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3321121,"total_server_duration_us":38,"operation_id":"0xd168","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":3321294},{"operation_name":"get","last_dispatch_duration_us":3214324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":3214324,"total_server_duration_us":32,"operation_id":"0xd126","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3214641}],"total_count":30}} INFO [2022-03-13 17:15:48,304] jepsen worker 43 - jepsen.util 583 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,307] jepsen worker 43 - jepsen.util 583 :invoke :write [14 17] INFO [2022-03-13 17:15:48,307] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,308] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:48,310] jepsen worker 14 - jepsen.util 374 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,310] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,311] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:15:48,311] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:15:48,327] jepsen worker 72 - jepsen.util 3312 :info :write [24 41] :RequestTimeoutException INFO [2022-03-13 17:15:48,327] jepsen worker 72 - jepsen.util 3402 :invoke :read [24 nil] INFO [2022-03-13 17:15:48,333] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,334] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:48,336] jepsen worker 37 - jepsen.util 2107 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,337] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 1] INFO [2022-03-13 17:15:48,382] jepsen worker 44 - jepsen.util 584 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,382] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,382] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:48,383] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] WARN [2022-03-13 17:15:48,479] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16494713,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18236665,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9231296,"encode_duration_us":19283,"operation_id":"0xccb6","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10001994},{"operation_name":"upsert","last_dispatch_duration_us":15912218,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16559400,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9361604,"encode_duration_us":19290,"operation_id":"0xcc42","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10009342},{"operation_name":"upsert","last_dispatch_duration_us":15994986,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17900706,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9293290,"encode_duration_us":25909,"operation_id":"0xcc78","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10007470}],"total_count":3}} INFO [2022-03-13 17:15:48,508] jepsen worker 13 - jepsen.util 643 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,508] jepsen worker 13 - jepsen.util 643 :invoke :write [4 9] INFO [2022-03-13 17:15:48,539] jepsen worker 33 - jepsen.util 753 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,540] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,539] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,540] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,540] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,541] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:15:48,541] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:48,542] jepsen worker 80 - jepsen.util 800 :invoke :write [26 43] INFO [2022-03-13 17:15:48,542] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:48,543] jepsen worker 51 - jepsen.util 771 :invoke :write [17 38] INFO [2022-03-13 17:15:48,546] jepsen worker 34 - jepsen.util 844 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,547] jepsen worker 34 - jepsen.util 844 :invoke :write [11 37] INFO [2022-03-13 17:15:48,547] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,548] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:15:48,550] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,550] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:48,558] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,559] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:48,563] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,563] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:48,566] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,567] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:15:48,596] jepsen worker 79 - jepsen.util 1069 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,597] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:48,613] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,614] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:15:48,615] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,615] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,617] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:15:48,618] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:48,619] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,620] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:48,622] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,623] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:48,624] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,625] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:15:48,643] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,645] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,645] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:15:48,646] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:48,728] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,728] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:48,729] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,730] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:15:48,736] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,737] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:48,756] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,756] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:15:48,793] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,817] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 10] INFO [2022-03-13 17:15:48,877] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:48,877] jepsen worker nemesis - couchbase.util Rebalance check count: 457 INFO [2022-03-13 17:15:48,877] jepsen worker nemesis - couchbase.util Rebalance stuck count: 357 INFO [2022-03-13 17:15:48,931] jepsen worker 52 - jepsen.util 592 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,932] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:48,936] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,936] jepsen worker 70 - jepsen.util 790 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,936] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:48,937] jepsen worker 70 - jepsen.util 790 :invoke :write [23 37] INFO [2022-03-13 17:15:48,939] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,940] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:48,940] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,941] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:48,948] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,948] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,948] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,948] jepsen worker 14 - jepsen.util 374 :invoke :write [4 5] INFO [2022-03-13 17:15:48,949] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:15:48,949] jepsen worker 33 - jepsen.util 753 :invoke :write [11 45] INFO [2022-03-13 17:15:48,959] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,959] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 28] INFO [2022-03-13 17:15:48,962] jepsen worker 37 - jepsen.util 2107 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,963] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:48,974] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:48,975] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 34] INFO [2022-03-13 17:15:49,005] jepsen worker 7 - jepsen.util 637 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,005] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:49,005] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,006] jepsen worker 53 - jepsen.util 503 :invoke :write [17 8] INFO [2022-03-13 17:15:49,007] jepsen worker 7 - jepsen.util 637 :invoke :write [2 29] INFO [2022-03-13 17:15:49,007] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:49,034] jepsen worker 13 - jepsen.util 643 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,035] jepsen worker 13 - jepsen.util 643 :invoke :write [4 21] INFO [2022-03-13 17:15:49,040] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,040] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:15:49,144] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,144] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:49,145] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,146] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:15:49,150] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,151] jepsen worker 22 - jepsen.util 652 :invoke :write [7 32] INFO [2022-03-13 17:15:49,160] jepsen worker 66 - jepsen.util 3126 :info :write [22 30] :RequestTimeoutException INFO [2022-03-13 17:15:49,161] jepsen worker 66 - jepsen.util 3216 :invoke :read [22 nil] INFO [2022-03-13 17:15:49,170] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,171] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:49,172] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,173] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:15:49,174] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,174] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:49,176] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,177] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:49,207] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,207] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,207] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,208] jepsen worker 77 - jepsen.util 707 :invoke :write [25 22] INFO [2022-03-13 17:15:49,208] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:49,209] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:49,210] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,210] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:15:49,218] jepsen worker 34 - jepsen.util 844 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,218] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,219] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:49,219] jepsen worker 52 - jepsen.util 592 :invoke :write [17 27] INFO [2022-03-13 17:15:49,223] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,224] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,224] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:15:49,225] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:49,225] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,226] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 33] INFO [2022-03-13 17:15:49,250] jepsen worker 86 - jepsen.util 3056 :info :write [28 49] :RequestTimeoutException INFO [2022-03-13 17:15:49,252] jepsen worker 86 - jepsen.util 3146 :invoke :write [28 33] INFO [2022-03-13 17:15:49,258] jepsen worker 80 - jepsen.util 800 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,258] jepsen worker 3 - jepsen.util 1983 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,259] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:49,259] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:15:49,355] jepsen worker 6 - jepsen.util 636 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,355] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,356] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:49,357] jepsen worker 26 - jepsen.util 476 :invoke :write [8 30] INFO [2022-03-13 17:15:49,372] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,372] jepsen worker 89 - jepsen.util 2519 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,373] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:49,374] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:49,374] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,375] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,375] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:49,376] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:49,376] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,377] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,379] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:15:49,380] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 30] INFO [2022-03-13 17:15:49,380] jepsen worker 88 - jepsen.util 2068 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,381] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:49,393] jepsen worker 13 - jepsen.util 643 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,393] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,393] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:49,394] jepsen worker 13 - jepsen.util 643 :invoke :write [4 13] INFO [2022-03-13 17:15:49,417] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,417] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:15:49,476] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,477] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:49,478] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,478] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:49,480] jepsen worker 37 - jepsen.util 2107 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,481] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:49,487] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,487] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:15:49,498] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,498] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:15:49,547] jepsen worker 69 - jepsen.util 699 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,548] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:49,555] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,555] jepsen worker 51 - jepsen.util 771 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,556] jepsen worker 21 - jepsen.util 831 :invoke :write [7 21] INFO [2022-03-13 17:15:49,558] jepsen worker 51 - jepsen.util 771 :invoke :write [17 15] INFO [2022-03-13 17:15:49,595] jepsen worker 8 - jepsen.util 728 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,595] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,596] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:49,596] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:15:49,597] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,598] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:49,608] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,608] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,610] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:49,611] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:15:49,630] jepsen worker 89 - jepsen.util 2519 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,630] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,631] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 8] INFO [2022-03-13 17:15:49,635] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:49,649] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,650] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:49,714] jepsen worker 9 - jepsen.util 2799 :info :write [3 14] :RequestTimeoutException INFO [2022-03-13 17:15:49,715] jepsen worker 9 - jepsen.util 2889 :invoke :read [3 nil] INFO [2022-03-13 17:15:49,720] jepsen worker 31 - jepsen.util 3451 :info :write [10 3] :RequestTimeoutException INFO [2022-03-13 17:15:49,721] jepsen worker 31 - jepsen.util 3541 :invoke :write [10 11] INFO [2022-03-13 17:15:49,726] jepsen worker 29 - jepsen.util 3449 :info :write [9 45] :RequestTimeoutException INFO [2022-03-13 17:15:49,726] jepsen worker 29 - jepsen.util 3539 :invoke :write [9 10] INFO [2022-03-13 17:15:49,766] jepsen worker 22 - jepsen.util 652 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,767] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:15:49,774] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,774] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,774] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:49,777] jepsen worker 14 - jepsen.util 374 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,777] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:49,777] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,778] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:49,778] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,778] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:49,778] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,779] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:49,779] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:15:49,779] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,780] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:49,785] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,786] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,787] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,788] jepsen worker 61 - jepsen.util 691 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,789] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,789] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:49,790] jepsen worker 61 - jepsen.util 691 :invoke :write [20 27] INFO [2022-03-13 17:15:49,791] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,791] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:15:49,792] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:15:49,792] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:15:49,793] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:49,882] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:49,882] jepsen worker nemesis - couchbase.util Rebalance check count: 458 INFO [2022-03-13 17:15:49,882] jepsen worker nemesis - couchbase.util Rebalance stuck count: 358 INFO [2022-03-13 17:15:49,890] jepsen worker 83 - jepsen.util 2693 :info :write [27 39] :RequestTimeoutException INFO [2022-03-13 17:15:49,890] jepsen worker 83 - jepsen.util 2783 :invoke :write [27 23] INFO [2022-03-13 17:15:49,968] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:49,968] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:50,016] jepsen worker 43 - jepsen.util 583 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,016] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,016] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:50,017] jepsen worker 43 - jepsen.util 583 :invoke :write [14 18] INFO [2022-03-13 17:15:50,018] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 28] INFO [2022-03-13 17:15:50,018] jepsen worker 42 - jepsen.util 402 :invoke :write [14 1] INFO [2022-03-13 17:15:50,023] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,024] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:50,026] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,026] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,026] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,027] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:15:50,031] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,031] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:15:50,034] jepsen worker 46 - jepsen.util 3286 :info :write [15 35] :RequestTimeoutException INFO [2022-03-13 17:15:50,034] jepsen worker 46 - jepsen.util 3376 :invoke :read [15 nil] INFO [2022-03-13 17:15:50,084] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,084] jepsen worker 75 - jepsen.util 435 :invoke :write [25 17] INFO [2022-03-13 17:15:50,090] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,090] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,091] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:15:50,091] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:50,109] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,109] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:15:50,115] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,116] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:15:50,122] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,122] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,123] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:15:50,123] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:50,159] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,159] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,160] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:15:50,160] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:15:50,168] jepsen worker 33 - jepsen.util 753 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,168] jepsen worker 33 - jepsen.util 753 :invoke :write [11 1] INFO [2022-03-13 17:15:50,179] jepsen worker 76 - jepsen.util 706 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,179] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:50,180] jepsen worker 76 - jepsen.util 706 :invoke :write [25 31] INFO [2022-03-13 17:15:50,180] jepsen worker 44 - jepsen.util 584 :invoke :write [14 31] INFO [2022-03-13 17:15:50,187] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,188] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,189] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,190] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:15:50,236] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,237] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,241] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,242] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,242] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,242] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,242] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:50,243] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:50,246] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,247] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:15:50,259] jepsen worker 37 - jepsen.util 2107 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,260] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,260] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 29] INFO [2022-03-13 17:15:50,261] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:15:50,264] jepsen worker 53 - jepsen.util 503 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,265] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:50,313] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,314] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,375] jepsen worker 75 - jepsen.util 435 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,375] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,375] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,375] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,375] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,376] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:50,376] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:15:50,377] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:50,377] jepsen worker 24 - jepsen.util 564 :invoke :write [8 14] INFO [2022-03-13 17:15:50,378] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:15:50,382] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,383] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:50,384] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,384] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:50,385] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,386] jepsen worker 75 - jepsen.util 435 :invoke :write [25 11] INFO [2022-03-13 17:15:50,421] jepsen worker 21 - jepsen.util 831 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,423] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:50,438] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,439] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,441] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,441] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,442] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:50,442] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:15:50,449] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,450] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:15:50,460] jepsen worker 50 - jepsen.util 3290 :ok :read [16 :nil] INFO [2022-03-13 17:15:50,461] jepsen worker 50 - jepsen.util 3290 :invoke :read [16 nil] INFO [2022-03-13 17:15:50,513] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,513] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,515] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,515] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:15:50,525] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,525] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,533] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,533] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:50,580] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,580] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,581] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 8] INFO [2022-03-13 17:15:50,581] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:50,594] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,594] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,595] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:50,595] jepsen worker 12 - jepsen.util 732 :invoke :write [4 11] INFO [2022-03-13 17:15:50,613] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,614] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:50,644] jepsen worker 51 - jepsen.util 771 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,644] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,644] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,645] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,645] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:15:50,646] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:50,646] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,647] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:15:50,648] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,648] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:15:50,649] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,651] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:15:50,659] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,660] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:50,691] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,691] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,693] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,693] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:15:50,795] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,796] jepsen worker 7 - jepsen.util 637 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,796] jepsen worker 6 - jepsen.util 636 :invoke :write [2 8] INFO [2022-03-13 17:15:50,797] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:15:50,799] jepsen worker 24 - jepsen.util 564 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,800] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:50,820] jepsen worker 52 - jepsen.util 592 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,820] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,821] jepsen worker 34 - jepsen.util 844 :invoke :write [11 44] INFO [2022-03-13 17:15:50,821] jepsen worker 52 - jepsen.util 592 :invoke :write [17 3] INFO [2022-03-13 17:15:50,844] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,844] jepsen worker 33 - jepsen.util 753 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,845] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:50,845] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:15:50,860] jepsen worker 37 - jepsen.util 2107 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,860] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,860] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,861] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:15:50,862] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:50,862] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,863] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,863] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:15:50,865] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,865] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:50,869] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,869] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,870] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,871] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:15:50,872] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 3] INFO [2022-03-13 17:15:50,877] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:50,878] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,879] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:15:50,888] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:50,888] jepsen worker nemesis - couchbase.util Rebalance check count: 459 INFO [2022-03-13 17:15:50,888] jepsen worker nemesis - couchbase.util Rebalance stuck count: 359 INFO [2022-03-13 17:15:50,896] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,896] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 36] INFO [2022-03-13 17:15:50,925] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,926] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,927] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,928] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:50,929] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,930] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:15:50,956] jepsen worker 65 - jepsen.util 3755 :info :write [21 25] :RequestTimeoutException INFO [2022-03-13 17:15:50,956] jepsen worker 65 - jepsen.util 3845 :invoke :read [21 nil] INFO [2022-03-13 17:15:50,996] jepsen worker 75 - jepsen.util 435 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,996] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,996] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,996] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:50,997] jepsen worker 71 - jepsen.util 521 :invoke :write [23 24] INFO [2022-03-13 17:15:50,997] jepsen worker 24 - jepsen.util 564 :invoke :write [8 36] INFO [2022-03-13 17:15:50,997] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,997] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,998] jepsen worker 70 - jepsen.util 790 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,998] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:50,998] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,998] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:50,999] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:50,999] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:50,999] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:50,999] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:15:51,047] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,047] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,048] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,048] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,049] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:15:51,049] jepsen worker 51 - jepsen.util 771 :invoke :write [17 41] INFO [2022-03-13 17:15:51,052] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:51,052] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:51,062] jepsen worker 89 - jepsen.util 2519 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,062] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,063] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:51,064] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:51,080] jepsen worker 24 - jepsen.util 564 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,081] jepsen worker 24 - jepsen.util 564 :invoke :write [8 5] INFO [2022-03-13 17:15:51,087] jepsen worker 26 - jepsen.util 476 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,087] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,088] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:15:51,089] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:51,105] jepsen worker 3 - jepsen.util 1983 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,106] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 28] INFO [2022-03-13 17:15:51,123] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,123] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:51,147] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,147] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,148] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:51,149] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:15:51,200] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,200] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,200] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:15:51,201] jepsen worker 33 - jepsen.util 753 :invoke :write [11 14] INFO [2022-03-13 17:15:51,228] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,229] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,229] jepsen worker 80 - jepsen.util 800 :invoke :write [26 27] INFO [2022-03-13 17:15:51,229] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,230] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:15:51,231] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:51,259] jepsen worker 37 - jepsen.util 2107 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,259] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,259] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 4] INFO [2022-03-13 17:15:51,260] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:15:51,268] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,269] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,269] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:15:51,278] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:15:51,311] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,312] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:51,357] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,357] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:15:51,422] jepsen worker 71 - jepsen.util 521 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,423] jepsen worker 77 - jepsen.util 707 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,423] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,424] jepsen worker 69 - jepsen.util 699 :invoke :write [23 27] INFO [2022-03-13 17:15:51,424] jepsen worker 77 - jepsen.util 707 :invoke :write [25 31] INFO [2022-03-13 17:15:51,425] jepsen worker 56 - jepsen.util 3386 :ok :read [18 21] INFO [2022-03-13 17:15:51,428] jepsen worker 56 - jepsen.util 3386 :invoke :read [18 nil] INFO [2022-03-13 17:15:51,433] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:15:51,435] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,436] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:15:51,471] jepsen worker 3 - jepsen.util 1983 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,472] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:51,473] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,473] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:51,474] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,475] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,475] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,475] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,475] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:15:51,483] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:15:51,484] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:15:51,486] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:15:51,499] jepsen worker 51 - jepsen.util 771 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,500] jepsen worker 51 - jepsen.util 771 :invoke :write [17 26] INFO [2022-03-13 17:15:51,512] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,513] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:51,518] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,519] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:15:51,619] jepsen worker 56 - jepsen.util 3386 :ok :read [18 21] INFO [2022-03-13 17:15:51,620] jepsen worker 56 - jepsen.util 3386 :invoke :write [18 45] INFO [2022-03-13 17:15:51,624] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,624] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:51,626] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,626] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:51,642] jepsen worker 41 - jepsen.util 3371 :ok :read [13 39] INFO [2022-03-13 17:15:51,642] jepsen worker 41 - jepsen.util 3371 :invoke :read [13 nil] INFO [2022-03-13 17:15:51,665] jepsen worker 72 - jepsen.util 3402 :ok :read [24 24] INFO [2022-03-13 17:15:51,666] jepsen worker 72 - jepsen.util 3402 :invoke :read [24 nil] INFO [2022-03-13 17:15:51,674] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,675] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:51,685] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,685] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:51,687] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,687] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:51,689] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,690] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:15:51,699] jepsen worker 61 - jepsen.util 691 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,700] jepsen worker 61 - jepsen.util 691 :invoke :write [20 45] INFO [2022-03-13 17:15:51,735] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,735] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,736] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:51,737] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:51,756] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,757] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:51,768] jepsen worker 72 - jepsen.util 3402 :ok :read [24 24] INFO [2022-03-13 17:15:51,768] jepsen worker 72 - jepsen.util 3402 :invoke :read [24 nil] INFO [2022-03-13 17:15:51,774] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,775] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:15:51,827] jepsen worker 24 - jepsen.util 564 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,828] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:51,839] jepsen worker 72 - jepsen.util 3402 :ok :read [24 24] INFO [2022-03-13 17:15:51,839] jepsen worker 72 - jepsen.util 3402 :invoke :read [24 nil] INFO [2022-03-13 17:15:51,855] jepsen worker 33 - jepsen.util 753 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,856] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:51,861] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,861] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,861] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,862] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:51,863] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:15:51,864] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,867] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:51,868] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:51,869] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,869] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,871] jepsen worker 71 - jepsen.util 521 :invoke :write [23 47] INFO [2022-03-13 17:15:51,875] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:15:51,892] jepsen worker 72 - jepsen.util 3402 :ok :read [24 24] INFO [2022-03-13 17:15:51,892] jepsen worker 72 - jepsen.util 3402 :invoke :write [24 21] INFO [2022-03-13 17:15:51,894] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:51,894] jepsen worker nemesis - couchbase.util Rebalance check count: 460 INFO [2022-03-13 17:15:51,894] jepsen worker nemesis - couchbase.util Rebalance stuck count: 360 INFO [2022-03-13 17:15:51,895] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,895] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:15:51,988] jepsen worker 85 - jepsen.util 2965 :info :write [28 37] :RequestTimeoutException INFO [2022-03-13 17:15:51,989] jepsen worker 85 - jepsen.util 3055 :invoke :write [28 4] INFO [2022-03-13 17:15:51,998] jepsen worker 67 - jepsen.util 2677 :info :write [22 32] :RequestTimeoutException INFO [2022-03-13 17:15:51,998] jepsen worker 40 - jepsen.util 3370 :info :write [13 3] :RequestTimeoutException INFO [2022-03-13 17:15:51,999] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,999] jepsen worker 37 - jepsen.util 2107 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:51,999] jepsen worker 67 - jepsen.util 2767 :invoke :write [22 14] INFO [2022-03-13 17:15:52,000] jepsen worker 40 - jepsen.util 3460 :invoke :read [13 nil] INFO [2022-03-13 17:15:52,000] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:52,001] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 16] INFO [2022-03-13 17:15:52,016] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,016] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,040] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,040] jepsen worker 42 - jepsen.util 402 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,040] jepsen worker 53 - jepsen.util 503 :invoke :write [17 26] INFO [2022-03-13 17:15:52,041] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,041] jepsen worker 42 - jepsen.util 402 :invoke :write [14 43] INFO [2022-03-13 17:15:52,043] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,044] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:52,045] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:52,046] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,046] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,046] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,047] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:15:52,047] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,048] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:15:52,048] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,049] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,050] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,051] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,052] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,052] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,054] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,054] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,055] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,057] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,057] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:52,059] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,059] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,059] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:15:52,062] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:15:52,062] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,064] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,065] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,078] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,079] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,082] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,082] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,084] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,084] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,088] jepsen worker 1 - jepsen.util 2701 :info :write [0 37] :RequestTimeoutException INFO [2022-03-13 17:15:52,089] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,089] jepsen worker 1 - jepsen.util 2791 :invoke :write [0 1] INFO [2022-03-13 17:15:52,091] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,098] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,099] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,103] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,104] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,107] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,107] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:15:52,109] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,110] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:15:52,126] jepsen worker 37 - jepsen.util 2107 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,126] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,126] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,128] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:52,129] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:15:52,130] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:15:52,132] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,133] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:15:52,162] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,164] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,203] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,203] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,205] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,205] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:15:52,209] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,209] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:15:52,246] jepsen worker 76 - jepsen.util 706 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,247] jepsen worker 76 - jepsen.util 706 :invoke :write [25 2] INFO [2022-03-13 17:15:52,270] jepsen worker 43 - jepsen.util 583 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,271] jepsen worker 43 - jepsen.util 583 :invoke :write [14 28] INFO [2022-03-13 17:15:52,274] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,275] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,278] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,278] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:15:52,291] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,292] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:15:52,307] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,307] jepsen worker 44 - jepsen.util 584 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,308] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:15:52,308] jepsen worker 44 - jepsen.util 584 :invoke :write [14 26] INFO [2022-03-13 17:15:52,327] jepsen worker 79 - jepsen.util 1069 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,327] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,327] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 14] INFO [2022-03-13 17:15:52,328] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:15:52,338] jepsen worker 17 - jepsen.util 3167 :info :write [5 19] :RequestTimeoutException INFO [2022-03-13 17:15:52,339] jepsen worker 17 - jepsen.util 3257 :invoke :write [5 4] INFO [2022-03-13 17:15:52,366] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,366] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,366] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:15:52,367] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:52,372] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,372] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:15:52,389] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,389] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,391] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,391] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,439] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,440] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,441] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,441] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,448] jepsen worker 12 - jepsen.util 732 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,448] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,448] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,448] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,448] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,449] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:52,450] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:52,450] jepsen worker 78 - jepsen.util 618 :invoke :write [26 46] INFO [2022-03-13 17:15:52,451] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:52,451] jepsen worker 60 - jepsen.util 600 :invoke :write [20 15] INFO [2022-03-13 17:15:52,459] jepsen worker 80 - jepsen.util 800 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,460] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:52,466] jepsen worker 13 - jepsen.util 643 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,467] jepsen worker 13 - jepsen.util 643 :invoke :read [4 nil] INFO [2022-03-13 17:15:52,480] jepsen worker 63 - jepsen.util 3303 :info :write [21 2] :RequestTimeoutException INFO [2022-03-13 17:15:52,481] jepsen worker 63 - jepsen.util 3393 :invoke :read [21 nil] INFO [2022-03-13 17:15:52,488] jepsen worker 6 - jepsen.util 636 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,488] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,488] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,488] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,489] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:52,490] jepsen worker 6 - jepsen.util 636 :invoke :write [2 29] INFO [2022-03-13 17:15:52,490] jepsen worker 26 - jepsen.util 476 :invoke :write [8 25] INFO [2022-03-13 17:15:52,491] jepsen worker 62 - jepsen.util 512 :invoke :write [20 41] INFO [2022-03-13 17:15:52,619] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,620] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,623] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,623] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,623] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,625] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:52,625] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:52,625] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:15:52,626] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,627] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,628] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,628] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,631] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,631] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 30] INFO [2022-03-13 17:15:52,635] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,635] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,635] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,636] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,637] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,638] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:15:52,650] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,650] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,651] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:15:52,651] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,653] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,653] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,654] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,655] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:15:52,664] jepsen worker 71 - jepsen.util 521 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,665] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:52,666] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,667] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,668] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,669] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,669] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,670] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,671] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:15:52,671] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:15:52,715] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,716] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,723] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,725] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:15:52,726] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,727] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:52,728] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,729] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:15:52,748] jepsen worker 2 - jepsen.util 3782 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:15:52,749] jepsen worker 2 - jepsen.util 3872 :invoke :write [0 4] INFO [2022-03-13 17:15:52,762] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,762] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,828] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,828] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,829] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,830] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:15:52,835] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,835] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,835] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,835] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:15:52,836] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:15:52,836] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:15:52,839] jepsen worker 37 - jepsen.util 2107 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,840] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 26] INFO [2022-03-13 17:15:52,883] jepsen worker 77 - jepsen.util 707 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,885] jepsen worker 77 - jepsen.util 707 :invoke :write [25 20] INFO [2022-03-13 17:15:52,891] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,892] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:52,893] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,894] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:15:52,899] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:52,899] jepsen worker nemesis - couchbase.util Rebalance check count: 461 INFO [2022-03-13 17:15:52,899] jepsen worker nemesis - couchbase.util Rebalance stuck count: 361 INFO [2022-03-13 17:15:52,900] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,901] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:15:52,909] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,909] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,909] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,910] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:52,911] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:15:52,911] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:15:52,929] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,929] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:52,936] jepsen worker 11 - jepsen.util 3251 :info :write [3 15] :RequestTimeoutException INFO [2022-03-13 17:15:52,937] jepsen worker 11 - jepsen.util 3341 :invoke :write [3 38] INFO [2022-03-13 17:15:52,946] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,947] jepsen worker 30 - jepsen.util 3090 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:15:52,948] jepsen worker 30 - jepsen.util 3180 :invoke :read [10 nil] INFO [2022-03-13 17:15:52,948] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:52,958] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,959] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:15:52,972] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,972] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:52,972] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 0] INFO [2022-03-13 17:15:52,972] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,057] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,057] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,057] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,058] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,059] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:53,059] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:15:53,062] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,062] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,062] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:15:53,063] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,063] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:53,063] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,064] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,065] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:53,066] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,066] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:15:53,072] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,072] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,073] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,074] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:15:53,081] jepsen worker 42 - jepsen.util 402 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,082] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:53,083] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,084] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:53,122] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,122] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,126] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,127] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:15:53,132] jepsen worker 61 - jepsen.util 691 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,133] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:53,170] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,170] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,176] jepsen worker 37 - jepsen.util 2107 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,176] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,177] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 41] INFO [2022-03-13 17:15:53,177] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:15:53,179] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,180] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:53,186] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,186] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:15:53,270] jepsen worker 20 - jepsen.util 3620 :info :write [6 20] :RequestTimeoutException INFO [2022-03-13 17:15:53,271] jepsen worker 20 - jepsen.util 3710 :invoke :write [6 32] INFO [2022-03-13 17:15:53,284] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,284] jepsen worker 53 - jepsen.util 503 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,284] jepsen worker 53 - jepsen.util 503 :invoke :write [17 18] INFO [2022-03-13 17:15:53,285] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:15:53,295] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,296] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:53,304] jepsen worker 39 - jepsen.util 3369 :info :write [13 44] :RequestTimeoutException INFO [2022-03-13 17:15:53,305] jepsen worker 39 - jepsen.util 3459 :invoke :read [13 nil] INFO [2022-03-13 17:15:53,318] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,319] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,320] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,320] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,321] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,322] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:15:53,329] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,330] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:15:53,333] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,334] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:15:53,342] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,343] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:15:53,401] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,401] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:15:53,405] jepsen worker 89 - jepsen.util 2519 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,405] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,405] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:15:53,406] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:53,408] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,408] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:15:53,414] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,415] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,496] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,496] jepsen worker 76 - jepsen.util 706 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,496] jepsen worker 23 - jepsen.util 563 :invoke :write [7 20] INFO [2022-03-13 17:15:53,497] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:53,504] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,504] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,504] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:53,505] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:53,511] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,511] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,511] jepsen worker 51 - jepsen.util 771 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,511] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,511] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,511] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,512] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,513] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:15:53,514] jepsen worker 80 - jepsen.util 800 :invoke :write [26 48] INFO [2022-03-13 17:15:53,516] jepsen worker 21 - jepsen.util 831 :invoke :write [7 9] INFO [2022-03-13 17:15:53,517] jepsen worker 51 - jepsen.util 771 :invoke :write [17 4] INFO [2022-03-13 17:15:53,518] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:15:53,519] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:53,521] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,521] jepsen worker 33 - jepsen.util 753 :invoke :write [11 6] INFO [2022-03-13 17:15:53,522] jepsen worker 37 - jepsen.util 2107 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,522] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:15:53,523] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:53,531] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,532] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,538] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 3] INFO [2022-03-13 17:15:53,540] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,558] jepsen worker 43 - jepsen.util 583 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,558] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,559] jepsen worker 43 - jepsen.util 583 :invoke :write [14 26] INFO [2022-03-13 17:15:53,560] jepsen worker 7 - jepsen.util 637 :invoke :write [2 42] INFO [2022-03-13 17:15:53,600] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,600] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,600] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,601] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:53,601] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:15:53,602] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:53,605] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,605] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,607] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:15:53,607] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:53,608] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,609] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:15:53,620] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,621] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:15:53,621] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,622] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:15:53,624] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,624] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,625] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:53,625] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:53,668] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,668] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,669] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:15:53,669] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 45] INFO [2022-03-13 17:15:53,700] jepsen worker 88 - jepsen.util 2068 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,700] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,700] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,701] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:15:53,703] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:15:53,703] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:53,705] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,705] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:15:53,715] jepsen worker 80 - jepsen.util 800 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,715] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,716] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:53,716] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 48] INFO [2022-03-13 17:15:53,719] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,719] jepsen worker 44 - jepsen.util 584 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,719] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,720] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:53,720] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:53,721] jepsen worker 44 - jepsen.util 584 :invoke :write [14 49] INFO [2022-03-13 17:15:53,730] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,734] jepsen worker 82 - jepsen.util 3232 :info :write [27 24] :SyncWriteAmbiguous INFO [2022-03-13 17:15:53,735] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:15:53,736] jepsen worker 82 - jepsen.util 3322 :invoke :read [27 nil] INFO [2022-03-13 17:15:53,789] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,790] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:15:53,812] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,823] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:53,882] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,883] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:15:53,891] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,891] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:15:53,898] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,899] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:15:53,904] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:53,904] jepsen worker nemesis - couchbase.util Rebalance check count: 462 INFO [2022-03-13 17:15:53,904] jepsen worker nemesis - couchbase.util Rebalance stuck count: 362 INFO [2022-03-13 17:15:53,918] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,919] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:15:53,936] jepsen worker 51 - jepsen.util 771 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,937] jepsen worker 51 - jepsen.util 771 :invoke :write [17 48] INFO [2022-03-13 17:15:53,990] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,991] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:15:53,998] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:53,999] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:15:54,047] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,048] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,095] jepsen worker 89 - jepsen.util 2519 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,095] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,095] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 35] INFO [2022-03-13 17:15:54,096] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:54,102] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,102] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,105] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,105] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:15:54,125] jepsen worker 52 - jepsen.util 592 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,126] jepsen worker 52 - jepsen.util 592 :invoke :read [17 nil] INFO [2022-03-13 17:15:54,126] jepsen worker 33 - jepsen.util 753 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,127] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:54,132] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,133] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:54,135] jepsen worker 79 - jepsen.util 1069 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,135] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,136] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 45] INFO [2022-03-13 17:15:54,136] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:15:54,167] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,168] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:54,204] jepsen worker 21 - jepsen.util 831 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,204] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:15:54,273] jepsen worker 58 - jepsen.util 3118 :ok :read [19 16] INFO [2022-03-13 17:15:54,274] jepsen worker 58 - jepsen.util 3118 :invoke :write [19 15] INFO [2022-03-13 17:15:54,296] jepsen worker 28 - jepsen.util 3448 :info :write [9 22] :RequestTimeoutException INFO [2022-03-13 17:15:54,296] jepsen worker 28 - jepsen.util 3538 :invoke :read [9 nil] INFO [2022-03-13 17:15:54,336] jepsen worker 78 - jepsen.util 618 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,337] jepsen worker 78 - jepsen.util 618 :invoke :write [26 13] INFO [2022-03-13 17:15:54,350] jepsen worker 54 - jepsen.util 2934 :info :write [18 5] :RequestTimeoutException INFO [2022-03-13 17:15:54,351] jepsen worker 54 - jepsen.util 3024 :invoke :read [18 nil] INFO [2022-03-13 17:15:54,356] jepsen worker 59 - jepsen.util 3209 :info :write [19 2] :RequestTimeoutException INFO [2022-03-13 17:15:54,357] jepsen worker 59 - jepsen.util 3299 :invoke :write [19 30] INFO [2022-03-13 17:15:54,361] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,362] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:54,364] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,364] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:54,486] jepsen worker 48 - jepsen.util 3198 :info :write [16 22] :RequestTimeoutException INFO [2022-03-13 17:15:54,487] jepsen worker 48 - jepsen.util 3288 :invoke :write [16 6] INFO [2022-03-13 17:15:54,536] jepsen worker 68 - jepsen.util 3488 :info :write [22 27] :RequestTimeoutException INFO [2022-03-13 17:15:54,537] jepsen worker 68 - jepsen.util 3578 :invoke :write [22 0] INFO [2022-03-13 17:15:54,555] jepsen worker 51 - jepsen.util 771 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,556] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:15:54,578] jepsen worker 6 - jepsen.util 636 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,579] jepsen worker 6 - jepsen.util 636 :invoke :write [2 27] INFO [2022-03-13 17:15:54,600] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,601] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,601] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,602] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:15:54,602] jepsen worker 24 - jepsen.util 564 :invoke :write [8 3] INFO [2022-03-13 17:15:54,603] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:54,604] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,604] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:15:54,606] jepsen worker 81 - jepsen.util 3681 :info :write [27 19] :RequestTimeoutException INFO [2022-03-13 17:15:54,607] jepsen worker 81 - jepsen.util 3771 :invoke :write [27 20] INFO [2022-03-13 17:15:54,613] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,613] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,613] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,613] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,614] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:15:54,615] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:54,616] jepsen worker 80 - jepsen.util 800 :invoke :write [26 34] INFO [2022-03-13 17:15:54,616] jepsen worker 33 - jepsen.util 753 :invoke :write [11 40] INFO [2022-03-13 17:15:54,626] jepsen worker 49 - jepsen.util 3199 :info :write [16 41] :RequestTimeoutException INFO [2022-03-13 17:15:54,627] jepsen worker 49 - jepsen.util 3289 :invoke :read [16 nil] INFO [2022-03-13 17:15:54,650] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,652] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:15:54,663] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,663] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:54,664] jepsen worker 37 - jepsen.util 2107 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,665] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 6] INFO [2022-03-13 17:15:54,690] jepsen worker 18 - jepsen.util 3168 :info :write [6 6] :RequestTimeoutException INFO [2022-03-13 17:15:54,691] jepsen worker 18 - jepsen.util 3258 :invoke :read [6 nil] INFO [2022-03-13 17:15:54,722] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,723] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,725] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,725] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:15:54,738] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,738] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,739] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:54,739] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,769] jepsen worker 49 - jepsen.util 3289 :ok :read [16 :nil] INFO [2022-03-13 17:15:54,769] jepsen worker 49 - jepsen.util 3289 :invoke :read [16 nil] INFO [2022-03-13 17:15:54,770] jepsen worker 49 - jepsen.util 3289 :ok :read [16 :nil] INFO [2022-03-13 17:15:54,771] jepsen worker 49 - jepsen.util 3289 :invoke :write [16 8] INFO [2022-03-13 17:15:54,789] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,789] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,789] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,790] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:15:54,791] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:15:54,791] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 33] INFO [2022-03-13 17:15:54,792] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,793] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,794] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,795] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:15:54,800] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,801] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,801] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:54,801] jepsen worker 60 - jepsen.util 600 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,801] jepsen worker 61 - jepsen.util 691 :invoke :write [20 6] INFO [2022-03-13 17:15:54,801] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,801] jepsen worker 60 - jepsen.util 600 :invoke :write [20 36] INFO [2022-03-13 17:15:54,802] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:54,807] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,807] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,807] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,808] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:54,809] jepsen worker 51 - jepsen.util 771 :invoke :write [17 28] INFO [2022-03-13 17:15:54,810] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:54,825] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,826] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,831] jepsen worker 33 - jepsen.util 753 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,836] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:54,839] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,839] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,841] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,842] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:15:54,850] jepsen worker 24 - jepsen.util 564 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,851] jepsen worker 24 - jepsen.util 564 :invoke :write [8 36] INFO [2022-03-13 17:15:54,861] jepsen worker 80 - jepsen.util 800 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,862] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:54,874] jepsen worker 89 - jepsen.util 2519 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,874] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,874] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:54,875] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:15:54,878] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,878] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:54,879] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,879] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:15:54,890] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,892] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:15:54,897] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,898] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:54,900] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,901] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,901] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,901] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,901] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,902] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:54,903] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:54,904] jepsen worker 21 - jepsen.util 831 :invoke :write [7 14] INFO [2022-03-13 17:15:54,904] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:15:54,906] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:15:54,915] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:54,915] jepsen worker nemesis - couchbase.util Rebalance check count: 463 INFO [2022-03-13 17:15:54,915] jepsen worker nemesis - couchbase.util Rebalance stuck count: 363 INFO [2022-03-13 17:15:54,955] jepsen worker 37 - jepsen.util 2107 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,955] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,956] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:54,956] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 10] INFO [2022-03-13 17:15:54,957] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,957] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:54,958] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,959] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 39] INFO [2022-03-13 17:15:54,966] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,967] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,968] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,969] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,969] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,970] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,974] jepsen worker 4 - jepsen.util 2254 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,974] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,983] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,984] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,987] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,988] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:54,989] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,991] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:15:54,996] jepsen worker 26 - jepsen.util 476 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:54,997] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:55,015] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,016] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:15:55,033] jepsen worker 47 - jepsen.util 3197 :info :write [15 36] :RequestTimeoutException INFO [2022-03-13 17:15:55,034] jepsen worker 47 - jepsen.util 3287 :invoke :read [15 nil] INFO [2022-03-13 17:15:55,052] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,053] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:15:55,066] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,067] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,069] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,069] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 18] INFO [2022-03-13 17:15:55,117] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,118] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,118] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:55,122] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:15:55,144] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,145] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,158] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,158] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,161] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,161] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,162] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,163] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:15:55,168] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,169] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:15:55,214] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,215] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:15:55,233] jepsen worker 37 - jepsen.util 2107 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,234] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 21] INFO [2022-03-13 17:15:55,246] jepsen worker 3 - jepsen.util 1983 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,247] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:15:55,294] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,294] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:15:55,320] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,321] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:15:55,325] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,325] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,325] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:55,326] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,366] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,366] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,366] jepsen worker 38 - jepsen.util 2468 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,367] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,367] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:15:55,368] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 0] INFO [2022-03-13 17:15:55,368] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,368] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:15:55,422] jepsen worker 37 - jepsen.util 2107 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,423] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 42] INFO [2022-03-13 17:15:55,426] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,427] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,435] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,436] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:15:55,474] jepsen worker 19 - jepsen.util 2989 :ok :read [6 16] INFO [2022-03-13 17:15:55,475] jepsen worker 19 - jepsen.util 2989 :invoke :read [6 nil] INFO [2022-03-13 17:15:55,477] jepsen worker 0 - jepsen.util 3150 :info :write [0 32] :RequestTimeoutException INFO [2022-03-13 17:15:55,477] jepsen worker 0 - jepsen.util 3240 :invoke :write [0 1] INFO [2022-03-13 17:15:55,483] jepsen worker 55 - jepsen.util 3205 :ok :read [18 21] INFO [2022-03-13 17:15:55,484] jepsen worker 55 - jepsen.util 3205 :invoke :write [18 46] INFO [2022-03-13 17:15:55,488] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,489] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:55,507] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,507] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,508] jepsen worker 77 - jepsen.util 707 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,508] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,508] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,509] jepsen worker 77 - jepsen.util 707 :invoke :write [25 20] INFO [2022-03-13 17:15:55,510] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:15:55,510] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,516] jepsen worker 51 - jepsen.util 771 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,516] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,516] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,516] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:15:55,517] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,517] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,517] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:55,517] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,518] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:55,518] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:55,519] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,519] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:55,520] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,521] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,522] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,523] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,524] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,524] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,525] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,526] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:15:55,537] jepsen worker 62 - jepsen.util 512 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,537] jepsen worker 62 - jepsen.util 512 :invoke :write [20 5] INFO [2022-03-13 17:15:55,583] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,584] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:15:55,594] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,595] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:15:55,599] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,599] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:15:55,618] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,618] jepsen worker 38 - jepsen.util 2468 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,619] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:15:55,619] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:55,624] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,624] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,624] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,625] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:15:55,627] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,627] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:15:55,629] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,629] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,630] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,631] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,632] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,632] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,634] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,634] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:15:55,648] jepsen worker 16 - jepsen.util 3166 :ok :read [5 43] INFO [2022-03-13 17:15:55,649] jepsen worker 16 - jepsen.util 3166 :invoke :write [5 11] INFO [2022-03-13 17:15:55,651] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,651] jepsen worker 23 - jepsen.util 563 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,651] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,651] jepsen worker 24 - jepsen.util 564 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,651] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,651] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:55,652] jepsen worker 23 - jepsen.util 563 :invoke :write [7 19] INFO [2022-03-13 17:15:55,653] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,653] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:55,655] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:55,657] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:15:55,664] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,664] jepsen worker 21 - jepsen.util 831 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,664] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,664] jepsen worker 71 - jepsen.util 521 :invoke :write [23 34] INFO [2022-03-13 17:15:55,665] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,665] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,666] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:55,667] jepsen worker 51 - jepsen.util 771 :invoke :write [17 6] INFO [2022-03-13 17:15:55,667] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,668] jepsen worker 14 - jepsen.util 374 :invoke :write [4 46] INFO [2022-03-13 17:15:55,672] jepsen worker 80 - jepsen.util 800 :invoke :write [26 17] INFO [2022-03-13 17:15:55,672] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:15:55,673] jepsen worker 33 - jepsen.util 753 :invoke :write [11 4] INFO [2022-03-13 17:15:55,679] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,683] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:55,685] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,685] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 23] INFO [2022-03-13 17:15:55,737] jepsen worker 82 - jepsen.util 3322 :ok :read [27 4] INFO [2022-03-13 17:15:55,738] jepsen worker 82 - jepsen.util 3322 :invoke :write [27 7] INFO [2022-03-13 17:15:55,741] jepsen worker 43 - jepsen.util 583 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,743] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,743] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,743] jepsen worker 43 - jepsen.util 583 :invoke :write [14 48] INFO [2022-03-13 17:15:55,745] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:55,745] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:55,754] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,754] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:15:55,780] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,780] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,781] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:55,783] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:15:55,790] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,791] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:15:55,826] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,827] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:15:55,830] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,830] jepsen worker 71 - jepsen.util 521 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,830] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,831] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:55,831] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:55,832] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:55,834] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,835] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:15:55,836] jepsen worker 37 - jepsen.util 2107 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,836] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,836] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:55,836] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,837] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] INFO [2022-03-13 17:15:55,837] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,838] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:15:55,838] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:55,838] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,839] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:15:55,839] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,840] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:15:55,843] jepsen worker 44 - jepsen.util 584 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,844] jepsen worker 44 - jepsen.util 584 :invoke :write [14 48] INFO [2022-03-13 17:15:55,867] jepsen worker 3 - jepsen.util 1983 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,868] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 13] INFO [2022-03-13 17:15:55,870] jepsen worker 80 - jepsen.util 800 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,872] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:15:55,887] jepsen worker 64 - jepsen.util 3214 :ok :read [21 :nil] INFO [2022-03-13 17:15:55,887] jepsen worker 27 - jepsen.util 3717 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,888] jepsen worker 64 - jepsen.util 3214 :invoke :read [21 nil] INFO [2022-03-13 17:15:55,891] jepsen worker 27 - jepsen.util 3717 :invoke :read [9 nil] INFO [2022-03-13 17:15:55,899] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,900] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:55,907] jepsen worker 73 - jepsen.util 3493 :ok :read [24 24] INFO [2022-03-13 17:15:55,907] jepsen worker 73 - jepsen.util 3493 :invoke :read [24 nil] INFO [2022-03-13 17:15:55,915] jepsen worker 74 - jepsen.util 3584 :ok :read [24 24] INFO [2022-03-13 17:15:55,916] jepsen worker 74 - jepsen.util 3584 :invoke :read [24 nil] INFO [2022-03-13 17:15:55,921] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:55,921] jepsen worker nemesis - couchbase.util Rebalance check count: 464 INFO [2022-03-13 17:15:55,921] jepsen worker nemesis - couchbase.util Rebalance stuck count: 364 INFO [2022-03-13 17:15:55,927] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,928] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:15:55,934] jepsen worker 7 - jepsen.util 637 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,935] jepsen worker 7 - jepsen.util 637 :invoke :write [2 46] INFO [2022-03-13 17:15:55,948] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,948] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,948] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:15:55,949] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 20] INFO [2022-03-13 17:15:55,961] jepsen worker 33 - jepsen.util 753 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,961] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:55,963] jepsen worker 30 - jepsen.util 3180 :ok :read [10 :nil] INFO [2022-03-13 17:15:55,964] jepsen worker 30 - jepsen.util 3180 :invoke :read [10 nil] INFO [2022-03-13 17:15:55,967] jepsen worker 47 - jepsen.util 3287 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,967] jepsen worker 64 - jepsen.util 3214 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,967] jepsen worker 27 - jepsen.util 3717 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,968] jepsen worker 47 - jepsen.util 3287 :invoke :read [15 nil] INFO [2022-03-13 17:15:55,968] jepsen worker 64 - jepsen.util 3214 :invoke :read [21 nil] INFO [2022-03-13 17:15:55,969] jepsen worker 27 - jepsen.util 3717 :invoke :write [9 40] INFO [2022-03-13 17:15:55,970] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,971] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:55,971] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:55,972] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:15:56,007] jepsen worker 3 - jepsen.util 1983 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,008] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 18] INFO [2022-03-13 17:15:56,021] jepsen worker 30 - jepsen.util 3180 :ok :read [10 :nil] INFO [2022-03-13 17:15:56,021] jepsen worker 30 - jepsen.util 3180 :invoke :read [10 nil] INFO [2022-03-13 17:15:56,032] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,033] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:15:56,042] jepsen worker 74 - jepsen.util 3584 :ok :read [24 24] INFO [2022-03-13 17:15:56,042] jepsen worker 74 - jepsen.util 3584 :invoke :write [24 27] INFO [2022-03-13 17:15:56,071] jepsen worker 18 - jepsen.util 3258 :ok :read [6 16] INFO [2022-03-13 17:15:56,071] jepsen worker 54 - jepsen.util 3024 :ok :read [18 21] INFO [2022-03-13 17:15:56,071] jepsen worker 18 - jepsen.util 3258 :invoke :read [6 nil] INFO [2022-03-13 17:15:56,072] jepsen worker 54 - jepsen.util 3024 :invoke :write [18 13] INFO [2022-03-13 17:15:56,081] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,082] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:15:56,095] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,095] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,095] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:15:56,096] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,105] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,105] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,148] jepsen worker 3 - jepsen.util 1983 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,148] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:15:56,151] jepsen worker 78 - jepsen.util 618 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,152] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:56,157] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,157] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:15:56,159] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,160] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,168] jepsen worker 73 - jepsen.util 3493 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,169] jepsen worker 73 - jepsen.util 3493 :invoke :read [24 nil] INFO [2022-03-13 17:15:56,173] jepsen worker 14 - jepsen.util 374 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,175] jepsen worker 14 - jepsen.util 374 :invoke :write [4 6] INFO [2022-03-13 17:15:56,187] jepsen worker 19 - jepsen.util 2989 :ok :read [6 16] INFO [2022-03-13 17:15:56,188] jepsen worker 19 - jepsen.util 2989 :invoke :write [6 34] INFO [2022-03-13 17:15:56,191] jepsen worker 64 - jepsen.util 3214 :ok :read [21 :nil] INFO [2022-03-13 17:15:56,192] jepsen worker 64 - jepsen.util 3214 :invoke :read [21 nil] INFO [2022-03-13 17:15:56,201] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,202] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,202] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:56,202] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:15:56,212] jepsen worker 47 - jepsen.util 3287 :ok :read [15 48] INFO [2022-03-13 17:15:56,213] jepsen worker 47 - jepsen.util 3287 :invoke :read [15 nil] INFO [2022-03-13 17:15:56,215] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,215] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,216] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 8] INFO [2022-03-13 17:15:56,216] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:15:56,227] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,228] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,249] jepsen worker 6 - jepsen.util 636 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,250] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:56,250] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:56,250] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,251] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:56,251] jepsen worker 8 - jepsen.util 728 :invoke :write [2 35] INFO [2022-03-13 17:15:56,259] jepsen worker 30 - jepsen.util 3180 :ok :read [10 :nil] INFO [2022-03-13 17:15:56,261] jepsen worker 30 - jepsen.util 3180 :invoke :write [10 15] INFO [2022-03-13 17:15:56,281] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,281] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,283] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,283] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,284] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,284] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,285] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,285] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,286] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,285] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,285] jepsen worker 37 - jepsen.util 2107 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,286] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:15:56,287] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,287] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,287] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 42] INFO [2022-03-13 17:15:56,288] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,289] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,289] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,289] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,289] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,290] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,290] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,290] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:15:56,291] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,291] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:15:56,292] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,296] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,297] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 9] INFO [2022-03-13 17:15:56,299] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,300] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 14] INFO [2022-03-13 17:15:56,316] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,316] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,364] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,365] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:15:56,370] jepsen worker 32 - jepsen.util 3362 :info :write [10 11] :RequestTimeoutException INFO [2022-03-13 17:15:56,371] jepsen worker 32 - jepsen.util 3452 :invoke :write [10 43] INFO [2022-03-13 17:15:56,376] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,376] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:56,377] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,377] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:56,414] jepsen worker 61 - jepsen.util 691 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,415] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:56,423] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,424] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,426] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,426] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,428] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,428] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,429] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,430] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,431] jepsen worker 51 - jepsen.util 771 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,431] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,432] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:15:56,432] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:15:56,433] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,433] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:15:56,476] jepsen worker 88 - jepsen.util 2068 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,476] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 27] INFO [2022-03-13 17:15:56,502] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,503] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,505] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,505] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 19] INFO [2022-03-13 17:15:56,524] jepsen worker 89 - jepsen.util 2519 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,524] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,551] jepsen worker 46 - jepsen.util 3376 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,552] jepsen worker 46 - jepsen.util 3376 :invoke :write [15 29] INFO [2022-03-13 17:15:56,563] jepsen worker 64 - jepsen.util 3214 :ok :read [21 :nil] INFO [2022-03-13 17:15:56,564] jepsen worker 64 - jepsen.util 3214 :invoke :read [21 nil] INFO [2022-03-13 17:15:56,567] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,567] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,568] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,568] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,579] jepsen worker 79 - jepsen.util 1069 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,580] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 8] INFO [2022-03-13 17:15:56,580] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,581] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:56,584] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,584] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,584] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:56,584] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,584] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,585] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:56,586] jepsen worker 24 - jepsen.util 564 :invoke :write [8 29] INFO [2022-03-13 17:15:56,586] jepsen worker 71 - jepsen.util 521 :invoke :write [23 12] INFO [2022-03-13 17:15:56,589] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,590] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:56,591] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,592] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:15:56,594] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,595] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:15:56,596] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15775411,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15775411,"total_server_duration_us":27043,"operation_id":"0xd122","timeout_ms":30000,"last_server_duration_us":27043,"total_duration_us":15778011},{"operation_name":"get","last_dispatch_duration_us":15680486,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15680486,"total_server_duration_us":19496,"operation_id":"0xd12c","timeout_ms":30000,"last_server_duration_us":19496,"total_duration_us":15682651},{"operation_name":"upsert","last_dispatch_duration_us":11214958,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11214958,"total_server_duration_us":80398,"encode_duration_us":20183,"operation_id":"0xd23c","timeout_ms":10000,"last_server_duration_us":80398,"total_duration_us":10007580},{"operation_name":"upsert","last_dispatch_duration_us":72259,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":465688,"total_server_duration_us":172878,"encode_duration_us":16839,"operation_id":"0xd0f9","timeout_ms":10000,"last_server_duration_us":29137,"total_duration_us":10004236},{"operation_name":"upsert","last_dispatch_duration_us":8807739,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9285014,"total_server_duration_us":218673,"encode_duration_us":18982,"operation_id":"0xd0f7","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10003077},{"operation_name":"get","last_dispatch_duration_us":9537180,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9537180,"total_server_duration_us":312571,"operation_id":"0xd2be","timeout_ms":30000,"last_server_duration_us":312571,"total_duration_us":9539904},{"operation_name":"upsert","last_dispatch_duration_us":4608575,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":4608575,"total_server_duration_us":548944,"encode_duration_us":28090,"operation_id":"0xd1f8","timeout_ms":10000,"last_server_duration_us":548944,"total_duration_us":4612527},{"operation_name":"upsert","last_dispatch_duration_us":4464826,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":4464826,"total_server_duration_us":536748,"encode_duration_us":16843,"operation_id":"0xd1ed","timeout_ms":10000,"last_server_duration_us":536748,"total_duration_us":4469110},{"operation_name":"upsert","last_dispatch_duration_us":1979392,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1979392,"total_server_duration_us":149387,"encode_duration_us":25235,"operation_id":"0xd4bf","timeout_ms":10000,"last_server_duration_us":149387,"total_duration_us":1985116},{"operation_name":"upsert","last_dispatch_duration_us":1833510,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1833510,"total_server_duration_us":375432,"encode_duration_us":20176,"operation_id":"0xd4aa","timeout_ms":10000,"last_server_duration_us":375432,"total_duration_us":1838332}],"total_count":80}} WARN [2022-03-13 17:15:56,596] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17692091,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24656898,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9305813,"encode_duration_us":33231,"operation_id":"0xcdbd","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10005894},{"operation_name":"upsert","last_dispatch_duration_us":9096156,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16627241,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9367987,"encode_duration_us":17518,"operation_id":"0xcdd9","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10008544}],"total_count":2}} INFO [2022-03-13 17:15:56,603] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,604] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:15:56,621] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,622] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:15:56,634] jepsen worker 18 - jepsen.util 3258 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,636] jepsen worker 18 - jepsen.util 3258 :invoke :read [6 nil] INFO [2022-03-13 17:15:56,646] jepsen worker 60 - jepsen.util 600 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,647] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,648] jepsen worker 76 - jepsen.util 706 :invoke :write [25 34] INFO [2022-03-13 17:15:56,648] jepsen worker 60 - jepsen.util 600 :invoke :write [20 35] INFO [2022-03-13 17:15:56,659] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,660] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:56,667] jepsen worker 47 - jepsen.util 3287 :ok :read [15 48] INFO [2022-03-13 17:15:56,669] jepsen worker 47 - jepsen.util 3287 :invoke :read [15 nil] INFO [2022-03-13 17:15:56,671] jepsen worker 88 - jepsen.util 2068 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,672] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,673] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,673] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,675] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 17] INFO [2022-03-13 17:15:56,675] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,679] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,679] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,680] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,681] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:15:56,681] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,681] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:15:56,693] jepsen worker 3 - jepsen.util 1983 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,694] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:15:56,698] jepsen worker 53 - jepsen.util 503 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,699] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:56,700] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,700] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,700] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,702] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:15:56,703] jepsen worker 36 - jepsen.util 2196 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,703] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,704] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:56,704] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:15:56,705] jepsen worker 80 - jepsen.util 800 :invoke :write [26 26] INFO [2022-03-13 17:15:56,710] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:15:56,714] jepsen worker 73 - jepsen.util 3493 :ok :read [24 24] INFO [2022-03-13 17:15:56,715] jepsen worker 73 - jepsen.util 3493 :invoke :write [24 32] INFO [2022-03-13 17:15:56,758] jepsen worker 37 - jepsen.util 2107 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,759] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 25] INFO [2022-03-13 17:15:56,766] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,766] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:56,791] jepsen worker 77 - jepsen.util 707 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,791] jepsen worker 77 - jepsen.util 707 :invoke :write [25 23] INFO [2022-03-13 17:15:56,798] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,799] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 9] INFO [2022-03-13 17:15:56,813] jepsen worker 71 - jepsen.util 521 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,815] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:56,818] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,818] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,819] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,820] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 43] INFO [2022-03-13 17:15:56,831] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,832] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,836] jepsen worker 14 - jepsen.util 374 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,836] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,837] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,836] jepsen worker 4 - jepsen.util 2254 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,837] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,836] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,837] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,838] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:15:56,838] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:15:56,839] jepsen worker 33 - jepsen.util 753 :invoke :write [11 37] INFO [2022-03-13 17:15:56,840] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:15:56,841] jepsen worker 14 - jepsen.util 374 :invoke :write [4 22] INFO [2022-03-13 17:15:56,846] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,846] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,847] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,848] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,849] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,850] jepsen worker 51 - jepsen.util 771 :invoke :write [17 5] INFO [2022-03-13 17:15:56,850] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,851] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,857] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,859] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:15:56,871] jepsen worker 62 - jepsen.util 512 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,872] jepsen worker 62 - jepsen.util 512 :invoke :write [20 30] INFO [2022-03-13 17:15:56,896] jepsen worker 45 - jepsen.util 3465 :info :write [15 13] :RequestTimeoutException INFO [2022-03-13 17:15:56,898] jepsen worker 45 - jepsen.util 3555 :invoke :read [15 nil] INFO [2022-03-13 17:15:56,912] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,913] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,915] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,916] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,917] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,917] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:56,925] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,926] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:15:56,926] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:56,926] jepsen worker nemesis - couchbase.util Rebalance check count: 465 INFO [2022-03-13 17:15:56,927] jepsen worker nemesis - couchbase.util Rebalance stuck count: 365 INFO [2022-03-13 17:15:56,935] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,936] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:56,939] jepsen worker 37 - jepsen.util 2107 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,939] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,940] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:56,940] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 14] INFO [2022-03-13 17:15:56,941] jepsen worker 64 - jepsen.util 3214 :ok :read [21 :nil] INFO [2022-03-13 17:15:56,941] jepsen worker 64 - jepsen.util 3214 :invoke :write [21 49] INFO [2022-03-13 17:15:56,980] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,981] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:15:56,988] jepsen worker 24 - jepsen.util 564 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,989] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:15:56,992] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:56,993] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:57,003] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,003] jepsen worker 80 - jepsen.util 800 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,004] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:15:57,005] jepsen worker 80 - jepsen.util 800 :invoke :write [26 40] INFO [2022-03-13 17:15:57,007] jepsen worker 18 - jepsen.util 3258 :ok :read [6 16] INFO [2022-03-13 17:15:57,008] jepsen worker 18 - jepsen.util 3258 :invoke :write [6 26] INFO [2022-03-13 17:15:57,034] jepsen worker 38 - jepsen.util 2468 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,037] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:57,046] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,046] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,046] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,047] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:15:57,048] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:57,048] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:15:57,054] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,054] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,054] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:15:57,055] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:57,056] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,057] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:15:57,061] jepsen worker 23 - jepsen.util 563 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,061] jepsen worker 23 - jepsen.util 563 :invoke :write [7 15] INFO [2022-03-13 17:15:57,075] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,076] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:57,082] jepsen worker 89 - jepsen.util 2519 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,083] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 33] INFO [2022-03-13 17:15:57,084] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,085] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:15:57,094] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,094] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,095] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:15:57,096] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:57,104] jepsen worker 47 - jepsen.util 3287 :ok :read [15 48] INFO [2022-03-13 17:15:57,104] jepsen worker 47 - jepsen.util 3287 :invoke :write [15 48] INFO [2022-03-13 17:15:57,137] jepsen worker 37 - jepsen.util 2107 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,138] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,140] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 10] INFO [2022-03-13 17:15:57,140] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 4] INFO [2022-03-13 17:15:57,167] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,168] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,169] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,169] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,170] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:15:57,170] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,203] jepsen worker 33 - jepsen.util 753 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,204] jepsen worker 33 - jepsen.util 753 :invoke :write [11 22] INFO [2022-03-13 17:15:57,216] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,217] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:15:57,219] jepsen worker 89 - jepsen.util 2519 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,220] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:57,229] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,230] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] INFO [2022-03-13 17:15:57,235] jepsen worker 43 - jepsen.util 583 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,237] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:57,240] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,241] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:15:57,285] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,286] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:57,298] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,298] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,299] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:15:57,299] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:15:57,315] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,315] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,320] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,321] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:15:57,341] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,341] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,342] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 30] INFO [2022-03-13 17:15:57,342] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:15:57,360] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,360] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,360] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,361] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:15:57,362] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:15:57,363] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:15:57,395] jepsen worker 38 - jepsen.util 2468 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,396] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:15:57,413] jepsen worker 10 - jepsen.util 3250 :info :write [3 31] :RequestTimeoutException INFO [2022-03-13 17:15:57,414] jepsen worker 10 - jepsen.util 3340 :invoke :write [3 41] INFO [2022-03-13 17:15:57,416] jepsen worker 34 - jepsen.util 844 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,416] jepsen worker 13 - jepsen.util 643 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,416] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,417] jepsen worker 13 - jepsen.util 643 :invoke :write [4 1] INFO [2022-03-13 17:15:57,417] jepsen worker 34 - jepsen.util 844 :invoke :read [11 nil] INFO [2022-03-13 17:15:57,417] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:15:57,427] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,427] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,427] jepsen worker 44 - jepsen.util 584 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,427] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,428] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:57,429] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:57,430] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:15:57,430] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:57,471] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,472] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:57,481] jepsen worker 37 - jepsen.util 2107 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,481] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:57,486] jepsen worker 14 - jepsen.util 374 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,486] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:15:57,521] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,522] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:57,526] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:57,526] jepsen worker 7 - jepsen.util 637 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,526] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,527] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,527] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:15:57,528] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:15:57,528] jepsen worker 6 - jepsen.util 636 :invoke :write [2 25] INFO [2022-03-13 17:15:57,528] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:15:57,541] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,542] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,547] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,551] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:15:57,609] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,609] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,610] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:57,610] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 31] INFO [2022-03-13 17:15:57,638] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,639] jepsen worker 21 - jepsen.util 831 :invoke :write [7 24] INFO [2022-03-13 17:15:57,642] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,643] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:57,648] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,649] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:57,653] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,653] jepsen worker 52 - jepsen.util 592 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,654] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:15:57,654] jepsen worker 52 - jepsen.util 592 :invoke :write [17 1] INFO [2022-03-13 17:15:57,656] jepsen worker 89 - jepsen.util 2519 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,656] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,656] jepsen worker 34 - jepsen.util 844 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,658] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:57,658] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,659] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 8] INFO [2022-03-13 17:15:57,659] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:15:57,661] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,661] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:15:57,663] jepsen worker 34 - jepsen.util 844 :invoke :write [11 10] INFO [2022-03-13 17:15:57,671] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,674] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:15:57,679] jepsen worker 69 - jepsen.util 699 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,680] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:15:57,728] jepsen worker 57 - jepsen.util 3387 :ok :read [19 16] INFO [2022-03-13 17:15:57,729] jepsen worker 57 - jepsen.util 3387 :invoke :write [19 6] INFO [2022-03-13 17:15:57,731] jepsen worker 80 - jepsen.util 800 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,731] jepsen worker 80 - jepsen.util 800 :invoke :write [26 6] INFO [2022-03-13 17:15:57,734] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,734] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,736] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:15:57,736] jepsen worker 24 - jepsen.util 564 :invoke :write [8 46] INFO [2022-03-13 17:15:57,737] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,738] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:15:57,752] jepsen worker 37 - jepsen.util 2107 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,753] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 28] INFO [2022-03-13 17:15:57,765] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,765] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:15:57,814] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,814] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,818] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,819] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:15:57,841] jepsen worker 8 - jepsen.util 728 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,841] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,842] jepsen worker 8 - jepsen.util 728 :invoke :write [2 22] INFO [2022-03-13 17:15:57,842] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:15:57,888] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,888] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,889] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:15:57,889] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,891] jepsen worker 33 - jepsen.util 753 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,892] jepsen worker 33 - jepsen.util 753 :invoke :write [11 15] INFO [2022-03-13 17:15:57,896] jepsen worker 24 - jepsen.util 564 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,897] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:15:57,927] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,929] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,930] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,931] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,932] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,932] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:57,932] jepsen worker nemesis - couchbase.util Rebalance check count: 466 INFO [2022-03-13 17:15:57,932] jepsen worker nemesis - couchbase.util Rebalance stuck count: 366 INFO [2022-03-13 17:15:57,933] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,935] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,936] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:15:57,957] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,957] jepsen worker 89 - jepsen.util 2519 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,958] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 25] INFO [2022-03-13 17:15:57,958] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] WARN [2022-03-13 17:15:57,989] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18039954,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18736136,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9369597,"encode_duration_us":17170,"operation_id":"0xcf95","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10001546},{"operation_name":"upsert","last_dispatch_duration_us":9135847,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15498419,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9075304,"encode_duration_us":16788,"operation_id":"0xcfe9","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10005880}],"total_count":2}} INFO [2022-03-13 17:15:57,990] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,990] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:57,991] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:57,992] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 34] INFO [2022-03-13 17:15:58,001] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,002] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,007] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,008] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,009] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,010] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:15:58,044] jepsen worker 60 - jepsen.util 600 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,044] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,044] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,045] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:58,045] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:58,046] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:15:58,059] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,059] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,060] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:15:58,062] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:15:58,111] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,112] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:15:58,136] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,136] jepsen worker 37 - jepsen.util 2107 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,139] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,139] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 6] INFO [2022-03-13 17:15:58,155] jepsen worker 88 - jepsen.util 2068 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,156] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:15:58,170] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,170] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] WARN [2022-03-13 17:15:58,192] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":26189638,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":26189638,"total_server_duration_us":49,"operation_id":"0xcbf1","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":26200227},{"operation_name":"get","last_dispatch_duration_us":8808045,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8808045,"total_server_duration_us":7541,"operation_id":"0xd05e","timeout_ms":30000,"last_server_duration_us":7541,"total_duration_us":13975358},{"operation_name":"upsert","last_dispatch_duration_us":31738,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8845747,"total_server_duration_us":216614,"encode_duration_us":20564,"operation_id":"0xd1f6","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10007933},{"operation_name":"upsert","last_dispatch_duration_us":8546480,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8658012,"total_server_duration_us":49977,"encode_duration_us":20786,"operation_id":"0xd129","timeout_ms":10000,"last_server_duration_us":26418,"total_duration_us":10007521},{"operation_name":"upsert","last_dispatch_duration_us":79547,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":199884,"total_server_duration_us":42780,"encode_duration_us":16822,"operation_id":"0xd2bf","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10005206},{"operation_name":"upsert","last_dispatch_duration_us":59331,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9387691,"total_server_duration_us":201311,"encode_duration_us":24672,"operation_id":"0xd1c2","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10004380},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":16671,"operation_id":"0xd135","timeout_ms":10000,"total_duration_us":10001951},{"operation_name":"upsert","last_dispatch_duration_us":73165,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":240963,"total_server_duration_us":41227,"encode_duration_us":19622,"operation_id":"0xd26b","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10001218},{"operation_name":"get","last_dispatch_duration_us":199020,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":199020,"total_server_duration_us":7856,"operation_id":"0xd353","timeout_ms":30000,"last_server_duration_us":7856,"total_duration_us":3323764},{"operation_name":"upsert","last_dispatch_duration_us":3214670,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3214670,"total_server_duration_us":197496,"encode_duration_us":21716,"operation_id":"0xd5a6","timeout_ms":10000,"last_server_duration_us":197496,"total_duration_us":3218281}],"total_count":26}} WARN [2022-03-13 17:15:58,197] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17606002,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17606002,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9064980,"encode_duration_us":21506,"operation_id":"0xcff4","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10004206},{"operation_name":"upsert","last_dispatch_duration_us":17597688,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24601536,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9303986,"encode_duration_us":17566,"operation_id":"0xcd1b","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10009521},{"operation_name":"upsert","last_dispatch_duration_us":16038111,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16038111,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9038428,"encode_duration_us":18114,"operation_id":"0xd13b","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10005638}],"total_count":3}} INFO [2022-03-13 17:15:58,242] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,243] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,245] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,246] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,246] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:15:58,247] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,264] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,264] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:58,270] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,270] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] WARN [2022-03-13 17:15:58,289] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17738239,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19643553,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9252370,"encode_duration_us":16928,"operation_id":"0xcfbb","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10001338},{"operation_name":"upsert","last_dispatch_duration_us":11689617,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11689617,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":77570,"encode_duration_us":25165,"operation_id":"0xd049","timeout_ms":10000,"last_server_duration_us":77570,"total_duration_us":10007057},{"operation_name":"upsert","last_dispatch_duration_us":9047421,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14062939,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9377636,"encode_duration_us":17353,"operation_id":"0xd12a","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10001782}],"total_count":3}} WARN [2022-03-13 17:15:58,290] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10283305,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10283305,"total_server_duration_us":6678,"operation_id":"0xd281","timeout_ms":30000,"last_server_duration_us":6678,"total_duration_us":10286483},{"operation_name":"upsert","last_dispatch_duration_us":32036,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1101524,"total_server_duration_us":328607,"encode_duration_us":21350,"operation_id":"0xd239","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10010406},{"operation_name":"upsert","last_dispatch_duration_us":34899,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":3855654,"total_server_duration_us":358187,"encode_duration_us":17302,"operation_id":"0xd175","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10007070},{"operation_name":"upsert","last_dispatch_duration_us":8654284,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8654284,"total_server_duration_us":35312,"encode_duration_us":18397,"operation_id":"0xd2e6","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":10003239},{"operation_name":"upsert","last_dispatch_duration_us":8680896,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":15011504,"total_server_duration_us":342026,"encode_duration_us":19499,"operation_id":"0xd0a7","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10002006},{"operation_name":"upsert","last_dispatch_duration_us":9047421,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":14062939,"total_server_duration_us":9377636,"encode_duration_us":17353,"operation_id":"0xd12a","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10001782},{"operation_name":"get","last_dispatch_duration_us":8538889,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":8538889,"total_server_duration_us":32,"operation_id":"0xd142","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8539092},{"operation_name":"upsert","last_dispatch_duration_us":2303776,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2303776,"total_server_duration_us":32645,"encode_duration_us":18803,"operation_id":"0xd2f1","timeout_ms":10000,"last_server_duration_us":32645,"total_duration_us":2308271},{"operation_name":"get","last_dispatch_duration_us":1982743,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1982743,"total_server_duration_us":13223,"operation_id":"0xd5ed","timeout_ms":30000,"last_server_duration_us":13223,"total_duration_us":1984905},{"operation_name":"get","last_dispatch_duration_us":1669204,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1669204,"total_server_duration_us":38,"operation_id":"0xd2ef","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":1669447}],"total_count":27}} WARN [2022-03-13 17:15:58,289] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16195450,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":16195450,"total_server_duration_us":323050,"operation_id":"0xd123","timeout_ms":30000,"last_server_duration_us":323050,"total_duration_us":16195895},{"operation_name":"get","last_dispatch_duration_us":12908457,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":12908457,"total_server_duration_us":62,"operation_id":"0xd1f2","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":12908784},{"operation_name":"upsert","last_dispatch_duration_us":108522,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9480963,"total_server_duration_us":174356,"encode_duration_us":17188,"operation_id":"0xd1a9","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10010637},{"operation_name":"upsert","last_dispatch_duration_us":106475,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9470582,"total_server_duration_us":154463,"encode_duration_us":17044,"operation_id":"0xd1a8","timeout_ms":10000,"last_server_duration_us":27222,"total_duration_us":10009466},{"operation_name":"upsert","last_dispatch_duration_us":178525,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":457736,"total_server_duration_us":60938,"encode_duration_us":16885,"operation_id":"0xd1ac","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10009076},{"operation_name":"upsert","last_dispatch_duration_us":141260,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":141260,"total_server_duration_us":20362,"encode_duration_us":23138,"operation_id":"0xd1e4","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10008499},{"operation_name":"upsert","last_dispatch_duration_us":107817,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9411649,"total_server_duration_us":130789,"encode_duration_us":26640,"operation_id":"0xd1aa","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10008006},{"operation_name":"upsert","last_dispatch_duration_us":115708,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9130104,"total_server_duration_us":86543,"encode_duration_us":17465,"operation_id":"0xd1b0","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10007083},{"operation_name":"get","last_dispatch_duration_us":2689924,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2689924,"total_server_duration_us":38,"operation_id":"0xd0d4","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9386984},{"operation_name":"upsert","last_dispatch_duration_us":3264380,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3264380,"total_server_duration_us":169390,"encode_duration_us":22029,"operation_id":"0xd128","timeout_ms":10000,"last_server_duration_us":169390,"total_duration_us":8437778}],"total_count":75}} INFO [2022-03-13 17:15:58,290] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,290] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,290] jepsen worker 38 - jepsen.util 2468 :fail :write [12 25] :ServerOutOfMemoryException WARN [2022-03-13 17:15:58,292] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18139992,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27050503,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9047197,"encode_duration_us":25344,"operation_id":"0xcbfc","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10001851},{"operation_name":"upsert","last_dispatch_duration_us":28531626,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":28531626,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9050106,"encode_duration_us":16648,"operation_id":"0xcb6e","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10008121}],"total_count":2}} INFO [2022-03-13 17:15:58,293] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:58,293] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,294] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] WARN [2022-03-13 17:15:58,298] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":25810,"operation_id":"0xd1f9","timeout_ms":10000,"total_duration_us":10009579},{"operation_name":"upsert","last_dispatch_duration_us":16038111,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16038111,"total_server_duration_us":9038428,"encode_duration_us":18114,"operation_id":"0xd13b","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10005638},{"operation_name":"upsert","last_dispatch_duration_us":8598847,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16332646,"total_server_duration_us":257312,"encode_duration_us":16831,"operation_id":"0xd127","timeout_ms":10000,"last_server_duration_us":83703,"total_duration_us":10005139},{"operation_name":"upsert","last_dispatch_duration_us":8684057,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8735651,"total_server_duration_us":114519,"encode_duration_us":17370,"operation_id":"0xd276","timeout_ms":10000,"last_server_duration_us":93195,"total_duration_us":10003959},{"operation_name":"upsert","last_dispatch_duration_us":38760,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9637880,"total_server_duration_us":381716,"encode_duration_us":16825,"operation_id":"0xd305","timeout_ms":10000,"last_server_duration_us":26418,"total_duration_us":10003873},{"operation_name":"get","last_dispatch_duration_us":6515311,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6515311,"total_server_duration_us":93195,"operation_id":"0xd411","timeout_ms":30000,"last_server_duration_us":93195,"total_duration_us":6515693},{"operation_name":"upsert","last_dispatch_duration_us":1280306,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1280306,"total_server_duration_us":41552,"encode_duration_us":18512,"operation_id":"0xd59f","timeout_ms":10000,"last_server_duration_us":41552,"total_duration_us":3394378},{"operation_name":"get","last_dispatch_duration_us":1575631,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1575631,"total_server_duration_us":27,"operation_id":"0xd590","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":3167005},{"operation_name":"upsert","last_dispatch_duration_us":3017815,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3017815,"total_server_duration_us":21163,"encode_duration_us":19402,"operation_id":"0xd53e","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":3025732},{"operation_name":"upsert","last_dispatch_duration_us":2544697,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2544697,"total_server_duration_us":132923,"encode_duration_us":17341,"operation_id":"0xd2ae","timeout_ms":10000,"last_server_duration_us":132923,"total_duration_us":2552166}],"total_count":66}} WARN [2022-03-13 17:15:58,300] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":66942,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9286417,"total_server_duration_us":320098,"encode_duration_us":17729,"operation_id":"0xd283","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10010958},{"operation_name":"upsert","last_dispatch_duration_us":121428,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1080415,"total_server_duration_us":147650,"encode_duration_us":20324,"operation_id":"0xd23d","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10008561},{"operation_name":"upsert","last_dispatch_duration_us":190078,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9544475,"total_server_duration_us":257509,"encode_duration_us":19474,"operation_id":"0xd245","timeout_ms":10000,"last_server_duration_us":38482,"total_duration_us":10006638},{"operation_name":"upsert","last_dispatch_duration_us":65254,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1052164,"total_server_duration_us":557145,"encode_duration_us":23897,"operation_id":"0xd1f0","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10005713},{"operation_name":"upsert","last_dispatch_duration_us":162942,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1026276,"total_server_duration_us":121069,"encode_duration_us":18575,"operation_id":"0xd255","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10003241},{"operation_name":"upsert","last_dispatch_duration_us":126426,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9282724,"total_server_duration_us":184242,"encode_duration_us":19988,"operation_id":"0xd26a","timeout_ms":10000,"last_server_duration_us":37241,"total_duration_us":10003027},{"operation_name":"upsert","last_dispatch_duration_us":149197,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1128662,"total_server_duration_us":148138,"encode_duration_us":16701,"operation_id":"0xd263","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10002543},{"operation_name":"upsert","last_dispatch_duration_us":9033350,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10082514,"total_server_duration_us":9492463,"encode_duration_us":18781,"operation_id":"0xd1f1","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10002310},{"operation_name":"upsert","last_dispatch_duration_us":9152699,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9152699,"total_server_duration_us":9047983,"encode_duration_us":21987,"operation_id":"0xd25c","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9156579},{"operation_name":"get","last_dispatch_duration_us":162125,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":162125,"total_server_duration_us":7698,"operation_id":"0xd2e5","timeout_ms":30000,"last_server_duration_us":7698,"total_duration_us":7372523}],"total_count":20}} INFO [2022-03-13 17:15:58,301] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,301] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,303] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:15:58,304] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,305] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,306] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,312] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,313] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,313] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,314] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:15:58,322] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,323] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:58,391] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,391] jepsen worker 37 - jepsen.util 2107 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,391] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,391] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 23] INFO [2022-03-13 17:15:58,393] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,393] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,395] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,396] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 11] INFO [2022-03-13 17:15:58,397] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,397] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,397] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,397] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:15:58,398] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:15:58,398] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:15:58,401] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException WARN [2022-03-13 17:15:58,401] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9033350,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10082514,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9492463,"encode_duration_us":18781,"operation_id":"0xd1f1","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10002310}],"total_count":1}} INFO [2022-03-13 17:15:58,401] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:15:58,437] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,437] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,437] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,438] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:15:58,439] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,440] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:15:58,477] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,478] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:15:58,489] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,489] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:15:58,539] jepsen worker 38 - jepsen.util 2468 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,540] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,549] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,550] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:58,550] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,551] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:15:58,608] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,609] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,611] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,611] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:15:58,627] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,627] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,664] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,665] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:15:58,675] jepsen worker 37 - jepsen.util 2107 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,675] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,676] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:15:58,676] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,681] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,681] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,681] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,682] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:15:58,682] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:15:58,684] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,689] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,690] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:15:58,694] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:58,694] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:15:58,694] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,695] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:15:58,695] jepsen worker 7 - jepsen.util 637 :invoke :write [2 33] INFO [2022-03-13 17:15:58,696] jepsen worker 42 - jepsen.util 402 :invoke :write [14 39] INFO [2022-03-13 17:15:58,704] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,705] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,706] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,707] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,708] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,709] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:15:58,751] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,751] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,752] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 23] INFO [2022-03-13 17:15:58,752] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,771] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,771] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:15:58,786] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,787] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:58,839] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,864] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,867] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,906] jepsen worker 77 - jepsen.util 707 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,914] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,930] jepsen worker 77 - jepsen.util 707 :invoke :write [25 10] INFO [2022-03-13 17:15:58,930] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:15:58,914] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,916] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,930] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:58,930] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,918] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,929] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:15:58,930] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,929] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 34] INFO [2022-03-13 17:15:58,931] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,937] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,938] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:15:58,940] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:58,940] jepsen worker nemesis - couchbase.util Rebalance check count: 467 INFO [2022-03-13 17:15:58,940] jepsen worker nemesis - couchbase.util Rebalance stuck count: 367 INFO [2022-03-13 17:15:58,957] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,959] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,959] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,959] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,965] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,966] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,966] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,967] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,968] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,969] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,969] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:58,970] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:15:58,972] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,973] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:15:58,988] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,988] jepsen worker 6 - jepsen.util 636 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,989] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:15:58,990] jepsen worker 26 - jepsen.util 476 :invoke :write [8 38] INFO [2022-03-13 17:15:58,994] jepsen worker 36 - jepsen.util 2196 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,994] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,995] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:58,995] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 7] INFO [2022-03-13 17:15:58,996] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:58,998] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:15:59,039] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,039] jepsen worker 89 - jepsen.util 2519 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,040] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,040] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,041] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,041] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,041] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,042] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,046] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,046] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,047] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,047] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 1] INFO [2022-03-13 17:15:59,048] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,051] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,054] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,055] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,057] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,058] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,106] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,107] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:15:59,128] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,129] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,133] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,134] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,141] jepsen worker 37 - jepsen.util 2107 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,141] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,142] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,143] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:15:59,191] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,192] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,191] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,192] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,193] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,193] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,193] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,193] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,193] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,194] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,194] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,194] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,195] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,195] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,195] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,195] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,196] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,196] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,196] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,196] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,196] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,197] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:15:59,197] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,197] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:15:59,198] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:15:59,200] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,200] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:15:59,207] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:15:59,219] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,219] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,273] jepsen worker 89 - jepsen.util 2519 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,273] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,274] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,275] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,276] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,276] jepsen worker 86 - jepsen.util 3146 :info :write [28 33] :RequestTimeoutException INFO [2022-03-13 17:15:59,276] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,277] jepsen worker 86 - jepsen.util 3236 :invoke :write [28 21] INFO [2022-03-13 17:15:59,278] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,278] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,279] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,279] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:15:59,280] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,280] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 41] INFO [2022-03-13 17:15:59,293] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,294] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,307] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,307] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:15:59,308] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,309] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,309] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:15:59,309] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,309] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,309] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:15:59,309] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,309] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:15:59,310] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:15:59,311] jepsen worker 69 - jepsen.util 699 :invoke :write [23 14] INFO [2022-03-13 17:15:59,311] jepsen worker 70 - jepsen.util 790 :invoke :write [23 10] INFO [2022-03-13 17:15:59,311] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:15:59,314] jepsen worker 23 - jepsen.util 563 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,315] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,315] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:15:59,317] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:15:59,319] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,320] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:59,338] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,339] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 5] INFO [2022-03-13 17:15:59,340] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,341] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,344] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,344] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:15:59,350] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,350] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,350] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:15:59,351] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 5] INFO [2022-03-13 17:15:59,357] jepsen worker 8 - jepsen.util 728 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,357] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,358] jepsen worker 8 - jepsen.util 728 :invoke :write [2 7] INFO [2022-03-13 17:15:59,358] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:15:59,374] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,377] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:15:59,401] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,402] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:15:59,471] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,471] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,473] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,473] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,475] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,475] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,476] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,477] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,482] jepsen worker 89 - jepsen.util 2519 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,482] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,483] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 19] INFO [2022-03-13 17:15:59,483] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,502] jepsen worker 15 - jepsen.util 3345 :ok :read [5 43] INFO [2022-03-13 17:15:59,503] jepsen worker 15 - jepsen.util 3345 :invoke :read [5 nil] INFO [2022-03-13 17:15:59,505] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,505] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:15:59,532] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,532] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,534] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:15:59,535] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:15:59,541] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,542] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:15:59,546] jepsen worker 3 - jepsen.util 1983 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,547] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,547] jepsen worker 50 - jepsen.util 3290 :ok :read [16 :nil] INFO [2022-03-13 17:15:59,548] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,548] jepsen worker 50 - jepsen.util 3290 :invoke :write [16 7] INFO [2022-03-13 17:15:59,550] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,551] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,552] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,555] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:15:59,556] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:15:59,567] jepsen worker 76 - jepsen.util 706 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,568] jepsen worker 76 - jepsen.util 706 :invoke :write [25 19] INFO [2022-03-13 17:15:59,593] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,594] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,595] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,596] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,653] jepsen worker 37 - jepsen.util 2107 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,653] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 39] INFO [2022-03-13 17:15:59,664] jepsen worker 15 - jepsen.util 3345 :ok :read [5 43] INFO [2022-03-13 17:15:59,664] jepsen worker 15 - jepsen.util 3345 :invoke :read [5 nil] INFO [2022-03-13 17:15:59,676] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,677] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,678] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,679] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:15:59,685] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,686] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,715] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,716] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:15:59,722] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,723] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:15:59,740] jepsen worker 31 - jepsen.util 3541 :info :write [10 11] :RequestTimeoutException INFO [2022-03-13 17:15:59,741] jepsen worker 31 - jepsen.util 3631 :invoke :read [10 nil] INFO [2022-03-13 17:15:59,742] jepsen worker 31 - jepsen.util 3631 :ok :read [10 :nil] INFO [2022-03-13 17:15:59,743] jepsen worker 31 - jepsen.util 3631 :invoke :write [10 40] INFO [2022-03-13 17:15:59,748] jepsen worker 29 - jepsen.util 3539 :info :write [9 10] :RequestTimeoutException INFO [2022-03-13 17:15:59,748] jepsen worker 29 - jepsen.util 3629 :invoke :read [9 nil] INFO [2022-03-13 17:15:59,757] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,757] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,757] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,758] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:59,759] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:15:59,760] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,760] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,760] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,760] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,760] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:59,761] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:15:59,762] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,763] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:59,763] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,764] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,764] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,765] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,765] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,766] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,767] jepsen worker 89 - jepsen.util 2519 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,767] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,769] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,769] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,770] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,771] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,771] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 23] INFO [2022-03-13 17:15:59,771] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,771] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:15:59,772] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,773] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:15:59,778] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,786] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:15:59,858] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,858] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,859] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,859] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,867] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,867] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,868] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,869] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,870] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,871] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,873] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,873] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:15:59,874] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,875] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:15:59,920] jepsen worker 83 - jepsen.util 2783 :info :write [27 23] :RequestTimeoutException INFO [2022-03-13 17:15:59,921] jepsen worker 83 - jepsen.util 2873 :invoke :write [27 30] INFO [2022-03-13 17:15:59,929] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,929] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,929] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:15:59,930] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:15:59,930] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,931] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:15:59,935] jepsen worker 21 - jepsen.util 831 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,935] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:15:59,936] jepsen worker 79 - jepsen.util 1069 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,936] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:15:59,938] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,938] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:15:59,939] jepsen worker 43 - jepsen.util 583 :invoke :write [14 16] INFO [2022-03-13 17:15:59,939] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:15:59,947] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:15:59,947] jepsen worker nemesis - couchbase.util Rebalance check count: 468 INFO [2022-03-13 17:15:59,947] jepsen worker nemesis - couchbase.util Rebalance stuck count: 368 INFO [2022-03-13 17:15:59,971] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,971] jepsen worker 37 - jepsen.util 2107 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:15:59,971] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:15:59,972] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,014] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,015] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:00,017] jepsen worker 89 - jepsen.util 2519 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,017] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,017] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,017] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,018] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 13] INFO [2022-03-13 17:16:00,019] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 42] INFO [2022-03-13 17:16:00,020] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:16:00,020] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,022] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,023] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,023] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:16:00,025] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,026] jepsen worker 80 - jepsen.util 800 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,027] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:16:00,027] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,027] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,028] jepsen worker 80 - jepsen.util 800 :invoke :write [26 40] INFO [2022-03-13 17:16:00,029] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,029] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,030] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,031] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,032] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,032] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,033] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,033] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,035] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,035] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,036] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,036] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,037] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,043] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,043] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:16:00,045] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,047] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,048] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,049] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:16:00,082] jepsen worker 89 - jepsen.util 2519 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,083] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,135] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,135] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,135] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,136] jepsen worker 77 - jepsen.util 707 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,136] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,136] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,136] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,136] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:00,136] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:00,136] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,137] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:00,137] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,137] jepsen worker 78 - jepsen.util 618 :invoke :write [26 12] INFO [2022-03-13 17:16:00,138] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:00,139] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:16:00,140] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:00,140] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:00,141] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:16:00,142] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:00,143] jepsen worker 44 - jepsen.util 584 :invoke :write [14 9] INFO [2022-03-13 17:16:00,146] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,147] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,147] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:16:00,148] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:16:00,178] jepsen worker 62 - jepsen.util 512 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,179] jepsen worker 37 - jepsen.util 2107 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,179] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:00,180] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 5] INFO [2022-03-13 17:16:00,198] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,199] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 2] INFO [2022-03-13 17:16:00,224] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,225] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,227] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,228] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:16:00,234] jepsen worker 33 - jepsen.util 753 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,235] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:00,247] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,247] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,248] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 12] INFO [2022-03-13 17:16:00,248] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,252] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,253] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:16:00,307] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,308] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:16:00,315] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,315] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,348] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,349] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:16:00,349] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,350] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:16:00,399] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,400] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,413] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,414] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:00,415] jepsen worker 37 - jepsen.util 2107 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,415] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 47] INFO [2022-03-13 17:16:00,459] jepsen worker 89 - jepsen.util 2519 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,460] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,461] jepsen worker 41 - jepsen.util 3371 :ok :read [13 39] INFO [2022-03-13 17:16:00,462] jepsen worker 41 - jepsen.util 3371 :invoke :write [13 22] INFO [2022-03-13 17:16:00,464] jepsen worker 40 - jepsen.util 3460 :ok :read [13 39] INFO [2022-03-13 17:16:00,465] jepsen worker 40 - jepsen.util 3460 :invoke :write [13 29] INFO [2022-03-13 17:16:00,472] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,472] jepsen worker 4 - jepsen.util 2254 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,472] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,473] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:16:00,473] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:16:00,474] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,476] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,476] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:16:00,484] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,485] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,487] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,488] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:16:00,506] jepsen worker 84 - jepsen.util 3684 :ok :read [28 44] INFO [2022-03-13 17:16:00,507] jepsen worker 84 - jepsen.util 3684 :invoke :write [28 14] INFO [2022-03-13 17:16:00,549] jepsen worker 52 - jepsen.util 592 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,549] jepsen worker 52 - jepsen.util 592 :invoke :write [17 35] INFO [2022-03-13 17:16:00,561] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,562] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:16:00,564] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,565] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:16:00,573] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,573] jepsen worker 70 - jepsen.util 790 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,574] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:00,575] jepsen worker 70 - jepsen.util 790 :invoke :write [23 27] INFO [2022-03-13 17:16:00,622] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,623] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,625] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,626] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:16:00,631] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,631] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,631] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,634] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 30] INFO [2022-03-13 17:16:00,634] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,634] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,635] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,635] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:16:00,645] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,646] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,651] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,651] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 32] INFO [2022-03-13 17:16:00,686] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,687] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:16:00,691] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,692] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,693] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,694] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,701] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,701] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:16:00,701] jepsen worker 37 - jepsen.util 2107 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,701] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,701] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:16:00,702] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,704] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,704] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,705] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,706] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,707] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,707] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,708] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,709] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,711] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,711] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,713] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,713] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,748] jepsen worker 56 - jepsen.util 3386 :info :write [18 45] :SyncWriteAmbiguous INFO [2022-03-13 17:16:00,749] jepsen worker 56 - jepsen.util 3476 :invoke :read [18 nil] INFO [2022-03-13 17:16:00,752] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,752] jepsen worker 42 - jepsen.util 402 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,752] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:16:00,753] jepsen worker 42 - jepsen.util 402 :invoke :write [14 25] INFO [2022-03-13 17:16:00,761] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,761] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,762] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:16:00,763] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:16:00,788] jepsen worker 89 - jepsen.util 2519 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,789] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,790] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:16:00,790] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 20] INFO [2022-03-13 17:16:00,815] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,816] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:16:00,837] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,838] jepsen worker 69 - jepsen.util 699 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,838] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,839] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:00,840] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,840] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,841] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,842] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,851] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,852] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,853] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,854] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:00,879] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,880] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:16:00,901] jepsen worker 36 - jepsen.util 2196 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,902] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:16:00,932] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,932] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,933] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:16:00,933] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:16:00,951] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,951] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:00,953] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:00,953] jepsen worker nemesis - couchbase.util Rebalance check count: 469 INFO [2022-03-13 17:16:00,953] jepsen worker nemesis - couchbase.util Rebalance stuck count: 369 INFO [2022-03-13 17:16:00,963] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,964] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:00,968] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,969] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:00,969] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:00,969] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:16:01,052] jepsen worker 89 - jepsen.util 2519 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,052] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 14] INFO [2022-03-13 17:16:01,073] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,075] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:16:01,077] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,077] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,106] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,106] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,106] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 49] INFO [2022-03-13 17:16:01,107] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:16:01,132] jepsen worker 72 - jepsen.util 3402 :info :write [24 21] :SyncWriteAmbiguous INFO [2022-03-13 17:16:01,133] jepsen worker 72 - jepsen.util 3492 :invoke :write [24 30] INFO [2022-03-13 17:16:01,144] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,144] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,144] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,145] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,145] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,146] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,146] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,146] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,146] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,147] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:16:01,147] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,147] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,148] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,148] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,148] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,148] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,149] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,149] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,150] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,150] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,150] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:16:01,150] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,151] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:16:01,151] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 19] INFO [2022-03-13 17:16:01,176] jepsen worker 51 - jepsen.util 771 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,177] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:01,212] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,213] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:16:01,221] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,221] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,222] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:16:01,223] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,223] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:01,224] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:01,304] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,304] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,327] jepsen worker 89 - jepsen.util 2519 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,327] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,328] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,328] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:16:01,332] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,332] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,333] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,334] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,336] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,337] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,339] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,339] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,340] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,341] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:16:01,383] jepsen worker 4 - jepsen.util 2254 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,384] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,412] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,413] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:16:01,479] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,479] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:16:01,494] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,495] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,495] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,495] jepsen worker 29 - jepsen.util 3629 :ok :read [9 26] INFO [2022-03-13 17:16:01,495] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,496] jepsen worker 29 - jepsen.util 3629 :invoke :read [9 nil] INFO [2022-03-13 17:16:01,496] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 6] INFO [2022-03-13 17:16:01,497] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:16:01,497] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,500] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:16:01,517] jepsen worker 37 - jepsen.util 2107 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,518] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,518] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,518] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,520] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:16:01,521] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:16:01,540] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,540] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:16:01,562] jepsen worker 29 - jepsen.util 3629 :ok :read [9 26] INFO [2022-03-13 17:16:01,563] jepsen worker 29 - jepsen.util 3629 :invoke :read [9 nil] INFO [2022-03-13 17:16:01,600] jepsen worker 88 - jepsen.util 2068 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,600] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,623] jepsen worker 29 - jepsen.util 3629 :ok :read [9 26] INFO [2022-03-13 17:16:01,623] jepsen worker 29 - jepsen.util 3629 :invoke :read [9 nil] INFO [2022-03-13 17:16:01,630] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,631] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,633] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,633] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:16:01,635] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,635] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,636] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,636] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,641] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,641] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:16:01,645] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,645] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,646] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,647] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:16:01,648] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,649] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,650] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,656] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,673] jepsen worker 29 - jepsen.util 3629 :ok :read [9 26] INFO [2022-03-13 17:16:01,673] jepsen worker 29 - jepsen.util 3629 :invoke :read [9 nil] INFO [2022-03-13 17:16:01,716] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,716] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,717] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,717] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,717] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,717] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,717] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:16:01,718] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,719] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:16:01,720] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,723] jepsen worker 29 - jepsen.util 3629 :ok :read [9 26] INFO [2022-03-13 17:16:01,725] jepsen worker 29 - jepsen.util 3629 :invoke :write [9 41] INFO [2022-03-13 17:16:01,769] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,770] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,773] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,774] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:16:01,821] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,822] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:01,822] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,823] jepsen worker 7 - jepsen.util 637 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,823] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:01,824] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:01,826] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,826] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,826] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:01,827] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:01,836] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,837] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:01,845] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,846] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,846] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,846] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,861] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,861] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,861] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:16:01,862] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 15] INFO [2022-03-13 17:16:01,871] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,871] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:16:01,896] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,897] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:16:01,907] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,908] jepsen worker 75 - jepsen.util 435 :invoke :write [25 3] INFO [2022-03-13 17:16:01,910] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,910] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,910] jepsen worker 71 - jepsen.util 521 :invoke :write [23 34] INFO [2022-03-13 17:16:01,911] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:01,912] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,913] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:01,914] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,915] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:01,916] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,917] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:01,919] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,919] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:16:01,925] jepsen worker 26 - jepsen.util 476 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,925] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,925] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,926] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:01,926] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:01,927] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:01,952] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,952] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,953] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,953] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 1] INFO [2022-03-13 17:16:01,954] jepsen worker 21 - jepsen.util 831 :invoke :write [7 26] INFO [2022-03-13 17:16:01,954] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,959] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:01,959] jepsen worker nemesis - couchbase.util Rebalance check count: 470 INFO [2022-03-13 17:16:01,959] jepsen worker nemesis - couchbase.util Rebalance stuck count: 370 INFO [2022-03-13 17:16:01,959] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,960] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,964] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,965] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,966] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,967] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:01,967] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,967] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,968] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,968] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:16:01,969] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,969] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,975] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,975] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,975] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,976] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:16:01,977] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:16:01,977] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,978] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,979] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:01,983] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,984] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,986] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,987] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,989] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,990] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,991] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,992] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:01,993] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,994] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:16:01,998] jepsen worker 76 - jepsen.util 706 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,998] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,998] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,999] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,999] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:01,999] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:02,000] jepsen worker 77 - jepsen.util 707 :invoke :write [25 29] INFO [2022-03-13 17:16:02,001] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:02,001] jepsen worker 60 - jepsen.util 600 :invoke :write [20 22] INFO [2022-03-13 17:16:02,005] jepsen worker 12 - jepsen.util 732 :invoke :write [4 46] INFO [2022-03-13 17:16:02,016] jepsen worker 85 - jepsen.util 3055 :info :write [28 4] :RequestTimeoutException INFO [2022-03-13 17:16:02,016] jepsen worker 85 - jepsen.util 3145 :invoke :read [28 nil] INFO [2022-03-13 17:16:02,021] jepsen worker 67 - jepsen.util 2767 :info :write [22 14] :RequestTimeoutException INFO [2022-03-13 17:16:02,023] jepsen worker 67 - jepsen.util 2857 :invoke :read [22 nil] INFO [2022-03-13 17:16:02,052] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,053] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:02,053] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,054] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:16:02,054] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,055] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,059] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,059] jepsen worker 71 - jepsen.util 521 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,059] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,060] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:02,062] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,063] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,064] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,065] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,065] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,066] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,066] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:16:02,067] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:16:02,077] jepsen worker 80 - jepsen.util 800 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,077] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,078] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:02,078] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:02,105] jepsen worker 75 - jepsen.util 435 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,106] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:02,110] jepsen worker 1 - jepsen.util 2791 :info :write [0 1] :RequestTimeoutException INFO [2022-03-13 17:16:02,111] jepsen worker 1 - jepsen.util 2881 :invoke :write [0 40] INFO [2022-03-13 17:16:02,126] jepsen worker 8 - jepsen.util 728 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,127] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:02,133] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,134] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:16:02,162] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,163] jepsen worker 14 - jepsen.util 374 :invoke :write [4 27] INFO [2022-03-13 17:16:02,176] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,177] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:16:02,189] jepsen worker 89 - jepsen.util 2519 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,190] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,190] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,191] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:02,225] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,225] jepsen worker 37 - jepsen.util 2107 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,225] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,226] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,226] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,227] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 0] INFO [2022-03-13 17:16:02,228] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:16:02,228] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:16:02,237] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,237] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,246] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,247] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,248] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,249] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 30] INFO [2022-03-13 17:16:02,251] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,252] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:02,275] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,275] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,275] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,276] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:02,277] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:16:02,277] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,277] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:02,278] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:16:02,279] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,280] jepsen worker 78 - jepsen.util 618 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,280] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:16:02,280] jepsen worker 78 - jepsen.util 618 :invoke :write [26 12] INFO [2022-03-13 17:16:02,293] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,293] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,324] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,325] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:16:02,338] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,339] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:02,364] jepsen worker 17 - jepsen.util 3257 :info :write [5 4] :RequestTimeoutException INFO [2022-03-13 17:16:02,365] jepsen worker 17 - jepsen.util 3347 :invoke :write [5 28] INFO [2022-03-13 17:16:02,369] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,369] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,415] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,415] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:16:02,418] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,418] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,419] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,419] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,419] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:16:02,419] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,425] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,425] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,425] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,426] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 5] INFO [2022-03-13 17:16:02,428] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,429] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,430] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,431] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,432] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,432] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 37] INFO [2022-03-13 17:16:02,474] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,475] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:16:02,476] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,476] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,477] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,477] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:02,478] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:02,479] jepsen worker 51 - jepsen.util 771 :invoke :write [17 0] INFO [2022-03-13 17:16:02,483] jepsen worker 43 - jepsen.util 583 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,483] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,484] jepsen worker 43 - jepsen.util 583 :invoke :write [14 33] INFO [2022-03-13 17:16:02,485] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:02,499] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,500] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,501] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,502] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:16:02,519] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,520] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:16:02,530] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,530] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,531] jepsen worker 21 - jepsen.util 831 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,532] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:02,533] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:02,534] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:02,552] jepsen worker 44 - jepsen.util 584 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,553] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,553] jepsen worker 37 - jepsen.util 2107 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,553] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,554] jepsen worker 44 - jepsen.util 584 :invoke :write [14 34] INFO [2022-03-13 17:16:02,555] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:02,555] jepsen worker 69 - jepsen.util 699 :invoke :write [23 25] INFO [2022-03-13 17:16:02,556] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:16:02,560] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,561] jepsen worker 66 - jepsen.util 3216 :ok :read [22 30] INFO [2022-03-13 17:16:02,561] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:16:02,566] jepsen worker 66 - jepsen.util 3216 :invoke :write [22 46] INFO [2022-03-13 17:16:02,569] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,570] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,572] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,573] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,575] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,576] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,578] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,578] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,580] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,581] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,589] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,590] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,592] jepsen worker 39 - jepsen.util 3459 :ok :read [13 39] INFO [2022-03-13 17:16:02,593] jepsen worker 39 - jepsen.util 3459 :invoke :write [13 29] INFO [2022-03-13 17:16:02,595] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,596] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:16:02,619] jepsen worker 88 - jepsen.util 2068 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,620] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:16:02,644] jepsen worker 89 - jepsen.util 2519 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,645] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,673] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,673] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,676] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,676] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:02,700] jepsen worker 14 - jepsen.util 374 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,700] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,700] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,701] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:02,701] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:02,702] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:16:02,707] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,707] jepsen worker 24 - jepsen.util 564 :invoke :write [8 2] INFO [2022-03-13 17:16:02,725] jepsen worker 42 - jepsen.util 402 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,725] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,725] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,726] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,726] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,726] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,726] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:02,726] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,726] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,727] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:02,727] jepsen worker 87 - jepsen.util 2607 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,728] jepsen worker 53 - jepsen.util 503 :invoke :write [17 7] INFO [2022-03-13 17:16:02,729] jepsen worker 26 - jepsen.util 476 :invoke :write [8 4] INFO [2022-03-13 17:16:02,730] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 18] INFO [2022-03-13 17:16:02,730] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:02,734] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:16:02,734] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:02,735] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,736] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:16:02,738] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:02,840] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,842] jepsen worker 2 - jepsen.util 3872 :info :write [0 4] :RequestTimeoutException INFO [2022-03-13 17:16:02,842] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,843] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,843] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,843] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,845] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:02,847] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,847] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:16:02,847] jepsen worker 2 - jepsen.util 3962 :invoke :read [0 nil] INFO [2022-03-13 17:16:02,847] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,847] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,864] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,864] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,865] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,866] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,867] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,868] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,868] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,869] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,870] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,871] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,871] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,871] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,872] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,873] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,873] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] INFO [2022-03-13 17:16:02,874] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:02,902] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,902] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:02,904] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,905] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,905] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,905] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 39] INFO [2022-03-13 17:16:02,906] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,906] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 42] INFO [2022-03-13 17:16:02,960] jepsen worker 11 - jepsen.util 3341 :info :write [3 38] :RequestTimeoutException INFO [2022-03-13 17:16:02,961] jepsen worker 11 - jepsen.util 3431 :invoke :read [3 nil] INFO [2022-03-13 17:16:02,968] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:02,968] jepsen worker nemesis - couchbase.util Rebalance check count: 471 INFO [2022-03-13 17:16:02,968] jepsen worker nemesis - couchbase.util Rebalance stuck count: 371 INFO [2022-03-13 17:16:02,988] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,988] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:02,997] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,997] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,998] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:02,999] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,000] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:16:03,001] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:16:03,007] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,008] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:16:03,038] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,038] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,039] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 21] INFO [2022-03-13 17:16:03,046] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:16:03,072] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,072] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,072] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,073] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,104] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,104] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,154] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,154] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,160] jepsen worker 3 - jepsen.util 1983 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,160] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:16:03,217] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,217] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,217] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,217] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,218] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,218] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,223] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,223] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,223] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,223] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 27] INFO [2022-03-13 17:16:03,224] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,224] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:16:03,227] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,227] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:16:03,245] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,245] jepsen worker 37 - jepsen.util 2107 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,245] jepsen worker 89 - jepsen.util 2519 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,245] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,246] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,246] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:16:03,251] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,251] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,258] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,258] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,258] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,259] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,259] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:16:03,260] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,263] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,263] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,263] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,264] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,265] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,265] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 1] INFO [2022-03-13 17:16:03,269] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,269] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,271] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,272] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:03,295] jepsen worker 70 - jepsen.util 790 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,296] jepsen worker 70 - jepsen.util 790 :invoke :write [23 2] INFO [2022-03-13 17:16:03,297] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,298] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:16:03,300] jepsen worker 20 - jepsen.util 3710 :info :write [6 32] :RequestTimeoutException INFO [2022-03-13 17:16:03,301] jepsen worker 20 - jepsen.util 3800 :invoke :write [6 13] INFO [2022-03-13 17:16:03,301] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,301] jepsen worker 51 - jepsen.util 771 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,302] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,303] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,303] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,303] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,304] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:03,304] jepsen worker 33 - jepsen.util 753 :invoke :write [11 49] INFO [2022-03-13 17:16:03,308] jepsen worker 14 - jepsen.util 374 :invoke :write [4 23] INFO [2022-03-13 17:16:03,308] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:03,311] jepsen worker 51 - jepsen.util 771 :invoke :write [17 33] INFO [2022-03-13 17:16:03,313] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:03,340] jepsen worker 88 - jepsen.util 2068 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,341] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,346] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,347] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,353] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,354] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:16:03,354] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,355] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,355] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 21] INFO [2022-03-13 17:16:03,356] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,357] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,358] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,359] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,360] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,366] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:03,366] jepsen worker 43 - jepsen.util 583 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,366] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,367] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,367] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,367] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,367] jepsen worker 42 - jepsen.util 402 :invoke :write [14 4] INFO [2022-03-13 17:16:03,368] jepsen worker 43 - jepsen.util 583 :invoke :write [14 36] INFO [2022-03-13 17:16:03,368] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:03,369] jepsen worker 6 - jepsen.util 636 :invoke :write [2 37] INFO [2022-03-13 17:16:03,370] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:03,370] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:03,386] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,387] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,424] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,424] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,427] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,427] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,429] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,429] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:16:03,439] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,441] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,446] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,446] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,446] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,446] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,446] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:03,447] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:16:03,448] jepsen worker 80 - jepsen.util 800 :invoke :write [26 3] INFO [2022-03-13 17:16:03,448] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:03,451] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,452] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:16:03,459] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,459] jepsen worker 24 - jepsen.util 564 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,460] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,460] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:03,492] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,492] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,492] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,493] jepsen worker 76 - jepsen.util 706 :invoke :write [25 10] INFO [2022-03-13 17:16:03,494] jepsen worker 61 - jepsen.util 691 :invoke :write [20 14] INFO [2022-03-13 17:16:03,495] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:03,506] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,508] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:16:03,535] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,535] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,535] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,536] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,536] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,536] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,536] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,537] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:16:03,537] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:16:03,540] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:03,541] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:16:03,541] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,541] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,542] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:16:03,545] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,546] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,548] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,549] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:16:03,555] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,556] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,557] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,557] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,558] jepsen worker 37 - jepsen.util 2107 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,558] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,559] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,560] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,581] jepsen worker 33 - jepsen.util 753 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,581] jepsen worker 33 - jepsen.util 753 :invoke :write [11 43] INFO [2022-03-13 17:16:03,584] jepsen worker 26 - jepsen.util 476 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,585] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:03,590] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,590] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,595] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,595] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,599] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,600] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:16:03,614] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,614] jepsen worker 89 - jepsen.util 2519 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,615] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,616] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,617] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,618] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 47] INFO [2022-03-13 17:16:03,619] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,623] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:16:03,668] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,668] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,669] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,669] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 37] INFO [2022-03-13 17:16:03,672] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,673] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 5] INFO [2022-03-13 17:16:03,681] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,682] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:16:03,703] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,704] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:16:03,728] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,728] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,728] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:03,729] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,752] jepsen worker 89 - jepsen.util 2519 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,753] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 3] INFO [2022-03-13 17:16:03,764] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,765] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:16:03,774] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,775] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:16:03,835] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,835] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:16:03,846] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,847] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:16:03,874] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,875] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,896] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,896] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,906] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,907] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,910] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,911] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:03,939] jepsen worker 38 - jepsen.util 2468 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,939] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,939] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,942] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:16:03,943] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:03,940] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,941] jepsen worker 12 - jepsen.util 732 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,942] jepsen worker 24 - jepsen.util 564 :invoke :write [8 12] INFO [2022-03-13 17:16:03,943] jepsen worker 12 - jepsen.util 732 :invoke :write [4 47] INFO [2022-03-13 17:16:03,943] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,949] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,949] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:03,951] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,951] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:16:03,953] jepsen worker 14 - jepsen.util 374 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,953] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:16:03,969] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,970] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:03,974] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:03,974] jepsen worker nemesis - couchbase.util Rebalance check count: 472 INFO [2022-03-13 17:16:03,974] jepsen worker nemesis - couchbase.util Rebalance stuck count: 372 INFO [2022-03-13 17:16:03,981] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,982] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:03,984] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:03,985] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:16:04,050] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,050] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,050] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,051] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 12] INFO [2022-03-13 17:16:04,051] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:04,052] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,053] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,053] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 35] INFO [2022-03-13 17:16:04,053] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,054] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:16:04,063] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,066] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:16:04,068] jepsen worker 89 - jepsen.util 2519 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,068] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,069] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:04,069] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:16:04,115] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,116] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:04,144] jepsen worker 79 - jepsen.util 1069 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,144] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,145] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:04,146] jepsen worker 37 - jepsen.util 2107 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,146] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:04,146] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,146] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,147] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:04,147] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:16:04,148] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:04,149] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,149] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,149] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:04,150] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:04,165] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,166] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 10] INFO [2022-03-13 17:16:04,167] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,167] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,167] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 8] INFO [2022-03-13 17:16:04,168] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:04,170] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,171] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:04,174] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,175] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 49] INFO [2022-03-13 17:16:04,202] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,202] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:16:04,232] jepsen worker 88 - jepsen.util 2068 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,232] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:04,245] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,246] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 14] INFO [2022-03-13 17:16:04,272] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,272] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 9] INFO [2022-03-13 17:16:04,296] jepsen worker 58 - jepsen.util 3118 :info :write [19 15] :RequestTimeoutException INFO [2022-03-13 17:16:04,297] jepsen worker 58 - jepsen.util 3208 :invoke :read [19 nil] INFO [2022-03-13 17:16:04,303] jepsen worker 87 - jepsen.util 2607 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,304] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:04,334] jepsen worker 51 - jepsen.util 771 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,334] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,334] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,334] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:04,335] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:04,335] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:04,341] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,342] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:16:04,349] jepsen worker 24 - jepsen.util 564 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,349] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:04,355] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,356] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:16:04,371] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,371] jepsen worker 4 - jepsen.util 2254 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,371] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,372] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:16:04,373] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:16:04,374] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:16:04,377] jepsen worker 59 - jepsen.util 3299 :info :write [19 30] :RequestTimeoutException INFO [2022-03-13 17:16:04,378] jepsen worker 59 - jepsen.util 3389 :invoke :write [19 23] INFO [2022-03-13 17:16:04,391] jepsen worker 3 - jepsen.util 1983 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,392] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,394] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,395] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:16:04,414] jepsen worker 37 - jepsen.util 2107 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,415] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 42] INFO [2022-03-13 17:16:04,498] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,499] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:16:04,503] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,504] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,510] jepsen worker 48 - jepsen.util 3288 :info :write [16 6] :RequestTimeoutException INFO [2022-03-13 17:16:04,511] jepsen worker 48 - jepsen.util 3378 :invoke :read [16 nil] INFO [2022-03-13 17:16:04,544] jepsen worker 36 - jepsen.util 2196 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,544] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:16:04,563] jepsen worker 68 - jepsen.util 3578 :info :write [22 0] :RequestTimeoutException INFO [2022-03-13 17:16:04,564] jepsen worker 68 - jepsen.util 3668 :invoke :read [22 nil] INFO [2022-03-13 17:16:04,586] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,586] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,586] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,587] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:04,588] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:16:04,588] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:04,601] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,602] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,604] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,605] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,606] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,606] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,608] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,608] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:16:04,627] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,630] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:04,636] jepsen worker 81 - jepsen.util 3771 :info :write [27 20] :RequestTimeoutException INFO [2022-03-13 17:16:04,636] jepsen worker 81 - jepsen.util 3861 :invoke :read [27 nil] INFO [2022-03-13 17:16:04,639] jepsen worker 65 - jepsen.util 3845 :ok :read [21 :nil] INFO [2022-03-13 17:16:04,639] jepsen worker 63 - jepsen.util 3393 :ok :read [21 :nil] INFO [2022-03-13 17:16:04,640] jepsen worker 65 - jepsen.util 3845 :invoke :read [21 nil] INFO [2022-03-13 17:16:04,640] jepsen worker 63 - jepsen.util 3393 :invoke :write [21 49] INFO [2022-03-13 17:16:04,649] jepsen worker 38 - jepsen.util 2468 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,649] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:16:04,689] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,691] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:16:04,697] jepsen worker 65 - jepsen.util 3845 :ok :read [21 :nil] INFO [2022-03-13 17:16:04,697] jepsen worker 65 - jepsen.util 3845 :invoke :read [21 nil] INFO [2022-03-13 17:16:04,703] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,703] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,704] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,704] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,704] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 6] INFO [2022-03-13 17:16:04,705] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:16:04,705] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,706] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,707] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,708] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:16:04,710] jepsen worker 45 - jepsen.util 3555 :ok :read [15 48] INFO [2022-03-13 17:16:04,710] jepsen worker 45 - jepsen.util 3555 :invoke :read [15 nil] INFO [2022-03-13 17:16:04,717] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,718] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 18] INFO [2022-03-13 17:16:04,719] jepsen worker 56 - jepsen.util 3476 :ok :read [18 21] INFO [2022-03-13 17:16:04,720] jepsen worker 56 - jepsen.util 3476 :invoke :write [18 6] INFO [2022-03-13 17:16:04,730] jepsen worker 65 - jepsen.util 3845 :ok :read [21 :nil] INFO [2022-03-13 17:16:04,731] jepsen worker 65 - jepsen.util 3845 :invoke :write [21 32] INFO [2022-03-13 17:16:04,746] jepsen worker 45 - jepsen.util 3555 :ok :read [15 48] INFO [2022-03-13 17:16:04,747] jepsen worker 80 - jepsen.util 800 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,748] jepsen worker 45 - jepsen.util 3555 :invoke :write [15 47] INFO [2022-03-13 17:16:04,749] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:04,758] jepsen worker 60 - jepsen.util 600 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,759] jepsen worker 60 - jepsen.util 600 :invoke :write [20 30] INFO [2022-03-13 17:16:04,762] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,762] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,763] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:04,764] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:04,776] jepsen worker 37 - jepsen.util 2107 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,777] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:04,796] jepsen worker 49 - jepsen.util 3289 :info :write [16 8] :RequestTimeoutException INFO [2022-03-13 17:16:04,797] jepsen worker 49 - jepsen.util 3379 :invoke :write [16 0] INFO [2022-03-13 17:16:04,805] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,806] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] INFO [2022-03-13 17:16:04,814] jepsen worker 88 - jepsen.util 2068 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,814] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:16:04,848] jepsen worker 53 - jepsen.util 503 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,849] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,849] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,849] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,849] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:04,850] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:04,850] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:04,851] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:04,856] jepsen worker 16 - jepsen.util 3166 :info :write [5 11] :SyncWriteAmbiguous INFO [2022-03-13 17:16:04,856] jepsen worker 16 - jepsen.util 3256 :invoke :read [5 nil] INFO [2022-03-13 17:16:04,864] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,864] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,865] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,865] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,866] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:04,866] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:04,867] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,867] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:04,867] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,867] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:16:04,868] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:04,868] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:16:04,869] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,869] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:04,870] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,871] jepsen worker 75 - jepsen.util 435 :invoke :write [25 33] INFO [2022-03-13 17:16:04,875] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,876] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:04,924] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,924] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,952] jepsen worker 44 - jepsen.util 584 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,952] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:04,975] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:04,978] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:04,981] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:04,981] jepsen worker nemesis - couchbase.util Rebalance check count: 473 INFO [2022-03-13 17:16:04,981] jepsen worker nemesis - couchbase.util Rebalance stuck count: 373 INFO [2022-03-13 17:16:04,996] jepsen worker 9 - jepsen.util 2889 :ok :read [3 25] INFO [2022-03-13 17:16:04,997] jepsen worker 9 - jepsen.util 2889 :invoke :read [3 nil] INFO [2022-03-13 17:16:05,007] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,007] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,007] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,007] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 43] INFO [2022-03-13 17:16:05,008] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 39] INFO [2022-03-13 17:16:05,009] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:16:05,010] jepsen worker 28 - jepsen.util 3538 :ok :read [9 26] INFO [2022-03-13 17:16:05,011] jepsen worker 28 - jepsen.util 3538 :invoke :write [9 6] INFO [2022-03-13 17:16:05,014] jepsen worker 33 - jepsen.util 753 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,015] jepsen worker 33 - jepsen.util 753 :invoke :write [11 31] INFO [2022-03-13 17:16:05,017] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,017] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,018] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,018] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:05,018] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,019] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:05,027] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,027] jepsen worker 24 - jepsen.util 564 :invoke :write [8 3] INFO [2022-03-13 17:16:05,031] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,031] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,031] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,031] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:05,032] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:05,033] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:16:05,033] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,034] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:05,034] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,036] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,037] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:16:05,037] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:16:05,048] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,049] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,050] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,050] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,051] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,052] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,054] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,055] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,056] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,057] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,061] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,061] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,062] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:16:05,063] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 21] INFO [2022-03-13 17:16:05,106] jepsen worker 89 - jepsen.util 2519 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,106] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 34] INFO [2022-03-13 17:16:05,169] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,170] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:16:05,189] jepsen worker 3 - jepsen.util 1983 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,190] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:16:05,252] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,252] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,253] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,254] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:05,325] jepsen worker 36 - jepsen.util 2196 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,325] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,326] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:16:05,331] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:16:05,338] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,340] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:16:05,370] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,370] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 9] INFO [2022-03-13 17:16:05,460] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,461] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:16:05,500] jepsen worker 0 - jepsen.util 3240 :info :write [0 1] :RequestTimeoutException INFO [2022-03-13 17:16:05,501] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,501] jepsen worker 0 - jepsen.util 3330 :invoke :write [0 18] INFO [2022-03-13 17:16:05,502] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,503] jepsen worker 6 - jepsen.util 636 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,504] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,505] jepsen worker 6 - jepsen.util 636 :invoke :write [2 35] INFO [2022-03-13 17:16:05,507] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,508] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,509] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:16:05,520] jepsen worker 55 - jepsen.util 3205 :info :write [18 46] :RequestTimeoutException INFO [2022-03-13 17:16:05,521] jepsen worker 55 - jepsen.util 3295 :invoke :read [18 nil] INFO [2022-03-13 17:16:05,534] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,535] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:05,571] jepsen worker 75 - jepsen.util 435 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,572] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:16:05,578] jepsen worker 89 - jepsen.util 2519 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,578] jepsen worker 37 - jepsen.util 2107 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,579] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 32] INFO [2022-03-13 17:16:05,583] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 6] INFO [2022-03-13 17:16:05,589] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,590] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,590] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,590] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,590] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,591] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,591] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:05,592] jepsen worker 51 - jepsen.util 771 :invoke :write [17 26] INFO [2022-03-13 17:16:05,592] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:16:05,593] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:05,594] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:16:05,606] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:05,614] jepsen worker 24 - jepsen.util 564 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,614] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,615] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:16:05,616] jepsen worker 24 - jepsen.util 564 :invoke :write [8 13] INFO [2022-03-13 17:16:05,629] jepsen worker 3 - jepsen.util 1983 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,630] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:16:05,636] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,637] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,661] jepsen worker 34 - jepsen.util 844 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,662] jepsen worker 34 - jepsen.util 844 :invoke :write [11 2] INFO [2022-03-13 17:16:05,673] jepsen worker 69 - jepsen.util 699 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,673] jepsen worker 69 - jepsen.util 699 :invoke :write [23 22] INFO [2022-03-13 17:16:05,711] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,712] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:16:05,719] jepsen worker 33 - jepsen.util 753 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,719] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,719] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,719] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,720] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:05,720] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:05,721] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:05,721] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:16:05,726] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,728] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,729] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,730] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,739] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,740] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:16:05,750] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,750] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 34] INFO [2022-03-13 17:16:05,763] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,764] jepsen worker 71 - jepsen.util 521 :invoke :write [23 47] INFO [2022-03-13 17:16:05,767] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,767] jepsen worker 42 - jepsen.util 402 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,767] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:05,768] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:05,776] jepsen worker 82 - jepsen.util 3322 :info :write [27 7] :RequestTimeoutException INFO [2022-03-13 17:16:05,777] jepsen worker 82 - jepsen.util 3412 :invoke :write [27 11] INFO [2022-03-13 17:16:05,803] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,803] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,803] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:05,804] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:16:05,806] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,807] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,808] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,808] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:16:05,815] jepsen worker 46 - jepsen.util 3376 :info :write [15 29] :SyncWriteAmbiguous INFO [2022-03-13 17:16:05,817] jepsen worker 46 - jepsen.util 3466 :invoke :write [15 25] INFO [2022-03-13 17:16:05,821] jepsen worker 24 - jepsen.util 564 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,821] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:05,869] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,870] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:05,871] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,872] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:16:05,910] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,912] jepsen worker 19 - jepsen.util 2989 :info :write [6 34] :SyncWriteAmbiguous INFO [2022-03-13 17:16:05,912] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:05,913] jepsen worker 19 - jepsen.util 3079 :invoke :read [6 nil] INFO [2022-03-13 17:16:05,913] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,914] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:16:05,917] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,918] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:16:05,969] jepsen worker 88 - jepsen.util 2068 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,969] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,970] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:16:05,971] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:16:05,973] jepsen worker 89 - jepsen.util 2519 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,973] jepsen worker 70 - jepsen.util 790 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,973] jepsen worker 37 - jepsen.util 2107 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,974] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:05,975] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:05,975] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 41] INFO [2022-03-13 17:16:05,975] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,976] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,976] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 1] INFO [2022-03-13 17:16:05,976] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:05,977] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,978] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:05,985] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,985] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:05,986] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:16:05,986] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 14] INFO [2022-03-13 17:16:05,988] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:05,988] jepsen worker nemesis - couchbase.util Rebalance check count: 474 INFO [2022-03-13 17:16:05,988] jepsen worker nemesis - couchbase.util Rebalance stuck count: 374 INFO [2022-03-13 17:16:06,018] jepsen worker 27 - jepsen.util 3717 :info :write [9 40] :RequestTimeoutException INFO [2022-03-13 17:16:06,019] jepsen worker 27 - jepsen.util 3807 :invoke :read [9 nil] INFO [2022-03-13 17:16:06,042] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,043] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:16:06,068] jepsen worker 74 - jepsen.util 3584 :info :write [24 27] :RequestTimeoutException INFO [2022-03-13 17:16:06,069] jepsen worker 74 - jepsen.util 3674 :invoke :read [24 nil] INFO [2022-03-13 17:16:06,092] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,093] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,095] jepsen worker 74 - jepsen.util 3674 :ok :read [24 24] INFO [2022-03-13 17:16:06,096] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,096] jepsen worker 74 - jepsen.util 3674 :invoke :write [24 14] INFO [2022-03-13 17:16:06,097] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:16:06,106] jepsen worker 54 - jepsen.util 3024 :info :write [18 13] :RequestTimeoutException INFO [2022-03-13 17:16:06,106] jepsen worker 54 - jepsen.util 3114 :invoke :write [18 37] INFO [2022-03-13 17:16:06,108] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,109] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:16:06,123] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,124] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:06,127] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,129] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:06,133] jepsen worker 19 - jepsen.util 3079 :ok :read [6 16] INFO [2022-03-13 17:16:06,134] jepsen worker 19 - jepsen.util 3079 :invoke :read [6 nil] INFO [2022-03-13 17:16:06,160] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,160] jepsen worker 51 - jepsen.util 771 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,160] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,160] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,160] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:06,161] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:06,162] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:06,162] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:06,181] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,182] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,183] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,184] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:16:06,195] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,198] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,230] jepsen worker 19 - jepsen.util 3079 :ok :read [6 16] INFO [2022-03-13 17:16:06,231] jepsen worker 19 - jepsen.util 3079 :invoke :write [6 2] INFO [2022-03-13 17:16:06,255] jepsen worker 89 - jepsen.util 2519 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,255] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,255] jepsen worker 38 - jepsen.util 2468 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,256] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,256] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,257] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,257] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:16:06,258] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,258] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,259] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,260] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,261] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 46] INFO [2022-03-13 17:16:06,272] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,272] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:06,273] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,273] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:16:06,288] jepsen worker 30 - jepsen.util 3180 :info :write [10 15] :RequestTimeoutException INFO [2022-03-13 17:16:06,288] jepsen worker 30 - jepsen.util 3270 :invoke :write [10 43] INFO [2022-03-13 17:16:06,292] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,292] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,338] jepsen worker 43 - jepsen.util 583 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,338] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,338] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,338] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,338] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,338] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,339] jepsen worker 53 - jepsen.util 503 :invoke :write [17 19] INFO [2022-03-13 17:16:06,340] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 13] INFO [2022-03-13 17:16:06,340] jepsen worker 43 - jepsen.util 583 :invoke :write [14 8] INFO [2022-03-13 17:16:06,343] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:06,343] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,344] jepsen worker 8 - jepsen.util 728 :invoke :write [2 34] INFO [2022-03-13 17:16:06,344] jepsen worker 62 - jepsen.util 512 :invoke :write [20 15] INFO [2022-03-13 17:16:06,346] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:16:06,350] jepsen worker 37 - jepsen.util 2107 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,354] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 11] INFO [2022-03-13 17:16:06,380] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,381] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,382] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,384] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:16:06,395] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,395] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,395] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,396] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,397] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,398] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:16:06,398] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,398] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,399] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,400] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 37] INFO [2022-03-13 17:16:06,401] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,401] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:16:06,403] jepsen worker 32 - jepsen.util 3452 :info :write [10 43] :RequestTimeoutException INFO [2022-03-13 17:16:06,410] jepsen worker 32 - jepsen.util 3542 :invoke :read [10 nil] INFO [2022-03-13 17:16:06,428] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,429] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,490] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,491] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:16:06,524] jepsen worker 89 - jepsen.util 2519 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,525] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:16:06,530] jepsen worker 71 - jepsen.util 521 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,530] jepsen worker 76 - jepsen.util 706 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,530] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,531] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,532] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:16:06,532] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:06,533] jepsen worker 75 - jepsen.util 435 :invoke :write [25 11] INFO [2022-03-13 17:16:06,533] jepsen worker 76 - jepsen.util 706 :invoke :write [25 13] INFO [2022-03-13 17:16:06,535] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,536] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:16:06,545] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,546] jepsen worker 14 - jepsen.util 374 :invoke :write [4 22] INFO [2022-03-13 17:16:06,564] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,564] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,578] jepsen worker 6 - jepsen.util 636 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,578] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,578] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:06,579] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,580] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:06,581] jepsen worker 6 - jepsen.util 636 :invoke :write [2 15] INFO [2022-03-13 17:16:06,582] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:06,582] jepsen worker 26 - jepsen.util 476 :invoke :write [8 36] INFO [2022-03-13 17:16:06,608] jepsen worker 88 - jepsen.util 2068 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,609] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:16:06,622] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,622] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,623] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,624] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,625] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,626] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] WARN [2022-03-13 17:16:06,646] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12059223,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12059223,"total_server_duration_us":32,"operation_id":"0xd2ab","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":12063241},{"operation_name":"upsert","last_dispatch_duration_us":97679,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10033877,"total_server_duration_us":135276,"encode_duration_us":17471,"operation_id":"0xd710","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10016009},{"operation_name":"upsert","last_dispatch_duration_us":9118797,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10400297,"total_server_duration_us":108685,"encode_duration_us":28193,"operation_id":"0xd6a5","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10007557},{"operation_name":"upsert","last_dispatch_duration_us":9007836,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":12991267,"total_server_duration_us":153582,"encode_duration_us":17236,"operation_id":"0xd584","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10005698},{"operation_name":"upsert","last_dispatch_duration_us":32079,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6282148,"total_server_duration_us":281567,"encode_duration_us":18004,"operation_id":"0xd4f7","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10002593},{"operation_name":"upsert","last_dispatch_duration_us":9399180,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15586980,"total_server_duration_us":9096026,"encode_duration_us":16830,"operation_id":"0xd3f7","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10001941},{"operation_name":"upsert","last_dispatch_duration_us":9694933,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9694933,"total_server_duration_us":9045859,"encode_duration_us":20865,"operation_id":"0xd71c","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":9701381},{"operation_name":"upsert","last_dispatch_duration_us":1921336,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2690643,"total_server_duration_us":551079,"encode_duration_us":35181,"operation_id":"0xd79f","timeout_ms":10000,"last_server_duration_us":350020,"total_duration_us":4289864},{"operation_name":"upsert","last_dispatch_duration_us":1518796,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1518796,"total_server_duration_us":43727,"encode_duration_us":22127,"operation_id":"0xd8fc","timeout_ms":10000,"last_server_duration_us":43727,"total_duration_us":2605610},{"operation_name":"upsert","last_dispatch_duration_us":1699665,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2290277,"total_server_duration_us":317506,"encode_duration_us":19473,"operation_id":"0xd825","timeout_ms":10000,"last_server_duration_us":184936,"total_duration_us":2322654}],"total_count":62}} WARN [2022-03-13 17:16:06,647] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9399180,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15586980,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9096026,"encode_duration_us":16830,"operation_id":"0xd3f7","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10001941},{"operation_name":"upsert","last_dispatch_duration_us":17837299,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18302988,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9197512,"encode_duration_us":16839,"operation_id":"0xd0f9","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10004236}],"total_count":2}} INFO [2022-03-13 17:16:06,665] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,665] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:16:06,667] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,667] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,687] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,687] jepsen worker 37 - jepsen.util 2107 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,687] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,688] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,698] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,698] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,700] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,701] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:06,738] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,738] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,738] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,739] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,738] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,739] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,739] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:06,740] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:06,741] jepsen worker 33 - jepsen.util 753 :invoke :write [11 46] INFO [2022-03-13 17:16:06,742] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:06,743] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:16:06,743] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:16:06,748] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,749] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:16:06,750] jepsen worker 73 - jepsen.util 3493 :info :write [24 32] :RequestTimeoutException INFO [2022-03-13 17:16:06,751] jepsen worker 73 - jepsen.util 3583 :invoke :write [24 26] INFO [2022-03-13 17:16:06,759] jepsen worker 61 - jepsen.util 691 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,759] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,760] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,760] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,761] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:06,761] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:16:06,762] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,762] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,763] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:16:06,763] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,764] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,766] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,768] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,768] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,779] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,779] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,780] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,781] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,782] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,783] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 19] INFO [2022-03-13 17:16:06,812] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,812] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,813] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,813] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,863] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,864] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,864] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,864] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:16:06,864] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:16:06,865] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,867] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,867] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,868] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,869] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,869] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,870] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,870] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,870] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,871] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,871] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,872] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,873] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:16:06,876] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,876] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,879] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,879] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,880] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,881] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,883] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,883] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,926] jepsen worker 3 - jepsen.util 1983 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,926] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,926] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,927] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,938] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,938] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,939] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,939] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:16:06,940] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,940] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,941] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,942] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,943] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,944] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,945] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,946] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:06,947] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,947] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 5] INFO [2022-03-13 17:16:06,962] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,962] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,963] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,963] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,964] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,964] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,964] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,965] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,967] jepsen worker 64 - jepsen.util 3214 :info :write [21 49] :RequestTimeoutException INFO [2022-03-13 17:16:06,968] jepsen worker 64 - jepsen.util 3304 :invoke :read [21 nil] INFO [2022-03-13 17:16:06,969] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,970] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:16:06,977] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,977] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,977] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,977] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,978] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:06,979] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:16:06,979] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,979] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,980] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,980] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,981] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,981] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 31] INFO [2022-03-13 17:16:06,982] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:16:06,987] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,988] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,989] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:06,989] jepsen worker 75 - jepsen.util 435 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,990] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:06,991] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,991] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 48] INFO [2022-03-13 17:16:06,993] jepsen worker 62 - jepsen.util 512 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:06,995] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:06,995] jepsen worker nemesis - couchbase.util Rebalance check count: 475 INFO [2022-03-13 17:16:06,995] jepsen worker nemesis - couchbase.util Rebalance stuck count: 375 INFO [2022-03-13 17:16:06,995] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:07,029] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,029] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,030] jepsen worker 18 - jepsen.util 3258 :info :write [6 26] :RequestTimeoutException INFO [2022-03-13 17:16:07,031] jepsen worker 18 - jepsen.util 3348 :invoke :write [6 17] INFO [2022-03-13 17:16:07,039] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,039] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:16:07,058] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,058] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:16:07,084] jepsen worker 89 - jepsen.util 2519 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,085] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 33] INFO [2022-03-13 17:16:07,113] jepsen worker 3 - jepsen.util 1983 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,113] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,126] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,127] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,128] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,129] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,130] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,130] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,131] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,132] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:16:07,135] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,136] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:16:07,142] jepsen worker 14 - jepsen.util 374 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,142] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,142] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,143] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,143] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:07,144] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:07,144] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:07,144] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:16:07,148] jepsen worker 47 - jepsen.util 3287 :info :write [15 48] :RequestTimeoutException INFO [2022-03-13 17:16:07,148] jepsen worker 47 - jepsen.util 3377 :invoke :write [15 14] INFO [2022-03-13 17:16:07,178] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,178] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:07,181] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,182] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:16:07,210] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,211] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:07,228] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,228] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:16:07,230] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,230] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,230] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,231] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:07,232] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,232] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:07,232] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,233] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:07,234] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:16:07,234] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:16:07,236] jepsen worker 33 - jepsen.util 753 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,237] jepsen worker 33 - jepsen.util 753 :invoke :write [11 13] INFO [2022-03-13 17:16:07,293] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,294] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:16:07,316] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,317] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:16:07,327] jepsen worker 37 - jepsen.util 2107 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,328] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 43] INFO [2022-03-13 17:16:07,344] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,345] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,345] jepsen worker 24 - jepsen.util 564 :invoke :write [8 15] INFO [2022-03-13 17:16:07,347] jepsen worker 60 - jepsen.util 600 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,348] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:07,350] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:16:07,366] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,367] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:07,376] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,377] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:16:07,403] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,403] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,403] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,403] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,404] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:16:07,405] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:07,406] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:07,406] jepsen worker 80 - jepsen.util 800 :invoke :write [26 39] INFO [2022-03-13 17:16:07,420] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,421] jepsen worker 89 - jepsen.util 2519 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,421] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 11] INFO [2022-03-13 17:16:07,422] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:16:07,427] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,428] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:07,431] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,431] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:16:07,436] jepsen worker 10 - jepsen.util 3340 :info :write [3 41] :RequestTimeoutException INFO [2022-03-13 17:16:07,436] jepsen worker 10 - jepsen.util 3430 :invoke :write [3 49] INFO [2022-03-13 17:16:07,446] jepsen worker 13 - jepsen.util 643 :info :write [4 1] :RequestTimeoutException INFO [2022-03-13 17:16:07,448] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:07,522] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,523] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:16:07,527] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,528] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:16:07,529] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,530] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:16:07,556] jepsen worker 79 - jepsen.util 1069 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,556] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:07,570] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,571] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:16:07,581] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,582] jepsen worker 75 - jepsen.util 435 :invoke :write [25 18] INFO [2022-03-13 17:16:07,594] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,595] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:16:07,609] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,610] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,612] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,612] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,612] jepsen worker 33 - jepsen.util 753 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,612] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,613] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:07,613] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:16:07,614] jepsen worker 51 - jepsen.util 771 :invoke :write [17 38] INFO [2022-03-13 17:16:07,614] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:16:07,644] jepsen worker 89 - jepsen.util 2519 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,645] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:16:07,686] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,687] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:16:07,695] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,696] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 9] INFO [2022-03-13 17:16:07,718] jepsen worker 81 - jepsen.util 3861 :ok :read [27 4] INFO [2022-03-13 17:16:07,718] jepsen worker 81 - jepsen.util 3861 :invoke :read [27 nil] INFO [2022-03-13 17:16:07,729] jepsen worker 37 - jepsen.util 2107 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,729] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:07,745] jepsen worker 81 - jepsen.util 3861 :ok :read [27 4] INFO [2022-03-13 17:16:07,745] jepsen worker 81 - jepsen.util 3861 :invoke :write [27 18] INFO [2022-03-13 17:16:07,770] jepsen worker 57 - jepsen.util 3387 :info :write [19 6] :RequestTimeoutException INFO [2022-03-13 17:16:07,771] jepsen worker 57 - jepsen.util 3477 :invoke :write [19 15] INFO [2022-03-13 17:16:07,778] jepsen worker 80 - jepsen.util 800 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,778] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:07,778] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,779] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:16:07,783] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,783] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,784] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,784] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:16:07,827] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,828] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:16:07,832] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,832] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:16:07,854] jepsen worker 24 - jepsen.util 564 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,855] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:07,858] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,858] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,860] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:07,862] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:16:07,867] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,867] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,867] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:16:07,868] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:16:07,870] jepsen worker 5 - jepsen.util 2435 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,871] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:07,880] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,881] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:16:07,898] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,898] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,899] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:07,900] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:16:07,901] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,902] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:07,913] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,914] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:07,916] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,917] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:16:07,952] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,953] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:07,963] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:07,964] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 18] INFO [2022-03-13 17:16:08,001] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:08,001] jepsen worker nemesis - couchbase.util Rebalance check count: 476 INFO [2022-03-13 17:16:08,001] jepsen worker nemesis - couchbase.util Rebalance stuck count: 376 INFO [2022-03-13 17:16:08,006] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,007] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 6] INFO [2022-03-13 17:16:08,041] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,041] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,047] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,047] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] WARN [2022-03-13 17:16:08,048] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17717246,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18174982,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9100427,"encode_duration_us":16885,"operation_id":"0xd1ac","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10009076}],"total_count":1}} INFO [2022-03-13 17:16:08,050] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,051] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,098] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,098] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:16:08,119] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,120] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:16:08,134] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,134] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] WARN [2022-03-13 17:16:08,151] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19221299,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19221299,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9032060,"encode_duration_us":25810,"operation_id":"0xd1f9","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10009579}],"total_count":1}} INFO [2022-03-13 17:16:08,172] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,173] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,198] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,199] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:16:08,206] jepsen worker 87 - jepsen.util 2607 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,206] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,206] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,207] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,207] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:16:08,208] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,208] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:16:08,208] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,209] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,210] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,211] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,211] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] WARN [2022-03-13 17:16:08,255] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13681931,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13681931,"total_server_duration_us":1306,"operation_id":"0xd486","timeout_ms":30000,"last_server_duration_us":1306,"total_duration_us":13682163},{"operation_name":"get","last_dispatch_duration_us":12157738,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":12157738,"total_server_duration_us":1590,"operation_id":"0xd535","timeout_ms":30000,"last_server_duration_us":1590,"total_duration_us":12157944},{"operation_name":"upsert","last_dispatch_duration_us":39416,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":2076559,"total_server_duration_us":327236,"encode_duration_us":17669,"operation_id":"0xd639","timeout_ms":10000,"last_server_duration_us":28860,"total_duration_us":10010513},{"operation_name":"upsert","last_dispatch_duration_us":7029668,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8052791,"total_server_duration_us":293829,"encode_duration_us":28505,"operation_id":"0xd6e1","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10008973},{"operation_name":"upsert","last_dispatch_duration_us":7113452,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14472805,"total_server_duration_us":327866,"encode_duration_us":21416,"operation_id":"0xd3ba","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10001887},{"operation_name":"upsert","last_dispatch_duration_us":9173153,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9173153,"total_server_duration_us":9054355,"encode_duration_us":28478,"operation_id":"0xd6d4","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9176903},{"operation_name":"get","last_dispatch_duration_us":7786378,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7786378,"total_server_duration_us":38,"operation_id":"0xd7a1","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":7788981},{"operation_name":"get","last_dispatch_duration_us":3943496,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":3943496,"total_server_duration_us":9686,"operation_id":"0xd9b2","timeout_ms":30000,"last_server_duration_us":9686,"total_duration_us":3946629},{"operation_name":"upsert","last_dispatch_duration_us":1224122,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1912870,"total_server_duration_us":625540,"encode_duration_us":40028,"operation_id":"0xd817","timeout_ms":10000,"last_server_duration_us":323050,"total_duration_us":1936761},{"operation_name":"upsert","last_dispatch_duration_us":986449,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1806206,"total_server_duration_us":216054,"encode_duration_us":32273,"operation_id":"0xd826","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":1827567}],"total_count":27}} INFO [2022-03-13 17:16:08,262] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,263] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,269] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,270] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,271] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,272] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,273] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,273] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,286] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,287] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,288] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,288] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,289] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,290] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,304] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,304] jepsen worker 89 - jepsen.util 2519 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,304] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,304] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:16:08,305] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,305] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:16:08,313] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,313] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:16:08,345] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,345] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,345] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,345] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] WARN [2022-03-13 17:16:08,347] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":25184710,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":25184710,"total_server_duration_us":27,"operation_id":"0xcf58","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":25187362},{"operation_name":"upsert","last_dispatch_duration_us":38582,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8573254,"total_server_duration_us":261801,"encode_duration_us":19871,"operation_id":"0xd59c","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10008605},{"operation_name":"upsert","last_dispatch_duration_us":9630485,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9630485,"total_server_duration_us":20362,"encode_duration_us":21393,"operation_id":"0xd40d","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10007838},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":17595,"operation_id":"0xd61a","timeout_ms":10000,"total_duration_us":10005000},{"operation_name":"upsert","last_dispatch_duration_us":7461240,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7461240,"total_server_duration_us":20045,"encode_duration_us":17601,"operation_id":"0xd50f","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10002842},{"operation_name":"upsert","last_dispatch_duration_us":4136635,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4136635,"total_server_duration_us":20441,"encode_duration_us":20326,"operation_id":"0xd69b","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10001496},{"operation_name":"upsert","last_dispatch_duration_us":9072016,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9181884,"total_server_duration_us":9072770,"encode_duration_us":23678,"operation_id":"0xd4f2","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":9214227},{"operation_name":"upsert","last_dispatch_duration_us":9103079,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9103079,"total_server_duration_us":9028877,"encode_duration_us":18558,"operation_id":"0xd4d5","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":9108538},{"operation_name":"get","last_dispatch_duration_us":9084707,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9084707,"total_server_duration_us":49,"operation_id":"0xd439","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9084957},{"operation_name":"upsert","last_dispatch_duration_us":4448675,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4448675,"total_server_duration_us":220150,"encode_duration_us":19114,"operation_id":"0xda3c","timeout_ms":10000,"last_server_duration_us":220150,"total_duration_us":4452366}],"total_count":12}} WARN [2022-03-13 17:16:08,349] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":8790995,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8790995,"total_server_duration_us":10571,"operation_id":"0xd241","timeout_ms":30000,"last_server_duration_us":10571,"total_duration_us":16002971},{"operation_name":"upsert","last_dispatch_duration_us":31280,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":2524903,"total_server_duration_us":449379,"encode_duration_us":38331,"operation_id":"0xd70a","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10010180},{"operation_name":"upsert","last_dispatch_duration_us":32059,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":1536553,"total_server_duration_us":326147,"encode_duration_us":34263,"operation_id":"0xd7bd","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10008675},{"operation_name":"upsert","last_dispatch_duration_us":37166,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":2140763,"total_server_duration_us":350519,"encode_duration_us":17676,"operation_id":"0xd726","timeout_ms":10000,"last_server_duration_us":27673,"total_duration_us":10008276},{"operation_name":"upsert","last_dispatch_duration_us":34780,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":1634679,"total_server_duration_us":337864,"encode_duration_us":17619,"operation_id":"0xd7a8","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10007770},{"operation_name":"upsert","last_dispatch_duration_us":29895,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":2324629,"total_server_duration_us":347250,"encode_duration_us":20212,"operation_id":"0xd70e","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10004901},{"operation_name":"upsert","last_dispatch_duration_us":40460,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8421103,"total_server_duration_us":311537,"encode_duration_us":17777,"operation_id":"0xd3f8","timeout_ms":10000,"last_server_duration_us":29695,"total_duration_us":10002713},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":17508,"operation_id":"0xd613","timeout_ms":10000,"total_duration_us":10002087},{"operation_name":"get","last_dispatch_duration_us":8796122,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8796122,"total_server_duration_us":6827,"operation_id":"0xd4d6","timeout_ms":30000,"last_server_duration_us":6827,"total_duration_us":8798534},{"operation_name":"get","last_dispatch_duration_us":8446524,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8446524,"total_server_duration_us":6678,"operation_id":"0xd4f3","timeout_ms":30000,"last_server_duration_us":6678,"total_duration_us":8448838}],"total_count":50}} WARN [2022-03-13 17:16:08,351] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15261186,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":15261186,"total_server_duration_us":232763,"operation_id":"0xd3f6","timeout_ms":30000,"last_server_duration_us":232763,"total_duration_us":15263530},{"operation_name":"get","last_dispatch_duration_us":10689457,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10689457,"total_server_duration_us":32,"operation_id":"0xd60e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":10692135},{"operation_name":"upsert","last_dispatch_duration_us":7979424,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8382801,"total_server_duration_us":101952,"encode_duration_us":19931,"operation_id":"0xd56d","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10073190},{"operation_name":"upsert","last_dispatch_duration_us":3669345,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12260053,"total_server_duration_us":236604,"encode_duration_us":18632,"operation_id":"0xd64c","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10006207},{"operation_name":"upsert","last_dispatch_duration_us":871832,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":871832,"total_server_duration_us":42525,"encode_duration_us":22698,"operation_id":"0xd7d9","timeout_ms":10000,"last_server_duration_us":42525,"total_duration_us":10005936},{"operation_name":"upsert","last_dispatch_duration_us":7801037,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8172694,"total_server_duration_us":87198,"encode_duration_us":20900,"operation_id":"0xd4f5","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10004974},{"operation_name":"upsert","last_dispatch_duration_us":11814035,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12731440,"total_server_duration_us":4120109,"encode_duration_us":17746,"operation_id":"0xd60d","timeout_ms":10000,"last_server_duration_us":3866221,"total_duration_us":10003692},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":19137,"operation_id":"0xd7d8","timeout_ms":10000,"total_duration_us":10001380},{"operation_name":"upsert","last_dispatch_duration_us":7959885,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7959885,"total_server_duration_us":64343,"encode_duration_us":28258,"operation_id":"0xd80f","timeout_ms":10000,"last_server_duration_us":64343,"total_duration_us":7968686},{"operation_name":"get","last_dispatch_duration_us":3061715,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":3061715,"total_server_duration_us":10935,"operation_id":"0xdbe0","timeout_ms":30000,"last_server_duration_us":10935,"total_duration_us":3063778}],"total_count":11}} WARN [2022-03-13 17:16:08,355] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17647091,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21502745,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9392370,"encode_duration_us":17302,"operation_id":"0xd175","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10007070},{"operation_name":"upsert","last_dispatch_duration_us":17588088,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18689613,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9359606,"encode_duration_us":21350,"operation_id":"0xd239","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10010406}],"total_count":2}} INFO [2022-03-13 17:16:08,355] jepsen worker 86 - jepsen.util 3236 :info :write [28 21] :SyncWriteAmbiguous INFO [2022-03-13 17:16:08,356] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,356] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,356] jepsen worker 86 - jepsen.util 3326 :invoke :read [28 nil] INFO [2022-03-13 17:16:08,357] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:08,358] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:16:08,387] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,388] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:16:08,388] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,388] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,388] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,389] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 37] INFO [2022-03-13 17:16:08,390] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,391] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,392] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,393] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] WARN [2022-03-13 17:16:08,448] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":35651666,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":43668130,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9290605,"encode_duration_us":26461,"operation_id":"0xc8cb","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10008423}],"total_count":1}} INFO [2022-03-13 17:16:08,451] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,451] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] WARN [2022-03-13 17:16:08,451] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9216067,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10344730,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9202493,"encode_duration_us":16701,"operation_id":"0xd263","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10002543},{"operation_name":"upsert","last_dispatch_duration_us":11814035,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12731440,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":4120109,"encode_duration_us":17746,"operation_id":"0xd60d","timeout_ms":10000,"last_server_duration_us":3866221,"total_duration_us":10003692},{"operation_name":"upsert","last_dispatch_duration_us":17171004,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18197280,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9172237,"encode_duration_us":18575,"operation_id":"0xd255","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10003241},{"operation_name":"upsert","last_dispatch_duration_us":17201168,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18281583,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9175466,"encode_duration_us":20324,"operation_id":"0xd23d","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10008561},{"operation_name":"upsert","last_dispatch_duration_us":17076365,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18128530,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9597696,"encode_duration_us":23897,"operation_id":"0xd1f0","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10005713}],"total_count":5}} WARN [2022-03-13 17:16:08,451] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13376010,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13376010,"total_server_duration_us":6628,"operation_id":"0xd3a9","timeout_ms":30000,"last_server_duration_us":6628,"total_duration_us":13380848},{"operation_name":"upsert","last_dispatch_duration_us":32926,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8700885,"total_server_duration_us":194594,"encode_duration_us":16816,"operation_id":"0xd627","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10008680},{"operation_name":"upsert","last_dispatch_duration_us":10201539,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10201539,"total_server_duration_us":21163,"encode_duration_us":17367,"operation_id":"0xd523","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10006568},{"operation_name":"upsert","last_dispatch_duration_us":36044,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":824083,"total_server_duration_us":289727,"encode_duration_us":27069,"operation_id":"0xd73a","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10004766},{"operation_name":"get","last_dispatch_duration_us":9267275,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9267275,"total_server_duration_us":43,"operation_id":"0xd5a0","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9270739},{"operation_name":"upsert","last_dispatch_duration_us":9240161,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9240161,"total_server_duration_us":9045859,"encode_duration_us":17932,"operation_id":"0xd75c","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":9243991},{"operation_name":"upsert","last_dispatch_duration_us":1657883,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3299677,"total_server_duration_us":566881,"encode_duration_us":23745,"operation_id":"0xd768","timeout_ms":10000,"last_server_duration_us":363714,"total_duration_us":3331348},{"operation_name":"upsert","last_dispatch_duration_us":1460902,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3256506,"total_server_duration_us":202660,"encode_duration_us":18917,"operation_id":"0xd7a0","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":3286364},{"operation_name":"upsert","last_dispatch_duration_us":3086943,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3086943,"total_server_duration_us":964887,"encode_duration_us":35911,"operation_id":"0xd886","timeout_ms":10000,"last_server_duration_us":964887,"total_duration_us":3091311},{"operation_name":"upsert","last_dispatch_duration_us":2630792,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2630792,"total_server_duration_us":356705,"encode_duration_us":27845,"operation_id":"0xdb28","timeout_ms":10000,"last_server_duration_us":356705,"total_duration_us":2941101}],"total_count":58}} INFO [2022-03-13 17:16:08,471] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,472] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:16:08,479] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,480] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,524] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,524] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,525] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:16:08,525] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:16:08,563] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,564] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:16:08,569] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,569] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,570] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:16:08,571] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,615] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,615] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,616] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,616] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,619] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,620] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,659] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,660] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:16:08,702] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,703] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,707] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,707] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,709] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,710] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:08,711] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,712] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:08,714] jepsen worker 89 - jepsen.util 2519 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,714] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,715] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,716] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:16:08,722] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,723] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,760] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,760] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,760] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,761] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 30] INFO [2022-03-13 17:16:08,762] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 13] INFO [2022-03-13 17:16:08,763] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,764] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,764] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,770] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,775] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,781] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,781] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,781] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,782] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:16:08,782] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,783] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 4] INFO [2022-03-13 17:16:08,783] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,785] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,786] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,787] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:16:08,797] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,797] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:16:08,799] jepsen worker 15 - jepsen.util 3345 :ok :read [5 43] INFO [2022-03-13 17:16:08,799] jepsen worker 15 - jepsen.util 3345 :invoke :write [5 37] INFO [2022-03-13 17:16:08,802] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,803] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,807] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:08,807] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 29] INFO [2022-03-13 17:16:08,868] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,869] jepsen worker 14 - jepsen.util 374 :invoke :write [4 5] INFO [2022-03-13 17:16:08,879] jepsen worker 12 - jepsen.util 732 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,879] jepsen worker 12 - jepsen.util 732 :invoke :write [4 36] INFO [2022-03-13 17:16:08,887] jepsen worker 87 - jepsen.util 2607 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,889] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:16:08,892] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,892] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:16:08,922] jepsen worker 51 - jepsen.util 771 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,922] jepsen worker 51 - jepsen.util 771 :invoke :write [17 41] INFO [2022-03-13 17:16:08,970] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,971] jepsen worker 71 - jepsen.util 521 :invoke :write [23 9] INFO [2022-03-13 17:16:08,971] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,972] jepsen worker 37 - jepsen.util 2107 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,972] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,972] jepsen worker 89 - jepsen.util 2519 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,972] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:16:08,973] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 5] INFO [2022-03-13 17:16:08,973] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,974] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 27] INFO [2022-03-13 17:16:08,982] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,983] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,984] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,985] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:08,987] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,988] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 38] INFO [2022-03-13 17:16:08,995] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:08,997] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:16:09,007] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:09,007] jepsen worker nemesis - couchbase.util Rebalance check count: 477 INFO [2022-03-13 17:16:09,007] jepsen worker nemesis - couchbase.util Rebalance stuck count: 377 INFO [2022-03-13 17:16:09,034] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,035] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:09,036] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,036] jepsen worker 80 - jepsen.util 800 :invoke :write [26 19] INFO [2022-03-13 17:16:09,041] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,042] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:16:09,043] jepsen worker 38 - jepsen.util 2468 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,043] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:16:09,055] jepsen worker 53 - jepsen.util 503 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,056] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:09,082] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,083] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,083] jepsen worker 75 - jepsen.util 435 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,083] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 20] INFO [2022-03-13 17:16:09,084] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:16:09,084] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:16:09,121] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,122] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:09,122] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,123] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:09,144] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,145] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:16:09,155] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,156] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:16:09,172] jepsen worker 37 - jepsen.util 2107 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,172] jepsen worker 36 - jepsen.util 2196 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,172] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:09,172] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 3] INFO [2022-03-13 17:16:09,179] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,180] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:16:09,203] jepsen worker 71 - jepsen.util 521 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,204] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:16:09,215] jepsen worker 69 - jepsen.util 699 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,216] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:09,222] jepsen worker 14 - jepsen.util 374 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,222] jepsen worker 14 - jepsen.util 374 :invoke :write [4 23] INFO [2022-03-13 17:16:09,225] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,226] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:16:09,242] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,243] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:16:09,274] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,274] jepsen worker 89 - jepsen.util 2519 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,275] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,276] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,276] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,276] jepsen worker 51 - jepsen.util 771 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,277] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,277] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:16:09,278] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:09,279] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:16:09,279] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:16:09,283] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:09,298] jepsen worker 88 - jepsen.util 2068 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,299] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,308] jepsen worker 43 - jepsen.util 583 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,308] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:09,308] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,308] jepsen worker 43 - jepsen.util 583 :invoke :write [14 42] INFO [2022-03-13 17:16:09,310] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:09,311] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:09,321] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,322] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:16:09,330] jepsen worker 37 - jepsen.util 2107 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,330] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:16:09,351] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,351] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,353] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,353] jepsen worker 80 - jepsen.util 800 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,353] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,353] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,354] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,355] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:09,355] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:09,356] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:09,356] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,357] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,357] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,357] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,359] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,359] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:16:09,400] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,400] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:09,408] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,408] jepsen worker 78 - jepsen.util 618 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,408] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:16:09,410] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:09,421] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,421] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,421] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,422] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:16:09,422] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:16:09,423] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,424] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,425] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,426] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,426] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:16:09,435] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,435] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,436] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:09,437] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:16:09,463] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,464] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,467] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,468] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,469] jepsen worker 26 - jepsen.util 476 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,470] jepsen worker 26 - jepsen.util 476 :invoke :write [8 28] INFO [2022-03-13 17:16:09,471] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,471] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,478] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,478] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,479] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,480] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:16:09,499] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,499] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,499] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,499] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,500] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:09,500] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:16:09,501] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,501] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,501] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,501] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:16:09,504] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,505] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:16:09,536] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,537] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:09,566] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,566] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:16:09,567] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,568] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:16:09,574] jepsen worker 50 - jepsen.util 3290 :info :write [16 7] :RequestTimeoutException INFO [2022-03-13 17:16:09,576] jepsen worker 50 - jepsen.util 3380 :invoke :read [16 nil] INFO [2022-03-13 17:16:09,588] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,589] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,589] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,588] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,589] jepsen worker 14 - jepsen.util 374 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,589] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,589] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,590] jepsen worker 51 - jepsen.util 771 :invoke :write [17 33] INFO [2022-03-13 17:16:09,591] jepsen worker 33 - jepsen.util 753 :invoke :write [11 39] INFO [2022-03-13 17:16:09,596] jepsen worker 80 - jepsen.util 800 :invoke :write [26 44] INFO [2022-03-13 17:16:09,597] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:16:09,598] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:09,599] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,600] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:16:09,601] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:16:09,603] jepsen worker 14 - jepsen.util 374 :invoke :write [4 40] INFO [2022-03-13 17:16:09,609] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,611] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:16:09,632] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,633] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:09,635] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,636] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,641] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,642] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,643] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,644] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:16:09,666] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,666] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,666] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,667] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 46] INFO [2022-03-13 17:16:09,667] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:16:09,668] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 20] INFO [2022-03-13 17:16:09,677] jepsen worker 67 - jepsen.util 2857 :ok :read [22 30] INFO [2022-03-13 17:16:09,678] jepsen worker 67 - jepsen.util 2857 :invoke :write [22 40] INFO [2022-03-13 17:16:09,683] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,684] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,707] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,707] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 22] INFO [2022-03-13 17:16:09,726] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,726] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:16:09,768] jepsen worker 31 - jepsen.util 3631 :info :write [10 40] :RequestTimeoutException INFO [2022-03-13 17:16:09,768] jepsen worker 31 - jepsen.util 3721 :invoke :read [10 nil] INFO [2022-03-13 17:16:09,787] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,788] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 21] INFO [2022-03-13 17:16:09,790] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,791] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,794] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,794] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:16:09,812] jepsen worker 76 - jepsen.util 706 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,812] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:09,812] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,812] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,812] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,813] jepsen worker 76 - jepsen.util 706 :invoke :write [25 36] INFO [2022-03-13 17:16:09,814] jepsen worker 51 - jepsen.util 771 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,819] jepsen worker 61 - jepsen.util 691 :invoke :write [20 17] INFO [2022-03-13 17:16:09,821] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,822] jepsen worker 44 - jepsen.util 584 :invoke :write [14 4] INFO [2022-03-13 17:16:09,823] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:09,824] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,825] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:09,825] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:09,826] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:09,826] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:16:09,840] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,840] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,841] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:09,842] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:16:09,875] jepsen worker 33 - jepsen.util 753 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,875] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:09,887] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,887] jepsen worker 37 - jepsen.util 2107 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,888] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:09,888] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 39] INFO [2022-03-13 17:16:09,899] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,899] jepsen worker 4 - jepsen.util 2254 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,900] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:16:09,900] jepsen worker 6 - jepsen.util 636 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,900] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,901] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:16:09,902] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:09,902] jepsen worker 62 - jepsen.util 512 :invoke :write [20 48] INFO [2022-03-13 17:16:09,932] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,932] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:09,936] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,936] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:09,944] jepsen worker 83 - jepsen.util 2873 :info :write [27 30] :RequestTimeoutException INFO [2022-03-13 17:16:09,944] jepsen worker 83 - jepsen.util 2963 :invoke :write [27 20] INFO [2022-03-13 17:16:09,983] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:09,985] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,003] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,004] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,005] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,006] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,007] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,007] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:16:10,012] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:10,013] jepsen worker nemesis - couchbase.util Rebalance check count: 478 INFO [2022-03-13 17:16:10,013] jepsen worker nemesis - couchbase.util Rebalance stuck count: 378 INFO [2022-03-13 17:16:10,013] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,014] jepsen worker 38 - jepsen.util 2468 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,015] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:16:10,017] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,017] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:16:10,022] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:10,048] jepsen worker 80 - jepsen.util 800 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,049] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:10,070] jepsen worker 89 - jepsen.util 2519 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,070] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,071] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 48] INFO [2022-03-13 17:16:10,072] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,075] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,076] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:16:10,083] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,083] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:16:10,096] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,097] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:16:10,145] jepsen worker 37 - jepsen.util 2107 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,146] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,169] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,169] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,169] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,169] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 10] INFO [2022-03-13 17:16:10,201] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,202] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,204] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,204] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,205] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,206] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:16:10,246] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,247] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,247] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,248] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,262] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,262] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,262] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:16:10,263] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,278] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,278] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,278] jepsen worker 89 - jepsen.util 2519 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,279] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,279] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,280] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:16:10,280] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,281] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 41] INFO [2022-03-13 17:16:10,286] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,287] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:16:10,321] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,321] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,322] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:16:10,322] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 42] INFO [2022-03-13 17:16:10,332] jepsen worker 87 - jepsen.util 2607 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,333] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,333] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:16:10,334] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:16:10,374] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,374] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,394] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,395] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:16:10,416] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,416] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:16:10,424] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,424] jepsen worker 77 - jepsen.util 707 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,424] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,424] jepsen worker 37 - jepsen.util 2107 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,425] jepsen worker 60 - jepsen.util 600 :invoke :write [20 8] INFO [2022-03-13 17:16:10,425] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:10,426] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 38] INFO [2022-03-13 17:16:10,426] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,428] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,428] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:10,432] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,432] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:16:10,459] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,460] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:16:10,478] jepsen worker 89 - jepsen.util 2519 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,479] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:16:10,483] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,484] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:16:10,488] jepsen worker 41 - jepsen.util 3371 :info :write [13 22] :RequestTimeoutException INFO [2022-03-13 17:16:10,489] jepsen worker 41 - jepsen.util 3461 :invoke :write [13 42] INFO [2022-03-13 17:16:10,495] jepsen worker 14 - jepsen.util 374 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,496] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,496] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,496] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:10,497] jepsen worker 51 - jepsen.util 771 :invoke :write [17 21] INFO [2022-03-13 17:16:10,497] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:10,497] jepsen worker 40 - jepsen.util 3460 :info :write [13 29] :RequestTimeoutException INFO [2022-03-13 17:16:10,498] jepsen worker 40 - jepsen.util 3550 :invoke :write [13 35] INFO [2022-03-13 17:16:10,505] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,506] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:10,511] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,511] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 49] INFO [2022-03-13 17:16:10,528] jepsen worker 84 - jepsen.util 3684 :info :write [28 14] :RequestTimeoutException INFO [2022-03-13 17:16:10,528] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,529] jepsen worker 84 - jepsen.util 3774 :invoke :write [28 36] INFO [2022-03-13 17:16:10,535] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:16:10,577] jepsen worker 52 - jepsen.util 592 :info :write [17 35] :RequestTimeoutException INFO [2022-03-13 17:16:10,577] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:10,585] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,585] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:16:10,589] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,590] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:16:10,631] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,632] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,632] jepsen worker 8 - jepsen.util 728 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,632] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,632] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,633] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,633] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:10,634] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,635] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,635] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,635] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,636] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:10,637] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 42] INFO [2022-03-13 17:16:10,638] jepsen worker 7 - jepsen.util 637 :invoke :write [2 35] INFO [2022-03-13 17:16:10,639] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:16:10,640] jepsen worker 80 - jepsen.util 800 :invoke :write [26 3] INFO [2022-03-13 17:16:10,641] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:16:10,643] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:16:10,647] jepsen worker 8 - jepsen.util 728 :invoke :write [2 17] INFO [2022-03-13 17:16:10,653] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:10,653] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,653] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,653] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,654] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,654] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:16:10,655] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:16:10,656] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,656] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:16:10,666] jepsen worker 37 - jepsen.util 2107 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,667] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:10,672] jepsen worker 12 - jepsen.util 732 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,673] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,673] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,673] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,674] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:10,674] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,675] jepsen worker 12 - jepsen.util 732 :invoke :write [4 28] INFO [2022-03-13 17:16:10,676] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:10,676] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:10,677] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:10,708] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,708] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 9] INFO [2022-03-13 17:16:10,732] jepsen worker 88 - jepsen.util 2068 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,733] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,735] jepsen worker 43 - jepsen.util 583 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,736] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:10,757] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,757] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,757] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:10,757] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,758] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:16:10,758] jepsen worker 33 - jepsen.util 753 :invoke :write [11 1] INFO [2022-03-13 17:16:10,780] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,781] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 19] INFO [2022-03-13 17:16:10,805] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,805] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,805] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,805] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,806] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,807] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,809] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,809] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,809] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,810] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,810] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,810] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,811] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:16:10,811] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 37] INFO [2022-03-13 17:16:10,812] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,812] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,813] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,814] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,814] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,814] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:10,815] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 1] INFO [2022-03-13 17:16:10,815] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,816] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:10,817] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,818] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:16:10,820] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:16:10,833] jepsen worker 27 - jepsen.util 3807 :ok :read [9 26] INFO [2022-03-13 17:16:10,835] jepsen worker 27 - jepsen.util 3807 :invoke :write [9 45] INFO [2022-03-13 17:16:10,838] jepsen worker 3 - jepsen.util 1983 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,840] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 1] INFO [2022-03-13 17:16:10,858] jepsen worker 76 - jepsen.util 706 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,859] jepsen worker 76 - jepsen.util 706 :invoke :write [25 40] INFO [2022-03-13 17:16:10,864] jepsen worker 51 - jepsen.util 771 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,865] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:10,880] jepsen worker 5 - jepsen.util 2435 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,881] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:16:10,888] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,888] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,888] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:16:10,891] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 46] INFO [2022-03-13 17:16:10,921] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,922] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:16:10,928] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,929] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:10,960] jepsen worker 3 - jepsen.util 1983 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,961] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:16:10,963] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,963] jepsen worker 26 - jepsen.util 476 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,964] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:10,964] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:10,980] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:10,981] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:16:11,018] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:11,018] jepsen worker nemesis - couchbase.util Rebalance check count: 479 INFO [2022-03-13 17:16:11,018] jepsen worker nemesis - couchbase.util Rebalance stuck count: 379 INFO [2022-03-13 17:16:11,019] jepsen worker 88 - jepsen.util 2068 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,020] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,042] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,043] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:16:11,075] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,076] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:11,083] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,084] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:16:11,086] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,087] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:11,105] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,105] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:11,109] jepsen worker 80 - jepsen.util 800 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,109] jepsen worker 80 - jepsen.util 800 :invoke :write [26 47] INFO [2022-03-13 17:16:11,134] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,135] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:16:11,145] jepsen worker 64 - jepsen.util 3304 :ok :read [21 :nil] INFO [2022-03-13 17:16:11,146] jepsen worker 64 - jepsen.util 3304 :invoke :write [21 49] INFO [2022-03-13 17:16:11,156] jepsen worker 72 - jepsen.util 3492 :info :write [24 30] :RequestTimeoutException INFO [2022-03-13 17:16:11,157] jepsen worker 72 - jepsen.util 3582 :invoke :write [24 22] INFO [2022-03-13 17:16:11,165] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,165] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:16:11,171] jepsen worker 89 - jepsen.util 2519 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,171] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,213] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,213] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,213] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:16:11,214] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:16:11,255] jepsen worker 37 - jepsen.util 2107 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,255] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,288] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,288] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,288] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,288] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:16:11,289] jepsen worker 14 - jepsen.util 374 :invoke :write [4 33] INFO [2022-03-13 17:16:11,290] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:11,292] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,293] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 21] INFO [2022-03-13 17:16:11,295] jepsen worker 44 - jepsen.util 584 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,295] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:11,299] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,299] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,300] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,300] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:16:11,301] jepsen worker 75 - jepsen.util 435 :invoke :write [25 5] INFO [2022-03-13 17:16:11,301] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:11,301] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,302] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,302] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 38] INFO [2022-03-13 17:16:11,303] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:16:11,304] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,305] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,306] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,306] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,307] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,308] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,309] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,310] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 22] INFO [2022-03-13 17:16:11,314] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,315] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:16:11,355] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,356] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:11,364] jepsen worker 62 - jepsen.util 512 :fail :write [20 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,365] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:11,367] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,367] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:11,368] jepsen worker 53 - jepsen.util 503 :invoke :write [17 17] INFO [2022-03-13 17:16:11,368] jepsen worker 42 - jepsen.util 402 :invoke :write [14 21] INFO [2022-03-13 17:16:11,382] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,382] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,383] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,383] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,391] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,391] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,392] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,393] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 43] INFO [2022-03-13 17:16:11,393] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,395] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:11,397] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,398] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:16:11,400] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,400] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:16:11,404] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,405] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:16:11,430] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,430] jepsen worker 33 - jepsen.util 753 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,431] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,431] jepsen worker 33 - jepsen.util 753 :invoke :write [11 7] INFO [2022-03-13 17:16:11,432] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:11,434] jepsen worker 51 - jepsen.util 771 :invoke :write [17 28] INFO [2022-03-13 17:16:11,452] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,452] jepsen worker 24 - jepsen.util 564 :invoke :write [8 46] INFO [2022-03-13 17:16:11,453] jepsen worker 3 - jepsen.util 1983 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,455] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:16:11,496] jepsen worker 36 - jepsen.util 2196 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,497] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:16:11,506] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,506] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:16:11,511] jepsen worker 80 - jepsen.util 800 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,513] jepsen worker 80 - jepsen.util 800 :invoke :write [26 30] INFO [2022-03-13 17:16:11,519] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,519] jepsen worker 61 - jepsen.util 691 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,520] jepsen worker 77 - jepsen.util 707 :invoke :write [25 38] INFO [2022-03-13 17:16:11,521] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:11,532] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,533] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:16:11,547] jepsen worker 14 - jepsen.util 374 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,549] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:16:11,578] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,579] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,579] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,579] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:11,580] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:11,581] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:16:11,592] jepsen worker 75 - jepsen.util 435 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,592] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:11,598] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,598] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:11,601] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,601] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:11,604] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,605] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 18] INFO [2022-03-13 17:16:11,608] jepsen worker 51 - jepsen.util 771 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,609] jepsen worker 51 - jepsen.util 771 :invoke :write [17 19] INFO [2022-03-13 17:16:11,613] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,613] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:16:11,618] jepsen worker 89 - jepsen.util 2519 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,618] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,662] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,663] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:16:11,679] jepsen worker 37 - jepsen.util 2107 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,680] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 21] INFO [2022-03-13 17:16:11,715] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,716] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:16:11,740] jepsen worker 39 - jepsen.util 3459 :info :write [13 29] :SyncWriteAmbiguous INFO [2022-03-13 17:16:11,740] jepsen worker 39 - jepsen.util 3549 :invoke :write [13 11] INFO [2022-03-13 17:16:11,742] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,743] jepsen worker 7 - jepsen.util 637 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,743] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,743] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:11,744] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,745] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,750] jepsen worker 29 - jepsen.util 3629 :info :write [9 41] :RequestTimeoutException INFO [2022-03-13 17:16:11,751] jepsen worker 29 - jepsen.util 3719 :invoke :read [9 nil] INFO [2022-03-13 17:16:11,761] jepsen worker 33 - jepsen.util 753 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,762] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:11,763] jepsen worker 3 - jepsen.util 1983 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,764] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:11,767] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,768] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 19] INFO [2022-03-13 17:16:11,801] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,801] jepsen worker 60 - jepsen.util 600 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,801] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,801] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,802] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:11,802] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,802] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:11,803] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,804] jepsen worker 78 - jepsen.util 618 :invoke :write [26 42] INFO [2022-03-13 17:16:11,804] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,804] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,804] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:11,805] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,805] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,806] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,808] jepsen worker 69 - jepsen.util 699 :invoke :write [23 2] INFO [2022-03-13 17:16:11,810] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,810] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,811] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 3] INFO [2022-03-13 17:16:11,811] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,812] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,813] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:16:11,820] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,820] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 14] INFO [2022-03-13 17:16:11,825] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,825] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:11,834] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,835] jepsen worker 71 - jepsen.util 521 :invoke :write [23 21] INFO [2022-03-13 17:16:11,867] jepsen worker 66 - jepsen.util 3216 :info :write [22 46] :SyncWriteAmbiguous INFO [2022-03-13 17:16:11,868] jepsen worker 3 - jepsen.util 1983 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,868] jepsen worker 66 - jepsen.util 3306 :invoke :read [22 nil] INFO [2022-03-13 17:16:11,869] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:16:11,880] jepsen worker 37 - jepsen.util 2107 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,880] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,881] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,881] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,884] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,885] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,886] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,886] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,888] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,888] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,889] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,889] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:11,894] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,894] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,895] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:16:11,896] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 41] INFO [2022-03-13 17:16:11,923] jepsen worker 5 - jepsen.util 2435 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,924] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:11,934] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,935] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:16:11,950] jepsen worker 89 - jepsen.util 2519 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,950] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:16:11,958] jepsen worker 48 - jepsen.util 3378 :ok :read [16 :nil] INFO [2022-03-13 17:16:11,958] jepsen worker 48 - jepsen.util 3378 :invoke :write [16 40] INFO [2022-03-13 17:16:11,967] jepsen worker 80 - jepsen.util 800 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,968] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:11,971] jepsen worker 8 - jepsen.util 728 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,972] jepsen worker 8 - jepsen.util 728 :invoke :write [2 24] INFO [2022-03-13 17:16:11,993] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,993] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:11,996] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,997] jepsen worker 24 - jepsen.util 564 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,997] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:11,997] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 13] INFO [2022-03-13 17:16:11,998] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:16:11,999] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,025] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:12,025] jepsen worker nemesis - couchbase.util Rebalance check count: 480 INFO [2022-03-13 17:16:12,025] jepsen worker nemesis - couchbase.util Rebalance stuck count: 380 INFO [2022-03-13 17:16:12,033] jepsen worker 71 - jepsen.util 521 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,033] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,034] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,034] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,037] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,038] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:12,046] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,047] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,047] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,047] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,060] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,060] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,060] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,061] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,061] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,062] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,080] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,081] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:12,082] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,083] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:16:12,111] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,111] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,111] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,112] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 16] INFO [2022-03-13 17:16:12,112] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,113] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,113] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,114] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,114] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,114] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,115] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,115] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,115] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,116] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,116] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,117] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:16:12,117] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,118] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:16:12,140] jepsen worker 1 - jepsen.util 2881 :info :write [0 40] :RequestTimeoutException INFO [2022-03-13 17:16:12,141] jepsen worker 1 - jepsen.util 2971 :invoke :write [0 28] INFO [2022-03-13 17:16:12,181] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,181] jepsen worker 4 - jepsen.util 2254 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,181] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,182] jepsen worker 37 - jepsen.util 2107 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,181] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,183] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,184] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 6] INFO [2022-03-13 17:16:12,184] jepsen worker 3 - jepsen.util 1983 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,184] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,185] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,185] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,185] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,185] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,185] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,185] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:16:12,187] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:16:12,188] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:12,189] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,191] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,191] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:16:12,191] jepsen worker 12 - jepsen.util 732 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,192] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 38] INFO [2022-03-13 17:16:12,193] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 46] INFO [2022-03-13 17:16:12,194] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,196] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,196] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:16:12,196] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 38] INFO [2022-03-13 17:16:12,200] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:16:12,204] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,205] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:12,205] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,207] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,208] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,210] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,212] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,212] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,213] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,214] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,215] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,216] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,216] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,217] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,217] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,218] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,220] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,220] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,239] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,240] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:16:12,259] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,259] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:16:12,277] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,277] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:12,277] jepsen worker 79 - jepsen.util 1069 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,277] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,277] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,278] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:12,279] jepsen worker 6 - jepsen.util 636 :invoke :write [2 24] INFO [2022-03-13 17:16:12,279] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 37] INFO [2022-03-13 17:16:12,280] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,281] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:12,320] jepsen worker 3 - jepsen.util 1983 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,320] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:16:12,322] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,322] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,322] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,323] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:16:12,323] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:16:12,323] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:16:12,351] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,351] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:16:12,387] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,387] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,388] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,388] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,388] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:16:12,389] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,392] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,392] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:16:12,393] jepsen worker 17 - jepsen.util 3347 :info :write [5 28] :RequestTimeoutException INFO [2022-03-13 17:16:12,394] jepsen worker 17 - jepsen.util 3437 :invoke :write [5 8] INFO [2022-03-13 17:16:12,406] jepsen worker 51 - jepsen.util 771 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,407] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,407] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:12,407] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:12,410] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,410] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,410] jepsen worker 80 - jepsen.util 800 :invoke :write [26 39] INFO [2022-03-13 17:16:12,411] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:12,429] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,429] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,429] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,430] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,449] jepsen worker 37 - jepsen.util 2107 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,450] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:12,457] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,458] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,458] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 6] INFO [2022-03-13 17:16:12,458] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,459] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,460] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,460] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,460] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,461] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,461] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,462] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,462] jepsen worker 75 - jepsen.util 435 :invoke :write [25 20] INFO [2022-03-13 17:16:12,463] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,463] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,482] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,483] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,490] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,490] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,494] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,494] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:16:12,538] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,539] jepsen worker 89 - jepsen.util 2519 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,539] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,539] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 22] INFO [2022-03-13 17:16:12,561] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,562] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,564] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,565] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:16:12,606] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,606] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,606] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,607] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,607] jepsen worker 76 - jepsen.util 706 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,607] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,607] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:12,608] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,609] jepsen worker 44 - jepsen.util 584 :invoke :write [14 10] INFO [2022-03-13 17:16:12,609] jepsen worker 61 - jepsen.util 691 :invoke :write [20 8] INFO [2022-03-13 17:16:12,618] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,619] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:12,628] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,628] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 29] INFO [2022-03-13 17:16:12,629] jepsen worker 38 - jepsen.util 2468 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,631] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:16:12,641] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,642] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,648] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,649] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:12,666] jepsen worker 75 - jepsen.util 435 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,667] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,666] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,667] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,667] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,668] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,668] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,669] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,670] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:16:12,670] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:16:12,672] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,672] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:16:12,731] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,731] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:12,731] jepsen worker 4 - jepsen.util 2254 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,732] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,733] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,733] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,734] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,734] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:12,736] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,736] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,736] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,736] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,737] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:16:12,737] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:16:12,740] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,741] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 44] INFO [2022-03-13 17:16:12,748] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,749] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,750] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,751] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:12,811] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,812] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,812] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,812] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,812] jepsen worker 21 - jepsen.util 831 :invoke :write [7 44] INFO [2022-03-13 17:16:12,813] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:16:12,814] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:12,814] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:12,817] jepsen worker 53 - jepsen.util 503 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,818] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:12,818] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,819] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,827] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,827] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,830] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,830] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:16:12,841] jepsen worker 37 - jepsen.util 2107 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,842] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 18] INFO [2022-03-13 17:16:12,868] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,869] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,877] jepsen worker 80 - jepsen.util 800 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,877] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,877] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,877] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,878] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,878] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:12,878] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:12,878] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:12,879] jepsen worker 33 - jepsen.util 753 :invoke :write [11 30] INFO [2022-03-13 17:16:12,879] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,879] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:12,880] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:16:12,900] jepsen worker 89 - jepsen.util 2519 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,900] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,901] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:12,901] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:16:12,905] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,906] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:12,911] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,911] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:16:12,921] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,922] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:12,925] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,926] jepsen worker 71 - jepsen.util 521 :invoke :write [23 24] INFO [2022-03-13 17:16:12,927] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,927] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:16:12,939] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,939] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,939] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,939] jepsen worker 77 - jepsen.util 707 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,940] jepsen worker 21 - jepsen.util 831 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,939] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:12,940] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:12,941] jepsen worker 60 - jepsen.util 600 :invoke :write [20 44] INFO [2022-03-13 17:16:12,941] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:12,942] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:16:12,991] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:12,992] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:16:13,002] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,002] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,003] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:13,004] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:16:13,012] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,013] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,013] jepsen worker 42 - jepsen.util 402 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,013] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 41] INFO [2022-03-13 17:16:13,014] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:13,015] jepsen worker 7 - jepsen.util 637 :invoke :write [2 33] INFO [2022-03-13 17:16:13,028] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,028] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:16:13,029] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,029] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,029] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,029] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,029] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,031] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:13,031] jepsen worker nemesis - couchbase.util Rebalance check count: 481 INFO [2022-03-13 17:16:13,031] jepsen worker nemesis - couchbase.util Rebalance stuck count: 381 INFO [2022-03-13 17:16:13,031] jepsen worker 80 - jepsen.util 800 :invoke :write [26 28] INFO [2022-03-13 17:16:13,031] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:13,032] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:13,033] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:13,042] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:13,103] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,104] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,110] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,111] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:16:13,124] jepsen worker 4 - jepsen.util 2254 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,125] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:16:13,153] jepsen worker 37 - jepsen.util 2107 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,153] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:16:13,190] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,191] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,197] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,197] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:16:13,219] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,220] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,233] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,233] jepsen worker 71 - jepsen.util 521 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,233] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:16:13,233] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:13,235] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,235] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,235] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 38] INFO [2022-03-13 17:16:13,236] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,240] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,240] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 10] INFO [2022-03-13 17:16:13,258] jepsen worker 33 - jepsen.util 753 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,259] jepsen worker 33 - jepsen.util 753 :invoke :write [11 13] INFO [2022-03-13 17:16:13,272] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,273] jepsen worker 78 - jepsen.util 618 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,273] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,273] jepsen worker 78 - jepsen.util 618 :invoke :write [26 48] INFO [2022-03-13 17:16:13,292] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,293] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,300] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,301] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:16:13,303] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,303] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,303] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,304] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,304] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,304] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:13,305] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:16:13,306] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:13,307] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:16:13,307] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:16:13,322] jepsen worker 38 - jepsen.util 2468 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,323] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 24] INFO [2022-03-13 17:16:13,340] jepsen worker 20 - jepsen.util 3800 :info :write [6 13] :RequestTimeoutException INFO [2022-03-13 17:16:13,341] jepsen worker 20 - jepsen.util 3890 :invoke :read [6 nil] INFO [2022-03-13 17:16:13,352] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,352] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,352] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,353] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:16:13,354] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 3] INFO [2022-03-13 17:16:13,355] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,356] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,356] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,358] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,359] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,360] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,361] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,362] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,362] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:16:13,366] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,366] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:16:13,436] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,436] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,436] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:13,437] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:13,437] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,437] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:16:13,439] jepsen worker 8 - jepsen.util 728 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,439] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,439] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,440] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:13,440] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:13,440] jepsen worker 26 - jepsen.util 476 :invoke :write [8 15] INFO [2022-03-13 17:16:13,441] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:13,442] jepsen worker 43 - jepsen.util 583 :invoke :write [14 15] INFO [2022-03-13 17:16:13,461] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,461] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,461] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:13,462] jepsen worker 52 - jepsen.util 682 :invoke :write [17 2] INFO [2022-03-13 17:16:13,467] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,468] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,470] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,471] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:16:13,474] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,474] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,474] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,475] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:16:13,475] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:13,475] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:16:13,477] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,477] jepsen worker 89 - jepsen.util 2519 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,477] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,479] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 20] INFO [2022-03-13 17:16:13,479] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:16:13,480] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 31] INFO [2022-03-13 17:16:13,500] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,500] jepsen worker 69 - jepsen.util 699 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,501] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,501] jepsen worker 69 - jepsen.util 699 :invoke :write [23 36] INFO [2022-03-13 17:16:13,504] jepsen worker 12 - jepsen.util 732 :invoke :write [4 49] INFO [2022-03-13 17:16:13,506] jepsen worker 88 - jepsen.util 2068 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,506] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:13,507] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:16:13,567] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,568] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:16:13,636] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,636] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,644] jepsen worker 38 - jepsen.util 2468 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,644] jepsen worker 36 - jepsen.util 2196 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,645] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,646] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:16:13,648] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,648] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,651] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,651] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,653] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,653] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,659] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,659] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:13,661] jepsen worker 6 - jepsen.util 636 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,662] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:13,669] jepsen worker 80 - jepsen.util 800 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,669] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:13,678] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,678] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:13,702] jepsen worker 44 - jepsen.util 584 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,703] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:13,737] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,738] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:16:13,748] jepsen worker 63 - jepsen.util 3393 :info :write [21 49] :SyncWriteAmbiguous INFO [2022-03-13 17:16:13,749] jepsen worker 63 - jepsen.util 3483 :invoke :write [21 28] INFO [2022-03-13 17:16:13,757] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,758] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:16:13,814] jepsen worker 56 - jepsen.util 3476 :info :write [18 6] :SyncWriteAmbiguous INFO [2022-03-13 17:16:13,815] jepsen worker 56 - jepsen.util 3566 :invoke :write [18 20] INFO [2022-03-13 17:16:13,817] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,817] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,817] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:16:13,822] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 1] INFO [2022-03-13 17:16:13,829] jepsen worker 89 - jepsen.util 2519 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,831] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:13,829] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,829] jepsen worker 37 - jepsen.util 2107 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,831] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,831] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,832] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,832] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,833] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,834] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,842] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,842] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,843] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,844] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,844] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,844] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:16:13,849] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,849] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 25] INFO [2022-03-13 17:16:13,852] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,853] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,856] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,857] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:16:13,864] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,864] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:13,869] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,869] jepsen worker 79 - jepsen.util 1069 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,870] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:13,971] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 4] INFO [2022-03-13 17:16:13,872] jepsen worker 53 - jepsen.util 503 :invoke :write [17 47] INFO [2022-03-13 17:16:13,971] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:13,886] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,909] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,971] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:16:13,971] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:13,909] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,909] jepsen worker 61 - jepsen.util 691 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,910] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,973] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:13,973] jepsen worker 61 - jepsen.util 691 :invoke :write [20 45] INFO [2022-03-13 17:16:13,933] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,974] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 12] INFO [2022-03-13 17:16:13,976] jepsen worker 68 - jepsen.util 3668 :ok :read [22 30] INFO [2022-03-13 17:16:13,977] jepsen worker 68 - jepsen.util 3668 :invoke :read [22 nil] INFO [2022-03-13 17:16:13,933] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,934] jepsen worker 33 - jepsen.util 753 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,978] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:13,934] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,978] jepsen worker 33 - jepsen.util 753 :invoke :write [11 9] INFO [2022-03-13 17:16:13,978] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:16:13,935] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,936] jepsen worker 7 - jepsen.util 637 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,978] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:13,978] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:13,979] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,981] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:16:13,982] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,983] jepsen worker 60 - jepsen.util 600 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,983] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:13,984] jepsen worker 60 - jepsen.util 600 :invoke :write [20 25] INFO [2022-03-13 17:16:13,936] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,936] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,948] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:13,994] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:13,994] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:13,967] jepsen worker 2 - jepsen.util 3962 :ok :read [0 41] INFO [2022-03-13 17:16:13,973] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:13,994] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:13,995] jepsen worker 2 - jepsen.util 3962 :invoke :read [0 nil] INFO [2022-03-13 17:16:14,036] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,037] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:14,042] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:14,042] jepsen worker nemesis - couchbase.util Rebalance check count: 482 INFO [2022-03-13 17:16:14,042] jepsen worker nemesis - couchbase.util Rebalance stuck count: 382 INFO [2022-03-13 17:16:14,045] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,046] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:16:14,064] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,065] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,065] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,066] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:16:14,066] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,067] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:16:14,083] jepsen worker 88 - jepsen.util 2068 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,083] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:16:14,092] jepsen worker 26 - jepsen.util 476 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,093] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:14,116] jepsen worker 37 - jepsen.util 2107 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,117] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,117] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 25] INFO [2022-03-13 17:16:14,117] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:16:14,132] jepsen worker 78 - jepsen.util 618 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,132] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,133] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:14,133] jepsen worker 76 - jepsen.util 706 :invoke :write [25 9] INFO [2022-03-13 17:16:14,161] jepsen worker 89 - jepsen.util 2519 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,162] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,193] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,194] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,222] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,222] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,257] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,257] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,257] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 44] INFO [2022-03-13 17:16:14,258] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:16:14,259] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,260] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,263] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,263] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,264] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,265] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,266] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,266] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,267] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 40] INFO [2022-03-13 17:16:14,267] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:16:14,269] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,270] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:16:14,288] jepsen worker 43 - jepsen.util 583 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,288] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,288] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,288] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,288] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:14,289] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,290] jepsen worker 43 - jepsen.util 583 :invoke :write [14 47] INFO [2022-03-13 17:16:14,290] jepsen worker 62 - jepsen.util 512 :invoke :write [20 12] INFO [2022-03-13 17:16:14,291] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:14,292] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:14,293] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:14,293] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:16:14,296] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,297] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:14,300] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,300] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,300] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,301] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,301] jepsen worker 75 - jepsen.util 435 :invoke :write [25 36] INFO [2022-03-13 17:16:14,309] jepsen worker 71 - jepsen.util 521 :invoke :write [23 30] INFO [2022-03-13 17:16:14,310] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:16:14,311] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:16:14,329] jepsen worker 37 - jepsen.util 2107 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,330] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 16] INFO [2022-03-13 17:16:14,334] jepsen worker 69 - jepsen.util 699 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,334] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:14,335] jepsen worker 69 - jepsen.util 699 :invoke :write [23 26] INFO [2022-03-13 17:16:14,335] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:14,370] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,370] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,370] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:16:14,371] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:16:14,382] jepsen worker 5 - jepsen.util 2435 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,383] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,394] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,395] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:16:14,410] jepsen worker 59 - jepsen.util 3389 :info :write [19 23] :RequestTimeoutException INFO [2022-03-13 17:16:14,411] jepsen worker 59 - jepsen.util 3479 :invoke :write [19 16] INFO [2022-03-13 17:16:14,434] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,434] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:16:14,462] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,463] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:16:14,480] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,480] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,482] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,482] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,486] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,487] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 47] INFO [2022-03-13 17:16:14,494] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,495] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,496] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,496] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,498] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,498] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:16:14,545] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,545] jepsen worker 53 - jepsen.util 503 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,546] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:16:14,547] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,547] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:14,548] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:16:14,572] jepsen worker 75 - jepsen.util 435 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,573] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,574] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:16:14,575] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,576] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,577] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,579] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,579] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:16:14,600] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,600] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,604] jepsen worker 37 - jepsen.util 2107 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,605] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:14,622] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,622] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,622] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:16:14,623] jepsen worker 80 - jepsen.util 800 :invoke :write [26 41] INFO [2022-03-13 17:16:14,626] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,626] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:14,631] jepsen worker 12 - jepsen.util 732 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,631] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,631] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,632] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,632] jepsen worker 12 - jepsen.util 732 :invoke :write [4 10] INFO [2022-03-13 17:16:14,633] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:14,633] jepsen worker 70 - jepsen.util 790 :invoke :write [23 9] INFO [2022-03-13 17:16:14,634] jepsen worker 77 - jepsen.util 707 :invoke :write [25 17] INFO [2022-03-13 17:16:14,650] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,650] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,672] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,673] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:16:14,686] jepsen worker 89 - jepsen.util 2519 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,687] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,703] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,704] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:16:14,727] jepsen worker 79 - jepsen.util 1069 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,727] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,727] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:14,727] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,727] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,728] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 11] INFO [2022-03-13 17:16:14,731] jepsen worker 7 - jepsen.util 637 :invoke :write [2 9] INFO [2022-03-13 17:16:14,731] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:14,732] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:14,732] jepsen worker 6 - jepsen.util 636 :invoke :write [2 42] INFO [2022-03-13 17:16:14,750] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,751] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:16:14,763] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,763] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,763] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,763] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 27] INFO [2022-03-13 17:16:14,764] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,764] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,765] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,765] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,766] jepsen worker 65 - jepsen.util 3845 :info :write [21 32] :RequestTimeoutException INFO [2022-03-13 17:16:14,766] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,767] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,768] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,768] jepsen worker 65 - jepsen.util 3935 :invoke :write [21 41] INFO [2022-03-13 17:16:14,768] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,770] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,771] jepsen worker 71 - jepsen.util 521 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,771] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,771] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,772] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:14,773] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,773] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,774] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:16:14,774] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,775] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,776] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,776] jepsen worker 45 - jepsen.util 3555 :info :write [15 47] :RequestTimeoutException INFO [2022-03-13 17:16:14,777] jepsen worker 45 - jepsen.util 3645 :invoke :read [15 nil] INFO [2022-03-13 17:16:14,778] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:16:14,788] jepsen worker 33 - jepsen.util 753 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,788] jepsen worker 33 - jepsen.util 753 :invoke :write [11 29] INFO [2022-03-13 17:16:14,796] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,796] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,797] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:16:14,798] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 16] INFO [2022-03-13 17:16:14,807] jepsen worker 45 - jepsen.util 3645 :ok :read [15 48] INFO [2022-03-13 17:16:14,808] jepsen worker 45 - jepsen.util 3645 :invoke :read [15 nil] INFO [2022-03-13 17:16:14,821] jepsen worker 49 - jepsen.util 3379 :info :write [16 0] :RequestTimeoutException INFO [2022-03-13 17:16:14,821] jepsen worker 49 - jepsen.util 3469 :invoke :read [16 nil] INFO [2022-03-13 17:16:14,830] jepsen worker 61 - jepsen.util 691 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,831] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:14,839] jepsen worker 45 - jepsen.util 3645 :ok :read [15 48] INFO [2022-03-13 17:16:14,839] jepsen worker 45 - jepsen.util 3645 :invoke :write [15 8] INFO [2022-03-13 17:16:14,843] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,844] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,848] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,849] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] INFO [2022-03-13 17:16:14,856] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,856] jepsen worker 36 - jepsen.util 2196 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,857] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:14,857] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:14,858] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,858] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,859] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:16:14,859] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 12] INFO [2022-03-13 17:16:14,901] jepsen worker 4 - jepsen.util 2254 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,902] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:16:14,903] jepsen worker 55 - jepsen.util 3295 :ok :read [18 21] INFO [2022-03-13 17:16:14,903] jepsen worker 55 - jepsen.util 3295 :invoke :write [18 3] INFO [2022-03-13 17:16:14,911] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,912] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,913] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,914] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:16:14,927] jepsen worker 37 - jepsen.util 2107 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,928] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 16] INFO [2022-03-13 17:16:14,964] cb-events - com.couchbase.metrics [com.couchbase.metrics][LatencyMetricsAggregatedEvent][600s] Aggregated Latency Metrics: {"operations":{"kv":{"get":{"total_count":3982,"percentiles_us":{"50.0":11337.727,"90.0":503316.479,"99.0":9865003.007,"99.9":2.0803747839E7,"100.0":3.0064771071E7}},"upsert":{"total_count":3870,"percentiles_us":{"50.0":227540.991,"90.0":5368709.119,"99.0":1.0066329599E7,"99.9":1.0066329599E7,"100.0":1.0200547327E7}}}},"meta":{"emit_interval_s":600}} INFO [2022-03-13 17:16:14,984] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,984] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:14,987] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:14,987] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:16:15,005] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,005] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,018] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,019] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:16:15,036] jepsen worker 28 - jepsen.util 3538 :info :write [9 6] :RequestTimeoutException INFO [2022-03-13 17:16:15,037] jepsen worker 28 - jepsen.util 3628 :invoke :read [9 nil] INFO [2022-03-13 17:16:15,048] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:15,048] jepsen worker nemesis - couchbase.util Rebalance check count: 483 INFO [2022-03-13 17:16:15,048] jepsen worker nemesis - couchbase.util Rebalance stuck count: 383 INFO [2022-03-13 17:16:15,054] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,055] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,056] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,057] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:16:15,071] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,071] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 4] INFO [2022-03-13 17:16:15,082] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,082] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:16:15,142] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,142] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,148] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,149] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,157] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,158] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 36] INFO [2022-03-13 17:16:15,165] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,165] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,166] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,166] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,193] jepsen worker 37 - jepsen.util 2107 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,193] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 8] INFO [2022-03-13 17:16:15,203] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,203] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:16:15,259] jepsen worker 3 - jepsen.util 1983 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,259] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:16:15,265] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,265] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,265] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,265] jepsen worker 89 - jepsen.util 2519 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,265] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:16:15,266] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,267] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 48] INFO [2022-03-13 17:16:15,267] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,268] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:16:15,271] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:16:15,321] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,322] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:15,342] jepsen worker 36 - jepsen.util 2196 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,343] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:16:15,379] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,379] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,383] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,384] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,386] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,387] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,396] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,397] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:16:15,398] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,399] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,442] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,443] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,444] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,444] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,445] jepsen worker 37 - jepsen.util 2107 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,445] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,445] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,446] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 44] INFO [2022-03-13 17:16:15,446] jepsen worker 89 - jepsen.util 2519 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,446] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:16:15,447] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:16:15,451] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 31] INFO [2022-03-13 17:16:15,478] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,479] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,495] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,496] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:16:15,530] jepsen worker 0 - jepsen.util 3330 :info :write [0 18] :RequestTimeoutException INFO [2022-03-13 17:16:15,530] jepsen worker 0 - jepsen.util 3420 :invoke :write [0 29] INFO [2022-03-13 17:16:15,544] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,544] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,544] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,544] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 30] INFO [2022-03-13 17:16:15,545] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:16:15,545] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,546] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,547] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,548] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,548] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,550] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,550] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,551] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,552] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 2] INFO [2022-03-13 17:16:15,566] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,567] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,568] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,569] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:16:15,633] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,634] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:16:15,642] jepsen worker 89 - jepsen.util 2519 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,642] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,643] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,643] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:15,656] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,656] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:16:15,686] jepsen worker 34 - jepsen.util 844 :info :write [11 2] :RequestTimeoutException INFO [2022-03-13 17:16:15,687] jepsen worker 34 - jepsen.util 934 :invoke :write [11 13] INFO [2022-03-13 17:16:15,696] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,697] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:16:15,732] jepsen worker 4 - jepsen.util 2254 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,733] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:16:15,765] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,765] jepsen worker 37 - jepsen.util 2107 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,766] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,767] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:16:15,767] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,768] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:16:15,768] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 47] INFO [2022-03-13 17:16:15,768] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,771] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,772] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,772] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,773] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:16:15,774] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,775] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:15,776] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,777] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 17] INFO [2022-03-13 17:16:15,796] jepsen worker 82 - jepsen.util 3412 :info :write [27 11] :RequestTimeoutException INFO [2022-03-13 17:16:15,797] jepsen worker 82 - jepsen.util 3502 :invoke :write [27 27] INFO [2022-03-13 17:16:15,813] jepsen worker 87 - jepsen.util 2607 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,813] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:16:15,846] jepsen worker 46 - jepsen.util 3466 :info :write [15 25] :RequestTimeoutException INFO [2022-03-13 17:16:15,847] jepsen worker 46 - jepsen.util 3556 :invoke :read [15 nil] INFO [2022-03-13 17:16:15,869] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,869] jepsen worker 46 - jepsen.util 3556 :ok :read [15 48] INFO [2022-03-13 17:16:15,870] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:16:15,870] jepsen worker 46 - jepsen.util 3556 :invoke :read [15 nil] INFO [2022-03-13 17:16:15,872] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,873] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:15,895] jepsen worker 46 - jepsen.util 3556 :ok :read [15 48] INFO [2022-03-13 17:16:15,896] jepsen worker 46 - jepsen.util 3556 :invoke :read [15 nil] INFO [2022-03-13 17:16:15,920] jepsen worker 46 - jepsen.util 3556 :ok :read [15 48] INFO [2022-03-13 17:16:15,921] jepsen worker 46 - jepsen.util 3556 :invoke :write [15 43] INFO [2022-03-13 17:16:15,933] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,933] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 3] INFO [2022-03-13 17:16:15,939] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,939] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:16:15,988] jepsen worker 3 - jepsen.util 1983 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,989] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:16:15,998] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:15,998] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:16:16,026] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,026] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,027] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,027] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,054] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:16,054] jepsen worker nemesis - couchbase.util Rebalance check count: 484 INFO [2022-03-13 17:16:16,054] jepsen worker nemesis - couchbase.util Rebalance stuck count: 384 INFO [2022-03-13 17:16:16,083] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,084] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:16:16,120] jepsen worker 74 - jepsen.util 3674 :info :write [24 14] :RequestTimeoutException INFO [2022-03-13 17:16:16,121] jepsen worker 74 - jepsen.util 3764 :invoke :read [24 nil] INFO [2022-03-13 17:16:16,134] jepsen worker 54 - jepsen.util 3114 :info :write [18 37] :RequestTimeoutException INFO [2022-03-13 17:16:16,134] jepsen worker 54 - jepsen.util 3204 :invoke :read [18 nil] INFO [2022-03-13 17:16:16,135] jepsen worker 37 - jepsen.util 2107 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,136] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,137] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 1] INFO [2022-03-13 17:16:16,137] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,139] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,139] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:16,156] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,156] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,157] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,157] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,158] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,158] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,159] jepsen worker 4 - jepsen.util 2254 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,159] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,160] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,160] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,160] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,161] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,161] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,162] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,162] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,162] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,162] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,163] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 16] INFO [2022-03-13 17:16:16,163] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,163] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,164] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,164] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:16:16,165] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,166] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,167] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,168] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:16:16,201] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,201] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 18] INFO [2022-03-13 17:16:16,222] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,223] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,229] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,229] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,231] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,232] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:16:16,250] jepsen worker 19 - jepsen.util 3079 :info :write [6 2] :RequestTimeoutException INFO [2022-03-13 17:16:16,251] jepsen worker 19 - jepsen.util 3169 :invoke :write [6 27] INFO [2022-03-13 17:16:16,275] jepsen worker 88 - jepsen.util 2068 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,275] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:16:16,292] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,293] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 10] INFO [2022-03-13 17:16:16,302] jepsen worker 43 - jepsen.util 583 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,303] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:16,307] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,308] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,311] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:16,312] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,312] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,312] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:16,316] jepsen worker 30 - jepsen.util 3270 :info :write [10 43] :RequestTimeoutException INFO [2022-03-13 17:16:16,318] jepsen worker 30 - jepsen.util 3360 :invoke :write [10 14] INFO [2022-03-13 17:16:16,352] jepsen worker 60 - jepsen.util 600 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,352] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,353] jepsen worker 60 - jepsen.util 600 :invoke :write [20 41] INFO [2022-03-13 17:16:16,354] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:16,356] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,357] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 27] INFO [2022-03-13 17:16:16,382] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,384] jepsen worker 8 - jepsen.util 728 :invoke :write [2 4] INFO [2022-03-13 17:16:16,394] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,395] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 41] INFO [2022-03-13 17:16:16,422] jepsen worker 37 - jepsen.util 2107 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,422] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,423] jepsen worker 76 - jepsen.util 706 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,423] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:16,424] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,424] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:16,426] jepsen worker 44 - jepsen.util 584 :invoke :write [14 26] INFO [2022-03-13 17:16:16,426] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,427] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,428] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,429] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,429] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,429] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,430] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:16:16,436] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,436] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,436] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,437] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:16,437] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:16,437] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:16:16,439] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,440] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:16:16,443] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,443] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,444] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] INFO [2022-03-13 17:16:16,444] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:16:16,454] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,455] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:16:16,459] jepsen worker 62 - jepsen.util 512 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,461] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:16,464] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,465] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:16,465] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,465] jepsen worker 53 - jepsen.util 503 :invoke :write [17 21] INFO [2022-03-13 17:16:16,465] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:16,466] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:16,482] jepsen worker 89 - jepsen.util 2519 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,482] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,483] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,484] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,484] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:16:16,485] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:16,519] jepsen worker 3 - jepsen.util 1983 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,520] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:16:16,539] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,539] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,540] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:16,540] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:16:16,547] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,548] jepsen worker 69 - jepsen.util 699 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,548] jepsen worker 23 - jepsen.util 563 :invoke :write [7 10] INFO [2022-03-13 17:16:16,550] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:16,560] cb-events - com.couchbase.metrics [com.couchbase.metrics][LatencyMetricsAggregatedEvent][600s] Aggregated Latency Metrics: {"operations":{"kv":{"get":{"total_count":3442,"percentiles_us":{"50.0":88604.671,"90.0":918552.575,"99.0":1.3019119615E7,"99.9":2.2414360575E7,"100.0":2.6306674687E7}},"upsert":{"total_count":3527,"percentiles_us":{"50.0":266338.303,"90.0":9462349.823,"99.0":1.0066329599E7,"99.9":1.0066329599E7,"100.0":1.0200547327E7}}}},"meta":{"emit_interval_s":600}} INFO [2022-03-13 17:16:16,560] cb-events - com.couchbase.metrics [com.couchbase.metrics][LatencyMetricsAggregatedEvent][600s] Aggregated Latency Metrics: {"operations":{"kv":{"get":{"total_count":2497,"percentiles_us":{"50.0":214958.079,"90.0":3523215.359,"99.0":1.6173236223E7,"99.9":2.8051505151E7,"100.0":3.0064771071E7}},"upsert":{"total_count":2358,"percentiles_us":{"50.0":905969.663,"90.0":1.0066329599E7,"99.0":1.0066329599E7,"99.9":1.0066329599E7,"100.0":1.0066329599E7}}}},"meta":{"emit_interval_s":600}} INFO [2022-03-13 17:16:16,566] cb-events - com.couchbase.metrics [com.couchbase.metrics][LatencyMetricsAggregatedEvent][600s] Aggregated Latency Metrics: {"operations":{"kv":{"get":{"total_count":2412,"percentiles_us":{"50.0":373293.055,"90.0":3170893.823,"99.0":1.2146704383E7,"99.9":1.7045651455E7,"100.0":2.2548578303E7}},"upsert":{"total_count":2396,"percentiles_us":{"50.0":1216348.159,"90.0":1.0066329599E7,"99.0":1.0066329599E7,"99.9":1.0066329599E7,"100.0":1.0066329599E7}}}},"meta":{"emit_interval_s":600}} INFO [2022-03-13 17:16:16,581] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,582] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:16:16,588] jepsen worker 79 - jepsen.util 1069 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,589] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 8] INFO [2022-03-13 17:16:16,616] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,616] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 1] INFO [2022-03-13 17:16:16,618] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,618] jepsen worker 36 - jepsen.util 2196 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,619] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,620] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:16:16,621] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,621] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,622] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,623] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,623] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:16:16,623] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,625] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:16:16,626] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:16:16,631] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,632] jepsen worker 13 - jepsen.util 733 :invoke :write [4 22] INFO [2022-03-13 17:16:16,639] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,639] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,649] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,649] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,649] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:16:16,649] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:16:16,666] cb-events - com.couchbase.metrics [com.couchbase.metrics][LatencyMetricsAggregatedEvent][600s] Aggregated Latency Metrics: {"operations":{"kv":{"get":{"total_count":2141,"percentiles_us":{"50.0":304087.039,"90.0":2030043.135,"99.0":1.4092861439E7,"99.9":1.9595788287E7,"100.0":2.4293408767E7}},"upsert":{"total_count":1983,"percentiles_us":{"50.0":884998.143,"90.0":1.0066329599E7,"99.0":1.0066329599E7,"99.9":1.0066329599E7,"100.0":1.0133438463E7}}}},"meta":{"emit_interval_s":600}} WARN [2022-03-13 17:16:16,666] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17998465,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17998465,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9035244,"encode_duration_us":19991,"operation_id":"0xd7ae","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10001813}],"total_count":1}} WARN [2022-03-13 17:16:16,667] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":69073,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9754763,"total_server_duration_us":212049,"encode_duration_us":27818,"operation_id":"0xd786","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10007078},{"operation_name":"upsert","last_dispatch_duration_us":32002,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":550570,"total_server_duration_us":288816,"encode_duration_us":34775,"operation_id":"0xd814","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10006258},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17575,"operation_id":"0xdbfb","timeout_ms":10000,"total_duration_us":10005231},{"operation_name":"upsert","last_dispatch_duration_us":6951943,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7224347,"total_server_duration_us":134480,"encode_duration_us":33706,"operation_id":"0xdb1a","timeout_ms":10000,"last_server_duration_us":46287,"total_duration_us":10004948},{"operation_name":"upsert","last_dispatch_duration_us":17998465,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":17998465,"total_server_duration_us":9035244,"encode_duration_us":19991,"operation_id":"0xd7ae","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10001813},{"operation_name":"upsert","last_dispatch_duration_us":40715,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":581940,"total_server_duration_us":254869,"encode_duration_us":17859,"operation_id":"0xdc9f","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10001235},{"operation_name":"upsert","last_dispatch_duration_us":9055036,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9055036,"total_server_duration_us":9041612,"encode_duration_us":17201,"operation_id":"0xd8c9","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9059133},{"operation_name":"upsert","last_dispatch_duration_us":2286564,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2286564,"total_server_duration_us":1492999,"encode_duration_us":39509,"operation_id":"0xe117","timeout_ms":10000,"last_server_duration_us":1492999,"total_duration_us":2290083},{"operation_name":"upsert","last_dispatch_duration_us":2090473,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2090473,"total_server_duration_us":62668,"encode_duration_us":30719,"operation_id":"0xe138","timeout_ms":10000,"last_server_duration_us":62668,"total_duration_us":2097271},{"operation_name":"upsert","last_dispatch_duration_us":1967191,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1967191,"total_server_duration_us":62668,"encode_duration_us":20309,"operation_id":"0xe15d","timeout_ms":10000,"last_server_duration_us":62668,"total_duration_us":1971943}],"total_count":56}} INFO [2022-03-13 17:16:16,672] cb-events - com.couchbase.metrics [com.couchbase.metrics][LatencyMetricsAggregatedEvent][600s] Aggregated Latency Metrics: {"operations":{"kv":{"get":{"total_count":4096,"percentiles_us":{"50.0":98041.855,"90.0":603979.775,"99.0":1.2683575295E7,"99.9":1.8119393279E7,"100.0":2.3353884671E7}},"upsert":{"total_count":3998,"percentiles_us":{"50.0":299892.735,"90.0":5972688.895,"99.0":1.0066329599E7,"99.9":1.0066329599E7,"100.0":1.0200547327E7}}}},"meta":{"emit_interval_s":600}} INFO [2022-03-13 17:16:16,672] jepsen worker 80 - jepsen.util 800 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,674] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:16,705] jepsen worker 12 - jepsen.util 732 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,705] jepsen worker 12 - jepsen.util 732 :invoke :write [4 5] INFO [2022-03-13 17:16:16,715] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,716] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,720] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,720] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:16,724] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,724] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,725] jepsen worker 87 - jepsen.util 2607 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,726] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:16:16,734] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,735] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,736] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,737] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,739] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,740] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,741] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,741] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,743] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,744] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,745] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,746] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 9] INFO [2022-03-13 17:16:16,770] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,770] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:16,781] jepsen worker 73 - jepsen.util 3583 :info :write [24 26] :RequestTimeoutException INFO [2022-03-13 17:16:16,781] jepsen worker 73 - jepsen.util 3673 :invoke :read [24 nil] INFO [2022-03-13 17:16:16,797] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,797] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:16:16,819] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,820] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:16,821] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,822] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:16:16,824] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,824] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:16,842] jepsen worker 7 - jepsen.util 637 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,843] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,843] jepsen worker 7 - jepsen.util 637 :invoke :write [2 2] INFO [2022-03-13 17:16:16,845] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:16:16,867] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,868] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,868] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 37] INFO [2022-03-13 17:16:16,869] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:16:16,907] jepsen worker 3 - jepsen.util 1983 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,908] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:16:16,915] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,916] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:16:16,988] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,989] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:16:16,994] jepsen worker 85 - jepsen.util 3145 :ok :read [28 44] INFO [2022-03-13 17:16:16,995] jepsen worker 85 - jepsen.util 3145 :invoke :read [28 nil] INFO [2022-03-13 17:16:16,996] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,996] jepsen worker 33 - jepsen.util 753 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:16,997] jepsen worker 33 - jepsen.util 753 :invoke :write [11 37] INFO [2022-03-13 17:16:16,997] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:17,015] jepsen worker 37 - jepsen.util 2107 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,015] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,016] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 33] INFO [2022-03-13 17:16:17,017] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,029] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,029] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:16:17,047] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,048] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,048] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,048] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:17,049] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:17,049] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:16:17,051] jepsen worker 77 - jepsen.util 707 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,051] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:17,058] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,058] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,059] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:17,060] jepsen worker 18 - jepsen.util 3348 :info :write [6 17] :RequestTimeoutException INFO [2022-03-13 17:16:17,060] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:16:17,060] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:17,060] jepsen worker nemesis - couchbase.util Rebalance check count: 485 INFO [2022-03-13 17:16:17,060] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,060] jepsen worker nemesis - couchbase.util Rebalance stuck count: 385 INFO [2022-03-13 17:16:17,061] jepsen worker 18 - jepsen.util 3438 :invoke :write [6 44] INFO [2022-03-13 17:16:17,061] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:16:17,070] jepsen worker 52 - jepsen.util 682 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,071] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,071] jepsen worker 52 - jepsen.util 682 :invoke :write [17 5] INFO [2022-03-13 17:16:17,072] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:16:17,074] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,076] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:16:17,087] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,087] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,088] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:16:17,089] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:16:17,102] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,102] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,103] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:17,104] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:17,104] jepsen worker 6 - jepsen.util 636 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,104] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:17,105] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:17,106] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:17,131] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,132] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:16:17,150] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,150] jepsen worker 24 - jepsen.util 564 :invoke :write [8 29] INFO [2022-03-13 17:16:17,156] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,156] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,173] jepsen worker 47 - jepsen.util 3377 :info :write [15 14] :RequestTimeoutException INFO [2022-03-13 17:16:17,174] jepsen worker 47 - jepsen.util 3467 :invoke :write [15 39] INFO [2022-03-13 17:16:17,198] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,199] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,200] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,201] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,208] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,208] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,210] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,210] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:16:17,211] jepsen worker 89 - jepsen.util 2519 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,212] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,212] jepsen worker 70 - jepsen.util 790 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,212] jepsen worker 70 - jepsen.util 790 :invoke :write [23 5] INFO [2022-03-13 17:16:17,212] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,213] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:17,221] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,222] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,223] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,223] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,224] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:16:17,225] jepsen worker 80 - jepsen.util 800 :invoke :write [26 30] INFO [2022-03-13 17:16:17,261] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,262] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:16:17,264] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,265] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:16:17,278] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,279] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,278] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,279] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,280] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:16:17,286] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,287] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,287] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,287] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:17,287] jepsen worker 8 - jepsen.util 728 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,287] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:16:17,288] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 45] INFO [2022-03-13 17:16:17,289] jepsen worker 42 - jepsen.util 402 :invoke :write [14 10] INFO [2022-03-13 17:16:17,290] jepsen worker 62 - jepsen.util 512 :invoke :write [20 32] INFO [2022-03-13 17:16:17,292] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:17,293] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:17,302] jepsen worker 37 - jepsen.util 2107 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,302] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,307] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,308] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,310] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,311] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,315] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,317] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:16:17,331] jepsen worker 34 - jepsen.util 934 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,331] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:17,342] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,343] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:16:17,365] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,365] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:17,387] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,388] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,417] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,418] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,427] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,427] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:17,442] jepsen worker 88 - jepsen.util 2068 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,442] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,456] jepsen worker 10 - jepsen.util 3430 :info :write [3 49] :RequestTimeoutException INFO [2022-03-13 17:16:17,457] jepsen worker 10 - jepsen.util 3520 :invoke :write [3 32] INFO [2022-03-13 17:16:17,457] jepsen worker 24 - jepsen.util 564 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,457] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:16:17,459] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,459] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,459] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:16:17,460] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:16:17,499] jepsen worker 60 - jepsen.util 600 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,499] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:17,500] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,501] jepsen worker 76 - jepsen.util 706 :invoke :write [25 16] INFO [2022-03-13 17:16:17,517] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,518] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:17,521] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,521] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,522] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:17,523] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:16:17,529] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,530] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,534] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,534] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,534] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,535] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,536] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:16:17,536] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,536] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:16:17,537] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,538] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,539] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,540] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,542] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,542] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 43] INFO [2022-03-13 17:16:17,543] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 24] INFO [2022-03-13 17:16:17,554] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,554] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,556] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:16:17,558] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 33] INFO [2022-03-13 17:16:17,585] jepsen worker 53 - jepsen.util 503 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,586] jepsen worker 53 - jepsen.util 503 :invoke :write [17 27] INFO [2022-03-13 17:16:17,632] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,633] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,635] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 25] INFO [2022-03-13 17:16:17,639] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:16:17,647] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,648] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:17,660] jepsen worker 44 - jepsen.util 584 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,662] jepsen worker 44 - jepsen.util 584 :invoke :write [14 2] INFO [2022-03-13 17:16:17,671] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,671] jepsen worker 33 - jepsen.util 753 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,671] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,671] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:17,672] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:17,672] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:16:17,683] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,683] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,710] jepsen worker 3 - jepsen.util 1983 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,710] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:16:17,721] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,724] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 42] INFO [2022-03-13 17:16:17,766] jepsen worker 81 - jepsen.util 3861 :info :write [27 18] :RequestTimeoutException INFO [2022-03-13 17:16:17,767] jepsen worker 81 - jepsen.util 3951 :invoke :write [27 33] INFO [2022-03-13 17:16:17,776] jepsen worker 37 - jepsen.util 2107 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,777] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,784] jepsen worker 5 - jepsen.util 2435 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,785] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:16:17,786] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,787] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,800] jepsen worker 57 - jepsen.util 3477 :info :write [19 15] :RequestTimeoutException INFO [2022-03-13 17:16:17,801] jepsen worker 57 - jepsen.util 3567 :invoke :read [19 nil] INFO [2022-03-13 17:16:17,848] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,849] jepsen worker 79 - jepsen.util 1069 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,849] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:17,850] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 24] INFO [2022-03-13 17:16:17,861] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,862] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:16:17,873] jepsen worker 4 - jepsen.util 2254 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,873] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,874] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,874] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,875] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,875] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:16:17,876] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:16:17,877] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:16:17,907] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,907] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,908] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,909] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,911] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,911] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,912] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 19] INFO [2022-03-13 17:16:17,913] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,914] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,915] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,915] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,916] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,916] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,917] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,917] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,918] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,919] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,919] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:17,920] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,921] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,922] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,923] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,925] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,926] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 48] INFO [2022-03-13 17:16:17,935] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,935] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,936] jepsen worker 71 - jepsen.util 521 :invoke :write [23 17] INFO [2022-03-13 17:16:17,936] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:17,937] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,938] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:16:17,938] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,938] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,939] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:16:17,940] jepsen worker 24 - jepsen.util 564 :invoke :write [8 26] INFO [2022-03-13 17:16:17,943] jepsen worker 13 - jepsen.util 733 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,944] jepsen worker 23 - jepsen.util 563 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,944] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,944] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:17,945] jepsen worker 69 - jepsen.util 699 :invoke :write [23 16] INFO [2022-03-13 17:16:17,946] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:16:17,971] jepsen worker 89 - jepsen.util 2519 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,971] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,972] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:17,973] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:17,982] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:17,982] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:16:18,003] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,003] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,022] jepsen worker 7 - jepsen.util 637 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,022] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,023] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:18,022] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,023] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,023] jepsen worker 6 - jepsen.util 636 :invoke :write [2 7] INFO [2022-03-13 17:16:18,024] jepsen worker 26 - jepsen.util 476 :invoke :write [8 41] INFO [2022-03-13 17:16:18,024] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:18,026] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:18,027] jepsen worker 43 - jepsen.util 583 :invoke :write [14 32] INFO [2022-03-13 17:16:18,036] jepsen worker 80 - jepsen.util 800 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,036] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,037] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,038] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:16:18,038] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:18,047] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:18,048] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,049] jepsen worker 5 - jepsen.util 2435 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,050] jepsen worker 13 - jepsen.util 733 :invoke :write [4 40] INFO [2022-03-13 17:16:18,051] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,053] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,054] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:16:18,063] jepsen worker 71 - jepsen.util 521 :fail :write [23 17] :ServerOutOfMemoryException WARN [2022-03-13 17:16:18,063] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9047692,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11372322,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9384616,"encode_duration_us":20212,"operation_id":"0xd70e","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10004901},{"operation_name":"upsert","last_dispatch_duration_us":17366327,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17507587,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9066221,"encode_duration_us":23138,"operation_id":"0xd1e4","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10008499}],"total_count":2}} INFO [2022-03-13 17:16:18,064] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:18,068] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:18,068] jepsen worker nemesis - couchbase.util Rebalance check count: 486 INFO [2022-03-13 17:16:18,068] jepsen worker nemesis - couchbase.util Rebalance stuck count: 386 INFO [2022-03-13 17:16:18,080] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,081] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 16] INFO [2022-03-13 17:16:18,109] jepsen worker 52 - jepsen.util 682 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,110] jepsen worker 52 - jepsen.util 682 :invoke :write [17 1] INFO [2022-03-13 17:16:18,118] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,118] jepsen worker 12 - jepsen.util 732 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,119] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,119] jepsen worker 12 - jepsen.util 732 :invoke :write [4 24] INFO [2022-03-13 17:16:18,119] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,119] jepsen worker 37 - jepsen.util 2107 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,119] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,119] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,120] jepsen worker 78 - jepsen.util 618 :invoke :write [26 34] INFO [2022-03-13 17:16:18,121] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:18,122] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 47] INFO [2022-03-13 17:16:18,123] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,124] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 8] INFO [2022-03-13 17:16:18,124] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 0] INFO [2022-03-13 17:16:18,125] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 41] INFO [2022-03-13 17:16:18,125] jepsen worker 77 - jepsen.util 707 :invoke :write [25 43] INFO [2022-03-13 17:16:18,134] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,135] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,135] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,136] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:16:18,142] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,144] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] WARN [2022-03-13 17:16:18,168] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17285806,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18109889,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9337710,"encode_duration_us":27069,"operation_id":"0xd73a","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10004766}],"total_count":1}} INFO [2022-03-13 17:16:18,190] jepsen worker 87 - jepsen.util 2607 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,190] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,190] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,191] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,219] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,220] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,221] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,222] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:16:18,264] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,264] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,264] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,265] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,265] jepsen worker 89 - jepsen.util 2519 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,265] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,265] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:16:18,266] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,266] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:16:18,267] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,267] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,268] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] WARN [2022-03-13 17:16:18,269] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":17948,"operation_id":"0xda6e","timeout_ms":10000,"total_duration_us":10010607},{"operation_name":"upsert","last_dispatch_duration_us":9122029,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9122029,"total_server_duration_us":19809,"encode_duration_us":17834,"operation_id":"0xdc3e","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10009736},{"operation_name":"upsert","last_dispatch_duration_us":18143429,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":18143429,"total_server_duration_us":9032060,"encode_duration_us":18090,"operation_id":"0xd8fe","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10007693},{"operation_name":"upsert","last_dispatch_duration_us":8775661,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8775661,"total_server_duration_us":19887,"encode_duration_us":23776,"operation_id":"0xdbb2","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10007344},{"operation_name":"upsert","last_dispatch_duration_us":33054,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7630227,"total_server_duration_us":302015,"encode_duration_us":21169,"operation_id":"0xdd33","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10006837},{"operation_name":"upsert","last_dispatch_duration_us":33526,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7997955,"total_server_duration_us":302986,"encode_duration_us":19336,"operation_id":"0xdc91","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10004499},{"operation_name":"upsert","last_dispatch_duration_us":9080848,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9665389,"total_server_duration_us":9383273,"encode_duration_us":21273,"operation_id":"0xda29","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":10003141},{"operation_name":"get","last_dispatch_duration_us":9112129,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9112129,"total_server_duration_us":38,"operation_id":"0xd910","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9114000},{"operation_name":"upsert","last_dispatch_duration_us":3566038,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3566038,"total_server_duration_us":73136,"encode_duration_us":36049,"operation_id":"0xe0be","timeout_ms":10000,"last_server_duration_us":73136,"total_duration_us":3571929},{"operation_name":"get","last_dispatch_duration_us":3391405,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3391405,"total_server_duration_us":38,"operation_id":"0xe0e7","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":3391582}],"total_count":15}} INFO [2022-03-13 17:16:18,269] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,269] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,269] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,270] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,270] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,271] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:16:18,272] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,272] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,274] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:16:18,274] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11100836,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11100836,"total_server_duration_us":6048,"operation_id":"0xdadf","timeout_ms":30000,"last_server_duration_us":6048,"total_duration_us":11103021},{"operation_name":"upsert","last_dispatch_duration_us":8245135,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8777623,"total_server_duration_us":268800,"encode_duration_us":19121,"operation_id":"0xdbf2","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10008844},{"operation_name":"upsert","last_dispatch_duration_us":7611655,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7611655,"total_server_duration_us":20601,"encode_duration_us":19902,"operation_id":"0xdca9","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10007142},{"operation_name":"upsert","last_dispatch_duration_us":31691,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":505990,"total_server_duration_us":207716,"encode_duration_us":29303,"operation_id":"0xdbef","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10005123},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17722,"operation_id":"0xdd5d","timeout_ms":10000,"total_duration_us":10001171},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17358,"operation_id":"0xdc73","timeout_ms":10000,"total_duration_us":10001067},{"operation_name":"get","last_dispatch_duration_us":9394219,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9394219,"total_server_duration_us":56,"operation_id":"0xdbca","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9396217},{"operation_name":"upsert","last_dispatch_duration_us":9082193,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9082193,"total_server_duration_us":9051168,"encode_duration_us":20645,"operation_id":"0xdbe2","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9085794},{"operation_name":"upsert","last_dispatch_duration_us":9065924,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9065924,"total_server_duration_us":9053293,"encode_duration_us":22149,"operation_id":"0xdbed","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9069787},{"operation_name":"upsert","last_dispatch_duration_us":1716353,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1716353,"total_server_duration_us":1503873,"encode_duration_us":16717,"operation_id":"0xe166","timeout_ms":10000,"last_server_duration_us":1503873,"total_duration_us":1720076}],"total_count":21}} WARN [2022-03-13 17:16:18,274] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16090691,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18167251,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9359296,"encode_duration_us":17669,"operation_id":"0xd639","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10010513}],"total_count":1}} INFO [2022-03-13 17:16:18,274] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:16:18,295] jepsen worker 62 - jepsen.util 512 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,296] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:16:18,297] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,298] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,298] jepsen worker 24 - jepsen.util 564 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,298] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,298] jepsen worker 14 - jepsen.util 374 :invoke :write [4 35] INFO [2022-03-13 17:16:18,299] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:18,301] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:18,301] jepsen worker 24 - jepsen.util 564 :invoke :write [8 48] INFO [2022-03-13 17:16:18,317] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,318] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:16:18,357] jepsen worker 37 - jepsen.util 2107 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,357] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] WARN [2022-03-13 17:16:18,364] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9078160,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9078160,"total_server_duration_us":25450,"encode_duration_us":21731,"operation_id":"0xd97b","timeout_ms":10000,"last_server_duration_us":25450,"total_duration_us":10010297},{"operation_name":"upsert","last_dispatch_duration_us":9043623,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9043623,"total_server_duration_us":18876,"encode_duration_us":17753,"operation_id":"0xd976","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10007958},{"operation_name":"upsert","last_dispatch_duration_us":8137029,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8730492,"total_server_duration_us":320934,"encode_duration_us":20675,"operation_id":"0xd91d","timeout_ms":10000,"last_server_duration_us":27402,"total_duration_us":10003576},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":18807,"operation_id":"0xd986","timeout_ms":10000,"total_duration_us":10001670},{"operation_name":"upsert","last_dispatch_duration_us":3221996,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3221996,"total_server_duration_us":495602,"encode_duration_us":27154,"operation_id":"0xda60","timeout_ms":10000,"last_server_duration_us":495602,"total_duration_us":8396174},{"operation_name":"get","last_dispatch_duration_us":7636011,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":7636011,"total_server_duration_us":7231,"operation_id":"0xda5f","timeout_ms":30000,"last_server_duration_us":7231,"total_duration_us":7638436},{"operation_name":"upsert","last_dispatch_duration_us":2956155,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2956155,"total_server_duration_us":61901,"encode_duration_us":23115,"operation_id":"0xda81","timeout_ms":10000,"last_server_duration_us":61901,"total_duration_us":7104184},{"operation_name":"upsert","last_dispatch_duration_us":3316497,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3316497,"total_server_duration_us":1256367,"encode_duration_us":19445,"operation_id":"0xdb86","timeout_ms":10000,"last_server_duration_us":1256367,"total_duration_us":4915165},{"operation_name":"get","last_dispatch_duration_us":2753633,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2753633,"total_server_duration_us":38,"operation_id":"0xdc0d","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":4858827},{"operation_name":"get","last_dispatch_duration_us":136954,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":136954,"total_server_duration_us":69,"operation_id":"0xdc8b","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":4796448}],"total_count":76}} INFO [2022-03-13 17:16:18,365] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,366] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] WARN [2022-03-13 17:16:18,369] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9080848,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9665389,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9383273,"encode_duration_us":21273,"operation_id":"0xda29","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":10003141},{"operation_name":"upsert","last_dispatch_duration_us":14126155,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14126155,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9045859,"encode_duration_us":17595,"operation_id":"0xd61a","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10005000},{"operation_name":"upsert","last_dispatch_duration_us":18143429,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18143429,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9032060,"encode_duration_us":18090,"operation_id":"0xd8fe","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10007693}],"total_count":3}} WARN [2022-03-13 17:16:18,369] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18367,"operation_id":"0xdc7c","timeout_ms":10000,"total_duration_us":10010215},{"operation_name":"upsert","last_dispatch_duration_us":12242499,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":12242499,"total_server_duration_us":185545,"encode_duration_us":18053,"operation_id":"0xd98d","timeout_ms":10000,"last_server_duration_us":185545,"total_duration_us":10008850},{"operation_name":"upsert","last_dispatch_duration_us":10947210,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":10947210,"total_server_duration_us":78555,"encode_duration_us":17514,"operation_id":"0xdc62","timeout_ms":10000,"last_server_duration_us":78555,"total_duration_us":10006137},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":20839,"operation_id":"0xdc15","timeout_ms":10000,"total_duration_us":10003908},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":21217,"operation_id":"0xd9db","timeout_ms":10000,"total_duration_us":10001224},{"operation_name":"upsert","last_dispatch_duration_us":31637,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":606723,"total_server_duration_us":385923,"encode_duration_us":19125,"operation_id":"0xdd8d","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10001035},{"operation_name":"get","last_dispatch_duration_us":9182644,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":9182644,"total_server_duration_us":32,"operation_id":"0xdd5a","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9182929},{"operation_name":"upsert","last_dispatch_duration_us":1272387,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1272387,"total_server_duration_us":83558,"encode_duration_us":32460,"operation_id":"0xe27d","timeout_ms":10000,"last_server_duration_us":83558,"total_duration_us":1278336}],"total_count":8}} INFO [2022-03-13 17:16:18,372] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,372] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,373] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:18,374] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:18,407] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,408] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:16:18,415] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,415] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:16:18,421] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,421] jepsen worker 70 - jepsen.util 790 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,421] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,421] jepsen worker 61 - jepsen.util 691 :invoke :write [20 33] INFO [2022-03-13 17:16:18,422] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:18,422] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:18,439] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,440] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] WARN [2022-03-13 17:16:18,469] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":32387,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":942877,"total_server_duration_us":190630,"encode_duration_us":17701,"operation_id":"0xda86","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10010415},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17727,"operation_id":"0xdc94","timeout_ms":10000,"total_duration_us":10008690},{"operation_name":"upsert","last_dispatch_duration_us":8279105,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11623892,"total_server_duration_us":212473,"encode_duration_us":18056,"operation_id":"0xd93b","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10003599},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":23534,"operation_id":"0xdd41","timeout_ms":10000,"total_duration_us":10000662},{"operation_name":"get","last_dispatch_duration_us":9357838,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9357838,"total_server_duration_us":27,"operation_id":"0xdc3f","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9359922},{"operation_name":"upsert","last_dispatch_duration_us":9153666,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9265532,"total_server_duration_us":9047940,"encode_duration_us":18804,"operation_id":"0xdabd","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":9280112},{"operation_name":"upsert","last_dispatch_duration_us":9120974,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9120974,"total_server_duration_us":9025695,"encode_duration_us":18906,"operation_id":"0xdac0","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9126118},{"operation_name":"get","last_dispatch_duration_us":7445722,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7445722,"total_server_duration_us":92,"operation_id":"0xdbbf","timeout_ms":30000,"last_server_duration_us":92,"total_duration_us":7446036},{"operation_name":"upsert","last_dispatch_duration_us":3164917,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3164917,"total_server_duration_us":183922,"encode_duration_us":37193,"operation_id":"0xdcb1","timeout_ms":10000,"last_server_duration_us":183922,"total_duration_us":4250515},{"operation_name":"upsert","last_dispatch_duration_us":3290640,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3290640,"total_server_duration_us":73136,"encode_duration_us":19442,"operation_id":"0xdce0","timeout_ms":10000,"last_server_duration_us":73136,"total_duration_us":3298988}],"total_count":72}} WARN [2022-03-13 17:16:18,469] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16595164,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16595164,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9037366,"encode_duration_us":19137,"operation_id":"0xd7d8","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10001380},{"operation_name":"upsert","last_dispatch_duration_us":10947210,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10947210,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":78555,"encode_duration_us":17514,"operation_id":"0xdc62","timeout_ms":10000,"last_server_duration_us":78555,"total_duration_us":10006137},{"operation_name":"upsert","last_dispatch_duration_us":10646462,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11518295,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":107388,"encode_duration_us":22698,"operation_id":"0xd7d9","timeout_ms":10000,"last_server_duration_us":64863,"total_duration_us":10005936},{"operation_name":"upsert","last_dispatch_duration_us":12242499,"last_local_id":"A312307A00000006/0000000066A14478","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12242499,"last_local_socket":"172.23.104.79:37416","total_server_duration_us":185545,"encode_duration_us":18053,"operation_id":"0xd98d","timeout_ms":10000,"last_server_duration_us":185545,"total_duration_us":10008850}],"total_count":4}} INFO [2022-03-13 17:16:18,477] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,478] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,479] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,479] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,508] jepsen worker 42 - jepsen.util 402 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,508] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:18,516] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,516] jepsen worker 24 - jepsen.util 564 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,516] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,517] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:16:18,517] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:18,518] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:16:18,518] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,519] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:16:18,529] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,530] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 49] INFO [2022-03-13 17:16:18,553] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,553] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,554] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:16:18,555] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 47] INFO [2022-03-13 17:16:18,566] jepsen worker 76 - jepsen.util 706 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,567] jepsen worker 76 - jepsen.util 706 :invoke :write [25 14] INFO [2022-03-13 17:16:18,585] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,586] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:18,594] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,594] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,594] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,595] jepsen worker 34 - jepsen.util 934 :invoke :write [11 19] INFO [2022-03-13 17:16:18,596] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:18,596] jepsen worker 80 - jepsen.util 800 :invoke :write [26 35] INFO [2022-03-13 17:16:18,624] jepsen worker 53 - jepsen.util 503 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,625] jepsen worker 53 - jepsen.util 503 :invoke :write [17 36] INFO [2022-03-13 17:16:18,654] jepsen worker 58 - jepsen.util 3208 :ok :read [19 6] INFO [2022-03-13 17:16:18,655] jepsen worker 58 - jepsen.util 3208 :invoke :read [19 nil] INFO [2022-03-13 17:16:18,667] jepsen worker 16 - jepsen.util 3256 :ok :read [5 43] INFO [2022-03-13 17:16:18,668] jepsen worker 16 - jepsen.util 3256 :invoke :write [5 3] INFO [2022-03-13 17:16:18,669] jepsen worker 86 - jepsen.util 3326 :ok :read [28 44] INFO [2022-03-13 17:16:18,670] jepsen worker 86 - jepsen.util 3326 :invoke :write [28 3] INFO [2022-03-13 17:16:18,698] jepsen worker 36 - jepsen.util 2196 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,698] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:18,720] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,720] jepsen worker 71 - jepsen.util 521 :invoke :write [23 10] INFO [2022-03-13 17:16:18,744] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,744] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,746] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,747] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,753] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,753] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,753] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,754] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:18,755] jepsen worker 33 - jepsen.util 753 :invoke :write [11 19] INFO [2022-03-13 17:16:18,755] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:16:18,765] jepsen worker 44 - jepsen.util 584 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,766] jepsen worker 44 - jepsen.util 584 :invoke :write [14 48] INFO [2022-03-13 17:16:18,766] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,771] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,773] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,774] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,776] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,776] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,793] jepsen worker 38 - jepsen.util 2468 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,794] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:16:18,803] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,804] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:16:18,808] jepsen worker 13 - jepsen.util 733 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,809] jepsen worker 13 - jepsen.util 733 :invoke :write [4 14] INFO [2022-03-13 17:16:18,820] jepsen worker 14 - jepsen.util 374 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,820] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:18,825] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,825] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:16:18,829] jepsen worker 79 - jepsen.util 1069 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,829] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,829] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,830] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 32] INFO [2022-03-13 17:16:18,830] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:18,831] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:18,837] jepsen worker 15 - jepsen.util 3345 :info :write [5 37] :RequestTimeoutException INFO [2022-03-13 17:16:18,839] jepsen worker 15 - jepsen.util 3435 :invoke :write [5 0] INFO [2022-03-13 17:16:18,865] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,865] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,866] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,867] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,868] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,883] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:16:18,885] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,885] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,887] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:18,887] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,887] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:18,887] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,894] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,895] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:18,896] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,897] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:16:18,904] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,904] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,905] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:18,906] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:18,908] jepsen worker 4 - jepsen.util 2254 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,908] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 47] INFO [2022-03-13 17:16:18,925] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,925] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,926] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 18] INFO [2022-03-13 17:16:18,926] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 44] INFO [2022-03-13 17:16:18,943] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,944] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:18,955] jepsen worker 69 - jepsen.util 699 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,955] jepsen worker 69 - jepsen.util 699 :invoke :write [23 45] INFO [2022-03-13 17:16:18,976] jepsen worker 37 - jepsen.util 2107 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,977] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 40] INFO [2022-03-13 17:16:18,985] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:18,986] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,003] jepsen worker 80 - jepsen.util 800 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,004] jepsen worker 80 - jepsen.util 800 :invoke :write [26 32] INFO [2022-03-13 17:16:19,017] jepsen worker 6 - jepsen.util 636 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,018] jepsen worker 6 - jepsen.util 636 :invoke :write [2 31] INFO [2022-03-13 17:16:19,026] jepsen worker 71 - jepsen.util 521 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,027] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:16:19,030] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,031] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:16:19,061] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,062] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,074] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:19,074] jepsen worker nemesis - couchbase.util Rebalance check count: 487 INFO [2022-03-13 17:16:19,074] jepsen worker nemesis - couchbase.util Rebalance stuck count: 387 INFO [2022-03-13 17:16:19,079] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,080] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,099] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,099] jepsen worker 52 - jepsen.util 682 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,100] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:19,101] jepsen worker 52 - jepsen.util 682 :invoke :write [17 11] INFO [2022-03-13 17:16:19,115] jepsen worker 36 - jepsen.util 2196 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,116] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,116] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,117] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,118] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,119] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,120] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,121] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,133] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,133] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:16:19,152] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,152] jepsen worker 34 - jepsen.util 934 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,153] jepsen worker 51 - jepsen.util 771 :invoke :write [17 7] INFO [2022-03-13 17:16:19,154] jepsen worker 34 - jepsen.util 934 :invoke :write [11 30] INFO [2022-03-13 17:16:19,159] jepsen worker 4 - jepsen.util 2254 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,160] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,160] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,159] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,160] jepsen worker 75 - jepsen.util 435 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,160] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:16:19,161] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,161] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:16:19,162] jepsen worker 75 - jepsen.util 435 :invoke :write [25 40] INFO [2022-03-13 17:16:19,165] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:19,167] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,168] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:16:19,168] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,169] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:16:19,216] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,218] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:16:19,228] jepsen worker 37 - jepsen.util 2107 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,228] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,229] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,229] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,238] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,238] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,238] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:19,239] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:19,283] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,284] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,286] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,287] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:16:19,287] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,287] jepsen worker 89 - jepsen.util 2519 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,287] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,288] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,289] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 34] INFO [2022-03-13 17:16:19,290] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 32] INFO [2022-03-13 17:16:19,291] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,292] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 37] INFO [2022-03-13 17:16:19,294] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,295] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,296] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,296] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,297] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,298] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,299] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,300] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:16:19,364] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,364] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:16:19,367] jepsen worker 26 - jepsen.util 476 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,368] jepsen worker 26 - jepsen.util 476 :invoke :write [8 1] INFO [2022-03-13 17:16:19,372] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,372] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:16:19,375] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,376] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,386] jepsen worker 75 - jepsen.util 435 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,386] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,387] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:19,387] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:19,389] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,390] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:19,392] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,393] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,393] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,393] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,394] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,394] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:19,395] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:16:19,396] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:16:19,398] jepsen worker 89 - jepsen.util 2519 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,398] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,399] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 27] INFO [2022-03-13 17:16:19,399] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:19,404] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,405] jepsen worker 24 - jepsen.util 564 :invoke :write [8 20] INFO [2022-03-13 17:16:19,441] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,441] jepsen worker 33 - jepsen.util 753 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,441] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,441] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,442] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:16:19,442] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:16:19,443] jepsen worker 14 - jepsen.util 374 :invoke :write [4 43] INFO [2022-03-13 17:16:19,444] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:19,461] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,461] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,461] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,463] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:16:19,463] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:16:19,464] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 41] INFO [2022-03-13 17:16:19,474] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,475] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:16:19,530] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,530] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:16:19,542] jepsen worker 80 - jepsen.util 800 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,543] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:19,555] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,555] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,555] jepsen worker 78 - jepsen.util 618 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,556] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:19,557] jepsen worker 70 - jepsen.util 790 :invoke :write [23 40] INFO [2022-03-13 17:16:19,557] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:19,569] jepsen worker 37 - jepsen.util 2107 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,570] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,570] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 8] INFO [2022-03-13 17:16:19,570] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,577] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,577] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:16:19,604] jepsen worker 34 - jepsen.util 934 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,604] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:19,606] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,607] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:16:19,608] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,609] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:16:19,628] jepsen worker 43 - jepsen.util 583 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,629] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:19,628] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:19,633] jepsen worker 42 - jepsen.util 402 :invoke :write [14 20] INFO [2022-03-13 17:16:19,644] jepsen worker 51 - jepsen.util 771 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,645] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:19,653] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,653] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,680] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,680] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,681] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:19,681] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:19,684] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,685] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,686] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,687] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 10] INFO [2022-03-13 17:16:19,707] jepsen worker 67 - jepsen.util 2857 :info :write [22 40] :RequestTimeoutException INFO [2022-03-13 17:16:19,708] jepsen worker 67 - jepsen.util 2947 :invoke :read [22 nil] INFO [2022-03-13 17:16:19,716] jepsen worker 61 - jepsen.util 691 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,716] jepsen worker 61 - jepsen.util 691 :invoke :write [20 20] INFO [2022-03-13 17:16:19,735] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,736] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:19,739] jepsen worker 88 - jepsen.util 2068 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,739] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:16:19,743] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,744] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:16:19,749] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,750] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:19,751] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,752] jepsen worker 38 - jepsen.util 2468 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,754] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,755] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,755] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 30] INFO [2022-03-13 17:16:19,756] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,758] jepsen worker 89 - jepsen.util 2519 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,758] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,759] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,760] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,763] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,764] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,763] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,764] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,767] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,767] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,767] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,768] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,769] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,769] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,769] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,770] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 11] INFO [2022-03-13 17:16:19,770] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,771] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,773] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,773] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:16:19,828] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,829] jepsen worker 62 - jepsen.util 512 :invoke :write [20 49] INFO [2022-03-13 17:16:19,834] jepsen worker 14 - jepsen.util 374 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,834] jepsen worker 37 - jepsen.util 2107 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,834] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,835] jepsen worker 14 - jepsen.util 374 :invoke :write [4 32] INFO [2022-03-13 17:16:19,836] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:16:19,836] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:19,838] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,839] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 48] INFO [2022-03-13 17:16:19,849] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,849] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,849] jepsen worker 3 - jepsen.util 1983 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,852] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,852] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:16:19,853] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:16:19,899] jepsen worker 24 - jepsen.util 564 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,900] jepsen worker 24 - jepsen.util 564 :invoke :write [8 34] INFO [2022-03-13 17:16:19,914] jepsen worker 5 - jepsen.util 2435 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,915] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:16:19,936] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,936] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,936] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,936] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,936] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,936] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,936] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,937] jepsen worker 51 - jepsen.util 771 :invoke :write [17 36] INFO [2022-03-13 17:16:19,937] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:19,938] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:19,938] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:19,939] jepsen worker 80 - jepsen.util 800 :invoke :write [26 1] INFO [2022-03-13 17:16:19,939] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:19,940] jepsen worker 35 - jepsen.util 935 :invoke :write [11 48] INFO [2022-03-13 17:16:19,953] jepsen worker 76 - jepsen.util 706 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,953] jepsen worker 76 - jepsen.util 706 :invoke :write [25 15] INFO [2022-03-13 17:16:19,958] jepsen worker 89 - jepsen.util 2519 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,959] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:19,967] jepsen worker 83 - jepsen.util 2963 :info :write [27 20] :RequestTimeoutException INFO [2022-03-13 17:16:19,968] jepsen worker 83 - jepsen.util 3053 :invoke :write [27 2] INFO [2022-03-13 17:16:19,987] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,987] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:19,989] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:16:19,995] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,011] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,012] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:20,024] jepsen worker 53 - jepsen.util 503 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,024] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,024] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,025] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:20,025] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:20,025] jepsen worker 13 - jepsen.util 733 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,026] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:20,026] jepsen worker 13 - jepsen.util 733 :invoke :write [4 26] INFO [2022-03-13 17:16:20,037] jepsen worker 14 - jepsen.util 374 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,037] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,037] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,037] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,037] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,037] jepsen worker 14 - jepsen.util 374 :invoke :write [4 5] INFO [2022-03-13 17:16:20,038] jepsen worker 34 - jepsen.util 934 :invoke :write [11 32] INFO [2022-03-13 17:16:20,039] jepsen worker 21 - jepsen.util 831 :invoke :write [7 16] INFO [2022-03-13 17:16:20,039] jepsen worker 33 - jepsen.util 753 :invoke :write [11 39] INFO [2022-03-13 17:16:20,043] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:20,048] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,049] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,058] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,058] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,059] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,059] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,060] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,061] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:20,086] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:20,087] jepsen worker nemesis - couchbase.util Rebalance check count: 488 INFO [2022-03-13 17:16:20,087] jepsen worker nemesis - couchbase.util Rebalance stuck count: 388 INFO [2022-03-13 17:16:20,091] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,091] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,092] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 1] INFO [2022-03-13 17:16:20,092] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,111] jepsen worker 35 - jepsen.util 935 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,112] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:16:20,129] jepsen worker 37 - jepsen.util 2107 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,129] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,129] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,129] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,130] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,130] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,137] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,137] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,137] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,138] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,142] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,143] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,143] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 49] INFO [2022-03-13 17:16:20,143] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 12] INFO [2022-03-13 17:16:20,162] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,163] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,164] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,164] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,165] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,165] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,167] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,168] jepsen worker 44 - jepsen.util 584 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,168] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:16:20,169] jepsen worker 44 - jepsen.util 584 :invoke :write [14 37] INFO [2022-03-13 17:16:20,172] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,173] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:20,175] jepsen worker 52 - jepsen.util 682 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,176] jepsen worker 52 - jepsen.util 682 :invoke :write [17 44] INFO [2022-03-13 17:16:20,233] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,233] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,234] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 36] INFO [2022-03-13 17:16:20,235] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,238] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,238] jepsen worker 88 - jepsen.util 2068 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,239] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,240] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,240] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:16:20,241] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 28] INFO [2022-03-13 17:16:20,243] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,249] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,250] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:16:20,256] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:16:20,257] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,258] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:16:20,279] jepsen worker 80 - jepsen.util 800 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,287] jepsen worker 80 - jepsen.util 800 :invoke :write [26 21] INFO [2022-03-13 17:16:20,298] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,298] jepsen worker 36 - jepsen.util 2196 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,298] jepsen worker 24 - jepsen.util 564 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,298] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,300] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:20,300] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,301] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:20,301] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:16:20,301] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,302] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,302] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:20,303] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:16:20,303] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,306] jepsen worker 24 - jepsen.util 564 :invoke :write [8 28] INFO [2022-03-13 17:16:20,357] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,358] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,367] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,367] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,369] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,369] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 17] INFO [2022-03-13 17:16:20,376] jepsen worker 51 - jepsen.util 771 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,376] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,376] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:16:20,376] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:20,384] jepsen worker 79 - jepsen.util 1069 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,385] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 25] INFO [2022-03-13 17:16:20,405] jepsen worker 89 - jepsen.util 2519 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,406] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,406] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 17] INFO [2022-03-13 17:16:20,406] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,411] jepsen worker 34 - jepsen.util 934 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,411] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:20,437] jepsen worker 88 - jepsen.util 2068 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,438] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 10] INFO [2022-03-13 17:16:20,459] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,459] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,461] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,461] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 47] INFO [2022-03-13 17:16:20,480] jepsen worker 37 - jepsen.util 2107 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,480] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,481] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:16:20,481] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,496] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,497] jepsen worker 71 - jepsen.util 521 :invoke :write [23 29] INFO [2022-03-13 17:16:20,516] jepsen worker 41 - jepsen.util 3461 :info :write [13 42] :RequestTimeoutException INFO [2022-03-13 17:16:20,517] jepsen worker 41 - jepsen.util 3551 :invoke :read [13 nil] INFO [2022-03-13 17:16:20,520] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,520] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,520] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,521] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:16:20,527] jepsen worker 14 - jepsen.util 374 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,527] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:20,530] jepsen worker 40 - jepsen.util 3550 :info :write [13 35] :RequestTimeoutException INFO [2022-03-13 17:16:20,531] jepsen worker 40 - jepsen.util 3640 :invoke :read [13 nil] INFO [2022-03-13 17:16:20,535] jepsen worker 69 - jepsen.util 699 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,535] jepsen worker 69 - jepsen.util 699 :invoke :write [23 29] INFO [2022-03-13 17:16:20,554] jepsen worker 5 - jepsen.util 2435 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,555] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,556] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,557] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:16:20,558] jepsen worker 84 - jepsen.util 3774 :info :write [28 36] :RequestTimeoutException INFO [2022-03-13 17:16:20,559] jepsen worker 84 - jepsen.util 3864 :invoke :read [28 nil] INFO [2022-03-13 17:16:20,605] jepsen worker 88 - jepsen.util 2068 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,606] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,609] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,609] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,610] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,610] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,610] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 7] INFO [2022-03-13 17:16:20,611] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,637] jepsen worker 3 - jepsen.util 1983 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,637] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:16:20,652] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,653] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:20,664] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,664] jepsen worker 89 - jepsen.util 2519 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,665] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:16:20,665] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,671] jepsen worker 21 - jepsen.util 831 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,672] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:20,685] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,685] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,685] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,685] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,686] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,686] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,691] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,691] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,692] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:16:20,692] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:16:20,692] jepsen worker 6 - jepsen.util 636 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,692] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,693] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:16:20,694] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:20,720] jepsen worker 5 - jepsen.util 2435 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,721] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 0] INFO [2022-03-13 17:16:20,724] jepsen worker 33 - jepsen.util 753 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,725] jepsen worker 33 - jepsen.util 753 :invoke :write [11 20] INFO [2022-03-13 17:16:20,736] jepsen worker 24 - jepsen.util 564 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,737] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:16:20,760] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,762] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,763] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,763] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,781] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,782] jepsen worker 35 - jepsen.util 935 :invoke :write [11 3] INFO [2022-03-13 17:16:20,791] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,791] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,794] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,794] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,796] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,796] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:20,838] jepsen worker 37 - jepsen.util 2107 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,838] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,838] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,838] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 2] INFO [2022-03-13 17:16:20,839] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,839] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,846] jepsen worker 71 - jepsen.util 521 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,846] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,846] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,847] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:16:20,847] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,847] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,848] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,848] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:20,848] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,849] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 17] INFO [2022-03-13 17:16:20,849] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,850] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:16:20,853] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,853] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,853] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:20,853] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,854] jepsen worker 78 - jepsen.util 618 :invoke :write [26 4] INFO [2022-03-13 17:16:20,854] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:20,868] jepsen worker 27 - jepsen.util 3807 :info :write [9 45] :RequestTimeoutException INFO [2022-03-13 17:16:20,869] jepsen worker 27 - jepsen.util 3897 :invoke :read [9 nil] INFO [2022-03-13 17:16:20,891] jepsen worker 5 - jepsen.util 2435 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,891] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,893] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,893] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 11] INFO [2022-03-13 17:16:20,897] jepsen worker 80 - jepsen.util 800 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,897] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,897] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:20,898] jepsen worker 80 - jepsen.util 800 :invoke :write [26 48] INFO [2022-03-13 17:16:20,904] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,905] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:16:20,930] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,930] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:20,960] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,960] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,960] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:20,961] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:20,968] jepsen worker 26 - jepsen.util 476 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,968] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:20,968] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:20,969] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:20,975] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,975] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,975] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,976] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:20,979] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,980] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,982] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,982] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:20,994] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:20,994] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 19] INFO [2022-03-13 17:16:21,032] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,032] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,033] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,033] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:21,034] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:16:21,034] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:21,034] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,035] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:16:21,035] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,036] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:21,037] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,037] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,038] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,038] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,038] jepsen worker 24 - jepsen.util 564 :invoke :write [8 13] INFO [2022-03-13 17:16:21,039] jepsen worker 14 - jepsen.util 374 :invoke :write [4 35] INFO [2022-03-13 17:16:21,040] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:16:21,040] jepsen worker 21 - jepsen.util 831 :invoke :write [7 33] INFO [2022-03-13 17:16:21,042] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,043] jepsen worker 34 - jepsen.util 934 :invoke :write [11 10] INFO [2022-03-13 17:16:21,050] jepsen worker 70 - jepsen.util 790 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,050] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:21,059] jepsen worker 5 - jepsen.util 2435 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,060] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:21,080] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,080] jepsen worker 37 - jepsen.util 2107 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,080] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,081] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:16:21,081] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:16:21,081] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,083] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,084] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,088] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,088] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:16:21,093] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:21,093] jepsen worker nemesis - couchbase.util Rebalance check count: 489 INFO [2022-03-13 17:16:21,093] jepsen worker nemesis - couchbase.util Rebalance stuck count: 389 INFO [2022-03-13 17:16:21,109] jepsen worker 42 - jepsen.util 402 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,111] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:16:21,123] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,123] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,123] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,123] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:16:21,124] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,124] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,127] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,127] jepsen worker 3 - jepsen.util 1983 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,127] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,128] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,129] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:16:21,129] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,130] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,130] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,132] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,132] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:16:21,133] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,133] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:16:21,164] jepsen worker 33 - jepsen.util 753 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,164] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:21,167] jepsen worker 64 - jepsen.util 3304 :info :write [21 49] :RequestTimeoutException INFO [2022-03-13 17:16:21,168] jepsen worker 64 - jepsen.util 3394 :invoke :write [21 42] INFO [2022-03-13 17:16:21,175] jepsen worker 89 - jepsen.util 2519 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,176] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 39] INFO [2022-03-13 17:16:21,176] jepsen worker 24 - jepsen.util 564 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,181] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:16:21,184] jepsen worker 72 - jepsen.util 3582 :info :write [24 22] :RequestTimeoutException INFO [2022-03-13 17:16:21,185] jepsen worker 72 - jepsen.util 3672 :invoke :write [24 25] INFO [2022-03-13 17:16:21,218] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,219] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,225] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,226] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:21,241] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,242] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 20] INFO [2022-03-13 17:16:21,304] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,304] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:16:21,314] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,314] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,315] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,316] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 25] INFO [2022-03-13 17:16:21,324] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,325] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,328] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,330] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,331] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,331] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,333] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,334] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,338] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,339] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:16:21,372] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,373] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:16:21,421] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,421] jepsen worker 89 - jepsen.util 2519 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,421] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,421] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 8] INFO [2022-03-13 17:16:21,422] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,422] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:16:21,423] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,424] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:16:21,428] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,428] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,431] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,432] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:16:21,435] jepsen worker 4 - jepsen.util 2254 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,435] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 4] INFO [2022-03-13 17:16:21,482] jepsen worker 40 - jepsen.util 3640 :ok :read [13 39] INFO [2022-03-13 17:16:21,483] jepsen worker 40 - jepsen.util 3640 :invoke :read [13 nil] INFO [2022-03-13 17:16:21,496] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,496] jepsen worker 61 - jepsen.util 691 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,496] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,497] jepsen worker 61 - jepsen.util 691 :invoke :write [20 45] INFO [2022-03-13 17:16:21,499] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,500] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,502] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,503] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,507] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,508] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,512] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,513] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,515] jepsen worker 37 - jepsen.util 2107 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,516] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,521] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,521] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:16:21,522] jepsen worker 35 - jepsen.util 935 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,522] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,523] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:21,523] jepsen worker 51 - jepsen.util 771 :invoke :write [17 7] INFO [2022-03-13 17:16:21,529] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,530] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,534] jepsen worker 13 - jepsen.util 733 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,535] jepsen worker 13 - jepsen.util 733 :invoke :write [4 11] INFO [2022-03-13 17:16:21,535] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,539] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:16:21,554] jepsen worker 40 - jepsen.util 3640 :ok :read [13 39] INFO [2022-03-13 17:16:21,554] jepsen worker 40 - jepsen.util 3640 :invoke :read [13 nil] INFO [2022-03-13 17:16:21,563] jepsen worker 62 - jepsen.util 512 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,563] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,563] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,564] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,564] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:21,565] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:21,565] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:21,566] jepsen worker 7 - jepsen.util 637 :invoke :write [2 35] INFO [2022-03-13 17:16:21,602] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,603] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,608] jepsen worker 76 - jepsen.util 706 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,608] jepsen worker 76 - jepsen.util 706 :invoke :write [25 24] INFO [2022-03-13 17:16:21,611] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,611] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,615] jepsen worker 89 - jepsen.util 2519 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,616] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,616] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,617] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,617] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 32] INFO [2022-03-13 17:16:21,617] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,618] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,625] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:16:21,629] jepsen worker 40 - jepsen.util 3640 :ok :read [13 39] INFO [2022-03-13 17:16:21,630] jepsen worker 40 - jepsen.util 3640 :invoke :read [13 nil] INFO [2022-03-13 17:16:21,635] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,635] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,635] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,636] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,636] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,637] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,638] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 7] INFO [2022-03-13 17:16:21,638] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:16:21,641] jepsen worker 79 - jepsen.util 1069 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,641] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,641] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:21,641] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,642] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:21,643] jepsen worker 26 - jepsen.util 476 :invoke :write [8 6] INFO [2022-03-13 17:16:21,643] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:21,643] jepsen worker 43 - jepsen.util 583 :invoke :write [14 37] INFO [2022-03-13 17:16:21,649] jepsen worker 80 - jepsen.util 800 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,649] jepsen worker 80 - jepsen.util 800 :invoke :write [26 0] INFO [2022-03-13 17:16:21,712] jepsen worker 40 - jepsen.util 3640 :ok :read [13 39] INFO [2022-03-13 17:16:21,713] jepsen worker 40 - jepsen.util 3640 :invoke :write [13 17] INFO [2022-03-13 17:16:21,713] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,714] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,715] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,715] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,717] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,718] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:21,718] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,719] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 13] INFO [2022-03-13 17:16:21,722] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,723] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:16:21,725] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,725] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,726] jepsen worker 4 - jepsen.util 2254 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,726] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,726] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,727] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,727] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,728] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 13] INFO [2022-03-13 17:16:21,728] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,728] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:16:21,729] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:21,729] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:16:21,735] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,735] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:16:21,766] jepsen worker 39 - jepsen.util 3549 :info :write [13 11] :RequestTimeoutException INFO [2022-03-13 17:16:21,770] jepsen worker 39 - jepsen.util 3639 :invoke :write [13 35] INFO [2022-03-13 17:16:21,791] jepsen worker 37 - jepsen.util 2107 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,792] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 44] INFO [2022-03-13 17:16:21,842] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,842] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,842] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,843] jepsen worker 62 - jepsen.util 512 :invoke :write [20 25] INFO [2022-03-13 17:16:21,843] jepsen worker 53 - jepsen.util 503 :invoke :write [17 1] INFO [2022-03-13 17:16:21,844] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:21,851] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,851] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,855] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,855] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,857] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,858] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:16:21,862] jepsen worker 38 - jepsen.util 2468 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,863] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 46] INFO [2022-03-13 17:16:21,881] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,882] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 40] INFO [2022-03-13 17:16:21,921] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,922] jepsen worker 33 - jepsen.util 753 :invoke :write [11 46] INFO [2022-03-13 17:16:21,953] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,953] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:16:21,976] jepsen worker 89 - jepsen.util 2519 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,977] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 41] INFO [2022-03-13 17:16:21,983] jepsen worker 48 - jepsen.util 3378 :info :write [16 40] :RequestTimeoutException INFO [2022-03-13 17:16:21,984] jepsen worker 48 - jepsen.util 3468 :invoke :write [16 32] INFO [2022-03-13 17:16:21,986] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,987] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:21,990] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:21,991] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:22,017] jepsen worker 87 - jepsen.util 2607 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,019] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:22,098] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:22,098] jepsen worker nemesis - couchbase.util Rebalance check count: 490 INFO [2022-03-13 17:16:22,098] jepsen worker nemesis - couchbase.util Rebalance stuck count: 390 INFO [2022-03-13 17:16:22,102] jepsen worker 7 - jepsen.util 637 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,102] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,102] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,103] jepsen worker 7 - jepsen.util 637 :invoke :write [2 34] INFO [2022-03-13 17:16:22,103] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 9] INFO [2022-03-13 17:16:22,104] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:22,108] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,109] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,113] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,114] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,114] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:22,115] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:22,124] jepsen worker 37 - jepsen.util 2107 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,125] jepsen worker 80 - jepsen.util 800 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,127] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:22,128] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,168] jepsen worker 1 - jepsen.util 2971 :info :write [0 28] :RequestTimeoutException INFO [2022-03-13 17:16:22,168] jepsen worker 1 - jepsen.util 3061 :invoke :write [0 9] INFO [2022-03-13 17:16:22,183] jepsen worker 78 - jepsen.util 618 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,183] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,183] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,183] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,185] jepsen worker 4 - jepsen.util 2254 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,185] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,185] jepsen worker 60 - jepsen.util 600 :invoke :write [20 45] INFO [2022-03-13 17:16:22,185] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,186] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:16:22,188] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 0] INFO [2022-03-13 17:16:22,190] jepsen worker 78 - jepsen.util 618 :invoke :write [26 36] INFO [2022-03-13 17:16:22,191] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:22,191] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:16:22,194] jepsen worker 70 - jepsen.util 790 :invoke :write [23 11] INFO [2022-03-13 17:16:22,196] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,199] jepsen worker 14 - jepsen.util 374 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,200] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,205] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,207] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:16:22,208] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:16:22,214] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,214] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,219] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,220] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 7] INFO [2022-03-13 17:16:22,251] jepsen worker 43 - jepsen.util 583 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,252] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:22,299] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,299] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:16:22,303] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,304] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 12] INFO [2022-03-13 17:16:22,315] jepsen worker 38 - jepsen.util 2468 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,315] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,316] jepsen worker 21 - jepsen.util 831 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,316] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,317] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:16:22,318] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,357] jepsen worker 77 - jepsen.util 707 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,358] jepsen worker 77 - jepsen.util 707 :invoke :write [25 16] INFO [2022-03-13 17:16:22,361] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,361] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:16:22,385] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,385] jepsen worker 89 - jepsen.util 2519 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,385] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,385] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,386] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 2] INFO [2022-03-13 17:16:22,387] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:16:22,387] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,388] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,389] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,390] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,390] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:16:22,394] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,395] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,396] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,397] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,398] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,399] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,399] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 7] INFO [2022-03-13 17:16:22,419] jepsen worker 26 - jepsen.util 476 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,420] jepsen worker 17 - jepsen.util 3437 :info :write [5 8] :RequestTimeoutException INFO [2022-03-13 17:16:22,421] jepsen worker 17 - jepsen.util 3527 :invoke :write [5 26] INFO [2022-03-13 17:16:22,422] jepsen worker 26 - jepsen.util 476 :invoke :write [8 49] INFO [2022-03-13 17:16:22,425] jepsen worker 3 - jepsen.util 1983 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,426] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,430] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,431] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:16:22,439] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,439] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:22,441] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,442] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:16:22,464] jepsen worker 12 - jepsen.util 732 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,465] jepsen worker 12 - jepsen.util 732 :invoke :write [4 11] INFO [2022-03-13 17:16:22,485] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,486] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,487] jepsen worker 8 - jepsen.util 728 :invoke :write [2 23] INFO [2022-03-13 17:16:22,487] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] INFO [2022-03-13 17:16:22,501] jepsen worker 89 - jepsen.util 2519 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,502] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:22,505] jepsen worker 87 - jepsen.util 2607 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,506] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:16:22,535] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,535] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,535] jepsen worker 34 - jepsen.util 934 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,536] jepsen worker 34 - jepsen.util 934 :invoke :write [11 40] INFO [2022-03-13 17:16:22,538] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,539] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:16:22,576] jepsen worker 33 - jepsen.util 753 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,576] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,577] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:22,577] jepsen worker 80 - jepsen.util 800 :invoke :write [26 6] INFO [2022-03-13 17:16:22,591] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,592] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:16:22,611] jepsen worker 4 - jepsen.util 2254 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,612] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:16:22,633] jepsen worker 37 - jepsen.util 2107 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,635] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,659] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,660] jepsen worker 76 - jepsen.util 706 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,660] jepsen worker 76 - jepsen.util 706 :invoke :write [25 12] INFO [2022-03-13 17:16:22,661] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:16:22,698] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,698] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,700] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:16:22,701] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:16:22,710] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,710] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,710] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,711] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:22,712] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,712] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,712] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,713] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:22,713] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,714] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,714] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,714] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:22,715] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,715] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,717] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,717] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,718] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:22,719] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:22,723] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,723] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,723] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:16:22,724] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:16:22,737] jepsen worker 53 - jepsen.util 503 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,737] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,737] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:22,738] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:22,789] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,790] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,791] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,791] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,792] jepsen worker 71 - jepsen.util 521 :invoke :write [23 37] INFO [2022-03-13 17:16:22,792] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:16:22,798] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,799] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,801] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,802] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,802] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,802] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,803] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 42] INFO [2022-03-13 17:16:22,804] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 5] INFO [2022-03-13 17:16:22,807] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,808] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 20] INFO [2022-03-13 17:16:22,810] jepsen worker 51 - jepsen.util 771 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,813] jepsen worker 51 - jepsen.util 771 :invoke :write [17 19] INFO [2022-03-13 17:16:22,837] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,837] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:16:22,850] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,851] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:16:22,871] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,872] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 19] INFO [2022-03-13 17:16:22,882] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,883] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:16:22,895] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,897] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:22,912] jepsen worker 44 - jepsen.util 584 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,912] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:22,921] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,922] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:22,938] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,938] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,960] jepsen worker 62 - jepsen.util 512 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,960] jepsen worker 62 - jepsen.util 512 :invoke :write [20 41] INFO [2022-03-13 17:16:22,962] jepsen worker 3 - jepsen.util 1983 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,963] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:22,964] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,964] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:16:22,983] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:22,984] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,014] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,014] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,048] jepsen worker 38 - jepsen.util 2468 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,048] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,049] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:16:23,050] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,054] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,054] jepsen worker 34 - jepsen.util 934 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,055] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:23,055] jepsen worker 33 - jepsen.util 753 :invoke :write [11 17] INFO [2022-03-13 17:16:23,061] jepsen worker 71 - jepsen.util 521 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,062] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:23,069] jepsen worker 69 - jepsen.util 699 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,069] jepsen worker 69 - jepsen.util 699 :invoke :write [23 33] INFO [2022-03-13 17:16:23,103] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,103] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,103] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,104] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:16:23,104] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,105] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,108] jepsen worker 11 - jepsen.util 3431 :ok :read [3 25] INFO [2022-03-13 17:16:23,108] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:23,109] jepsen worker nemesis - couchbase.util Rebalance check count: 491 INFO [2022-03-13 17:16:23,109] jepsen worker nemesis - couchbase.util Rebalance stuck count: 391 INFO [2022-03-13 17:16:23,109] jepsen worker 11 - jepsen.util 3431 :invoke :write [3 14] INFO [2022-03-13 17:16:23,110] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,110] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,111] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:16:23,111] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 31] INFO [2022-03-13 17:16:23,127] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,128] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,131] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,132] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:16:23,136] jepsen worker 9 - jepsen.util 2889 :ok :read [3 25] INFO [2022-03-13 17:16:23,137] jepsen worker 9 - jepsen.util 2889 :invoke :write [3 25] INFO [2022-03-13 17:16:23,169] jepsen worker 89 - jepsen.util 2519 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,169] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,170] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 19] INFO [2022-03-13 17:16:23,170] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,182] jepsen worker 32 - jepsen.util 3542 :ok :read [10 :nil] INFO [2022-03-13 17:16:23,183] jepsen worker 32 - jepsen.util 3542 :invoke :write [10 21] INFO [2022-03-13 17:16:23,187] jepsen worker 7 - jepsen.util 637 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,188] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:23,209] jepsen worker 29 - jepsen.util 3719 :ok :read [9 26] INFO [2022-03-13 17:16:23,210] jepsen worker 29 - jepsen.util 3719 :invoke :read [9 nil] INFO [2022-03-13 17:16:23,231] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,232] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,232] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,232] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:16:23,238] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,238] jepsen worker 80 - jepsen.util 800 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,239] jepsen worker 80 - jepsen.util 800 :invoke :write [26 41] INFO [2022-03-13 17:16:23,239] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:23,244] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,244] jepsen worker 37 - jepsen.util 2107 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,244] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,244] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,244] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,245] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,247] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 49] INFO [2022-03-13 17:16:23,248] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:16:23,255] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:16:23,255] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,255] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:23,256] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,268] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,268] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,316] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,317] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 34] INFO [2022-03-13 17:16:23,319] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,319] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,327] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,328] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,329] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,330] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 49] INFO [2022-03-13 17:16:23,331] jepsen worker 89 - jepsen.util 2519 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,332] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,332] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,333] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,334] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,334] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,335] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,335] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,336] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,336] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,337] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,337] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,338] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,338] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,347] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,347] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,348] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:16:23,348] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,350] jepsen worker 60 - jepsen.util 600 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,350] jepsen worker 60 - jepsen.util 600 :invoke :write [20 24] INFO [2022-03-13 17:16:23,355] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,360] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:23,375] jepsen worker 88 - jepsen.util 2068 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,375] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,375] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,376] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,386] jepsen worker 29 - jepsen.util 3719 :ok :read [9 26] INFO [2022-03-13 17:16:23,387] jepsen worker 29 - jepsen.util 3719 :invoke :read [9 nil] INFO [2022-03-13 17:16:23,406] jepsen worker 37 - jepsen.util 2107 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,407] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,407] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,408] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,408] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,409] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,409] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 10] INFO [2022-03-13 17:16:23,410] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:16:23,429] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,430] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:23,436] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,436] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,436] jepsen worker 87 - jepsen.util 2607 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,436] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,437] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,438] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,440] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,440] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,440] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,441] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,442] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:16:23,442] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,442] jepsen worker 79 - jepsen.util 1069 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,443] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,444] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,444] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,444] jepsen worker 6 - jepsen.util 636 :invoke :write [2 12] INFO [2022-03-13 17:16:23,445] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:16:23,446] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:16:23,446] jepsen worker 51 - jepsen.util 771 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,446] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 26] INFO [2022-03-13 17:16:23,447] jepsen worker 51 - jepsen.util 771 :invoke :write [17 40] INFO [2022-03-13 17:16:23,452] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,453] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:16:23,458] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,460] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,462] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,464] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:16:23,520] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,520] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:23,528] jepsen worker 13 - jepsen.util 733 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,528] jepsen worker 13 - jepsen.util 733 :invoke :write [4 27] INFO [2022-03-13 17:16:23,532] jepsen worker 37 - jepsen.util 2107 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,533] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,542] jepsen worker 29 - jepsen.util 3719 :ok :read [9 26] INFO [2022-03-13 17:16:23,543] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,543] jepsen worker 29 - jepsen.util 3719 :invoke :write [9 25] INFO [2022-03-13 17:16:23,543] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:16:23,551] jepsen worker 3 - jepsen.util 1983 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,551] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:16:23,555] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,555] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,555] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,556] jepsen worker 75 - jepsen.util 435 :invoke :write [25 40] INFO [2022-03-13 17:16:23,557] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:16:23,557] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:23,615] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,615] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:16:23,620] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,621] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,621] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,621] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,622] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,622] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,673] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,673] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,673] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,673] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,674] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,675] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 41] INFO [2022-03-13 17:16:23,675] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,676] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,676] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:16:23,677] jepsen worker 24 - jepsen.util 564 :invoke :write [8 11] INFO [2022-03-13 17:16:23,680] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,681] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,684] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,686] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,687] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,688] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,689] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,690] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 13] INFO [2022-03-13 17:16:23,691] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,692] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:16:23,692] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,692] jepsen worker 77 - jepsen.util 707 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,693] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:16:23,693] jepsen worker 77 - jepsen.util 707 :invoke :write [25 10] INFO [2022-03-13 17:16:23,726] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,727] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,748] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,749] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:16:23,751] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,751] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 26] INFO [2022-03-13 17:16:23,760] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,761] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,765] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,767] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:16:23,776] jepsen worker 63 - jepsen.util 3483 :info :write [21 28] :RequestTimeoutException INFO [2022-03-13 17:16:23,777] jepsen worker 63 - jepsen.util 3573 :invoke :write [21 46] INFO [2022-03-13 17:16:23,828] jepsen worker 37 - jepsen.util 2107 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,828] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 0] INFO [2022-03-13 17:16:23,829] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,829] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,829] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:16:23,830] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,832] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,832] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,832] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,832] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,833] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,833] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:16:23,834] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:23,834] jepsen worker 75 - jepsen.util 435 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,835] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:23,836] jepsen worker 56 - jepsen.util 3566 :info :write [18 20] :RequestTimeoutException INFO [2022-03-13 17:16:23,836] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:23,836] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,836] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,836] jepsen worker 56 - jepsen.util 3656 :invoke :read [18 nil] INFO [2022-03-13 17:16:23,837] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 12] INFO [2022-03-13 17:16:23,837] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,838] jepsen worker 26 - jepsen.util 476 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,838] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,838] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:23,839] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,839] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,840] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,841] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 15] INFO [2022-03-13 17:16:23,846] jepsen worker 34 - jepsen.util 934 :invoke :write [11 25] INFO [2022-03-13 17:16:23,847] jepsen worker 3 - jepsen.util 1983 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,847] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:16:23,890] jepsen worker 52 - jepsen.util 682 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,890] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:23,892] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,900] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:23,909] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,910] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,911] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,911] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:23,912] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,913] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:23,914] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,915] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:16:23,925] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,925] jepsen worker 24 - jepsen.util 564 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,925] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,926] jepsen worker 24 - jepsen.util 564 :invoke :write [8 19] INFO [2022-03-13 17:16:23,926] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:23,927] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:23,956] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,957] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 21] INFO [2022-03-13 17:16:23,964] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,964] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:16:23,973] jepsen worker 31 - jepsen.util 3721 :ok :read [10 :nil] INFO [2022-03-13 17:16:23,974] jepsen worker 31 - jepsen.util 3721 :invoke :write [10 34] INFO [2022-03-13 17:16:23,976] jepsen worker 12 - jepsen.util 732 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,976] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:23,977] jepsen worker 12 - jepsen.util 732 :invoke :write [4 25] INFO [2022-03-13 17:16:23,978] jepsen worker 44 - jepsen.util 584 :invoke :write [14 34] INFO [2022-03-13 17:16:23,982] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,983] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,984] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,984] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:16:23,985] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:23,986] jepsen worker 75 - jepsen.util 435 :invoke :write [25 47] INFO [2022-03-13 17:16:23,992] jepsen worker 33 - jepsen.util 753 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,992] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:23,993] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:23,994] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:24,024] jepsen worker 8 - jepsen.util 728 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,025] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:24,102] jepsen worker 4 - jepsen.util 2254 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,103] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,112] jepsen worker 89 - jepsen.util 2519 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,112] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,112] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,113] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:24,114] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:24,114] jepsen worker nemesis - couchbase.util Rebalance check count: 492 INFO [2022-03-13 17:16:24,114] jepsen worker nemesis - couchbase.util Rebalance stuck count: 392 INFO [2022-03-13 17:16:24,119] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,119] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,125] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,126] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:16:24,134] jepsen worker 76 - jepsen.util 706 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,135] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:24,143] jepsen worker 24 - jepsen.util 564 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,144] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,144] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:16:24,145] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:16:24,159] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,160] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:16:24,171] jepsen worker 37 - jepsen.util 2107 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,172] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,172] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,173] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:24,174] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:16:24,174] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,177] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,178] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,178] jepsen worker 88 - jepsen.util 2068 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,179] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 33] INFO [2022-03-13 17:16:24,180] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,182] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,184] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,185] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:16:24,217] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,218] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:16:24,237] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,238] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,238] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 46] INFO [2022-03-13 17:16:24,239] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 2] INFO [2022-03-13 17:16:24,250] jepsen worker 87 - jepsen.util 2607 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,250] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,251] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 26] INFO [2022-03-13 17:16:24,251] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:16:24,280] jepsen worker 69 - jepsen.util 699 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,281] jepsen worker 69 - jepsen.util 699 :invoke :write [23 8] INFO [2022-03-13 17:16:24,288] jepsen worker 80 - jepsen.util 800 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,289] jepsen worker 80 - jepsen.util 800 :invoke :write [26 37] INFO [2022-03-13 17:16:24,307] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,307] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:24,307] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,307] jepsen worker 62 - jepsen.util 512 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,308] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:24,309] jepsen worker 53 - jepsen.util 503 :invoke :write [17 4] INFO [2022-03-13 17:16:24,310] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:24,311] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,311] jepsen worker 62 - jepsen.util 512 :invoke :write [20 31] INFO [2022-03-13 17:16:24,311] jepsen worker 88 - jepsen.util 2068 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,312] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,312] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:16:24,323] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,324] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,325] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,325] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,330] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,330] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,331] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,332] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,337] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,338] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,340] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,340] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:16:24,345] jepsen worker 75 - jepsen.util 435 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,346] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:16:24,349] jepsen worker 78 - jepsen.util 618 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,350] jepsen worker 78 - jepsen.util 618 :invoke :write [26 0] INFO [2022-03-13 17:16:24,384] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,384] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,385] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,386] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:16:24,420] jepsen worker 87 - jepsen.util 2607 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,420] jepsen worker 51 - jepsen.util 771 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,421] jepsen worker 51 - jepsen.util 771 :invoke :write [17 25] INFO [2022-03-13 17:16:24,421] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,429] jepsen worker 38 - jepsen.util 2468 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,430] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 35] INFO [2022-03-13 17:16:24,440] jepsen worker 59 - jepsen.util 3479 :info :write [19 16] :RequestTimeoutException INFO [2022-03-13 17:16:24,441] jepsen worker 59 - jepsen.util 3569 :invoke :write [19 37] INFO [2022-03-13 17:16:24,457] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,458] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,459] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,459] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,460] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,461] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 44] INFO [2022-03-13 17:16:24,470] jepsen worker 6 - jepsen.util 636 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,471] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:24,492] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,492] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,492] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,493] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 38] INFO [2022-03-13 17:16:24,494] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,495] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:16:24,495] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,496] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 39] INFO [2022-03-13 17:16:24,496] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,498] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,499] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,500] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 47] INFO [2022-03-13 17:16:24,558] jepsen worker 37 - jepsen.util 2107 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,559] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:24,560] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,560] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:24,576] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,576] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,577] jepsen worker 14 - jepsen.util 374 :invoke :write [4 14] INFO [2022-03-13 17:16:24,579] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:16:24,580] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,580] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:16:24,613] jepsen worker 87 - jepsen.util 2607 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,614] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,631] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,631] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:16:24,644] jepsen worker 70 - jepsen.util 790 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,644] jepsen worker 70 - jepsen.util 790 :invoke :write [23 28] INFO [2022-03-13 17:16:24,686] jepsen worker 5 - jepsen.util 2435 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,687] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,693] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,694] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:24,709] jepsen worker 88 - jepsen.util 2068 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,710] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 27] INFO [2022-03-13 17:16:24,711] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,712] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:16:24,716] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,717] jepsen worker 24 - jepsen.util 564 :invoke :write [8 34] INFO [2022-03-13 17:16:24,726] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,727] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,736] jepsen worker 79 - jepsen.util 1069 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,736] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,737] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,737] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:24,738] jepsen worker 8 - jepsen.util 728 :invoke :write [2 34] INFO [2022-03-13 17:16:24,739] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,743] jepsen worker 26 - jepsen.util 476 :invoke :write [8 39] INFO [2022-03-13 17:16:24,743] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,748] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,749] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,750] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,751] jepsen worker 34 - jepsen.util 934 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,751] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,751] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,752] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,753] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,753] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:24,754] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:24,755] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:24,756] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:24,763] jepsen worker 38 - jepsen.util 2468 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,763] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,764] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:24,765] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:24,798] jepsen worker 65 - jepsen.util 3935 :info :write [21 41] :RequestTimeoutException INFO [2022-03-13 17:16:24,799] jepsen worker 65 - jepsen.util 4025 :invoke :write [21 21] INFO [2022-03-13 17:16:24,804] jepsen worker 89 - jepsen.util 2519 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,804] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,804] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,805] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,805] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:16:24,805] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:24,809] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,810] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,811] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,811] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,812] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,812] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:16:24,813] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,813] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:24,813] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:16:24,814] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:16:24,815] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,816] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 48] INFO [2022-03-13 17:16:24,838] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,839] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:16:24,860] jepsen worker 45 - jepsen.util 3645 :info :write [15 8] :RequestTimeoutException INFO [2022-03-13 17:16:24,861] jepsen worker 45 - jepsen.util 3735 :invoke :write [15 16] INFO [2022-03-13 17:16:24,864] jepsen worker 77 - jepsen.util 707 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,864] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:16:24,871] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,872] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,873] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,874] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:24,877] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,878] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:24,882] jepsen worker 80 - jepsen.util 800 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,882] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:24,906] jepsen worker 88 - jepsen.util 2068 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,906] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:24,908] jepsen worker 51 - jepsen.util 771 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,909] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:24,923] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,924] jepsen worker 55 - jepsen.util 3295 :info :write [18 3] :RequestTimeoutException INFO [2022-03-13 17:16:24,924] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:16:24,924] jepsen worker 55 - jepsen.util 3385 :invoke :read [18 nil] INFO [2022-03-13 17:16:24,948] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,948] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,949] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:24,949] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 45] INFO [2022-03-13 17:16:24,952] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:24,953] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,004] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,004] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,005] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,005] jepsen worker 4 - jepsen.util 2254 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,006] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,006] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:16:25,023] jepsen worker 24 - jepsen.util 564 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,024] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,024] jepsen worker 71 - jepsen.util 521 :invoke :write [23 43] INFO [2022-03-13 17:16:25,025] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:16:25,032] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:25,032] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,032] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,033] jepsen worker 43 - jepsen.util 583 :invoke :write [14 24] INFO [2022-03-13 17:16:25,033] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:25,034] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:25,040] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,041] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 23] INFO [2022-03-13 17:16:25,101] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,101] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:25,103] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,104] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 23] INFO [2022-03-13 17:16:25,119] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:25,120] jepsen worker nemesis - couchbase.util Rebalance check count: 493 INFO [2022-03-13 17:16:25,120] jepsen worker nemesis - couchbase.util Rebalance stuck count: 393 INFO [2022-03-13 17:16:25,123] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,123] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,124] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,124] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,141] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,141] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,142] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,143] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 11] INFO [2022-03-13 17:16:25,184] jepsen worker 13 - jepsen.util 733 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,184] jepsen worker 13 - jepsen.util 733 :invoke :write [4 2] INFO [2022-03-13 17:16:25,189] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,190] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,190] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:16:25,190] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,190] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,190] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,191] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,192] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,195] jepsen worker 3 - jepsen.util 1983 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,195] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,195] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,195] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,196] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,196] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,197] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,198] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,198] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,198] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,199] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:16:25,199] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:16:25,200] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,200] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,201] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:16:25,201] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 43] INFO [2022-03-13 17:16:25,207] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,208] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,210] jepsen worker 89 - jepsen.util 2519 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,210] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,210] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,211] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,212] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:16:25,212] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,223] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,223] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,229] jepsen worker 14 - jepsen.util 374 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,230] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:16:25,241] jepsen worker 38 - jepsen.util 2468 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,242] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,242] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,243] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 45] INFO [2022-03-13 17:16:25,244] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,245] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:16:25,277] jepsen worker 71 - jepsen.util 521 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,278] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:25,277] jepsen worker 53 - jepsen.util 503 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,279] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:25,279] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,279] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,279] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:16:25,281] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,284] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,285] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:16:25,306] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,306] jepsen worker 5 - jepsen.util 2435 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,307] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:16:25,307] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 17] INFO [2022-03-13 17:16:25,341] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,342] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 37] INFO [2022-03-13 17:16:25,351] jepsen worker 89 - jepsen.util 2519 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,351] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,355] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,355] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,355] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,355] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,355] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,355] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,355] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:25,356] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,357] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:25,358] jepsen worker 33 - jepsen.util 753 :invoke :write [11 15] INFO [2022-03-13 17:16:25,359] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:16:25,359] jepsen worker 80 - jepsen.util 800 :invoke :write [26 16] INFO [2022-03-13 17:16:25,361] jepsen worker 34 - jepsen.util 934 :invoke :write [11 11] INFO [2022-03-13 17:16:25,365] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,366] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:25,366] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:25,437] jepsen worker 66 - jepsen.util 3306 :ok :read [22 30] INFO [2022-03-13 17:16:25,437] jepsen worker 66 - jepsen.util 3306 :invoke :read [22 nil] INFO [2022-03-13 17:16:25,450] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,450] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,451] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:16:25,451] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,469] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,469] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,476] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,476] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 3] INFO [2022-03-13 17:16:25,483] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,483] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,506] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,506] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,506] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,507] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 20] INFO [2022-03-13 17:16:25,521] jepsen worker 69 - jepsen.util 699 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,521] jepsen worker 69 - jepsen.util 699 :invoke :write [23 49] INFO [2022-03-13 17:16:25,529] jepsen worker 3 - jepsen.util 1983 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,529] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 15] INFO [2022-03-13 17:16:25,556] jepsen worker 0 - jepsen.util 3420 :info :write [0 29] :RequestTimeoutException INFO [2022-03-13 17:16:25,556] jepsen worker 87 - jepsen.util 2607 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,558] jepsen worker 0 - jepsen.util 3510 :invoke :read [0 nil] INFO [2022-03-13 17:16:25,558] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:16:25,560] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,560] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,560] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,561] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:25,561] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:16:25,562] jepsen worker 24 - jepsen.util 564 :invoke :write [8 36] INFO [2022-03-13 17:16:25,616] jepsen worker 5 - jepsen.util 2435 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,617] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,622] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,622] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,624] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,625] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:16:25,631] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,631] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,632] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:16:25,632] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,636] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,636] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:25,646] jepsen worker 4 - jepsen.util 2254 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,646] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,670] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,671] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:25,676] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,676] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,676] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,676] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,676] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:25,677] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:25,677] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:25,678] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:16:25,691] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,692] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,692] jepsen worker 62 - jepsen.util 512 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,693] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:25,693] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 37] INFO [2022-03-13 17:16:25,694] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:25,700] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,701] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 6] INFO [2022-03-13 17:16:25,703] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,704] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,710] jepsen worker 3 - jepsen.util 1983 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,711] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,713] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,714] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:16:25,714] jepsen worker 33 - jepsen.util 753 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,715] jepsen worker 33 - jepsen.util 753 :invoke :write [11 10] INFO [2022-03-13 17:16:25,727] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,728] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,728] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 14] INFO [2022-03-13 17:16:25,727] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,729] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,729] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:16:25,731] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,731] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 1] INFO [2022-03-13 17:16:25,742] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,743] jepsen worker 22 - jepsen.util 652 :invoke :write [7 28] INFO [2022-03-13 17:16:25,746] jepsen worker 34 - jepsen.util 934 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,747] jepsen worker 34 - jepsen.util 934 :invoke :write [11 18] INFO [2022-03-13 17:16:25,780] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,780] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,781] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,782] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:16:25,828] jepsen worker 82 - jepsen.util 3502 :info :write [27 27] :RequestTimeoutException INFO [2022-03-13 17:16:25,828] jepsen worker 82 - jepsen.util 3592 :invoke :write [27 6] INFO [2022-03-13 17:16:25,831] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,832] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,835] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,836] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,837] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,838] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:16:25,843] jepsen worker 36 - jepsen.util 2196 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,844] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:16:25,861] jepsen worker 24 - jepsen.util 564 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,862] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] INFO [2022-03-13 17:16:25,864] jepsen worker 4 - jepsen.util 2254 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,864] jepsen worker 78 - jepsen.util 618 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,865] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:16:25,866] jepsen worker 78 - jepsen.util 618 :invoke :write [26 11] INFO [2022-03-13 17:16:25,889] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,890] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:25,893] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,894] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 15] INFO [2022-03-13 17:16:25,937] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,937] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,938] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,938] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,944] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,944] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,944] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,945] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,947] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,947] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,948] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,948] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,950] jepsen worker 46 - jepsen.util 3556 :info :write [15 43] :RequestTimeoutException INFO [2022-03-13 17:16:25,950] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,950] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,951] jepsen worker 46 - jepsen.util 3646 :invoke :write [15 38] INFO [2022-03-13 17:16:25,951] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,951] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 20] INFO [2022-03-13 17:16:25,952] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,952] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 5] INFO [2022-03-13 17:16:25,968] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,969] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:25,977] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,978] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:16:25,991] jepsen worker 88 - jepsen.util 2068 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:25,992] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:16:26,014] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,015] jepsen worker 52 - jepsen.util 682 :invoke :write [17 22] INFO [2022-03-13 17:16:26,019] jepsen worker 8 - jepsen.util 728 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,020] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:26,032] jepsen worker 80 - jepsen.util 800 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,033] jepsen worker 80 - jepsen.util 800 :invoke :write [26 6] INFO [2022-03-13 17:16:26,047] jepsen worker 5 - jepsen.util 2435 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,047] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,050] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,051] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:26,086] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,086] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,087] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:16:26,088] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:26,088] jepsen worker 89 - jepsen.util 2519 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,089] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 24] INFO [2022-03-13 17:16:26,101] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,101] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:16:26,102] jepsen worker 12 - jepsen.util 732 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,103] jepsen worker 12 - jepsen.util 732 :invoke :write [4 11] INFO [2022-03-13 17:16:26,126] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:26,126] jepsen worker nemesis - couchbase.util Rebalance check count: 494 INFO [2022-03-13 17:16:26,126] jepsen worker nemesis - couchbase.util Rebalance stuck count: 394 INFO [2022-03-13 17:16:26,130] jepsen worker 3 - jepsen.util 1983 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,131] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,132] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,132] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,134] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,135] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,138] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,138] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,140] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,141] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,146] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,147] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:16:26,175] jepsen worker 37 - jepsen.util 2107 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,176] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 17] INFO [2022-03-13 17:16:26,203] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,204] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:16:26,205] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,206] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:16:26,222] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,227] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,238] jepsen worker 26 - jepsen.util 476 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,238] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,238] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:26,239] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:26,240] jepsen worker 42 - jepsen.util 402 :invoke :write [14 40] INFO [2022-03-13 17:16:26,241] jepsen worker 7 - jepsen.util 637 :invoke :write [2 44] INFO [2022-03-13 17:16:26,244] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,244] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,244] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,244] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,246] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:26,247] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:16:26,249] jepsen worker 51 - jepsen.util 771 :invoke :write [17 46] INFO [2022-03-13 17:16:26,250] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:26,269] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,271] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:16:26,276] jepsen worker 19 - jepsen.util 3169 :info :write [6 27] :RequestTimeoutException INFO [2022-03-13 17:16:26,277] jepsen worker 19 - jepsen.util 3259 :invoke :read [6 nil] INFO [2022-03-13 17:16:26,284] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,284] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,285] jepsen worker 24 - jepsen.util 564 :invoke :write [8 39] INFO [2022-03-13 17:16:26,287] jepsen worker 71 - jepsen.util 521 :invoke :write [23 26] INFO [2022-03-13 17:16:26,305] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,306] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,306] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,307] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,340] jepsen worker 30 - jepsen.util 3360 :info :write [10 14] :RequestTimeoutException INFO [2022-03-13 17:16:26,340] jepsen worker 70 - jepsen.util 790 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,341] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,341] jepsen worker 30 - jepsen.util 3450 :invoke :write [10 23] INFO [2022-03-13 17:16:26,342] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:26,342] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:26,342] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,343] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:16:26,364] jepsen worker 89 - jepsen.util 2519 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,365] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,365] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,366] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,370] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,371] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,371] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,372] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,372] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,373] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,373] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:16:26,374] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,377] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,378] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:16:26,380] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,380] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,408] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,408] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,409] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:16:26,410] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,415] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,417] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 1] INFO [2022-03-13 17:16:26,426] jepsen worker 37 - jepsen.util 2107 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,426] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,427] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,427] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:16:26,428] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,429] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 32] INFO [2022-03-13 17:16:26,430] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,431] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 11] INFO [2022-03-13 17:16:26,442] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,442] jepsen worker 43 - jepsen.util 583 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,442] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,442] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,444] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:26,445] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:26,446] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:26,447] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:16:26,454] jepsen worker 33 - jepsen.util 753 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,455] jepsen worker 33 - jepsen.util 753 :invoke :write [11 32] INFO [2022-03-13 17:16:26,485] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,485] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,491] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,491] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,492] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,493] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:16:26,504] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,504] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 22] INFO [2022-03-13 17:16:26,513] jepsen worker 34 - jepsen.util 934 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,514] jepsen worker 34 - jepsen.util 934 :invoke :write [11 30] INFO [2022-03-13 17:16:26,517] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,517] jepsen worker 77 - jepsen.util 707 :invoke :write [25 33] INFO [2022-03-13 17:16:26,528] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,528] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,529] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 22] INFO [2022-03-13 17:16:26,529] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,553] jepsen worker 3 - jepsen.util 1983 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,553] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,556] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,557] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 2] INFO [2022-03-13 17:16:26,612] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,613] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:16:26,616] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,617] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,623] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,623] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:16:26,632] jepsen worker 37 - jepsen.util 2107 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,633] jepsen worker 36 - jepsen.util 2196 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,633] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 7] INFO [2022-03-13 17:16:26,634] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:16:26,648] jepsen worker 24 - jepsen.util 564 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,648] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] WARN [2022-03-13 17:16:26,687] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17566488,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17566488,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":8343334,"encode_duration_us":27735,"operation_id":"0xdd97","timeout_ms":10000,"last_server_duration_us":8343334,"total_duration_us":10000835},{"operation_name":"upsert","last_dispatch_duration_us":16984018,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17534589,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":8842547,"encode_duration_us":34775,"operation_id":"0xd814","timeout_ms":10000,"last_server_duration_us":8553731,"total_duration_us":10006258},{"operation_name":"upsert","last_dispatch_duration_us":16200116,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16782056,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9292235,"encode_duration_us":17859,"operation_id":"0xdc9f","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10001235}],"total_count":3}} WARN [2022-03-13 17:16:26,687] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14958546,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":14958546,"total_server_duration_us":7231,"operation_id":"0xda5c","timeout_ms":30000,"last_server_duration_us":7231,"total_duration_us":14961801},{"operation_name":"get","last_dispatch_duration_us":13550694,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":13550694,"total_server_duration_us":6877,"operation_id":"0xdfc0","timeout_ms":30000,"last_server_duration_us":6877,"total_duration_us":13552808},{"operation_name":"upsert","last_dispatch_duration_us":14881782,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":14881782,"total_server_duration_us":19574,"encode_duration_us":20734,"operation_id":"0xdeee","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10010517},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":20409,"operation_id":"0xe140","timeout_ms":10000,"total_duration_us":10007924},{"operation_name":"upsert","last_dispatch_duration_us":8561907,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16261190,"total_server_duration_us":306831,"encode_duration_us":22507,"operation_id":"0xdcfc","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10006259},{"operation_name":"upsert","last_dispatch_duration_us":43463,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8343023,"total_server_duration_us":309902,"encode_duration_us":18334,"operation_id":"0xe235","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10003331},{"operation_name":"upsert","last_dispatch_duration_us":17566488,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":17566488,"total_server_duration_us":8343334,"encode_duration_us":27735,"operation_id":"0xdd97","timeout_ms":10000,"last_server_duration_us":8343334,"total_duration_us":10000835},{"operation_name":"upsert","last_dispatch_duration_us":798969,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1507944,"total_server_duration_us":219193,"encode_duration_us":28982,"operation_id":"0xe518","timeout_ms":10000,"last_server_duration_us":200008,"total_duration_us":2605576},{"operation_name":"upsert","last_dispatch_duration_us":2180919,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2180919,"total_server_duration_us":170761,"encode_duration_us":17895,"operation_id":"0xe195","timeout_ms":10000,"last_server_duration_us":170761,"total_duration_us":2188786},{"operation_name":"upsert","last_dispatch_duration_us":2140096,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2140096,"total_server_duration_us":100910,"encode_duration_us":26325,"operation_id":"0xe173","timeout_ms":10000,"last_server_duration_us":100910,"total_duration_us":2147366}],"total_count":93}} INFO [2022-03-13 17:16:26,692] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,692] jepsen worker 88 - jepsen.util 2068 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,692] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,693] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:16:26,718] jepsen worker 3 - jepsen.util 1983 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,718] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,721] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,722] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,723] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,724] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 46] INFO [2022-03-13 17:16:26,741] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,741] jepsen worker 79 - jepsen.util 1069 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,741] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,743] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:26,743] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:26,744] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:26,755] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,755] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,756] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,756] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,780] jepsen worker 50 - jepsen.util 3380 :ok :read [16 :nil] INFO [2022-03-13 17:16:26,781] jepsen worker 50 - jepsen.util 3380 :invoke :write [16 11] INFO [2022-03-13 17:16:26,788] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,790] jepsen worker 23 - jepsen.util 563 :invoke :write [7 6] INFO [2022-03-13 17:16:26,807] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,807] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,807] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,808] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,809] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,809] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,811] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,811] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,811] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,812] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,812] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,813] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 0] INFO [2022-03-13 17:16:26,813] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,813] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,814] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,815] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:16:26,815] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,816] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:16:26,827] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,827] jepsen worker 89 - jepsen.util 2519 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,828] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 28] INFO [2022-03-13 17:16:26,828] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:26,841] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,842] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 12] INFO [2022-03-13 17:16:26,895] jepsen worker 3 - jepsen.util 1983 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,895] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,899] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,899] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 36] INFO [2022-03-13 17:16:26,934] jepsen worker 4 - jepsen.util 2254 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,934] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:26,940] jepsen worker 42 - jepsen.util 402 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,940] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:26,942] jepsen worker 71 - jepsen.util 521 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,943] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:26,943] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,943] jepsen worker 80 - jepsen.util 800 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,943] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,944] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:26,945] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:26,945] jepsen worker 14 - jepsen.util 374 :invoke :write [4 32] INFO [2022-03-13 17:16:26,954] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,955] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,955] jepsen worker 37 - jepsen.util 2107 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,955] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,956] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 42] INFO [2022-03-13 17:16:26,957] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 24] INFO [2022-03-13 17:16:26,959] jepsen worker 5 - jepsen.util 2435 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,960] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 21] INFO [2022-03-13 17:16:26,963] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,964] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,967] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,968] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:26,974] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,975] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:16:26,980] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:26,981] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:27,037] jepsen worker 3 - jepsen.util 1983 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,038] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 17] INFO [2022-03-13 17:16:27,038] jepsen worker 38 - jepsen.util 2468 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,039] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:16:27,040] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,040] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:16:27,100] jepsen worker 18 - jepsen.util 3438 :info :write [6 44] :RequestTimeoutException INFO [2022-03-13 17:16:27,101] jepsen worker 18 - jepsen.util 3528 :invoke :write [6 0] INFO [2022-03-13 17:16:27,110] jepsen worker 5 - jepsen.util 2435 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,110] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:16:27,126] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,127] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,127] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,127] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,132] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:27,132] jepsen worker nemesis - couchbase.util Rebalance check count: 495 INFO [2022-03-13 17:16:27,132] jepsen worker nemesis - couchbase.util Rebalance stuck count: 395 INFO [2022-03-13 17:16:27,183] jepsen worker 51 - jepsen.util 771 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,183] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,183] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:27,184] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:27,187] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,187] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,187] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,187] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,188] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,188] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:16:27,188] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,189] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,189] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,189] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,190] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,190] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,190] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,191] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 38] INFO [2022-03-13 17:16:27,191] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,192] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,193] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,193] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:16:27,194] jepsen worker 47 - jepsen.util 3467 :info :write [15 39] :RequestTimeoutException INFO [2022-03-13 17:16:27,195] jepsen worker 47 - jepsen.util 3557 :invoke :read [15 nil] INFO [2022-03-13 17:16:27,207] jepsen worker 37 - jepsen.util 2107 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,207] jepsen worker 89 - jepsen.util 2519 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,208] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 43] INFO [2022-03-13 17:16:27,208] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:27,223] jepsen worker 3 - jepsen.util 1983 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,224] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,227] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,227] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 26] INFO [2022-03-13 17:16:27,291] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,291] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,293] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,293] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,295] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,296] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:27,316] jepsen worker 4 - jepsen.util 2254 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,317] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:16:27,360] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,360] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,360] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,361] jepsen worker 69 - jepsen.util 699 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,362] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:27,362] jepsen worker 69 - jepsen.util 699 :invoke :write [23 9] INFO [2022-03-13 17:16:27,364] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:16:27,370] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:27,371] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,372] jepsen worker 24 - jepsen.util 564 :invoke :write [8 1] INFO [2022-03-13 17:16:27,372] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,373] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:27,374] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,375] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:16:27,386] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,386] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,387] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,387] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 39] INFO [2022-03-13 17:16:27,387] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 18] INFO [2022-03-13 17:16:27,388] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:16:27,396] jepsen worker 3 - jepsen.util 1983 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,400] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:16:27,412] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,422] jepsen worker 33 - jepsen.util 753 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,422] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,423] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:16:27,424] jepsen worker 7 - jepsen.util 637 :fail :write [2 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,424] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,425] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,425] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:27,426] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:27,427] jepsen worker 7 - jepsen.util 637 :invoke :write [2 1] INFO [2022-03-13 17:16:27,428] jepsen worker 43 - jepsen.util 583 :invoke :write [14 1] INFO [2022-03-13 17:16:27,428] jepsen worker 53 - jepsen.util 503 :invoke :write [17 7] INFO [2022-03-13 17:16:27,496] jepsen worker 10 - jepsen.util 3520 :info :write [3 32] :RequestTimeoutException INFO [2022-03-13 17:16:27,497] jepsen worker 10 - jepsen.util 3610 :invoke :read [3 nil] INFO [2022-03-13 17:16:27,502] jepsen worker 78 - jepsen.util 618 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,502] jepsen worker 78 - jepsen.util 618 :invoke :write [26 3] INFO [2022-03-13 17:16:27,564] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,564] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:27,583] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,584] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,586] jepsen worker 34 - jepsen.util 934 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,586] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,586] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,587] jepsen worker 80 - jepsen.util 800 :invoke :write [26 1] INFO [2022-03-13 17:16:27,587] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:27,588] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:27,589] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,589] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,591] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,591] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,592] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,592] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,594] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,595] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,595] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,595] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,622] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,623] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:16:27,652] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,652] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,652] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,653] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 0] INFO [2022-03-13 17:16:27,676] jepsen worker 36 - jepsen.util 2196 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,676] jepsen worker 37 - jepsen.util 2107 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,677] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 15] INFO [2022-03-13 17:16:27,677] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 18] INFO [2022-03-13 17:16:27,679] jepsen worker 87 - jepsen.util 2607 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,679] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,680] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,680] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:16:27,680] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,681] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,682] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,682] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:16:27,682] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,683] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:16:27,751] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,751] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,770] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,770] jepsen worker 62 - jepsen.util 512 :invoke :write [20 12] INFO [2022-03-13 17:16:27,772] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,772] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,773] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:27,772] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,772] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,775] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 44] INFO [2022-03-13 17:16:27,775] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:27,776] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:27,776] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:27,777] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:27,786] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,786] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:16:27,797] jepsen worker 81 - jepsen.util 3951 :info :write [27 33] :RequestTimeoutException INFO [2022-03-13 17:16:27,797] jepsen worker 81 - jepsen.util 4041 :invoke :write [27 20] INFO [2022-03-13 17:16:27,803] jepsen worker 14 - jepsen.util 374 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,803] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,803] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,803] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,804] jepsen worker 21 - jepsen.util 831 :invoke :write [7 9] INFO [2022-03-13 17:16:27,805] jepsen worker 14 - jepsen.util 374 :invoke :write [4 4] INFO [2022-03-13 17:16:27,806] jepsen worker 51 - jepsen.util 771 :invoke :write [17 30] INFO [2022-03-13 17:16:27,810] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:27,819] jepsen worker 88 - jepsen.util 2068 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,819] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,820] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:16:27,821] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:27,838] jepsen worker 12 - jepsen.util 732 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,838] jepsen worker 12 - jepsen.util 732 :invoke :write [4 38] INFO [2022-03-13 17:16:27,857] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,858] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,871] jepsen worker 58 - jepsen.util 3208 :ok :read [19 15] INFO [2022-03-13 17:16:27,871] jepsen worker 58 - jepsen.util 3208 :invoke :read [19 nil] INFO [2022-03-13 17:16:27,878] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,878] jepsen worker 89 - jepsen.util 2519 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,878] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 17] INFO [2022-03-13 17:16:27,879] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 26] INFO [2022-03-13 17:16:27,910] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,910] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,911] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 11] INFO [2022-03-13 17:16:27,911] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:27,946] jepsen worker 37 - jepsen.util 2107 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,946] jepsen worker 36 - jepsen.util 2196 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,947] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:27,947] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:27,976] jepsen worker 24 - jepsen.util 564 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,977] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,979] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:16:27,982] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:16:27,988] jepsen worker 13 - jepsen.util 733 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:27,989] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:16:28,015] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,015] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,015] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:28,016] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:28,019] jepsen worker 58 - jepsen.util 3208 :ok :read [19 15] INFO [2022-03-13 17:16:28,020] jepsen worker 89 - jepsen.util 2519 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,020] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,020] jepsen worker 38 - jepsen.util 2468 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,020] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,020] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,020] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,021] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,021] jepsen worker 58 - jepsen.util 3208 :invoke :write [19 15] INFO [2022-03-13 17:16:28,023] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:16:28,024] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:28,025] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:16:28,026] jepsen worker 34 - jepsen.util 934 :invoke :write [11 43] INFO [2022-03-13 17:16:28,027] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,027] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:16:28,028] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:28,029] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,030] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:16:28,030] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 3] INFO [2022-03-13 17:16:28,036] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:28,036] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,037] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,038] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:16:28,038] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:16:28,045] jepsen worker 87 - jepsen.util 2607 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,045] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,045] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,048] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:28,048] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:28,049] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:16:28,051] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,054] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,055] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:28,055] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:16:28,056] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,057] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] WARN [2022-03-13 17:16:28,077] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18023808,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18023808,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9032060,"encode_duration_us":18807,"operation_id":"0xd986","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10001670},{"operation_name":"upsert","last_dispatch_duration_us":16934846,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18471400,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9362452,"encode_duration_us":34263,"operation_id":"0xd7bd","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10008675},{"operation_name":"upsert","last_dispatch_duration_us":17238899,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17238899,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9026756,"encode_duration_us":19048,"operation_id":"0xdef7","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008061}],"total_count":3}} INFO [2022-03-13 17:16:28,138] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:28,138] jepsen worker nemesis - couchbase.util Rebalance check count: 496 INFO [2022-03-13 17:16:28,138] jepsen worker nemesis - couchbase.util Rebalance stuck count: 396 INFO [2022-03-13 17:16:28,178] jepsen worker 7 - jepsen.util 637 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,178] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] WARN [2022-03-13 17:16:28,184] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17302254,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26003140,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9225593,"encode_duration_us":16816,"operation_id":"0xd627","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10008680},{"operation_name":"upsert","last_dispatch_duration_us":17293961,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18236839,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9226935,"encode_duration_us":17701,"operation_id":"0xda86","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10010415},{"operation_name":"upsert","last_dispatch_duration_us":17805669,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17805669,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9049045,"encode_duration_us":17727,"operation_id":"0xdc94","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10008690}],"total_count":3}} INFO [2022-03-13 17:16:28,198] jepsen worker 80 - jepsen.util 800 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,198] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,199] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:28,200] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:28,229] jepsen worker 14 - jepsen.util 374 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,229] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:16:28,274] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,274] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] WARN [2022-03-13 17:16:28,284] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17203328,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17203328,"total_server_duration_us":27,"operation_id":"0xde71","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":17203648},{"operation_name":"upsert","last_dispatch_duration_us":9035794,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":18016305,"total_server_duration_us":43220,"encode_duration_us":20157,"operation_id":"0xde10","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10016260},{"operation_name":"upsert","last_dispatch_duration_us":113361,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":616133,"total_server_duration_us":418581,"encode_duration_us":28209,"operation_id":"0xe2ac","timeout_ms":10000,"last_server_duration_us":102326,"total_duration_us":10009978},{"operation_name":"upsert","last_dispatch_duration_us":39767,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":642195,"total_server_duration_us":380607,"encode_duration_us":19719,"operation_id":"0xe210","timeout_ms":10000,"last_server_duration_us":27854,"total_duration_us":10008999},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":17277,"operation_id":"0xe011","timeout_ms":10000,"total_duration_us":10007556},{"operation_name":"upsert","last_dispatch_duration_us":89173,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":855989,"total_server_duration_us":389530,"encode_duration_us":18073,"operation_id":"0xe19d","timeout_ms":10000,"last_server_duration_us":24325,"total_duration_us":10002398},{"operation_name":"upsert","last_dispatch_duration_us":3691550,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3691550,"total_server_duration_us":35613,"encode_duration_us":18080,"operation_id":"0xe48b","timeout_ms":10000,"last_server_duration_us":35613,"total_duration_us":3695221},{"operation_name":"upsert","last_dispatch_duration_us":3353935,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3353935,"total_server_duration_us":72589,"encode_duration_us":20145,"operation_id":"0xe5c3","timeout_ms":10000,"last_server_duration_us":72589,"total_duration_us":3359800},{"operation_name":"upsert","last_dispatch_duration_us":2972274,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2972274,"total_server_duration_us":48449,"encode_duration_us":18474,"operation_id":"0xe3f9","timeout_ms":10000,"last_server_duration_us":48449,"total_duration_us":2977127},{"operation_name":"get","last_dispatch_duration_us":2112935,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2112935,"total_server_duration_us":43,"operation_id":"0xe6c2","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":2113191}],"total_count":16}} WARN [2022-03-13 17:16:28,378] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14338605,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":14338605,"total_server_duration_us":27,"operation_id":"0xdba6","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14339910},{"operation_name":"get","last_dispatch_duration_us":13786681,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":13786681,"total_server_duration_us":15764,"operation_id":"0xdc02","timeout_ms":30000,"last_server_duration_us":15764,"total_duration_us":13790410},{"operation_name":"get","last_dispatch_duration_us":10296325,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10296325,"total_server_duration_us":27,"operation_id":"0xddc5","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":10298751},{"operation_name":"upsert","last_dispatch_duration_us":36637,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":374156,"total_server_duration_us":221067,"encode_duration_us":21834,"operation_id":"0xdf36","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10010876},{"operation_name":"upsert","encode_duration_us":21491,"operation_id":"0xe209","timeout_ms":10000,"total_duration_us":10008640},{"operation_name":"upsert","last_dispatch_duration_us":9184244,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18228949,"total_server_duration_us":39962,"encode_duration_us":20266,"operation_id":"0xde91","timeout_ms":10000,"last_server_duration_us":18799,"total_duration_us":10008516},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","encode_duration_us":17771,"operation_id":"0xdfe7","timeout_ms":10000,"total_duration_us":10008480},{"operation_name":"upsert","last_dispatch_duration_us":17238899,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":17238899,"total_server_duration_us":9026756,"encode_duration_us":19048,"operation_id":"0xdef7","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008061},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":24492,"operation_id":"0xe18f","timeout_ms":10000,"total_duration_us":10004839},{"operation_name":"upsert","last_dispatch_duration_us":8345127,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8612152,"total_server_duration_us":136423,"encode_duration_us":18175,"operation_id":"0xdf4b","timeout_ms":10000,"last_server_duration_us":27583,"total_duration_us":10003263}],"total_count":60}} INFO [2022-03-13 17:16:28,382] jepsen worker 60 - jepsen.util 600 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,383] jepsen worker 60 - jepsen.util 600 :invoke :write [20 1] WARN [2022-03-13 17:16:28,386] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":20127014,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":20127014,"total_server_duration_us":43,"operation_id":"0xdaf2","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":20129592},{"operation_name":"get","last_dispatch_duration_us":18102765,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":18102765,"total_server_duration_us":23,"operation_id":"0xdc0f","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":18108034},{"operation_name":"get","last_dispatch_duration_us":16770373,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16770373,"total_server_duration_us":38,"operation_id":"0xdcb7","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":16770703},{"operation_name":"get","last_dispatch_duration_us":11439693,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11439693,"total_server_duration_us":62,"operation_id":"0xdfa7","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":11442442},{"operation_name":"upsert","last_dispatch_duration_us":32139,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4253336,"total_server_duration_us":281930,"encode_duration_us":19233,"operation_id":"0xe316","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10009860},{"operation_name":"upsert","last_dispatch_duration_us":153040,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7442857,"total_server_duration_us":93013,"encode_duration_us":17670,"operation_id":"0xe22e","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10006593},{"operation_name":"upsert","last_dispatch_duration_us":54371,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":6424409,"total_server_duration_us":274296,"encode_duration_us":20974,"operation_id":"0xe1e2","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10004148},{"operation_name":"upsert","last_dispatch_duration_us":9038779,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9430363,"total_server_duration_us":9270462,"encode_duration_us":23930,"operation_id":"0xdeed","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10002731},{"operation_name":"upsert","last_dispatch_duration_us":122117,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9806935,"total_server_duration_us":85407,"encode_duration_us":18511,"operation_id":"0xe0f8","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10000919},{"operation_name":"upsert","last_dispatch_duration_us":2766763,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2766763,"total_server_duration_us":214065,"encode_duration_us":30114,"operation_id":"0xe77a","timeout_ms":10000,"last_server_duration_us":214065,"total_duration_us":2772789}],"total_count":16}} WARN [2022-03-13 17:16:28,387] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":31535,"operation_id":"0xe2e2","timeout_ms":10000,"total_duration_us":10006690},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18471,"operation_id":"0xdfa8","timeout_ms":10000,"total_duration_us":10006466},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":16972,"operation_id":"0xdeb1","timeout_ms":10000,"total_duration_us":10004345},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":22358,"operation_id":"0xe0eb","timeout_ms":10000,"total_duration_us":10003684},{"operation_name":"upsert","last_dispatch_duration_us":626533,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":970748,"total_server_duration_us":280734,"encode_duration_us":29425,"operation_id":"0xe581","timeout_ms":10000,"last_server_duration_us":154616,"total_duration_us":991453},{"operation_name":"upsert","last_dispatch_duration_us":433571,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":969571,"total_server_duration_us":136617,"encode_duration_us":25177,"operation_id":"0xe52d","timeout_ms":10000,"last_server_duration_us":73821,"total_duration_us":989395},{"operation_name":"upsert","last_dispatch_duration_us":959456,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":959456,"total_server_duration_us":369144,"encode_duration_us":17189,"operation_id":"0xe829","timeout_ms":10000,"last_server_duration_us":369144,"total_duration_us":964655},{"operation_name":"upsert","last_dispatch_duration_us":588526,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":865102,"total_server_duration_us":225429,"encode_duration_us":20027,"operation_id":"0xe561","timeout_ms":10000,"last_server_duration_us":168413,"total_duration_us":879071},{"operation_name":"get","last_dispatch_duration_us":711382,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":711382,"total_server_duration_us":43,"operation_id":"0xe861","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":711589},{"operation_name":"upsert","last_dispatch_duration_us":627907,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":627907,"total_server_duration_us":164530,"encode_duration_us":22891,"operation_id":"0xe81d","timeout_ms":10000,"last_server_duration_us":164530,"total_duration_us":632035}],"total_count":16}} WARN [2022-03-13 17:16:28,388] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17263635,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17263635,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9062854,"encode_duration_us":17358,"operation_id":"0xdc73","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10001067},{"operation_name":"upsert","last_dispatch_duration_us":17263214,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17769205,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9262071,"encode_duration_us":29303,"operation_id":"0xdbef","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10005123}],"total_count":2}} INFO [2022-03-13 17:16:28,392] jepsen worker 21 - jepsen.util 831 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,393] jepsen worker 21 - jepsen.util 831 :invoke :write [7 26] WARN [2022-03-13 17:16:28,485] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14203800,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":14203800,"total_server_duration_us":38,"operation_id":"0xde94","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":14204108},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":18573,"operation_id":"0xdf4d","timeout_ms":10000,"total_duration_us":10007662},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":21199,"operation_id":"0xdfce","timeout_ms":10000,"total_duration_us":10003136},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":16869,"operation_id":"0xe1a3","timeout_ms":10000,"total_duration_us":10003081},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17169,"operation_id":"0xe2b9","timeout_ms":10000,"total_duration_us":10001642},{"operation_name":"get","last_dispatch_duration_us":974043,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":974043,"total_server_duration_us":11,"operation_id":"0xe5bc","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":2055257},{"operation_name":"upsert","last_dispatch_duration_us":1683123,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1683123,"total_server_duration_us":56647,"encode_duration_us":30316,"operation_id":"0xe45c","timeout_ms":10000,"last_server_duration_us":56647,"total_duration_us":1703235},{"operation_name":"upsert","last_dispatch_duration_us":1644671,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1644671,"total_server_duration_us":18646,"encode_duration_us":19650,"operation_id":"0xe5dc","timeout_ms":10000,"last_server_duration_us":18646,"total_duration_us":1651824},{"operation_name":"upsert","last_dispatch_duration_us":1545349,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1545349,"total_server_duration_us":141324,"encode_duration_us":24623,"operation_id":"0xe3d4","timeout_ms":10000,"last_server_duration_us":141324,"total_duration_us":1649503},{"operation_name":"upsert","last_dispatch_duration_us":1572123,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1572123,"total_server_duration_us":65775,"encode_duration_us":19981,"operation_id":"0xe344","timeout_ms":10000,"last_server_duration_us":65775,"total_duration_us":1580799}],"total_count":84}} WARN [2022-03-13 17:16:28,485] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":24646037,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24646037,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9054355,"encode_duration_us":17948,"operation_id":"0xda6e","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10010607}],"total_count":1}} WARN [2022-03-13 17:16:28,486] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21917285,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21917285,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9029938,"encode_duration_us":21217,"operation_id":"0xd9db","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10001224},{"operation_name":"upsert","last_dispatch_duration_us":9038779,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9430363,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9270462,"encode_duration_us":23930,"operation_id":"0xdeed","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10002731}],"total_count":2}} INFO [2022-03-13 17:16:28,580] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,580] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:16:28,589] jepsen worker 53 - jepsen.util 503 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,591] jepsen worker 53 - jepsen.util 503 :invoke :write [17 38] INFO [2022-03-13 17:16:28,614] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,614] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,615] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:28,615] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:28,688] jepsen worker 16 - jepsen.util 3256 :info :write [5 3] :RequestTimeoutException INFO [2022-03-13 17:16:28,689] jepsen worker 16 - jepsen.util 3346 :invoke :write [5 45] INFO [2022-03-13 17:16:28,697] jepsen worker 86 - jepsen.util 3326 :info :write [28 3] :RequestTimeoutException INFO [2022-03-13 17:16:28,698] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:28,744] jepsen worker 51 - jepsen.util 771 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,745] jepsen worker 51 - jepsen.util 771 :invoke :write [17 3] INFO [2022-03-13 17:16:28,786] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,786] jepsen worker 38 - jepsen.util 2468 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,787] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:28,787] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 12] INFO [2022-03-13 17:16:28,797] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,798] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:28,804] jepsen worker 22 - jepsen.util 652 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,805] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:16:28,834] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,834] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:16:28,836] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,836] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,837] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,837] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:28,838] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:16:28,838] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:28,868] jepsen worker 15 - jepsen.util 3435 :info :write [5 0] :RequestTimeoutException INFO [2022-03-13 17:16:28,869] jepsen worker 15 - jepsen.util 3525 :invoke :write [5 25] INFO [2022-03-13 17:16:28,900] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,901] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:28,902] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,903] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:28,907] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,907] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,908] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:28,908] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 28] INFO [2022-03-13 17:16:28,921] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,922] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,922] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,922] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] INFO [2022-03-13 17:16:28,922] jepsen worker 23 - jepsen.util 563 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,923] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:28,923] jepsen worker 23 - jepsen.util 563 :invoke :write [7 30] INFO [2022-03-13 17:16:28,923] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:28,930] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,931] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,931] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:28,932] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:16:28,932] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,933] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:16:28,943] jepsen worker 34 - jepsen.util 934 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:28,944] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:29,005] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,005] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,005] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,006] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,007] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:16:29,008] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,008] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 36] INFO [2022-03-13 17:16:29,009] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,010] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,011] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:16:29,015] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,016] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:16:29,068] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,068] jepsen worker 38 - jepsen.util 2468 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,068] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,069] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,070] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,070] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,070] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,071] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 33] INFO [2022-03-13 17:16:29,072] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,072] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,073] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,073] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:16:29,099] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,100] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:16:29,103] jepsen worker 43 - jepsen.util 583 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,103] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:29,103] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,103] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,103] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,104] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:29,105] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:29,105] jepsen worker 26 - jepsen.util 476 :invoke :write [8 34] INFO [2022-03-13 17:16:29,106] jepsen worker 42 - jepsen.util 402 :invoke :write [14 1] INFO [2022-03-13 17:16:29,113] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,114] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,118] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:29,130] jepsen worker 89 - jepsen.util 2519 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,130] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 3] INFO [2022-03-13 17:16:29,144] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:29,144] jepsen worker nemesis - couchbase.util Rebalance check count: 497 INFO [2022-03-13 17:16:29,144] jepsen worker nemesis - couchbase.util Rebalance stuck count: 397 INFO [2022-03-13 17:16:29,150] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,151] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:16:29,218] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,218] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,227] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,227] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:16:29,294] jepsen worker 51 - jepsen.util 771 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,295] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,295] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,294] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,295] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:29,296] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:16:29,297] jepsen worker 51 - jepsen.util 771 :invoke :write [17 48] INFO [2022-03-13 17:16:29,298] jepsen worker 33 - jepsen.util 753 :invoke :write [11 36] INFO [2022-03-13 17:16:29,300] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,301] jepsen worker 38 - jepsen.util 2468 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,302] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:29,303] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 44] INFO [2022-03-13 17:16:29,327] jepsen worker 88 - jepsen.util 2068 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,327] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,327] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,327] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,328] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 8] INFO [2022-03-13 17:16:29,328] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 9] INFO [2022-03-13 17:16:29,329] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,329] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:29,330] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,331] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 25] INFO [2022-03-13 17:16:29,407] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,407] jepsen worker 89 - jepsen.util 2519 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,408] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,409] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,423] jepsen worker 4 - jepsen.util 2254 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,424] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,450] jepsen worker 77 - jepsen.util 707 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,451] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:29,483] jepsen worker 88 - jepsen.util 2068 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,483] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,486] jepsen worker 62 - jepsen.util 512 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,487] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:29,530] jepsen worker 38 - jepsen.util 2468 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,531] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,531] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,531] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,531] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 45] INFO [2022-03-13 17:16:29,532] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 34] INFO [2022-03-13 17:16:29,533] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:16:29,533] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,534] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,535] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,536] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,536] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:16:29,544] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,544] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,545] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,546] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:16:29,546] jepsen worker 87 - jepsen.util 2607 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,547] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,548] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,549] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,549] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 31] INFO [2022-03-13 17:16:29,550] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,551] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,551] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,553] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,554] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 33] INFO [2022-03-13 17:16:29,584] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,585] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,586] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,587] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,588] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,588] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:29,630] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,630] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,631] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:29,631] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:29,641] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,641] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,641] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:29,642] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,642] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:16:29,643] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:16:29,646] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,646] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:16:29,652] jepsen worker 3 - jepsen.util 1983 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,652] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,661] jepsen worker 4 - jepsen.util 2254 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,661] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,661] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,662] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:16:29,663] jepsen worker 69 - jepsen.util 699 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,663] jepsen worker 69 - jepsen.util 699 :invoke :write [23 37] INFO [2022-03-13 17:16:29,707] jepsen worker 37 - jepsen.util 2107 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,707] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,745] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,746] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,747] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,748] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,749] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,750] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:16:29,760] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,760] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 34] INFO [2022-03-13 17:16:29,778] jepsen worker 21 - jepsen.util 831 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,779] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:16:29,781] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,783] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 43] INFO [2022-03-13 17:16:29,818] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,819] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 5] INFO [2022-03-13 17:16:29,826] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,827] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:29,827] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,827] jepsen worker 80 - jepsen.util 800 :invoke :write [26 42] INFO [2022-03-13 17:16:29,828] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,828] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:16:29,841] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,841] jepsen worker 79 - jepsen.util 1069 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,841] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:29,842] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:29,857] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,857] jepsen worker 87 - jepsen.util 2607 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,858] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,858] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:16:29,872] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,872] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,873] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,874] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 19] INFO [2022-03-13 17:16:29,891] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,892] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:29,893] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,893] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:16:29,902] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,903] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:16:29,928] jepsen worker 89 - jepsen.util 2519 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,928] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,929] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:29,929] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:29,932] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,932] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:29,989] jepsen worker 51 - jepsen.util 771 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,990] jepsen worker 51 - jepsen.util 771 :invoke :write [17 41] INFO [2022-03-13 17:16:29,991] jepsen worker 78 - jepsen.util 618 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,991] jepsen worker 78 - jepsen.util 618 :invoke :write [26 9] INFO [2022-03-13 17:16:29,994] jepsen worker 83 - jepsen.util 3053 :info :write [27 2] :RequestTimeoutException INFO [2022-03-13 17:16:29,995] jepsen worker 83 - jepsen.util 3143 :invoke :write [27 13] INFO [2022-03-13 17:16:29,996] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,996] jepsen worker 88 - jepsen.util 2068 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:29,997] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:16:29,997] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,004] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,004] jepsen worker 36 - jepsen.util 2196 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,004] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:16:30,005] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,008] jepsen worker 49 - jepsen.util 3469 :ok :read [16 :nil] INFO [2022-03-13 17:16:30,008] jepsen worker 49 - jepsen.util 3469 :invoke :write [16 14] INFO [2022-03-13 17:16:30,021] jepsen worker 3 - jepsen.util 1983 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,022] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:16:30,031] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,031] jepsen worker 38 - jepsen.util 2468 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,031] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,032] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:16:30,032] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,033] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 46] INFO [2022-03-13 17:16:30,033] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,034] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 30] INFO [2022-03-13 17:16:30,072] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,073] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 8] INFO [2022-03-13 17:16:30,099] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,100] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:30,130] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,131] jepsen worker 37 - jepsen.util 2107 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,131] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 17] INFO [2022-03-13 17:16:30,132] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 0] INFO [2022-03-13 17:16:30,147] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,148] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:16:30,155] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:30,155] jepsen worker nemesis - couchbase.util Rebalance check count: 498 INFO [2022-03-13 17:16:30,155] jepsen worker nemesis - couchbase.util Rebalance stuck count: 398 INFO [2022-03-13 17:16:30,181] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,182] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,183] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,184] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,227] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,228] jepsen worker 71 - jepsen.util 521 :invoke :write [23 26] INFO [2022-03-13 17:16:30,230] jepsen worker 38 - jepsen.util 2468 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,230] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,232] jepsen worker 53 - jepsen.util 503 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,232] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,232] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,232] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:30,233] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:30,233] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:30,234] jepsen worker 43 - jepsen.util 583 :invoke :write [14 27] INFO [2022-03-13 17:16:30,234] jepsen worker 8 - jepsen.util 728 :invoke :write [2 45] INFO [2022-03-13 17:16:30,236] jepsen worker 33 - jepsen.util 753 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,236] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,237] jepsen worker 33 - jepsen.util 753 :invoke :write [11 45] INFO [2022-03-13 17:16:30,237] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:16:30,240] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,240] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,241] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:30,241] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:16:30,242] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,243] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,244] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,245] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 42] INFO [2022-03-13 17:16:30,272] jepsen worker 36 - jepsen.util 2196 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,272] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,349] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,350] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,353] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,353] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:16:30,357] jepsen worker 37 - jepsen.util 2107 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,357] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,358] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,358] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,358] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,359] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:30,359] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 0] INFO [2022-03-13 17:16:30,360] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,361] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,363] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:30,363] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:30,369] jepsen worker 61 - jepsen.util 691 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,370] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:30,373] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:30,386] jepsen worker 5 - jepsen.util 2435 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,386] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 32] INFO [2022-03-13 17:16:30,418] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,418] jepsen worker 71 - jepsen.util 521 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,418] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,418] jepsen worker 75 - jepsen.util 435 :invoke :write [25 0] INFO [2022-03-13 17:16:30,419] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:16:30,420] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:30,425] jepsen worker 89 - jepsen.util 2519 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,426] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,426] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,426] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,427] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,427] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:16:30,427] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,428] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,429] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,430] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,433] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,433] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:16:30,442] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,443] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:16:30,450] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,450] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,451] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,451] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,456] jepsen worker 26 - jepsen.util 476 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,457] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:30,459] jepsen worker 80 - jepsen.util 800 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,459] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,460] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:30,461] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:30,489] jepsen worker 37 - jepsen.util 2107 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,489] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,490] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,490] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,491] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,491] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 1] INFO [2022-03-13 17:16:30,491] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,492] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:16:30,500] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,501] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,505] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,505] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 13] INFO [2022-03-13 17:16:30,511] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,511] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,511] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,512] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,512] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:16:30,513] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:16:30,514] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,515] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 5] INFO [2022-03-13 17:16:30,529] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,530] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 48] INFO [2022-03-13 17:16:30,555] jepsen worker 51 - jepsen.util 771 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,555] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,556] jepsen worker 51 - jepsen.util 771 :invoke :write [17 15] INFO [2022-03-13 17:16:30,556] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:30,563] jepsen worker 44 - jepsen.util 584 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,564] jepsen worker 44 - jepsen.util 584 :invoke :write [14 4] INFO [2022-03-13 17:16:30,583] jepsen worker 5 - jepsen.util 2435 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,583] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 24] INFO [2022-03-13 17:16:30,620] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,621] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 47] INFO [2022-03-13 17:16:30,634] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,635] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:16:30,656] jepsen worker 37 - jepsen.util 2107 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,656] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,659] jepsen worker 75 - jepsen.util 435 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,660] jepsen worker 75 - jepsen.util 435 :invoke :write [25 45] INFO [2022-03-13 17:16:30,674] jepsen worker 3 - jepsen.util 1983 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,675] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:30,703] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,703] jepsen worker 42 - jepsen.util 402 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,703] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,703] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,703] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,703] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,704] jepsen worker 62 - jepsen.util 512 :invoke :write [20 29] INFO [2022-03-13 17:16:30,705] jepsen worker 7 - jepsen.util 637 :invoke :write [2 27] INFO [2022-03-13 17:16:30,706] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:30,706] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:30,707] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:30,708] jepsen worker 53 - jepsen.util 503 :invoke :write [17 27] INFO [2022-03-13 17:16:30,716] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,716] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:16:30,724] jepsen worker 36 - jepsen.util 2196 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,725] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,744] jepsen worker 4 - jepsen.util 2254 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,744] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,776] jepsen worker 5 - jepsen.util 2435 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,777] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:16:30,785] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,785] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,786] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:30,787] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 19] INFO [2022-03-13 17:16:30,791] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,791] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:16:30,813] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,814] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 26] INFO [2022-03-13 17:16:30,825] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,826] jepsen worker 24 - jepsen.util 564 :invoke :write [8 18] INFO [2022-03-13 17:16:30,828] jepsen worker 52 - jepsen.util 682 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,830] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:30,832] jepsen worker 33 - jepsen.util 753 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,832] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,833] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,833] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,833] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,833] jepsen worker 14 - jepsen.util 374 :invoke :write [4 29] INFO [2022-03-13 17:16:30,833] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,834] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:30,834] jepsen worker 80 - jepsen.util 800 :invoke :write [26 28] INFO [2022-03-13 17:16:30,835] jepsen worker 33 - jepsen.util 753 :invoke :write [11 14] INFO [2022-03-13 17:16:30,836] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:30,837] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:30,843] jepsen worker 40 - jepsen.util 3640 :info :write [13 17] :SyncWriteAmbiguous INFO [2022-03-13 17:16:30,843] jepsen worker 40 - jepsen.util 3730 :invoke :read [13 nil] INFO [2022-03-13 17:16:30,866] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,867] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 24] INFO [2022-03-13 17:16:30,878] jepsen worker 23 - jepsen.util 563 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,878] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,879] jepsen worker 23 - jepsen.util 563 :invoke :write [7 27] INFO [2022-03-13 17:16:30,880] jepsen worker 77 - jepsen.util 707 :invoke :write [25 43] INFO [2022-03-13 17:16:30,881] jepsen worker 51 - jepsen.util 771 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,881] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:30,908] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,908] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,909] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,909] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,909] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:30,910] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:30,911] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:30,912] jepsen worker 35 - jepsen.util 935 :invoke :write [11 48] INFO [2022-03-13 17:16:30,917] jepsen worker 89 - jepsen.util 2519 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,917] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,918] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,918] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,918] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,918] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:16:30,919] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,920] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 34] INFO [2022-03-13 17:16:30,924] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,925] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:30,926] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,927] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 7] INFO [2022-03-13 17:16:30,950] jepsen worker 80 - jepsen.util 800 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,951] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:30,958] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,960] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,967] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,968] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:30,969] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,970] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 29] INFO [2022-03-13 17:16:30,980] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:30,981] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:16:31,000] jepsen worker 88 - jepsen.util 2068 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,000] jepsen worker 37 - jepsen.util 2107 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,000] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,001] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,001] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,002] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,018] jepsen worker 43 - jepsen.util 583 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,018] jepsen worker 43 - jepsen.util 583 :invoke :write [14 9] INFO [2022-03-13 17:16:31,027] jepsen worker 3 - jepsen.util 1983 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,028] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,032] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,032] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:16:31,068] jepsen worker 4 - jepsen.util 2254 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,068] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,068] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,069] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 0] INFO [2022-03-13 17:16:31,070] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,070] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 19] INFO [2022-03-13 17:16:31,071] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,071] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 9] INFO [2022-03-13 17:16:31,117] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,117] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,117] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,118] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 46] INFO [2022-03-13 17:16:31,119] jepsen worker 14 - jepsen.util 374 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,120] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:31,140] jepsen worker 75 - jepsen.util 435 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,141] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:31,159] jepsen worker 33 - jepsen.util 753 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,159] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,159] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,159] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,160] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,161] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:31,161] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:31,162] jepsen worker nemesis - couchbase.util Rebalance check count: 499 INFO [2022-03-13 17:16:31,162] jepsen worker nemesis - couchbase.util Rebalance stuck count: 399 INFO [2022-03-13 17:16:31,162] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:16:31,162] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:31,163] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:31,163] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:31,169] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,169] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,169] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,170] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:16:31,177] jepsen worker 4 - jepsen.util 2254 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,178] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,191] jepsen worker 64 - jepsen.util 3394 :info :write [21 42] :RequestTimeoutException INFO [2022-03-13 17:16:31,191] jepsen worker 64 - jepsen.util 3484 :invoke :write [21 45] INFO [2022-03-13 17:16:31,199] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,200] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,201] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,202] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 45] INFO [2022-03-13 17:16:31,214] jepsen worker 72 - jepsen.util 3672 :info :write [24 25] :RequestTimeoutException INFO [2022-03-13 17:16:31,214] jepsen worker 72 - jepsen.util 3762 :invoke :write [24 30] INFO [2022-03-13 17:16:31,249] jepsen worker 88 - jepsen.util 2068 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,250] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 1] INFO [2022-03-13 17:16:31,261] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,261] jepsen worker 89 - jepsen.util 2519 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,262] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 40] INFO [2022-03-13 17:16:31,263] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 6] INFO [2022-03-13 17:16:31,268] jepsen worker 5 - jepsen.util 2435 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,268] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,269] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,270] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,274] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,275] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,278] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,278] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 46] INFO [2022-03-13 17:16:31,287] jepsen worker 69 - jepsen.util 699 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,288] jepsen worker 69 - jepsen.util 699 :invoke :write [23 3] INFO [2022-03-13 17:16:31,294] jepsen worker 35 - jepsen.util 935 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,294] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,294] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,294] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,294] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,295] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,295] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,295] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:31,296] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:31,296] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:31,297] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:31,297] jepsen worker 35 - jepsen.util 935 :invoke :write [11 19] INFO [2022-03-13 17:16:31,302] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:31,303] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:31,339] jepsen worker 3 - jepsen.util 1983 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,340] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,341] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,341] jepsen worker 24 - jepsen.util 564 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,342] jepsen worker 24 - jepsen.util 564 :invoke :write [8 41] INFO [2022-03-13 17:16:31,342] jepsen worker 37 - jepsen.util 2107 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,342] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 33] INFO [2022-03-13 17:16:31,343] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 5] INFO [2022-03-13 17:16:31,354] jepsen worker 8 - jepsen.util 728 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,357] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,357] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:31,357] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,357] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,359] jepsen worker 26 - jepsen.util 476 :invoke :write [8 7] INFO [2022-03-13 17:16:31,359] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:31,360] jepsen worker 8 - jepsen.util 728 :invoke :write [2 36] INFO [2022-03-13 17:16:31,360] jepsen worker 6 - jepsen.util 636 :invoke :write [2 39] INFO [2022-03-13 17:16:31,366] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:31,366] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,366] jepsen worker 87 - jepsen.util 2607 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,367] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 36] INFO [2022-03-13 17:16:31,368] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,369] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,369] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,369] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,369] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,369] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,371] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,372] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,375] jepsen worker 51 - jepsen.util 771 :invoke :write [17 26] INFO [2022-03-13 17:16:31,376] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:31,376] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:16:31,377] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:16:31,377] jepsen worker 33 - jepsen.util 753 :invoke :write [11 14] INFO [2022-03-13 17:16:31,379] jepsen worker 34 - jepsen.util 934 :invoke :write [11 18] INFO [2022-03-13 17:16:31,382] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:16:31,406] jepsen worker 5 - jepsen.util 2435 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,411] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 26] INFO [2022-03-13 17:16:31,433] jepsen worker 35 - jepsen.util 935 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,433] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,435] jepsen worker 12 - jepsen.util 732 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,435] jepsen worker 80 - jepsen.util 800 :invoke :write [26 34] INFO [2022-03-13 17:16:31,436] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,436] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:31,437] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:31,438] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:31,440] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,440] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,441] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:16:31,441] jepsen worker 75 - jepsen.util 435 :invoke :write [25 12] INFO [2022-03-13 17:16:31,451] jepsen worker 88 - jepsen.util 2068 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,451] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,451] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,452] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:16:31,469] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,470] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 4] INFO [2022-03-13 17:16:31,494] jepsen worker 7 - jepsen.util 637 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,494] jepsen worker 7 - jepsen.util 637 :invoke :write [2 43] INFO [2022-03-13 17:16:31,498] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,498] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:31,505] jepsen worker 89 - jepsen.util 2519 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,506] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,522] jepsen worker 3 - jepsen.util 1983 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,523] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 31] INFO [2022-03-13 17:16:31,567] jepsen worker 87 - jepsen.util 2607 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,567] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,568] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,568] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 39] INFO [2022-03-13 17:16:31,581] jepsen worker 37 - jepsen.util 2107 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,582] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 39] INFO [2022-03-13 17:16:31,629] jepsen worker 5 - jepsen.util 2435 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,630] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,634] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,635] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,636] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,637] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,647] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,647] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,647] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,648] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:16:31,651] jepsen worker 36 - jepsen.util 2196 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,652] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,653] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,653] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,656] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,656] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,657] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,657] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,658] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,658] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,659] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,659] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 8] INFO [2022-03-13 17:16:31,662] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,663] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 1] INFO [2022-03-13 17:16:31,701] jepsen worker 3 - jepsen.util 1983 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,701] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,702] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 25] INFO [2022-03-13 17:16:31,703] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:31,706] jepsen worker 78 - jepsen.util 618 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,706] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,707] jepsen worker 78 - jepsen.util 618 :invoke :write [26 14] INFO [2022-03-13 17:16:31,707] jepsen worker 61 - jepsen.util 691 :invoke :write [20 37] INFO [2022-03-13 17:16:31,708] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,709] jepsen worker 70 - jepsen.util 790 :invoke :write [23 4] INFO [2022-03-13 17:16:31,710] jepsen worker 24 - jepsen.util 564 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,710] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:31,722] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,723] jepsen worker 87 - jepsen.util 2607 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,723] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,723] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:16:31,727] jepsen worker 33 - jepsen.util 753 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,728] jepsen worker 33 - jepsen.util 753 :invoke :write [11 9] INFO [2022-03-13 17:16:31,734] jepsen worker 38 - jepsen.util 2468 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,735] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,745] jepsen worker 53 - jepsen.util 503 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,746] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:31,796] jepsen worker 39 - jepsen.util 3639 :info :write [13 35] :RequestTimeoutException INFO [2022-03-13 17:16:31,797] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,797] jepsen worker 39 - jepsen.util 3729 :invoke :read [13 nil] INFO [2022-03-13 17:16:31,798] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,798] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,798] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,842] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,843] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,869] jepsen worker 89 - jepsen.util 2519 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,869] jepsen worker 37 - jepsen.util 2107 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,869] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,870] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 26] INFO [2022-03-13 17:16:31,870] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,870] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,871] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:16:31,871] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,871] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 30] INFO [2022-03-13 17:16:31,873] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:16:31,873] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 18] INFO [2022-03-13 17:16:31,874] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,876] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,876] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,878] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,878] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,879] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,880] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,881] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,882] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:31,883] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,883] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 49] INFO [2022-03-13 17:16:31,909] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,909] jepsen worker 3 - jepsen.util 1983 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,910] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,910] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:16:31,911] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,912] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,913] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,914] jepsen worker 60 - jepsen.util 600 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,914] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:31,915] jepsen worker 60 - jepsen.util 600 :invoke :write [20 1] INFO [2022-03-13 17:16:31,916] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,917] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:16:31,921] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,922] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:31,931] jepsen worker 36 - jepsen.util 2196 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,932] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:31,992] jepsen worker 88 - jepsen.util 2068 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:31,993] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:32,016] jepsen worker 48 - jepsen.util 3468 :info :write [16 32] :RequestTimeoutException INFO [2022-03-13 17:16:32,017] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 34] INFO [2022-03-13 17:16:32,017] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,018] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,027] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,027] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:16:32,069] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,070] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,075] jepsen worker 28 - jepsen.util 3628 :ok :read [9 26] INFO [2022-03-13 17:16:32,076] jepsen worker 28 - jepsen.util 3628 :invoke :write [9 16] INFO [2022-03-13 17:16:32,085] jepsen worker 54 - jepsen.util 3204 :ok :read [18 21] INFO [2022-03-13 17:16:32,086] jepsen worker 54 - jepsen.util 3204 :invoke :write [18 26] INFO [2022-03-13 17:16:32,087] jepsen worker 89 - jepsen.util 2519 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,087] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,088] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,088] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:32,089] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:32,089] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:32,091] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,091] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,092] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 49] INFO [2022-03-13 17:16:32,092] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:32,101] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,101] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,102] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,102] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:32,103] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,104] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 36] INFO [2022-03-13 17:16:32,181] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,181] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,181] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,181] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,181] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,181] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,181] jepsen worker 75 - jepsen.util 435 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,181] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,182] jepsen worker 62 - jepsen.util 512 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,182] jepsen worker 87 - jepsen.util 2607 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,182] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,184] jepsen worker 44 - jepsen.util 584 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,187] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,188] jepsen worker 75 - jepsen.util 435 :invoke :write [25 37] INFO [2022-03-13 17:16:32,188] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:32,188] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,189] jepsen worker nemesis - couchbase.util Rebalance check count: 500 INFO [2022-03-13 17:16:32,189] jepsen worker nemesis - couchbase.util Rebalance stuck count: 400 INFO [2022-03-13 17:16:32,189] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:16:32,190] jepsen worker 1 - jepsen.util 3061 :info :write [0 9] :RequestTimeoutException INFO [2022-03-13 17:16:32,191] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:16:32,193] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:16:32,194] jepsen worker 43 - jepsen.util 583 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,194] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:16:32,194] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,194] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:32,196] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:32,197] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:16:32,198] jepsen worker 1 - jepsen.util 3151 :invoke :write [0 16] INFO [2022-03-13 17:16:32,199] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:16:32,199] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,199] jepsen worker 37 - jepsen.util 2107 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,200] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:16:32,205] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:32,205] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:32,205] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:32,205] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 36] INFO [2022-03-13 17:16:32,206] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 16] INFO [2022-03-13 17:16:32,207] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:16:32,208] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:16:32,210] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:32,214] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,222] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:32,266] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,267] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,268] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,269] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,270] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,270] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,271] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,272] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,273] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,274] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:16:32,302] jepsen worker 89 - jepsen.util 2519 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,303] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 30] INFO [2022-03-13 17:16:32,326] jepsen worker 34 - jepsen.util 934 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,327] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:32,332] jepsen worker 23 - jepsen.util 563 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,333] jepsen worker 23 - jepsen.util 563 :invoke :write [7 14] INFO [2022-03-13 17:16:32,356] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,357] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,359] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,360] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,361] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,362] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:16:32,365] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,366] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,370] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,371] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 16] INFO [2022-03-13 17:16:32,375] jepsen worker 32 - jepsen.util 3542 :info :write [10 21] :SyncWriteAmbiguous INFO [2022-03-13 17:16:32,376] jepsen worker 32 - jepsen.util 3632 :invoke :write [10 0] INFO [2022-03-13 17:16:32,411] jepsen worker 36 - jepsen.util 2196 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,411] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 11] INFO [2022-03-13 17:16:32,440] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,440] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 16] INFO [2022-03-13 17:16:32,458] jepsen worker 17 - jepsen.util 3527 :info :write [5 26] :RequestTimeoutException INFO [2022-03-13 17:16:32,459] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,459] jepsen worker 17 - jepsen.util 3617 :invoke :write [5 19] INFO [2022-03-13 17:16:32,460] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 30] INFO [2022-03-13 17:16:32,484] jepsen worker 37 - jepsen.util 2107 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,485] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 26] INFO [2022-03-13 17:16:32,529] jepsen worker 75 - jepsen.util 435 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,529] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,530] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:32,531] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:32,536] jepsen worker 5 - jepsen.util 2435 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,537] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,541] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,542] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,543] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,544] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,546] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,546] jepsen worker 8 - jepsen.util 728 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,546] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,546] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,547] jepsen worker 8 - jepsen.util 728 :invoke :write [2 28] INFO [2022-03-13 17:16:32,547] jepsen worker 51 - jepsen.util 771 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,548] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,548] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:32,549] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 28] INFO [2022-03-13 17:16:32,549] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 30] INFO [2022-03-13 17:16:32,551] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,552] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:16:32,594] jepsen worker 89 - jepsen.util 2519 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,595] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 21] INFO [2022-03-13 17:16:32,647] jepsen worker 87 - jepsen.util 2607 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,648] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:32,669] jepsen worker 3 - jepsen.util 1983 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,671] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:16:32,680] jepsen worker 19 - jepsen.util 3259 :ok :read [6 16] INFO [2022-03-13 17:16:32,681] jepsen worker 19 - jepsen.util 3259 :invoke :write [6 30] INFO [2022-03-13 17:16:32,683] jepsen worker 38 - jepsen.util 2468 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,683] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:32,728] jepsen worker 88 - jepsen.util 2068 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,729] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:32,759] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,759] jepsen worker 80 - jepsen.util 800 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,760] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:32,760] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:32,783] jepsen worker 4 - jepsen.util 2254 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,783] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,783] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,783] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:32,784] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,784] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:16:32,784] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,785] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:32,786] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,787] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,787] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,788] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 11] INFO [2022-03-13 17:16:32,788] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,788] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,790] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,790] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,791] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,792] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:16:32,803] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,803] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,809] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,810] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:32,812] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,813] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:32,814] jepsen worker 36 - jepsen.util 2196 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,817] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 2] INFO [2022-03-13 17:16:32,868] jepsen worker 37 - jepsen.util 2107 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,868] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,868] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 26] INFO [2022-03-13 17:16:32,869] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:32,870] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,871] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:32,872] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,872] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 34] INFO [2022-03-13 17:16:32,878] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,879] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 8] INFO [2022-03-13 17:16:32,879] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,880] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 27] INFO [2022-03-13 17:16:32,936] jepsen worker 89 - jepsen.util 2519 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,937] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:32,938] jepsen worker 77 - jepsen.util 707 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,938] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,938] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,939] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:16:32,939] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:32,940] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:32,945] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,946] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:16:32,958] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,958] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,958] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,959] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:32,959] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:16:32,959] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:32,960] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,960] jepsen worker 33 - jepsen.util 753 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,960] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:32,961] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:32,961] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,962] jepsen worker 24 - jepsen.util 564 :invoke :write [8 29] INFO [2022-03-13 17:16:32,984] jepsen worker 88 - jepsen.util 2068 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:32,985] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 13] INFO [2022-03-13 17:16:33,030] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,031] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,034] jepsen worker 3 - jepsen.util 1983 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,035] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 5] INFO [2022-03-13 17:16:33,041] jepsen worker 20 - jepsen.util 3890 :ok :read [6 16] INFO [2022-03-13 17:16:33,042] jepsen worker 20 - jepsen.util 3890 :invoke :read [6 nil] INFO [2022-03-13 17:16:33,045] jepsen worker 36 - jepsen.util 2196 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,045] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,046] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,047] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 6] INFO [2022-03-13 17:16:33,078] jepsen worker 74 - jepsen.util 3764 :ok :read [24 24] INFO [2022-03-13 17:16:33,079] jepsen worker 74 - jepsen.util 3764 :invoke :read [24 nil] INFO [2022-03-13 17:16:33,094] jepsen worker 87 - jepsen.util 2607 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,095] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 35] INFO [2022-03-13 17:16:33,127] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,127] jepsen worker 38 - jepsen.util 2468 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,127] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 42] INFO [2022-03-13 17:16:33,128] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,134] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,135] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:16:33,140] jepsen worker 11 - jepsen.util 3431 :info :write [3 14] :RequestTimeoutException INFO [2022-03-13 17:16:33,140] jepsen worker 11 - jepsen.util 3521 :invoke :write [3 2] INFO [2022-03-13 17:16:33,147] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,147] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:16:33,151] jepsen worker 6 - jepsen.util 636 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,152] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:33,167] jepsen worker 9 - jepsen.util 2889 :info :write [3 25] :RequestTimeoutException INFO [2022-03-13 17:16:33,168] jepsen worker 9 - jepsen.util 2979 :invoke :read [3 nil] INFO [2022-03-13 17:16:33,175] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,175] jepsen worker 88 - jepsen.util 2068 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,175] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,176] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,176] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,177] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:33,181] jepsen worker 56 - jepsen.util 3656 :ok :read [18 21] INFO [2022-03-13 17:16:33,181] jepsen worker 56 - jepsen.util 3656 :invoke :write [18 37] INFO [2022-03-13 17:16:33,195] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:33,195] jepsen worker nemesis - couchbase.util Rebalance check count: 501 INFO [2022-03-13 17:16:33,195] jepsen worker nemesis - couchbase.util Rebalance stuck count: 401 INFO [2022-03-13 17:16:33,196] jepsen worker 9 - jepsen.util 2979 :ok :read [3 25] INFO [2022-03-13 17:16:33,197] jepsen worker 9 - jepsen.util 2979 :invoke :read [3 nil] INFO [2022-03-13 17:16:33,220] jepsen worker 9 - jepsen.util 2979 :ok :read [3 25] INFO [2022-03-13 17:16:33,220] jepsen worker 9 - jepsen.util 2979 :invoke :read [3 nil] INFO [2022-03-13 17:16:33,230] jepsen worker 87 - jepsen.util 2607 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,230] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,230] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,231] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,231] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 7] INFO [2022-03-13 17:16:33,231] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,234] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,234] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,234] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:16:33,235] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 32] INFO [2022-03-13 17:16:33,248] jepsen worker 9 - jepsen.util 2979 :ok :read [3 25] INFO [2022-03-13 17:16:33,248] jepsen worker 9 - jepsen.util 2979 :invoke :read [3 nil] INFO [2022-03-13 17:16:33,254] jepsen worker 3 - jepsen.util 1983 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,254] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,256] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,257] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:16:33,262] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,262] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 48] INFO [2022-03-13 17:16:33,264] jepsen worker 37 - jepsen.util 2107 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,265] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 34] INFO [2022-03-13 17:16:33,279] jepsen worker 9 - jepsen.util 2979 :ok :read [3 25] INFO [2022-03-13 17:16:33,279] jepsen worker 9 - jepsen.util 2979 :invoke :write [3 45] INFO [2022-03-13 17:16:33,316] jepsen worker 36 - jepsen.util 2196 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,317] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,345] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,346] jepsen worker 75 - jepsen.util 435 :invoke :write [25 23] INFO [2022-03-13 17:16:33,350] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,350] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:16:33,363] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,363] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,364] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:16:33,364] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:33,371] jepsen worker 87 - jepsen.util 2607 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,372] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,387] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,387] jepsen worker 89 - jepsen.util 2519 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,387] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:16:33,388] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 1] INFO [2022-03-13 17:16:33,396] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,397] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,397] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,397] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,397] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,397] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,397] jepsen worker 21 - jepsen.util 831 :invoke :write [7 24] INFO [2022-03-13 17:16:33,398] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,398] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:33,399] jepsen worker 14 - jepsen.util 374 :invoke :write [4 22] INFO [2022-03-13 17:16:33,400] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:33,400] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:33,401] jepsen worker 34 - jepsen.util 934 :invoke :write [11 32] INFO [2022-03-13 17:16:33,402] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:33,432] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,433] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,434] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,435] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:16:33,460] jepsen worker 38 - jepsen.util 2468 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,461] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,495] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,496] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 43] INFO [2022-03-13 17:16:33,504] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,505] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,507] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,508] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,509] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,509] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 34] INFO [2022-03-13 17:16:33,559] jepsen worker 24 - jepsen.util 564 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,560] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] INFO [2022-03-13 17:16:33,563] jepsen worker 69 - jepsen.util 699 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,564] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:33,570] jepsen worker 37 - jepsen.util 2107 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,570] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,571] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,572] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 36] INFO [2022-03-13 17:16:33,576] jepsen worker 29 - jepsen.util 3719 :info :write [9 25] :RequestTimeoutException INFO [2022-03-13 17:16:33,577] jepsen worker 29 - jepsen.util 3809 :invoke :write [9 11] INFO [2022-03-13 17:16:33,586] jepsen worker 88 - jepsen.util 2068 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,587] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,587] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,588] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,596] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,596] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,596] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,596] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,596] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,597] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:16:33,597] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:16:33,598] jepsen worker 89 - jepsen.util 2519 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,599] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,599] jepsen worker 33 - jepsen.util 753 :invoke :write [11 37] INFO [2022-03-13 17:16:33,599] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,600] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,600] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:33,600] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:33,605] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,605] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,618] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,619] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:33,637] jepsen worker 36 - jepsen.util 2196 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,638] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,640] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,640] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,659] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,659] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,659] jepsen worker 37 - jepsen.util 2107 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,659] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,659] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 44] INFO [2022-03-13 17:16:33,660] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 9] INFO [2022-03-13 17:16:33,660] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:16:33,661] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,662] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,662] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 42] INFO [2022-03-13 17:16:33,684] jepsen worker 4 - jepsen.util 2254 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,684] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,684] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,684] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:16:33,685] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,685] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 44] INFO [2022-03-13 17:16:33,691] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,692] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,694] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,695] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 28] INFO [2022-03-13 17:16:33,698] jepsen worker 5 - jepsen.util 2435 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,699] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,700] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,700] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,706] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,707] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:33,752] jepsen worker 7 - jepsen.util 637 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,753] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,753] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:33,754] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:33,758] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,758] jepsen worker 13 - jepsen.util 733 :invoke :write [4 31] INFO [2022-03-13 17:16:33,766] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,766] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:16:33,770] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,771] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:33,771] jepsen worker 89 - jepsen.util 2519 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,771] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,789] jepsen worker 88 - jepsen.util 2068 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,790] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,799] jepsen worker 63 - jepsen.util 3573 :info :write [21 46] :RequestTimeoutException INFO [2022-03-13 17:16:33,800] jepsen worker 63 - jepsen.util 3663 :invoke :read [21 nil] INFO [2022-03-13 17:16:33,801] jepsen worker 21 - jepsen.util 831 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,801] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:33,814] jepsen worker 37 - jepsen.util 2107 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,815] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 40] INFO [2022-03-13 17:16:33,832] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,833] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 6] INFO [2022-03-13 17:16:33,873] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,873] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,888] jepsen worker 36 - jepsen.util 2196 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,888] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,889] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:33,889] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 18] INFO [2022-03-13 17:16:33,889] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,890] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:16:33,890] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,891] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:16:33,941] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,942] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:16:33,958] jepsen worker 5 - jepsen.util 2435 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,959] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 27] INFO [2022-03-13 17:16:33,976] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,976] jepsen worker 87 - jepsen.util 2607 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,976] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,977] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,977] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 24] INFO [2022-03-13 17:16:33,977] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:33,979] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,979] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,979] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 23] INFO [2022-03-13 17:16:33,980] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,981] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,982] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:33,984] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:33,985] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:16:34,005] jepsen worker 75 - jepsen.util 435 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,006] jepsen worker 31 - jepsen.util 3721 :info :write [10 34] :RequestTimeoutException INFO [2022-03-13 17:16:34,007] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:34,008] jepsen worker 31 - jepsen.util 3811 :invoke :write [10 12] INFO [2022-03-13 17:16:34,029] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,030] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:16:34,044] jepsen worker 37 - jepsen.util 2107 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,044] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 35] INFO [2022-03-13 17:16:34,090] jepsen worker 88 - jepsen.util 2068 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,091] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:34,098] jepsen worker 89 - jepsen.util 2519 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,099] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:16:34,129] jepsen worker 5 - jepsen.util 2435 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,130] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:16:34,167] jepsen worker 61 - jepsen.util 691 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,168] jepsen worker 61 - jepsen.util 691 :invoke :write [20 2] INFO [2022-03-13 17:16:34,183] jepsen worker 4 - jepsen.util 2254 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,184] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,185] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,185] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 40] INFO [2022-03-13 17:16:34,186] jepsen worker 14 - jepsen.util 374 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,186] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:34,187] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,188] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 8] INFO [2022-03-13 17:16:34,202] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:34,202] jepsen worker nemesis - couchbase.util Rebalance check count: 502 INFO [2022-03-13 17:16:34,202] jepsen worker nemesis - couchbase.util Rebalance stuck count: 402 INFO [2022-03-13 17:16:34,204] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,204] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,205] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,205] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:34,205] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:34,206] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:16:34,207] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,207] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,207] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:34,208] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:16:34,209] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,210] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:34,237] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,238] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:16:34,257] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,258] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,263] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,263] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:34,271] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,271] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,271] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,272] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 44] INFO [2022-03-13 17:16:34,273] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,273] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:16:34,275] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,275] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,277] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,277] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,279] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,279] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,283] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,284] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:16:34,309] jepsen worker 85 - jepsen.util 3145 :ok :read [28 44] INFO [2022-03-13 17:16:34,309] jepsen worker 85 - jepsen.util 3145 :invoke :write [28 14] INFO [2022-03-13 17:16:34,319] jepsen worker 57 - jepsen.util 3567 :ok :read [19 16] INFO [2022-03-13 17:16:34,320] jepsen worker 57 - jepsen.util 3567 :invoke :read [19 nil] INFO [2022-03-13 17:16:34,352] jepsen worker 3 - jepsen.util 1983 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,353] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 35] INFO [2022-03-13 17:16:34,353] jepsen worker 84 - jepsen.util 3864 :ok :read [28 44] INFO [2022-03-13 17:16:34,354] jepsen worker 84 - jepsen.util 3864 :invoke :write [28 15] INFO [2022-03-13 17:16:34,374] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,375] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:16:34,378] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,378] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,378] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:34,378] jepsen worker 26 - jepsen.util 476 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,378] jepsen worker 87 - jepsen.util 2607 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,380] jepsen worker 43 - jepsen.util 583 :invoke :write [14 33] INFO [2022-03-13 17:16:34,381] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 45] INFO [2022-03-13 17:16:34,381] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:34,383] jepsen worker 26 - jepsen.util 476 :invoke :write [8 5] INFO [2022-03-13 17:16:34,383] jepsen worker 66 - jepsen.util 3306 :ok :read [22 30] INFO [2022-03-13 17:16:34,383] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 34] INFO [2022-03-13 17:16:34,384] jepsen worker 66 - jepsen.util 3306 :invoke :read [22 nil] INFO [2022-03-13 17:16:34,395] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,395] jepsen worker 37 - jepsen.util 2107 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,396] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:16:34,398] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 10] INFO [2022-03-13 17:16:34,409] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:34,410] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:34,435] jepsen worker 39 - jepsen.util 3729 :ok :read [13 39] INFO [2022-03-13 17:16:34,436] jepsen worker 39 - jepsen.util 3729 :invoke :read [13 nil] INFO [2022-03-13 17:16:34,458] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,459] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:16:34,461] jepsen worker 57 - jepsen.util 3567 :ok :read [19 16] INFO [2022-03-13 17:16:34,462] jepsen worker 57 - jepsen.util 3567 :invoke :read [19 nil] INFO [2022-03-13 17:16:34,462] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,463] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,465] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,466] jepsen worker 59 - jepsen.util 3569 :info :write [19 37] :RequestTimeoutException INFO [2022-03-13 17:16:34,466] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,466] jepsen worker 59 - jepsen.util 3659 :invoke :read [19 nil] INFO [2022-03-13 17:16:34,467] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:34,467] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 31] INFO [2022-03-13 17:16:34,481] jepsen worker 66 - jepsen.util 3306 :ok :read [22 30] INFO [2022-03-13 17:16:34,482] jepsen worker 66 - jepsen.util 3306 :invoke :read [22 nil] INFO [2022-03-13 17:16:34,492] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:34,493] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:34,523] jepsen worker 39 - jepsen.util 3729 :ok :read [13 39] INFO [2022-03-13 17:16:34,524] jepsen worker 39 - jepsen.util 3729 :invoke :write [13 48] INFO [2022-03-13 17:16:34,539] jepsen worker 3 - jepsen.util 1983 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,540] jepsen worker 34 - jepsen.util 934 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,540] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,540] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,541] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,541] jepsen worker 80 - jepsen.util 800 :invoke :write [26 23] INFO [2022-03-13 17:16:34,542] jepsen worker 34 - jepsen.util 934 :invoke :write [11 11] INFO [2022-03-13 17:16:34,545] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:34,547] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,547] jepsen worker 78 - jepsen.util 618 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,547] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,547] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,548] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:16:34,549] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,550] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,551] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,552] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,554] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,554] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,554] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,555] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,555] jepsen worker 52 - jepsen.util 682 :invoke :write [17 28] INFO [2022-03-13 17:16:34,555] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:34,556] jepsen worker 78 - jepsen.util 618 :invoke :write [26 39] INFO [2022-03-13 17:16:34,558] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,559] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 30] INFO [2022-03-13 17:16:34,581] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,582] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 44] INFO [2022-03-13 17:16:34,584] jepsen worker 38 - jepsen.util 2468 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,584] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:34,589] jepsen worker 4 - jepsen.util 2254 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,590] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 36] INFO [2022-03-13 17:16:34,611] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,612] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:16:34,613] jepsen worker 57 - jepsen.util 3567 :ok :read [19 16] INFO [2022-03-13 17:16:34,614] jepsen worker 57 - jepsen.util 3567 :invoke :read [19 nil] INFO [2022-03-13 17:16:34,624] jepsen worker 66 - jepsen.util 3306 :ok :read [22 30] INFO [2022-03-13 17:16:34,625] jepsen worker 66 - jepsen.util 3306 :invoke :write [22 2] INFO [2022-03-13 17:16:34,637] jepsen worker 5 - jepsen.util 2435 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,638] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:16:34,649] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,650] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:34,654] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:34,654] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:34,662] jepsen worker 87 - jepsen.util 2607 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,663] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 1] INFO [2022-03-13 17:16:34,697] jepsen worker 57 - jepsen.util 3567 :ok :read [19 16] INFO [2022-03-13 17:16:34,698] jepsen worker 57 - jepsen.util 3567 :invoke :read [19 nil] INFO [2022-03-13 17:16:34,712] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,713] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:34,731] jepsen worker 3 - jepsen.util 1983 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,732] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,734] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,735] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,737] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,738] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,739] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,739] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:34,740] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,741] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,741] jepsen worker 37 - jepsen.util 2107 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,741] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,742] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:34,742] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 40] INFO [2022-03-13 17:16:34,742] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 35] INFO [2022-03-13 17:16:34,743] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 8] INFO [2022-03-13 17:16:34,746] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,746] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 17] INFO [2022-03-13 17:16:34,750] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,751] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:34,753] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,753] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:34,762] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,763] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:16:34,766] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,766] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:34,767] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,768] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:34,768] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,769] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:34,776] jepsen worker 70 - jepsen.util 790 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,776] jepsen worker 70 - jepsen.util 790 :invoke :write [23 44] INFO [2022-03-13 17:16:34,779] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,780] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:34,805] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,805] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 43] INFO [2022-03-13 17:16:34,808] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:34,809] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:34,826] jepsen worker 36 - jepsen.util 2196 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,826] jepsen worker 65 - jepsen.util 4025 :info :write [21 21] :RequestTimeoutException INFO [2022-03-13 17:16:34,826] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 16] INFO [2022-03-13 17:16:34,827] jepsen worker 65 - jepsen.util 4115 :invoke :read [21 nil] INFO [2022-03-13 17:16:34,867] jepsen worker 4 - jepsen.util 2254 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,868] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:16:34,875] jepsen worker 57 - jepsen.util 3567 :ok :read [19 16] INFO [2022-03-13 17:16:34,876] jepsen worker 57 - jepsen.util 3567 :invoke :write [19 42] INFO [2022-03-13 17:16:34,884] jepsen worker 3 - jepsen.util 1983 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,884] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:16:34,890] jepsen worker 45 - jepsen.util 3735 :info :write [15 16] :RequestTimeoutException INFO [2022-03-13 17:16:34,891] jepsen worker 45 - jepsen.util 3825 :invoke :read [15 nil] INFO [2022-03-13 17:16:34,913] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,913] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,914] jepsen worker 38 - jepsen.util 2468 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,914] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,914] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,914] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 16] INFO [2022-03-13 17:16:34,915] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:34,915] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:34,916] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,916] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:34,916] jepsen worker 24 - jepsen.util 564 :invoke :write [8 47] INFO [2022-03-13 17:16:34,917] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:34,921] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,921] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,921] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:16:34,922] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:34,923] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,924] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:16:34,949] jepsen worker 87 - jepsen.util 2607 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,949] jepsen worker 5 - jepsen.util 2435 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,950] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:34,951] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:16:34,955] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:34,956] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:34,968] jepsen worker 89 - jepsen.util 2519 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,969] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:34,969] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:34,969] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,027] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,028] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 12] INFO [2022-03-13 17:16:35,054] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,054] jepsen worker 36 - jepsen.util 2196 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,055] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:16:35,058] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:35,072] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,072] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,072] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,072] jepsen worker 8 - jepsen.util 728 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,072] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,074] jepsen worker 53 - jepsen.util 503 :invoke :write [17 47] INFO [2022-03-13 17:16:35,074] jepsen worker 88 - jepsen.util 2068 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,075] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:35,076] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 32] INFO [2022-03-13 17:16:35,076] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:35,077] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:35,082] jepsen worker 8 - jepsen.util 728 :invoke :write [2 38] INFO [2022-03-13 17:16:35,083] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,083] jepsen worker 33 - jepsen.util 753 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,083] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,084] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,084] jepsen worker 33 - jepsen.util 753 :invoke :write [11 30] INFO [2022-03-13 17:16:35,085] jepsen worker 51 - jepsen.util 771 :invoke :write [17 35] INFO [2022-03-13 17:16:35,085] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:35,086] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:16:35,104] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,105] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,107] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,110] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,113] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,113] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 1] INFO [2022-03-13 17:16:35,133] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,133] jepsen worker 37 - jepsen.util 2107 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,134] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,134] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 12] INFO [2022-03-13 17:16:35,135] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,136] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 7] INFO [2022-03-13 17:16:35,137] jepsen worker 60 - jepsen.util 600 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,137] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,138] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:35,138] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:35,146] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,147] jepsen worker 76 - jepsen.util 706 :invoke :write [25 32] INFO [2022-03-13 17:16:35,150] jepsen worker 24 - jepsen.util 564 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,150] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,150] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:35,151] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:16:35,163] jepsen worker 79 - jepsen.util 1069 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,165] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:35,196] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:35,196] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:35,207] jepsen worker 3 - jepsen.util 1983 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,208] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 10] INFO [2022-03-13 17:16:35,211] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,212] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,217] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:35,218] jepsen worker nemesis - couchbase.util Rebalance check count: 503 INFO [2022-03-13 17:16:35,218] jepsen worker nemesis - couchbase.util Rebalance stuck count: 403 INFO [2022-03-13 17:16:35,219] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,220] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 19] INFO [2022-03-13 17:16:35,268] jepsen worker 5 - jepsen.util 2435 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,269] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:16:35,276] jepsen worker 34 - jepsen.util 934 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,277] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:35,305] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,305] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:35,306] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,306] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:35,307] jepsen worker 69 - jepsen.util 699 :invoke :write [23 43] INFO [2022-03-13 17:16:35,307] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:35,325] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,326] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 22] INFO [2022-03-13 17:16:35,352] jepsen worker 3 - jepsen.util 1983 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,353] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 11] INFO [2022-03-13 17:16:35,372] jepsen worker 89 - jepsen.util 2519 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,372] jepsen worker 37 - jepsen.util 2107 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,373] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:35,373] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,405] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:35,406] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:35,432] jepsen worker 88 - jepsen.util 2068 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,432] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,432] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 40] INFO [2022-03-13 17:16:35,433] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,451] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,452] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,453] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,454] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,458] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,459] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,460] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,461] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:35,472] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,472] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,472] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,472] jepsen worker 80 - jepsen.util 800 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,473] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:35,473] jepsen worker 14 - jepsen.util 374 :invoke :write [4 4] INFO [2022-03-13 17:16:35,474] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:35,475] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:35,484] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:35,485] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:35,488] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,489] jepsen worker 71 - jepsen.util 521 :invoke :write [23 24] INFO [2022-03-13 17:16:35,521] jepsen worker 33 - jepsen.util 753 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,521] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:35,534] jepsen worker 87 - jepsen.util 2607 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,534] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,535] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:35,536] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:16:35,546] jepsen worker 3 - jepsen.util 1983 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,546] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:16:35,558] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,558] jepsen worker 36 - jepsen.util 2196 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,558] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 29] INFO [2022-03-13 17:16:35,559] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,560] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,560] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,560] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,561] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 27] INFO [2022-03-13 17:16:35,568] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,568] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,569] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,569] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 36] INFO [2022-03-13 17:16:35,570] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,570] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,572] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,573] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 25] INFO [2022-03-13 17:16:35,630] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:35,631] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:35,652] jepsen worker 26 - jepsen.util 476 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,653] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:35,660] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:35,660] jepsen worker 86 - jepsen.util 3416 :invoke :read [28 nil] INFO [2022-03-13 17:16:35,663] jepsen worker 38 - jepsen.util 2468 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,663] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,666] jepsen worker 88 - jepsen.util 2068 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,667] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,667] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:35,668] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:16:35,670] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,671] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 4] INFO [2022-03-13 17:16:35,705] jepsen worker 89 - jepsen.util 2519 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,706] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 4] INFO [2022-03-13 17:16:35,714] jepsen worker 51 - jepsen.util 771 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,714] jepsen worker 51 - jepsen.util 771 :invoke :write [17 39] INFO [2022-03-13 17:16:35,729] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,730] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,732] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,732] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:16:35,749] jepsen worker 86 - jepsen.util 3416 :ok :read [28 44] INFO [2022-03-13 17:16:35,750] jepsen worker 86 - jepsen.util 3416 :invoke :write [28 9] INFO [2022-03-13 17:16:35,763] jepsen worker 5 - jepsen.util 2435 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,764] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 39] INFO [2022-03-13 17:16:35,810] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,811] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,812] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,812] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:16:35,822] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,822] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,823] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:35,824] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:35,844] jepsen worker 37 - jepsen.util 2107 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,845] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,845] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,846] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 34] INFO [2022-03-13 17:16:35,848] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,848] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 44] INFO [2022-03-13 17:16:35,854] jepsen worker 82 - jepsen.util 3592 :info :write [27 6] :RequestTimeoutException INFO [2022-03-13 17:16:35,854] jepsen worker 82 - jepsen.util 3682 :invoke :write [27 29] INFO [2022-03-13 17:16:35,861] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,861] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,862] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:35,863] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:35,897] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,898] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:35,898] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,898] jepsen worker 5 - jepsen.util 2435 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,899] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,899] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,899] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,899] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 45] INFO [2022-03-13 17:16:35,900] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:16:35,901] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:35,901] jepsen worker 34 - jepsen.util 934 :invoke :write [11 45] INFO [2022-03-13 17:16:35,902] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:35,916] jepsen worker 23 - jepsen.util 563 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,917] jepsen worker 23 - jepsen.util 563 :invoke :write [7 36] INFO [2022-03-13 17:16:35,939] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,939] jepsen worker 71 - jepsen.util 521 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,940] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:35,940] jepsen worker 71 - jepsen.util 521 :invoke :write [23 22] INFO [2022-03-13 17:16:35,939] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,944] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:35,958] jepsen worker 67 - jepsen.util 2947 :ok :read [22 30] INFO [2022-03-13 17:16:35,959] jepsen worker 67 - jepsen.util 2947 :invoke :write [22 13] INFO [2022-03-13 17:16:35,967] jepsen worker 36 - jepsen.util 2196 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:35,967] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:35,976] jepsen worker 46 - jepsen.util 3646 :info :write [15 38] :RequestTimeoutException INFO [2022-03-13 17:16:35,977] jepsen worker 46 - jepsen.util 3736 :invoke :read [15 nil] INFO [2022-03-13 17:16:36,005] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,005] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,021] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,021] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,021] jepsen worker 0 - jepsen.util 3510 :ok :read [0 41] INFO [2022-03-13 17:16:36,022] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,023] jepsen worker 0 - jepsen.util 3510 :invoke :write [0 43] INFO [2022-03-13 17:16:36,023] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:16:36,043] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,043] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,044] jepsen worker 24 - jepsen.util 564 :invoke :write [8 8] INFO [2022-03-13 17:16:36,047] jepsen worker 75 - jepsen.util 435 :invoke :write [25 13] INFO [2022-03-13 17:16:36,060] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,060] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,060] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,061] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 39] INFO [2022-03-13 17:16:36,066] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:36,067] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 2] INFO [2022-03-13 17:16:36,069] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,070] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 40] INFO [2022-03-13 17:16:36,112] jepsen worker 5 - jepsen.util 2435 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,113] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,117] jepsen worker 14 - jepsen.util 374 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,117] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,117] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:16:36,118] jepsen worker 33 - jepsen.util 753 :invoke :write [11 24] INFO [2022-03-13 17:16:36,120] jepsen worker 43 - jepsen.util 583 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,121] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:36,127] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,128] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,133] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,134] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,138] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,138] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,141] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,142] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,148] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,148] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 28] INFO [2022-03-13 17:16:36,175] jepsen worker 89 - jepsen.util 2519 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,175] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,176] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:36,176] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 29] INFO [2022-03-13 17:16:36,208] jepsen worker 4 - jepsen.util 2254 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,209] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 22] INFO [2022-03-13 17:16:36,213] jepsen worker 71 - jepsen.util 521 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,214] jepsen worker 71 - jepsen.util 521 :invoke :write [23 26] INFO [2022-03-13 17:16:36,224] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:36,224] jepsen worker nemesis - couchbase.util Rebalance check count: 504 INFO [2022-03-13 17:16:36,224] jepsen worker nemesis - couchbase.util Rebalance stuck count: 404 INFO [2022-03-13 17:16:36,245] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,246] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,247] jepsen worker 38 - jepsen.util 2468 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,247] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 21] INFO [2022-03-13 17:16:36,250] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,250] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,251] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 7] INFO [2022-03-13 17:16:36,252] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,256] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,257] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,262] jepsen worker 61 - jepsen.util 691 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,262] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,263] jepsen worker 61 - jepsen.util 691 :invoke :write [20 22] INFO [2022-03-13 17:16:36,263] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,265] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,265] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,273] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,274] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,293] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,294] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,322] jepsen worker 88 - jepsen.util 2068 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,323] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:16:36,345] jepsen worker 37 - jepsen.util 2107 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,346] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:16:36,351] jepsen worker 5 - jepsen.util 2435 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,351] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,351] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,352] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,363] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,363] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,364] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 5] INFO [2022-03-13 17:16:36,364] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 37] INFO [2022-03-13 17:16:36,370] jepsen worker 30 - jepsen.util 3450 :info :write [10 23] :RequestTimeoutException INFO [2022-03-13 17:16:36,371] jepsen worker 30 - jepsen.util 3540 :invoke :read [10 nil] INFO [2022-03-13 17:16:36,392] jepsen worker 51 - jepsen.util 771 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,392] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:36,395] jepsen worker 36 - jepsen.util 2196 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,395] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,395] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,395] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,396] jepsen worker 87 - jepsen.util 2607 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,396] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:36,397] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:16:36,398] jepsen worker 80 - jepsen.util 800 :invoke :write [26 16] INFO [2022-03-13 17:16:36,398] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:36,399] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 15] INFO [2022-03-13 17:16:36,403] jepsen worker 24 - jepsen.util 564 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,404] jepsen worker 24 - jepsen.util 564 :invoke :write [8 9] INFO [2022-03-13 17:16:36,431] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,431] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,431] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,431] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,432] jepsen worker 6 - jepsen.util 636 :invoke :write [2 7] INFO [2022-03-13 17:16:36,433] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:36,434] jepsen worker 7 - jepsen.util 637 :invoke :write [2 49] INFO [2022-03-13 17:16:36,434] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:36,464] jepsen worker 4 - jepsen.util 2254 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,465] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 35] INFO [2022-03-13 17:16:36,476] jepsen worker 13 - jepsen.util 733 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,476] jepsen worker 13 - jepsen.util 733 :invoke :write [4 16] INFO [2022-03-13 17:16:36,502] jepsen worker 3 - jepsen.util 1983 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,503] jepsen worker 38 - jepsen.util 2468 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,503] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,505] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:36,510] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,511] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,546] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,547] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:16:36,553] jepsen worker 75 - jepsen.util 435 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,554] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:36,557] jepsen worker 78 - jepsen.util 618 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,558] jepsen worker 78 - jepsen.util 618 :invoke :write [26 14] INFO [2022-03-13 17:16:36,570] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,570] jepsen worker 89 - jepsen.util 2519 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,572] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,572] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:36,572] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 38] INFO [2022-03-13 17:16:36,573] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 9] INFO [2022-03-13 17:16:36,586] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,587] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 26] INFO [2022-03-13 17:16:36,594] jepsen worker 5 - jepsen.util 2435 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,595] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,595] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 49] INFO [2022-03-13 17:16:36,596] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,597] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,599] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,600] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,601] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:16:36,651] jepsen worker 87 - jepsen.util 2607 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,652] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:36,703] jepsen worker 53 - jepsen.util 503 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,703] jepsen worker 53 - jepsen.util 503 :invoke :write [17 19] INFO [2022-03-13 17:16:36,705] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,705] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,705] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:36,706] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 43] INFO [2022-03-13 17:16:36,706] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,707] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 4] WARN [2022-03-13 17:16:36,712] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17293761,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":17293761,"total_server_duration_us":6927,"operation_id":"0xe29e","timeout_ms":30000,"last_server_duration_us":6927,"total_duration_us":17296325},{"operation_name":"get","last_dispatch_duration_us":16499805,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":16499805,"total_server_duration_us":56,"operation_id":"0xe31e","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":16502151},{"operation_name":"get","last_dispatch_duration_us":13775961,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":13775961,"total_server_duration_us":49,"operation_id":"0xe4c5","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":13778216},{"operation_name":"upsert","last_dispatch_duration_us":40338,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1064045,"total_server_duration_us":628224,"encode_duration_us":18833,"operation_id":"0xe756","timeout_ms":10000,"last_server_duration_us":26596,"total_duration_us":10010043},{"operation_name":"upsert","last_dispatch_duration_us":33456,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":730110,"total_server_duration_us":435651,"encode_duration_us":19852,"operation_id":"0xe821","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10008376},{"operation_name":"upsert","last_dispatch_duration_us":12147245,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12147245,"total_server_duration_us":21895,"encode_duration_us":18065,"operation_id":"0xe5af","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10002862},{"operation_name":"get","last_dispatch_duration_us":8924530,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8924530,"total_server_duration_us":25800,"operation_id":"0xe797","timeout_ms":30000,"last_server_duration_us":25800,"total_duration_us":8926805},{"operation_name":"get","last_dispatch_duration_us":5670747,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":5670747,"total_server_duration_us":23,"operation_id":"0xe950","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":5672893},{"operation_name":"get","last_dispatch_duration_us":2605029,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2605029,"total_server_duration_us":7962,"operation_id":"0xeb0e","timeout_ms":30000,"last_server_duration_us":7962,"total_duration_us":2612200},{"operation_name":"upsert","last_dispatch_duration_us":1453517,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1453517,"total_server_duration_us":287877,"encode_duration_us":21688,"operation_id":"0xec12","timeout_ms":10000,"last_server_duration_us":287877,"total_duration_us":1462150}],"total_count":88}} WARN [2022-03-13 17:16:36,712] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19833827,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19833827,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":8842976,"encode_duration_us":20409,"operation_id":"0xe140","timeout_ms":10000,"last_server_duration_us":8842976,"total_duration_us":10007924}],"total_count":1}} INFO [2022-03-13 17:16:36,728] jepsen worker 5 - jepsen.util 2435 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,730] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:16:36,737] jepsen worker 4 - jepsen.util 2254 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,738] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,785] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,786] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:16:36,799] jepsen worker 34 - jepsen.util 934 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,800] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:36,803] jepsen worker 89 - jepsen.util 2519 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,804] jepsen worker 71 - jepsen.util 521 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,804] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:36,805] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:16:36,810] jepsen worker 50 - jepsen.util 3380 :info :write [16 11] :RequestTimeoutException INFO [2022-03-13 17:16:36,812] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] INFO [2022-03-13 17:16:36,838] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,838] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,838] jepsen worker 88 - jepsen.util 2068 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,839] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 27] INFO [2022-03-13 17:16:36,839] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,840] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:36,842] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,843] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,843] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:36,843] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:36,844] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,845] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 11] INFO [2022-03-13 17:16:36,845] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,846] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:16:36,854] jepsen worker 70 - jepsen.util 790 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,854] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,855] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,855] jepsen worker 12 - jepsen.util 732 :invoke :write [4 41] INFO [2022-03-13 17:16:36,855] jepsen worker 60 - jepsen.util 600 :invoke :write [20 32] INFO [2022-03-13 17:16:36,860] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:36,873] jepsen worker 36 - jepsen.util 2196 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,873] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,874] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:36,875] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 7] INFO [2022-03-13 17:16:36,875] jepsen worker 33 - jepsen.util 753 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,876] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:36,896] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,896] jepsen worker 24 - jepsen.util 564 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,897] jepsen worker 24 - jepsen.util 564 :invoke :write [8 17] INFO [2022-03-13 17:16:36,898] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,899] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 18] INFO [2022-03-13 17:16:36,899] jepsen worker 75 - jepsen.util 435 :invoke :write [25 6] INFO [2022-03-13 17:16:36,922] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,922] jepsen worker 38 - jepsen.util 2468 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,922] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:36,923] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:36,936] jepsen worker 8 - jepsen.util 728 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,936] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,937] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:36,937] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:36,937] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,937] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,937] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:36,938] jepsen worker 62 - jepsen.util 512 :invoke :write [20 49] INFO [2022-03-13 17:16:36,938] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:36,939] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:36,939] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:36,940] jepsen worker 8 - jepsen.util 728 :invoke :write [2 41] INFO [2022-03-13 17:16:36,943] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,943] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,943] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,944] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:36,944] jepsen worker 51 - jepsen.util 771 :invoke :write [17 12] INFO [2022-03-13 17:16:36,944] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:16:36,965] jepsen worker 87 - jepsen.util 2607 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,966] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:36,983] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:36,985] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 4] INFO [2022-03-13 17:16:37,021] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,021] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:37,028] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,028] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:37,073] jepsen worker 4 - jepsen.util 2254 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,073] jepsen worker 76 - jepsen.util 706 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,074] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 18] INFO [2022-03-13 17:16:37,074] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,074] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:37,074] jepsen worker 76 - jepsen.util 706 :invoke :write [25 45] INFO [2022-03-13 17:16:37,075] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:37,075] jepsen worker 77 - jepsen.util 707 :invoke :write [25 32] INFO [2022-03-13 17:16:37,093] jepsen worker 37 - jepsen.util 2107 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,094] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,095] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,096] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:16:37,107] jepsen worker 5 - jepsen.util 2435 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,108] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 7] INFO [2022-03-13 17:16:37,128] jepsen worker 18 - jepsen.util 3528 :info :write [6 0] :RequestTimeoutException INFO [2022-03-13 17:16:37,128] jepsen worker 18 - jepsen.util 3618 :invoke :write [6 22] INFO [2022-03-13 17:16:37,130] jepsen worker 6 - jepsen.util 636 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,130] jepsen worker 24 - jepsen.util 564 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,130] jepsen worker 6 - jepsen.util 636 :invoke :write [2 18] INFO [2022-03-13 17:16:37,131] jepsen worker 24 - jepsen.util 564 :invoke :write [8 33] INFO [2022-03-13 17:16:37,147] jepsen worker 80 - jepsen.util 800 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,148] jepsen worker 80 - jepsen.util 800 :invoke :write [26 23] INFO [2022-03-13 17:16:37,156] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,156] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 4] INFO [2022-03-13 17:16:37,156] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,157] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:16:37,166] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,166] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,166] jepsen worker 3 - jepsen.util 1983 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,166] jepsen worker 89 - jepsen.util 2519 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,167] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,168] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,168] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 3] INFO [2022-03-13 17:16:37,168] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,168] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,169] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 10] INFO [2022-03-13 17:16:37,169] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,170] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,180] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,180] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,181] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,182] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 46] INFO [2022-03-13 17:16:37,202] jepsen worker 5 - jepsen.util 2435 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,202] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:16:37,231] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:37,231] jepsen worker nemesis - couchbase.util Rebalance check count: 505 INFO [2022-03-13 17:16:37,231] jepsen worker nemesis - couchbase.util Rebalance stuck count: 405 INFO [2022-03-13 17:16:37,232] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,232] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,232] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,232] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,233] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:37,233] jepsen worker 34 - jepsen.util 934 :invoke :write [11 24] INFO [2022-03-13 17:16:37,234] jepsen worker 33 - jepsen.util 753 :invoke :write [11 18] INFO [2022-03-13 17:16:37,234] jepsen worker 14 - jepsen.util 374 :invoke :write [4 6] INFO [2022-03-13 17:16:37,249] jepsen worker 4 - jepsen.util 2254 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,249] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 41] INFO [2022-03-13 17:16:37,264] jepsen worker 51 - jepsen.util 771 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,264] jepsen worker 51 - jepsen.util 771 :invoke :write [17 24] INFO [2022-03-13 17:16:37,278] jepsen worker 3 - jepsen.util 1983 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,279] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,283] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,283] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 9] INFO [2022-03-13 17:16:37,288] jepsen worker 38 - jepsen.util 2468 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,288] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,289] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,289] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:16:37,303] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,304] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,323] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,323] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,348] jepsen worker 75 - jepsen.util 435 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,348] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,348] jepsen worker 71 - jepsen.util 521 :invoke :write [23 3] INFO [2022-03-13 17:16:37,349] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,350] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,350] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:37,352] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,352] jepsen worker 69 - jepsen.util 699 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,352] jepsen worker 69 - jepsen.util 699 :invoke :write [23 38] INFO [2022-03-13 17:16:37,353] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,359] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,360] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,361] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,362] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 13] INFO [2022-03-13 17:16:37,367] jepsen worker 52 - jepsen.util 682 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,368] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:37,373] jepsen worker 89 - jepsen.util 2519 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,373] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,373] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,374] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 15] INFO [2022-03-13 17:16:37,402] jepsen worker 24 - jepsen.util 564 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,403] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,403] jepsen worker 88 - jepsen.util 2068 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,403] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:16:37,404] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:16:37,405] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,443] jepsen worker 3 - jepsen.util 1983 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,444] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:16:37,465] jepsen worker 36 - jepsen.util 2196 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,465] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,465] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:16:37,466] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 12] INFO [2022-03-13 17:16:37,499] jepsen worker 7 - jepsen.util 637 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,500] jepsen worker 7 - jepsen.util 637 :invoke :write [2 34] INFO [2022-03-13 17:16:37,503] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,503] jepsen worker 4 - jepsen.util 2254 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,503] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,504] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,504] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,505] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 29] INFO [2022-03-13 17:16:37,506] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,506] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:37,507] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 29] INFO [2022-03-13 17:16:37,508] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 48] INFO [2022-03-13 17:16:37,511] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:37,515] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,544] jepsen worker 71 - jepsen.util 521 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,545] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:37,558] jepsen worker 5 - jepsen.util 2435 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,559] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,570] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,570] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,574] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,574] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,579] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,580] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,581] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,582] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 2] INFO [2022-03-13 17:16:37,584] jepsen worker 37 - jepsen.util 2107 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,584] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,585] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 13] INFO [2022-03-13 17:16:37,586] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,616] jepsen worker 4 - jepsen.util 2254 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,616] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,639] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,639] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:16:37,646] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,646] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,647] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,647] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 12] INFO [2022-03-13 17:16:37,648] jepsen worker 23 - jepsen.util 563 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,649] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:37,650] jepsen worker 80 - jepsen.util 800 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,650] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,651] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:37,651] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:37,701] jepsen worker 88 - jepsen.util 2068 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,702] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,708] jepsen worker 33 - jepsen.util 753 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,708] jepsen worker 33 - jepsen.util 753 :invoke :write [11 8] INFO [2022-03-13 17:16:37,721] jepsen worker 89 - jepsen.util 2519 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,721] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,725] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,726] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:37,737] jepsen worker 5 - jepsen.util 2435 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,738] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,740] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,741] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:16:37,781] jepsen worker 38 - jepsen.util 2468 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,781] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,782] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 24] INFO [2022-03-13 17:16:37,782] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 6] INFO [2022-03-13 17:16:37,788] jepsen worker 53 - jepsen.util 503 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,788] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:37,788] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,788] jepsen worker 87 - jepsen.util 2607 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,788] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,788] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,788] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:37,789] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,790] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:37,791] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:37,792] jepsen worker 42 - jepsen.util 402 :invoke :write [14 11] INFO [2022-03-13 17:16:37,793] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 13] INFO [2022-03-13 17:16:37,793] jepsen worker 53 - jepsen.util 503 :invoke :write [17 23] INFO [2022-03-13 17:16:37,794] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 7] INFO [2022-03-13 17:16:37,794] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,795] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:37,795] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,802] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:16:37,802] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,803] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,805] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,806] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,808] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,815] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:37,826] jepsen worker 81 - jepsen.util 4041 :info :write [27 20] :RequestTimeoutException INFO [2022-03-13 17:16:37,827] jepsen worker 81 - jepsen.util 4131 :invoke :write [27 26] INFO [2022-03-13 17:16:37,848] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,848] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,849] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 8] INFO [2022-03-13 17:16:37,849] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 13] INFO [2022-03-13 17:16:37,872] jepsen worker 87 - jepsen.util 2607 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,872] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,880] jepsen worker 36 - jepsen.util 2196 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,880] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,881] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:37,881] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:37,888] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,889] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 25] INFO [2022-03-13 17:16:37,898] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:37,899] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 7] INFO [2022-03-13 17:16:37,964] jepsen worker 73 - jepsen.util 3673 :ok :read [24 24] INFO [2022-03-13 17:16:37,965] jepsen worker 73 - jepsen.util 3673 :invoke :write [24 26] INFO [2022-03-13 17:16:38,002] jepsen worker 27 - jepsen.util 3897 :ok :read [9 26] INFO [2022-03-13 17:16:38,004] jepsen worker 27 - jepsen.util 3897 :invoke :read [9 nil] INFO [2022-03-13 17:16:38,078] jepsen worker 58 - jepsen.util 3208 :info :write [19 15] :RequestTimeoutException INFO [2022-03-13 17:16:38,078] jepsen worker 58 - jepsen.util 3298 :invoke :read [19 nil] INFO [2022-03-13 17:16:38,080] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,080] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] WARN [2022-03-13 17:16:38,092] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":23130243,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23130243,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9028877,"encode_duration_us":24492,"operation_id":"0xe18f","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10004839},{"operation_name":"upsert","last_dispatch_duration_us":24695010,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":24695010,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9042674,"encode_duration_us":17771,"operation_id":"0xdfe7","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10008480},{"operation_name":"upsert","last_dispatch_duration_us":18227468,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18227468,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9026756,"encode_duration_us":17826,"operation_id":"0xe3a7","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10009414},{"operation_name":"upsert","last_dispatch_duration_us":17291202,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17665359,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9245701,"encode_duration_us":21834,"operation_id":"0xdf36","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10010876}],"total_count":4}} INFO [2022-03-13 17:16:38,147] jepsen worker 49 - jepsen.util 3469 :info :write [16 14] :SyncWriteAmbiguous INFO [2022-03-13 17:16:38,149] jepsen worker 49 - jepsen.util 3559 :invoke :read [16 nil] INFO [2022-03-13 17:16:38,160] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,161] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,177] jepsen worker 58 - jepsen.util 3298 :ok :read [19 16] INFO [2022-03-13 17:16:38,178] jepsen worker 58 - jepsen.util 3298 :invoke :read [19 nil] INFO [2022-03-13 17:16:38,225] jepsen worker 40 - jepsen.util 3730 :ok :read [13 39] INFO [2022-03-13 17:16:38,226] jepsen worker 40 - jepsen.util 3730 :invoke :write [13 18] INFO [2022-03-13 17:16:38,237] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:38,237] jepsen worker nemesis - couchbase.util Rebalance check count: 506 INFO [2022-03-13 17:16:38,237] jepsen worker nemesis - couchbase.util Rebalance stuck count: 406 INFO [2022-03-13 17:16:38,259] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,260] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,273] jepsen worker 3 - jepsen.util 1983 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,273] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,275] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,276] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 37] INFO [2022-03-13 17:16:38,286] jepsen worker 58 - jepsen.util 3298 :ok :read [19 16] INFO [2022-03-13 17:16:38,286] jepsen worker 58 - jepsen.util 3298 :invoke :write [19 1] INFO [2022-03-13 17:16:38,288] jepsen worker 36 - jepsen.util 2196 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,289] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] WARN [2022-03-13 17:16:38,299] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16226432,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":16226432,"total_server_duration_us":93802,"operation_id":"0xe43f","timeout_ms":30000,"last_server_duration_us":93802,"total_duration_us":16229359},{"operation_name":"get","last_dispatch_duration_us":10446017,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10446017,"total_server_duration_us":70690,"operation_id":"0xe7b3","timeout_ms":30000,"last_server_duration_us":70690,"total_duration_us":10448395},{"operation_name":"upsert","last_dispatch_duration_us":308150,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9962781,"total_server_duration_us":141479,"encode_duration_us":21094,"operation_id":"0xe874","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10007837},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":18037,"operation_id":"0xe6b1","timeout_ms":10000,"total_duration_us":10003841},{"operation_name":"upsert","last_dispatch_duration_us":8291015,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8291015,"total_server_duration_us":32743,"encode_duration_us":18901,"operation_id":"0xe52a","timeout_ms":10000,"last_server_duration_us":32743,"total_duration_us":10003028},{"operation_name":"upsert","last_dispatch_duration_us":5635210,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5635210,"total_server_duration_us":83848,"encode_duration_us":21445,"operation_id":"0xe95c","timeout_ms":10000,"last_server_duration_us":83848,"total_duration_us":5638665},{"operation_name":"upsert","last_dispatch_duration_us":4790134,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4790134,"total_server_duration_us":115616,"encode_duration_us":17942,"operation_id":"0xe7fe","timeout_ms":10000,"last_server_duration_us":115616,"total_duration_us":4794333},{"operation_name":"get","last_dispatch_duration_us":3723302,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3723302,"total_server_duration_us":32,"operation_id":"0xea6a","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3723594},{"operation_name":"upsert","last_dispatch_duration_us":3026810,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3026810,"total_server_duration_us":30069,"encode_duration_us":28835,"operation_id":"0xe7dc","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":3032222},{"operation_name":"upsert","last_dispatch_duration_us":2784444,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2784444,"total_server_duration_us":97174,"encode_duration_us":20444,"operation_id":"0xec92","timeout_ms":10000,"last_server_duration_us":97174,"total_duration_us":2791460}],"total_count":14}} WARN [2022-03-13 17:16:38,301] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18021559,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18021559,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9024634,"encode_duration_us":21199,"operation_id":"0xdfce","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10003136},{"operation_name":"upsert","last_dispatch_duration_us":21835034,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21835034,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9029938,"encode_duration_us":18573,"operation_id":"0xdf4d","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10007662}],"total_count":2}} WARN [2022-03-13 17:16:38,320] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17019091,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":17019091,"total_server_duration_us":43,"operation_id":"0xe1aa","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":17020941},{"operation_name":"get","last_dispatch_duration_us":15928299,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":15928299,"total_server_duration_us":84,"operation_id":"0xe219","timeout_ms":30000,"last_server_duration_us":84,"total_duration_us":15930385},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":21582,"operation_id":"0xe901","timeout_ms":10000,"total_duration_us":10006517},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18314,"operation_id":"0xe701","timeout_ms":10000,"total_duration_us":10006121},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":18598,"operation_id":"0xe7f9","timeout_ms":10000,"total_duration_us":10005794},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":24039,"operation_id":"0xe593","timeout_ms":10000,"total_duration_us":10001711},{"operation_name":"get","last_dispatch_duration_us":1725349,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1725349,"total_server_duration_us":56,"operation_id":"0xe907","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":1725685},{"operation_name":"upsert","last_dispatch_duration_us":900465,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":900465,"total_server_duration_us":324596,"encode_duration_us":19922,"operation_id":"0xecd4","timeout_ms":10000,"last_server_duration_us":324596,"total_duration_us":916565},{"operation_name":"upsert","last_dispatch_duration_us":711086,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":711086,"total_server_duration_us":182910,"encode_duration_us":22290,"operation_id":"0xeac9","timeout_ms":10000,"last_server_duration_us":182910,"total_duration_us":716355},{"operation_name":"upsert","last_dispatch_duration_us":708185,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":708185,"total_server_duration_us":373240,"encode_duration_us":48142,"operation_id":"0xeb47","timeout_ms":10000,"last_server_duration_us":373240,"total_duration_us":711988}],"total_count":20}} WARN [2022-03-13 17:16:38,320] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":24480779,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24480779,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9047983,"encode_duration_us":17722,"operation_id":"0xdd5d","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10001171},{"operation_name":"upsert","last_dispatch_duration_us":22220903,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":22220903,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9039489,"encode_duration_us":16972,"operation_id":"0xdeb1","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10004345}],"total_count":2}} INFO [2022-03-13 17:16:38,324] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,324] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,325] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,325] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 39] INFO [2022-03-13 17:16:38,341] jepsen worker 5 - jepsen.util 2435 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,342] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,344] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,347] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 33] INFO [2022-03-13 17:16:38,389] jepsen worker 34 - jepsen.util 934 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,389] jepsen worker 34 - jepsen.util 934 :invoke :write [11 3] WARN [2022-03-13 17:16:38,393] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":21164313,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":21164313,"total_server_duration_us":6239,"operation_id":"0xe28d","timeout_ms":30000,"last_server_duration_us":6239,"total_duration_us":21166158},{"operation_name":"get","last_dispatch_duration_us":17114110,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17114110,"total_server_duration_us":56,"operation_id":"0xe4fb","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":17117111},{"operation_name":"upsert","last_dispatch_duration_us":8420204,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":17990428,"total_server_duration_us":177424,"encode_duration_us":18026,"operation_id":"0xe3c1","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10010360},{"operation_name":"upsert","last_dispatch_duration_us":18227468,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18227468,"total_server_duration_us":9026756,"encode_duration_us":17826,"operation_id":"0xe3a7","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10009414},{"operation_name":"upsert","last_dispatch_duration_us":8323519,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8409603,"total_server_duration_us":40487,"encode_duration_us":27478,"operation_id":"0xe5d0","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10008567},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":16825,"operation_id":"0xe895","timeout_ms":10000,"total_duration_us":10008488},{"operation_name":"upsert","last_dispatch_duration_us":8413823,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":18128773,"total_server_duration_us":166155,"encode_duration_us":16578,"operation_id":"0xe3a6","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10002824},{"operation_name":"upsert","last_dispatch_duration_us":65638,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9172755,"total_server_duration_us":241192,"encode_duration_us":55018,"operation_id":"0xe922","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10001163},{"operation_name":"upsert","last_dispatch_duration_us":2146074,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2146074,"total_server_duration_us":67089,"encode_duration_us":19363,"operation_id":"0xe554","timeout_ms":10000,"last_server_duration_us":67089,"total_duration_us":8852167},{"operation_name":"get","last_dispatch_duration_us":1117674,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1117674,"total_server_duration_us":62,"operation_id":"0xe6da","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":7299642}],"total_count":51}} INFO [2022-03-13 17:16:38,408] jepsen worker 38 - jepsen.util 2468 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,409] jepsen worker 4 - jepsen.util 2254 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,409] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:38,409] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,410] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,411] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:16:38,425] jepsen worker 78 - jepsen.util 618 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,425] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:38,446] jepsen worker 3 - jepsen.util 1983 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,446] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,454] jepsen worker 62 - jepsen.util 512 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,455] jepsen worker 62 - jepsen.util 512 :invoke :write [20 44] INFO [2022-03-13 17:16:38,456] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,457] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 42] INFO [2022-03-13 17:16:38,459] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,459] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,460] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:38,460] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:16:38,466] jepsen worker 89 - jepsen.util 2519 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,466] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,466] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,467] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,467] jepsen worker 24 - jepsen.util 564 :invoke :write [8 15] INFO [2022-03-13 17:16:38,468] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 31] INFO [2022-03-13 17:16:38,472] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,473] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:38,474] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,475] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 11] INFO [2022-03-13 17:16:38,480] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,480] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:38,482] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,482] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:38,484] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,484] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:16:38,495] jepsen worker 88 - jepsen.util 2068 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,495] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,496] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,497] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] WARN [2022-03-13 17:16:38,499] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":23412865,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":23412865,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9037366,"encode_duration_us":17277,"operation_id":"0xe011","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10007556},{"operation_name":"upsert","last_dispatch_duration_us":9087329,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9729525,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9406302,"encode_duration_us":19719,"operation_id":"0xe210","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10008999}],"total_count":2}} WARN [2022-03-13 17:16:38,501] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":190040,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9444443,"total_server_duration_us":108308,"encode_duration_us":19378,"operation_id":"0xe639","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10009371},{"operation_name":"upsert","last_dispatch_duration_us":164439,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":164439,"total_server_duration_us":32258,"encode_duration_us":17340,"operation_id":"0xe74d","timeout_ms":10000,"last_server_duration_us":32258,"total_duration_us":10009222},{"operation_name":"upsert","last_dispatch_duration_us":200144,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9435763,"total_server_duration_us":115491,"encode_duration_us":18749,"operation_id":"0xe634","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10007863},{"operation_name":"upsert","last_dispatch_duration_us":142321,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":142321,"total_server_duration_us":27583,"encode_duration_us":28760,"operation_id":"0xe685","timeout_ms":10000,"last_server_duration_us":27583,"total_duration_us":10003571},{"operation_name":"upsert","last_dispatch_duration_us":44572,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7392280,"total_server_duration_us":337326,"encode_duration_us":27756,"operation_id":"0xe5a4","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10003354},{"operation_name":"upsert","last_dispatch_duration_us":9164943,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9164943,"total_server_duration_us":9038428,"encode_duration_us":20158,"operation_id":"0xe63d","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":9170649},{"operation_name":"upsert","last_dispatch_duration_us":9081675,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9081675,"total_server_duration_us":9027816,"encode_duration_us":29932,"operation_id":"0xe58b","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":9096585},{"operation_name":"get","last_dispatch_duration_us":197336,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":197336,"total_server_duration_us":62,"operation_id":"0xe818","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":6388504},{"operation_name":"upsert","last_dispatch_duration_us":3583465,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3583465,"total_server_duration_us":187583,"encode_duration_us":21482,"operation_id":"0xe9fc","timeout_ms":10000,"last_server_duration_us":187583,"total_duration_us":3587619},{"operation_name":"upsert","last_dispatch_duration_us":2680565,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2680565,"total_server_duration_us":80826,"encode_duration_us":33003,"operation_id":"0xec2c","timeout_ms":10000,"last_server_duration_us":80826,"total_duration_us":2684256}],"total_count":11}} INFO [2022-03-13 17:16:38,502] jepsen worker 14 - jepsen.util 374 :fail :write [4 6] :ServerOutOfMemoryException WARN [2022-03-13 17:16:38,502] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":19682241,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":19682241,"total_server_duration_us":56,"operation_id":"0xe0a1","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":19684125},{"operation_name":"get","last_dispatch_duration_us":16939732,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":16939732,"total_server_duration_us":32,"operation_id":"0xe218","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":16942363},{"operation_name":"get","last_dispatch_duration_us":15184762,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":15184762,"total_server_duration_us":27,"operation_id":"0xe199","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":15185175},{"operation_name":"upsert","last_dispatch_duration_us":9155317,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9155317,"total_server_duration_us":22141,"encode_duration_us":21126,"operation_id":"0xe6d0","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10010528},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":18359,"operation_id":"0xe52c","timeout_ms":10000,"total_duration_us":10009970},{"operation_name":"upsert","last_dispatch_duration_us":7607488,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12124067,"total_server_duration_us":192402,"encode_duration_us":17108,"operation_id":"0xe7e8","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10007254},{"operation_name":"upsert","last_dispatch_duration_us":10057968,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10057968,"total_server_duration_us":31776,"encode_duration_us":20713,"operation_id":"0xe46b","timeout_ms":10000,"last_server_duration_us":31776,"total_duration_us":10004737},{"operation_name":"get","last_dispatch_duration_us":9148520,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9148520,"total_server_duration_us":32,"operation_id":"0xe6b7","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9327648},{"operation_name":"upsert","last_dispatch_duration_us":8096254,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8096254,"total_server_duration_us":8042024,"encode_duration_us":32985,"operation_id":"0xe9f3","timeout_ms":10000,"last_server_duration_us":8042024,"total_duration_us":8101605},{"operation_name":"get","last_dispatch_duration_us":7365036,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7365036,"total_server_duration_us":7645,"operation_id":"0xea6f","timeout_ms":30000,"last_server_duration_us":7645,"total_duration_us":7367099}],"total_count":77}} INFO [2022-03-13 17:16:38,503] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:16:38,503] jepsen worker 47 - jepsen.util 3557 :ok :read [15 48] INFO [2022-03-13 17:16:38,504] jepsen worker 47 - jepsen.util 3557 :invoke :write [15 35] INFO [2022-03-13 17:16:38,510] jepsen worker 61 - jepsen.util 691 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,510] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,511] jepsen worker 61 - jepsen.util 691 :invoke :write [20 27] INFO [2022-03-13 17:16:38,511] jepsen worker 70 - jepsen.util 790 :invoke :write [23 13] INFO [2022-03-13 17:16:38,512] jepsen worker 63 - jepsen.util 3663 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,513] jepsen worker 63 - jepsen.util 3663 :invoke :write [21 24] INFO [2022-03-13 17:16:38,541] jepsen worker 5 - jepsen.util 2435 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,542] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 41] INFO [2022-03-13 17:16:38,544] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,544] jepsen worker 37 - jepsen.util 2107 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,545] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 42] INFO [2022-03-13 17:16:38,546] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 23] INFO [2022-03-13 17:16:38,551] jepsen worker 46 - jepsen.util 3736 :ok :read [15 48] INFO [2022-03-13 17:16:38,552] jepsen worker 46 - jepsen.util 3736 :invoke :read [15 nil] INFO [2022-03-13 17:16:38,581] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,581] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,582] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 14] INFO [2022-03-13 17:16:38,582] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,591] jepsen worker 8 - jepsen.util 728 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,592] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:38,617] jepsen worker 13 - jepsen.util 733 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,618] jepsen worker 13 - jepsen.util 733 :invoke :write [4 13] INFO [2022-03-13 17:16:38,618] jepsen worker 3 - jepsen.util 1983 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,619] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,621] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,622] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,624] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,625] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,626] jepsen worker 51 - jepsen.util 771 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,626] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,626] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,626] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,627] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,628] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:16:38,629] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:38,630] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:38,631] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,631] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:38,632] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,633] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,634] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 39] INFO [2022-03-13 17:16:38,637] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:38,638] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,639] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:38,671] jepsen worker 36 - jepsen.util 2196 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,671] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:38,683] jepsen worker 46 - jepsen.util 3736 :ok :read [15 48] INFO [2022-03-13 17:16:38,684] jepsen worker 46 - jepsen.util 3736 :invoke :read [15 nil] INFO [2022-03-13 17:16:38,690] jepsen worker 5 - jepsen.util 2435 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,692] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,693] jepsen worker 33 - jepsen.util 753 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,693] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:38,697] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,697] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,699] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,699] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 20] INFO [2022-03-13 17:16:38,712] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,712] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,712] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:16:38,715] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 6] INFO [2022-03-13 17:16:38,717] jepsen worker 16 - jepsen.util 3346 :info :write [5 45] :RequestTimeoutException INFO [2022-03-13 17:16:38,718] jepsen worker 16 - jepsen.util 3436 :invoke :write [5 37] INFO [2022-03-13 17:16:38,721] jepsen worker 46 - jepsen.util 3736 :ok :read [15 48] INFO [2022-03-13 17:16:38,721] jepsen worker 46 - jepsen.util 3736 :invoke :read [15 nil] INFO [2022-03-13 17:16:38,729] jepsen worker 60 - jepsen.util 600 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,730] jepsen worker 60 - jepsen.util 600 :invoke :write [20 43] INFO [2022-03-13 17:16:38,749] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,751] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:38,759] jepsen worker 3 - jepsen.util 1983 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,760] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,761] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,763] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,767] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,768] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 14] INFO [2022-03-13 17:16:38,788] jepsen worker 46 - jepsen.util 3736 :ok :read [15 48] INFO [2022-03-13 17:16:38,789] jepsen worker 46 - jepsen.util 3736 :invoke :read [15 nil] INFO [2022-03-13 17:16:38,801] jepsen worker 34 - jepsen.util 934 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,802] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:38,811] jepsen worker 6 - jepsen.util 636 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,812] jepsen worker 6 - jepsen.util 636 :invoke :write [2 23] INFO [2022-03-13 17:16:38,813] jepsen worker 24 - jepsen.util 564 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,813] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,813] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,813] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:38,814] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:38,815] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:38,823] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,824] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,824] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,824] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:38,825] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:38,826] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:16:38,828] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,828] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,829] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:38,830] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:38,831] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,832] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,832] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:38,833] jepsen worker 24 - jepsen.util 564 :invoke :write [8 15] INFO [2022-03-13 17:16:38,833] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,835] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:38,836] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,837] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:38,839] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,840] jepsen worker 71 - jepsen.util 521 :invoke :write [23 34] INFO [2022-03-13 17:16:38,841] jepsen worker 4 - jepsen.util 2254 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,842] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:38,845] jepsen worker 89 - jepsen.util 2519 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,845] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,846] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,847] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 33] INFO [2022-03-13 17:16:38,855] jepsen worker 46 - jepsen.util 3736 :ok :read [15 48] INFO [2022-03-13 17:16:38,856] jepsen worker 46 - jepsen.util 3736 :invoke :read [15 nil] INFO [2022-03-13 17:16:38,862] jepsen worker 5 - jepsen.util 2435 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,863] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 35] INFO [2022-03-13 17:16:38,868] jepsen worker 76 - jepsen.util 706 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,869] jepsen worker 76 - jepsen.util 706 :invoke :write [25 35] INFO [2022-03-13 17:16:38,888] jepsen worker 15 - jepsen.util 3525 :info :write [5 25] :RequestTimeoutException INFO [2022-03-13 17:16:38,889] jepsen worker 15 - jepsen.util 3615 :invoke :read [5 nil] INFO [2022-03-13 17:16:38,893] jepsen worker 46 - jepsen.util 3736 :ok :read [15 48] INFO [2022-03-13 17:16:38,893] jepsen worker 46 - jepsen.util 3736 :invoke :read [15 nil] INFO [2022-03-13 17:16:38,912] jepsen worker 37 - jepsen.util 2107 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,912] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,912] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 13] INFO [2022-03-13 17:16:38,913] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,913] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,913] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,913] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,914] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,914] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:16:38,915] jepsen worker 51 - jepsen.util 771 :invoke :write [17 19] INFO [2022-03-13 17:16:38,915] jepsen worker 14 - jepsen.util 374 :invoke :write [4 0] INFO [2022-03-13 17:16:38,917] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:16:38,918] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,922] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:38,927] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,928] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:38,949] jepsen worker 46 - jepsen.util 3736 :ok :read [15 48] INFO [2022-03-13 17:16:38,950] jepsen worker 46 - jepsen.util 3736 :invoke :write [15 22] INFO [2022-03-13 17:16:38,951] jepsen worker 7 - jepsen.util 637 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,027] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:39,027] jepsen worker 87 - jepsen.util 2607 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,027] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,027] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,951] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,951] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,028] jepsen worker 26 - jepsen.util 476 :invoke :write [8 23] INFO [2022-03-13 17:16:38,951] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:39,028] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:39,028] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:38,957] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,958] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,028] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:16:39,028] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:38,970] jepsen worker 3 - jepsen.util 1983 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:38,971] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,028] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:39,006] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,006] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,028] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:16:39,008] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,028] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:39,008] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,029] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:39,029] jepsen worker 80 - jepsen.util 800 :invoke :write [26 39] INFO [2022-03-13 17:16:39,019] jepsen worker 38 - jepsen.util 2468 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,028] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:39,029] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 15] INFO [2022-03-13 17:16:39,030] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:39,030] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 7] INFO [2022-03-13 17:16:39,031] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 46] INFO [2022-03-13 17:16:39,035] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,047] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:16:39,073] jepsen worker 36 - jepsen.util 2196 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,074] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,074] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:39,075] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 19] INFO [2022-03-13 17:16:39,083] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,084] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,084] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 17] INFO [2022-03-13 17:16:39,084] jepsen worker 5 - jepsen.util 2435 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,085] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:39,086] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:39,089] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,089] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,091] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 6] INFO [2022-03-13 17:16:39,091] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:39,092] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,093] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:39,094] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,095] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 10] INFO [2022-03-13 17:16:39,126] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,127] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:39,128] jepsen worker 24 - jepsen.util 564 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,128] jepsen worker 24 - jepsen.util 564 :invoke :write [8 8] INFO [2022-03-13 17:16:39,137] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,137] jepsen worker 69 - jepsen.util 699 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,138] jepsen worker 69 - jepsen.util 699 :invoke :write [23 34] INFO [2022-03-13 17:16:39,139] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:16:39,207] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,208] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:16:39,214] jepsen worker 42 - jepsen.util 402 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,215] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:39,244] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:39,244] jepsen worker nemesis - couchbase.util Rebalance check count: 507 INFO [2022-03-13 17:16:39,244] jepsen worker nemesis - couchbase.util Rebalance stuck count: 407 INFO [2022-03-13 17:16:39,259] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,259] jepsen worker 71 - jepsen.util 521 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,260] jepsen worker 71 - jepsen.util 521 :invoke :write [23 0] INFO [2022-03-13 17:16:39,261] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:39,315] jepsen worker 12 - jepsen.util 732 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,315] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,316] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:39,317] jepsen worker 14 - jepsen.util 374 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,317] jepsen worker 12 - jepsen.util 732 :invoke :write [4 47] INFO [2022-03-13 17:16:39,318] jepsen worker 14 - jepsen.util 374 :invoke :write [4 40] INFO [2022-03-13 17:16:39,383] jepsen worker 53 - jepsen.util 503 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,383] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:39,389] jepsen worker 24 - jepsen.util 564 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,389] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,389] jepsen worker 24 - jepsen.util 564 :invoke :write [8 25] INFO [2022-03-13 17:16:39,390] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:39,454] jepsen worker 71 - jepsen.util 521 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,454] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,455] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:39,455] jepsen worker 71 - jepsen.util 521 :invoke :write [23 48] INFO [2022-03-13 17:16:39,472] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,472] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:16:39,490] jepsen worker 24 - jepsen.util 564 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,490] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,490] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:16:39,491] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:39,604] jepsen worker 71 - jepsen.util 521 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,604] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,604] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:39,605] jepsen worker 75 - jepsen.util 435 :invoke :write [25 48] INFO [2022-03-13 17:16:39,610] jepsen worker 13 - jepsen.util 733 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,611] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:39,628] jepsen worker 70 - jepsen.util 790 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,628] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:39,652] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,652] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:39,654] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,654] jepsen worker 71 - jepsen.util 521 :invoke :write [23 7] INFO [2022-03-13 17:16:39,697] jepsen worker 75 - jepsen.util 435 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,697] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,698] jepsen worker 75 - jepsen.util 435 :invoke :write [25 10] INFO [2022-03-13 17:16:39,698] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:16:39,758] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,758] jepsen worker 62 - jepsen.util 512 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,759] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:39,760] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:39,764] jepsen worker 51 - jepsen.util 771 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,765] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,765] jepsen worker 51 - jepsen.util 771 :invoke :write [17 29] INFO [2022-03-13 17:16:39,767] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,768] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:39,769] jepsen worker 33 - jepsen.util 753 :invoke :write [11 27] INFO [2022-03-13 17:16:39,778] jepsen worker 71 - jepsen.util 521 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,779] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:39,888] jepsen worker 75 - jepsen.util 435 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,888] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:16:39,932] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,932] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:39,932] jepsen worker 24 - jepsen.util 564 :invoke :write [8 22] INFO [2022-03-13 17:16:39,933] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:40,020] jepsen worker 83 - jepsen.util 3143 :info :write [27 13] :RequestTimeoutException INFO [2022-03-13 17:16:40,020] jepsen worker 83 - jepsen.util 3233 :invoke :write [27 1] INFO [2022-03-13 17:16:40,028] jepsen worker 80 - jepsen.util 800 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,029] jepsen worker 80 - jepsen.util 800 :invoke :write [26 18] INFO [2022-03-13 17:16:40,038] jepsen worker 61 - jepsen.util 691 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,039] jepsen worker 61 - jepsen.util 691 :invoke :write [20 40] INFO [2022-03-13 17:16:40,078] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,079] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:40,079] jepsen worker 6 - jepsen.util 636 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,078] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,080] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:40,080] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,080] jepsen worker 43 - jepsen.util 583 :invoke :write [14 3] INFO [2022-03-13 17:16:40,081] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 28] INFO [2022-03-13 17:16:40,081] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:40,082] jepsen worker 7 - jepsen.util 637 :invoke :write [2 9] INFO [2022-03-13 17:16:40,115] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,115] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,116] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:40,117] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:40,120] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,120] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,120] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:40,121] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:40,179] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,180] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:16:40,187] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,188] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,187] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,189] jepsen worker 13 - jepsen.util 733 :invoke :write [4 5] INFO [2022-03-13 17:16:40,189] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:16:40,193] jepsen worker 52 - jepsen.util 682 :invoke :write [17 48] INFO [2022-03-13 17:16:40,248] jepsen worker 24 - jepsen.util 564 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,248] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,248] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,249] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:40,249] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:40,249] jepsen worker 71 - jepsen.util 521 :invoke :write [23 23] INFO [2022-03-13 17:16:40,250] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,250] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:40,250] jepsen worker nemesis - couchbase.util Rebalance check count: 508 INFO [2022-03-13 17:16:40,251] jepsen worker nemesis - couchbase.util Rebalance stuck count: 408 INFO [2022-03-13 17:16:40,251] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:16:40,257] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,258] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:16:40,324] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,324] jepsen worker 60 - jepsen.util 600 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,325] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:40,326] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:40,377] jepsen worker 14 - jepsen.util 374 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,378] jepsen worker 14 - jepsen.util 374 :invoke :write [4 31] INFO [2022-03-13 17:16:40,491] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,492] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:40,514] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:40,514] jepsen worker 26 - jepsen.util 476 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,514] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,515] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,515] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,515] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,515] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:40,516] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:40,516] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:16:40,516] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:40,517] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:40,517] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:40,612] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,614] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:16:40,614] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,616] jepsen worker 34 - jepsen.util 934 :invoke :write [11 5] INFO [2022-03-13 17:16:40,632] jepsen worker 71 - jepsen.util 521 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,632] jepsen worker 71 - jepsen.util 521 :invoke :write [23 49] INFO [2022-03-13 17:16:40,696] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,697] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:40,697] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,697] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:16:40,817] jepsen worker 76 - jepsen.util 706 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,817] jepsen worker 76 - jepsen.util 706 :invoke :write [25 31] INFO [2022-03-13 17:16:40,824] jepsen worker 51 - jepsen.util 771 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,824] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:40,867] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,867] jepsen worker 71 - jepsen.util 521 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,867] jepsen worker 79 - jepsen.util 1069 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,868] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:16:40,871] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:40,872] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 39] INFO [2022-03-13 17:16:40,996] jepsen worker 33 - jepsen.util 753 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:40,997] jepsen worker 33 - jepsen.util 753 :invoke :write [11 0] INFO [2022-03-13 17:16:41,037] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,037] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,038] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:41,039] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:41,074] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,074] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,075] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:41,076] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:16:41,160] jepsen worker 69 - jepsen.util 699 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,161] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:41,184] jepsen worker 43 - jepsen.util 583 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,184] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:41,188] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,188] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,188] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,189] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:41,190] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:16:41,190] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:41,191] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,191] jepsen worker 75 - jepsen.util 435 :invoke :write [25 16] INFO [2022-03-13 17:16:41,208] jepsen worker 68 - jepsen.util 3668 :ok :read [22 30] INFO [2022-03-13 17:16:41,209] jepsen worker 68 - jepsen.util 3668 :invoke :write [22 11] INFO [2022-03-13 17:16:41,216] jepsen worker 64 - jepsen.util 3484 :info :write [21 45] :RequestTimeoutException INFO [2022-03-13 17:16:41,217] jepsen worker 64 - jepsen.util 3574 :invoke :write [21 29] INFO [2022-03-13 17:16:41,236] jepsen worker 72 - jepsen.util 3762 :info :write [24 30] :RequestTimeoutException INFO [2022-03-13 17:16:41,237] jepsen worker 72 - jepsen.util 3852 :invoke :read [24 nil] INFO [2022-03-13 17:16:41,258] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:41,258] jepsen worker nemesis - couchbase.util Rebalance check count: 509 INFO [2022-03-13 17:16:41,258] jepsen worker nemesis - couchbase.util Rebalance stuck count: 409 INFO [2022-03-13 17:16:41,343] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,343] jepsen worker 80 - jepsen.util 800 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,344] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:41,344] jepsen worker 80 - jepsen.util 800 :invoke :write [26 4] INFO [2022-03-13 17:16:41,373] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,373] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,373] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:41,374] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:41,523] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,523] jepsen worker 24 - jepsen.util 564 :invoke :write [8 24] INFO [2022-03-13 17:16:41,540] jepsen worker 7 - jepsen.util 637 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,540] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,541] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,541] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,541] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,541] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:41,541] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,542] jepsen worker 6 - jepsen.util 636 :invoke :write [2 17] INFO [2022-03-13 17:16:41,542] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:41,543] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:41,543] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:41,545] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:41,547] jepsen worker 14 - jepsen.util 374 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,547] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:41,576] jepsen worker 12 - jepsen.util 732 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,576] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:41,624] jepsen worker 2 - jepsen.util 3962 :ok :read [0 41] INFO [2022-03-13 17:16:41,624] jepsen worker 2 - jepsen.util 3962 :invoke :write [0 29] INFO [2022-03-13 17:16:41,628] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,629] jepsen worker 21 - jepsen.util 831 :invoke :write [7 5] INFO [2022-03-13 17:16:41,632] jepsen worker 87 - jepsen.util 2607 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,633] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:41,636] jepsen worker 41 - jepsen.util 3551 :ok :read [13 39] INFO [2022-03-13 17:16:41,637] jepsen worker 41 - jepsen.util 3551 :invoke :write [13 5] INFO [2022-03-13 17:16:41,682] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,682] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,683] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 14] INFO [2022-03-13 17:16:41,683] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:41,685] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,686] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:16:41,695] jepsen worker 5 - jepsen.util 2435 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,696] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:41,700] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,700] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:41,701] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,702] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 38] INFO [2022-03-13 17:16:41,715] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,716] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:41,728] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,728] jepsen worker 75 - jepsen.util 435 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,729] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:41,729] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:41,759] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,759] jepsen worker 34 - jepsen.util 934 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,760] jepsen worker 34 - jepsen.util 934 :invoke :write [11 19] INFO [2022-03-13 17:16:41,761] jepsen worker 89 - jepsen.util 2519 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,761] jepsen worker 38 - jepsen.util 2468 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,762] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:41,764] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,764] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 43] INFO [2022-03-13 17:16:41,768] jepsen worker 59 - jepsen.util 3659 :ok :read [19 16] INFO [2022-03-13 17:16:41,769] jepsen worker 59 - jepsen.util 3659 :invoke :read [19 nil] INFO [2022-03-13 17:16:41,787] jepsen worker 15 - jepsen.util 3615 :ok :read [5 43] INFO [2022-03-13 17:16:41,788] jepsen worker 15 - jepsen.util 3615 :invoke :read [5 nil] INFO [2022-03-13 17:16:41,791] jepsen worker 48 - jepsen.util 3558 :fail :write [16 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,791] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 39] INFO [2022-03-13 17:16:41,805] jepsen worker 87 - jepsen.util 2607 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,807] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 25] INFO [2022-03-13 17:16:41,854] jepsen worker 36 - jepsen.util 2196 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,855] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 3] INFO [2022-03-13 17:16:41,885] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,885] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:41,891] jepsen worker 5 - jepsen.util 2435 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,891] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 5] INFO [2022-03-13 17:16:41,898] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,899] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,902] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,902] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,903] jepsen worker 37 - jepsen.util 2107 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,903] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,903] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,904] jepsen worker 61 - jepsen.util 691 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,904] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,904] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,904] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:41,904] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,905] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,905] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:41,905] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,906] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 22] INFO [2022-03-13 17:16:41,912] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,914] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:41,917] jepsen worker 70 - jepsen.util 790 :invoke :write [23 30] INFO [2022-03-13 17:16:41,918] jepsen worker 45 - jepsen.util 3825 :ok :read [15 48] INFO [2022-03-13 17:16:41,918] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:41,918] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,919] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,919] jepsen worker 45 - jepsen.util 3825 :invoke :write [15 47] INFO [2022-03-13 17:16:41,921] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,922] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,923] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,923] jepsen worker 24 - jepsen.util 564 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,925] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:41,926] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,926] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,926] jepsen worker 59 - jepsen.util 3659 :ok :read [19 16] INFO [2022-03-13 17:16:41,926] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:41,927] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:41,928] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,928] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:16:41,928] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,932] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:16:41,934] jepsen worker 79 - jepsen.util 1069 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,934] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:41,934] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,934] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,935] jepsen worker 24 - jepsen.util 564 :invoke :write [8 39] INFO [2022-03-13 17:16:41,935] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,935] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,935] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 38] INFO [2022-03-13 17:16:41,938] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,944] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:41,945] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:41,944] jepsen worker 59 - jepsen.util 3659 :invoke :write [19 14] INFO [2022-03-13 17:16:41,946] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:41,947] jepsen worker 53 - jepsen.util 503 :invoke :write [17 11] INFO [2022-03-13 17:16:41,947] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:41,948] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:41,967] jepsen worker 48 - jepsen.util 3558 :fail :write [16 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,968] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 33] INFO [2022-03-13 17:16:41,978] jepsen worker 4 - jepsen.util 2254 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:41,979] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:41,979] jepsen worker 15 - jepsen.util 3615 :ok :read [5 43] INFO [2022-03-13 17:16:41,980] jepsen worker 15 - jepsen.util 3615 :invoke :read [5 nil] INFO [2022-03-13 17:16:42,017] jepsen worker 89 - jepsen.util 2519 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,018] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:42,051] jepsen worker 5 - jepsen.util 2435 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,053] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:42,055] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,056] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 47] INFO [2022-03-13 17:16:42,064] jepsen worker 87 - jepsen.util 2607 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,064] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,064] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,065] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:42,066] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 42] INFO [2022-03-13 17:16:42,066] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:42,074] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,075] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 24] INFO [2022-03-13 17:16:42,075] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,075] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 42] INFO [2022-03-13 17:16:42,102] jepsen worker 36 - jepsen.util 2196 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,102] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,103] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 13] INFO [2022-03-13 17:16:42,103] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:42,107] jepsen worker 28 - jepsen.util 3628 :info :write [9 16] :RequestTimeoutException INFO [2022-03-13 17:16:42,107] jepsen worker 28 - jepsen.util 3718 :invoke :read [9 nil] INFO [2022-03-13 17:16:42,110] jepsen worker 33 - jepsen.util 753 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,110] jepsen worker 33 - jepsen.util 753 :invoke :write [11 27] INFO [2022-03-13 17:16:42,111] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,111] jepsen worker 13 - jepsen.util 733 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,112] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:42,113] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:42,116] jepsen worker 54 - jepsen.util 3204 :info :write [18 26] :RequestTimeoutException INFO [2022-03-13 17:16:42,117] jepsen worker 54 - jepsen.util 3294 :invoke :write [18 3] INFO [2022-03-13 17:16:42,136] jepsen worker 77 - jepsen.util 707 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,137] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:42,142] jepsen worker 48 - jepsen.util 3558 :fail :write [16 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,143] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 26] INFO [2022-03-13 17:16:42,162] jepsen worker 15 - jepsen.util 3615 :ok :read [5 43] INFO [2022-03-13 17:16:42,162] jepsen worker 15 - jepsen.util 3615 :invoke :read [5 nil] INFO [2022-03-13 17:16:42,171] jepsen worker 4 - jepsen.util 2254 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,172] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 25] INFO [2022-03-13 17:16:42,175] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,175] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:42,184] jepsen worker 37 - jepsen.util 2107 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,184] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,184] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,185] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,186] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,187] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:16:42,206] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,206] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,211] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,212] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 33] INFO [2022-03-13 17:16:42,220] jepsen worker 1 - jepsen.util 3151 :info :write [0 16] :RequestTimeoutException INFO [2022-03-13 17:16:42,223] jepsen worker 1 - jepsen.util 3241 :invoke :read [0 nil] INFO [2022-03-13 17:16:42,264] jepsen worker 88 - jepsen.util 2068 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,264] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 19] INFO [2022-03-13 17:16:42,265] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:42,265] jepsen worker nemesis - couchbase.util Rebalance check count: 510 INFO [2022-03-13 17:16:42,265] jepsen worker nemesis - couchbase.util Rebalance stuck count: 410 INFO [2022-03-13 17:16:42,268] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,268] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:16:42,277] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,277] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:42,280] jepsen worker 6 - jepsen.util 636 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,280] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:42,280] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,280] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,281] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:42,281] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,281] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,282] jepsen worker 42 - jepsen.util 402 :invoke :write [14 26] INFO [2022-03-13 17:16:42,284] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:42,285] jepsen worker 62 - jepsen.util 512 :invoke :write [20 6] INFO [2022-03-13 17:16:42,285] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:42,286] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:42,286] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:42,287] jepsen worker 43 - jepsen.util 583 :invoke :write [14 21] INFO [2022-03-13 17:16:42,294] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,295] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 31] INFO [2022-03-13 17:16:42,295] jepsen worker 36 - jepsen.util 2196 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,296] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,304] jepsen worker 5 - jepsen.util 2435 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,304] jepsen worker 3 - jepsen.util 1983 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,305] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:42,306] jepsen worker 15 - jepsen.util 3615 :ok :read [5 43] INFO [2022-03-13 17:16:42,307] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 44] INFO [2022-03-13 17:16:42,308] jepsen worker 15 - jepsen.util 3615 :invoke :write [5 8] INFO [2022-03-13 17:16:42,310] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,311] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 22] INFO [2022-03-13 17:16:42,320] jepsen worker 76 - jepsen.util 706 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,320] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,320] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,320] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,320] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:42,321] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,321] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,321] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,321] jepsen worker 76 - jepsen.util 706 :invoke :write [25 0] INFO [2022-03-13 17:16:42,322] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:42,323] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:42,323] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:42,324] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:16:42,325] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:42,325] jepsen worker 61 - jepsen.util 691 :invoke :write [20 24] INFO [2022-03-13 17:16:42,326] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:42,361] jepsen worker 87 - jepsen.util 2607 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,362] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 47] INFO [2022-03-13 17:16:42,396] jepsen worker 1 - jepsen.util 3241 :ok :read [0 41] INFO [2022-03-13 17:16:42,397] jepsen worker 1 - jepsen.util 3241 :invoke :read [0 nil] INFO [2022-03-13 17:16:42,404] jepsen worker 32 - jepsen.util 3632 :info :write [10 0] :RequestTimeoutException INFO [2022-03-13 17:16:42,405] jepsen worker 32 - jepsen.util 3722 :invoke :read [10 nil] INFO [2022-03-13 17:16:42,419] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,419] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,419] jepsen worker 48 - jepsen.util 3558 :fail :write [16 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,420] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,421] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 5] INFO [2022-03-13 17:16:42,426] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 6] INFO [2022-03-13 17:16:42,441] jepsen worker 65 - jepsen.util 4115 :ok :read [21 :nil] INFO [2022-03-13 17:16:42,441] jepsen worker 32 - jepsen.util 3722 :ok :read [10 :nil] INFO [2022-03-13 17:16:42,441] jepsen worker 65 - jepsen.util 4115 :invoke :write [21 12] INFO [2022-03-13 17:16:42,442] jepsen worker 32 - jepsen.util 3722 :invoke :write [10 13] INFO [2022-03-13 17:16:42,443] jepsen worker 79 - jepsen.util 1069 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,445] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 17] INFO [2022-03-13 17:16:42,454] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,457] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:42,465] jepsen worker 80 - jepsen.util 800 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,465] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,466] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:42,467] jepsen worker 80 - jepsen.util 800 :invoke :write [26 49] INFO [2022-03-13 17:16:42,474] jepsen worker 4 - jepsen.util 2254 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,474] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:16:42,475] jepsen worker 10 - jepsen.util 3610 :ok :read [3 25] INFO [2022-03-13 17:16:42,476] jepsen worker 10 - jepsen.util 3610 :invoke :write [3 30] INFO [2022-03-13 17:16:42,493] jepsen worker 70 - jepsen.util 790 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,493] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,493] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,493] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:42,493] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:42,494] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,494] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,494] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,496] jepsen worker 37 - jepsen.util 2107 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,496] jepsen worker 17 - jepsen.util 3617 :info :write [5 19] :RequestTimeoutException INFO [2022-03-13 17:16:42,496] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,497] jepsen worker 78 - jepsen.util 618 :invoke :write [26 44] INFO [2022-03-13 17:16:42,498] jepsen worker 12 - jepsen.util 732 :invoke :read [4 nil] INFO [2022-03-13 17:16:42,499] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:42,500] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,502] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:42,503] jepsen worker 60 - jepsen.util 600 :invoke :write [20 45] INFO [2022-03-13 17:16:42,503] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,504] jepsen worker 17 - jepsen.util 3707 :invoke :write [5 1] INFO [2022-03-13 17:16:42,506] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:42,564] jepsen worker 88 - jepsen.util 2068 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,564] jepsen worker 20 - jepsen.util 3890 :ok :read [6 16] INFO [2022-03-13 17:16:42,565] jepsen worker 20 - jepsen.util 3890 :invoke :write [6 23] INFO [2022-03-13 17:16:42,565] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 38] INFO [2022-03-13 17:16:42,572] jepsen worker 74 - jepsen.util 3764 :ok :read [24 24] INFO [2022-03-13 17:16:42,573] jepsen worker 74 - jepsen.util 3764 :invoke :read [24 nil] INFO [2022-03-13 17:16:42,599] jepsen worker 1 - jepsen.util 3241 :ok :read [0 41] INFO [2022-03-13 17:16:42,600] jepsen worker 5 - jepsen.util 2435 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,600] jepsen worker 1 - jepsen.util 3241 :invoke :read [0 nil] INFO [2022-03-13 17:16:42,600] jepsen worker 3 - jepsen.util 1983 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,600] jepsen worker 5 - jepsen.util 2435 :invoke :read [1 nil] INFO [2022-03-13 17:16:42,601] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 48] INFO [2022-03-13 17:16:42,603] jepsen worker 38 - jepsen.util 2468 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,603] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,603] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,604] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 16] INFO [2022-03-13 17:16:42,615] jepsen worker 53 - jepsen.util 503 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,615] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,615] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,615] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,615] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,616] jepsen worker 6 - jepsen.util 636 :invoke :write [2 5] INFO [2022-03-13 17:16:42,617] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:42,618] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:42,619] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:42,620] jepsen worker 21 - jepsen.util 831 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,620] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,620] jepsen worker 21 - jepsen.util 831 :invoke :write [7 23] INFO [2022-03-13 17:16:42,621] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:42,630] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:42,633] jepsen worker 24 - jepsen.util 564 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,634] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,634] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:42,634] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,634] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:42,635] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:42,638] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,639] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:42,639] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,640] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,640] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:42,641] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:42,641] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,642] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:42,651] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,651] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,651] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,654] jepsen worker 87 - jepsen.util 2607 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,655] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:42,656] jepsen worker 24 - jepsen.util 564 :invoke :write [8 1] INFO [2022-03-13 17:16:42,656] jepsen worker 71 - jepsen.util 521 :invoke :write [23 11] INFO [2022-03-13 17:16:42,657] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 5] INFO [2022-03-13 17:16:42,660] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,660] jepsen worker 75 - jepsen.util 435 :invoke :write [25 7] INFO [2022-03-13 17:16:42,667] jepsen worker 89 - jepsen.util 2519 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,667] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,667] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,668] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:42,668] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,669] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 36] INFO [2022-03-13 17:16:42,672] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,672] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,673] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,674] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:42,674] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,675] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,675] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 45] INFO [2022-03-13 17:16:42,678] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:42,686] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,686] jepsen worker 23 - jepsen.util 563 :invoke :write [7 35] INFO [2022-03-13 17:16:42,692] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,693] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 9] INFO [2022-03-13 17:16:42,706] jepsen worker 19 - jepsen.util 3259 :info :write [6 30] :RequestTimeoutException INFO [2022-03-13 17:16:42,707] jepsen worker 19 - jepsen.util 3349 :invoke :write [6 27] INFO [2022-03-13 17:16:42,723] jepsen worker 48 - jepsen.util 3558 :fail :write [16 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,724] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,725] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 45] INFO [2022-03-13 17:16:42,725] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 29] INFO [2022-03-13 17:16:42,776] jepsen worker 52 - jepsen.util 682 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,777] jepsen worker 52 - jepsen.util 682 :invoke :write [17 11] INFO [2022-03-13 17:16:42,778] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,779] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:42,783] jepsen worker 37 - jepsen.util 2107 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,784] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,787] jepsen worker 43 - jepsen.util 583 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,788] jepsen worker 34 - jepsen.util 934 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,789] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,789] jepsen worker 34 - jepsen.util 934 :invoke :write [11 35] INFO [2022-03-13 17:16:42,791] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:42,791] jepsen worker 43 - jepsen.util 583 :invoke :read [14 nil] INFO [2022-03-13 17:16:42,814] jepsen worker 3 - jepsen.util 1983 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,814] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 22] INFO [2022-03-13 17:16:42,815] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,815] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:16:42,821] jepsen worker 88 - jepsen.util 2068 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,822] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 21] INFO [2022-03-13 17:16:42,832] jepsen worker 24 - jepsen.util 564 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,833] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:42,893] jepsen worker 48 - jepsen.util 3558 :fail :write [16 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,894] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 49] INFO [2022-03-13 17:16:42,908] jepsen worker 87 - jepsen.util 2607 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,908] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 3] INFO [2022-03-13 17:16:42,934] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,934] jepsen worker 38 - jepsen.util 2468 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,934] jepsen worker 36 - jepsen.util 2196 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,935] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,936] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,936] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:42,940] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,940] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,940] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 6] INFO [2022-03-13 17:16:42,941] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 22] INFO [2022-03-13 17:16:42,956] jepsen worker 3 - jepsen.util 1983 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,956] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:42,962] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:42,963] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 34] INFO [2022-03-13 17:16:43,002] jepsen worker 89 - jepsen.util 2519 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,002] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,003] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 30] INFO [2022-03-13 17:16:43,006] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 16] INFO [2022-03-13 17:16:43,020] jepsen worker 4 - jepsen.util 2254 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,021] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:43,031] jepsen worker 48 - jepsen.util 3558 :fail :write [16 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,031] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 45] INFO [2022-03-13 17:16:43,077] jepsen worker 37 - jepsen.util 2107 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,078] jepsen worker 37 - jepsen.util 2107 :invoke :read [12 nil] INFO [2022-03-13 17:16:43,093] jepsen worker 3 - jepsen.util 1983 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,094] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:43,110] jepsen worker 88 - jepsen.util 2068 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,111] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:43,159] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,159] jepsen worker 48 - jepsen.util 3558 :fail :write [16 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,160] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 15] INFO [2022-03-13 17:16:43,160] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 32] INFO [2022-03-13 17:16:43,161] jepsen worker 87 - jepsen.util 2607 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,162] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,162] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,162] jepsen worker 87 - jepsen.util 2607 :invoke :read [29 nil] INFO [2022-03-13 17:16:43,162] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 23] INFO [2022-03-13 17:16:43,163] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:43,163] jepsen worker 87 - jepsen.util 2607 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,164] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 17] INFO [2022-03-13 17:16:43,164] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,164] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 26] INFO [2022-03-13 17:16:43,174] jepsen worker 11 - jepsen.util 3521 :info :write [3 2] :RequestTimeoutException INFO [2022-03-13 17:16:43,175] jepsen worker 11 - jepsen.util 3611 :invoke :read [3 nil] INFO [2022-03-13 17:16:43,177] jepsen worker 38 - jepsen.util 2468 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,177] jepsen worker 37 - jepsen.util 2107 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,178] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 18] INFO [2022-03-13 17:16:43,178] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 14] INFO [2022-03-13 17:16:43,204] jepsen worker 56 - jepsen.util 3656 :info :write [18 37] :RequestTimeoutException INFO [2022-03-13 17:16:43,204] jepsen worker 56 - jepsen.util 3746 :invoke :write [18 19] INFO [2022-03-13 17:16:43,238] jepsen worker 36 - jepsen.util 2196 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,239] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:43,268] jepsen worker 3 - jepsen.util 1983 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,269] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:43,269] jepsen worker 4 - jepsen.util 2254 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,270] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 46] INFO [2022-03-13 17:16:43,271] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6140350877192983}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:43,271] jepsen worker nemesis - couchbase.util Rebalance check count: 511 INFO [2022-03-13 17:16:43,271] jepsen worker nemesis - couchbase.util Rebalance stuck count: 411 INFO [2022-03-13 17:16:43,311] jepsen worker 9 - jepsen.util 2979 :info :write [3 45] :RequestTimeoutException INFO [2022-03-13 17:16:43,311] jepsen worker 9 - jepsen.util 3069 :invoke :write [3 36] INFO [2022-03-13 17:16:43,326] jepsen worker 88 - jepsen.util 2068 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,326] jepsen worker 48 - jepsen.util 3558 :fail :write [16 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,327] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,327] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 46] INFO [2022-03-13 17:16:43,328] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:16:43,331] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 41] INFO [2022-03-13 17:16:43,370] jepsen worker 89 - jepsen.util 2519 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,370] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,371] jepsen worker 87 - jepsen.util 2607 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,371] jepsen worker 36 - jepsen.util 2196 :invoke :read [12 nil] INFO [2022-03-13 17:16:43,371] jepsen worker 89 - jepsen.util 2519 :invoke :read [29 nil] INFO [2022-03-13 17:16:43,372] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 42] INFO [2022-03-13 17:16:43,414] jepsen worker 3 - jepsen.util 1983 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,415] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:43,416] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,416] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 29] INFO [2022-03-13 17:16:43,426] jepsen worker 4 - jepsen.util 2254 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,427] jepsen worker 4 - jepsen.util 2254 :invoke :read [1 nil] INFO [2022-03-13 17:16:43,435] jepsen worker 37 - jepsen.util 2107 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,435] jepsen worker 37 - jepsen.util 2107 :invoke :write [12 26] INFO [2022-03-13 17:16:43,438] jepsen worker 36 - jepsen.util 2196 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,438] jepsen worker 89 - jepsen.util 2519 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,439] jepsen worker 36 - jepsen.util 2196 :invoke :write [12 37] INFO [2022-03-13 17:16:43,440] jepsen worker 89 - jepsen.util 2519 :invoke :write [29 45] INFO [2022-03-13 17:16:43,510] jepsen worker 88 - jepsen.util 2068 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,510] jepsen worker 88 - jepsen.util 2068 :invoke :read [29 nil] INFO [2022-03-13 17:16:43,558] jepsen worker 3 - jepsen.util 1983 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,559] jepsen worker 3 - jepsen.util 1983 :invoke :read [1 nil] INFO [2022-03-13 17:16:43,589] jepsen worker 87 - jepsen.util 2607 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,589] jepsen worker 4 - jepsen.util 2254 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,589] jepsen worker 88 - jepsen.util 2068 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,590] jepsen worker 4 - jepsen.util 2254 :invoke :write [1 32] INFO [2022-03-13 17:16:43,590] jepsen worker 87 - jepsen.util 2607 :invoke :write [29 23] INFO [2022-03-13 17:16:43,591] jepsen worker 88 - jepsen.util 2068 :invoke :write [29 15] INFO [2022-03-13 17:16:43,595] jepsen worker 38 - jepsen.util 2468 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,596] jepsen worker 38 - jepsen.util 2468 :invoke :read [12 nil] INFO [2022-03-13 17:16:43,597] jepsen worker 29 - jepsen.util 3809 :info :write [9 11] :RequestTimeoutException INFO [2022-03-13 17:16:43,598] jepsen worker 29 - jepsen.util 3899 :invoke :write [9 6] INFO [2022-03-13 17:16:43,621] jepsen worker 3 - jepsen.util 1983 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,621] jepsen worker 3 - jepsen.util 1983 :invoke :write [1 28] INFO [2022-03-13 17:16:43,623] jepsen worker 5 - jepsen.util 2435 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,624] jepsen worker 5 - jepsen.util 2435 :invoke :write [1 36] INFO [2022-03-13 17:16:43,748] jepsen worker 38 - jepsen.util 2468 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,748] jepsen worker 38 - jepsen.util 2468 :invoke :write [12 45] INFO [2022-03-13 17:16:43,844] jepsen worker 76 - jepsen.util 706 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,845] jepsen worker 76 - jepsen.util 706 :invoke :write [25 13] INFO [2022-03-13 17:16:43,848] jepsen worker 33 - jepsen.util 753 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,849] jepsen worker 33 - jepsen.util 753 :invoke :write [11 12] INFO [2022-03-13 17:16:43,849] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,849] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:43,851] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,852] jepsen worker 14 - jepsen.util 374 :invoke :write [4 29] INFO [2022-03-13 17:16:43,901] jepsen worker 75 - jepsen.util 435 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,902] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:43,922] jepsen worker 42 - jepsen.util 402 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,922] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:43,929] jepsen worker 80 - jepsen.util 800 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,929] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,930] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:43,930] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:16:43,967] jepsen worker 71 - jepsen.util 521 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,967] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,967] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,967] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:43,971] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:43,968] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:43,984] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,984] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,984] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,984] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:43,985] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:16:43,985] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:43,986] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,986] jepsen worker 75 - jepsen.util 435 :invoke :write [25 44] INFO [2022-03-13 17:16:43,987] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,987] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:43,989] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:43,990] jepsen worker 71 - jepsen.util 521 :invoke :write [23 16] INFO [2022-03-13 17:16:43,997] jepsen worker 66 - jepsen.util 3306 :info :write [22 2] :SyncWriteAmbiguous INFO [2022-03-13 17:16:43,998] jepsen worker 66 - jepsen.util 3396 :invoke :read [22 nil] INFO [2022-03-13 17:16:44,011] jepsen worker 61 - jepsen.util 691 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,011] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,012] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:44,012] jepsen worker 12 - jepsen.util 732 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,012] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,012] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,013] jepsen worker 12 - jepsen.util 732 :invoke :write [4 45] INFO [2022-03-13 17:16:44,014] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:44,015] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:44,016] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,017] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:44,018] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,024] jepsen worker 21 - jepsen.util 831 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,025] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:16:44,031] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,032] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:44,043] jepsen worker 31 - jepsen.util 3811 :info :write [10 12] :RequestTimeoutException INFO [2022-03-13 17:16:44,044] jepsen worker 31 - jepsen.util 3901 :invoke :read [10 nil] INFO [2022-03-13 17:16:44,094] jepsen worker 34 - jepsen.util 934 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,094] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,095] jepsen worker 34 - jepsen.util 934 :invoke :write [11 14] INFO [2022-03-13 17:16:44,095] jepsen worker 35 - jepsen.util 935 :invoke :write [11 22] INFO [2022-03-13 17:16:44,100] jepsen worker 62 - jepsen.util 512 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,100] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:44,129] jepsen worker 57 - jepsen.util 3567 :info :write [19 42] :SyncWriteAmbiguous INFO [2022-03-13 17:16:44,131] jepsen worker 57 - jepsen.util 3657 :invoke :write [19 46] INFO [2022-03-13 17:16:44,137] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,137] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:44,168] jepsen worker 33 - jepsen.util 753 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,169] jepsen worker 33 - jepsen.util 753 :invoke :write [11 45] INFO [2022-03-13 17:16:44,205] jepsen worker 78 - jepsen.util 618 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,206] jepsen worker 78 - jepsen.util 618 :invoke :read [26 nil] INFO [2022-03-13 17:16:44,222] jepsen worker 75 - jepsen.util 435 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,223] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:16:44,247] jepsen worker 14 - jepsen.util 374 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,247] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,249] jepsen worker 14 - jepsen.util 374 :invoke :write [4 34] INFO [2022-03-13 17:16:44,249] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:44,278] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6198830409356726}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:44,278] jepsen worker nemesis - couchbase.util Rebalance check count: 512 INFO [2022-03-13 17:16:44,278] jepsen worker nemesis - couchbase.util Rebalance stuck count: 411 INFO [2022-03-13 17:16:44,287] jepsen worker 79 - jepsen.util 1069 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,287] jepsen worker 79 - jepsen.util 1069 :invoke :read [26 nil] INFO [2022-03-13 17:16:44,289] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,289] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,289] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,289] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,290] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:44,291] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:44,292] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:44,292] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:44,306] jepsen worker 71 - jepsen.util 521 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,306] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,307] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:16:44,307] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:44,330] jepsen worker 85 - jepsen.util 3145 :info :write [28 14] :RequestTimeoutException INFO [2022-03-13 17:16:44,332] jepsen worker 85 - jepsen.util 3235 :invoke :write [28 19] INFO [2022-03-13 17:16:44,350] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,350] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,351] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:16:44,351] jepsen worker 51 - jepsen.util 771 :invoke :write [17 45] INFO [2022-03-13 17:16:44,380] jepsen worker 84 - jepsen.util 3864 :info :write [28 15] :RequestTimeoutException INFO [2022-03-13 17:16:44,381] jepsen worker 84 - jepsen.util 3954 :invoke :write [28 14] INFO [2022-03-13 17:16:44,397] jepsen worker 60 - jepsen.util 600 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,397] jepsen worker 60 - jepsen.util 600 :invoke :read [20 nil] INFO [2022-03-13 17:16:44,408] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,408] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,408] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:16:44,409] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:44,419] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,419] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:44,479] jepsen worker 6 - jepsen.util 636 :fail :write [2 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,479] jepsen worker 43 - jepsen.util 583 :ok :read [14 :nil] INFO [2022-03-13 17:16:44,479] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:44,479] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,480] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,479] jepsen worker 79 - jepsen.util 1069 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,480] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,480] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,480] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,481] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:16:44,483] jepsen worker 62 - jepsen.util 512 :invoke :write [20 34] INFO [2022-03-13 17:16:44,484] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:44,485] jepsen worker 26 - jepsen.util 476 :invoke :read [8 nil] INFO [2022-03-13 17:16:44,486] jepsen worker 43 - jepsen.util 583 :invoke :write [14 1] INFO [2022-03-13 17:16:44,486] jepsen worker 79 - jepsen.util 1069 :invoke :write [26 43] INFO [2022-03-13 17:16:44,488] jepsen worker 7 - jepsen.util 637 :invoke :write [2 21] INFO [2022-03-13 17:16:44,488] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:44,490] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:44,491] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,491] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,492] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,492] jepsen worker 6 - jepsen.util 636 :invoke :write [2 3] INFO [2022-03-13 17:16:44,493] jepsen worker 53 - jepsen.util 503 :invoke :read [17 nil] INFO [2022-03-13 17:16:44,494] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:44,496] jepsen worker 26 - jepsen.util 476 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,496] jepsen worker 26 - jepsen.util 476 :invoke :write [8 39] INFO [2022-03-13 17:16:44,507] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,507] jepsen worker 34 - jepsen.util 934 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,508] jepsen worker 34 - jepsen.util 934 :invoke :write [11 36] INFO [2022-03-13 17:16:44,508] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,510] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,511] jepsen worker 24 - jepsen.util 564 :invoke :write [8 23] INFO [2022-03-13 17:16:44,543] jepsen worker 23 - jepsen.util 563 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,544] jepsen worker 53 - jepsen.util 503 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,544] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,545] jepsen worker 23 - jepsen.util 563 :invoke :write [7 20] INFO [2022-03-13 17:16:44,546] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:44,546] jepsen worker 53 - jepsen.util 503 :invoke :write [17 23] INFO [2022-03-13 17:16:44,550] jepsen worker 39 - jepsen.util 3729 :info :write [13 48] :RequestTimeoutException INFO [2022-03-13 17:16:44,551] jepsen worker 39 - jepsen.util 3819 :invoke :read [13 nil] INFO [2022-03-13 17:16:44,552] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,553] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,553] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:44,553] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,603] jepsen worker 35 - jepsen.util 935 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,604] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:16:44,611] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,611] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,611] jepsen worker 24 - jepsen.util 564 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,612] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,613] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:44,613] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,614] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:16:44,615] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,619] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,619] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,620] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:16:44,621] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,622] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,623] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:16:44,648] jepsen worker 76 - jepsen.util 706 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,648] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,648] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:44,648] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,649] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,649] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,650] jepsen worker 61 - jepsen.util 691 :invoke :read [20 nil] INFO [2022-03-13 17:16:44,651] jepsen worker 44 - jepsen.util 584 :invoke :read [14 nil] INFO [2022-03-13 17:16:44,652] jepsen worker 77 - jepsen.util 707 :invoke :write [25 38] INFO [2022-03-13 17:16:44,653] jepsen worker 69 - jepsen.util 699 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,654] jepsen worker 76 - jepsen.util 706 :invoke :read [25 nil] INFO [2022-03-13 17:16:44,654] jepsen worker 70 - jepsen.util 790 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,671] jepsen worker 33 - jepsen.util 753 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,672] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,672] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:44,673] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:44,801] jepsen worker 62 - jepsen.util 512 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,802] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:44,803] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,804] jepsen worker 24 - jepsen.util 564 :invoke :write [8 19] INFO [2022-03-13 17:16:44,850] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,851] jepsen worker 75 - jepsen.util 435 :invoke :write [25 30] INFO [2022-03-13 17:16:44,857] jepsen worker 14 - jepsen.util 374 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,858] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:44,910] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,911] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:44,935] jepsen worker 12 - jepsen.util 732 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,935] jepsen worker 60 - jepsen.util 600 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,935] jepsen worker 78 - jepsen.util 618 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:44,936] jepsen worker 78 - jepsen.util 618 :invoke :write [26 12] INFO [2022-03-13 17:16:44,936] jepsen worker 60 - jepsen.util 600 :invoke :write [20 25] INFO [2022-03-13 17:16:44,936] jepsen worker 12 - jepsen.util 732 :invoke :write [4 48] INFO [2022-03-13 17:16:45,048] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,049] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:45,058] jepsen worker 24 - jepsen.util 564 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,059] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:45,059] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,060] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:16:45,135] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,135] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,136] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:45,137] jepsen worker 24 - jepsen.util 564 :invoke :write [8 4] INFO [2022-03-13 17:16:45,137] jepsen worker 75 - jepsen.util 435 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,137] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,138] jepsen worker 71 - jepsen.util 521 :invoke :write [23 28] INFO [2022-03-13 17:16:45,139] jepsen worker 75 - jepsen.util 435 :invoke :write [25 24] INFO [2022-03-13 17:16:45,177] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,177] jepsen worker 51 - jepsen.util 771 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,178] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:45,179] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:45,186] jepsen worker 23 - jepsen.util 563 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,186] jepsen worker 61 - jepsen.util 691 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,186] jepsen worker 44 - jepsen.util 584 :ok :read [14 :nil] INFO [2022-03-13 17:16:45,186] jepsen worker 69 - jepsen.util 699 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,187] jepsen worker 76 - jepsen.util 706 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,187] jepsen worker 70 - jepsen.util 790 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,187] jepsen worker 43 - jepsen.util 583 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,188] jepsen worker 43 - jepsen.util 583 :invoke :write [14 19] INFO [2022-03-13 17:16:45,189] jepsen worker 61 - jepsen.util 691 :invoke :write [20 9] INFO [2022-03-13 17:16:45,190] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:16:45,191] jepsen worker 44 - jepsen.util 584 :invoke :write [14 10] INFO [2022-03-13 17:16:45,192] jepsen worker 76 - jepsen.util 706 :invoke :write [25 28] INFO [2022-03-13 17:16:45,193] jepsen worker 69 - jepsen.util 699 :invoke :write [23 16] INFO [2022-03-13 17:16:45,194] jepsen worker 70 - jepsen.util 790 :invoke :write [23 38] INFO [2022-03-13 17:16:45,234] jepsen worker 71 - jepsen.util 521 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,235] jepsen worker 39 - jepsen.util 3819 :ok :read [13 39] INFO [2022-03-13 17:16:45,235] jepsen worker 71 - jepsen.util 521 :invoke :write [23 31] INFO [2022-03-13 17:16:45,236] jepsen worker 39 - jepsen.util 3819 :invoke :read [13 nil] INFO [2022-03-13 17:16:45,284] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3460410557184751}, :ns_1@172.23.105.187 {:progress 0.6198830409356726}, :ns_1@172.23.107.227 {:progress 0.3460410557184751}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:45,284] jepsen worker nemesis - couchbase.util Rebalance check count: 513 INFO [2022-03-13 17:16:45,284] jepsen worker nemesis - couchbase.util Rebalance stuck count: 412 INFO [2022-03-13 17:16:45,296] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,297] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:45,299] jepsen worker 39 - jepsen.util 3819 :ok :read [13 39] INFO [2022-03-13 17:16:45,300] jepsen worker 39 - jepsen.util 3819 :invoke :write [13 31] INFO [2022-03-13 17:16:45,317] jepsen worker 34 - jepsen.util 934 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,318] jepsen worker 34 - jepsen.util 934 :invoke :write [11 25] INFO [2022-03-13 17:16:45,327] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,328] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:45,384] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,385] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,385] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,385] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,385] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,385] jepsen worker 13 - jepsen.util 733 :invoke :write [4 15] INFO [2022-03-13 17:16:45,386] jepsen worker 14 - jepsen.util 374 :invoke :write [4 31] INFO [2022-03-13 17:16:45,387] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:45,388] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:16:45,389] jepsen worker 80 - jepsen.util 800 :invoke :write [26 8] INFO [2022-03-13 17:16:45,780] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,780] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,780] jepsen worker 86 - jepsen.util 3416 :info :write [28 9] :RequestTimeoutException INFO [2022-03-13 17:16:45,780] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,780] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:45,781] jepsen worker 86 - jepsen.util 3506 :invoke :read [28 nil] INFO [2022-03-13 17:16:45,781] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:45,782] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:45,782] jepsen worker 24 - jepsen.util 564 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,783] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:45,788] jepsen worker 6 - jepsen.util 636 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,788] jepsen worker 77 - jepsen.util 707 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,788] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:16:45,789] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:16:45,841] jepsen worker 34 - jepsen.util 934 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,841] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,842] jepsen worker 33 - jepsen.util 753 :invoke :write [11 43] INFO [2022-03-13 17:16:45,847] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:45,867] jepsen worker 52 - jepsen.util 682 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,868] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:45,887] jepsen worker 82 - jepsen.util 3682 :info :write [27 29] :RequestTimeoutException INFO [2022-03-13 17:16:45,888] jepsen worker 82 - jepsen.util 3772 :invoke :write [27 5] INFO [2022-03-13 17:16:45,906] jepsen worker 75 - jepsen.util 435 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,907] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:45,921] jepsen worker 14 - jepsen.util 374 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,922] jepsen worker 14 - jepsen.util 374 :invoke :write [4 22] INFO [2022-03-13 17:16:45,980] jepsen worker 67 - jepsen.util 2947 :info :write [22 13] :RequestTimeoutException INFO [2022-03-13 17:16:45,981] jepsen worker 67 - jepsen.util 3037 :invoke :read [22 nil] INFO [2022-03-13 17:16:45,989] jepsen worker 71 - jepsen.util 521 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,989] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:45,991] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,991] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,992] jepsen worker 75 - jepsen.util 435 :invoke :write [25 2] INFO [2022-03-13 17:16:45,995] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:45,996] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,997] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:45,997] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,997] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:45,998] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,999] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:45,999] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:45,999] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:46,000] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,000] jepsen worker 71 - jepsen.util 521 :invoke :write [23 19] INFO [2022-03-13 17:16:46,001] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,001] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:46,002] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,003] jepsen worker 24 - jepsen.util 564 :invoke :write [8 4] INFO [2022-03-13 17:16:46,032] jepsen worker 80 - jepsen.util 800 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,033] jepsen worker 80 - jepsen.util 800 :invoke :write [26 36] INFO [2022-03-13 17:16:46,046] jepsen worker 13 - jepsen.util 733 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,047] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:16:46,050] jepsen worker 0 - jepsen.util 3510 :info :write [0 43] :RequestTimeoutException INFO [2022-03-13 17:16:46,050] jepsen worker 0 - jepsen.util 3600 :invoke :read [0 nil] INFO [2022-03-13 17:16:46,101] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,102] jepsen worker 22 - jepsen.util 652 :invoke :write [7 45] INFO [2022-03-13 17:16:46,112] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,112] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:46,114] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,115] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:46,117] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,118] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:46,119] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,120] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:46,131] jepsen worker 75 - jepsen.util 435 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,131] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,132] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:16:46,132] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:46,169] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,169] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,169] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,170] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,170] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:46,170] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:46,170] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,170] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:16:46,171] jepsen worker 51 - jepsen.util 771 :invoke :write [17 31] INFO [2022-03-13 17:16:46,171] jepsen worker 34 - jepsen.util 934 :invoke :write [11 17] INFO [2022-03-13 17:16:46,191] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,192] jepsen worker 13 - jepsen.util 733 :invoke :write [4 1] INFO [2022-03-13 17:16:46,289] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6198830409356726}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:46,290] jepsen worker nemesis - couchbase.util Rebalance check count: 514 INFO [2022-03-13 17:16:46,290] jepsen worker nemesis - couchbase.util Rebalance stuck count: 412 INFO [2022-03-13 17:16:46,434] jepsen worker 71 - jepsen.util 521 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,435] jepsen worker 71 - jepsen.util 521 :invoke :write [23 20] INFO [2022-03-13 17:16:46,472] jepsen worker 33 - jepsen.util 753 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,473] jepsen worker 33 - jepsen.util 753 :invoke :write [11 32] INFO [2022-03-13 17:16:46,480] jepsen worker 22 - jepsen.util 652 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,480] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,481] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:46,482] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:16:46,482] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,483] jepsen worker 52 - jepsen.util 682 :invoke :write [17 25] INFO [2022-03-13 17:16:46,497] jepsen worker 24 - jepsen.util 564 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,498] jepsen worker 24 - jepsen.util 564 :invoke :write [8 3] INFO [2022-03-13 17:16:46,552] jepsen worker 13 - jepsen.util 733 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,553] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:46,555] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,556] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:46,557] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,558] jepsen worker 13 - jepsen.util 733 :invoke :write [4 8] INFO [2022-03-13 17:16:46,604] jepsen worker 14 - jepsen.util 374 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,605] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:46,610] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,611] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:16:46,632] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,632] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:46,686] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,686] jepsen worker 80 - jepsen.util 800 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,687] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,687] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:46,687] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:46,688] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:16:46,713] jepsen worker 71 - jepsen.util 521 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,714] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] WARN [2022-03-13 17:16:46,729] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9038590,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9768700,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9462407,"encode_duration_us":19852,"operation_id":"0xe821","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10008376}],"total_count":1}} WARN [2022-03-13 17:16:46,729] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8200606,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9282054,"total_server_duration_us":316562,"encode_duration_us":16590,"operation_id":"0xec60","timeout_ms":10000,"last_server_duration_us":58128,"total_duration_us":10009557},{"operation_name":"upsert","last_dispatch_duration_us":8305370,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9296228,"total_server_duration_us":252377,"encode_duration_us":19552,"operation_id":"0xec88","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10005599},{"operation_name":"upsert","last_dispatch_duration_us":97174,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":97174,"total_server_duration_us":56034,"encode_duration_us":24904,"operation_id":"0xed42","timeout_ms":10000,"last_server_duration_us":56034,"total_duration_us":10004724},{"operation_name":"upsert","last_dispatch_duration_us":33849,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7537713,"total_server_duration_us":329550,"encode_duration_us":29556,"operation_id":"0xebcf","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10001454},{"operation_name":"upsert","last_dispatch_duration_us":7839176,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8863056,"total_server_duration_us":280861,"encode_duration_us":18909,"operation_id":"0xec5f","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":10001291},{"operation_name":"upsert","last_dispatch_duration_us":9159901,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9285945,"total_server_duration_us":9074085,"encode_duration_us":29583,"operation_id":"0xeca8","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9339918},{"operation_name":"upsert","last_dispatch_duration_us":9220241,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9220241,"total_server_duration_us":9051168,"encode_duration_us":24976,"operation_id":"0xecca","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9225766},{"operation_name":"get","last_dispatch_duration_us":7003551,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7003551,"total_server_duration_us":27,"operation_id":"0xecc9","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":7005679},{"operation_name":"upsert","last_dispatch_duration_us":212303,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":2606587,"total_server_duration_us":542605,"encode_duration_us":21839,"operation_id":"0xef08","timeout_ms":10000,"last_server_duration_us":38795,"total_duration_us":2963349},{"operation_name":"upsert","last_dispatch_duration_us":246304,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":2478703,"total_server_duration_us":460570,"encode_duration_us":22484,"operation_id":"0xef16","timeout_ms":10000,"last_server_duration_us":65775,"total_duration_us":2708772}],"total_count":81}} INFO [2022-03-13 17:16:46,751] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,751] jepsen worker 52 - jepsen.util 682 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,752] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:46,753] jepsen worker 52 - jepsen.util 682 :invoke :write [17 47] INFO [2022-03-13 17:16:46,782] jepsen worker 24 - jepsen.util 564 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,783] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:46,788] jepsen worker 34 - jepsen.util 934 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,789] jepsen worker 34 - jepsen.util 934 :invoke :write [11 6] INFO [2022-03-13 17:16:46,803] jepsen worker 13 - jepsen.util 733 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,804] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,804] jepsen worker 13 - jepsen.util 733 :invoke :write [4 44] INFO [2022-03-13 17:16:46,805] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:16:46,866] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,866] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:46,928] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,928] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,928] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,928] jepsen worker 52 - jepsen.util 682 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,928] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:46,929] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:46,930] jepsen worker 71 - jepsen.util 521 :invoke :write [23 27] INFO [2022-03-13 17:16:46,930] jepsen worker 52 - jepsen.util 682 :invoke :write [17 32] INFO [2022-03-13 17:16:46,931] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,932] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:16:46,935] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,936] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:46,938] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,938] jepsen worker 24 - jepsen.util 564 :invoke :write [8 16] INFO [2022-03-13 17:16:46,991] jepsen worker 51 - jepsen.util 771 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:46,992] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:47,102] jepsen worker 71 - jepsen.util 521 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,102] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:16:47,109] jepsen worker 33 - jepsen.util 753 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,109] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,109] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,109] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,110] jepsen worker 33 - jepsen.util 753 :invoke :write [11 39] INFO [2022-03-13 17:16:47,111] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,112] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:16:47,116] jepsen worker 80 - jepsen.util 800 :invoke :write [26 9] INFO [2022-03-13 17:16:47,156] jepsen worker 18 - jepsen.util 3618 :info :write [6 22] :RequestTimeoutException INFO [2022-03-13 17:16:47,157] jepsen worker 18 - jepsen.util 3708 :invoke :read [6 nil] INFO [2022-03-13 17:16:47,225] jepsen worker 13 - jepsen.util 733 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,225] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,226] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,226] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:16:47,227] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,228] jepsen worker 13 - jepsen.util 733 :invoke :write [4 13] INFO [2022-03-13 17:16:47,286] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,287] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:47,289] jepsen worker 24 - jepsen.util 564 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,289] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:16:47,296] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6198830409356726}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4117647058823529}} INFO [2022-03-13 17:16:47,296] jepsen worker nemesis - couchbase.util Rebalance check count: 515 INFO [2022-03-13 17:16:47,296] jepsen worker nemesis - couchbase.util Rebalance stuck count: 413 INFO [2022-03-13 17:16:47,300] jepsen worker 52 - jepsen.util 682 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,300] jepsen worker 52 - jepsen.util 682 :invoke :write [17 20] INFO [2022-03-13 17:16:47,426] jepsen worker 34 - jepsen.util 934 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,427] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,427] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,426] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,427] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:47,428] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:16:47,429] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:47,430] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,534] jepsen worker 13 - jepsen.util 733 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,534] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,534] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,535] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:47,535] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,536] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:47,536] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,537] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,538] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,538] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,540] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,540] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,542] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,542] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,544] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,544] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:47,546] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,546] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:16:47,564] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,565] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:47,669] jepsen worker 47 - jepsen.util 3557 :info :write [15 35] :SyncWriteAmbiguous INFO [2022-03-13 17:16:47,670] jepsen worker 47 - jepsen.util 3647 :invoke :read [15 nil] INFO [2022-03-13 17:16:47,678] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,679] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:47,719] jepsen worker 80 - jepsen.util 800 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,720] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:47,733] jepsen worker 52 - jepsen.util 682 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,733] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:47,735] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,735] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:16:47,834] jepsen worker 33 - jepsen.util 753 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,834] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,834] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,834] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,834] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,834] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:47,835] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,835] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,835] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:47,835] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,836] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,836] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:47,837] jepsen worker 52 - jepsen.util 682 :invoke :write [17 15] INFO [2022-03-13 17:16:47,838] jepsen worker 71 - jepsen.util 521 :invoke :write [23 42] INFO [2022-03-13 17:16:47,839] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:47,840] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:16:47,840] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:47,841] jepsen worker 14 - jepsen.util 374 :invoke :write [4 34] INFO [2022-03-13 17:16:47,843] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,847] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:47,848] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,849] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:47,850] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,850] jepsen worker 81 - jepsen.util 4131 :info :write [27 26] :RequestTimeoutException INFO [2022-03-13 17:16:47,851] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:47,851] jepsen worker 81 - jepsen.util 4221 :invoke :write [27 24] INFO [2022-03-13 17:16:47,852] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,852] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:16:47,878] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,878] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,878] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,879] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:16:47,880] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:47,880] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:47,882] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,882] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,882] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,882] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:47,882] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:16:47,883] jepsen worker 51 - jepsen.util 771 :invoke :write [17 31] INFO [2022-03-13 17:16:47,883] jepsen worker 34 - jepsen.util 934 :invoke :write [11 35] INFO [2022-03-13 17:16:47,884] jepsen worker 33 - jepsen.util 753 :invoke :write [11 38] INFO [2022-03-13 17:16:47,988] jepsen worker 73 - jepsen.util 3673 :info :write [24 26] :RequestTimeoutException INFO [2022-03-13 17:16:47,989] jepsen worker 73 - jepsen.util 3763 :invoke :write [24 45] INFO [2022-03-13 17:16:48,018] jepsen worker 14 - jepsen.util 374 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,019] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,019] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,019] jepsen worker 14 - jepsen.util 374 :invoke :write [4 21] INFO [2022-03-13 17:16:48,020] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:16:48,021] jepsen worker 80 - jepsen.util 800 :invoke :write [26 32] WARN [2022-03-13 17:16:48,107] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":27113041,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":35725194,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9169544,"encode_duration_us":18175,"operation_id":"0xdf4b","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10003263}],"total_count":1}} INFO [2022-03-13 17:16:48,135] jepsen worker 71 - jepsen.util 521 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,135] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,136] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:48,136] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:48,143] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,144] jepsen worker 13 - jepsen.util 733 :invoke :write [4 46] WARN [2022-03-13 17:16:48,218] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":20943377,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20943377,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9046921,"encode_duration_us":18359,"operation_id":"0xe52c","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10009970}],"total_count":1}} INFO [2022-03-13 17:16:48,254] jepsen worker 40 - jepsen.util 3730 :info :write [13 18] :RequestTimeoutException INFO [2022-03-13 17:16:48,254] jepsen worker 40 - jepsen.util 3820 :invoke :read [13 nil] INFO [2022-03-13 17:16:48,262] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,262] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:16:48,290] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,290] jepsen worker 24 - jepsen.util 564 :invoke :write [8 45] INFO [2022-03-13 17:16:48,297] jepsen worker 60 - jepsen.util 600 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,299] jepsen worker 60 - jepsen.util 600 :invoke :write [20 38] INFO [2022-03-13 17:16:48,301] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:48,301] jepsen worker nemesis - couchbase.util Rebalance check count: 516 INFO [2022-03-13 17:16:48,301] jepsen worker nemesis - couchbase.util Rebalance stuck count: 413 INFO [2022-03-13 17:16:48,308] jepsen worker 58 - jepsen.util 3298 :info :write [19 1] :RequestTimeoutException INFO [2022-03-13 17:16:48,309] jepsen worker 58 - jepsen.util 3388 :invoke :read [19 nil] WARN [2022-03-13 17:16:48,318] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11290688,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11290688,"total_server_duration_us":32,"operation_id":"0xe8a1","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11293234},{"operation_name":"upsert","last_dispatch_duration_us":8523279,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9172623,"total_server_duration_us":293903,"encode_duration_us":20149,"operation_id":"0xed63","timeout_ms":10000,"last_server_duration_us":29508,"total_duration_us":10006000},{"operation_name":"upsert","last_dispatch_duration_us":8528667,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13023725,"total_server_duration_us":117661,"encode_duration_us":18389,"operation_id":"0xeb3c","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10002887},{"operation_name":"upsert","last_dispatch_duration_us":323297,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":768170,"total_server_duration_us":96292,"encode_duration_us":18107,"operation_id":"0xed5f","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10002685},{"operation_name":"upsert","last_dispatch_duration_us":9168313,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":15176491,"total_server_duration_us":9067870,"encode_duration_us":21183,"operation_id":"0xe9ef","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10002392},{"operation_name":"upsert","last_dispatch_duration_us":9060930,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9108186,"total_server_duration_us":9052468,"encode_duration_us":33564,"operation_id":"0xeeae","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":9129961},{"operation_name":"get","last_dispatch_duration_us":4709349,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":4709349,"total_server_duration_us":84,"operation_id":"0xec30","timeout_ms":30000,"last_server_duration_us":84,"total_duration_us":4709875},{"operation_name":"upsert","last_dispatch_duration_us":4163541,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4163541,"total_server_duration_us":58128,"encode_duration_us":25400,"operation_id":"0xecbf","timeout_ms":10000,"last_server_duration_us":58128,"total_duration_us":4167651},{"operation_name":"upsert","last_dispatch_duration_us":3256854,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3256854,"total_server_duration_us":49603,"encode_duration_us":20851,"operation_id":"0xf0cd","timeout_ms":10000,"last_server_duration_us":49603,"total_duration_us":3264656},{"operation_name":"upsert","last_dispatch_duration_us":3065376,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":3065376,"total_server_duration_us":67882,"encode_duration_us":19377,"operation_id":"0xf0c9","timeout_ms":10000,"last_server_duration_us":67882,"total_duration_us":3070273}],"total_count":22}} WARN [2022-03-13 17:16:48,334] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":27606197,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":27606197,"total_server_duration_us":416106,"operation_id":"0xe10e","timeout_ms":30000,"last_server_duration_us":416106,"total_duration_us":27609088},{"operation_name":"get","last_dispatch_duration_us":27206194,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":27206194,"total_server_duration_us":6628,"operation_id":"0xe105","timeout_ms":30000,"last_server_duration_us":6628,"total_duration_us":27217502},{"operation_name":"get","last_dispatch_duration_us":21095825,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":21095825,"total_server_duration_us":38,"operation_id":"0xe4b8","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":21098519},{"operation_name":"upsert","last_dispatch_duration_us":48222,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9220527,"total_server_duration_us":91869,"encode_duration_us":20450,"operation_id":"0xeb31","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10009168},{"operation_name":"upsert","last_dispatch_duration_us":188278,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10060598,"total_server_duration_us":163732,"encode_duration_us":29630,"operation_id":"0xeb5f","timeout_ms":10000,"last_server_duration_us":27043,"total_duration_us":10006326},{"operation_name":"upsert","last_dispatch_duration_us":8944004,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8944004,"total_server_duration_us":26507,"encode_duration_us":18435,"operation_id":"0xea9e","timeout_ms":10000,"last_server_duration_us":26507,"total_duration_us":10005289},{"operation_name":"upsert","last_dispatch_duration_us":43779,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9355821,"total_server_duration_us":134794,"encode_duration_us":28069,"operation_id":"0xeb30","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10002225},{"operation_name":"get","last_dispatch_duration_us":7285660,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7285660,"total_server_duration_us":43,"operation_id":"0xec6f","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":7287009},{"operation_name":"get","last_dispatch_duration_us":2877999,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":2877999,"total_server_duration_us":8123,"operation_id":"0xeeff","timeout_ms":30000,"last_server_duration_us":8123,"total_duration_us":2881715},{"operation_name":"upsert","last_dispatch_duration_us":175485,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1978411,"total_server_duration_us":687863,"encode_duration_us":21110,"operation_id":"0xef25","timeout_ms":10000,"last_server_duration_us":54094,"total_duration_us":2578610}],"total_count":25}} WARN [2022-03-13 17:16:48,334] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":29411061,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":29411061,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9041612,"encode_duration_us":18471,"operation_id":"0xdfa8","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10006466},{"operation_name":"upsert","last_dispatch_duration_us":23609802,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":23609802,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9037366,"encode_duration_us":31535,"operation_id":"0xe2e2","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10006690}],"total_count":2}} INFO [2022-03-13 17:16:48,343] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,344] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] WARN [2022-03-13 17:16:48,409] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":507994,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1161749,"total_server_duration_us":301257,"encode_duration_us":22150,"operation_id":"0xed53","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10010051},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":19818,"operation_id":"0xee7b","timeout_ms":10000,"total_duration_us":10002829},{"operation_name":"upsert","last_dispatch_duration_us":164708,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8911713,"total_server_duration_us":259674,"encode_duration_us":16752,"operation_id":"0xe961","timeout_ms":10000,"last_server_duration_us":22720,"total_duration_us":10001639},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":18862,"operation_id":"0xec04","timeout_ms":10000,"total_duration_us":10001185},{"operation_name":"upsert","last_dispatch_duration_us":218321,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":990733,"total_server_duration_us":305214,"encode_duration_us":17721,"operation_id":"0xeb27","timeout_ms":10000,"last_server_duration_us":69615,"total_duration_us":9755419},{"operation_name":"upsert","last_dispatch_duration_us":1404015,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1404015,"total_server_duration_us":176294,"encode_duration_us":18008,"operation_id":"0xedf5","timeout_ms":10000,"last_server_duration_us":176294,"total_duration_us":5042259},{"operation_name":"get","last_dispatch_duration_us":1196831,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1196831,"total_server_duration_us":23,"operation_id":"0xedf3","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":4828589},{"operation_name":"upsert","last_dispatch_duration_us":220132,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":2056441,"total_server_duration_us":783212,"encode_duration_us":18957,"operation_id":"0xef24","timeout_ms":10000,"last_server_duration_us":62668,"total_duration_us":3193876},{"operation_name":"upsert","last_dispatch_duration_us":1344287,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1344287,"total_server_duration_us":67485,"encode_duration_us":26497,"operation_id":"0xeddf","timeout_ms":10000,"last_server_duration_us":67485,"total_duration_us":2433195},{"operation_name":"get","last_dispatch_duration_us":1186891,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1186891,"total_server_duration_us":27,"operation_id":"0xef5a","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":2275547}],"total_count":60}} INFO [2022-03-13 17:16:48,416] jepsen worker 52 - jepsen.util 682 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,417] jepsen worker 52 - jepsen.util 682 :invoke :write [17 47] INFO [2022-03-13 17:16:48,437] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,437] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,437] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:16:48,438] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:48,447] jepsen worker 13 - jepsen.util 733 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,448] jepsen worker 13 - jepsen.util 733 :invoke :write [4 44] INFO [2022-03-13 17:16:48,449] jepsen worker 7 - jepsen.util 637 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,449] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:16:48,490] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,491] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] WARN [2022-03-13 17:16:48,518] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":92072,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9002152,"total_server_duration_us":241639,"encode_duration_us":18293,"operation_id":"0xebbb","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10014971},{"operation_name":"upsert","last_dispatch_duration_us":9294075,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12259407,"total_server_duration_us":9712778,"encode_duration_us":16732,"operation_id":"0xe951","timeout_ms":10000,"last_server_duration_us":9274292,"total_duration_us":10010429},{"operation_name":"upsert","last_dispatch_duration_us":59197,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":115484,"total_server_duration_us":67079,"encode_duration_us":18274,"operation_id":"0xedfa","timeout_ms":10000,"last_server_duration_us":47192,"total_duration_us":10008197},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18725,"operation_id":"0xeb88","timeout_ms":10000,"total_duration_us":10006367},{"operation_name":"upsert","last_dispatch_duration_us":8886040,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8886040,"total_server_duration_us":24239,"encode_duration_us":17965,"operation_id":"0xeaa0","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10002668},{"operation_name":"get","last_dispatch_duration_us":403780,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":403780,"total_server_duration_us":402727,"operation_id":"0xecc6","timeout_ms":30000,"last_server_duration_us":402727,"total_duration_us":7612703},{"operation_name":"upsert","last_dispatch_duration_us":2997731,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2997731,"total_server_duration_us":91081,"encode_duration_us":22791,"operation_id":"0xf04f","timeout_ms":10000,"last_server_duration_us":91081,"total_duration_us":3004035},{"operation_name":"upsert","last_dispatch_duration_us":2114161,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2114161,"total_server_duration_us":50184,"encode_duration_us":22135,"operation_id":"0xed9b","timeout_ms":10000,"last_server_duration_us":50184,"total_duration_us":2117830},{"operation_name":"upsert","last_dispatch_duration_us":1899870,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1899870,"total_server_duration_us":193546,"encode_duration_us":18707,"operation_id":"0xef7b","timeout_ms":10000,"last_server_duration_us":193546,"total_duration_us":1903615},{"operation_name":"upsert","last_dispatch_duration_us":962746,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":962746,"total_server_duration_us":131690,"encode_duration_us":24786,"operation_id":"0xeed3","timeout_ms":10000,"last_server_duration_us":131690,"total_duration_us":966816}],"total_count":13}} WARN [2022-03-13 17:16:48,518] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14954490,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":14954490,"total_server_duration_us":289103,"operation_id":"0xe8c4","timeout_ms":30000,"last_server_duration_us":289103,"total_duration_us":14957918},{"operation_name":"upsert","last_dispatch_duration_us":33515,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":744623,"total_server_duration_us":295748,"encode_duration_us":17115,"operation_id":"0xee82","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10010285},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":24671,"operation_id":"0xec43","timeout_ms":10000,"total_duration_us":10009997},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":22031,"operation_id":"0xeb55","timeout_ms":10000,"total_duration_us":10004986},{"operation_name":"upsert","last_dispatch_duration_us":9486388,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9486388,"total_server_duration_us":21324,"encode_duration_us":18193,"operation_id":"0xebc3","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10003465},{"operation_name":"get","last_dispatch_duration_us":9496901,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9496901,"total_server_duration_us":12569,"operation_id":"0xebb2","timeout_ms":30000,"last_server_duration_us":12569,"total_duration_us":9500425},{"operation_name":"get","last_dispatch_duration_us":9475445,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9475445,"total_server_duration_us":29881,"operation_id":"0xebb6","timeout_ms":30000,"last_server_duration_us":29881,"total_duration_us":9477254},{"operation_name":"upsert","last_dispatch_duration_us":306456,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2543232,"total_server_duration_us":466730,"encode_duration_us":17821,"operation_id":"0xef23","timeout_ms":10000,"last_server_duration_us":41337,"total_duration_us":2751791},{"operation_name":"upsert","last_dispatch_duration_us":274258,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2448466,"total_server_duration_us":431069,"encode_duration_us":19406,"operation_id":"0xef22","timeout_ms":10000,"last_server_duration_us":43838,"total_duration_us":2666008},{"operation_name":"upsert","last_dispatch_duration_us":1828715,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1828715,"total_server_duration_us":60380,"encode_duration_us":29587,"operation_id":"0xf0ac","timeout_ms":10000,"last_server_duration_us":60380,"total_duration_us":1832474}],"total_count":57}} WARN [2022-03-13 17:16:48,518] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17793791,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27600726,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9121712,"encode_duration_us":18511,"operation_id":"0xe0f8","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10000919},{"operation_name":"upsert","last_dispatch_duration_us":17829971,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17972293,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9076628,"encode_duration_us":28760,"operation_id":"0xe685","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10003571},{"operation_name":"upsert","last_dispatch_duration_us":9294075,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12259407,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9712778,"encode_duration_us":16732,"operation_id":"0xe951","timeout_ms":10000,"last_server_duration_us":9274292,"total_duration_us":10010429}],"total_count":3}} WARN [2022-03-13 17:16:48,519] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9168313,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15176491,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9067870,"encode_duration_us":21183,"operation_id":"0xe9ef","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10002392},{"operation_name":"upsert","last_dispatch_duration_us":17572918,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18189052,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9448519,"encode_duration_us":28209,"operation_id":"0xe2ac","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10009978},{"operation_name":"upsert","last_dispatch_duration_us":17534151,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18390140,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9420529,"encode_duration_us":18073,"operation_id":"0xe19d","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10002398}],"total_count":3}} INFO [2022-03-13 17:16:48,532] jepsen worker 52 - jepsen.util 682 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,533] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:48,538] jepsen worker 51 - jepsen.util 771 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,539] jepsen worker 51 - jepsen.util 771 :invoke :write [17 45] INFO [2022-03-13 17:16:48,545] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,545] jepsen worker 24 - jepsen.util 564 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,546] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:48,546] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:16:48,550] jepsen worker 63 - jepsen.util 3663 :info :write [21 24] :RequestTimeoutException INFO [2022-03-13 17:16:48,550] jepsen worker 63 - jepsen.util 3753 :invoke :write [21 44] INFO [2022-03-13 17:16:48,574] jepsen worker 13 - jepsen.util 733 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,574] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:48,583] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,584] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:48,585] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,585] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:16:48,594] jepsen worker 34 - jepsen.util 934 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,595] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:48,677] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,677] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,678] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,678] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,678] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:48,679] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:48,679] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:48,680] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,680] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:48,681] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,681] jepsen worker 52 - jepsen.util 682 :invoke :write [17 27] INFO [2022-03-13 17:16:48,682] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:16:48,703] jepsen worker 33 - jepsen.util 753 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,704] jepsen worker 33 - jepsen.util 753 :invoke :write [11 44] INFO [2022-03-13 17:16:48,753] jepsen worker 16 - jepsen.util 3436 :info :write [5 37] :RequestTimeoutException INFO [2022-03-13 17:16:48,754] jepsen worker 16 - jepsen.util 3526 :invoke :read [5 nil] INFO [2022-03-13 17:16:48,808] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,808] jepsen worker 13 - jepsen.util 733 :invoke :write [4 4] INFO [2022-03-13 17:16:48,864] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,865] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:48,901] jepsen worker 55 - jepsen.util 3385 :ok :read [18 21] INFO [2022-03-13 17:16:48,901] jepsen worker 55 - jepsen.util 3385 :invoke :read [18 nil] INFO [2022-03-13 17:16:48,953] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,953] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,953] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,954] jepsen worker 24 - jepsen.util 564 :invoke :write [8 35] INFO [2022-03-13 17:16:48,954] jepsen worker 71 - jepsen.util 521 :invoke :write [23 41] INFO [2022-03-13 17:16:48,955] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:48,957] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,957] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:48,958] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,959] jepsen worker 75 - jepsen.util 435 :invoke :write [25 34] INFO [2022-03-13 17:16:48,961] jepsen worker 52 - jepsen.util 682 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,961] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:48,977] jepsen worker 80 - jepsen.util 800 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:48,978] jepsen worker 80 - jepsen.util 800 :invoke :write [26 31] INFO [2022-03-13 17:16:49,029] jepsen worker 14 - jepsen.util 374 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,050] jepsen worker 46 - jepsen.util 3736 :info :write [15 22] :RequestTimeoutException INFO [2022-03-13 17:16:49,380] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:49,380] jepsen worker 46 - jepsen.util 3826 :invoke :write [15 37] INFO [2022-03-13 17:16:49,102] jepsen worker 71 - jepsen.util 521 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,111] jepsen worker 51 - jepsen.util 771 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,382] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:49,382] jepsen worker 51 - jepsen.util 771 :invoke :write [17 42] INFO [2022-03-13 17:16:49,111] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,383] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:49,111] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,111] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,383] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:16:49,113] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,383] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:49,383] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:49,113] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,158] jepsen worker 24 - jepsen.util 564 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,384] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:49,249] jepsen worker 13 - jepsen.util 733 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,384] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:49,384] jepsen worker 13 - jepsen.util 733 :invoke :write [4 17] INFO [2022-03-13 17:16:49,249] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,384] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:16:49,249] jepsen worker 33 - jepsen.util 753 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,279] jepsen worker 75 - jepsen.util 435 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,390] jepsen worker 33 - jepsen.util 753 :invoke :write [11 33] INFO [2022-03-13 17:16:49,307] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:49,390] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:49,390] jepsen worker nemesis - couchbase.util Rebalance check count: 517 INFO [2022-03-13 17:16:49,390] jepsen worker nemesis - couchbase.util Rebalance stuck count: 414 INFO [2022-03-13 17:16:49,396] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,396] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,396] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:16:49,397] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:49,402] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,402] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,402] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,403] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,403] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:49,404] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,404] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:16:49,405] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,407] jepsen worker 34 - jepsen.util 934 :invoke :write [11 37] INFO [2022-03-13 17:16:49,408] jepsen worker 75 - jepsen.util 435 :invoke :write [25 26] INFO [2022-03-13 17:16:49,408] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,409] jepsen worker 24 - jepsen.util 564 :invoke :write [8 43] INFO [2022-03-13 17:16:49,414] jepsen worker 52 - jepsen.util 682 :invoke :write [17 19] INFO [2022-03-13 17:16:49,421] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:49,423] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,423] jepsen worker 71 - jepsen.util 521 :invoke :write [23 5] INFO [2022-03-13 17:16:49,482] jepsen worker 27 - jepsen.util 3897 :ok :read [9 26] INFO [2022-03-13 17:16:49,482] jepsen worker 27 - jepsen.util 3897 :invoke :read [9 nil] INFO [2022-03-13 17:16:49,490] jepsen worker 80 - jepsen.util 800 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,490] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:49,534] jepsen worker 24 - jepsen.util 564 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,534] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:49,553] jepsen worker 13 - jepsen.util 733 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,554] jepsen worker 13 - jepsen.util 733 :invoke :write [4 15] INFO [2022-03-13 17:16:49,680] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,680] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,681] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:16:49,681] jepsen worker 14 - jepsen.util 374 :invoke :write [4 20] INFO [2022-03-13 17:16:49,682] jepsen worker 71 - jepsen.util 521 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,683] jepsen worker 71 - jepsen.util 521 :invoke :write [23 44] INFO [2022-03-13 17:16:49,701] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,701] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:49,714] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,714] jepsen worker 75 - jepsen.util 435 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,715] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:49,715] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:49,857] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,857] jepsen worker 71 - jepsen.util 521 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,857] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,858] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:16:49,859] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:49,860] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:49,860] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,861] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,861] jepsen worker 24 - jepsen.util 564 :invoke :write [8 5] INFO [2022-03-13 17:16:49,861] jepsen worker 75 - jepsen.util 435 :invoke :write [25 29] INFO [2022-03-13 17:16:49,889] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,891] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:49,992] jepsen worker 52 - jepsen.util 682 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,992] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:49,993] jepsen worker 52 - jepsen.util 682 :invoke :write [17 48] INFO [2022-03-13 17:16:49,993] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:50,019] jepsen worker 13 - jepsen.util 733 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,019] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,020] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:50,020] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:50,026] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,027] jepsen worker 13 - jepsen.util 733 :invoke :write [4 44] INFO [2022-03-13 17:16:50,028] jepsen worker 33 - jepsen.util 753 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,029] jepsen worker 33 - jepsen.util 753 :invoke :write [11 7] INFO [2022-03-13 17:16:50,030] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,031] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:50,047] jepsen worker 83 - jepsen.util 3233 :info :write [27 1] :RequestTimeoutException INFO [2022-03-13 17:16:50,048] jepsen worker 83 - jepsen.util 3323 :invoke :write [27 12] INFO [2022-03-13 17:16:50,137] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,137] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:16:50,169] jepsen worker 51 - jepsen.util 771 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,170] jepsen worker 51 - jepsen.util 771 :invoke :write [17 45] INFO [2022-03-13 17:16:50,284] jepsen worker 52 - jepsen.util 682 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,284] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,285] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:50,285] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:50,286] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,286] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,287] jepsen worker 52 - jepsen.util 682 :invoke :write [17 31] INFO [2022-03-13 17:16:50,287] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:16:50,312] jepsen worker 13 - jepsen.util 733 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,312] jepsen worker 13 - jepsen.util 733 :invoke :write [4 26] INFO [2022-03-13 17:16:50,347] jepsen worker 52 - jepsen.util 682 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,348] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:50,395] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:50,395] jepsen worker nemesis - couchbase.util Rebalance check count: 518 INFO [2022-03-13 17:16:50,395] jepsen worker nemesis - couchbase.util Rebalance stuck count: 415 INFO [2022-03-13 17:16:50,438] jepsen worker 75 - jepsen.util 435 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,439] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:50,444] jepsen worker 34 - jepsen.util 934 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,445] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:50,446] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,447] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,447] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:16:50,448] jepsen worker 52 - jepsen.util 682 :invoke :write [17 37] INFO [2022-03-13 17:16:50,567] jepsen worker 13 - jepsen.util 733 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,568] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:50,579] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,580] jepsen worker 13 - jepsen.util 733 :invoke :write [4 46] INFO [2022-03-13 17:16:50,606] jepsen worker 52 - jepsen.util 682 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,607] jepsen worker 52 - jepsen.util 682 :invoke :write [17 17] INFO [2022-03-13 17:16:50,712] jepsen worker 24 - jepsen.util 564 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,713] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:50,722] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,723] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:16:50,731] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,731] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:16:50,783] jepsen worker 71 - jepsen.util 521 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,783] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,783] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,784] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:50,785] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:50,785] jepsen worker 24 - jepsen.util 564 :invoke :write [8 29] INFO [2022-03-13 17:16:50,797] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,797] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,798] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:50,798] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:50,799] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,800] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,800] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:50,800] jepsen worker 75 - jepsen.util 435 :invoke :write [25 40] INFO [2022-03-13 17:16:50,801] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,802] jepsen worker 71 - jepsen.util 521 :invoke :write [23 46] INFO [2022-03-13 17:16:50,813] jepsen worker 13 - jepsen.util 733 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,814] jepsen worker 13 - jepsen.util 733 :invoke :write [4 20] INFO [2022-03-13 17:16:50,830] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,830] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,831] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:16:50,831] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:16:50,843] jepsen worker 2 - jepsen.util 3962 :info :write [0 29] :SyncWriteAmbiguous INFO [2022-03-13 17:16:50,845] jepsen worker 2 - jepsen.util 4052 :invoke :read [0 nil] INFO [2022-03-13 17:16:50,870] jepsen worker 1 - jepsen.util 3241 :ok :read [0 41] INFO [2022-03-13 17:16:50,871] jepsen worker 1 - jepsen.util 3241 :invoke :write [0 23] INFO [2022-03-13 17:16:50,886] jepsen worker 24 - jepsen.util 564 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,888] jepsen worker 24 - jepsen.util 564 :invoke :write [8 37] INFO [2022-03-13 17:16:50,968] jepsen worker 52 - jepsen.util 682 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,968] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:50,975] jepsen worker 14 - jepsen.util 374 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:50,975] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:51,029] jepsen worker 78 - jepsen.util 618 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:51,029] jepsen worker 78 - jepsen.util 618 :invoke :write [26 6] INFO [2022-03-13 17:16:51,041] jepsen worker 18 - jepsen.util 3708 :ok :read [6 16] INFO [2022-03-13 17:16:51,041] jepsen worker 18 - jepsen.util 3708 :invoke :write [6 48] INFO [2022-03-13 17:16:51,100] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:51,100] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:51,100] jepsen worker 33 - jepsen.util 753 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:51,101] jepsen worker 52 - jepsen.util 682 :invoke :write [17 5] INFO [2022-03-13 17:16:51,102] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:51,102] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:51,104] jepsen worker 75 - jepsen.util 435 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:51,104] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:16:51,236] jepsen worker 68 - jepsen.util 3668 :info :write [22 11] :RequestTimeoutException INFO [2022-03-13 17:16:51,237] jepsen worker 68 - jepsen.util 3758 :invoke :write [22 39] INFO [2022-03-13 17:16:51,241] jepsen worker 64 - jepsen.util 3574 :info :write [21 29] :RequestTimeoutException INFO [2022-03-13 17:16:51,241] jepsen worker 64 - jepsen.util 3664 :invoke :read [21 nil] INFO [2022-03-13 17:16:51,323] jepsen worker 64 - jepsen.util 3664 :ok :read [21 :nil] INFO [2022-03-13 17:16:51,324] jepsen worker 64 - jepsen.util 3664 :invoke :read [21 nil] INFO [2022-03-13 17:16:51,390] jepsen worker 64 - jepsen.util 3664 :ok :read [21 :nil] INFO [2022-03-13 17:16:51,391] jepsen worker 64 - jepsen.util 3664 :invoke :read [21 nil] INFO [2022-03-13 17:16:51,400] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:51,400] jepsen worker nemesis - couchbase.util Rebalance check count: 519 INFO [2022-03-13 17:16:51,400] jepsen worker nemesis - couchbase.util Rebalance stuck count: 416 INFO [2022-03-13 17:16:51,454] jepsen worker 64 - jepsen.util 3664 :ok :read [21 :nil] INFO [2022-03-13 17:16:51,455] jepsen worker 64 - jepsen.util 3664 :invoke :write [21 4] INFO [2022-03-13 17:16:51,567] jepsen worker 30 - jepsen.util 3540 :ok :read [10 :nil] INFO [2022-03-13 17:16:51,567] jepsen worker 30 - jepsen.util 3540 :invoke :read [10 nil] INFO [2022-03-13 17:16:51,666] jepsen worker 41 - jepsen.util 3551 :info :write [13 5] :RequestTimeoutException INFO [2022-03-13 17:16:51,666] jepsen worker 41 - jepsen.util 3641 :invoke :write [13 26] INFO [2022-03-13 17:16:51,670] jepsen worker 74 - jepsen.util 3764 :ok :read [24 24] INFO [2022-03-13 17:16:51,670] jepsen worker 74 - jepsen.util 3764 :invoke :write [24 33] INFO [2022-03-13 17:16:51,970] jepsen worker 45 - jepsen.util 3825 :info :write [15 47] :RequestTimeoutException INFO [2022-03-13 17:16:51,971] jepsen worker 45 - jepsen.util 3915 :invoke :write [15 0] INFO [2022-03-13 17:16:51,976] jepsen worker 59 - jepsen.util 3659 :info :write [19 14] :RequestTimeoutException INFO [2022-03-13 17:16:51,977] jepsen worker 59 - jepsen.util 3749 :invoke :read [19 nil] INFO [2022-03-13 17:16:52,144] jepsen worker 54 - jepsen.util 3294 :info :write [18 3] :RequestTimeoutException INFO [2022-03-13 17:16:52,144] jepsen worker 54 - jepsen.util 3384 :invoke :write [18 27] INFO [2022-03-13 17:16:52,336] jepsen worker 15 - jepsen.util 3615 :info :write [5 8] :RequestTimeoutException INFO [2022-03-13 17:16:52,337] jepsen worker 15 - jepsen.util 3705 :invoke :read [5 nil] INFO [2022-03-13 17:16:52,405] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:52,405] jepsen worker nemesis - couchbase.util Rebalance check count: 520 INFO [2022-03-13 17:16:52,406] jepsen worker nemesis - couchbase.util Rebalance stuck count: 417 INFO [2022-03-13 17:16:52,476] jepsen worker 32 - jepsen.util 3722 :info :write [10 13] :RequestTimeoutException INFO [2022-03-13 17:16:52,476] jepsen worker 32 - jepsen.util 3812 :invoke :write [10 3] INFO [2022-03-13 17:16:52,486] jepsen worker 65 - jepsen.util 4115 :info :write [21 12] :RequestTimeoutException INFO [2022-03-13 17:16:52,487] jepsen worker 65 - jepsen.util 4205 :invoke :read [21 nil] INFO [2022-03-13 17:16:52,503] jepsen worker 10 - jepsen.util 3610 :info :write [3 30] :RequestTimeoutException INFO [2022-03-13 17:16:52,504] jepsen worker 10 - jepsen.util 3700 :invoke :read [3 nil] INFO [2022-03-13 17:16:52,531] jepsen worker 17 - jepsen.util 3707 :info :write [5 1] :RequestTimeoutException INFO [2022-03-13 17:16:52,531] jepsen worker 17 - jepsen.util 3797 :invoke :write [5 22] INFO [2022-03-13 17:16:52,594] jepsen worker 20 - jepsen.util 3890 :info :write [6 23] :RequestTimeoutException INFO [2022-03-13 17:16:52,594] jepsen worker 20 - jepsen.util 3980 :invoke :read [6 nil] INFO [2022-03-13 17:16:52,738] jepsen worker 19 - jepsen.util 3349 :info :write [6 27] :RequestTimeoutException INFO [2022-03-13 17:16:52,739] jepsen worker 19 - jepsen.util 3439 :invoke :write [6 20] INFO [2022-03-13 17:16:52,754] jepsen worker 13 - jepsen.util 733 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,755] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,755] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:52,755] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:52,757] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,758] jepsen worker 13 - jepsen.util 733 :invoke :write [4 41] INFO [2022-03-13 17:16:52,760] jepsen worker 26 - jepsen.util 476 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,760] jepsen worker 26 - jepsen.util 476 :invoke :write [8 22] INFO [2022-03-13 17:16:52,796] jepsen worker 51 - jepsen.util 771 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,796] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:52,796] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,796] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,797] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:52,798] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:52,853] jepsen worker 71 - jepsen.util 521 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,854] jepsen worker 71 - jepsen.util 521 :invoke :write [23 14] INFO [2022-03-13 17:16:52,861] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,862] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:16:52,912] jepsen worker 52 - jepsen.util 682 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,913] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:16:52,914] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,914] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:16:52,914] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,915] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:16:52,967] jepsen worker 24 - jepsen.util 564 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:52,967] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:16:53,003] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,003] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,004] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,004] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,005] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:16:53,006] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,006] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,007] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,010] jepsen worker 13 - jepsen.util 733 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,011] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,011] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:53,012] jepsen worker 33 - jepsen.util 753 :invoke :write [11 22] INFO [2022-03-13 17:16:53,012] jepsen worker 34 - jepsen.util 934 :invoke :write [11 3] INFO [2022-03-13 17:16:53,013] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:53,064] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,064] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,068] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,068] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,069] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,070] jepsen worker 13 - jepsen.util 733 :invoke :write [4 10] INFO [2022-03-13 17:16:53,073] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,073] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:53,113] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,114] jepsen worker 21 - jepsen.util 831 :invoke :write [7 16] INFO [2022-03-13 17:16:53,124] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,125] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:53,163] jepsen worker 71 - jepsen.util 521 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,164] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:53,212] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,213] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,213] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,212] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,213] jepsen worker 80 - jepsen.util 800 :invoke :write [26 36] INFO [2022-03-13 17:16:53,214] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,215] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:53,215] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:53,234] jepsen worker 56 - jepsen.util 3746 :info :write [18 19] :RequestTimeoutException INFO [2022-03-13 17:16:53,235] jepsen worker 56 - jepsen.util 3836 :invoke :read [18 nil] INFO [2022-03-13 17:16:53,300] jepsen worker 66 - jepsen.util 3396 :ok :read [22 30] INFO [2022-03-13 17:16:53,301] jepsen worker 66 - jepsen.util 3396 :invoke :read [22 nil] INFO [2022-03-13 17:16:53,303] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,304] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:53,311] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,311] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,311] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,312] jepsen worker 24 - jepsen.util 564 :invoke :write [8 2] INFO [2022-03-13 17:16:53,314] jepsen worker 71 - jepsen.util 521 :invoke :write [23 25] INFO [2022-03-13 17:16:53,315] jepsen worker 75 - jepsen.util 435 :invoke :write [25 35] INFO [2022-03-13 17:16:53,336] jepsen worker 9 - jepsen.util 3069 :info :write [3 36] :RequestTimeoutException INFO [2022-03-13 17:16:53,336] jepsen worker 9 - jepsen.util 3159 :invoke :write [3 29] INFO [2022-03-13 17:16:53,389] jepsen worker 86 - jepsen.util 3506 :ok :read [28 44] INFO [2022-03-13 17:16:53,389] jepsen worker 86 - jepsen.util 3506 :invoke :write [28 20] INFO [2022-03-13 17:16:53,392] jepsen worker 33 - jepsen.util 753 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,392] jepsen worker 33 - jepsen.util 753 :invoke :write [11 22] INFO [2022-03-13 17:16:53,405] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,405] jepsen worker 13 - jepsen.util 733 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,405] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,406] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:53,407] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,408] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:53,409] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,410] jepsen worker 52 - jepsen.util 682 :invoke :write [17 9] INFO [2022-03-13 17:16:53,410] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,413] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:53,413] jepsen worker nemesis - couchbase.util Rebalance check count: 521 INFO [2022-03-13 17:16:53,413] jepsen worker nemesis - couchbase.util Rebalance stuck count: 418 INFO [2022-03-13 17:16:53,415] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:16:53,417] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,424] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,426] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,427] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:16:53,430] jepsen worker 24 - jepsen.util 564 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,430] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:53,432] jepsen worker 66 - jepsen.util 3396 :ok :read [22 30] INFO [2022-03-13 17:16:53,434] jepsen worker 66 - jepsen.util 3396 :invoke :write [22 32] INFO [2022-03-13 17:16:53,460] jepsen worker 37 - jepsen.util 2107 :info :write [12 26] :RequestTimeoutException INFO [2022-03-13 17:16:53,461] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:53,464] jepsen worker 36 - jepsen.util 2196 :info :write [12 37] :RequestTimeoutException INFO [2022-03-13 17:16:53,464] jepsen worker 89 - jepsen.util 2519 :info :write [29 45] :RequestTimeoutException INFO [2022-03-13 17:16:53,464] jepsen worker 71 - jepsen.util 521 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,464] jepsen worker 36 - jepsen.util 2286 :invoke :read [12 nil] INFO [2022-03-13 17:16:53,465] jepsen worker 89 - jepsen.util 2609 :invoke :write [29 4] INFO [2022-03-13 17:16:53,466] jepsen worker 71 - jepsen.util 521 :invoke :write [23 27] INFO [2022-03-13 17:16:53,470] jepsen worker 59 - jepsen.util 3749 :ok :read [19 16] INFO [2022-03-13 17:16:53,471] jepsen worker 59 - jepsen.util 3749 :invoke :read [19 nil] INFO [2022-03-13 17:16:53,485] jepsen worker 36 - jepsen.util 2286 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,486] jepsen worker 36 - jepsen.util 2286 :invoke :write [12 22] INFO [2022-03-13 17:16:53,507] jepsen worker 34 - jepsen.util 934 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,508] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:53,531] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,531] jepsen worker 37 - jepsen.util 2197 :invoke :write [12 36] INFO [2022-03-13 17:16:53,539] jepsen worker 75 - jepsen.util 435 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,539] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,540] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:53,541] jepsen worker 24 - jepsen.util 564 :invoke :write [8 3] INFO [2022-03-13 17:16:53,588] jepsen worker 59 - jepsen.util 3749 :ok :read [19 16] INFO [2022-03-13 17:16:53,588] jepsen worker 59 - jepsen.util 3749 :invoke :write [19 20] INFO [2022-03-13 17:16:53,610] jepsen worker 88 - jepsen.util 2068 :info :write [29 15] :RequestTimeoutException INFO [2022-03-13 17:16:53,611] jepsen worker 88 - jepsen.util 2158 :invoke :read [29 nil] INFO [2022-03-13 17:16:53,616] jepsen worker 52 - jepsen.util 682 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,616] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:16:53,620] jepsen worker 87 - jepsen.util 2607 :info :write [29 23] :RequestTimeoutException INFO [2022-03-13 17:16:53,621] jepsen worker 4 - jepsen.util 2254 :info :write [1 32] :RequestTimeoutException INFO [2022-03-13 17:16:53,621] jepsen worker 87 - jepsen.util 2697 :invoke :read [29 nil] INFO [2022-03-13 17:16:53,621] jepsen worker 4 - jepsen.util 2344 :invoke :read [1 nil] INFO [2022-03-13 17:16:53,624] jepsen worker 21 - jepsen.util 831 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,624] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,624] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,625] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:16:53,625] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,626] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:53,626] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:16:53,627] jepsen worker 29 - jepsen.util 3899 :info :write [9 6] :RequestTimeoutException INFO [2022-03-13 17:16:53,627] jepsen worker 14 - jepsen.util 374 :invoke :write [4 31] INFO [2022-03-13 17:16:53,627] jepsen worker 29 - jepsen.util 3989 :invoke :read [9 nil] INFO [2022-03-13 17:16:53,648] jepsen worker 3 - jepsen.util 1983 :info :write [1 28] :RequestTimeoutException INFO [2022-03-13 17:16:53,649] jepsen worker 3 - jepsen.util 2073 :invoke :write [1 39] INFO [2022-03-13 17:16:53,656] jepsen worker 5 - jepsen.util 2435 :info :write [1 36] :RequestTimeoutException INFO [2022-03-13 17:16:53,657] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 9] INFO [2022-03-13 17:16:53,665] jepsen worker 87 - jepsen.util 2697 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,666] jepsen worker 87 - jepsen.util 2697 :invoke :read [29 nil] INFO [2022-03-13 17:16:53,668] jepsen worker 71 - jepsen.util 521 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,669] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,671] jepsen worker 75 - jepsen.util 435 :invoke :write [25 22] INFO [2022-03-13 17:16:53,671] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:53,674] jepsen worker 87 - jepsen.util 2697 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,675] jepsen worker 87 - jepsen.util 2697 :invoke :read [29 nil] INFO [2022-03-13 17:16:53,679] jepsen worker 80 - jepsen.util 800 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,679] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,680] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:53,680] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:16:53,687] jepsen worker 87 - jepsen.util 2697 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,688] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 41] INFO [2022-03-13 17:16:53,703] jepsen worker 88 - jepsen.util 2158 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,703] jepsen worker 88 - jepsen.util 2158 :invoke :write [29 2] INFO [2022-03-13 17:16:53,704] jepsen worker 4 - jepsen.util 2344 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,705] jepsen worker 4 - jepsen.util 2344 :invoke :write [1 5] INFO [2022-03-13 17:16:53,741] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,742] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,744] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,744] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,745] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,746] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:53,747] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:53,748] jepsen worker 13 - jepsen.util 733 :invoke :write [4 19] INFO [2022-03-13 17:16:53,770] jepsen worker 38 - jepsen.util 2468 :info :write [12 45] :RequestTimeoutException INFO [2022-03-13 17:16:53,771] jepsen worker 38 - jepsen.util 2558 :invoke :write [12 5] INFO [2022-03-13 17:16:54,170] jepsen worker 57 - jepsen.util 3657 :info :write [19 46] :RequestTimeoutException INFO [2022-03-13 17:16:54,171] jepsen worker 57 - jepsen.util 3747 :invoke :read [19 nil] INFO [2022-03-13 17:16:54,354] jepsen worker 16 - jepsen.util 3526 :ok :read [5 43] INFO [2022-03-13 17:16:54,363] jepsen worker 85 - jepsen.util 3235 :info :write [28 19] :RequestTimeoutException INFO [2022-03-13 17:16:54,395] jepsen worker 89 - jepsen.util 2609 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,444] jepsen worker 85 - jepsen.util 3325 :invoke :read [28 nil] INFO [2022-03-13 17:16:54,444] jepsen worker 89 - jepsen.util 2609 :invoke :write [29 49] INFO [2022-03-13 17:16:54,410] jepsen worker 84 - jepsen.util 3954 :info :write [28 14] :RequestTimeoutException INFO [2022-03-13 17:16:54,419] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:54,444] jepsen worker 16 - jepsen.util 3526 :invoke :write [5 36] INFO [2022-03-13 17:16:54,444] jepsen worker nemesis - couchbase.util Rebalance check count: 522 INFO [2022-03-13 17:16:54,444] jepsen worker 84 - jepsen.util 4044 :invoke :write [28 0] INFO [2022-03-13 17:16:54,444] jepsen worker nemesis - couchbase.util Rebalance stuck count: 419 INFO [2022-03-13 17:16:54,449] jepsen worker 3 - jepsen.util 2073 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,449] jepsen worker 3 - jepsen.util 2073 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,453] jepsen worker 5 - jepsen.util 2525 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,454] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 25] INFO [2022-03-13 17:16:54,475] jepsen worker 87 - jepsen.util 2697 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,476] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 47] INFO [2022-03-13 17:16:54,486] jepsen worker 88 - jepsen.util 2158 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,487] jepsen worker 88 - jepsen.util 2158 :invoke :write [29 42] INFO [2022-03-13 17:16:54,523] jepsen worker 85 - jepsen.util 3325 :ok :read [28 44] INFO [2022-03-13 17:16:54,524] jepsen worker 85 - jepsen.util 3325 :invoke :read [28 nil] INFO [2022-03-13 17:16:54,534] jepsen worker 79 - jepsen.util 1069 :info :write [26 43] :RequestTimeoutException INFO [2022-03-13 17:16:54,535] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:16:54,559] jepsen worker 37 - jepsen.util 2197 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,559] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,562] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,563] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,566] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,566] jepsen worker 37 - jepsen.util 2197 :invoke :write [12 45] INFO [2022-03-13 17:16:54,570] jepsen worker 89 - jepsen.util 2609 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,571] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,573] jepsen worker 53 - jepsen.util 503 :info :write [17 23] :RequestTimeoutException INFO [2022-03-13 17:16:54,574] jepsen worker 53 - jepsen.util 593 :invoke :write [17 19] INFO [2022-03-13 17:16:54,599] jepsen worker 36 - jepsen.util 2286 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,599] jepsen worker 3 - jepsen.util 2073 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,600] jepsen worker 3 - jepsen.util 2073 :invoke :write [1 21] INFO [2022-03-13 17:16:54,600] jepsen worker 36 - jepsen.util 2286 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,605] jepsen worker 36 - jepsen.util 2286 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,606] jepsen worker 36 - jepsen.util 2286 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,613] jepsen worker 36 - jepsen.util 2286 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,613] jepsen worker 36 - jepsen.util 2286 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,616] jepsen worker 36 - jepsen.util 2286 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,617] jepsen worker 36 - jepsen.util 2286 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,624] jepsen worker 36 - jepsen.util 2286 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,624] jepsen worker 36 - jepsen.util 2286 :invoke :write [12 0] INFO [2022-03-13 17:16:54,628] jepsen worker 5 - jepsen.util 2525 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,629] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,631] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,631] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,631] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,632] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,632] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,633] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,633] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,634] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,634] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,635] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,635] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,636] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,636] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,637] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,637] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,638] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,638] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 36] INFO [2022-03-13 17:16:54,639] jepsen worker 89 - jepsen.util 2609 :invoke :write [29 35] INFO [2022-03-13 17:16:54,670] jepsen worker 4 - jepsen.util 2344 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,671] jepsen worker 4 - jepsen.util 2344 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,676] jepsen worker 85 - jepsen.util 3325 :ok :read [28 44] INFO [2022-03-13 17:16:54,677] jepsen worker 85 - jepsen.util 3325 :invoke :write [28 1] INFO [2022-03-13 17:16:54,679] jepsen worker 4 - jepsen.util 2344 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,679] jepsen worker 4 - jepsen.util 2344 :invoke :write [1 31] INFO [2022-03-13 17:16:54,719] jepsen worker 5 - jepsen.util 2525 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,720] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 2] INFO [2022-03-13 17:16:54,754] jepsen worker 87 - jepsen.util 2697 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,755] jepsen worker 87 - jepsen.util 2697 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,758] jepsen worker 87 - jepsen.util 2697 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,759] jepsen worker 87 - jepsen.util 2697 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,762] jepsen worker 87 - jepsen.util 2697 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,762] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 9] INFO [2022-03-13 17:16:54,785] jepsen worker 89 - jepsen.util 2609 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,786] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,815] jepsen worker 38 - jepsen.util 2558 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,815] jepsen worker 38 - jepsen.util 2558 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,856] jepsen worker 3 - jepsen.util 2073 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,857] jepsen worker 3 - jepsen.util 2073 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,892] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,892] jepsen worker 5 - jepsen.util 2525 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,893] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,893] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:54,895] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,895] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,895] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,896] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 5] INFO [2022-03-13 17:16:54,897] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,897] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:54,901] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,901] jepsen worker 89 - jepsen.util 2609 :invoke :write [29 45] INFO [2022-03-13 17:16:54,923] jepsen worker 37 - jepsen.util 2197 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,924] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,927] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,928] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:54,940] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,940] jepsen worker 37 - jepsen.util 2197 :invoke :write [12 44] INFO [2022-03-13 17:16:54,954] jepsen worker 36 - jepsen.util 2286 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,954] jepsen worker 3 - jepsen.util 2073 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:54,955] jepsen worker 36 - jepsen.util 2286 :invoke :write [12 6] INFO [2022-03-13 17:16:54,959] jepsen worker 3 - jepsen.util 2073 :invoke :write [1 25] INFO [2022-03-13 17:16:54,968] jepsen worker 12 - jepsen.util 732 :info :write [4 48] :RequestTimeoutException INFO [2022-03-13 17:16:54,969] jepsen worker 12 - jepsen.util 822 :invoke :write [4 9] INFO [2022-03-13 17:16:55,009] jepsen worker 87 - jepsen.util 2697 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,010] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 9] INFO [2022-03-13 17:16:55,068] jepsen worker 5 - jepsen.util 2525 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,069] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,141] jepsen worker 88 - jepsen.util 2158 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,141] jepsen worker 38 - jepsen.util 2558 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,141] jepsen worker 4 - jepsen.util 2344 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,141] jepsen worker 88 - jepsen.util 2158 :invoke :write [29 11] INFO [2022-03-13 17:16:55,142] jepsen worker 38 - jepsen.util 2558 :invoke :write [12 42] INFO [2022-03-13 17:16:55,142] jepsen worker 4 - jepsen.util 2344 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,146] jepsen worker 4 - jepsen.util 2344 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,147] jepsen worker 4 - jepsen.util 2344 :invoke :write [1 15] INFO [2022-03-13 17:16:55,155] jepsen worker 36 - jepsen.util 2286 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,156] jepsen worker 36 - jepsen.util 2286 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,203] jepsen worker 89 - jepsen.util 2609 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,203] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,203] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,204] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,209] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,209] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,210] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,211] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 13] INFO [2022-03-13 17:16:55,211] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,212] jepsen worker 89 - jepsen.util 2609 :invoke :write [29 7] INFO [2022-03-13 17:16:55,218] jepsen worker 69 - jepsen.util 699 :info :write [23 16] :RequestTimeoutException INFO [2022-03-13 17:16:55,218] jepsen worker 76 - jepsen.util 706 :info :write [25 28] :RequestTimeoutException INFO [2022-03-13 17:16:55,219] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:16:55,220] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:55,227] jepsen worker 37 - jepsen.util 2197 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,228] jepsen worker 61 - jepsen.util 691 :info :write [20 9] :RequestTimeoutException INFO [2022-03-13 17:16:55,228] jepsen worker 44 - jepsen.util 584 :info :write [14 10] :RequestTimeoutException INFO [2022-03-13 17:16:55,228] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,229] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:16:55,230] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:16:55,231] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,234] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,234] jepsen worker 43 - jepsen.util 583 :info :write [14 19] :RequestTimeoutException INFO [2022-03-13 17:16:55,234] jepsen worker 87 - jepsen.util 2697 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,235] jepsen worker 43 - jepsen.util 673 :invoke :write [14 15] INFO [2022-03-13 17:16:55,236] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 1] INFO [2022-03-13 17:16:55,237] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,237] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,238] jepsen worker 70 - jepsen.util 790 :info :write [23 38] :RequestTimeoutException INFO [2022-03-13 17:16:55,238] jepsen worker 70 - jepsen.util 880 :invoke :write [23 43] INFO [2022-03-13 17:16:55,248] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,249] jepsen worker 37 - jepsen.util 2197 :invoke :write [12 11] INFO [2022-03-13 17:16:55,286] jepsen worker 36 - jepsen.util 2286 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,286] jepsen worker 3 - jepsen.util 2073 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,287] jepsen worker 3 - jepsen.util 2073 :invoke :write [1 26] INFO [2022-03-13 17:16:55,287] jepsen worker 36 - jepsen.util 2286 :invoke :write [12 17] INFO [2022-03-13 17:16:55,327] jepsen worker 39 - jepsen.util 3819 :info :write [13 31] :RequestTimeoutException INFO [2022-03-13 17:16:55,327] jepsen worker 39 - jepsen.util 3909 :invoke :read [13 nil] INFO [2022-03-13 17:16:55,343] jepsen worker 5 - jepsen.util 2525 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,344] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 2] INFO [2022-03-13 17:16:55,366] jepsen worker 87 - jepsen.util 2697 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,367] jepsen worker 87 - jepsen.util 2697 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,371] jepsen worker 87 - jepsen.util 2697 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,371] jepsen worker 87 - jepsen.util 2697 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,373] jepsen worker 87 - jepsen.util 2697 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,373] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 41] INFO [2022-03-13 17:16:55,391] jepsen worker 4 - jepsen.util 2344 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,391] jepsen worker 38 - jepsen.util 2558 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,392] jepsen worker 4 - jepsen.util 2344 :invoke :write [1 25] INFO [2022-03-13 17:16:55,392] jepsen worker 38 - jepsen.util 2558 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,432] jepsen worker 89 - jepsen.util 2609 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,433] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,450] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:55,450] jepsen worker nemesis - couchbase.util Rebalance check count: 523 INFO [2022-03-13 17:16:55,450] jepsen worker nemesis - couchbase.util Rebalance stuck count: 420 INFO [2022-03-13 17:16:55,455] jepsen worker 36 - jepsen.util 2286 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,455] jepsen worker 36 - jepsen.util 2286 :invoke :write [12 45] INFO [2022-03-13 17:16:55,490] jepsen worker 37 - jepsen.util 2197 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,491] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,515] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,515] jepsen worker 5 - jepsen.util 2525 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,516] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,517] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,519] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,519] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,519] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,520] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,520] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,521] jepsen worker 5 - jepsen.util 2525 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,521] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,522] jepsen worker 5 - jepsen.util 2525 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,522] jepsen worker 89 - jepsen.util 2609 :invoke :write [29 30] INFO [2022-03-13 17:16:55,522] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 47] INFO [2022-03-13 17:16:55,554] jepsen worker 38 - jepsen.util 2558 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,554] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,554] jepsen worker 88 - jepsen.util 2158 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,554] jepsen worker 38 - jepsen.util 2558 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,555] jepsen worker 88 - jepsen.util 2158 :invoke :write [29 33] INFO [2022-03-13 17:16:55,556] jepsen worker 38 - jepsen.util 2558 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,557] jepsen worker 38 - jepsen.util 2558 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,557] jepsen worker 37 - jepsen.util 2197 :invoke :read [12 nil] INFO [2022-03-13 17:16:55,559] jepsen worker 37 - jepsen.util 2197 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,559] jepsen worker 38 - jepsen.util 2558 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,560] jepsen worker 37 - jepsen.util 2197 :invoke :write [12 37] INFO [2022-03-13 17:16:55,560] jepsen worker 38 - jepsen.util 2558 :invoke :write [12 49] INFO [2022-03-13 17:16:55,593] jepsen worker 3 - jepsen.util 2073 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,594] jepsen worker 3 - jepsen.util 2073 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,620] jepsen worker 24 - jepsen.util 564 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,620] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:16:55,623] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,623] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:16:55,628] jepsen worker 33 - jepsen.util 753 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,628] jepsen worker 89 - jepsen.util 2609 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,628] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,628] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,630] jepsen worker 51 - jepsen.util 771 :invoke :write [17 49] INFO [2022-03-13 17:16:55,631] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,632] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:55,633] jepsen worker 33 - jepsen.util 753 :invoke :write [11 42] INFO [2022-03-13 17:16:55,639] jepsen worker 4 - jepsen.util 2344 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,640] jepsen worker 4 - jepsen.util 2344 :invoke :write [1 14] INFO [2022-03-13 17:16:55,643] jepsen worker 87 - jepsen.util 2697 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,643] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 6] INFO [2022-03-13 17:16:55,678] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,679] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:55,694] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,694] jepsen worker 5 - jepsen.util 2525 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,695] jepsen worker 89 - jepsen.util 2609 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,695] jepsen worker 5 - jepsen.util 2525 :invoke :write [1 12] INFO [2022-03-13 17:16:55,696] jepsen worker 89 - jepsen.util 2609 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,699] jepsen worker 89 - jepsen.util 2609 :invoke :write [29 15] INFO [2022-03-13 17:16:55,730] jepsen worker 36 - jepsen.util 2286 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,731] jepsen worker 36 - jepsen.util 2286 :invoke :write [12 42] INFO [2022-03-13 17:16:55,732] jepsen worker 3 - jepsen.util 2073 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,733] jepsen worker 3 - jepsen.util 2073 :invoke :read [1 nil] INFO [2022-03-13 17:16:55,740] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,741] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:55,741] jepsen worker 75 - jepsen.util 435 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,742] jepsen worker 3 - jepsen.util 2073 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,742] jepsen worker 75 - jepsen.util 435 :invoke :write [25 28] INFO [2022-03-13 17:16:55,744] jepsen worker 3 - jepsen.util 2073 :invoke :write [1 35] INFO [2022-03-13 17:16:55,803] jepsen worker 87 - jepsen.util 2697 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,804] jepsen worker 87 - jepsen.util 2697 :invoke :write [29 41] INFO [2022-03-13 17:16:55,812] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,813] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:55,820] jepsen worker 88 - jepsen.util 2158 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,820] jepsen worker 88 - jepsen.util 2158 :invoke :read [29 nil] INFO [2022-03-13 17:16:55,829] jepsen worker 14 - jepsen.util 374 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,830] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,831] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:16:55,831] jepsen worker 80 - jepsen.util 800 :invoke :write [26 19] INFO [2022-03-13 17:16:55,847] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,847] jepsen worker 13 - jepsen.util 733 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,848] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:55,848] jepsen worker 13 - jepsen.util 733 :invoke :write [4 1] INFO [2022-03-13 17:16:55,854] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,855] jepsen worker 76 - jepsen.util 796 :invoke :write [25 31] INFO [2022-03-13 17:16:55,882] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,883] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:16:55,918] jepsen worker 82 - jepsen.util 3772 :info :write [27 5] :RequestTimeoutException INFO [2022-03-13 17:16:55,919] jepsen worker 82 - jepsen.util 3862 :invoke :write [27 5] INFO [2022-03-13 17:16:55,924] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,925] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:55,940] jepsen worker 53 - jepsen.util 593 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,941] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:16:55,946] jepsen worker 88 - jepsen.util 2158 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,946] jepsen worker 88 - jepsen.util 2158 :invoke :write [29 42] INFO [2022-03-13 17:16:55,958] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:55,959] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:56,011] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,011] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:16:56,011] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,012] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:16:56,013] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:16:56,013] jepsen worker 34 - jepsen.util 934 :invoke :write [11 26] INFO [2022-03-13 17:16:56,130] jepsen worker 51 - jepsen.util 771 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,131] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:56,131] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,132] jepsen worker 71 - jepsen.util 521 :invoke :write [23 1] INFO [2022-03-13 17:16:56,135] jepsen worker 43 - jepsen.util 673 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,135] jepsen worker 13 - jepsen.util 733 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,135] jepsen worker 43 - jepsen.util 673 :invoke :write [14 34] INFO [2022-03-13 17:16:56,136] jepsen worker 13 - jepsen.util 733 :invoke :write [4 0] INFO [2022-03-13 17:16:56,137] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,137] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,137] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,138] jepsen worker 52 - jepsen.util 682 :invoke :write [17 30] INFO [2022-03-13 17:16:56,139] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:16:56,139] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:56,141] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,141] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,142] jepsen worker 53 - jepsen.util 593 :invoke :write [17 42] INFO [2022-03-13 17:16:56,143] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:16:56,220] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,220] jepsen worker 75 - jepsen.util 435 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,220] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,220] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:16:56,221] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:56,222] jepsen worker 69 - jepsen.util 789 :invoke :write [23 23] INFO [2022-03-13 17:16:56,227] jepsen worker 76 - jepsen.util 796 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,228] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:56,234] jepsen worker 33 - jepsen.util 753 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,234] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,235] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:56,235] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:56,300] jepsen worker 80 - jepsen.util 800 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,301] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:56,334] jepsen worker 71 - jepsen.util 521 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,334] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,335] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:16:56,335] jepsen worker 24 - jepsen.util 564 :invoke :write [8 40] INFO [2022-03-13 17:16:56,432] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,432] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,432] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:16:56,433] jepsen worker 14 - jepsen.util 374 :invoke :write [4 34] INFO [2022-03-13 17:16:56,433] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:56,433] jepsen worker 44 - jepsen.util 674 :invoke :write [14 0] INFO [2022-03-13 17:16:56,444] jepsen worker 69 - jepsen.util 789 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,445] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:16:56,456] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:56,456] jepsen worker nemesis - couchbase.util Rebalance check count: 524 INFO [2022-03-13 17:16:56,456] jepsen worker nemesis - couchbase.util Rebalance stuck count: 421 INFO [2022-03-13 17:16:56,491] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,491] jepsen worker 13 - jepsen.util 733 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,491] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,491] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:16:56,492] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:56,493] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:56,493] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,494] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,494] jepsen worker 13 - jepsen.util 733 :invoke :write [4 41] INFO [2022-03-13 17:16:56,495] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:56,497] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,497] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:56,512] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,559] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,857] jepsen worker 76 - jepsen.util 796 :invoke :write [25 5] INFO [2022-03-13 17:16:56,559] jepsen worker 34 - jepsen.util 934 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,559] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,857] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:56,857] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:56,559] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,560] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,858] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:56,560] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,567] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,859] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:56,859] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:56,859] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:16:56,567] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,611] jepsen worker 43 - jepsen.util 673 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,859] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:56,859] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:16:56,649] jepsen worker 44 - jepsen.util 674 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,678] jepsen worker 24 - jepsen.util 564 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,708] jepsen worker 10 - jepsen.util 3700 :ok :read [3 25] INFO [2022-03-13 17:16:56,859] jepsen worker 24 - jepsen.util 564 :invoke :write [8 6] INFO [2022-03-13 17:16:56,741] jepsen worker 29 - jepsen.util 3989 :ok :read [9 26] INFO [2022-03-13 17:16:56,859] jepsen worker 10 - jepsen.util 3700 :invoke :read [3 nil] INFO [2022-03-13 17:16:56,859] jepsen worker 29 - jepsen.util 3989 :invoke :write [9 2] WARN [2022-03-13 17:16:56,744] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17316003,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17413178,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9114638,"encode_duration_us":24904,"operation_id":"0xed42","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10004724},{"operation_name":"upsert","last_dispatch_duration_us":9081487,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10620247,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9514674,"encode_duration_us":17414,"operation_id":"0xefc7","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10005642}],"total_count":2}} INFO [2022-03-13 17:16:56,762] jepsen worker 13 - jepsen.util 733 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,860] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] WARN [2022-03-13 17:16:56,860] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8460360,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9437088,"total_server_duration_us":460248,"encode_duration_us":40079,"operation_id":"0xf028","timeout_ms":10000,"last_server_duration_us":25800,"total_duration_us":10010068},{"operation_name":"upsert","last_dispatch_duration_us":82364,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":3079446,"total_server_duration_us":1236812,"encode_duration_us":18354,"operation_id":"0xf0f7","timeout_ms":10000,"last_server_duration_us":66168,"total_duration_us":10005685},{"operation_name":"upsert","last_dispatch_duration_us":9081487,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10620247,"total_server_duration_us":9514674,"encode_duration_us":17414,"operation_id":"0xefc7","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10005642},{"operation_name":"upsert","last_dispatch_duration_us":15412807,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":15412807,"total_server_duration_us":40695,"encode_duration_us":17266,"operation_id":"0xee73","timeout_ms":10000,"last_server_duration_us":40695,"total_duration_us":10005507},{"operation_name":"upsert","last_dispatch_duration_us":8719027,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8719027,"total_server_duration_us":23054,"encode_duration_us":21036,"operation_id":"0xf084","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10005137},{"operation_name":"upsert","last_dispatch_duration_us":65401,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":3008892,"total_server_duration_us":899081,"encode_duration_us":20483,"operation_id":"0xf121","timeout_ms":10000,"last_server_duration_us":52064,"total_duration_us":10000929},{"operation_name":"get","last_dispatch_duration_us":9112378,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9112378,"total_server_duration_us":49,"operation_id":"0xf136","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9285820},{"operation_name":"get","last_dispatch_duration_us":7589274,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7589274,"total_server_duration_us":9171,"operation_id":"0xf1e8","timeout_ms":30000,"last_server_duration_us":9171,"total_duration_us":7591455},{"operation_name":"get","last_dispatch_duration_us":2216490,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2216490,"total_server_duration_us":19,"operation_id":"0xf321","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":3305828},{"operation_name":"upsert","last_dispatch_duration_us":2601808,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2601808,"total_server_duration_us":1656002,"encode_duration_us":17825,"operation_id":"0xf357","timeout_ms":10000,"last_server_duration_us":1656002,"total_duration_us":2607411}],"total_count":63}} INFO [2022-03-13 17:16:56,763] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,863] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:16:56,764] jepsen worker 14 - jepsen.util 374 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,776] jepsen worker 28 - jepsen.util 3718 :ok :read [9 26] INFO [2022-03-13 17:16:56,797] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,866] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:16:56,866] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:16:56,866] jepsen worker 28 - jepsen.util 3718 :invoke :read [9 nil] INFO [2022-03-13 17:16:56,872] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,880] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,880] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,880] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,798] jepsen worker 11 - jepsen.util 3611 :ok :read [3 25] INFO [2022-03-13 17:16:56,880] jepsen worker 11 - jepsen.util 3611 :invoke :write [3 31] INFO [2022-03-13 17:16:56,880] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,880] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,817] jepsen worker 53 - jepsen.util 593 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,857] jepsen worker 51 - jepsen.util 771 :invoke :write [17 49] INFO [2022-03-13 17:16:56,881] jepsen worker 53 - jepsen.util 593 :invoke :write [17 21] INFO [2022-03-13 17:16:56,859] jepsen worker 44 - jepsen.util 674 :invoke :write [14 2] INFO [2022-03-13 17:16:56,887] jepsen worker 80 - jepsen.util 800 :invoke :write [26 37] INFO [2022-03-13 17:16:56,890] jepsen worker 75 - jepsen.util 435 :invoke :write [25 4] INFO [2022-03-13 17:16:56,897] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:16:56,898] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:56,899] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:56,899] jepsen worker 21 - jepsen.util 831 :invoke :write [7 15] INFO [2022-03-13 17:16:56,911] jepsen worker 52 - jepsen.util 682 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:16:56,913] jepsen worker 52 - jepsen.util 682 :invoke :write [17 12] INFO [2022-03-13 17:16:57,005] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,008] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:16:57,035] jepsen worker 10 - jepsen.util 3700 :ok :read [3 25] INFO [2022-03-13 17:16:57,036] jepsen worker 10 - jepsen.util 3700 :invoke :read [3 nil] INFO [2022-03-13 17:16:57,096] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:16:57,096] jepsen worker 76 - jepsen.util 796 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,096] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,096] jepsen worker 76 - jepsen.util 796 :invoke :write [25 40] INFO [2022-03-13 17:16:57,097] jepsen worker 43 - jepsen.util 673 :invoke :write [14 7] INFO [2022-03-13 17:16:57,097] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:57,103] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,104] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:16:57,108] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,108] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,108] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:16:57,108] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,109] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:16:57,109] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:16:57,110] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:16:57,111] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:16:57,136] jepsen worker 24 - jepsen.util 564 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,137] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:57,151] jepsen worker 10 - jepsen.util 3700 :ok :read [3 25] INFO [2022-03-13 17:16:57,152] jepsen worker 10 - jepsen.util 3700 :invoke :write [3 30] INFO [2022-03-13 17:16:57,176] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,177] jepsen worker 13 - jepsen.util 733 :invoke :write [4 15] INFO [2022-03-13 17:16:57,220] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,220] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:16:57,227] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,227] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,228] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,228] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:16:57,228] jepsen worker 33 - jepsen.util 753 :invoke :write [11 48] INFO [2022-03-13 17:16:57,230] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:57,240] jepsen worker 53 - jepsen.util 593 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,240] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:16:57,313] jepsen worker 75 - jepsen.util 435 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,313] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:57,356] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,356] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:57,396] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,396] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,396] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,397] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,397] jepsen worker 71 - jepsen.util 521 :invoke :write [23 38] INFO [2022-03-13 17:16:57,399] jepsen worker 24 - jepsen.util 564 :invoke :write [8 31] INFO [2022-03-13 17:16:57,400] jepsen worker 75 - jepsen.util 435 :invoke :write [25 42] INFO [2022-03-13 17:16:57,402] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:16:57,462] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3489736070381232}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3489736070381232}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:57,462] jepsen worker nemesis - couchbase.util Rebalance check count: 525 INFO [2022-03-13 17:16:57,462] jepsen worker nemesis - couchbase.util Rebalance stuck count: 422 INFO [2022-03-13 17:16:57,478] jepsen worker 76 - jepsen.util 796 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,478] jepsen worker 52 - jepsen.util 682 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,479] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:57,479] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:57,479] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,480] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:16:57,480] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,481] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:57,569] jepsen worker 43 - jepsen.util 673 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,569] jepsen worker 13 - jepsen.util 733 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,569] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,569] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,570] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,570] jepsen worker 53 - jepsen.util 593 :invoke :write [17 6] INFO [2022-03-13 17:16:57,570] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,571] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:16:57,571] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:57,572] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:57,572] jepsen worker 52 - jepsen.util 682 :invoke :write [17 10] INFO [2022-03-13 17:16:57,573] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:57,573] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,574] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:16:57,574] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,574] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:16:57,574] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,575] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:16:57,575] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:16:57,576] jepsen worker 80 - jepsen.util 800 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,576] jepsen worker 13 - jepsen.util 733 :invoke :write [4 18] INFO [2022-03-13 17:16:57,579] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:57,607] jepsen worker 71 - jepsen.util 521 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,608] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:16:57,617] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:16:57,618] jepsen worker 43 - jepsen.util 673 :invoke :write [14 9] INFO [2022-03-13 17:16:57,706] jepsen worker 24 - jepsen.util 564 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,707] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:16:57,746] jepsen worker 21 - jepsen.util 831 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,747] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:16:57,781] jepsen worker 52 - jepsen.util 682 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,781] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,782] jepsen worker 52 - jepsen.util 682 :invoke :write [17 9] INFO [2022-03-13 17:16:57,782] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:16:57,881] jepsen worker 81 - jepsen.util 4221 :info :write [27 24] :RequestTimeoutException INFO [2022-03-13 17:16:57,882] jepsen worker 81 - jepsen.util 4311 :invoke :write [27 34] INFO [2022-03-13 17:16:57,884] jepsen worker 75 - jepsen.util 435 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,885] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:16:57,947] jepsen worker 44 - jepsen.util 674 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:16:57,948] jepsen worker 44 - jepsen.util 674 :invoke :write [14 31] INFO [2022-03-13 17:16:58,018] jepsen worker 73 - jepsen.util 3763 :info :write [24 45] :RequestTimeoutException INFO [2022-03-13 17:16:58,019] jepsen worker 73 - jepsen.util 3853 :invoke :read [24 nil] INFO [2022-03-13 17:16:58,034] jepsen worker 72 - jepsen.util 3852 :ok :read [24 24] INFO [2022-03-13 17:16:58,035] jepsen worker 72 - jepsen.util 3852 :invoke :read [24 nil] INFO [2022-03-13 17:16:58,046] jepsen worker 31 - jepsen.util 3901 :ok :read [10 :nil] INFO [2022-03-13 17:16:58,046] jepsen worker 31 - jepsen.util 3901 :invoke :read [10 nil] INFO [2022-03-13 17:16:58,056] jepsen worker 51 - jepsen.util 771 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,056] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,056] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,057] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:58,058] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:58,059] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:16:58,065] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,065] jepsen worker 69 - jepsen.util 789 :invoke :write [23 46] WARN [2022-03-13 17:16:58,128] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9337134,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10296614,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9480634,"encode_duration_us":18751,"operation_id":"0xee87","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":10002527},{"operation_name":"upsert","last_dispatch_duration_us":12138298,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12138298,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":91985,"encode_duration_us":22079,"operation_id":"0xf1ad","timeout_ms":10000,"last_server_duration_us":91985,"total_duration_us":10009153},{"operation_name":"upsert","last_dispatch_duration_us":20010575,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20010575,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9043735,"encode_duration_us":19818,"operation_id":"0xee7b","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10002829}],"total_count":3}} INFO [2022-03-13 17:16:58,144] jepsen worker 13 - jepsen.util 733 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,144] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,145] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:16:58,145] jepsen worker 13 - jepsen.util 733 :invoke :write [4 11] INFO [2022-03-13 17:16:58,196] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,196] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,196] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,197] jepsen worker 71 - jepsen.util 521 :invoke :write [23 45] INFO [2022-03-13 17:16:58,198] jepsen worker 75 - jepsen.util 435 :invoke :write [25 1] INFO [2022-03-13 17:16:58,198] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:16:58,231] jepsen worker 55 - jepsen.util 3385 :ok :read [18 21] INFO [2022-03-13 17:16:58,232] jepsen worker 55 - jepsen.util 3385 :invoke :write [18 11] WARN [2022-03-13 17:16:58,234] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19128454,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19128454,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9039489,"encode_duration_us":22031,"operation_id":"0xeb55","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10004986},{"operation_name":"upsert","last_dispatch_duration_us":9061741,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9806365,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9345854,"encode_duration_us":17115,"operation_id":"0xee82","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10010285},{"operation_name":"upsert","last_dispatch_duration_us":12548226,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":12548226,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":118961,"encode_duration_us":39955,"operation_id":"0xf175","timeout_ms":10000,"last_server_duration_us":118961,"total_duration_us":10006874}],"total_count":3}} INFO [2022-03-13 17:16:58,288] jepsen worker 33 - jepsen.util 753 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,288] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,288] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,288] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,288] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,288] jepsen worker 33 - jepsen.util 753 :invoke :write [11 24] INFO [2022-03-13 17:16:58,290] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:16:58,290] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:16:58,291] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:16:58,291] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:16:58,327] jepsen worker 69 - jepsen.util 789 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,328] jepsen worker 60 - jepsen.util 600 :info :write [20 38] :RequestTimeoutException INFO [2022-03-13 17:16:58,328] jepsen worker 69 - jepsen.util 789 :invoke :write [23 42] INFO [2022-03-13 17:16:58,329] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] WARN [2022-03-13 17:16:58,332] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14648054,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":14648054,"total_server_duration_us":27,"operation_id":"0xf039","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14650382},{"operation_name":"get","last_dispatch_duration_us":13601267,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13601267,"total_server_duration_us":19,"operation_id":"0xf0e1","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":13604971},{"operation_name":"upsert","last_dispatch_duration_us":79979,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":4000417,"total_server_duration_us":948134,"encode_duration_us":18833,"operation_id":"0xf116","timeout_ms":10000,"last_server_duration_us":59374,"total_duration_us":10010503},{"operation_name":"upsert","last_dispatch_duration_us":40626,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":4074674,"total_server_duration_us":989426,"encode_duration_us":19834,"operation_id":"0xf117","timeout_ms":10000,"last_server_duration_us":28218,"total_duration_us":10010479},{"operation_name":"upsert","last_dispatch_duration_us":8102751,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8545698,"total_server_duration_us":255910,"encode_duration_us":27937,"operation_id":"0xeeb6","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10008305},{"operation_name":"upsert","last_dispatch_duration_us":10242361,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10242361,"total_server_duration_us":44279,"encode_duration_us":32357,"operation_id":"0xf147","timeout_ms":10000,"last_server_duration_us":44279,"total_duration_us":10006010},{"operation_name":"upsert","last_dispatch_duration_us":17118761,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":17453078,"total_server_duration_us":9232032,"encode_duration_us":20841,"operation_id":"0xef15","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10001464},{"operation_name":"upsert","last_dispatch_duration_us":49523,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":3899841,"total_server_duration_us":1020037,"encode_duration_us":17474,"operation_id":"0xf115","timeout_ms":10000,"last_server_duration_us":36729,"total_duration_us":10000866},{"operation_name":"upsert","last_dispatch_duration_us":2231089,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2231089,"total_server_duration_us":1908630,"encode_duration_us":25820,"operation_id":"0xf42a","timeout_ms":10000,"last_server_duration_us":1908630,"total_duration_us":2236833},{"operation_name":"upsert","last_dispatch_duration_us":2162751,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2162751,"total_server_duration_us":74786,"encode_duration_us":28140,"operation_id":"0xf45c","timeout_ms":10000,"last_server_duration_us":74786,"total_duration_us":2167095}],"total_count":24}} WARN [2022-03-13 17:16:58,347] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":184090,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":880718,"total_server_duration_us":137551,"encode_duration_us":20099,"operation_id":"0xeff7","timeout_ms":10000,"last_server_duration_us":25976,"total_duration_us":10007673},{"operation_name":"upsert","last_dispatch_duration_us":8169610,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9578391,"total_server_duration_us":178238,"encode_duration_us":19062,"operation_id":"0xefc6","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10007309},{"operation_name":"upsert","last_dispatch_duration_us":73064,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":2399169,"total_server_duration_us":1399938,"encode_duration_us":25475,"operation_id":"0xf120","timeout_ms":10000,"last_server_duration_us":46175,"total_duration_us":10006600},{"operation_name":"upsert","last_dispatch_duration_us":30190,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8430813,"total_server_duration_us":273114,"encode_duration_us":17137,"operation_id":"0xf0ee","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10006460},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17206,"operation_id":"0xf158","timeout_ms":10000,"total_duration_us":10006419},{"operation_name":"upsert","last_dispatch_duration_us":8197843,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8927017,"total_server_duration_us":143126,"encode_duration_us":25856,"operation_id":"0xf02b","timeout_ms":10000,"last_server_duration_us":24841,"total_duration_us":10004544},{"operation_name":"upsert","last_dispatch_duration_us":8194738,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8535008,"total_server_duration_us":80555,"encode_duration_us":23419,"operation_id":"0xf05b","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10004242},{"operation_name":"upsert","last_dispatch_duration_us":9183030,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9183030,"total_server_duration_us":9028877,"encode_duration_us":27610,"operation_id":"0xeff6","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":9187867},{"operation_name":"get","last_dispatch_duration_us":8252469,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8252469,"total_server_duration_us":43,"operation_id":"0xf095","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8255470},{"operation_name":"get","last_dispatch_duration_us":3863597,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":3863597,"total_server_duration_us":375158,"operation_id":"0xf25d","timeout_ms":30000,"last_server_duration_us":375158,"total_duration_us":3865661}],"total_count":24}} WARN [2022-03-13 17:16:58,348] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9039162,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18394983,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9161550,"encode_duration_us":28069,"operation_id":"0xeb30","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10002225},{"operation_name":"upsert","last_dispatch_duration_us":9050945,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18271472,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9117564,"encode_duration_us":20450,"operation_id":"0xeb31","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009168}],"total_count":2}} WARN [2022-03-13 17:16:58,429] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":23959384,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":23959384,"total_server_duration_us":1887860,"operation_id":"0xe75c","timeout_ms":30000,"last_server_duration_us":1887860,"total_duration_us":23961338},{"operation_name":"get","last_dispatch_duration_us":16779590,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":16779590,"total_server_duration_us":6927,"operation_id":"0xefc8","timeout_ms":30000,"last_server_duration_us":6927,"total_duration_us":16782009},{"operation_name":"get","last_dispatch_duration_us":14000417,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":14000417,"total_server_duration_us":27,"operation_id":"0xf141","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14000750},{"operation_name":"get","last_dispatch_duration_us":11916854,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":11916854,"total_server_duration_us":49,"operation_id":"0xf1b7","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":11917146},{"operation_name":"get","last_dispatch_duration_us":11458764,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11458764,"total_server_duration_us":477833,"operation_id":"0xee7c","timeout_ms":30000,"last_server_duration_us":477833,"total_duration_us":11461051},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":17516,"operation_id":"0xf2b0","timeout_ms":10000,"total_duration_us":10010801},{"operation_name":"upsert","last_dispatch_duration_us":55182,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8187177,"total_server_duration_us":195639,"encode_duration_us":18596,"operation_id":"0xf1f7","timeout_ms":10000,"last_server_duration_us":23222,"total_duration_us":10010627},{"operation_name":"upsert","last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","encode_duration_us":28098,"operation_id":"0xf1ba","timeout_ms":10000,"total_duration_us":10010162},{"operation_name":"upsert","last_dispatch_duration_us":12138298,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":12138298,"total_server_duration_us":91985,"encode_duration_us":22079,"operation_id":"0xf1ad","timeout_ms":10000,"last_server_duration_us":91985,"total_duration_us":10009153},{"operation_name":"upsert","last_dispatch_duration_us":15312422,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":15312422,"total_server_duration_us":20601,"encode_duration_us":21015,"operation_id":"0xf0bc","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10008872}],"total_count":22}} INFO [2022-03-13 17:16:58,431] jepsen worker 53 - jepsen.util 593 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,432] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,433] jepsen worker 53 - jepsen.util 593 :invoke :write [17 37] INFO [2022-03-13 17:16:58,433] jepsen worker 76 - jepsen.util 796 :invoke :write [25 0] INFO [2022-03-13 17:16:58,438] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,438] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,439] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:58,440] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:58,438] jepsen worker 44 - jepsen.util 674 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,440] jepsen worker 44 - jepsen.util 674 :invoke :write [14 44] INFO [2022-03-13 17:16:58,468] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3519061583577713}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3519061583577713}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:58,468] jepsen worker nemesis - couchbase.util Rebalance check count: 526 INFO [2022-03-13 17:16:58,468] jepsen worker nemesis - couchbase.util Rebalance stuck count: 422 INFO [2022-03-13 17:16:58,481] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,482] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] WARN [2022-03-13 17:16:58,533] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18139,"operation_id":"0xf11e","timeout_ms":10000,"total_duration_us":10009615},{"operation_name":"upsert","last_dispatch_duration_us":8058301,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8260038,"total_server_duration_us":43543,"encode_duration_us":41561,"operation_id":"0xf072","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10002719},{"operation_name":"upsert","last_dispatch_duration_us":56662,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9274378,"total_server_duration_us":268073,"encode_duration_us":23509,"operation_id":"0xf1c1","timeout_ms":10000,"last_server_duration_us":25450,"total_duration_us":10002344},{"operation_name":"upsert","last_dispatch_duration_us":8094747,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8235410,"total_server_duration_us":44141,"encode_duration_us":31069,"operation_id":"0xf06a","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10002216},{"operation_name":"upsert","last_dispatch_duration_us":9075511,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":14224934,"total_server_duration_us":9081129,"encode_duration_us":24142,"operation_id":"0xef6e","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10001259},{"operation_name":"get","last_dispatch_duration_us":5583350,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5583350,"total_server_duration_us":27,"operation_id":"0xf2f1","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":5585248},{"operation_name":"upsert","last_dispatch_duration_us":2071022,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2071022,"total_server_duration_us":33622,"encode_duration_us":22319,"operation_id":"0xf46d","timeout_ms":10000,"last_server_duration_us":33622,"total_duration_us":2075763},{"operation_name":"upsert","last_dispatch_duration_us":1914418,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1914418,"total_server_duration_us":1648788,"encode_duration_us":18526,"operation_id":"0xf394","timeout_ms":10000,"last_server_duration_us":1648788,"total_duration_us":1921006},{"operation_name":"upsert","last_dispatch_duration_us":298241,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":917232,"total_server_duration_us":245650,"encode_duration_us":18075,"operation_id":"0xf469","timeout_ms":10000,"last_server_duration_us":95789,"total_duration_us":946773},{"operation_name":"get","last_dispatch_duration_us":626601,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":626601,"total_server_duration_us":32,"operation_id":"0xf4d9","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":626797}],"total_count":11}} WARN [2022-03-13 17:16:58,533] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17118761,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17453078,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9232032,"encode_duration_us":20841,"operation_id":"0xef15","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10001464},{"operation_name":"upsert","last_dispatch_duration_us":17551519,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18319690,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9138966,"encode_duration_us":18107,"operation_id":"0xed5f","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10002685}],"total_count":2}} WARN [2022-03-13 17:16:58,533] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9075511,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14224934,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9081129,"encode_duration_us":24142,"operation_id":"0xef6e","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10001259},{"operation_name":"upsert","last_dispatch_duration_us":17990846,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27426610,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9166659,"encode_duration_us":18749,"operation_id":"0xe634","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10007863},{"operation_name":"upsert","last_dispatch_duration_us":17989204,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17989204,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9033121,"encode_duration_us":18725,"operation_id":"0xeb88","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10006367}],"total_count":3}} WARN [2022-03-13 17:16:58,535] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15194229,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":15194229,"total_server_duration_us":38,"operation_id":"0xed86","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":15194544},{"operation_name":"get","last_dispatch_duration_us":12548211,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":12548211,"total_server_duration_us":155,"operation_id":"0xf17e","timeout_ms":30000,"last_server_duration_us":155,"total_duration_us":12561279},{"operation_name":"get","last_dispatch_duration_us":12004579,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":12004579,"total_server_duration_us":32,"operation_id":"0xf1a6","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":12305524},{"operation_name":"get","last_dispatch_duration_us":12033893,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":12033893,"total_server_duration_us":23,"operation_id":"0xf1ae","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":12058495},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17857,"operation_id":"0xf0e9","timeout_ms":10000,"total_duration_us":10011216},{"operation_name":"upsert","last_dispatch_duration_us":50925,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":834509,"total_server_duration_us":355212,"encode_duration_us":25648,"operation_id":"0xeede","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10009124},{"operation_name":"upsert","last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","encode_duration_us":17738,"operation_id":"0xf183","timeout_ms":10000,"total_duration_us":10008895},{"operation_name":"upsert","last_dispatch_duration_us":9125901,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9125901,"total_server_duration_us":27043,"encode_duration_us":17866,"operation_id":"0xf074","timeout_ms":10000,"last_server_duration_us":27043,"total_duration_us":10008684},{"operation_name":"upsert","last_dispatch_duration_us":9096755,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9096755,"total_server_duration_us":25275,"encode_duration_us":21320,"operation_id":"0xf093","timeout_ms":10000,"last_server_duration_us":25275,"total_duration_us":10007059},{"operation_name":"upsert","last_dispatch_duration_us":67549,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":3135913,"total_server_duration_us":921574,"encode_duration_us":16959,"operation_id":"0xf0f8","timeout_ms":10000,"last_server_duration_us":24928,"total_duration_us":10006914}],"total_count":21}} INFO [2022-03-13 17:16:58,568] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,568] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,569] jepsen worker 35 - jepsen.util 935 :invoke :write [11 48] INFO [2022-03-13 17:16:58,569] jepsen worker 80 - jepsen.util 800 :invoke :write [26 19] INFO [2022-03-13 17:16:58,576] jepsen worker 63 - jepsen.util 3753 :info :write [21 44] :RequestTimeoutException INFO [2022-03-13 17:16:58,576] jepsen worker 63 - jepsen.util 3843 :invoke :write [21 22] INFO [2022-03-13 17:16:58,621] jepsen worker 75 - jepsen.util 435 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,622] jepsen worker 75 - jepsen.util 435 :invoke :write [25 14] INFO [2022-03-13 17:16:58,716] jepsen worker 13 - jepsen.util 733 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,716] jepsen worker 43 - jepsen.util 673 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,718] jepsen worker 43 - jepsen.util 673 :invoke :write [14 19] INFO [2022-03-13 17:16:58,718] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:16:58,725] jepsen worker 33 - jepsen.util 753 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,726] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:16:58,860] jepsen worker 71 - jepsen.util 521 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,861] jepsen worker 71 - jepsen.util 521 :invoke :write [23 18] INFO [2022-03-13 17:16:58,868] jepsen worker 52 - jepsen.util 682 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,869] jepsen worker 76 - jepsen.util 796 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,869] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,870] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:16:58,871] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:58,871] jepsen worker 13 - jepsen.util 733 :invoke :write [4 11] INFO [2022-03-13 17:16:58,872] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,873] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:16:58,874] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,875] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:16:58,878] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,878] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:16:58,930] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,930] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,931] jepsen worker 60 - jepsen.util 690 :invoke :write [20 5] INFO [2022-03-13 17:16:58,931] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:16:58,968] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,968] jepsen worker 69 - jepsen.util 789 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:16:58,969] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:16:58,971] jepsen worker 69 - jepsen.util 789 :invoke :write [23 41] INFO [2022-03-13 17:16:59,008] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:59,008] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:16:59,008] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:16:59,009] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:16:59,009] jepsen worker 21 - jepsen.util 831 :invoke :write [7 40] INFO [2022-03-13 17:16:59,011] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:16:59,321] jepsen worker 0 - jepsen.util 3600 :ok :read [0 41] INFO [2022-03-13 17:16:59,323] jepsen worker 0 - jepsen.util 3600 :invoke :write [0 11] INFO [2022-03-13 17:16:59,332] jepsen worker 58 - jepsen.util 3388 :ok :read [19 16] INFO [2022-03-13 17:16:59,333] jepsen worker 58 - jepsen.util 3388 :invoke :read [19 nil] INFO [2022-03-13 17:16:59,410] jepsen worker 46 - jepsen.util 3826 :info :write [15 37] :RequestTimeoutException INFO [2022-03-13 17:16:59,412] jepsen worker 46 - jepsen.util 3916 :invoke :write [15 39] INFO [2022-03-13 17:16:59,473] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3519061583577713}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3519061583577713}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:16:59,473] jepsen worker nemesis - couchbase.util Rebalance check count: 527 INFO [2022-03-13 17:16:59,473] jepsen worker nemesis - couchbase.util Rebalance stuck count: 423 INFO [2022-03-13 17:16:59,949] jepsen worker 65 - jepsen.util 4205 :ok :read [21 :nil] INFO [2022-03-13 17:16:59,949] jepsen worker 65 - jepsen.util 4205 :invoke :read [21 nil] INFO [2022-03-13 17:16:59,964] jepsen worker 47 - jepsen.util 3647 :ok :read [15 48] INFO [2022-03-13 17:16:59,965] jepsen worker 47 - jepsen.util 3647 :invoke :read [15 nil] INFO [2022-03-13 17:17:00,077] jepsen worker 83 - jepsen.util 3323 :info :write [27 12] :RequestTimeoutException INFO [2022-03-13 17:17:00,078] jepsen worker 83 - jepsen.util 3413 :invoke :write [27 37] INFO [2022-03-13 17:17:00,218] jepsen worker 39 - jepsen.util 3909 :ok :read [13 39] INFO [2022-03-13 17:17:00,219] jepsen worker 39 - jepsen.util 3909 :invoke :write [13 0] INFO [2022-03-13 17:17:00,250] jepsen worker 57 - jepsen.util 3747 :ok :read [19 16] INFO [2022-03-13 17:17:00,251] jepsen worker 57 - jepsen.util 3747 :invoke :read [19 nil] INFO [2022-03-13 17:17:00,331] jepsen worker 57 - jepsen.util 3747 :ok :read [19 16] INFO [2022-03-13 17:17:00,332] jepsen worker 57 - jepsen.util 3747 :invoke :read [19 nil] INFO [2022-03-13 17:17:00,368] jepsen worker 57 - jepsen.util 3747 :ok :read [19 16] INFO [2022-03-13 17:17:00,369] jepsen worker 57 - jepsen.util 3747 :invoke :read [19 nil] INFO [2022-03-13 17:17:00,415] jepsen worker 57 - jepsen.util 3747 :ok :read [19 16] INFO [2022-03-13 17:17:00,415] jepsen worker 57 - jepsen.util 3747 :invoke :read [19 nil] INFO [2022-03-13 17:17:00,463] jepsen worker 57 - jepsen.util 3747 :ok :read [19 16] INFO [2022-03-13 17:17:00,464] jepsen worker 57 - jepsen.util 3747 :invoke :write [19 36] INFO [2022-03-13 17:17:00,478] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3519061583577713}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3519061583577713}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:17:00,478] jepsen worker nemesis - couchbase.util Rebalance check count: 528 INFO [2022-03-13 17:17:00,478] jepsen worker nemesis - couchbase.util Rebalance stuck count: 424 INFO [2022-03-13 17:17:00,743] jepsen worker 30 - jepsen.util 3540 :ok :read [10 :nil] INFO [2022-03-13 17:17:00,743] jepsen worker 30 - jepsen.util 3540 :invoke :read [10 nil] INFO [2022-03-13 17:17:00,832] jepsen worker 44 - jepsen.util 674 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,834] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:00,877] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,878] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:00,887] jepsen worker 53 - jepsen.util 593 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,888] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:00,892] jepsen worker 75 - jepsen.util 435 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,892] jepsen worker 75 - jepsen.util 435 :invoke :write [25 10] INFO [2022-03-13 17:17:00,896] jepsen worker 1 - jepsen.util 3241 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:17:00,897] jepsen worker 1 - jepsen.util 3331 :invoke :read [0 nil] INFO [2022-03-13 17:17:00,914] jepsen worker 35 - jepsen.util 935 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,915] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:00,928] jepsen worker 30 - jepsen.util 3540 :ok :read [10 :nil] INFO [2022-03-13 17:17:00,929] jepsen worker 30 - jepsen.util 3540 :invoke :write [10 30] INFO [2022-03-13 17:17:00,954] jepsen worker 80 - jepsen.util 800 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,955] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,955] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,955] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,956] jepsen worker 51 - jepsen.util 771 :invoke :write [17 22] INFO [2022-03-13 17:17:00,957] jepsen worker 80 - jepsen.util 800 :invoke :write [26 36] INFO [2022-03-13 17:17:00,958] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:17:00,962] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:00,988] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:00,989] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:01,016] jepsen worker 71 - jepsen.util 521 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,017] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:01,044] jepsen worker 21 - jepsen.util 831 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,044] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,045] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:01,045] jepsen worker 33 - jepsen.util 753 :invoke :write [11 12] INFO [2022-03-13 17:17:01,046] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:01,046] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,046] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,047] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,047] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:01,048] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:01,049] jepsen worker 34 - jepsen.util 934 :invoke :write [11 14] INFO [2022-03-13 17:17:01,050] jepsen worker 44 - jepsen.util 674 :invoke :write [14 10] INFO [2022-03-13 17:17:01,058] jepsen worker 78 - jepsen.util 618 :info :write [26 6] :RequestTimeoutException INFO [2022-03-13 17:17:01,059] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:01,066] jepsen worker 18 - jepsen.util 3708 :info :write [6 48] :RequestTimeoutException INFO [2022-03-13 17:17:01,067] jepsen worker 18 - jepsen.util 3798 :invoke :read [6 nil] INFO [2022-03-13 17:17:01,134] jepsen worker 69 - jepsen.util 789 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,134] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:01,135] jepsen worker 43 - jepsen.util 673 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,135] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,135] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,135] jepsen worker 43 - jepsen.util 673 :invoke :write [14 3] INFO [2022-03-13 17:17:01,136] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:01,136] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:01,164] jepsen worker 51 - jepsen.util 771 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,166] jepsen worker 51 - jepsen.util 771 :invoke :write [17 1] INFO [2022-03-13 17:17:01,210] jepsen worker 24 - jepsen.util 564 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,211] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:17:01,264] jepsen worker 68 - jepsen.util 3758 :info :write [22 39] :RequestTimeoutException INFO [2022-03-13 17:17:01,265] jepsen worker 68 - jepsen.util 3848 :invoke :read [22 nil] INFO [2022-03-13 17:17:01,268] jepsen worker 13 - jepsen.util 733 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,268] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:01,270] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,271] jepsen worker 76 - jepsen.util 796 :invoke :write [25 25] INFO [2022-03-13 17:17:01,282] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,282] jepsen worker 13 - jepsen.util 733 :invoke :write [4 19] INFO [2022-03-13 17:17:01,283] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,284] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:17:01,341] jepsen worker 75 - jepsen.util 435 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,341] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,341] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,341] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:17:01,342] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:01,342] jepsen worker 75 - jepsen.util 435 :invoke :write [25 31] INFO [2022-03-13 17:17:01,371] jepsen worker 80 - jepsen.util 800 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,371] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,371] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,371] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,372] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:01,372] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:01,373] jepsen worker 21 - jepsen.util 831 :invoke :write [7 49] INFO [2022-03-13 17:17:01,373] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:01,430] jepsen worker 33 - jepsen.util 753 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,430] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:01,437] jepsen worker 60 - jepsen.util 690 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,438] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:01,440] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,440] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,441] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:01,441] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,441] jepsen worker 53 - jepsen.util 593 :invoke :write [17 12] INFO [2022-03-13 17:17:01,442] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:01,458] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,458] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,458] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:17:01,459] jepsen worker 24 - jepsen.util 564 :invoke :write [8 44] INFO [2022-03-13 17:17:01,476] jepsen worker 64 - jepsen.util 3664 :info :write [21 4] :RequestTimeoutException INFO [2022-03-13 17:17:01,476] jepsen worker 64 - jepsen.util 3754 :invoke :write [21 21] INFO [2022-03-13 17:17:01,484] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3519061583577713}, :ns_1@172.23.105.187 {:progress 0.6257309941520468}, :ns_1@172.23.107.227 {:progress 0.3519061583577713}, :ns_1@172.23.107.228 {:progress 0.4176470588235294}} INFO [2022-03-13 17:17:01,484] jepsen worker nemesis - couchbase.util Rebalance check count: 529 INFO [2022-03-13 17:17:01,484] jepsen worker nemesis - couchbase.util Rebalance stuck count: 425 INFO [2022-03-13 17:17:01,488] jepsen worker 34 - jepsen.util 934 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,488] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:01,555] jepsen worker 44 - jepsen.util 674 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,556] jepsen worker 44 - jepsen.util 674 :invoke :write [14 9] INFO [2022-03-13 17:17:01,577] jepsen worker 32 - jepsen.util 3812 :info :write [10 3] :SyncWriteAmbiguous INFO [2022-03-13 17:17:01,577] jepsen worker 32 - jepsen.util 3902 :invoke :write [10 21] INFO [2022-03-13 17:17:01,599] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,600] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:17:01,612] jepsen worker 76 - jepsen.util 796 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,613] jepsen worker 76 - jepsen.util 796 :invoke :write [25 20] INFO [2022-03-13 17:17:01,676] jepsen worker 51 - jepsen.util 771 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,676] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:01,678] jepsen worker 75 - jepsen.util 435 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,679] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:17:01,687] jepsen worker 41 - jepsen.util 3641 :info :write [13 26] :RequestTimeoutException INFO [2022-03-13 17:17:01,688] jepsen worker 41 - jepsen.util 3731 :invoke :read [13 nil] INFO [2022-03-13 17:17:01,693] jepsen worker 74 - jepsen.util 3764 :info :write [24 33] :RequestTimeoutException INFO [2022-03-13 17:17:01,694] jepsen worker 74 - jepsen.util 3854 :invoke :read [24 nil] INFO [2022-03-13 17:17:01,715] jepsen worker 43 - jepsen.util 673 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,715] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,715] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,715] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,715] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:01,716] jepsen worker 43 - jepsen.util 673 :invoke :write [14 2] INFO [2022-03-13 17:17:01,719] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:01,720] jepsen worker 22 - jepsen.util 652 :invoke :write [7 5] INFO [2022-03-13 17:17:01,729] jepsen worker 56 - jepsen.util 3836 :ok :read [18 21] INFO [2022-03-13 17:17:01,730] jepsen worker 56 - jepsen.util 3836 :invoke :read [18 nil] INFO [2022-03-13 17:17:01,800] jepsen worker 56 - jepsen.util 3836 :ok :read [18 21] INFO [2022-03-13 17:17:01,800] jepsen worker 56 - jepsen.util 3836 :invoke :write [18 3] INFO [2022-03-13 17:17:01,816] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,816] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,816] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,816] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,817] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:01,818] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:17:01,818] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,819] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:01,820] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:01,820] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:17:01,829] jepsen worker 13 - jepsen.util 733 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,830] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:01,885] jepsen worker 24 - jepsen.util 564 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,885] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:01,928] jepsen worker 21 - jepsen.util 831 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,928] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,928] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,928] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:01,929] jepsen worker 33 - jepsen.util 753 :invoke :write [11 35] INFO [2022-03-13 17:17:01,929] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:01,951] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,951] jepsen worker 53 - jepsen.util 593 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,951] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,952] jepsen worker 53 - jepsen.util 593 :invoke :write [17 45] INFO [2022-03-13 17:17:01,952] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,952] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:01,953] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:01,953] jepsen worker 60 - jepsen.util 690 :invoke :write [20 32] INFO [2022-03-13 17:17:01,982] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,982] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,983] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,983] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,983] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:01,984] jepsen worker 71 - jepsen.util 521 :invoke :write [23 42] INFO [2022-03-13 17:17:01,984] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:01,985] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:01,987] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,987] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,987] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,988] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:01,988] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:01,989] jepsen worker 69 - jepsen.util 789 :invoke :write [23 41] INFO [2022-03-13 17:17:01,990] jepsen worker 45 - jepsen.util 3915 :info :write [15 0] :RequestTimeoutException INFO [2022-03-13 17:17:01,991] jepsen worker 45 - jepsen.util 4005 :invoke :read [15 nil] INFO [2022-03-13 17:17:01,999] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,999] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:01,999] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:02,001] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,001] jepsen worker 75 - jepsen.util 435 :invoke :write [25 25] INFO [2022-03-13 17:17:02,001] jepsen worker 24 - jepsen.util 564 :invoke :write [8 38] INFO [2022-03-13 17:17:02,040] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,040] jepsen worker 76 - jepsen.util 796 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,040] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:02,041] jepsen worker 13 - jepsen.util 733 :invoke :write [4 48] INFO [2022-03-13 17:17:02,042] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,043] jepsen worker 76 - jepsen.util 796 :invoke :write [25 35] INFO [2022-03-13 17:17:02,060] jepsen worker 44 - jepsen.util 674 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,060] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,062] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,064] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,066] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:02,067] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:17:02,068] jepsen worker 44 - jepsen.util 674 :invoke :write [14 40] INFO [2022-03-13 17:17:02,068] jepsen worker 51 - jepsen.util 771 :invoke :write [17 11] INFO [2022-03-13 17:17:02,106] jepsen worker 69 - jepsen.util 789 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,106] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:02,152] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,153] jepsen worker 14 - jepsen.util 374 :invoke :write [4 32] INFO [2022-03-13 17:17:02,159] jepsen worker 43 - jepsen.util 673 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,160] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:02,170] jepsen worker 54 - jepsen.util 3384 :info :write [18 27] :RequestTimeoutException INFO [2022-03-13 17:17:02,171] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:02,223] jepsen worker 71 - jepsen.util 521 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,223] jepsen worker 71 - jepsen.util 521 :invoke :write [23 12] INFO [2022-03-13 17:17:02,267] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,268] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,268] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,268] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:02,269] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:02,269] jepsen worker 34 - jepsen.util 934 :invoke :write [11 12] INFO [2022-03-13 17:17:02,270] jepsen worker 13 - jepsen.util 733 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,270] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:02,333] jepsen worker 22 - jepsen.util 652 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,334] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,334] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:17:02,335] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:02,344] jepsen worker 24 - jepsen.util 564 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,345] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:02,351] jepsen worker 33 - jepsen.util 753 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,351] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,352] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:02,352] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:02,417] jepsen worker 76 - jepsen.util 796 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,417] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,418] jepsen worker 76 - jepsen.util 796 :invoke :write [25 35] INFO [2022-03-13 17:17:02,419] jepsen worker 13 - jepsen.util 733 :invoke :write [4 28] INFO [2022-03-13 17:17:02,462] jepsen worker 51 - jepsen.util 771 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,462] jepsen worker 51 - jepsen.util 771 :invoke :write [17 38] INFO [2022-03-13 17:17:02,490] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.631578947368421}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4235294117647059}} INFO [2022-03-13 17:17:02,490] jepsen worker nemesis - couchbase.util Rebalance check count: 530 INFO [2022-03-13 17:17:02,490] jepsen worker nemesis - couchbase.util Rebalance stuck count: 425 INFO [2022-03-13 17:17:02,496] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,496] jepsen worker 75 - jepsen.util 435 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,497] jepsen worker 75 - jepsen.util 435 :invoke :write [25 21] INFO [2022-03-13 17:17:02,497] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:02,515] jepsen worker 60 - jepsen.util 690 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,515] jepsen worker 60 - jepsen.util 690 :invoke :write [20 22] INFO [2022-03-13 17:17:02,532] jepsen worker 86 - jepsen.util 3506 :info :write [28 20] :SyncWriteAmbiguous INFO [2022-03-13 17:17:02,532] jepsen worker 86 - jepsen.util 3596 :invoke :write [28 47] INFO [2022-03-13 17:17:02,552] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,552] jepsen worker 71 - jepsen.util 521 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,552] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,553] jepsen worker 24 - jepsen.util 564 :invoke :write [8 48] INFO [2022-03-13 17:17:02,554] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:17:02,555] jepsen worker 71 - jepsen.util 521 :invoke :write [23 15] INFO [2022-03-13 17:17:02,558] jepsen worker 17 - jepsen.util 3797 :info :write [5 22] :RequestTimeoutException INFO [2022-03-13 17:17:02,560] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:02,623] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,624] jepsen worker 35 - jepsen.util 935 :invoke :write [11 15] INFO [2022-03-13 17:17:02,670] jepsen worker 75 - jepsen.util 435 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,670] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:02,724] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:02,724] jepsen worker 53 - jepsen.util 593 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,724] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,724] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:02,725] jepsen worker 52 - jepsen.util 682 :invoke :write [17 44] INFO [2022-03-13 17:17:02,725] jepsen worker 43 - jepsen.util 673 :invoke :write [14 10] INFO [2022-03-13 17:17:02,752] jepsen worker 44 - jepsen.util 674 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,753] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:02,758] jepsen worker 76 - jepsen.util 796 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,758] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:02,764] jepsen worker 19 - jepsen.util 3439 :info :write [6 20] :RequestTimeoutException INFO [2022-03-13 17:17:02,764] jepsen worker 19 - jepsen.util 3529 :invoke :read [6 nil] INFO [2022-03-13 17:17:02,776] jepsen worker 59 - jepsen.util 3749 :info :write [19 20] :SyncWriteAmbiguous INFO [2022-03-13 17:17:02,776] jepsen worker 59 - jepsen.util 3839 :invoke :write [19 0] INFO [2022-03-13 17:17:02,794] jepsen worker 26 - jepsen.util 476 :info :write [8 22] :RequestTimeoutException INFO [2022-03-13 17:17:02,794] jepsen worker 26 - jepsen.util 566 :invoke :write [8 31] INFO [2022-03-13 17:17:02,860] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,860] jepsen worker 22 - jepsen.util 652 :invoke :write [7 47] INFO [2022-03-13 17:17:02,880] jepsen worker 13 - jepsen.util 733 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,881] jepsen worker 13 - jepsen.util 733 :invoke :write [4 25] INFO [2022-03-13 17:17:02,882] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,884] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:02,885] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:02,886] jepsen worker 76 - jepsen.util 796 :invoke :write [25 41] INFO [2022-03-13 17:17:03,366] jepsen worker 9 - jepsen.util 3159 :info :write [3 29] :RequestTimeoutException INFO [2022-03-13 17:17:03,367] jepsen worker 9 - jepsen.util 3249 :invoke :write [3 33] INFO [2022-03-13 17:17:03,461] jepsen worker 66 - jepsen.util 3396 :info :write [22 32] :RequestTimeoutException INFO [2022-03-13 17:17:03,461] jepsen worker 66 - jepsen.util 3486 :invoke :write [22 44] INFO [2022-03-13 17:17:03,484] jepsen worker 67 - jepsen.util 3037 :ok :read [22 30] INFO [2022-03-13 17:17:03,485] jepsen worker 67 - jepsen.util 3037 :invoke :read [22 nil] INFO [2022-03-13 17:17:03,485] jepsen worker 40 - jepsen.util 3820 :ok :read [13 39] INFO [2022-03-13 17:17:03,485] jepsen worker 40 - jepsen.util 3820 :invoke :write [13 45] INFO [2022-03-13 17:17:03,496] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.631578947368421}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4235294117647059}} INFO [2022-03-13 17:17:03,496] jepsen worker nemesis - couchbase.util Rebalance check count: 531 INFO [2022-03-13 17:17:03,496] jepsen worker nemesis - couchbase.util Rebalance stuck count: 426 INFO [2022-03-13 17:17:03,497] jepsen worker 2 - jepsen.util 4052 :ok :read [0 41] INFO [2022-03-13 17:17:03,497] jepsen worker 2 - jepsen.util 4052 :invoke :write [0 45] INFO [2022-03-13 17:17:03,510] jepsen worker 67 - jepsen.util 3037 :ok :read [22 30] INFO [2022-03-13 17:17:03,510] jepsen worker 67 - jepsen.util 3037 :invoke :read [22 nil] INFO [2022-03-13 17:17:03,577] jepsen worker 67 - jepsen.util 3037 :ok :read [22 30] INFO [2022-03-13 17:17:03,577] jepsen worker 67 - jepsen.util 3037 :invoke :read [22 nil] INFO [2022-03-13 17:17:03,645] jepsen worker 67 - jepsen.util 3037 :ok :read [22 30] INFO [2022-03-13 17:17:03,646] jepsen worker 67 - jepsen.util 3037 :invoke :write [22 45] INFO [2022-03-13 17:17:04,297] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,298] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:04,299] jepsen worker 60 - jepsen.util 690 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,300] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,301] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:04,301] jepsen worker 60 - jepsen.util 690 :invoke :write [20 49] INFO [2022-03-13 17:17:04,305] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,306] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:17:04,306] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,306] jepsen worker 14 - jepsen.util 374 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,307] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:04,307] jepsen worker 14 - jepsen.util 374 :invoke :write [4 49] INFO [2022-03-13 17:17:04,404] jepsen worker 34 - jepsen.util 934 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,404] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,405] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:04,405] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,405] jepsen worker 33 - jepsen.util 753 :invoke :write [11 18] INFO [2022-03-13 17:17:04,406] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:04,417] jepsen worker 24 - jepsen.util 564 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,417] jepsen worker 24 - jepsen.util 564 :invoke :write [8 7] INFO [2022-03-13 17:17:04,470] jepsen worker 84 - jepsen.util 4044 :info :write [28 0] :RequestTimeoutException INFO [2022-03-13 17:17:04,471] jepsen worker 84 - jepsen.util 4134 :invoke :write [28 38] INFO [2022-03-13 17:17:04,476] jepsen worker 16 - jepsen.util 3526 :info :write [5 36] :RequestTimeoutException INFO [2022-03-13 17:17:04,477] jepsen worker 16 - jepsen.util 3616 :invoke :read [5 nil] INFO [2022-03-13 17:17:04,485] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,485] jepsen worker 71 - jepsen.util 521 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,486] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:04,486] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:04,487] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,487] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:17:04,502] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.631578947368421}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4235294117647059}} INFO [2022-03-13 17:17:04,502] jepsen worker nemesis - couchbase.util Rebalance check count: 532 INFO [2022-03-13 17:17:04,502] jepsen worker nemesis - couchbase.util Rebalance stuck count: 427 INFO [2022-03-13 17:17:04,515] jepsen worker 51 - jepsen.util 771 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,516] jepsen worker 51 - jepsen.util 771 :invoke :write [17 48] INFO [2022-03-13 17:17:04,642] jepsen worker 35 - jepsen.util 935 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,643] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:17:04,643] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:04,644] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:04,646] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,646] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,647] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,647] jepsen worker 24 - jepsen.util 564 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,647] jepsen worker 75 - jepsen.util 435 :invoke :write [25 38] INFO [2022-03-13 17:17:04,648] jepsen worker 71 - jepsen.util 521 :invoke :write [23 0] INFO [2022-03-13 17:17:04,649] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:04,649] jepsen worker 24 - jepsen.util 564 :invoke :write [8 49] INFO [2022-03-13 17:17:04,663] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,663] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,664] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:04,665] jepsen worker 6 - jepsen.util 636 :invoke :write [2 28] INFO [2022-03-13 17:17:04,697] jepsen worker 85 - jepsen.util 3325 :info :write [28 1] :RequestTimeoutException INFO [2022-03-13 17:17:04,698] jepsen worker 85 - jepsen.util 3415 :invoke :write [28 22] INFO [2022-03-13 17:17:04,731] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,732] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:04,752] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,753] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:04,831] jepsen worker 37 - jepsen.util 2197 :info :write [12 37] :SyncWriteAmbiguous INFO [2022-03-13 17:17:04,832] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 22] INFO [2022-03-13 17:17:04,879] jepsen worker 71 - jepsen.util 521 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,880] jepsen worker 71 - jepsen.util 521 :invoke :write [23 32] INFO [2022-03-13 17:17:04,919] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,919] jepsen worker 61 - jepsen.util 781 :invoke :write [20 37] INFO [2022-03-13 17:17:04,988] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,988] jepsen worker 14 - jepsen.util 374 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,988] jepsen worker 34 - jepsen.util 934 :invoke :write [11 40] INFO [2022-03-13 17:17:04,988] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:04,990] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:04,991] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:04,994] jepsen worker 12 - jepsen.util 822 :info :write [4 9] :RequestTimeoutException INFO [2022-03-13 17:17:04,995] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:05,002] jepsen worker 75 - jepsen.util 435 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,003] jepsen worker 75 - jepsen.util 435 :invoke :write [25 19] INFO [2022-03-13 17:17:05,062] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,062] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:05,062] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,063] jepsen worker 8 - jepsen.util 728 :invoke :write [2 20] INFO [2022-03-13 17:17:05,064] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:05,065] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:05,075] jepsen worker 33 - jepsen.util 753 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,076] jepsen worker 33 - jepsen.util 753 :invoke :write [11 0] INFO [2022-03-13 17:17:05,137] jepsen worker 24 - jepsen.util 564 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,137] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,138] jepsen worker 24 - jepsen.util 564 :invoke :write [8 32] INFO [2022-03-13 17:17:05,138] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,144] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,144] jepsen worker 51 - jepsen.util 771 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,144] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:05,144] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,145] jepsen worker 44 - jepsen.util 674 :invoke :write [14 43] INFO [2022-03-13 17:17:05,146] jepsen worker 51 - jepsen.util 771 :invoke :write [17 48] INFO [2022-03-13 17:17:05,200] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,200] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,201] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,201] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,201] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,202] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:05,203] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:05,204] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:05,258] jepsen worker 70 - jepsen.util 880 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,258] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,259] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,259] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:05,267] jepsen worker 71 - jepsen.util 521 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,268] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,336] jepsen worker 34 - jepsen.util 934 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,336] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,401] jepsen worker 75 - jepsen.util 435 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,401] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,402] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:05,403] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,403] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,403] jepsen worker 33 - jepsen.util 753 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,404] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,404] jepsen worker 33 - jepsen.util 753 :invoke :write [11 37] INFO [2022-03-13 17:17:05,455] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,455] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,455] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,455] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,456] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,456] jepsen worker 78 - jepsen.util 708 :invoke :write [26 36] INFO [2022-03-13 17:17:05,457] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 41] INFO [2022-03-13 17:17:05,457] jepsen worker 77 - jepsen.util 707 :invoke :write [25 7] INFO [2022-03-13 17:17:05,470] jepsen worker 44 - jepsen.util 674 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,472] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:05,495] jepsen worker 24 - jepsen.util 564 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,495] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,495] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,495] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,496] jepsen worker 24 - jepsen.util 564 :invoke :write [8 0] INFO [2022-03-13 17:17:05,496] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:05,497] jepsen worker 71 - jepsen.util 521 :invoke :write [23 39] INFO [2022-03-13 17:17:05,497] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,501] jepsen worker 6 - jepsen.util 636 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,501] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,501] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,501] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:05,501] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:05,502] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:05,503] jepsen worker 12 - jepsen.util 912 :invoke :write [4 19] INFO [2022-03-13 17:17:05,503] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,503] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,504] jepsen worker 62 - jepsen.util 512 :invoke :write [20 11] INFO [2022-03-13 17:17:05,504] jepsen worker 75 - jepsen.util 435 :invoke :write [25 41] INFO [2022-03-13 17:17:05,505] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:17:05,508] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.631578947368421}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4235294117647059}} INFO [2022-03-13 17:17:05,508] jepsen worker nemesis - couchbase.util Rebalance check count: 533 INFO [2022-03-13 17:17:05,508] jepsen worker nemesis - couchbase.util Rebalance stuck count: 428 INFO [2022-03-13 17:17:05,593] jepsen worker 38 - jepsen.util 2558 :info :write [12 49] :RequestTimeoutException INFO [2022-03-13 17:17:05,594] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:05,595] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,595] jepsen worker 51 - jepsen.util 771 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,595] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,596] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,596] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,596] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:05,596] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,596] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,597] jepsen worker 51 - jepsen.util 771 :invoke :write [17 8] INFO [2022-03-13 17:17:05,598] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:05,598] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,599] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:17:05,599] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:05,600] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,640] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,640] jepsen worker 61 - jepsen.util 781 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,640] jepsen worker 23 - jepsen.util 563 :invoke :write [7 27] INFO [2022-03-13 17:17:05,641] jepsen worker 61 - jepsen.util 781 :invoke :write [20 40] INFO [2022-03-13 17:17:05,648] jepsen worker 71 - jepsen.util 521 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,649] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,666] jepsen worker 4 - jepsen.util 2344 :info :write [1 14] :RequestTimeoutException INFO [2022-03-13 17:17:05,667] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 23] INFO [2022-03-13 17:17:05,712] jepsen worker 33 - jepsen.util 753 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,712] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:05,713] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,713] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,713] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,713] jepsen worker 8 - jepsen.util 728 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,713] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,713] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,713] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,713] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,713] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:05,714] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,714] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,714] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:05,715] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:05,716] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:05,717] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,718] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,719] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:05,719] jepsen worker 5 - jepsen.util 2525 :info :write [1 12] :RequestTimeoutException INFO [2022-03-13 17:17:05,720] jepsen worker 89 - jepsen.util 2609 :info :write [29 15] :RequestTimeoutException INFO [2022-03-13 17:17:05,720] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:05,721] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 5] INFO [2022-03-13 17:17:05,722] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:05,724] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:05,724] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:05,724] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,725] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:17:05,728] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,728] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 49] INFO [2022-03-13 17:17:05,762] jepsen worker 24 - jepsen.util 564 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,763] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:05,767] jepsen worker 3 - jepsen.util 2073 :info :write [1 35] :RequestTimeoutException INFO [2022-03-13 17:17:05,767] jepsen worker 36 - jepsen.util 2286 :info :write [12 42] :RequestTimeoutException INFO [2022-03-13 17:17:05,768] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 11] INFO [2022-03-13 17:17:05,768] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:05,826] jepsen worker 87 - jepsen.util 2697 :info :write [29 41] :RequestTimeoutException INFO [2022-03-13 17:17:05,826] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 19] INFO [2022-03-13 17:17:05,831] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,832] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 23] INFO [2022-03-13 17:17:05,840] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,840] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,847] jepsen worker 75 - jepsen.util 435 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,848] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:05,858] jepsen worker 79 - jepsen.util 1159 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,858] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 7] INFO [2022-03-13 17:17:05,895] jepsen worker 51 - jepsen.util 771 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,896] jepsen worker 51 - jepsen.util 771 :invoke :write [17 30] INFO [2022-03-13 17:17:05,896] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,896] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,897] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,897] jepsen worker 80 - jepsen.util 800 :invoke :write [26 4] INFO [2022-03-13 17:17:05,897] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,898] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:05,898] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,898] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,898] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,899] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,900] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:05,900] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:05,900] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,901] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,901] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,901] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,901] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,901] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,901] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,901] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,902] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:05,903] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:17:05,904] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,905] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:17:05,906] jepsen worker 33 - jepsen.util 753 :invoke :write [11 25] INFO [2022-03-13 17:17:05,906] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,906] jepsen worker 71 - jepsen.util 521 :invoke :write [23 8] INFO [2022-03-13 17:17:05,906] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,909] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:05,910] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,911] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,914] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,929] jepsen worker 20 - jepsen.util 3980 :ok :read [6 16] INFO [2022-03-13 17:17:05,930] jepsen worker 20 - jepsen.util 3980 :invoke :write [6 36] INFO [2022-03-13 17:17:05,947] jepsen worker 82 - jepsen.util 3862 :info :write [27 5] :RequestTimeoutException INFO [2022-03-13 17:17:05,947] jepsen worker 82 - jepsen.util 3952 :invoke :write [27 4] INFO [2022-03-13 17:17:05,953] jepsen worker 28 - jepsen.util 3718 :ok :read [9 26] INFO [2022-03-13 17:17:05,954] jepsen worker 28 - jepsen.util 3718 :invoke :write [9 30] INFO [2022-03-13 17:17:05,973] jepsen worker 88 - jepsen.util 2158 :info :write [29 42] :RequestTimeoutException INFO [2022-03-13 17:17:05,974] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:05,976] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,976] jepsen worker 18 - jepsen.util 3798 :ok :read [6 16] INFO [2022-03-13 17:17:05,977] jepsen worker 24 - jepsen.util 564 :invoke :write [8 21] INFO [2022-03-13 17:17:05,977] jepsen worker 18 - jepsen.util 3798 :invoke :read [6 nil] INFO [2022-03-13 17:17:05,982] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:05,983] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,983] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,983] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,984] jepsen worker 44 - jepsen.util 674 :invoke :write [14 10] INFO [2022-03-13 17:17:05,984] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,985] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,988] jepsen worker 12 - jepsen.util 912 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:05,988] jepsen worker 75 - jepsen.util 435 :invoke :write [25 15] INFO [2022-03-13 17:17:05,989] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:05,991] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:05,991] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:05,994] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:05,994] jepsen worker 45 - jepsen.util 4005 :ok :read [15 48] INFO [2022-03-13 17:17:05,995] jepsen worker 34 - jepsen.util 934 :invoke :write [11 31] INFO [2022-03-13 17:17:05,996] jepsen worker 45 - jepsen.util 4005 :invoke :write [15 31] INFO [2022-03-13 17:17:06,093] jepsen worker 18 - jepsen.util 3798 :ok :read [6 16] INFO [2022-03-13 17:17:06,094] jepsen worker 18 - jepsen.util 3798 :invoke :write [6 34] INFO [2022-03-13 17:17:06,099] jepsen worker 71 - jepsen.util 521 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,100] jepsen worker 71 - jepsen.util 521 :invoke :write [23 36] INFO [2022-03-13 17:17:06,101] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,102] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:17:06,120] jepsen worker 29 - jepsen.util 3989 :info :write [9 2] :SyncWriteAmbiguous INFO [2022-03-13 17:17:06,121] jepsen worker 29 - jepsen.util 4079 :invoke :write [9 34] INFO [2022-03-13 17:17:06,128] jepsen worker 80 - jepsen.util 800 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,129] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:06,136] jepsen worker 78 - jepsen.util 708 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,137] jepsen worker 62 - jepsen.util 512 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,138] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,138] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,138] jepsen worker 78 - jepsen.util 708 :invoke :write [26 41] INFO [2022-03-13 17:17:06,138] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,138] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,139] jepsen worker 7 - jepsen.util 637 :invoke :write [2 28] INFO [2022-03-13 17:17:06,139] jepsen worker 70 - jepsen.util 880 :invoke :write [23 49] INFO [2022-03-13 17:17:06,140] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:06,141] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:17:06,143] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:17:06,167] jepsen worker 24 - jepsen.util 564 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,168] jepsen worker 24 - jepsen.util 564 :invoke :write [8 4] INFO [2022-03-13 17:17:06,192] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,193] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:17:06,220] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,221] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:06,222] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,223] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:06,224] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,226] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 34] INFO [2022-03-13 17:17:06,256] jepsen worker 19 - jepsen.util 3529 :ok :read [6 16] INFO [2022-03-13 17:17:06,257] jepsen worker 19 - jepsen.util 3529 :invoke :read [6 nil] INFO [2022-03-13 17:17:06,265] jepsen worker 75 - jepsen.util 435 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,266] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:06,284] jepsen worker 19 - jepsen.util 3529 :ok :read [6 16] INFO [2022-03-13 17:17:06,286] jepsen worker 19 - jepsen.util 3529 :invoke :read [6 nil] INFO [2022-03-13 17:17:06,291] jepsen worker 43 - jepsen.util 673 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,293] jepsen worker 43 - jepsen.util 673 :invoke :write [14 5] INFO [2022-03-13 17:17:06,312] jepsen worker 19 - jepsen.util 3529 :ok :read [6 16] INFO [2022-03-13 17:17:06,312] jepsen worker 19 - jepsen.util 3529 :invoke :read [6 nil] INFO [2022-03-13 17:17:06,338] jepsen worker 19 - jepsen.util 3529 :ok :read [6 16] INFO [2022-03-13 17:17:06,339] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,339] jepsen worker 19 - jepsen.util 3529 :invoke :read [6 nil] INFO [2022-03-13 17:17:06,340] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:17:06,351] jepsen worker 13 - jepsen.util 733 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,351] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:06,359] jepsen worker 71 - jepsen.util 521 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,360] jepsen worker 71 - jepsen.util 521 :invoke :write [23 4] INFO [2022-03-13 17:17:06,367] jepsen worker 77 - jepsen.util 707 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,368] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:06,376] jepsen worker 19 - jepsen.util 3529 :ok :read [6 16] INFO [2022-03-13 17:17:06,376] jepsen worker 19 - jepsen.util 3529 :invoke :write [6 27] INFO [2022-03-13 17:17:06,540] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.631578947368421}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4235294117647059}} INFO [2022-03-13 17:17:06,540] jepsen worker nemesis - couchbase.util Rebalance check count: 534 INFO [2022-03-13 17:17:06,540] jepsen worker nemesis - couchbase.util Rebalance stuck count: 429 INFO [2022-03-13 17:17:06,817] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:17:06,818] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] WARN [2022-03-13 17:17:06,829] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9121811,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16621045,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9149021,"encode_duration_us":22185,"operation_id":"0xf30c","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10007506}],"total_count":1}} WARN [2022-03-13 17:17:06,829] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8750209,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":14626153,"total_server_duration_us":192363,"encode_duration_us":18614,"operation_id":"0xf2a7","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10010199},{"operation_name":"upsert","last_dispatch_duration_us":8756741,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9044195,"total_server_duration_us":112935,"encode_duration_us":18684,"operation_id":"0xf42c","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10007856},{"operation_name":"upsert","last_dispatch_duration_us":9121811,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16621045,"total_server_duration_us":9149021,"encode_duration_us":22185,"operation_id":"0xf30c","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10007506},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":19136,"operation_id":"0xf479","timeout_ms":10000,"total_duration_us":10005971},{"operation_name":"upsert","last_dispatch_duration_us":7222835,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":12565552,"total_server_duration_us":334069,"encode_duration_us":17702,"operation_id":"0xf3cf","timeout_ms":10000,"last_server_duration_us":44389,"total_duration_us":10001075},{"operation_name":"upsert","last_dispatch_duration_us":9244604,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9244604,"total_server_duration_us":9085181,"encode_duration_us":19062,"operation_id":"0xf50f","timeout_ms":10000,"last_server_duration_us":9085181,"total_duration_us":9249502},{"operation_name":"upsert","last_dispatch_duration_us":9114100,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9202920,"total_server_duration_us":9053822,"encode_duration_us":35835,"operation_id":"0xf594","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9222071},{"operation_name":"upsert","last_dispatch_duration_us":9108743,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9108743,"total_server_duration_us":9049045,"encode_duration_us":24986,"operation_id":"0xf425","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":9115812},{"operation_name":"get","last_dispatch_duration_us":4186388,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":4186388,"total_server_duration_us":5674,"operation_id":"0xf3da","timeout_ms":30000,"last_server_duration_us":5674,"total_duration_us":4188986},{"operation_name":"get","last_dispatch_duration_us":3472960,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":3472960,"total_server_duration_us":27,"operation_id":"0xf75e","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":3475969}],"total_count":54}} INFO [2022-03-13 17:17:06,910] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,911] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:06,913] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,914] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,914] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,914] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:06,915] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:06,915] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:06,916] jepsen worker 33 - jepsen.util 753 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,917] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:06,930] jepsen worker 11 - jepsen.util 3611 :info :write [3 31] :RequestTimeoutException INFO [2022-03-13 17:17:06,931] jepsen worker 11 - jepsen.util 3701 :invoke :read [3 nil] INFO [2022-03-13 17:17:06,940] jepsen worker 24 - jepsen.util 564 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,940] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:06,941] jepsen worker 24 - jepsen.util 564 :invoke :write [8 48] INFO [2022-03-13 17:17:06,941] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:07,010] jepsen worker 23 - jepsen.util 563 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,011] jepsen worker 23 - jepsen.util 563 :invoke :write [7 3] INFO [2022-03-13 17:17:07,035] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,035] jepsen worker 71 - jepsen.util 521 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,035] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,035] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:07,035] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:07,036] jepsen worker 71 - jepsen.util 521 :invoke :write [23 15] INFO [2022-03-13 17:17:07,055] jepsen worker 51 - jepsen.util 771 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,055] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,055] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,056] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:07,057] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:07,057] jepsen worker 51 - jepsen.util 771 :invoke :write [17 49] INFO [2022-03-13 17:17:07,076] jepsen worker 7 - jepsen.util 637 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,076] jepsen worker 7 - jepsen.util 637 :invoke :write [2 20] INFO [2022-03-13 17:17:07,096] jepsen worker 24 - jepsen.util 564 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,096] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,097] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,097] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:07,098] jepsen worker 24 - jepsen.util 564 :invoke :write [8 42] INFO [2022-03-13 17:17:07,099] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:07,104] jepsen worker 31 - jepsen.util 3901 :ok :read [10 :nil] INFO [2022-03-13 17:17:07,105] jepsen worker 31 - jepsen.util 3901 :invoke :read [10 nil] INFO [2022-03-13 17:17:07,120] jepsen worker 27 - jepsen.util 3897 :ok :read [9 26] INFO [2022-03-13 17:17:07,120] jepsen worker 27 - jepsen.util 3897 :invoke :read [9 nil] INFO [2022-03-13 17:17:07,145] jepsen worker 44 - jepsen.util 674 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,145] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,146] jepsen worker 80 - jepsen.util 800 :invoke :write [26 16] INFO [2022-03-13 17:17:07,146] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:07,162] jepsen worker 73 - jepsen.util 3853 :ok :read [24 24] INFO [2022-03-13 17:17:07,163] jepsen worker 73 - jepsen.util 3853 :invoke :write [24 9] INFO [2022-03-13 17:17:07,165] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,165] jepsen worker 71 - jepsen.util 521 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,165] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,165] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:07,166] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:17:07,167] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:07,172] jepsen worker 72 - jepsen.util 3852 :ok :read [24 24] INFO [2022-03-13 17:17:07,173] jepsen worker 72 - jepsen.util 3852 :invoke :read [24 nil] INFO [2022-03-13 17:17:07,181] jepsen worker 10 - jepsen.util 3700 :info :write [3 30] :RequestTimeoutException INFO [2022-03-13 17:17:07,181] jepsen worker 10 - jepsen.util 3790 :invoke :write [3 42] INFO [2022-03-13 17:17:07,204] jepsen worker 61 - jepsen.util 781 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,204] jepsen worker 61 - jepsen.util 781 :invoke :write [20 39] INFO [2022-03-13 17:17:07,218] jepsen worker 31 - jepsen.util 3901 :ok :read [10 :nil] INFO [2022-03-13 17:17:07,219] jepsen worker 31 - jepsen.util 3901 :invoke :write [10 48] INFO [2022-03-13 17:17:07,227] jepsen worker 74 - jepsen.util 3854 :ok :read [24 24] INFO [2022-03-13 17:17:07,227] jepsen worker 74 - jepsen.util 3854 :invoke :read [24 nil] INFO [2022-03-13 17:17:07,237] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,238] jepsen worker 62 - jepsen.util 512 :invoke :write [20 30] INFO [2022-03-13 17:17:07,278] jepsen worker 34 - jepsen.util 934 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,278] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:07,281] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,282] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,282] jepsen worker 24 - jepsen.util 564 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,283] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:07,283] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:17:07,284] jepsen worker 71 - jepsen.util 521 :invoke :write [23 13] INFO [2022-03-13 17:17:07,293] jepsen worker 27 - jepsen.util 3897 :ok :read [9 26] INFO [2022-03-13 17:17:07,294] jepsen worker 27 - jepsen.util 3897 :invoke :read [9 nil] INFO [2022-03-13 17:17:07,331] jepsen worker 72 - jepsen.util 3852 :ok :read [24 24] INFO [2022-03-13 17:17:07,331] jepsen worker 72 - jepsen.util 3852 :invoke :read [24 nil] INFO [2022-03-13 17:17:07,339] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,340] jepsen worker 79 - jepsen.util 1159 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,340] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:07,341] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 41] INFO [2022-03-13 17:17:07,401] jepsen worker 74 - jepsen.util 3854 :ok :read [24 24] INFO [2022-03-13 17:17:07,401] jepsen worker 74 - jepsen.util 3854 :invoke :write [24 38] INFO [2022-03-13 17:17:07,435] jepsen worker 27 - jepsen.util 3897 :ok :read [9 26] INFO [2022-03-13 17:17:07,436] jepsen worker 27 - jepsen.util 3897 :invoke :read [9 nil] INFO [2022-03-13 17:17:07,460] jepsen worker 78 - jepsen.util 708 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,460] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:07,461] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,461] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:07,468] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,469] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:07,473] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,473] jepsen worker 72 - jepsen.util 3852 :ok :read [24 24] INFO [2022-03-13 17:17:07,474] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:17:07,475] jepsen worker 72 - jepsen.util 3852 :invoke :read [24 nil] INFO [2022-03-13 17:17:07,475] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,476] jepsen worker 75 - jepsen.util 435 :invoke :write [25 46] INFO [2022-03-13 17:17:07,510] jepsen worker 70 - jepsen.util 880 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,510] jepsen worker 70 - jepsen.util 880 :invoke :write [23 7] INFO [2022-03-13 17:17:07,514] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,514] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:07,514] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,515] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:07,516] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:07,517] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:17:07,541] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,541] jepsen worker 14 - jepsen.util 374 :invoke :write [4 27] INFO [2022-03-13 17:17:07,542] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,542] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:17:07,543] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,543] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,543] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:17:07,544] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:07,546] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.631578947368421}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4235294117647059}} INFO [2022-03-13 17:17:07,546] jepsen worker nemesis - couchbase.util Rebalance check count: 535 INFO [2022-03-13 17:17:07,546] jepsen worker nemesis - couchbase.util Rebalance stuck count: 430 INFO [2022-03-13 17:17:07,578] jepsen worker 27 - jepsen.util 3897 :ok :read [9 26] INFO [2022-03-13 17:17:07,578] jepsen worker 27 - jepsen.util 3897 :invoke :write [9 22] INFO [2022-03-13 17:17:07,610] jepsen worker 72 - jepsen.util 3852 :ok :read [24 24] INFO [2022-03-13 17:17:07,610] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,611] jepsen worker 72 - jepsen.util 3852 :invoke :read [24 nil] INFO [2022-03-13 17:17:07,612] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:07,617] jepsen worker 51 - jepsen.util 771 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,618] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:07,619] jepsen worker 51 - jepsen.util 771 :invoke :write [17 42] INFO [2022-03-13 17:17:07,620] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:07,740] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,740] jepsen worker 80 - jepsen.util 800 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,740] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,740] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,741] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:07,741] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:17:07,742] jepsen worker 34 - jepsen.util 934 :invoke :write [11 29] INFO [2022-03-13 17:17:07,742] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:07,745] jepsen worker 23 - jepsen.util 563 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,746] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:07,804] jepsen worker 71 - jepsen.util 521 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,805] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:17:07,805] jepsen worker 7 - jepsen.util 637 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,806] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:07,884] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,885] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:07,910] jepsen worker 81 - jepsen.util 4311 :info :write [27 34] :RequestTimeoutException INFO [2022-03-13 17:17:07,911] jepsen worker 81 - jepsen.util 4401 :invoke :read [27 nil] INFO [2022-03-13 17:17:07,919] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,919] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:17:07,964] jepsen worker 61 - jepsen.util 781 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:07,964] jepsen worker 61 - jepsen.util 781 :invoke :write [20 44] INFO [2022-03-13 17:17:08,084] jepsen worker 75 - jepsen.util 435 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,084] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:08,086] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,087] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:08,095] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,095] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:08,096] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:08,096] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:08,134] jepsen worker 62 - jepsen.util 512 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,134] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,134] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,135] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,135] jepsen worker 62 - jepsen.util 512 :invoke :write [20 20] INFO [2022-03-13 17:17:08,136] jepsen worker 6 - jepsen.util 636 :invoke :write [2 21] INFO [2022-03-13 17:17:08,136] jepsen worker 8 - jepsen.util 728 :invoke :write [2 38] INFO [2022-03-13 17:17:08,138] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] WARN [2022-03-13 17:17:08,142] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19231968,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19231968,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":73273,"encode_duration_us":23550,"operation_id":"0xf1b8","timeout_ms":10000,"last_server_duration_us":73273,"total_duration_us":10001290},{"operation_name":"upsert","last_dispatch_duration_us":11976809,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11976809,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":111491,"encode_duration_us":32288,"operation_id":"0xf1bb","timeout_ms":10000,"last_server_duration_us":111491,"total_duration_us":10004668},{"operation_name":"upsert","last_dispatch_duration_us":19326133,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19326133,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":61774,"encode_duration_us":24174,"operation_id":"0xf1b9","timeout_ms":10000,"last_server_duration_us":61774,"total_duration_us":10001463},{"operation_name":"upsert","last_dispatch_duration_us":9056173,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17243350,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9242560,"encode_duration_us":18596,"operation_id":"0xf1f7","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10010627},{"operation_name":"upsert","last_dispatch_duration_us":19514042,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19514042,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":45055,"encode_duration_us":28098,"operation_id":"0xf1ba","timeout_ms":10000,"last_server_duration_us":45055,"total_duration_us":10010162},{"operation_name":"upsert","last_dispatch_duration_us":9213466,"last_local_id":"A312307A00000002/0000000012D3A6D4","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":9213466,"last_local_socket":"172.23.104.79:37400","total_server_duration_us":76312,"encode_duration_us":22507,"operation_id":"0xf2bb","timeout_ms":10000,"last_server_duration_us":76312,"total_duration_us":10006174},{"operation_name":"upsert","last_dispatch_duration_us":19076155,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19076155,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9032060,"encode_duration_us":17516,"operation_id":"0xf2b0","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10010801}],"total_count":7}} INFO [2022-03-13 17:17:08,160] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,162] jepsen worker 49 - jepsen.util 3559 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:17:08,163] jepsen worker 71 - jepsen.util 521 :invoke :write [23 6] INFO [2022-03-13 17:17:08,166] jepsen worker 49 - jepsen.util 3559 :invoke :read [16 nil] INFO [2022-03-13 17:17:08,176] jepsen worker 14 - jepsen.util 374 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,177] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] WARN [2022-03-13 17:17:08,248] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11993111,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11993111,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":40270,"encode_duration_us":30722,"operation_id":"0xf3fc","timeout_ms":10000,"last_server_duration_us":40270,"total_duration_us":10001980},{"operation_name":"upsert","last_dispatch_duration_us":18044769,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27531158,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9066121,"encode_duration_us":18193,"operation_id":"0xebc3","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10003465},{"operation_name":"upsert","last_dispatch_duration_us":10070489,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":10070489,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":135047,"encode_duration_us":17113,"operation_id":"0xf4ce","timeout_ms":10000,"last_server_duration_us":135047,"total_duration_us":10007267},{"operation_name":"upsert","last_dispatch_duration_us":16518974,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17353484,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9391517,"encode_duration_us":25648,"operation_id":"0xeede","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10009124},{"operation_name":"upsert","last_dispatch_duration_us":19428223,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19428223,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":83558,"encode_duration_us":39122,"operation_id":"0xf1bc","timeout_ms":10000,"last_server_duration_us":83558,"total_duration_us":10006340},{"operation_name":"upsert","last_dispatch_duration_us":19317465,"last_local_id":"A312307A00000004/00000000C4C33556","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":19317465,"last_local_socket":"172.23.104.79:37436","total_server_duration_us":53974,"encode_duration_us":17738,"operation_id":"0xf183","timeout_ms":10000,"last_server_duration_us":53974,"total_duration_us":10008895}],"total_count":6}} INFO [2022-03-13 17:17:08,257] jepsen worker 55 - jepsen.util 3385 :info :write [18 11] :RequestTimeoutException INFO [2022-03-13 17:17:08,258] jepsen worker 55 - jepsen.util 3475 :invoke :read [18 nil] INFO [2022-03-13 17:17:08,342] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,342] jepsen worker 51 - jepsen.util 771 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,342] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,343] jepsen worker 51 - jepsen.util 771 :invoke :write [17 41] INFO [2022-03-13 17:17:08,344] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:08,344] jepsen worker 35 - jepsen.util 935 :invoke :write [11 3] WARN [2022-03-13 17:17:08,346] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":528,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":18061093,"total_server_duration_us":3020,"operation_id":"0xee7f","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":30010235},{"operation_name":"get","last_dispatch_duration_us":17478955,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":17478955,"total_server_duration_us":32,"operation_id":"0xf1fa","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":17480948},{"operation_name":"get","last_dispatch_duration_us":15212397,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":15212397,"total_server_duration_us":32,"operation_id":"0xf2b7","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":15214707},{"operation_name":"get","last_dispatch_duration_us":13308622,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13308622,"total_server_duration_us":8015,"operation_id":"0xf3ee","timeout_ms":30000,"last_server_duration_us":8015,"total_duration_us":13312317},{"operation_name":"get","last_dispatch_duration_us":12629015,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":12629015,"total_server_duration_us":23,"operation_id":"0xf395","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":12631215},{"operation_name":"upsert","last_dispatch_duration_us":13715749,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13715749,"total_server_duration_us":20124,"encode_duration_us":17611,"operation_id":"0xf3d0","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10007818},{"operation_name":"upsert","last_dispatch_duration_us":83672,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9520118,"total_server_duration_us":153273,"encode_duration_us":43789,"operation_id":"0xf59d","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10005726},{"operation_name":"upsert","last_dispatch_duration_us":105371,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":2295715,"total_server_duration_us":1084352,"encode_duration_us":17963,"operation_id":"0xf528","timeout_ms":10000,"last_server_duration_us":33918,"total_duration_us":10005681},{"operation_name":"upsert","last_dispatch_duration_us":114680,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":2306620,"total_server_duration_us":984114,"encode_duration_us":18064,"operation_id":"0xf529","timeout_ms":10000,"last_server_duration_us":36017,"total_duration_us":10001892},{"operation_name":"get","last_dispatch_duration_us":9054529,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9054529,"total_server_duration_us":10874,"operation_id":"0xf58e","timeout_ms":30000,"last_server_duration_us":10874,"total_duration_us":9067979}],"total_count":32}} INFO [2022-03-13 17:17:08,376] jepsen worker 15 - jepsen.util 3705 :ok :read [5 43] INFO [2022-03-13 17:17:08,378] jepsen worker 15 - jepsen.util 3705 :invoke :write [5 9] INFO [2022-03-13 17:17:08,408] jepsen worker 58 - jepsen.util 3388 :ok :read [19 16] INFO [2022-03-13 17:17:08,409] jepsen worker 58 - jepsen.util 3388 :invoke :read [19 nil] WARN [2022-03-13 17:17:08,431] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":480,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":9042555,"total_server_duration_us":1733689,"operation_id":"0xedd2","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":30003285},{"operation_name":"upsert","last_dispatch_duration_us":8827537,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":11943476,"total_server_duration_us":274096,"encode_duration_us":17036,"operation_id":"0xf2d9","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10007543},{"operation_name":"upsert","last_dispatch_duration_us":8682450,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9530390,"total_server_duration_us":231819,"encode_duration_us":18850,"operation_id":"0xf39d","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10005309},{"operation_name":"upsert","last_dispatch_duration_us":38766,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9621563,"total_server_duration_us":234998,"encode_duration_us":20025,"operation_id":"0xf398","timeout_ms":10000,"last_server_duration_us":24669,"total_duration_us":10004331},{"operation_name":"upsert","last_dispatch_duration_us":8810556,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9528395,"total_server_duration_us":242793,"encode_duration_us":17305,"operation_id":"0xf3a6","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10003172},{"operation_name":"upsert","last_dispatch_duration_us":30352,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":2099167,"total_server_duration_us":676686,"encode_duration_us":19004,"operation_id":"0xf537","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10002129},{"operation_name":"upsert","last_dispatch_duration_us":9075399,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9075399,"total_server_duration_us":9033121,"encode_duration_us":19028,"operation_id":"0xf3d9","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":9079400},{"operation_name":"get","last_dispatch_duration_us":8475925,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8475925,"total_server_duration_us":32,"operation_id":"0xf41b","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":8478344},{"operation_name":"get","last_dispatch_duration_us":119409,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":119409,"total_server_duration_us":43,"operation_id":"0xf475","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5791871},{"operation_name":"get","last_dispatch_duration_us":216531,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":216531,"total_server_duration_us":76,"operation_id":"0xf4e8","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":4876846}],"total_count":26}} WARN [2022-03-13 17:17:08,431] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17245797,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18126515,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9159003,"encode_duration_us":20099,"operation_id":"0xeff7","timeout_ms":10000,"last_server_duration_us":9021452,"total_duration_us":10007673}],"total_count":1}} WARN [2022-03-13 17:17:08,443] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":727688,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":727688,"total_server_duration_us":19,"operation_id":"0xf1ef","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":19666184},{"operation_name":"get","last_dispatch_duration_us":17616491,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17616491,"total_server_duration_us":49,"operation_id":"0xf320","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":17620122},{"operation_name":"get","last_dispatch_duration_us":656970,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":656970,"total_server_duration_us":15,"operation_id":"0xf48b","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":10919600},{"operation_name":"upsert","last_dispatch_duration_us":29584,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":2559211,"total_server_duration_us":951600,"encode_duration_us":26444,"operation_id":"0xf52c","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10009116},{"operation_name":"upsert","last_dispatch_duration_us":8907872,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10542563,"total_server_duration_us":261873,"encode_duration_us":17560,"operation_id":"0xf3e7","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10008522},{"operation_name":"upsert","last_dispatch_duration_us":33836,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":4079709,"total_server_duration_us":215336,"encode_duration_us":20324,"operation_id":"0xf355","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10008522},{"operation_name":"upsert","last_dispatch_duration_us":9213466,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9213466,"total_server_duration_us":76312,"encode_duration_us":22507,"operation_id":"0xf2bb","timeout_ms":10000,"last_server_duration_us":76312,"total_duration_us":10006174},{"operation_name":"upsert","encode_duration_us":24364,"operation_id":"0xf39c","timeout_ms":10000,"total_duration_us":10003922},{"operation_name":"upsert","last_dispatch_duration_us":51614,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":2714804,"total_server_duration_us":884864,"encode_duration_us":20947,"operation_id":"0xf52e","timeout_ms":10000,"last_server_duration_us":28952,"total_duration_us":10002116},{"operation_name":"get","last_dispatch_duration_us":9121409,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":9121409,"total_server_duration_us":32,"operation_id":"0xf4dd","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9687878}],"total_count":27}} INFO [2022-03-13 17:17:08,467] jepsen worker 68 - jepsen.util 3848 :ok :read [22 30] INFO [2022-03-13 17:17:08,468] jepsen worker 68 - jepsen.util 3848 :invoke :write [22 39] INFO [2022-03-13 17:17:08,524] jepsen worker 58 - jepsen.util 3388 :ok :read [19 16] INFO [2022-03-13 17:17:08,524] jepsen worker 58 - jepsen.util 3388 :invoke :write [19 47] WARN [2022-03-13 17:17:08,546] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18179993,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27352617,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9329147,"encode_duration_us":20149,"operation_id":"0xed63","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10006000},{"operation_name":"upsert","last_dispatch_duration_us":17608570,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26154269,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9316639,"encode_duration_us":27937,"operation_id":"0xeeb6","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10008305}],"total_count":2}} WARN [2022-03-13 17:17:08,546] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9150603,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9150603,"total_server_duration_us":7180,"operation_id":"0xf290","timeout_ms":30000,"last_server_duration_us":7180,"total_duration_us":12277706},{"operation_name":"upsert","last_dispatch_duration_us":9035315,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10264498,"total_server_duration_us":9464781,"encode_duration_us":22125,"operation_id":"0xf47a","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009126},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":20710,"operation_id":"0xf421","timeout_ms":10000,"total_duration_us":10008444},{"operation_name":"upsert","last_dispatch_duration_us":31080,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":2034942,"total_server_duration_us":724473,"encode_duration_us":18310,"operation_id":"0xf523","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10007483},{"operation_name":"upsert","last_dispatch_duration_us":34960,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":689396,"total_server_duration_us":379307,"encode_duration_us":20326,"operation_id":"0xf53f","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10005516},{"operation_name":"upsert","last_dispatch_duration_us":8708619,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12555512,"total_server_duration_us":512796,"encode_duration_us":17302,"operation_id":"0xf3c6","timeout_ms":10000,"last_server_duration_us":24669,"total_duration_us":10002162},{"operation_name":"upsert","last_dispatch_duration_us":31101,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1033964,"total_server_duration_us":415160,"encode_duration_us":17419,"operation_id":"0xf4b4","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10001990},{"operation_name":"get","last_dispatch_duration_us":7460702,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7460702,"total_server_duration_us":621,"operation_id":"0xf3d8","timeout_ms":30000,"last_server_duration_us":621,"total_duration_us":7460973},{"operation_name":"upsert","last_dispatch_duration_us":3447144,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3447144,"total_server_duration_us":57385,"encode_duration_us":18224,"operation_id":"0xf769","timeout_ms":10000,"last_server_duration_us":57385,"total_duration_us":3450855},{"operation_name":"upsert","last_dispatch_duration_us":2361384,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2361384,"total_server_duration_us":125946,"encode_duration_us":27012,"operation_id":"0xf641","timeout_ms":10000,"last_server_duration_us":125946,"total_duration_us":2368885}],"total_count":14}} WARN [2022-03-13 17:17:08,547] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16893751,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17009235,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9107630,"encode_duration_us":18274,"operation_id":"0xedfa","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10008197},{"operation_name":"upsert","last_dispatch_duration_us":17031394,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26033547,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9282190,"encode_duration_us":18293,"operation_id":"0xebbb","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10014971},{"operation_name":"upsert","last_dispatch_duration_us":9035315,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10264498,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9464781,"encode_duration_us":22125,"operation_id":"0xf47a","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009126}],"total_count":3}} WARN [2022-03-13 17:17:08,549] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17279259,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":17279259,"total_server_duration_us":69,"operation_id":"0xf1f0","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":18873373},{"operation_name":"get","last_dispatch_duration_us":16212763,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":16212763,"total_server_duration_us":23,"operation_id":"0xf2bf","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":16212968},{"operation_name":"get","last_dispatch_duration_us":16021877,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16021877,"total_server_duration_us":38,"operation_id":"0xf3d1","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":16023924},{"operation_name":"get","last_dispatch_duration_us":13253626,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13253626,"total_server_duration_us":62,"operation_id":"0xf205","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":13256311},{"operation_name":"get","last_dispatch_duration_us":11007118,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":11007118,"total_server_duration_us":49,"operation_id":"0xf2ba","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":11008277},{"operation_name":"upsert","last_dispatch_duration_us":54707,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2472402,"total_server_duration_us":918753,"encode_duration_us":17872,"operation_id":"0xf542","timeout_ms":10000,"last_server_duration_us":43948,"total_duration_us":10008370},{"operation_name":"upsert","last_dispatch_duration_us":17092752,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17092752,"total_server_duration_us":9032060,"encode_duration_us":18043,"operation_id":"0xf3a3","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10008354},{"operation_name":"upsert","last_dispatch_duration_us":10070489,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":10070489,"total_server_duration_us":135047,"encode_duration_us":17113,"operation_id":"0xf4ce","timeout_ms":10000,"last_server_duration_us":135047,"total_duration_us":10007267},{"operation_name":"upsert","last_dispatch_duration_us":72734,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":72734,"total_server_duration_us":20441,"encode_duration_us":19793,"operation_id":"0xf3f7","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10004522},{"operation_name":"upsert","last_dispatch_duration_us":53009,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2483204,"total_server_duration_us":800510,"encode_duration_us":27795,"operation_id":"0xf516","timeout_ms":10000,"last_server_duration_us":29695,"total_duration_us":10004421}],"total_count":38}} INFO [2022-03-13 17:17:08,551] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6432748538011697}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4352941176470588}} INFO [2022-03-13 17:17:08,552] jepsen worker nemesis - couchbase.util Rebalance check count: 536 INFO [2022-03-13 17:17:08,552] jepsen worker nemesis - couchbase.util Rebalance stuck count: 430 INFO [2022-03-13 17:17:08,580] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:08,581] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:08,604] jepsen worker 63 - jepsen.util 3843 :info :write [21 22] :RequestTimeoutException INFO [2022-03-13 17:17:08,604] jepsen worker 63 - jepsen.util 3933 :invoke :write [21 48] INFO [2022-03-13 17:17:08,654] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:08,654] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:08,743] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:08,744] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:08,802] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:08,803] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:08,868] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:08,868] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:08,926] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:08,926] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:08,988] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,988] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,989] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,989] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,989] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:08,990] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:08,991] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:08,991] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:17:08,996] jepsen worker 79 - jepsen.util 1159 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,996] jepsen worker 70 - jepsen.util 880 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,996] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,996] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,996] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:08,998] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 46] INFO [2022-03-13 17:17:08,999] jepsen worker 37 - jepsen.util 2287 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,000] jepsen worker 70 - jepsen.util 880 :invoke :write [23 33] INFO [2022-03-13 17:17:09,001] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:09,001] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:09,002] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:17:09,003] jepsen worker 17 - jepsen.util 3887 :invoke :read [5 nil] INFO [2022-03-13 17:17:09,003] jepsen worker 23 - jepsen.util 563 :invoke :write [7 8] INFO [2022-03-13 17:17:09,004] jepsen worker 78 - jepsen.util 708 :invoke :write [26 45] INFO [2022-03-13 17:17:09,011] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,021] jepsen worker 65 - jepsen.util 4205 :ok :read [21 :nil] INFO [2022-03-13 17:17:09,021] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:09,022] jepsen worker 65 - jepsen.util 4205 :invoke :read [21 nil] INFO [2022-03-13 17:17:09,023] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,027] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 12] INFO [2022-03-13 17:17:09,031] jepsen worker 5 - jepsen.util 2615 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,033] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 6] INFO [2022-03-13 17:17:09,050] jepsen worker 47 - jepsen.util 3647 :ok :read [15 48] INFO [2022-03-13 17:17:09,051] jepsen worker 47 - jepsen.util 3647 :invoke :read [15 nil] INFO [2022-03-13 17:17:09,054] jepsen worker 17 - jepsen.util 3887 :ok :read [5 43] INFO [2022-03-13 17:17:09,055] jepsen worker 17 - jepsen.util 3887 :invoke :write [5 13] INFO [2022-03-13 17:17:09,088] jepsen worker 65 - jepsen.util 4205 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,089] jepsen worker 65 - jepsen.util 4205 :invoke :read [21 nil] INFO [2022-03-13 17:17:09,110] jepsen worker 55 - jepsen.util 3475 :ok :read [18 21] INFO [2022-03-13 17:17:09,110] jepsen worker 55 - jepsen.util 3475 :invoke :write [18 0] INFO [2022-03-13 17:17:09,117] jepsen worker 65 - jepsen.util 4205 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,118] jepsen worker 65 - jepsen.util 4205 :invoke :write [21 17] INFO [2022-03-13 17:17:09,134] jepsen worker 71 - jepsen.util 521 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,135] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,135] jepsen worker 61 - jepsen.util 781 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,136] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,136] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:09,136] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,136] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:09,136] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,138] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:09,138] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,138] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,138] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,138] jepsen worker 34 - jepsen.util 934 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,139] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:09,139] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,139] jepsen worker 47 - jepsen.util 3647 :ok :read [15 48] INFO [2022-03-13 17:17:09,140] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:09,142] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:09,143] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:09,143] jepsen worker 61 - jepsen.util 781 :invoke :write [20 8] INFO [2022-03-13 17:17:09,144] jepsen worker 69 - jepsen.util 789 :invoke :write [23 6] INFO [2022-03-13 17:17:09,145] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,145] jepsen worker 47 - jepsen.util 3647 :invoke :write [15 15] INFO [2022-03-13 17:17:09,146] jepsen worker 33 - jepsen.util 753 :invoke :write [11 41] INFO [2022-03-13 17:17:09,147] jepsen worker 75 - jepsen.util 435 :invoke :write [25 20] INFO [2022-03-13 17:17:09,147] jepsen worker 71 - jepsen.util 521 :invoke :write [23 2] INFO [2022-03-13 17:17:09,147] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:09,150] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:09,157] jepsen worker 5 - jepsen.util 2615 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,158] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 33] INFO [2022-03-13 17:17:09,158] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,159] jepsen worker 24 - jepsen.util 564 :invoke :write [8 10] INFO [2022-03-13 17:17:09,167] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,167] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,168] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,169] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:09,171] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:09,171] jepsen worker 37 - jepsen.util 2287 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,172] jepsen worker 12 - jepsen.util 912 :invoke :write [4 24] INFO [2022-03-13 17:17:09,172] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 8] INFO [2022-03-13 17:17:09,203] jepsen worker 88 - jepsen.util 2248 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,203] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:09,211] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,214] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:09,215] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,216] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:09,218] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,219] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 1] INFO [2022-03-13 17:17:09,250] jepsen worker 51 - jepsen.util 771 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,251] jepsen worker 51 - jepsen.util 771 :invoke :write [17 47] INFO [2022-03-13 17:17:09,273] jepsen worker 75 - jepsen.util 435 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,274] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:09,277] jepsen worker 36 - jepsen.util 2376 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,278] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 33] INFO [2022-03-13 17:17:09,307] jepsen worker 5 - jepsen.util 2615 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,308] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:09,310] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,311] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 34] INFO [2022-03-13 17:17:09,354] jepsen worker 0 - jepsen.util 3600 :info :write [0 11] :RequestTimeoutException INFO [2022-03-13 17:17:09,357] jepsen worker 39 - jepsen.util 3909 :info :write [13 0] :SyncWriteAmbiguous INFO [2022-03-13 17:17:09,358] jepsen worker 0 - jepsen.util 3690 :invoke :read [0 nil] INFO [2022-03-13 17:17:09,359] jepsen worker 39 - jepsen.util 3999 :invoke :write [13 18] INFO [2022-03-13 17:17:09,359] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,359] jepsen worker 62 - jepsen.util 512 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,359] jepsen worker 35 - jepsen.util 935 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,359] jepsen worker 62 - jepsen.util 512 :invoke :write [20 47] INFO [2022-03-13 17:17:09,360] jepsen worker 77 - jepsen.util 707 :invoke :write [25 10] INFO [2022-03-13 17:17:09,361] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:17:09,361] jepsen worker 69 - jepsen.util 789 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,362] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:17:09,436] jepsen worker 46 - jepsen.util 3916 :info :write [15 39] :RequestTimeoutException INFO [2022-03-13 17:17:09,437] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:09,448] jepsen worker 24 - jepsen.util 564 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,448] jepsen worker 24 - jepsen.util 564 :invoke :read [8 nil] INFO [2022-03-13 17:17:09,488] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,489] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:09,489] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,489] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,489] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:17:09,489] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,489] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,491] jepsen worker 14 - jepsen.util 374 :invoke :write [4 39] INFO [2022-03-13 17:17:09,492] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:09,492] jepsen worker 34 - jepsen.util 934 :invoke :write [11 23] INFO [2022-03-13 17:17:09,493] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:09,494] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:09,531] jepsen worker 8 - jepsen.util 728 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,532] jepsen worker 8 - jepsen.util 728 :invoke :write [2 26] INFO [2022-03-13 17:17:09,534] jepsen worker 23 - jepsen.util 563 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,534] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:09,538] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,538] jepsen worker 71 - jepsen.util 521 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,539] jepsen worker 75 - jepsen.util 435 :invoke :read [25 nil] INFO [2022-03-13 17:17:09,539] jepsen worker 71 - jepsen.util 521 :invoke :read [23 nil] INFO [2022-03-13 17:17:09,562] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6432748538011697}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4352941176470588}} INFO [2022-03-13 17:17:09,562] jepsen worker nemesis - couchbase.util Rebalance check count: 537 INFO [2022-03-13 17:17:09,562] jepsen worker nemesis - couchbase.util Rebalance stuck count: 431 INFO [2022-03-13 17:17:09,601] jepsen worker 33 - jepsen.util 753 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,602] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:09,658] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,658] jepsen worker 75 - jepsen.util 435 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,658] jepsen worker 71 - jepsen.util 521 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,658] jepsen worker 24 - jepsen.util 564 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:09,659] jepsen worker 75 - jepsen.util 435 :invoke :write [25 39] INFO [2022-03-13 17:17:09,660] jepsen worker 24 - jepsen.util 564 :invoke :write [8 46] INFO [2022-03-13 17:17:09,660] jepsen worker 71 - jepsen.util 521 :invoke :write [23 33] INFO [2022-03-13 17:17:09,663] jepsen worker 69 - jepsen.util 789 :invoke :write [23 48] INFO [2022-03-13 17:17:10,026] jepsen worker 36 - jepsen.util 2376 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,027] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:10,043] jepsen worker 5 - jepsen.util 2615 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,044] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 13] INFO [2022-03-13 17:17:10,052] jepsen worker 88 - jepsen.util 2248 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,053] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:10,063] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,064] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:10,066] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,066] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:10,075] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,075] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:10,077] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,077] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:10,083] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,084] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 31] INFO [2022-03-13 17:17:10,097] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,097] jepsen worker 37 - jepsen.util 2287 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,097] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:10,098] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:10,101] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,102] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,102] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:10,103] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 29] INFO [2022-03-13 17:17:10,106] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,107] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:10,108] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,109] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 43] INFO [2022-03-13 17:17:10,117] jepsen worker 83 - jepsen.util 3413 :info :write [27 37] :RequestTimeoutException INFO [2022-03-13 17:17:10,118] jepsen worker 83 - jepsen.util 3503 :invoke :read [27 nil] INFO [2022-03-13 17:17:10,123] jepsen worker 3 - jepsen.util 2163 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,124] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 20] INFO [2022-03-13 17:17:10,179] jepsen worker 5 - jepsen.util 2615 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,180] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 4] INFO [2022-03-13 17:17:10,187] jepsen worker 37 - jepsen.util 2287 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,188] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 36] INFO [2022-03-13 17:17:10,198] jepsen worker 88 - jepsen.util 2248 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,203] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 46] INFO [2022-03-13 17:17:10,242] jepsen worker 54 - jepsen.util 3474 :ok :read [18 21] INFO [2022-03-13 17:17:10,242] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:10,272] jepsen worker 36 - jepsen.util 2376 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,272] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 1] INFO [2022-03-13 17:17:10,278] jepsen worker 5 - jepsen.util 2615 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,279] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 47] INFO [2022-03-13 17:17:10,281] jepsen worker 3 - jepsen.util 2163 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,282] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 12] INFO [2022-03-13 17:17:10,322] jepsen worker 88 - jepsen.util 2248 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,322] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:10,326] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,326] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:10,327] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,328] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 19] INFO [2022-03-13 17:17:10,347] jepsen worker 37 - jepsen.util 2287 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,351] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 0] INFO [2022-03-13 17:17:10,416] jepsen worker 75 - jepsen.util 435 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,417] jepsen worker 75 - jepsen.util 435 :invoke :write [25 9] INFO [2022-03-13 17:17:10,427] jepsen worker 51 - jepsen.util 771 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,429] jepsen worker 51 - jepsen.util 771 :invoke :write [17 28] INFO [2022-03-13 17:17:10,464] jepsen worker 69 - jepsen.util 789 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,464] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:10,480] jepsen worker 79 - jepsen.util 1159 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,480] jepsen worker 6 - jepsen.util 636 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,480] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 46] INFO [2022-03-13 17:17:10,481] jepsen worker 6 - jepsen.util 636 :invoke :write [2 45] INFO [2022-03-13 17:17:10,483] jepsen worker 36 - jepsen.util 2376 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,484] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 13] INFO [2022-03-13 17:17:10,486] jepsen worker 57 - jepsen.util 3747 :info :write [19 36] :RequestTimeoutException INFO [2022-03-13 17:17:10,487] jepsen worker 57 - jepsen.util 3837 :invoke :write [19 31] INFO [2022-03-13 17:17:10,488] jepsen worker 5 - jepsen.util 2615 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,488] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:10,489] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,490] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 26] INFO [2022-03-13 17:17:10,537] jepsen worker 52 - jepsen.util 682 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,537] jepsen worker 52 - jepsen.util 682 :invoke :write [17 35] INFO [2022-03-13 17:17:10,564] jepsen worker 78 - jepsen.util 708 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,565] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:10,567] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6549707602339181}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4470588235294117}} INFO [2022-03-13 17:17:10,568] jepsen worker nemesis - couchbase.util Rebalance check count: 538 INFO [2022-03-13 17:17:10,568] jepsen worker nemesis - couchbase.util Rebalance stuck count: 431 INFO [2022-03-13 17:17:10,568] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,569] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,569] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,569] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:10,569] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:10,570] jepsen worker 80 - jepsen.util 800 :invoke :write [26 30] INFO [2022-03-13 17:17:10,571] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:17:10,574] jepsen worker 44 - jepsen.util 674 :invoke :write [14 10] INFO [2022-03-13 17:17:10,584] jepsen worker 76 - jepsen.util 796 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,585] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:10,588] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,589] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,589] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:10,590] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:10,590] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,591] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,591] jepsen worker 13 - jepsen.util 733 :invoke :write [4 30] INFO [2022-03-13 17:17:10,591] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,591] jepsen worker 70 - jepsen.util 880 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,591] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:10,592] jepsen worker 76 - jepsen.util 796 :invoke :write [25 34] INFO [2022-03-13 17:17:10,593] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:10,593] jepsen worker 42 - jepsen.util 402 :invoke :write [14 47] INFO [2022-03-13 17:17:10,594] jepsen worker 70 - jepsen.util 880 :invoke :write [23 28] INFO [2022-03-13 17:17:10,660] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,660] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:17:10,678] jepsen worker 3 - jepsen.util 2163 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,679] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:10,680] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,681] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:10,683] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,683] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 17] INFO [2022-03-13 17:17:10,710] jepsen worker 37 - jepsen.util 2287 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,710] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:10,718] jepsen worker 26 - jepsen.util 566 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,719] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:10,746] jepsen worker 14 - jepsen.util 374 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,747] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:17:10,753] jepsen worker 12 - jepsen.util 912 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,753] jepsen worker 61 - jepsen.util 781 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,753] jepsen worker 61 - jepsen.util 781 :invoke :write [20 9] INFO [2022-03-13 17:17:10,754] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:10,784] jepsen worker 13 - jepsen.util 733 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,784] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:17:10,837] jepsen worker 22 - jepsen.util 652 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,837] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,838] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:10,838] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:10,847] jepsen worker 34 - jepsen.util 934 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,847] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,848] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:10,848] jepsen worker 34 - jepsen.util 934 :invoke :write [11 21] INFO [2022-03-13 17:17:10,895] jepsen worker 77 - jepsen.util 707 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,895] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:10,896] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,896] jepsen worker 62 - jepsen.util 512 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,897] jepsen worker 23 - jepsen.util 563 :invoke :write [7 14] INFO [2022-03-13 17:17:10,897] jepsen worker 62 - jepsen.util 512 :invoke :write [20 47] INFO [2022-03-13 17:17:10,953] jepsen worker 30 - jepsen.util 3540 :info :write [10 30] :RequestTimeoutException INFO [2022-03-13 17:17:10,954] jepsen worker 30 - jepsen.util 3630 :invoke :write [10 31] INFO [2022-03-13 17:17:10,960] jepsen worker 76 - jepsen.util 796 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,960] jepsen worker 76 - jepsen.util 796 :invoke :write [25 22] INFO [2022-03-13 17:17:10,971] jepsen worker 51 - jepsen.util 771 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,972] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:10,972] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:17:10,973] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:11,014] jepsen worker 60 - jepsen.util 690 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,015] jepsen worker 60 - jepsen.util 690 :invoke :write [20 39] INFO [2022-03-13 17:17:11,086] jepsen worker 5 - jepsen.util 2615 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,087] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:11,087] jepsen worker 88 - jepsen.util 2248 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,088] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:11,098] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,098] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 30] INFO [2022-03-13 17:17:11,130] jepsen worker 89 - jepsen.util 2699 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,130] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,131] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 36] INFO [2022-03-13 17:17:11,131] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:11,143] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,143] jepsen worker 13 - jepsen.util 733 :invoke :write [4 40] INFO [2022-03-13 17:17:11,147] jepsen worker 80 - jepsen.util 800 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,148] jepsen worker 80 - jepsen.util 800 :invoke :write [26 34] INFO [2022-03-13 17:17:11,290] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,290] jepsen worker 3 - jepsen.util 2163 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,291] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 20] INFO [2022-03-13 17:17:11,291] jepsen worker 43 - jepsen.util 673 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,292] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:11,292] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 1] INFO [2022-03-13 17:17:11,314] jepsen worker 79 - jepsen.util 1159 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,315] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,315] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:11,316] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:11,316] jepsen worker 8 - jepsen.util 728 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,318] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:11,341] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,342] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:17:11,464] jepsen worker 76 - jepsen.util 796 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,464] jepsen worker 76 - jepsen.util 796 :invoke :write [25 3] INFO [2022-03-13 17:17:11,496] jepsen worker 52 - jepsen.util 682 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,496] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,496] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,496] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,496] jepsen worker 36 - jepsen.util 2376 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,496] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,497] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:11,498] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:11,498] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:11,499] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 25] INFO [2022-03-13 17:17:11,499] jepsen worker 52 - jepsen.util 682 :invoke :write [17 49] INFO [2022-03-13 17:17:11,500] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 13] INFO [2022-03-13 17:17:11,505] jepsen worker 64 - jepsen.util 3754 :info :write [21 21] :RequestTimeoutException INFO [2022-03-13 17:17:11,507] jepsen worker 64 - jepsen.util 3844 :invoke :write [21 40] INFO [2022-03-13 17:17:11,573] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6549707602339181}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4470588235294117}} INFO [2022-03-13 17:17:11,573] jepsen worker nemesis - couchbase.util Rebalance check count: 539 INFO [2022-03-13 17:17:11,573] jepsen worker nemesis - couchbase.util Rebalance stuck count: 432 INFO [2022-03-13 17:17:11,606] jepsen worker 32 - jepsen.util 3902 :info :write [10 21] :RequestTimeoutException INFO [2022-03-13 17:17:11,607] jepsen worker 32 - jepsen.util 3992 :invoke :write [10 22] INFO [2022-03-13 17:17:11,640] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,640] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:11,647] jepsen worker 61 - jepsen.util 781 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,647] jepsen worker 6 - jepsen.util 636 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,647] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,648] jepsen worker 77 - jepsen.util 707 :invoke :write [25 23] INFO [2022-03-13 17:17:11,649] jepsen worker 6 - jepsen.util 636 :invoke :write [2 21] INFO [2022-03-13 17:17:11,649] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:11,650] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:11,651] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:11,707] jepsen worker 1 - jepsen.util 3331 :ok :read [0 41] INFO [2022-03-13 17:17:11,707] jepsen worker 1 - jepsen.util 3331 :invoke :write [0 19] INFO [2022-03-13 17:17:11,719] jepsen worker 41 - jepsen.util 3731 :ok :read [13 39] INFO [2022-03-13 17:17:11,720] jepsen worker 41 - jepsen.util 3731 :invoke :write [13 6] INFO [2022-03-13 17:17:11,751] jepsen worker 16 - jepsen.util 3616 :ok :read [5 43] INFO [2022-03-13 17:17:11,752] jepsen worker 16 - jepsen.util 3616 :invoke :write [5 48] INFO [2022-03-13 17:17:11,826] jepsen worker 56 - jepsen.util 3836 :info :write [18 3] :RequestTimeoutException INFO [2022-03-13 17:17:11,827] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 5] INFO [2022-03-13 17:17:12,551] jepsen worker 0 - jepsen.util 3690 :ok :read [0 41] INFO [2022-03-13 17:17:12,551] jepsen worker 0 - jepsen.util 3690 :invoke :read [0 nil] INFO [2022-03-13 17:17:12,555] jepsen worker 86 - jepsen.util 3596 :info :write [28 47] :RequestTimeoutException INFO [2022-03-13 17:17:12,556] jepsen worker 86 - jepsen.util 3686 :invoke :read [28 nil] INFO [2022-03-13 17:17:12,579] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6549707602339181}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4470588235294117}} INFO [2022-03-13 17:17:12,579] jepsen worker nemesis - couchbase.util Rebalance check count: 540 INFO [2022-03-13 17:17:12,579] jepsen worker nemesis - couchbase.util Rebalance stuck count: 433 INFO [2022-03-13 17:17:12,590] jepsen worker 2 - jepsen.util 4052 :info :write [0 45] :SyncWriteAmbiguous INFO [2022-03-13 17:17:12,590] jepsen worker 2 - jepsen.util 4142 :invoke :read [0 nil] INFO [2022-03-13 17:17:12,619] jepsen worker 0 - jepsen.util 3690 :ok :read [0 41] INFO [2022-03-13 17:17:12,620] jepsen worker 0 - jepsen.util 3690 :invoke :write [0 5] INFO [2022-03-13 17:17:12,632] jepsen worker 2 - jepsen.util 4142 :ok :read [0 41] INFO [2022-03-13 17:17:12,632] jepsen worker 2 - jepsen.util 4142 :invoke :read [0 nil] INFO [2022-03-13 17:17:12,807] jepsen worker 59 - jepsen.util 3839 :info :write [19 0] :RequestTimeoutException INFO [2022-03-13 17:17:12,808] jepsen worker 59 - jepsen.util 3929 :invoke :read [19 nil] INFO [2022-03-13 17:17:12,939] jepsen worker 13 - jepsen.util 733 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:12,940] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:17:12,949] jepsen worker 89 - jepsen.util 2699 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:12,950] jepsen worker 89 - jepsen.util 2699 :invoke :read [29 nil] INFO [2022-03-13 17:17:12,950] jepsen worker 5 - jepsen.util 2615 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:12,951] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:12,960] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:12,961] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 7] INFO [2022-03-13 17:17:12,965] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:12,966] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:13,021] jepsen worker 70 - jepsen.util 880 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,021] jepsen worker 23 - jepsen.util 563 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,021] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,021] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,021] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:13,022] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,022] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:13,022] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:13,023] jepsen worker 78 - jepsen.util 708 :invoke :write [26 9] INFO [2022-03-13 17:17:13,023] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:13,040] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,041] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:17:13,044] jepsen worker 60 - jepsen.util 690 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,044] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:13,044] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,044] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,045] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:13,045] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,045] jepsen worker 26 - jepsen.util 566 :invoke :write [8 41] INFO [2022-03-13 17:17:13,046] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:13,047] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:17:13,049] jepsen worker 53 - jepsen.util 593 :invoke :write [17 49] INFO [2022-03-13 17:17:13,084] jepsen worker 37 - jepsen.util 2287 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,084] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 12] INFO [2022-03-13 17:17:13,103] jepsen worker 36 - jepsen.util 2376 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,103] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 44] INFO [2022-03-13 17:17:13,118] jepsen worker 76 - jepsen.util 796 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,119] jepsen worker 87 - jepsen.util 2787 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,119] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:13,120] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:13,123] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,124] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:13,159] jepsen worker 37 - jepsen.util 2287 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,160] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:13,188] jepsen worker 5 - jepsen.util 2615 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,188] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,189] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,189] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:13,190] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 25] INFO [2022-03-13 17:17:13,191] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 19] INFO [2022-03-13 17:17:13,244] jepsen worker 44 - jepsen.util 674 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,245] jepsen worker 44 - jepsen.util 674 :invoke :write [14 29] INFO [2022-03-13 17:17:13,256] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:13,256] jepsen worker 52 - jepsen.util 682 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,257] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,258] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:13,259] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:13,261] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:13,275] jepsen worker 77 - jepsen.util 707 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,276] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:13,276] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,276] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,277] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,277] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:17:13,277] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:13,277] jepsen worker 42 - jepsen.util 402 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,278] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,278] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 28] INFO [2022-03-13 17:17:13,278] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:13,279] jepsen worker 12 - jepsen.util 912 :invoke :write [4 17] INFO [2022-03-13 17:17:13,339] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:17:13,339] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:17:13,358] jepsen worker 88 - jepsen.util 2248 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,359] jepsen worker 89 - jepsen.util 2699 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,362] jepsen worker 3 - jepsen.util 2163 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,363] jepsen worker 89 - jepsen.util 2699 :invoke :read [29 nil] INFO [2022-03-13 17:17:13,364] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:13,364] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 39] INFO [2022-03-13 17:17:13,394] jepsen worker 9 - jepsen.util 3249 :info :write [3 33] :RequestTimeoutException INFO [2022-03-13 17:17:13,395] jepsen worker 9 - jepsen.util 3339 :invoke :read [3 nil] INFO [2022-03-13 17:17:13,463] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,463] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,463] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,463] jepsen worker 34 - jepsen.util 934 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,463] jepsen worker 13 - jepsen.util 733 :invoke :write [4 4] INFO [2022-03-13 17:17:13,464] jepsen worker 76 - jepsen.util 796 :invoke :write [25 13] INFO [2022-03-13 17:17:13,465] jepsen worker 34 - jepsen.util 934 :invoke :write [11 8] INFO [2022-03-13 17:17:13,467] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:13,486] jepsen worker 66 - jepsen.util 3486 :info :write [22 44] :RequestTimeoutException INFO [2022-03-13 17:17:13,487] jepsen worker 66 - jepsen.util 3576 :invoke :write [22 19] INFO [2022-03-13 17:17:13,510] jepsen worker 40 - jepsen.util 3820 :info :write [13 45] :RequestTimeoutException INFO [2022-03-13 17:17:13,512] jepsen worker 40 - jepsen.util 3910 :invoke :write [13 41] INFO [2022-03-13 17:17:13,567] jepsen worker 4 - jepsen.util 2434 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,567] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,567] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:13,568] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,568] jepsen worker 89 - jepsen.util 2699 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,569] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:13,570] jepsen worker 89 - jepsen.util 2699 :invoke :read [29 nil] INFO [2022-03-13 17:17:13,570] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 30] INFO [2022-03-13 17:17:13,571] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,571] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 26] INFO [2022-03-13 17:17:13,584] jepsen worker 36 - jepsen.util 2376 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,584] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,585] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6549707602339181}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4470588235294117}} INFO [2022-03-13 17:17:13,585] jepsen worker nemesis - couchbase.util Rebalance check count: 541 INFO [2022-03-13 17:17:13,585] jepsen worker nemesis - couchbase.util Rebalance stuck count: 434 INFO [2022-03-13 17:17:13,585] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 6] INFO [2022-03-13 17:17:13,585] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:13,587] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,588] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:13,589] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,589] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:13,591] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,591] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 36] INFO [2022-03-13 17:17:13,599] jepsen worker 88 - jepsen.util 2248 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,599] jepsen worker 89 - jepsen.util 2699 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,602] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 10] INFO [2022-03-13 17:17:13,607] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 1] INFO [2022-03-13 17:17:13,625] jepsen worker 53 - jepsen.util 593 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,626] jepsen worker 53 - jepsen.util 593 :invoke :write [17 15] INFO [2022-03-13 17:17:13,633] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,633] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,633] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,633] jepsen worker 62 - jepsen.util 512 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,633] jepsen worker 78 - jepsen.util 708 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,633] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,634] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:13,634] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:13,635] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:13,635] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:13,636] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:13,637] jepsen worker 7 - jepsen.util 637 :invoke :write [2 30] INFO [2022-03-13 17:17:13,652] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,652] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,653] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:17:13,654] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:17:13,670] jepsen worker 67 - jepsen.util 3037 :info :write [22 45] :RequestTimeoutException INFO [2022-03-13 17:17:13,671] jepsen worker 67 - jepsen.util 3127 :invoke :read [22 nil] INFO [2022-03-13 17:17:13,790] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,791] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:13,792] jepsen worker 80 - jepsen.util 800 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,792] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,793] jepsen worker 80 - jepsen.util 800 :invoke :write [26 41] INFO [2022-03-13 17:17:13,793] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:13,911] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,911] jepsen worker 6 - jepsen.util 636 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,911] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:13,911] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:13,912] jepsen worker 42 - jepsen.util 402 :invoke :write [14 45] INFO [2022-03-13 17:17:13,915] jepsen worker 70 - jepsen.util 880 :invoke :write [23 5] INFO [2022-03-13 17:17:13,953] jepsen worker 13 - jepsen.util 733 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,954] jepsen worker 13 - jepsen.util 733 :invoke :write [4 43] INFO [2022-03-13 17:17:13,979] jepsen worker 3 - jepsen.util 2163 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,979] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:13,996] jepsen worker 5 - jepsen.util 2615 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,997] jepsen worker 38 - jepsen.util 2648 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,997] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:13,997] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:13,998] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:13,998] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:14,075] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,076] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:14,084] jepsen worker 79 - jepsen.util 1159 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,086] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:14,112] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,113] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:14,141] jepsen worker 44 - jepsen.util 674 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,141] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,142] jepsen worker 51 - jepsen.util 771 :invoke :write [17 34] INFO [2022-03-13 17:17:14,143] jepsen worker 44 - jepsen.util 674 :invoke :write [14 21] INFO [2022-03-13 17:17:14,185] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,185] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,186] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,186] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 48] INFO [2022-03-13 17:17:14,186] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 39] INFO [2022-03-13 17:17:14,187] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:14,260] jepsen worker 34 - jepsen.util 934 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,260] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:14,271] jepsen worker 26 - jepsen.util 566 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,271] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:14,272] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,272] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,272] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:14,273] jepsen worker 43 - jepsen.util 673 :invoke :write [14 45] INFO [2022-03-13 17:17:14,273] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:14,273] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:14,290] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,290] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,290] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:17:14,291] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:14,387] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,387] jepsen worker 4 - jepsen.util 2434 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,387] jepsen worker 36 - jepsen.util 2376 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,388] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:14,388] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:14,388] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:14,389] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,390] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:14,391] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,391] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 34] INFO [2022-03-13 17:17:14,402] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,402] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,402] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 18] INFO [2022-03-13 17:17:14,403] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 45] INFO [2022-03-13 17:17:14,467] jepsen worker 80 - jepsen.util 800 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,467] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,468] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,468] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:14,868] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:14,469] jepsen worker 76 - jepsen.util 796 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,868] jepsen worker 35 - jepsen.util 935 :invoke :write [11 49] INFO [2022-03-13 17:17:14,868] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:14,475] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,475] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,476] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,476] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,869] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:14,869] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:14,476] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,869] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:14,476] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,476] jepsen worker 12 - jepsen.util 912 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,869] jepsen worker 12 - jepsen.util 912 :invoke :write [4 28] INFO [2022-03-13 17:17:14,477] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,496] jepsen worker 84 - jepsen.util 4134 :info :write [28 38] :RequestTimeoutException INFO [2022-03-13 17:17:14,871] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:14,871] jepsen worker 84 - jepsen.util 4224 :invoke :write [28 24] INFO [2022-03-13 17:17:14,523] jepsen worker 37 - jepsen.util 2287 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,871] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 26] INFO [2022-03-13 17:17:14,523] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,532] jepsen worker 38 - jepsen.util 2648 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,872] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:14,872] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:14,533] jepsen worker 89 - jepsen.util 2699 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,872] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 5] INFO [2022-03-13 17:17:14,590] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6549707602339181}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4470588235294117}} INFO [2022-03-13 17:17:14,659] jepsen worker 24 - jepsen.util 564 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,883] jepsen worker nemesis - couchbase.util Rebalance check count: 542 INFO [2022-03-13 17:17:14,883] jepsen worker 24 - jepsen.util 564 :invoke :write [8 27] INFO [2022-03-13 17:17:14,676] jepsen worker 7 - jepsen.util 637 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,883] jepsen worker nemesis - couchbase.util Rebalance stuck count: 435 INFO [2022-03-13 17:17:14,676] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,883] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:14,693] jepsen worker 51 - jepsen.util 771 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,883] jepsen worker 51 - jepsen.util 771 :invoke :write [17 17] INFO [2022-03-13 17:17:14,884] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:14,728] jepsen worker 85 - jepsen.util 3415 :info :write [28 22] :RequestTimeoutException INFO [2022-03-13 17:17:14,889] jepsen worker 85 - jepsen.util 3505 :invoke :write [28 25] INFO [2022-03-13 17:17:14,736] jepsen worker 13 - jepsen.util 733 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,737] jepsen worker 53 - jepsen.util 593 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,737] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,737] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,893] jepsen worker 53 - jepsen.util 593 :invoke :write [17 43] INFO [2022-03-13 17:17:14,737] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,893] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:14,893] jepsen worker 60 - jepsen.util 690 :invoke :write [20 22] INFO [2022-03-13 17:17:14,738] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,795] jepsen worker 88 - jepsen.util 2248 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,893] jepsen worker 52 - jepsen.util 682 :invoke :write [17 19] INFO [2022-03-13 17:17:14,795] jepsen worker 5 - jepsen.util 2615 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,819] jepsen worker 42 - jepsen.util 402 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,894] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:14,894] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 0] INFO [2022-03-13 17:17:14,894] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:14,822] jepsen worker 44 - jepsen.util 674 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,822] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,898] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:14,900] jepsen worker 70 - jepsen.util 880 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,822] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,869] jepsen worker 78 - jepsen.util 708 :invoke :write [26 43] INFO [2022-03-13 17:17:14,869] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:14,869] jepsen worker 77 - jepsen.util 707 :invoke :write [25 17] INFO [2022-03-13 17:17:14,893] jepsen worker 13 - jepsen.util 733 :invoke :write [4 1] INFO [2022-03-13 17:17:14,893] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:17:14,904] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,906] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,906] jepsen worker 43 - jepsen.util 673 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,907] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,898] jepsen worker 44 - jepsen.util 674 :invoke :write [14 39] INFO [2022-03-13 17:17:14,900] jepsen worker 33 - jepsen.util 753 :invoke :write [11 33] INFO [2022-03-13 17:17:14,906] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,907] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,916] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,916] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,916] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,917] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,918] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:14,919] jepsen worker 26 - jepsen.util 566 :invoke :write [8 44] INFO [2022-03-13 17:17:14,919] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:14,924] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:14,925] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:14,926] jepsen worker 23 - jepsen.util 563 :invoke :write [7 40] INFO [2022-03-13 17:17:14,928] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:14,928] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:14,929] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:14,930] jepsen worker 76 - jepsen.util 796 :invoke :write [25 43] INFO [2022-03-13 17:17:14,934] jepsen worker 7 - jepsen.util 637 :invoke :write [2 3] INFO [2022-03-13 17:17:14,997] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,997] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:14,999] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:14,999] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:15,005] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:15,008] jepsen worker 43 - jepsen.util 673 :invoke :write [14 32] INFO [2022-03-13 17:17:15,008] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,009] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:17:15,036] jepsen worker 87 - jepsen.util 2787 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,037] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:15,039] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,040] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,040] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:15,041] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:15,042] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,042] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,043] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:15,044] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 38] INFO [2022-03-13 17:17:15,045] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,046] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 24] INFO [2022-03-13 17:17:15,067] jepsen worker 3 - jepsen.util 2163 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,067] jepsen worker 4 - jepsen.util 2434 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,067] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,068] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 21] INFO [2022-03-13 17:17:15,069] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 43] INFO [2022-03-13 17:17:15,070] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 0] INFO [2022-03-13 17:17:15,072] jepsen worker 5 - jepsen.util 2615 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,072] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 28] INFO [2022-03-13 17:17:15,086] jepsen worker 53 - jepsen.util 593 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,087] jepsen worker 53 - jepsen.util 593 :invoke :write [17 39] INFO [2022-03-13 17:17:15,193] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,193] jepsen worker 21 - jepsen.util 831 :invoke :write [7 40] INFO [2022-03-13 17:17:15,194] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,194] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:17:15,197] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,197] jepsen worker 14 - jepsen.util 374 :invoke :write [4 27] INFO [2022-03-13 17:17:15,199] jepsen worker 13 - jepsen.util 733 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,200] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:15,206] jepsen worker 37 - jepsen.util 2287 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,207] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,207] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:15,208] jepsen worker 12 - jepsen.util 912 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,209] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:15,211] jepsen worker 12 - jepsen.util 912 :invoke :write [4 35] INFO [2022-03-13 17:17:15,210] jepsen worker 45 - jepsen.util 4005 :info :write [15 31] :SyncWriteAmbiguous INFO [2022-03-13 17:17:15,210] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,211] jepsen worker 6 - jepsen.util 636 :invoke :write [2 28] INFO [2022-03-13 17:17:15,212] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:15,214] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:15,214] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 0] INFO [2022-03-13 17:17:15,211] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,215] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:15,211] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,215] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,215] jepsen worker 62 - jepsen.util 512 :invoke :write [20 10] INFO [2022-03-13 17:17:15,215] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:15,275] jepsen worker 28 - jepsen.util 3718 :info :write [9 30] :SyncWriteAmbiguous INFO [2022-03-13 17:17:15,276] jepsen worker 28 - jepsen.util 3808 :invoke :write [9 27] INFO [2022-03-13 17:17:15,286] jepsen worker 78 - jepsen.util 708 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,287] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,287] jepsen worker 78 - jepsen.util 708 :invoke :write [26 10] INFO [2022-03-13 17:17:15,288] jepsen worker 61 - jepsen.util 781 :invoke :write [20 46] INFO [2022-03-13 17:17:15,299] jepsen worker 26 - jepsen.util 566 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,300] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:15,414] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,414] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:15,414] jepsen worker 87 - jepsen.util 2787 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,415] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:15,422] jepsen worker 11 - jepsen.util 3701 :ok :read [3 25] INFO [2022-03-13 17:17:15,423] jepsen worker 11 - jepsen.util 3701 :invoke :write [3 32] INFO [2022-03-13 17:17:15,449] jepsen worker 38 - jepsen.util 2648 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,449] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,450] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 3] INFO [2022-03-13 17:17:15,450] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:15,619] jepsen worker 89 - jepsen.util 2699 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,619] jepsen worker 89 - jepsen.util 2699 :invoke :read [29 nil] INFO [2022-03-13 17:17:15,650] jepsen worker 5 - jepsen.util 2615 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:15,651] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 44] INFO [2022-03-13 17:17:15,895] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6666666666666667}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4588235294117647}} INFO [2022-03-13 17:17:15,895] jepsen worker nemesis - couchbase.util Rebalance check count: 543 INFO [2022-03-13 17:17:15,895] jepsen worker nemesis - couchbase.util Rebalance stuck count: 435 INFO [2022-03-13 17:17:15,960] jepsen worker 20 - jepsen.util 3980 :info :write [6 36] :RequestTimeoutException INFO [2022-03-13 17:17:15,960] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:15,976] jepsen worker 82 - jepsen.util 3952 :info :write [27 4] :RequestTimeoutException INFO [2022-03-13 17:17:15,977] jepsen worker 82 - jepsen.util 4042 :invoke :write [27 15] INFO [2022-03-13 17:17:16,035] jepsen worker 4 - jepsen.util 2434 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,035] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 23] INFO [2022-03-13 17:17:16,036] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,036] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:16,120] jepsen worker 18 - jepsen.util 3798 :info :write [6 34] :RequestTimeoutException INFO [2022-03-13 17:17:16,121] jepsen worker 18 - jepsen.util 3888 :invoke :read [6 nil] INFO [2022-03-13 17:17:16,148] jepsen worker 29 - jepsen.util 4079 :info :write [9 34] :RequestTimeoutException INFO [2022-03-13 17:17:16,149] jepsen worker 29 - jepsen.util 4169 :invoke :write [9 39] INFO [2022-03-13 17:17:16,239] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,239] jepsen worker 36 - jepsen.util 2376 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,239] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 45] INFO [2022-03-13 17:17:16,240] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:16,241] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,241] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:16,243] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,243] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:16,244] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,245] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 0] INFO [2022-03-13 17:17:16,259] jepsen worker 38 - jepsen.util 2648 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,260] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 27] INFO [2022-03-13 17:17:16,261] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,262] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:16,351] jepsen worker 73 - jepsen.util 3853 :info :write [24 9] :SyncWriteAmbiguous INFO [2022-03-13 17:17:16,351] jepsen worker 73 - jepsen.util 3943 :invoke :write [24 12] INFO [2022-03-13 17:17:16,410] jepsen worker 19 - jepsen.util 3529 :info :write [6 27] :RequestTimeoutException INFO [2022-03-13 17:17:16,411] jepsen worker 19 - jepsen.util 3619 :invoke :write [6 2] INFO [2022-03-13 17:17:16,417] jepsen worker 31 - jepsen.util 3901 :info :write [10 48] :SyncWriteAmbiguous INFO [2022-03-13 17:17:16,418] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 2] INFO [2022-03-13 17:17:16,443] jepsen worker 5 - jepsen.util 2615 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,443] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:16,461] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,461] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,462] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:16,463] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:16,476] jepsen worker 35 - jepsen.util 935 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,476] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,477] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:16,477] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:16,537] jepsen worker 76 - jepsen.util 796 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,538] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,538] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:16,539] jepsen worker 13 - jepsen.util 733 :invoke :write [4 23] INFO [2022-03-13 17:17:16,541] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,541] jepsen worker 76 - jepsen.util 796 :invoke :write [25 18] INFO [2022-03-13 17:17:16,547] jepsen worker 7 - jepsen.util 637 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,547] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,547] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,547] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:16,548] jepsen worker 7 - jepsen.util 637 :invoke :write [2 49] INFO [2022-03-13 17:17:16,549] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:16,549] jepsen worker 8 - jepsen.util 728 :invoke :write [2 17] INFO [2022-03-13 17:17:16,550] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:16,573] jepsen worker 51 - jepsen.util 771 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,573] jepsen worker 51 - jepsen.util 771 :invoke :write [17 1] INFO [2022-03-13 17:17:16,646] jepsen worker 52 - jepsen.util 682 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,647] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:16,653] jepsen worker 77 - jepsen.util 707 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,654] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:17:16,664] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,665] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 17] INFO [2022-03-13 17:17:16,692] jepsen worker 72 - jepsen.util 3852 :ok :read [24 24] INFO [2022-03-13 17:17:16,693] jepsen worker 72 - jepsen.util 3852 :invoke :read [24 nil] INFO [2022-03-13 17:17:16,713] jepsen worker 33 - jepsen.util 753 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,714] jepsen worker 33 - jepsen.util 753 :invoke :write [11 24] INFO [2022-03-13 17:17:16,752] jepsen worker 37 - jepsen.util 2287 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,753] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 36] INFO [2022-03-13 17:17:16,763] jepsen worker 88 - jepsen.util 2248 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,763] jepsen worker 3 - jepsen.util 2163 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,763] jepsen worker 89 - jepsen.util 2699 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,763] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 5] INFO [2022-03-13 17:17:16,765] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:16,766] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 26] INFO [2022-03-13 17:17:16,811] jepsen worker 12 - jepsen.util 912 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,812] jepsen worker 12 - jepsen.util 912 :invoke :write [4 41] INFO [2022-03-13 17:17:16,819] jepsen worker 60 - jepsen.util 690 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,823] jepsen worker 60 - jepsen.util 690 :invoke :write [20 14] INFO [2022-03-13 17:17:16,836] jepsen worker 44 - jepsen.util 674 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,837] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] WARN [2022-03-13 17:17:16,843] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17182941,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17182941,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9049045,"encode_duration_us":19136,"operation_id":"0xf479","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005971}],"total_count":1}} WARN [2022-03-13 17:17:16,843] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10793099,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10793099,"total_server_duration_us":32,"operation_id":"0xf6a1","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":10794920},{"operation_name":"get","last_dispatch_duration_us":10012799,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10012799,"total_server_duration_us":15,"operation_id":"0xf6f1","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":10014673},{"operation_name":"upsert","last_dispatch_duration_us":40894,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7461736,"total_server_duration_us":330824,"encode_duration_us":18441,"operation_id":"0xf5ad","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10009722},{"operation_name":"upsert","last_dispatch_duration_us":13777683,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":13777683,"total_server_duration_us":21405,"encode_duration_us":18727,"operation_id":"0xf5f4","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10008635},{"operation_name":"upsert","last_dispatch_duration_us":51937,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1185093,"total_server_duration_us":639070,"encode_duration_us":28807,"operation_id":"0xf882","timeout_ms":10000,"last_server_duration_us":23983,"total_duration_us":10004178},{"operation_name":"get","last_dispatch_duration_us":7257487,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":7257487,"total_server_duration_us":27,"operation_id":"0xf7a1","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":7259498},{"operation_name":"upsert","last_dispatch_duration_us":646292,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":2182252,"total_server_duration_us":729890,"encode_duration_us":24478,"operation_id":"0xf839","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":5383907},{"operation_name":"upsert","last_dispatch_duration_us":1991796,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1991796,"total_server_duration_us":271179,"encode_duration_us":19471,"operation_id":"0xfa29","timeout_ms":10000,"last_server_duration_us":271179,"total_duration_us":3083083},{"operation_name":"upsert","last_dispatch_duration_us":126220,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1292073,"total_server_duration_us":441710,"encode_duration_us":20622,"operation_id":"0xf877","timeout_ms":10000,"last_server_duration_us":90781,"total_duration_us":2955453},{"operation_name":"get","last_dispatch_duration_us":2242192,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2242192,"total_server_duration_us":49,"operation_id":"0xfa21","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":2803434}],"total_count":78}} INFO [2022-03-13 17:17:16,862] jepsen worker 9 - jepsen.util 3339 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,863] jepsen worker 9 - jepsen.util 3339 :invoke :read [3 nil] INFO [2022-03-13 17:17:16,867] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,867] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 25] INFO [2022-03-13 17:17:16,887] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,887] jepsen worker 38 - jepsen.util 2648 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,887] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:16,888] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 49] INFO [2022-03-13 17:17:16,900] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6666666666666667}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4588235294117647}} INFO [2022-03-13 17:17:16,900] jepsen worker nemesis - couchbase.util Rebalance check count: 544 INFO [2022-03-13 17:17:16,900] jepsen worker nemesis - couchbase.util Rebalance stuck count: 436 INFO [2022-03-13 17:17:16,904] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,905] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:16,910] jepsen worker 4 - jepsen.util 2434 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,910] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:16,913] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,914] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:16,972] jepsen worker 76 - jepsen.util 796 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,972] jepsen worker 76 - jepsen.util 796 :invoke :write [25 3] INFO [2022-03-13 17:17:16,976] jepsen worker 72 - jepsen.util 3852 :ok :read [24 24] INFO [2022-03-13 17:17:16,976] jepsen worker 72 - jepsen.util 3852 :invoke :write [24 13] INFO [2022-03-13 17:17:16,980] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,980] jepsen worker 23 - jepsen.util 563 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,981] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:16,982] jepsen worker 23 - jepsen.util 563 :invoke :write [7 16] INFO [2022-03-13 17:17:16,995] jepsen worker 9 - jepsen.util 3339 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,995] jepsen worker 45 - jepsen.util 4095 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:16,996] jepsen worker 9 - jepsen.util 3339 :invoke :read [3 nil] INFO [2022-03-13 17:17:16,997] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:17,040] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,041] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:17:17,064] jepsen worker 6 - jepsen.util 636 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,064] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:17,136] jepsen worker 9 - jepsen.util 3339 :ok :read [3 25] INFO [2022-03-13 17:17:17,136] jepsen worker 9 - jepsen.util 3339 :invoke :write [3 18] INFO [2022-03-13 17:17:17,144] jepsen worker 45 - jepsen.util 4095 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,145] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:17,152] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,152] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:17,156] jepsen worker 13 - jepsen.util 733 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,156] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:17,195] jepsen worker 78 - jepsen.util 708 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,195] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:17,213] jepsen worker 10 - jepsen.util 3790 :info :write [3 42] :RequestTimeoutException INFO [2022-03-13 17:17:17,214] jepsen worker 10 - jepsen.util 3880 :invoke :write [3 42] INFO [2022-03-13 17:17:17,283] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,283] jepsen worker 88 - jepsen.util 2248 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,283] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:17,284] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 35] INFO [2022-03-13 17:17:17,292] jepsen worker 36 - jepsen.util 2376 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,292] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:17,292] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:17,293] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:17,327] jepsen worker 3 - jepsen.util 2163 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,328] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 18] INFO [2022-03-13 17:17:17,352] jepsen worker 21 - jepsen.util 831 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,352] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:17,392] jepsen worker 62 - jepsen.util 512 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,393] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,393] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:17,393] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:17,393] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:17,394] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:17,428] jepsen worker 74 - jepsen.util 3854 :info :write [24 38] :RequestTimeoutException INFO [2022-03-13 17:17:17,430] jepsen worker 74 - jepsen.util 3944 :invoke :write [24 36] INFO [2022-03-13 17:17:17,487] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,488] jepsen worker 89 - jepsen.util 2699 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,488] jepsen worker 5 - jepsen.util 2615 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,489] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 42] INFO [2022-03-13 17:17:17,490] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 2] INFO [2022-03-13 17:17:17,490] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 29] INFO [2022-03-13 17:17:17,501] jepsen worker 15 - jepsen.util 3705 :info :write [5 9] :SyncWriteAmbiguous INFO [2022-03-13 17:17:17,502] jepsen worker 15 - jepsen.util 3795 :invoke :read [5 nil] INFO [2022-03-13 17:17:17,550] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:17,551] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:17,598] jepsen worker 68 - jepsen.util 3848 :info :write [22 39] :SyncWriteAmbiguous INFO [2022-03-13 17:17:17,598] jepsen worker 27 - jepsen.util 3897 :info :write [9 22] :RequestTimeoutException INFO [2022-03-13 17:17:17,599] jepsen worker 68 - jepsen.util 3938 :invoke :write [22 20] INFO [2022-03-13 17:17:17,600] jepsen worker 27 - jepsen.util 3987 :invoke :read [9 nil] INFO [2022-03-13 17:17:17,698] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,698] jepsen worker 37 - jepsen.util 2287 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,699] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 34] INFO [2022-03-13 17:17:17,700] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 28] INFO [2022-03-13 17:17:17,712] jepsen worker 4 - jepsen.util 2434 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,714] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 48] INFO [2022-03-13 17:17:17,792] jepsen worker 43 - jepsen.util 673 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,792] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:17,795] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,796] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:17,798] jepsen worker 61 - jepsen.util 781 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,799] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:17,836] jepsen worker 8 - jepsen.util 728 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,837] jepsen worker 8 - jepsen.util 728 :invoke :write [2 31] INFO [2022-03-13 17:17:17,859] jepsen worker 76 - jepsen.util 796 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,860] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:17,860] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,861] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:17,868] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,868] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,868] jepsen worker 13 - jepsen.util 733 :invoke :write [4 30] INFO [2022-03-13 17:17:17,869] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:17,870] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,871] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:17,872] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,872] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:17,873] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,874] jepsen worker 76 - jepsen.util 796 :invoke :write [25 4] INFO [2022-03-13 17:17:17,887] jepsen worker 14 - jepsen.util 374 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,888] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:17,894] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,894] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,896] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 39] INFO [2022-03-13 17:17:17,896] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:17:17,906] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6666666666666667}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4588235294117647}} INFO [2022-03-13 17:17:17,906] jepsen worker nemesis - couchbase.util Rebalance check count: 545 INFO [2022-03-13 17:17:17,906] jepsen worker nemesis - couchbase.util Rebalance stuck count: 437 INFO [2022-03-13 17:17:17,907] jepsen worker 38 - jepsen.util 2648 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,907] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,907] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:17,908] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:17,909] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,909] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,909] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 48] INFO [2022-03-13 17:17:17,910] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:17,911] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,912] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 12] INFO [2022-03-13 17:17:17,927] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,927] jepsen worker 53 - jepsen.util 593 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,928] jepsen worker 52 - jepsen.util 682 :invoke :write [17 29] INFO [2022-03-13 17:17:17,929] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:17,935] jepsen worker 3 - jepsen.util 2163 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,936] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 35] INFO [2022-03-13 17:17:17,939] jepsen worker 33 - jepsen.util 753 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:17,940] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:17,940] jepsen worker 33 - jepsen.util 753 :invoke :write [11 23] INFO [2022-03-13 17:17:17,946] jepsen worker 44 - jepsen.util 674 :invoke :write [14 35] INFO [2022-03-13 17:17:18,000] jepsen worker 7 - jepsen.util 637 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,001] jepsen worker 7 - jepsen.util 637 :invoke :write [2 37] INFO [2022-03-13 17:17:18,024] jepsen worker 23 - jepsen.util 563 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,024] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,024] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,025] jepsen worker 23 - jepsen.util 563 :invoke :write [7 11] INFO [2022-03-13 17:17:18,027] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:18,027] jepsen worker 78 - jepsen.util 708 :invoke :write [26 44] INFO [2022-03-13 17:17:18,064] jepsen worker 12 - jepsen.util 912 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,064] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,064] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,064] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,065] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:18,065] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:18,066] jepsen worker 6 - jepsen.util 636 :invoke :write [2 3] INFO [2022-03-13 17:17:18,067] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:18,067] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:18,068] jepsen worker 70 - jepsen.util 880 :invoke :write [23 6] INFO [2022-03-13 17:17:18,084] jepsen worker 51 - jepsen.util 771 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,085] jepsen worker 60 - jepsen.util 690 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,086] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,086] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,086] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:18,086] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,086] jepsen worker 51 - jepsen.util 771 :invoke :write [17 9] INFO [2022-03-13 17:17:18,087] jepsen worker 26 - jepsen.util 566 :invoke :write [8 34] INFO [2022-03-13 17:17:18,088] jepsen worker 60 - jepsen.util 690 :invoke :write [20 18] INFO [2022-03-13 17:17:18,089] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:18,090] jepsen worker 53 - jepsen.util 593 :invoke :write [17 9] INFO [2022-03-13 17:17:18,092] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:18,102] jepsen worker 5 - jepsen.util 2615 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,103] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 48] INFO [2022-03-13 17:17:18,139] jepsen worker 79 - jepsen.util 1159 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,140] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] WARN [2022-03-13 17:17:18,158] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17922829,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":22002538,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9272878,"encode_duration_us":20324,"operation_id":"0xf355","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10008522}],"total_count":1}} INFO [2022-03-13 17:17:18,200] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,200] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,200] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,200] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,201] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:18,202] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,202] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,202] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:18,203] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:18,204] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:17:18,204] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:18,205] jepsen worker 14 - jepsen.util 374 :invoke :write [4 15] INFO [2022-03-13 17:17:18,207] jepsen worker 13 - jepsen.util 733 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,208] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:18,233] jepsen worker 44 - jepsen.util 674 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,234] jepsen worker 44 - jepsen.util 674 :invoke :write [14 16] INFO [2022-03-13 17:17:18,311] jepsen worker 36 - jepsen.util 2376 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,311] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 35] INFO [2022-03-13 17:17:18,328] jepsen worker 87 - jepsen.util 2787 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,328] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] WARN [2022-03-13 17:17:18,358] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8810232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9192797,"total_server_duration_us":98105,"encode_duration_us":18461,"operation_id":"0xf868","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10006644},{"operation_name":"upsert","last_dispatch_duration_us":63251,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8088036,"total_server_duration_us":274775,"encode_duration_us":18317,"operation_id":"0xf774","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10006250},{"operation_name":"upsert","last_dispatch_duration_us":57670,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8425241,"total_server_duration_us":267595,"encode_duration_us":18737,"operation_id":"0xf790","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10004959},{"operation_name":"upsert","last_dispatch_duration_us":8784350,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9227576,"total_server_duration_us":124553,"encode_duration_us":24448,"operation_id":"0xf85a","timeout_ms":10000,"last_server_duration_us":33721,"total_duration_us":10004918},{"operation_name":"upsert","last_dispatch_duration_us":9208160,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9283248,"total_server_duration_us":9073496,"encode_duration_us":22234,"operation_id":"0xf85d","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9295225},{"operation_name":"upsert","last_dispatch_duration_us":9178271,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9178271,"total_server_duration_us":9066042,"encode_duration_us":29143,"operation_id":"0xf867","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":9182550},{"operation_name":"upsert","last_dispatch_duration_us":9065003,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9065003,"total_server_duration_us":9023573,"encode_duration_us":19323,"operation_id":"0xf77c","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9070120},{"operation_name":"get","last_dispatch_duration_us":8474577,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8474577,"total_server_duration_us":7078,"operation_id":"0xf8ad","timeout_ms":30000,"last_server_duration_us":7078,"total_duration_us":8476499},{"operation_name":"upsert","last_dispatch_duration_us":7950659,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":7950659,"total_server_duration_us":45838,"encode_duration_us":19582,"operation_id":"0xf767","timeout_ms":10000,"last_server_duration_us":45838,"total_duration_us":7956436},{"operation_name":"upsert","last_dispatch_duration_us":7901436,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":7901436,"total_server_duration_us":57632,"encode_duration_us":16866,"operation_id":"0xf766","timeout_ms":10000,"last_server_duration_us":57632,"total_duration_us":7906207}],"total_count":56}} WARN [2022-03-13 17:17:18,363] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18207653,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27661271,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9079682,"encode_duration_us":19351,"operation_id":"0xf03e","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10006504},{"operation_name":"upsert","last_dispatch_duration_us":17092752,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17092752,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9032060,"encode_duration_us":18043,"operation_id":"0xf3a3","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10008354},{"operation_name":"upsert","last_dispatch_duration_us":18208451,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27305207,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9076443,"encode_duration_us":21320,"operation_id":"0xf093","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10007059}],"total_count":3}} INFO [2022-03-13 17:17:18,380] jepsen worker 33 - jepsen.util 753 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,380] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:18,393] jepsen worker 8 - jepsen.util 728 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,394] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] WARN [2022-03-13 17:17:18,448] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9036188,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9718176,"total_server_duration_us":9280131,"encode_duration_us":20056,"operation_id":"0xf6d3","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10008300},{"operation_name":"upsert","last_dispatch_duration_us":6980277,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7885299,"total_server_duration_us":463574,"encode_duration_us":18468,"operation_id":"0xf700","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10006243},{"operation_name":"upsert","last_dispatch_duration_us":49023,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":3252285,"total_server_duration_us":552921,"encode_duration_us":17983,"operation_id":"0xf64b","timeout_ms":10000,"last_server_duration_us":34116,"total_duration_us":10005422},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18472,"operation_id":"0xf76c","timeout_ms":10000,"total_duration_us":10005153},{"operation_name":"upsert","last_dispatch_duration_us":30607,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":443711,"total_server_duration_us":230008,"encode_duration_us":18859,"operation_id":"0xf67e","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10002465},{"operation_name":"upsert","last_dispatch_duration_us":6946961,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":6946961,"total_server_duration_us":22389,"encode_duration_us":20795,"operation_id":"0xf746","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10001418},{"operation_name":"upsert","last_dispatch_duration_us":9109461,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9109461,"total_server_duration_us":9067105,"encode_duration_us":19658,"operation_id":"0xf673","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":9114736},{"operation_name":"get","last_dispatch_duration_us":7369952,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":7369952,"total_server_duration_us":38,"operation_id":"0xf81a","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":7370263},{"operation_name":"upsert","last_dispatch_duration_us":7266830,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":7266830,"total_server_duration_us":64343,"encode_duration_us":20482,"operation_id":"0xf845","timeout_ms":10000,"last_server_duration_us":64343,"total_duration_us":7270650},{"operation_name":"upsert","last_dispatch_duration_us":4956535,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":4956535,"total_server_duration_us":205497,"encode_duration_us":37441,"operation_id":"0xf9b7","timeout_ms":10000,"last_server_duration_us":205497,"total_duration_us":4960855}],"total_count":23}} WARN [2022-03-13 17:17:18,449] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10324685,"last_local_id":"A312307A00000005/00000000F4F37B9B","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":12423852,"last_local_socket":"172.23.104.79:33502","total_server_duration_us":10989632,"encode_duration_us":19004,"operation_id":"0xf537","timeout_ms":10000,"last_server_duration_us":10312946,"total_duration_us":10002129},{"operation_name":"upsert","last_dispatch_duration_us":9036188,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9718176,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9280131,"encode_duration_us":20056,"operation_id":"0xf6d3","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10008300}],"total_count":2}} WARN [2022-03-13 17:17:18,458] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":1363936,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":5158989,"total_server_duration_us":289061,"operation_id":"0xf0ed","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":30009301},{"operation_name":"upsert","last_dispatch_duration_us":29785,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8295453,"total_server_duration_us":279312,"encode_duration_us":18567,"operation_id":"0xf7cd","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10010442},{"operation_name":"upsert","last_dispatch_duration_us":333428,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10192287,"total_server_duration_us":146985,"encode_duration_us":18576,"operation_id":"0xf8d1","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10007699},{"operation_name":"upsert","last_dispatch_duration_us":9010916,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18284942,"total_server_duration_us":109903,"encode_duration_us":20599,"operation_id":"0xf602","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10004181},{"operation_name":"upsert","last_dispatch_duration_us":73323,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9946265,"total_server_duration_us":110184,"encode_duration_us":26928,"operation_id":"0xf765","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10003299},{"operation_name":"upsert","last_dispatch_duration_us":8959047,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10277529,"total_server_duration_us":94541,"encode_duration_us":23006,"operation_id":"0xf86d","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10003044},{"operation_name":"upsert","last_dispatch_duration_us":317757,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10119729,"total_server_duration_us":138626,"encode_duration_us":17665,"operation_id":"0xf8f9","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10001529},{"operation_name":"upsert","last_dispatch_duration_us":9165471,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9165471,"total_server_duration_us":9027816,"encode_duration_us":26281,"operation_id":"0xf8c7","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":9169380},{"operation_name":"upsert","last_dispatch_duration_us":9161372,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9161372,"total_server_duration_us":9032060,"encode_duration_us":20489,"operation_id":"0xf8c2","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":9165392},{"operation_name":"get","last_dispatch_duration_us":9053909,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9053909,"total_server_duration_us":40589,"operation_id":"0xf8fa","timeout_ms":30000,"last_server_duration_us":40589,"total_duration_us":9056247}],"total_count":57}} INFO [2022-03-13 17:17:18,520] jepsen worker 81 - jepsen.util 4401 :ok :read [27 4] INFO [2022-03-13 17:17:18,520] jepsen worker 81 - jepsen.util 4401 :invoke :write [27 42] INFO [2022-03-13 17:17:18,546] jepsen worker 67 - jepsen.util 3127 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,547] jepsen worker 67 - jepsen.util 3127 :invoke :read [22 nil] INFO [2022-03-13 17:17:18,553] jepsen worker 83 - jepsen.util 3503 :ok :read [27 4] INFO [2022-03-13 17:17:18,554] jepsen worker 83 - jepsen.util 3503 :invoke :read [27 nil] INFO [2022-03-13 17:17:18,554] jepsen worker 58 - jepsen.util 3388 :info :write [19 47] :RequestTimeoutException INFO [2022-03-13 17:17:18,555] jepsen worker 58 - jepsen.util 3478 :invoke :write [19 17] INFO [2022-03-13 17:17:18,561] jepsen worker 86 - jepsen.util 3686 :ok :read [28 44] INFO [2022-03-13 17:17:18,562] jepsen worker 86 - jepsen.util 3686 :invoke :write [28 33] WARN [2022-03-13 17:17:18,563] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9046924,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":18279301,"total_server_duration_us":93124,"encode_duration_us":16929,"operation_id":"0xf64a","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10009911},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":18212,"operation_id":"0xf76b","timeout_ms":10000,"total_duration_us":10008249},{"operation_name":"upsert","last_dispatch_duration_us":8386367,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8461485,"total_server_duration_us":57371,"encode_duration_us":19139,"operation_id":"0xf62a","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":10007729},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":26386,"operation_id":"0xf8c4","timeout_ms":10000,"total_duration_us":10005122},{"operation_name":"upsert","last_dispatch_duration_us":8403126,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8888146,"total_server_duration_us":199988,"encode_duration_us":20388,"operation_id":"0xf6a4","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10003107},{"operation_name":"upsert","last_dispatch_duration_us":9102871,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9102871,"total_server_duration_us":9053293,"encode_duration_us":18554,"operation_id":"0xf92c","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9108537},{"operation_name":"upsert","last_dispatch_duration_us":9099252,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9099252,"total_server_duration_us":9042674,"encode_duration_us":17299,"operation_id":"0xf924","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9103639},{"operation_name":"get","last_dispatch_duration_us":8053434,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8053434,"total_server_duration_us":25888,"operation_id":"0xf723","timeout_ms":30000,"last_server_duration_us":25888,"total_duration_us":8055285},{"operation_name":"get","last_dispatch_duration_us":7185036,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7185036,"total_server_duration_us":43,"operation_id":"0xf6b9","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":7187483},{"operation_name":"upsert","last_dispatch_duration_us":94471,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":1600162,"total_server_duration_us":545331,"encode_duration_us":17430,"operation_id":"0xf847","timeout_ms":10000,"last_server_duration_us":24928,"total_duration_us":4272902}],"total_count":63}} INFO [2022-03-13 17:17:18,564] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,564] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,564] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,565] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,565] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,565] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:18,566] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:18,567] jepsen worker 80 - jepsen.util 800 :invoke :write [26 14] WARN [2022-03-13 17:17:18,568] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":64471,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9058638,"total_server_duration_us":252026,"encode_duration_us":26306,"operation_id":"0xf672","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10010907},{"operation_name":"upsert","last_dispatch_duration_us":50618,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8470675,"total_server_duration_us":254217,"encode_duration_us":17783,"operation_id":"0xf6e7","timeout_ms":10000,"last_server_duration_us":31108,"total_duration_us":10009828},{"operation_name":"upsert","last_dispatch_duration_us":33487,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":284639,"total_server_duration_us":125331,"encode_duration_us":24104,"operation_id":"0xf85b","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10003930},{"operation_name":"upsert","last_dispatch_duration_us":32806,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4710161,"total_server_duration_us":277555,"encode_duration_us":22254,"operation_id":"0xf7aa","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10002178},{"operation_name":"get","last_dispatch_duration_us":9070767,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9070767,"total_server_duration_us":894,"operation_id":"0xf670","timeout_ms":30000,"last_server_duration_us":894,"total_duration_us":9071108},{"operation_name":"get","last_dispatch_duration_us":9051834,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9051834,"total_server_duration_us":38,"operation_id":"0xf671","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9055063},{"operation_name":"upsert","last_dispatch_duration_us":7868578,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":7868578,"total_server_duration_us":204013,"encode_duration_us":26068,"operation_id":"0xf82b","timeout_ms":10000,"last_server_duration_us":204013,"total_duration_us":7872232},{"operation_name":"upsert","last_dispatch_duration_us":7676037,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7676037,"total_server_duration_us":36220,"encode_duration_us":18088,"operation_id":"0xf76a","timeout_ms":10000,"last_server_duration_us":36220,"total_duration_us":7679601},{"operation_name":"get","last_dispatch_duration_us":6002888,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":6002888,"total_server_duration_us":49,"operation_id":"0xf747","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":6005330},{"operation_name":"get","last_dispatch_duration_us":4236113,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4236113,"total_server_duration_us":43,"operation_id":"0xf87e","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":4236382}],"total_count":24}} INFO [2022-03-13 17:17:18,568] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] WARN [2022-03-13 17:17:18,568] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10555952,"last_local_id":"A312307A00000006/000000005DE126B4","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":12590895,"last_local_socket":"172.23.104.79:33480","total_server_duration_us":11266387,"encode_duration_us":18310,"operation_id":"0xf523","timeout_ms":10000,"last_server_duration_us":10541914,"total_duration_us":10007483},{"operation_name":"upsert","last_dispatch_duration_us":15642020,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15642020,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9049045,"encode_duration_us":20710,"operation_id":"0xf421","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10008444}],"total_count":2}} INFO [2022-03-13 17:17:18,568] jepsen worker 34 - jepsen.util 934 :invoke :write [11 12] INFO [2022-03-13 17:17:18,576] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,576] jepsen worker 52 - jepsen.util 682 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,576] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:18,577] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,577] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:18,578] jepsen worker 43 - jepsen.util 673 :invoke :write [14 31] INFO [2022-03-13 17:17:18,579] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,580] jepsen worker 13 - jepsen.util 733 :invoke :write [4 10] INFO [2022-03-13 17:17:18,582] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:18,582] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:18,613] jepsen worker 67 - jepsen.util 3127 :fail :read [22 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,614] jepsen worker 67 - jepsen.util 3127 :invoke :write [22 17] INFO [2022-03-13 17:17:18,630] jepsen worker 63 - jepsen.util 3933 :info :write [21 48] :RequestTimeoutException INFO [2022-03-13 17:17:18,631] jepsen worker 63 - jepsen.util 4023 :invoke :read [21 nil] INFO [2022-03-13 17:17:18,664] jepsen worker 83 - jepsen.util 3503 :ok :read [27 4] INFO [2022-03-13 17:17:18,664] jepsen worker 83 - jepsen.util 3503 :invoke :write [27 39] INFO [2022-03-13 17:17:18,714] jepsen worker 37 - jepsen.util 2287 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,715] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 0] INFO [2022-03-13 17:17:18,731] jepsen worker 88 - jepsen.util 2248 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,732] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 38] INFO [2022-03-13 17:17:18,748] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,748] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:18,748] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,750] jepsen worker 12 - jepsen.util 912 :invoke :write [4 4] INFO [2022-03-13 17:17:18,750] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:18,751] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:18,756] jepsen worker 36 - jepsen.util 2376 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,756] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 18] INFO [2022-03-13 17:17:18,912] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6666666666666667}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4588235294117647}} INFO [2022-03-13 17:17:18,912] jepsen worker nemesis - couchbase.util Rebalance check count: 546 INFO [2022-03-13 17:17:18,912] jepsen worker nemesis - couchbase.util Rebalance stuck count: 438 INFO [2022-03-13 17:17:18,919] jepsen worker 4 - jepsen.util 2434 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:18,919] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 37] INFO [2022-03-13 17:17:18,924] jepsen worker 55 - jepsen.util 3475 :info :write [18 0] :SyncWriteAmbiguous INFO [2022-03-13 17:17:18,925] jepsen worker 55 - jepsen.util 3565 :invoke :read [18 nil] INFO [2022-03-13 17:17:19,011] jepsen worker 25 - jepsen.util 565 :ok :read [8 :nil] INFO [2022-03-13 17:17:19,012] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:19,019] jepsen worker 33 - jepsen.util 753 :ok :read [11 7] INFO [2022-03-13 17:17:19,020] jepsen worker 33 - jepsen.util 753 :invoke :write [11 26] INFO [2022-03-13 17:17:19,026] jepsen worker 35 - jepsen.util 935 :ok :read [11 7] INFO [2022-03-13 17:17:19,027] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:17:19,076] jepsen worker 17 - jepsen.util 3887 :info :write [5 13] :RequestTimeoutException INFO [2022-03-13 17:17:19,077] jepsen worker 17 - jepsen.util 3977 :invoke :read [5 nil] INFO [2022-03-13 17:17:19,091] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,091] jepsen worker 38 - jepsen.util 2648 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,092] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:19,092] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:19,093] jepsen worker 89 - jepsen.util 2699 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,094] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 7] INFO [2022-03-13 17:17:19,118] jepsen worker 3 - jepsen.util 2163 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,119] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:19,121] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,121] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:19,123] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,123] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:19,125] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,125] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:19,126] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,127] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 10] INFO [2022-03-13 17:17:19,134] jepsen worker 18 - jepsen.util 3888 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,134] jepsen worker 18 - jepsen.util 3888 :invoke :write [6 15] INFO [2022-03-13 17:17:19,147] jepsen worker 65 - jepsen.util 4205 :info :write [21 17] :RequestTimeoutException INFO [2022-03-13 17:17:19,147] jepsen worker 65 - jepsen.util 4295 :invoke :read [21 nil] INFO [2022-03-13 17:17:19,159] jepsen worker 54 - jepsen.util 3474 :ok :read [18 21] INFO [2022-03-13 17:17:19,159] jepsen worker 88 - jepsen.util 2248 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,160] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:19,160] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:19,187] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,187] jepsen worker 47 - jepsen.util 3647 :info :write [15 15] :RequestTimeoutException INFO [2022-03-13 17:17:19,188] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 17] INFO [2022-03-13 17:17:19,189] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,191] jepsen worker 61 - jepsen.util 781 :invoke :write [20 46] INFO [2022-03-13 17:17:19,192] jepsen worker 47 - jepsen.util 3737 :invoke :read [15 nil] INFO [2022-03-13 17:17:19,201] jepsen worker 46 - jepsen.util 4006 :ok :read [15 48] INFO [2022-03-13 17:17:19,202] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 18] INFO [2022-03-13 17:17:19,291] jepsen worker 4 - jepsen.util 2434 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,291] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 28] INFO [2022-03-13 17:17:19,294] jepsen worker 55 - jepsen.util 3565 :ok :read [18 21] INFO [2022-03-13 17:17:19,294] jepsen worker 55 - jepsen.util 3565 :invoke :write [18 6] INFO [2022-03-13 17:17:19,306] jepsen worker 37 - jepsen.util 2287 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,307] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 21] INFO [2022-03-13 17:17:19,313] jepsen worker 5 - jepsen.util 2615 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,314] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:19,316] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,317] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 35] INFO [2022-03-13 17:17:19,319] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,320] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,320] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:19,321] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 33] INFO [2022-03-13 17:17:19,325] jepsen worker 20 - jepsen.util 4070 :ok :read [6 16] INFO [2022-03-13 17:17:19,326] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:19,327] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,328] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:19,330] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,330] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 41] INFO [2022-03-13 17:17:19,332] jepsen worker 89 - jepsen.util 2699 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,332] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 42] INFO [2022-03-13 17:17:19,342] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,343] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 27] INFO [2022-03-13 17:17:19,390] jepsen worker 39 - jepsen.util 3999 :info :write [13 18] :RequestTimeoutException INFO [2022-03-13 17:17:19,391] jepsen worker 39 - jepsen.util 4089 :invoke :write [13 33] INFO [2022-03-13 17:17:19,427] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,428] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,428] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:19,429] jepsen worker 62 - jepsen.util 512 :invoke :write [20 32] INFO [2022-03-13 17:17:19,429] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:19,430] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:19,464] jepsen worker 54 - jepsen.util 3474 :ok :read [18 21] INFO [2022-03-13 17:17:19,465] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:19,497] jepsen worker 36 - jepsen.util 2376 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,499] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:19,503] jepsen worker 3 - jepsen.util 2163 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,559] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 40] INFO [2022-03-13 17:17:19,615] jepsen worker 20 - jepsen.util 4070 :ok :read [6 16] INFO [2022-03-13 17:17:19,615] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:19,686] jepsen worker 54 - jepsen.util 3474 :ok :read [18 21] INFO [2022-03-13 17:17:19,687] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 17] INFO [2022-03-13 17:17:19,706] jepsen worker 71 - jepsen.util 521 :info :write [23 33] :RequestTimeoutException INFO [2022-03-13 17:17:19,707] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:19,718] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,718] jepsen worker 37 - jepsen.util 2287 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,718] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:19,719] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 32] INFO [2022-03-13 17:17:19,766] jepsen worker 20 - jepsen.util 4070 :ok :read [6 16] INFO [2022-03-13 17:17:19,767] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:19,838] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,839] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:19,842] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,842] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,843] jepsen worker 22 - jepsen.util 652 :invoke :write [7 13] INFO [2022-03-13 17:17:19,843] jepsen worker 52 - jepsen.util 682 :invoke :write [17 23] INFO [2022-03-13 17:17:19,917] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6666666666666667}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4588235294117647}} INFO [2022-03-13 17:17:19,917] jepsen worker nemesis - couchbase.util Rebalance check count: 547 INFO [2022-03-13 17:17:19,917] jepsen worker nemesis - couchbase.util Rebalance stuck count: 439 INFO [2022-03-13 17:17:19,960] jepsen worker 20 - jepsen.util 4070 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:19,960] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 32] INFO [2022-03-13 17:17:20,039] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,041] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:20,194] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,195] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:20,206] jepsen worker 4 - jepsen.util 2434 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,207] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,216] jepsen worker 38 - jepsen.util 2648 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,216] jepsen worker 5 - jepsen.util 2615 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,217] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,218] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 19] INFO [2022-03-13 17:17:20,227] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,228] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 7] INFO [2022-03-13 17:17:20,237] jepsen worker 89 - jepsen.util 2699 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,238] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 41] INFO [2022-03-13 17:17:20,242] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,244] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 30] INFO [2022-03-13 17:17:20,262] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:20,262] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,266] jepsen worker 8 - jepsen.util 728 :invoke :write [2 31] INFO [2022-03-13 17:17:20,267] jepsen worker 42 - jepsen.util 402 :invoke :write [14 44] INFO [2022-03-13 17:17:20,331] jepsen worker 36 - jepsen.util 2376 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,331] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,332] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 29] INFO [2022-03-13 17:17:20,332] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 11] INFO [2022-03-13 17:17:20,372] jepsen worker 3 - jepsen.util 2163 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,372] jepsen worker 88 - jepsen.util 2248 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,373] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 1] INFO [2022-03-13 17:17:20,373] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 29] INFO [2022-03-13 17:17:20,385] jepsen worker 56 - jepsen.util 3926 :fail :write [18 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,386] jepsen worker 56 - jepsen.util 3926 :invoke :read [18 nil] INFO [2022-03-13 17:17:20,394] jepsen worker 87 - jepsen.util 2787 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,394] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:20,446] jepsen worker 75 - jepsen.util 435 :info :write [25 9] :RequestTimeoutException INFO [2022-03-13 17:17:20,447] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:17:20,449] jepsen worker 36 - jepsen.util 2376 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,450] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 9] INFO [2022-03-13 17:17:20,455] jepsen worker 54 - jepsen.util 3474 :fail :write [18 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,455] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:20,477] jepsen worker 5 - jepsen.util 2615 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,477] jepsen worker 89 - jepsen.util 2699 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,478] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,479] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 36] INFO [2022-03-13 17:17:20,484] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,485] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,486] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,487] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 9] INFO [2022-03-13 17:17:20,498] jepsen worker 31 - jepsen.util 3991 :fail :write [10 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,499] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 21] INFO [2022-03-13 17:17:20,516] jepsen worker 57 - jepsen.util 3837 :info :write [19 31] :RequestTimeoutException INFO [2022-03-13 17:17:20,517] jepsen worker 57 - jepsen.util 3927 :invoke :read [19 nil] INFO [2022-03-13 17:17:20,520] jepsen worker 4 - jepsen.util 2434 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,521] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,526] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,527] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,528] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,529] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 14] INFO [2022-03-13 17:17:20,537] jepsen worker 20 - jepsen.util 4070 :fail :write [6 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,538] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 10] INFO [2022-03-13 17:17:20,582] jepsen worker 38 - jepsen.util 2648 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,583] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,583] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 37] INFO [2022-03-13 17:17:20,583] jepsen worker 46 - jepsen.util 4006 :fail :write [15 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,584] jepsen worker 56 - jepsen.util 3926 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,584] jepsen worker 37 - jepsen.util 2287 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,585] jepsen worker 56 - jepsen.util 3926 :invoke :read [18 nil] INFO [2022-03-13 17:17:20,585] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 41] INFO [2022-03-13 17:17:20,586] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 23] INFO [2022-03-13 17:17:20,590] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:20,590] jepsen worker 3 - jepsen.util 2163 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,590] jepsen worker 54 - jepsen.util 3474 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,591] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,591] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,591] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 34] INFO [2022-03-13 17:17:20,593] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 8] INFO [2022-03-13 17:17:20,599] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,600] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 5] INFO [2022-03-13 17:17:20,629] jepsen worker 31 - jepsen.util 3991 :fail :write [10 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,631] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 21] INFO [2022-03-13 17:17:20,655] jepsen worker 56 - jepsen.util 3926 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,655] jepsen worker 20 - jepsen.util 4070 :fail :write [6 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,657] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:20,657] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 29] INFO [2022-03-13 17:17:20,693] jepsen worker 46 - jepsen.util 4006 :fail :write [15 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,693] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:20,729] jepsen worker 54 - jepsen.util 3474 :fail :write [18 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,730] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 44] INFO [2022-03-13 17:17:20,758] jepsen worker 31 - jepsen.util 3991 :fail :write [10 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,758] jepsen worker 20 - jepsen.util 4070 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,760] jepsen worker 31 - jepsen.util 3991 :invoke :read [10 nil] INFO [2022-03-13 17:17:20,760] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 15] INFO [2022-03-13 17:17:20,793] jepsen worker 46 - jepsen.util 4006 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,794] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:20,794] jepsen worker 56 - jepsen.util 3926 :fail :write [18 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,794] jepsen worker 56 - jepsen.util 3926 :invoke :read [18 nil] INFO [2022-03-13 17:17:20,809] jepsen worker 41 - jepsen.util 3731 :info :write [13 6] :SyncWriteAmbiguous INFO [2022-03-13 17:17:20,810] jepsen worker 41 - jepsen.util 3821 :invoke :write [13 1] INFO [2022-03-13 17:17:20,821] jepsen worker 31 - jepsen.util 3991 :ok :read [10 :nil] INFO [2022-03-13 17:17:20,821] jepsen worker 31 - jepsen.util 3991 :invoke :read [10 nil] INFO [2022-03-13 17:17:20,822] jepsen worker 54 - jepsen.util 3474 :fail :write [18 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,822] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 3] INFO [2022-03-13 17:17:20,849] jepsen worker 46 - jepsen.util 4006 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,849] jepsen worker 20 - jepsen.util 4070 :fail :write [6 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,850] jepsen worker 56 - jepsen.util 3926 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,850] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:20,850] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 19] INFO [2022-03-13 17:17:20,850] jepsen worker 31 - jepsen.util 3991 :ok :read [10 :nil] INFO [2022-03-13 17:17:20,854] jepsen worker 56 - jepsen.util 3926 :invoke :read [18 nil] INFO [2022-03-13 17:17:20,855] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 10] INFO [2022-03-13 17:17:20,865] jepsen worker 59 - jepsen.util 3929 :ok :read [19 16] INFO [2022-03-13 17:17:20,865] jepsen worker 59 - jepsen.util 3929 :invoke :write [19 31] INFO [2022-03-13 17:17:20,884] jepsen worker 46 - jepsen.util 4006 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,884] jepsen worker 56 - jepsen.util 3926 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,885] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:20,885] jepsen worker 54 - jepsen.util 3474 :fail :write [18 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,885] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 14] INFO [2022-03-13 17:17:20,891] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 19] INFO [2022-03-13 17:17:20,892] jepsen worker 36 - jepsen.util 2376 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,893] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 19] INFO [2022-03-13 17:17:20,915] jepsen worker 88 - jepsen.util 2248 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,916] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 34] INFO [2022-03-13 17:17:20,920] jepsen worker 5 - jepsen.util 2615 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,921] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,925] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6783625730994152}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4705882352941176}} INFO [2022-03-13 17:17:20,925] jepsen worker nemesis - couchbase.util Rebalance check count: 548 INFO [2022-03-13 17:17:20,925] jepsen worker nemesis - couchbase.util Rebalance stuck count: 439 INFO [2022-03-13 17:17:20,927] jepsen worker 20 - jepsen.util 4070 :fail :write [6 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,927] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:20,932] jepsen worker 51 - jepsen.util 771 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,933] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:20,935] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,935] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,937] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,938] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:20,959] jepsen worker 46 - jepsen.util 4006 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,959] jepsen worker 31 - jepsen.util 3991 :fail :write [10 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,960] jepsen worker 31 - jepsen.util 3991 :invoke :read [10 nil] INFO [2022-03-13 17:17:20,960] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:20,966] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,967] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 11] INFO [2022-03-13 17:17:20,981] jepsen worker 30 - jepsen.util 3630 :info :write [10 31] :RequestTimeoutException INFO [2022-03-13 17:17:20,982] jepsen worker 30 - jepsen.util 3720 :invoke :write [10 24] INFO [2022-03-13 17:17:20,988] jepsen worker 37 - jepsen.util 2287 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,989] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:20,993] jepsen worker 56 - jepsen.util 3926 :fail :write [18 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:20,994] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 43] INFO [2022-03-13 17:17:21,022] jepsen worker 31 - jepsen.util 3991 :ok :read [10 :nil] INFO [2022-03-13 17:17:21,022] jepsen worker 46 - jepsen.util 4006 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,022] jepsen worker 54 - jepsen.util 3474 :fail :write [18 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,022] jepsen worker 20 - jepsen.util 4070 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,023] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 5] INFO [2022-03-13 17:17:21,023] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 46] INFO [2022-03-13 17:17:21,024] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:21,024] jepsen worker 31 - jepsen.util 3991 :invoke :read [10 nil] INFO [2022-03-13 17:17:21,049] jepsen worker 36 - jepsen.util 2376 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,049] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,050] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 25] INFO [2022-03-13 17:17:21,050] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:21,053] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,053] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:21,055] jepsen worker 56 - jepsen.util 3926 :fail :write [18 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,055] jepsen worker 20 - jepsen.util 4070 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,055] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,055] jepsen worker 31 - jepsen.util 3991 :ok :read [10 :nil] INFO [2022-03-13 17:17:21,055] jepsen worker 56 - jepsen.util 3926 :invoke :read [18 nil] INFO [2022-03-13 17:17:21,057] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:21,057] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 10] INFO [2022-03-13 17:17:21,058] jepsen worker 87 - jepsen.util 2787 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,058] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 19] INFO [2022-03-13 17:17:21,059] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:21,079] jepsen worker 4 - jepsen.util 2434 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,079] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 11] INFO [2022-03-13 17:17:21,085] jepsen worker 46 - jepsen.util 4006 :fail :write [15 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,086] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 28] INFO [2022-03-13 17:17:21,132] jepsen worker 36 - jepsen.util 2376 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,133] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:21,138] jepsen worker 20 - jepsen.util 4070 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,138] jepsen worker 54 - jepsen.util 3474 :fail :write [18 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,138] jepsen worker 56 - jepsen.util 3926 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,139] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:21,139] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 41] INFO [2022-03-13 17:17:21,140] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 5] INFO [2022-03-13 17:17:21,180] jepsen worker 31 - jepsen.util 3991 :fail :write [10 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,181] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 33] INFO [2022-03-13 17:17:21,184] jepsen worker 38 - jepsen.util 2648 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,184] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,184] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 3] INFO [2022-03-13 17:17:21,186] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:21,198] jepsen worker 37 - jepsen.util 2287 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,198] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,199] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 21] INFO [2022-03-13 17:17:21,199] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:21,204] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,204] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:21,206] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,207] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 32] INFO [2022-03-13 17:17:21,217] jepsen worker 46 - jepsen.util 4006 :fail :write [15 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,218] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 4] INFO [2022-03-13 17:17:21,218] jepsen worker 54 - jepsen.util 3474 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,219] jepsen worker 3 - jepsen.util 2163 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,219] jepsen worker 89 - jepsen.util 2699 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,219] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 21] INFO [2022-03-13 17:17:21,220] jepsen worker 89 - jepsen.util 2699 :invoke :read [29 nil] INFO [2022-03-13 17:17:21,223] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 43] INFO [2022-03-13 17:17:21,255] jepsen worker 56 - jepsen.util 3926 :fail :write [18 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,256] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 31] INFO [2022-03-13 17:17:21,284] jepsen worker 13 - jepsen.util 733 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,285] jepsen worker 13 - jepsen.util 733 :invoke :write [4 16] INFO [2022-03-13 17:17:21,308] jepsen worker 20 - jepsen.util 4070 :fail :write [6 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,308] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 41] INFO [2022-03-13 17:17:21,340] jepsen worker 31 - jepsen.util 3991 :fail :write [10 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,341] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 38] INFO [2022-03-13 17:17:21,380] jepsen worker 54 - jepsen.util 3474 :fail :write [18 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,381] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:21,447] jepsen worker 46 - jepsen.util 4006 :fail :write [15 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,447] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 21] INFO [2022-03-13 17:17:21,479] jepsen worker 36 - jepsen.util 2376 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,479] jepsen worker 5 - jepsen.util 2615 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,479] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,480] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:21,480] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:21,481] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 10] INFO [2022-03-13 17:17:21,488] jepsen worker 56 - jepsen.util 3926 :fail :write [18 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,489] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 15] INFO [2022-03-13 17:17:21,531] jepsen worker 80 - jepsen.util 800 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,531] jepsen worker 80 - jepsen.util 800 :invoke :write [26 37] INFO [2022-03-13 17:17:21,532] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,532] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 36] INFO [2022-03-13 17:17:21,537] jepsen worker 70 - jepsen.util 880 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,537] jepsen worker 37 - jepsen.util 2287 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,538] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 20] INFO [2022-03-13 17:17:21,538] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:21,538] jepsen worker 64 - jepsen.util 3844 :info :write [21 40] :RequestTimeoutException INFO [2022-03-13 17:17:21,539] jepsen worker 64 - jepsen.util 3934 :invoke :read [21 nil] INFO [2022-03-13 17:17:21,553] jepsen worker 2 - jepsen.util 4142 :ok :read [0 41] INFO [2022-03-13 17:17:21,554] jepsen worker 2 - jepsen.util 4142 :invoke :write [0 15] INFO [2022-03-13 17:17:21,567] jepsen worker 34 - jepsen.util 934 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,568] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:21,587] jepsen worker 89 - jepsen.util 2699 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,588] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 23] INFO [2022-03-13 17:17:21,630] jepsen worker 32 - jepsen.util 3992 :info :write [10 22] :RequestTimeoutException INFO [2022-03-13 17:17:21,631] jepsen worker 32 - jepsen.util 4082 :invoke :write [10 19] INFO [2022-03-13 17:17:21,706] jepsen worker 37 - jepsen.util 2287 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,707] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 33] INFO [2022-03-13 17:17:21,721] jepsen worker 61 - jepsen.util 781 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,721] jepsen worker 61 - jepsen.util 781 :invoke :write [20 38] INFO [2022-03-13 17:17:21,731] jepsen worker 1 - jepsen.util 3331 :info :write [0 19] :RequestTimeoutException INFO [2022-03-13 17:17:21,732] jepsen worker 1 - jepsen.util 3421 :invoke :write [0 19] INFO [2022-03-13 17:17:21,769] jepsen worker 0 - jepsen.util 3690 :info :write [0 5] :SyncWriteAmbiguous INFO [2022-03-13 17:17:21,769] jepsen worker 0 - jepsen.util 3780 :invoke :read [0 nil] INFO [2022-03-13 17:17:21,780] jepsen worker 16 - jepsen.util 3616 :info :write [5 48] :RequestTimeoutException INFO [2022-03-13 17:17:21,781] jepsen worker 16 - jepsen.util 3706 :invoke :write [5 10] INFO [2022-03-13 17:17:21,834] jepsen worker 20 - jepsen.util 4070 :fail :write [6 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,836] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 1] INFO [2022-03-13 17:17:21,866] jepsen worker 31 - jepsen.util 3991 :fail :write [10 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,866] jepsen worker 54 - jepsen.util 3474 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,866] jepsen worker 31 - jepsen.util 3991 :invoke :write [10 1] INFO [2022-03-13 17:17:21,867] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:21,895] jepsen worker 46 - jepsen.util 4006 :fail :write [15 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,896] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 23] INFO [2022-03-13 17:17:21,926] jepsen worker 60 - jepsen.util 690 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,927] jepsen worker 60 - jepsen.util 690 :invoke :write [20 35] INFO [2022-03-13 17:17:21,929] jepsen worker 56 - jepsen.util 3926 :fail :write [18 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,929] jepsen worker 56 - jepsen.util 3926 :invoke :read [18 nil] INFO [2022-03-13 17:17:21,930] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,930] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:17:21,932] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6842105263157895}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4764705882352941}} INFO [2022-03-13 17:17:21,932] jepsen worker nemesis - couchbase.util Rebalance check count: 549 INFO [2022-03-13 17:17:21,932] jepsen worker nemesis - couchbase.util Rebalance stuck count: 439 INFO [2022-03-13 17:17:21,935] jepsen worker 64 - jepsen.util 3934 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,935] jepsen worker 64 - jepsen.util 3934 :invoke :read [21 nil] INFO [2022-03-13 17:17:21,961] jepsen worker 62 - jepsen.util 512 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,961] jepsen worker 62 - jepsen.util 512 :invoke :write [20 12] INFO [2022-03-13 17:17:21,967] jepsen worker 20 - jepsen.util 4070 :fail :write [6 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,967] jepsen worker 54 - jepsen.util 3474 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:21,968] jepsen worker 20 - jepsen.util 4070 :invoke :read [6 nil] INFO [2022-03-13 17:17:21,968] jepsen worker 54 - jepsen.util 3474 :invoke :read [18 nil] INFO [2022-03-13 17:17:22,073] jepsen worker 56 - jepsen.util 3926 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,073] jepsen worker 64 - jepsen.util 3934 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,073] jepsen worker 20 - jepsen.util 4070 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,073] jepsen worker 54 - jepsen.util 3474 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,075] jepsen worker 64 - jepsen.util 3934 :invoke :write [21 12] INFO [2022-03-13 17:17:22,075] jepsen worker 54 - jepsen.util 3474 :invoke :write [18 11] INFO [2022-03-13 17:17:22,076] jepsen worker 20 - jepsen.util 4070 :invoke :write [6 22] INFO [2022-03-13 17:17:22,076] jepsen worker 56 - jepsen.util 3926 :invoke :write [18 13] INFO [2022-03-13 17:17:22,173] jepsen worker 4 - jepsen.util 2434 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,174] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 36] INFO [2022-03-13 17:17:22,175] jepsen worker 38 - jepsen.util 2648 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,175] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,175] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:22,175] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 23] INFO [2022-03-13 17:17:22,180] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,181] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 14] INFO [2022-03-13 17:17:22,186] jepsen worker 36 - jepsen.util 2376 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,187] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 1] INFO [2022-03-13 17:17:22,231] jepsen worker 37 - jepsen.util 2287 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,232] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 41] INFO [2022-03-13 17:17:22,249] jepsen worker 5 - jepsen.util 2615 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,250] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 17] INFO [2022-03-13 17:17:22,282] jepsen worker 87 - jepsen.util 2787 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,282] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 17] INFO [2022-03-13 17:17:22,325] jepsen worker 38 - jepsen.util 2648 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,326] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:22,336] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,336] jepsen worker 14 - jepsen.util 374 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,336] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:17:22,337] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,337] jepsen worker 14 - jepsen.util 374 :invoke :write [4 24] INFO [2022-03-13 17:17:22,337] jepsen worker 77 - jepsen.util 707 :invoke :write [25 18] INFO [2022-03-13 17:17:22,356] jepsen worker 36 - jepsen.util 2376 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,357] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 15] INFO [2022-03-13 17:17:22,368] jepsen worker 3 - jepsen.util 2163 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,369] jepsen worker 5 - jepsen.util 2615 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,369] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:22,370] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:22,372] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,374] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 35] INFO [2022-03-13 17:17:22,439] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,439] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:22,447] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,448] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 18] INFO [2022-03-13 17:17:22,484] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,489] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:22,546] jepsen worker 76 - jepsen.util 796 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,547] jepsen worker 76 - jepsen.util 796 :invoke :write [25 36] INFO [2022-03-13 17:17:22,548] jepsen worker 4 - jepsen.util 2434 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,548] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:22,557] jepsen worker 36 - jepsen.util 2376 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,558] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:22,561] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,562] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 29] INFO [2022-03-13 17:17:22,573] jepsen worker 5 - jepsen.util 2615 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,574] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 16] INFO [2022-03-13 17:17:22,633] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,633] jepsen worker 44 - jepsen.util 674 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,634] jepsen worker 87 - jepsen.util 2787 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,634] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,634] jepsen worker 44 - jepsen.util 674 :invoke :write [14 0] INFO [2022-03-13 17:17:22,635] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 36] INFO [2022-03-13 17:17:22,635] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 4] INFO [2022-03-13 17:17:22,637] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:22,748] jepsen worker 26 - jepsen.util 566 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,748] jepsen worker 26 - jepsen.util 566 :invoke :write [8 14] INFO [2022-03-13 17:17:22,750] jepsen worker 7 - jepsen.util 637 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:22,751] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:22,938] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6842105263157895}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4764705882352941}} INFO [2022-03-13 17:17:22,938] jepsen worker nemesis - couchbase.util Rebalance check count: 550 INFO [2022-03-13 17:17:22,938] jepsen worker nemesis - couchbase.util Rebalance stuck count: 440 INFO [2022-03-13 17:17:23,123] jepsen worker 89 - jepsen.util 2699 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,125] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 14] INFO [2022-03-13 17:17:23,129] jepsen worker 37 - jepsen.util 2287 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,129] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,132] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,133] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,133] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 15] INFO [2022-03-13 17:17:23,133] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,135] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,136] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,137] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,139] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,140] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,140] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,142] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,142] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,144] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,144] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,145] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,148] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,149] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,150] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,150] jepsen worker 23 - jepsen.util 563 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,151] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:23,152] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,153] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 4] INFO [2022-03-13 17:17:23,156] jepsen worker 80 - jepsen.util 800 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,156] jepsen worker 80 - jepsen.util 800 :invoke :write [26 29] INFO [2022-03-13 17:17:23,157] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,157] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:23,167] jepsen worker 5 - jepsen.util 2615 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,168] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:23,169] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,170] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:23,176] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,176] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:23,183] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,184] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 18] INFO [2022-03-13 17:17:23,192] jepsen worker 12 - jepsen.util 912 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,193] jepsen worker 12 - jepsen.util 912 :invoke :write [4 26] INFO [2022-03-13 17:17:23,233] jepsen worker 87 - jepsen.util 2787 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,234] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:23,245] jepsen worker 3 - jepsen.util 2163 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,246] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:23,259] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,260] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:23,261] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,261] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 17] INFO [2022-03-13 17:17:23,263] jepsen worker 36 - jepsen.util 2376 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,264] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 25] INFO [2022-03-13 17:17:23,291] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,291] jepsen worker 38 - jepsen.util 2648 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,292] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,292] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 29] INFO [2022-03-13 17:17:23,332] jepsen worker 37 - jepsen.util 2287 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,333] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 48] INFO [2022-03-13 17:17:23,334] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,335] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 35] INFO [2022-03-13 17:17:23,343] jepsen worker 88 - jepsen.util 2248 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,344] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 24] INFO [2022-03-13 17:17:23,406] jepsen worker 36 - jepsen.util 2376 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,407] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,411] jepsen worker 4 - jepsen.util 2434 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,412] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 13] INFO [2022-03-13 17:17:23,430] jepsen worker 87 - jepsen.util 2787 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,430] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:23,495] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,495] jepsen worker 37 - jepsen.util 2287 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,496] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,497] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 39] INFO [2022-03-13 17:17:23,502] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,503] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,504] jepsen worker 5 - jepsen.util 2615 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,505] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,506] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 27] INFO [2022-03-13 17:17:23,507] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 41] INFO [2022-03-13 17:17:23,510] jepsen worker 66 - jepsen.util 3576 :info :write [22 19] :RequestTimeoutException INFO [2022-03-13 17:17:23,511] jepsen worker 66 - jepsen.util 3666 :invoke :read [22 nil] INFO [2022-03-13 17:17:23,533] jepsen worker 40 - jepsen.util 3910 :info :write [13 41] :RequestTimeoutException INFO [2022-03-13 17:17:23,534] jepsen worker 40 - jepsen.util 4000 :invoke :write [13 48] INFO [2022-03-13 17:17:23,540] jepsen worker 33 - jepsen.util 753 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,541] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:23,541] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,542] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:23,555] jepsen worker 3 - jepsen.util 2163 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,555] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:23,580] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,581] jepsen worker 35 - jepsen.util 935 :invoke :write [11 15] INFO [2022-03-13 17:17:23,754] jepsen worker 78 - jepsen.util 708 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,754] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:23,781] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,782] jepsen worker 21 - jepsen.util 831 :invoke :write [7 48] INFO [2022-03-13 17:17:23,919] jepsen worker 38 - jepsen.util 2648 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,919] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,919] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 49] INFO [2022-03-13 17:17:23,920] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:23,925] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,925] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:23,934] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,935] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 26] INFO [2022-03-13 17:17:23,939] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,940] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:23,945] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6842105263157895}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4764705882352941}} INFO [2022-03-13 17:17:23,945] jepsen worker nemesis - couchbase.util Rebalance check count: 551 INFO [2022-03-13 17:17:23,945] jepsen worker nemesis - couchbase.util Rebalance stuck count: 441 INFO [2022-03-13 17:17:23,946] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,946] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:23,952] jepsen worker 53 - jepsen.util 593 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,952] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:23,967] jepsen worker 37 - jepsen.util 2287 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,968] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:23,982] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,983] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:23,998] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,998] jepsen worker 36 - jepsen.util 2376 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:23,999] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,000] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,000] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,001] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,001] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,002] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,002] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,003] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,003] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,004] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,004] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,005] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,005] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,006] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 8] INFO [2022-03-13 17:17:24,006] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,006] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 41] INFO [2022-03-13 17:17:24,026] jepsen worker 5 - jepsen.util 2615 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,027] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 9] INFO [2022-03-13 17:17:24,054] jepsen worker 4 - jepsen.util 2434 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,055] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 27] INFO [2022-03-13 17:17:24,072] jepsen worker 38 - jepsen.util 2648 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,073] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,073] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 22] INFO [2022-03-13 17:17:24,073] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:24,074] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,075] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 47] INFO [2022-03-13 17:17:24,129] jepsen worker 5 - jepsen.util 2615 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,129] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,130] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,131] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,132] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,133] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,134] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,134] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,137] jepsen worker 88 - jepsen.util 2248 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,138] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 41] INFO [2022-03-13 17:17:24,142] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,143] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 5] INFO [2022-03-13 17:17:24,147] jepsen worker 37 - jepsen.util 2287 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,147] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,152] jepsen worker 13 - jepsen.util 733 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,152] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:24,157] jepsen worker 6 - jepsen.util 636 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,157] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:24,191] jepsen worker 3 - jepsen.util 2163 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,192] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 24] INFO [2022-03-13 17:17:24,196] jepsen worker 38 - jepsen.util 2648 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,197] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 34] INFO [2022-03-13 17:17:24,231] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,231] jepsen worker 36 - jepsen.util 2376 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,231] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 25] INFO [2022-03-13 17:17:24,232] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 41] INFO [2022-03-13 17:17:24,239] jepsen worker 4 - jepsen.util 2434 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,240] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 31] INFO [2022-03-13 17:17:24,270] jepsen worker 89 - jepsen.util 2699 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,271] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 17] INFO [2022-03-13 17:17:24,300] jepsen worker 3 - jepsen.util 2163 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,300] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 28] INFO [2022-03-13 17:17:24,329] jepsen worker 87 - jepsen.util 2787 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,329] jepsen worker 87 - jepsen.util 2787 :invoke :read [29 nil] INFO [2022-03-13 17:17:24,333] jepsen worker 88 - jepsen.util 2248 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,334] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:24,363] jepsen worker 37 - jepsen.util 2287 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,364] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,369] jepsen worker 14 - jepsen.util 374 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,370] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,370] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:24,371] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:24,373] jepsen worker 22 - jepsen.util 652 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,374] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:24,398] jepsen worker 4 - jepsen.util 2434 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,399] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,400] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,401] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,402] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,403] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,404] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,404] jepsen worker 76 - jepsen.util 796 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,404] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,405] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,405] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:24,406] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:24,407] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,407] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 37] INFO [2022-03-13 17:17:24,408] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,409] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:24,416] jepsen worker 5 - jepsen.util 2615 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,416] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 29] INFO [2022-03-13 17:17:24,456] jepsen worker 89 - jepsen.util 2699 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,457] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,457] jepsen worker 89 - jepsen.util 2699 :invoke :read [29 nil] INFO [2022-03-13 17:17:24,458] jepsen worker 88 - jepsen.util 2248 :invoke :read [29 nil] INFO [2022-03-13 17:17:24,477] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,477] jepsen worker 36 - jepsen.util 2376 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,478] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,478] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 18] INFO [2022-03-13 17:17:24,482] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,483] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 3] INFO [2022-03-13 17:17:24,532] jepsen worker 87 - jepsen.util 2787 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,532] jepsen worker 38 - jepsen.util 2648 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,532] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 44] INFO [2022-03-13 17:17:24,533] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,533] jepsen worker 89 - jepsen.util 2699 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,533] jepsen worker 3 - jepsen.util 2163 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,533] jepsen worker 88 - jepsen.util 2248 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,622] jepsen worker 88 - jepsen.util 2248 :invoke :write [29 47] INFO [2022-03-13 17:17:24,572] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,572] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,622] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:24,572] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,622] jepsen worker 76 - jepsen.util 796 :invoke :write [25 28] INFO [2022-03-13 17:17:24,622] jepsen worker 13 - jepsen.util 733 :invoke :write [4 27] INFO [2022-03-13 17:17:24,588] jepsen worker 37 - jepsen.util 2287 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,615] jepsen worker 4 - jepsen.util 2434 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,622] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,621] jepsen worker 89 - jepsen.util 2699 :invoke :read [29 nil] INFO [2022-03-13 17:17:24,623] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,622] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 10] INFO [2022-03-13 17:17:24,629] jepsen worker 89 - jepsen.util 2699 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,630] jepsen worker 89 - jepsen.util 2699 :invoke :write [29 23] INFO [2022-03-13 17:17:24,639] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,640] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,641] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,641] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 16] INFO [2022-03-13 17:17:24,646] jepsen worker 5 - jepsen.util 2615 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,650] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:24,652] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,652] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 23] INFO [2022-03-13 17:17:24,654] jepsen worker 36 - jepsen.util 2376 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,654] jepsen worker 36 - jepsen.util 2376 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,658] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,659] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,664] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,664] jepsen worker 36 - jepsen.util 2376 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,664] jepsen worker 36 - jepsen.util 2376 :invoke :write [12 26] INFO [2022-03-13 17:17:24,665] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,667] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,668] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,669] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,670] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,672] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,673] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 9] INFO [2022-03-13 17:17:24,763] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,764] jepsen worker 38 - jepsen.util 2648 :invoke :read [12 nil] INFO [2022-03-13 17:17:24,771] jepsen worker 38 - jepsen.util 2648 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,772] jepsen worker 38 - jepsen.util 2648 :invoke :write [12 25] INFO [2022-03-13 17:17:24,787] jepsen worker 79 - jepsen.util 1159 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,788] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 20] INFO [2022-03-13 17:17:24,791] jepsen worker 80 - jepsen.util 800 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,792] jepsen worker 80 - jepsen.util 800 :invoke :write [26 32] INFO [2022-03-13 17:17:24,795] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,796] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:24,798] jepsen worker 52 - jepsen.util 682 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,799] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:24,801] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,802] jepsen worker 75 - jepsen.util 525 :invoke :write [25 17] INFO [2022-03-13 17:17:24,869] jepsen worker 37 - jepsen.util 2287 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,869] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 19] INFO [2022-03-13 17:17:24,880] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,880] jepsen worker 42 - jepsen.util 402 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,882] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:17:24,882] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:24,885] jepsen worker 43 - jepsen.util 673 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,887] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:24,897] jepsen worker 84 - jepsen.util 4224 :info :write [28 24] :RequestTimeoutException INFO [2022-03-13 17:17:24,897] jepsen worker 84 - jepsen.util 4314 :invoke :read [28 nil] INFO [2022-03-13 17:17:24,930] jepsen worker 27 - jepsen.util 3987 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,930] jepsen worker 47 - jepsen.util 3737 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,931] jepsen worker 47 - jepsen.util 3737 :invoke :write [15 46] INFO [2022-03-13 17:17:24,931] jepsen worker 27 - jepsen.util 3987 :invoke :write [9 6] INFO [2022-03-13 17:17:24,947] jepsen worker 4 - jepsen.util 2434 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,947] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 45] INFO [2022-03-13 17:17:24,951] jepsen worker 85 - jepsen.util 3505 :info :write [28 25] :RequestTimeoutException INFO [2022-03-13 17:17:24,951] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.6842105263157895}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4764705882352941}} INFO [2022-03-13 17:17:24,951] jepsen worker nemesis - couchbase.util Rebalance check count: 552 INFO [2022-03-13 17:17:24,951] jepsen worker nemesis - couchbase.util Rebalance stuck count: 442 INFO [2022-03-13 17:17:24,951] jepsen worker 85 - jepsen.util 3595 :invoke :write [28 0] INFO [2022-03-13 17:17:24,959] jepsen worker 24 - jepsen.util 564 :info :write [8 27] :RequestTimeoutException INFO [2022-03-13 17:17:24,960] jepsen worker 24 - jepsen.util 654 :invoke :write [8 44] INFO [2022-03-13 17:17:24,989] jepsen worker 35 - jepsen.util 935 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,989] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,990] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:24,991] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:17:24,991] jepsen worker 13 - jepsen.util 733 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:24,992] jepsen worker 13 - jepsen.util 733 :invoke :write [4 16] INFO [2022-03-13 17:17:25,025] jepsen worker 63 - jepsen.util 4023 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,026] jepsen worker 63 - jepsen.util 4023 :invoke :read [21 nil] INFO [2022-03-13 17:17:25,135] jepsen worker 87 - jepsen.util 2787 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,136] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 14] INFO [2022-03-13 17:17:25,202] jepsen worker 61 - jepsen.util 781 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,203] jepsen worker 61 - jepsen.util 781 :invoke :write [20 42] INFO [2022-03-13 17:17:25,220] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,220] jepsen worker 26 - jepsen.util 566 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,222] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:25,222] jepsen worker 53 - jepsen.util 593 :invoke :write [17 41] INFO [2022-03-13 17:17:25,244] jepsen worker 63 - jepsen.util 4023 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,245] jepsen worker 63 - jepsen.util 4023 :invoke :read [21 nil] INFO [2022-03-13 17:17:25,300] jepsen worker 28 - jepsen.util 3808 :info :write [9 27] :RequestTimeoutException INFO [2022-03-13 17:17:25,301] jepsen worker 28 - jepsen.util 3898 :invoke :write [9 3] INFO [2022-03-13 17:17:25,330] jepsen worker 63 - jepsen.util 4023 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,331] jepsen worker 63 - jepsen.util 4023 :invoke :read [21 nil] INFO [2022-03-13 17:17:25,338] jepsen worker 3 - jepsen.util 2163 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,339] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:25,340] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,342] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:25,344] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,345] jepsen worker 3 - jepsen.util 2163 :invoke :read [1 nil] INFO [2022-03-13 17:17:25,370] jepsen worker 3 - jepsen.util 2163 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,371] jepsen worker 3 - jepsen.util 2163 :invoke :write [1 22] INFO [2022-03-13 17:17:25,411] jepsen worker 63 - jepsen.util 4023 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,411] jepsen worker 63 - jepsen.util 4023 :invoke :read [21 nil] INFO [2022-03-13 17:17:25,450] jepsen worker 11 - jepsen.util 3701 :info :write [3 32] :RequestTimeoutException INFO [2022-03-13 17:17:25,450] jepsen worker 11 - jepsen.util 3791 :invoke :read [3 nil] INFO [2022-03-13 17:17:25,542] jepsen worker 37 - jepsen.util 2287 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,542] jepsen worker 37 - jepsen.util 2287 :invoke :read [12 nil] INFO [2022-03-13 17:17:25,553] jepsen worker 63 - jepsen.util 4023 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,553] jepsen worker 63 - jepsen.util 4023 :invoke :write [21 26] INFO [2022-03-13 17:17:25,604] jepsen worker 44 - jepsen.util 674 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,605] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:25,606] jepsen worker 8 - jepsen.util 728 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,606] jepsen worker 8 - jepsen.util 728 :invoke :write [2 36] INFO [2022-03-13 17:17:25,615] jepsen worker 76 - jepsen.util 796 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,616] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:25,636] jepsen worker 37 - jepsen.util 2287 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,637] jepsen worker 37 - jepsen.util 2287 :invoke :write [12 43] INFO [2022-03-13 17:17:25,759] jepsen worker 73 - jepsen.util 3943 :info :write [24 12] :SyncWriteAmbiguous INFO [2022-03-13 17:17:25,760] jepsen worker 73 - jepsen.util 4033 :invoke :read [24 nil] INFO [2022-03-13 17:17:25,769] jepsen worker 5 - jepsen.util 2615 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,770] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 37] INFO [2022-03-13 17:17:25,809] jepsen worker 77 - jepsen.util 707 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,809] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,810] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:25,811] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:25,809] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,811] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:25,815] jepsen worker 60 - jepsen.util 690 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,815] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,815] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:25,816] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,816] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,817] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:25,818] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:25,819] jepsen worker 26 - jepsen.util 566 :invoke :write [8 31] INFO [2022-03-13 17:17:25,819] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:25,820] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:17:25,857] jepsen worker 4 - jepsen.util 2434 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,857] jepsen worker 4 - jepsen.util 2434 :invoke :read [1 nil] INFO [2022-03-13 17:17:25,858] jepsen worker 4 - jepsen.util 2434 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,859] jepsen worker 4 - jepsen.util 2434 :invoke :write [1 11] INFO [2022-03-13 17:17:25,919] jepsen worker 73 - jepsen.util 4033 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,919] jepsen worker 73 - jepsen.util 4033 :invoke :read [24 nil] INFO [2022-03-13 17:17:25,932] jepsen worker 45 - jepsen.util 4095 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,933] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:25,959] jepsen worker 87 - jepsen.util 2787 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:25,959] jepsen worker 87 - jepsen.util 2787 :invoke :write [29 16] INFO [2022-03-13 17:17:25,962] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7017543859649122}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4941176470588236}} INFO [2022-03-13 17:17:25,962] jepsen worker nemesis - couchbase.util Rebalance check count: 553 INFO [2022-03-13 17:17:25,962] jepsen worker nemesis - couchbase.util Rebalance stuck count: 442 INFO [2022-03-13 17:17:26,005] jepsen worker 82 - jepsen.util 4042 :info :write [27 15] :RequestTimeoutException INFO [2022-03-13 17:17:26,006] jepsen worker 82 - jepsen.util 4132 :invoke :read [27 nil] INFO [2022-03-13 17:17:26,024] jepsen worker 73 - jepsen.util 4033 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,025] jepsen worker 73 - jepsen.util 4033 :invoke :read [24 nil] INFO [2022-03-13 17:17:26,028] jepsen worker 11 - jepsen.util 3791 :ok :read [3 25] INFO [2022-03-13 17:17:26,029] jepsen worker 11 - jepsen.util 3791 :invoke :write [3 7] INFO [2022-03-13 17:17:26,090] jepsen worker 73 - jepsen.util 4033 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,090] jepsen worker 73 - jepsen.util 4033 :invoke :read [24 nil] INFO [2022-03-13 17:17:26,092] jepsen worker 73 - jepsen.util 4033 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,093] jepsen worker 73 - jepsen.util 4033 :invoke :write [24 29] INFO [2022-03-13 17:17:26,180] jepsen worker 29 - jepsen.util 4169 :info :write [9 39] :RequestTimeoutException INFO [2022-03-13 17:17:26,181] jepsen worker 29 - jepsen.util 4259 :invoke :read [9 nil] INFO [2022-03-13 17:17:26,238] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,238] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,238] jepsen worker 21 - jepsen.util 831 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,238] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,239] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,240] jepsen worker 14 - jepsen.util 374 :invoke :write [4 20] INFO [2022-03-13 17:17:26,241] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:26,242] jepsen worker 5 - jepsen.util 2615 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,242] jepsen worker 33 - jepsen.util 753 :invoke :write [11 46] INFO [2022-03-13 17:17:26,244] jepsen worker 34 - jepsen.util 934 :invoke :write [11 34] INFO [2022-03-13 17:17:26,244] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:17:26,245] jepsen worker 5 - jepsen.util 2615 :invoke :read [1 nil] INFO [2022-03-13 17:17:26,246] jepsen worker 5 - jepsen.util 2615 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,247] jepsen worker 5 - jepsen.util 2615 :invoke :write [1 5] INFO [2022-03-13 17:17:26,308] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,308] jepsen worker 80 - jepsen.util 800 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,308] jepsen worker 80 - jepsen.util 800 :invoke :write [26 0] INFO [2022-03-13 17:17:26,309] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:26,318] jepsen worker 75 - jepsen.util 525 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,319] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:26,440] jepsen worker 62 - jepsen.util 512 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,440] jepsen worker 19 - jepsen.util 3619 :info :write [6 2] :RequestTimeoutException INFO [2022-03-13 17:17:26,440] jepsen worker 62 - jepsen.util 512 :invoke :write [20 33] INFO [2022-03-13 17:17:26,441] jepsen worker 19 - jepsen.util 3709 :invoke :write [6 12] INFO [2022-03-13 17:17:26,495] jepsen worker 24 - jepsen.util 654 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,496] jepsen worker 24 - jepsen.util 654 :invoke :write [8 5] INFO [2022-03-13 17:17:26,704] jepsen worker 79 - jepsen.util 1159 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,704] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] WARN [2022-03-13 17:17:26,857] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":57546,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":565699,"total_server_duration_us":242490,"encode_duration_us":18129,"operation_id":"0xfa67","timeout_ms":10000,"last_server_duration_us":40802,"total_duration_us":10010124},{"operation_name":"upsert","last_dispatch_duration_us":8035640,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8513069,"total_server_duration_us":241723,"encode_duration_us":16832,"operation_id":"0xfa65","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10005603},{"operation_name":"upsert","last_dispatch_duration_us":44806,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1098921,"total_server_duration_us":622172,"encode_duration_us":19714,"operation_id":"0xf947","timeout_ms":10000,"last_server_duration_us":32161,"total_duration_us":10005181},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":56564,"operation_id":"0xfb6a","timeout_ms":10000,"total_duration_us":10004500},{"operation_name":"upsert","last_dispatch_duration_us":32278,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":970800,"total_server_duration_us":498409,"encode_duration_us":21289,"operation_id":"0xfa27","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10004239},{"operation_name":"upsert","last_dispatch_duration_us":30541,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":942111,"total_server_duration_us":262475,"encode_duration_us":26284,"operation_id":"0xfbc3","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10004158},{"operation_name":"upsert","last_dispatch_duration_us":9065833,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9065833,"total_server_duration_us":9033121,"encode_duration_us":17215,"operation_id":"0xfa66","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":9069330},{"operation_name":"get","last_dispatch_duration_us":8040693,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8040693,"total_server_duration_us":49,"operation_id":"0xfa98","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8041809},{"operation_name":"upsert","last_dispatch_duration_us":1913551,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1913551,"total_server_duration_us":26774,"encode_duration_us":19825,"operation_id":"0xfccd","timeout_ms":10000,"last_server_duration_us":26774,"total_duration_us":4532824},{"operation_name":"upsert","last_dispatch_duration_us":1875930,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1875930,"total_server_duration_us":339174,"encode_duration_us":22181,"operation_id":"0xfccc","timeout_ms":10000,"last_server_duration_us":339174,"total_duration_us":4497021}],"total_count":66}} INFO [2022-03-13 17:17:26,859] jepsen worker 64 - jepsen.util 3934 :fail :write [21 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,859] jepsen worker 64 - jepsen.util 3934 :invoke :write [21 26] INFO [2022-03-13 17:17:26,874] jepsen worker 66 - jepsen.util 3666 :ok :read [22 30] INFO [2022-03-13 17:17:26,875] jepsen worker 66 - jepsen.util 3666 :invoke :write [22 49] INFO [2022-03-13 17:17:26,895] jepsen worker 73 - jepsen.util 4033 :fail :write [24 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,896] jepsen worker 73 - jepsen.util 4033 :invoke :read [24 nil] INFO [2022-03-13 17:17:26,900] jepsen worker 53 - jepsen.util 593 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,900] jepsen worker 13 - jepsen.util 733 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,900] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:26,900] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,900] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,901] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,901] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,901] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:26,902] jepsen worker 75 - jepsen.util 525 :invoke :write [25 9] INFO [2022-03-13 17:17:26,903] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:26,904] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:26,904] jepsen worker 60 - jepsen.util 690 :invoke :write [20 45] INFO [2022-03-13 17:17:26,906] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:26,906] jepsen worker 76 - jepsen.util 796 :invoke :write [25 37] INFO [2022-03-13 17:17:26,928] jepsen worker 73 - jepsen.util 4033 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,928] jepsen worker 64 - jepsen.util 3934 :fail :write [21 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,929] jepsen worker 73 - jepsen.util 4033 :invoke :write [24 10] INFO [2022-03-13 17:17:26,930] jepsen worker 64 - jepsen.util 3934 :invoke :write [21 1] INFO [2022-03-13 17:17:26,960] jepsen worker 46 - jepsen.util 4006 :fail :write [15 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:26,961] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 24] INFO [2022-03-13 17:17:26,970] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7076023391812866}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4941176470588236}} INFO [2022-03-13 17:17:26,970] jepsen worker nemesis - couchbase.util Rebalance check count: 554 INFO [2022-03-13 17:17:26,970] jepsen worker nemesis - couchbase.util Rebalance stuck count: 442 INFO [2022-03-13 17:17:27,008] jepsen worker 72 - jepsen.util 3852 :info :write [24 13] :RequestTimeoutException INFO [2022-03-13 17:17:27,009] jepsen worker 72 - jepsen.util 3942 :invoke :write [24 30] INFO [2022-03-13 17:17:27,029] jepsen worker 73 - jepsen.util 4033 :fail :write [24 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,031] jepsen worker 73 - jepsen.util 4033 :invoke :read [24 nil] INFO [2022-03-13 17:17:27,104] jepsen worker 64 - jepsen.util 3934 :fail :write [21 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,105] jepsen worker 64 - jepsen.util 3934 :invoke :write [21 31] INFO [2022-03-13 17:17:27,133] jepsen worker 46 - jepsen.util 4006 :fail :write [15 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,134] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:27,158] jepsen worker 9 - jepsen.util 3339 :info :write [3 18] :RequestTimeoutException INFO [2022-03-13 17:17:27,159] jepsen worker 9 - jepsen.util 3429 :invoke :read [3 nil] INFO [2022-03-13 17:17:27,168] jepsen worker 73 - jepsen.util 4033 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,169] jepsen worker 73 - jepsen.util 4033 :invoke :write [24 33] INFO [2022-03-13 17:17:27,198] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,199] jepsen worker 46 - jepsen.util 4006 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,203] jepsen worker 46 - jepsen.util 4006 :invoke :read [15 nil] INFO [2022-03-13 17:17:27,203] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:27,236] jepsen worker 10 - jepsen.util 3880 :info :write [3 42] :RequestTimeoutException INFO [2022-03-13 17:17:27,237] jepsen worker 10 - jepsen.util 3970 :invoke :write [3 5] INFO [2022-03-13 17:17:27,272] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,272] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:27,274] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:27,274] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,275] jepsen worker 35 - jepsen.util 935 :invoke :write [11 14] INFO [2022-03-13 17:17:27,275] jepsen worker 44 - jepsen.util 674 :invoke :write [14 12] INFO [2022-03-13 17:17:27,276] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,277] jepsen worker 21 - jepsen.util 831 :invoke :write [7 18] INFO [2022-03-13 17:17:27,278] jepsen worker 12 - jepsen.util 912 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,278] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,278] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,279] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,279] jepsen worker 7 - jepsen.util 637 :invoke :write [2 3] INFO [2022-03-13 17:17:27,280] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:27,280] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:27,281] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:27,316] jepsen worker 46 - jepsen.util 4006 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,317] jepsen worker 46 - jepsen.util 4006 :invoke :write [15 49] INFO [2022-03-13 17:17:27,327] jepsen worker 75 - jepsen.util 525 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,327] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:27,424] jepsen worker 14 - jepsen.util 374 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,424] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:27,432] jepsen worker 61 - jepsen.util 781 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,432] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,432] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,432] jepsen worker 61 - jepsen.util 781 :invoke :write [20 35] INFO [2022-03-13 17:17:27,433] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,433] jepsen worker 78 - jepsen.util 708 :invoke :write [26 43] INFO [2022-03-13 17:17:27,433] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:27,434] jepsen worker 77 - jepsen.util 707 :invoke :write [25 16] INFO [2022-03-13 17:17:27,438] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,440] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 24] INFO [2022-03-13 17:17:27,449] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,450] jepsen worker 23 - jepsen.util 563 :invoke :write [7 40] INFO [2022-03-13 17:17:27,456] jepsen worker 74 - jepsen.util 3944 :info :write [24 36] :RequestTimeoutException INFO [2022-03-13 17:17:27,457] jepsen worker 74 - jepsen.util 4034 :invoke :read [24 nil] INFO [2022-03-13 17:17:27,481] jepsen worker 26 - jepsen.util 566 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,482] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:27,485] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,485] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,485] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:27,486] jepsen worker 53 - jepsen.util 593 :invoke :write [17 47] INFO [2022-03-13 17:17:27,486] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:27,487] jepsen worker 43 - jepsen.util 673 :invoke :write [14 18] INFO [2022-03-13 17:17:27,509] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,510] jepsen worker 42 - jepsen.util 402 :invoke :write [14 1] INFO [2022-03-13 17:17:27,511] jepsen worker 33 - jepsen.util 753 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,512] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:27,628] jepsen worker 68 - jepsen.util 3938 :info :write [22 20] :RequestTimeoutException INFO [2022-03-13 17:17:27,629] jepsen worker 68 - jepsen.util 4028 :invoke :read [22 nil] INFO [2022-03-13 17:17:27,637] jepsen worker 76 - jepsen.util 796 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,637] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,638] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,639] jepsen worker 75 - jepsen.util 525 :invoke :write [25 34] INFO [2022-03-13 17:17:27,640] jepsen worker 13 - jepsen.util 733 :invoke :write [4 16] INFO [2022-03-13 17:17:27,641] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:27,651] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,652] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:17:27,687] jepsen worker 8 - jepsen.util 728 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,687] jepsen worker 8 - jepsen.util 728 :invoke :write [2 43] INFO [2022-03-13 17:17:27,796] jepsen worker 58 - jepsen.util 3478 :info :write [19 17] :SyncWriteAmbiguous INFO [2022-03-13 17:17:27,797] jepsen worker 58 - jepsen.util 3568 :invoke :write [19 6] INFO [2022-03-13 17:17:27,843] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,844] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:17:27,847] jepsen worker 60 - jepsen.util 690 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,847] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,848] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,848] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:27,849] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:27,849] jepsen worker 52 - jepsen.util 682 :invoke :write [17 14] INFO [2022-03-13 17:17:27,850] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:27,851] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:28,003] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7076023391812866}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.4941176470588236}} INFO [2022-03-13 17:17:28,003] jepsen worker nemesis - couchbase.util Rebalance check count: 555 INFO [2022-03-13 17:17:28,003] jepsen worker nemesis - couchbase.util Rebalance stuck count: 443 INFO [2022-03-13 17:17:28,046] jepsen worker 78 - jepsen.util 708 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,047] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:28,103] jepsen worker 80 - jepsen.util 800 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,104] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:28,171] jepsen worker 75 - jepsen.util 525 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,171] jepsen worker 53 - jepsen.util 593 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,172] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:28,173] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:28,247] jepsen worker 34 - jepsen.util 934 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,247] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,248] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:28,248] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:28,252] jepsen worker 62 - jepsen.util 512 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,252] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,252] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,252] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,252] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:28,253] jepsen worker 12 - jepsen.util 912 :invoke :write [4 37] INFO [2022-03-13 17:17:28,254] jepsen worker 6 - jepsen.util 636 :invoke :write [2 33] INFO [2022-03-13 17:17:28,254] jepsen worker 70 - jepsen.util 880 :invoke :write [23 22] INFO [2022-03-13 17:17:28,307] jepsen worker 35 - jepsen.util 935 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,307] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:28,364] jepsen worker 18 - jepsen.util 3888 :info :write [6 15] :SyncWriteAmbiguous INFO [2022-03-13 17:17:28,365] jepsen worker 18 - jepsen.util 3978 :invoke :read [6 nil] WARN [2022-03-13 17:17:28,376] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":443929,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10310540,"total_server_duration_us":212095,"encode_duration_us":16658,"operation_id":"0xfb95","timeout_ms":10000,"last_server_duration_us":37241,"total_duration_us":10009835},{"operation_name":"upsert","last_dispatch_duration_us":391555,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10114270,"total_server_duration_us":203446,"encode_duration_us":17692,"operation_id":"0xfb8f","timeout_ms":10000,"last_server_duration_us":77851,"total_duration_us":10005807},{"operation_name":"upsert","last_dispatch_duration_us":192147,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":818963,"total_server_duration_us":102787,"encode_duration_us":22702,"operation_id":"0xfbcc","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":10005786},{"operation_name":"upsert","last_dispatch_duration_us":12843872,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":12843872,"total_server_duration_us":55182,"encode_duration_us":18390,"operation_id":"0xfa5f","timeout_ms":10000,"last_server_duration_us":55182,"total_duration_us":10004483},{"operation_name":"upsert","last_dispatch_duration_us":9063431,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9629390,"total_server_duration_us":9401448,"encode_duration_us":18107,"operation_id":"0xfacb","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10004132},{"operation_name":"upsert","last_dispatch_duration_us":37865,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":4104087,"total_server_duration_us":511088,"encode_duration_us":28897,"operation_id":"0xf987","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10001764},{"operation_name":"get","last_dispatch_duration_us":119363,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":119363,"total_server_duration_us":32,"operation_id":"0xfc47","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":7328764},{"operation_name":"upsert","last_dispatch_duration_us":3319549,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5178532,"total_server_duration_us":377938,"encode_duration_us":24558,"operation_id":"0xfcb1","timeout_ms":10000,"last_server_duration_us":47647,"total_duration_us":6281994},{"operation_name":"upsert","last_dispatch_duration_us":4093141,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":5808511,"total_server_duration_us":689431,"encode_duration_us":21617,"operation_id":"0xfc7e","timeout_ms":10000,"last_server_duration_us":342070,"total_duration_us":5839675},{"operation_name":"get","last_dispatch_duration_us":59049,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":59049,"total_server_duration_us":19,"operation_id":"0xfcdb","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":5736220}],"total_count":48}} WARN [2022-03-13 17:17:28,379] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17456210,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17528944,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9086483,"encode_duration_us":19793,"operation_id":"0xf3f7","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10004522}],"total_count":1}} INFO [2022-03-13 17:17:28,455] jepsen worker 77 - jepsen.util 707 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,456] jepsen worker 77 - jepsen.util 707 :invoke :write [25 7] WARN [2022-03-13 17:17:28,463] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10589337,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10589337,"total_server_duration_us":9227,"operation_id":"0xf912","timeout_ms":30000,"last_server_duration_us":9227,"total_duration_us":10591195},{"operation_name":"upsert","last_dispatch_duration_us":9397872,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9397872,"total_server_duration_us":358048,"encode_duration_us":19320,"operation_id":"0xf9da","timeout_ms":10000,"last_server_duration_us":358048,"total_duration_us":10009764},{"operation_name":"upsert","last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","encode_duration_us":64963,"operation_id":"0xfb6d","timeout_ms":10000,"total_duration_us":10009396},{"operation_name":"upsert","last_dispatch_duration_us":18350237,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":18350237,"total_server_duration_us":9483465,"encode_duration_us":39056,"operation_id":"0xf9b8","timeout_ms":10000,"last_server_duration_us":9483465,"total_duration_us":10006084},{"operation_name":"upsert","last_dispatch_duration_us":51657,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":3797928,"total_server_duration_us":282294,"encode_duration_us":21716,"operation_id":"0xfbbf","timeout_ms":10000,"last_server_duration_us":29323,"total_duration_us":10006034},{"operation_name":"upsert","last_dispatch_duration_us":160626,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8159303,"total_server_duration_us":202923,"encode_duration_us":18202,"operation_id":"0xfc1f","timeout_ms":10000,"last_server_duration_us":57632,"total_duration_us":10002568},{"operation_name":"get","last_dispatch_duration_us":9372953,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":9372953,"total_server_duration_us":92,"operation_id":"0xf9dc","timeout_ms":30000,"last_server_duration_us":92,"total_duration_us":9373371},{"operation_name":"get","last_dispatch_duration_us":8415466,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8415466,"total_server_duration_us":43,"operation_id":"0xf9cb","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8417574},{"operation_name":"get","last_dispatch_duration_us":6393292,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6393292,"total_server_duration_us":56,"operation_id":"0xfcba","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":6393621},{"operation_name":"get","last_dispatch_duration_us":5986348,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":5986348,"total_server_duration_us":8778,"operation_id":"0xfa82","timeout_ms":30000,"last_server_duration_us":8778,"total_duration_us":5988792}],"total_count":22}} WARN [2022-03-13 17:17:28,463] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16385862,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16829574,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9275867,"encode_duration_us":18859,"operation_id":"0xf67e","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10002465},{"operation_name":"upsert","last_dispatch_duration_us":16036176,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19288462,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9589226,"encode_duration_us":17983,"operation_id":"0xf64b","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10005422},{"operation_name":"upsert","last_dispatch_duration_us":18350237,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":18350237,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":9483465,"encode_duration_us":39056,"operation_id":"0xf9b8","timeout_ms":10000,"last_server_duration_us":9483465,"total_duration_us":10006084}],"total_count":3}} WARN [2022-03-13 17:17:28,476] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":198795,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":453741,"total_server_duration_us":63118,"encode_duration_us":20021,"operation_id":"0xfc15","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10010863},{"operation_name":"upsert","last_dispatch_duration_us":45732,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":4618014,"total_server_duration_us":196967,"encode_duration_us":19284,"operation_id":"0xfc4e","timeout_ms":10000,"last_server_duration_us":25275,"total_duration_us":10009245},{"operation_name":"upsert","last_dispatch_duration_us":8301339,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":14306294,"total_server_duration_us":157622,"encode_duration_us":22470,"operation_id":"0xfa4c","timeout_ms":10000,"last_server_duration_us":35011,"total_duration_us":10008126},{"operation_name":"upsert","last_dispatch_duration_us":8342677,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8843818,"total_server_duration_us":119088,"encode_duration_us":18412,"operation_id":"0xfc1c","timeout_ms":10000,"last_server_duration_us":77711,"total_duration_us":10002697},{"operation_name":"upsert","last_dispatch_duration_us":9315877,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9315877,"total_server_duration_us":9035244,"encode_duration_us":17387,"operation_id":"0xfbcb","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":9388031},{"operation_name":"upsert","last_dispatch_duration_us":9122344,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9122344,"total_server_duration_us":9043735,"encode_duration_us":18040,"operation_id":"0xfa91","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":9128700},{"operation_name":"get","last_dispatch_duration_us":8380965,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8380965,"total_server_duration_us":109859,"operation_id":"0xfc40","timeout_ms":30000,"last_server_duration_us":109859,"total_duration_us":8381308},{"operation_name":"upsert","last_dispatch_duration_us":3711246,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5682367,"total_server_duration_us":749979,"encode_duration_us":18652,"operation_id":"0xfc73","timeout_ms":10000,"last_server_duration_us":556382,"total_duration_us":5707365},{"operation_name":"upsert","last_dispatch_duration_us":3903754,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5561506,"total_server_duration_us":1197113,"encode_duration_us":17776,"operation_id":"0xfcdc","timeout_ms":10000,"last_server_duration_us":602237,"total_duration_us":5580379},{"operation_name":"upsert","last_dispatch_duration_us":3297863,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5071369,"total_server_duration_us":570783,"encode_duration_us":19193,"operation_id":"0xfc72","timeout_ms":10000,"last_server_duration_us":387030,"total_duration_us":5105741}],"total_count":31}} INFO [2022-03-13 17:17:28,512] jepsen worker 55 - jepsen.util 3565 :info :write [18 6] :SyncWriteAmbiguous INFO [2022-03-13 17:17:28,513] jepsen worker 55 - jepsen.util 3655 :invoke :read [18 nil] INFO [2022-03-13 17:17:28,518] jepsen worker 21 - jepsen.util 831 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,518] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:28,546] jepsen worker 81 - jepsen.util 4401 :info :write [27 42] :RequestTimeoutException INFO [2022-03-13 17:17:28,547] jepsen worker 81 - jepsen.util 4491 :invoke :write [27 10] WARN [2022-03-13 17:17:28,576] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9063431,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9629390,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9401448,"encode_duration_us":18107,"operation_id":"0xfacb","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10004132},{"operation_name":"upsert","last_dispatch_duration_us":17811522,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27331641,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9186394,"encode_duration_us":43789,"operation_id":"0xf59d","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10005726}],"total_count":2}} WARN [2022-03-13 17:17:28,579] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":30720,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9543080,"total_server_duration_us":224922,"encode_duration_us":20010,"operation_id":"0xf939","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10008845},{"operation_name":"upsert","last_dispatch_duration_us":7957390,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12678640,"total_server_duration_us":303314,"encode_duration_us":17221,"operation_id":"0xfad2","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10002970},{"operation_name":"get","last_dispatch_duration_us":9173032,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9173032,"total_server_duration_us":106305,"operation_id":"0xf98a","timeout_ms":30000,"last_server_duration_us":106305,"total_duration_us":9740707},{"operation_name":"upsert","last_dispatch_duration_us":9106952,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9190687,"total_server_duration_us":9093261,"encode_duration_us":26939,"operation_id":"0xfca9","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9212336},{"operation_name":"get","last_dispatch_duration_us":8893424,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8893424,"total_server_duration_us":10631,"operation_id":"0xf9d2","timeout_ms":30000,"last_server_duration_us":10631,"total_duration_us":8896283},{"operation_name":"upsert","last_dispatch_duration_us":336738,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8470041,"total_server_duration_us":199515,"encode_duration_us":23433,"operation_id":"0xfa6e","timeout_ms":10000,"last_server_duration_us":50301,"total_duration_us":8534064},{"operation_name":"upsert","last_dispatch_duration_us":3888421,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":6041113,"total_server_duration_us":652607,"encode_duration_us":21016,"operation_id":"0xfc77","timeout_ms":10000,"last_server_duration_us":363443,"total_duration_us":6069326},{"operation_name":"upsert","last_dispatch_duration_us":3708785,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3708785,"total_server_duration_us":363714,"encode_duration_us":30348,"operation_id":"0xfd29","timeout_ms":10000,"last_server_duration_us":363714,"total_duration_us":5308907},{"operation_name":"upsert","last_dispatch_duration_us":51909,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":1894374,"total_server_duration_us":517534,"encode_duration_us":17336,"operation_id":"0xfde7","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":5046794},{"operation_name":"upsert","last_dispatch_duration_us":91384,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":1601758,"total_server_duration_us":410827,"encode_duration_us":18039,"operation_id":"0xfdef","timeout_ms":10000,"last_server_duration_us":26241,"total_duration_us":4765370}],"total_count":49}} WARN [2022-03-13 17:17:28,582] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8528301,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8982958,"total_server_duration_us":302667,"encode_duration_us":18424,"operation_id":"0xf9e6","timeout_ms":10000,"last_server_duration_us":22554,"total_duration_us":10010033},{"operation_name":"upsert","last_dispatch_duration_us":90302,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":2162357,"total_server_duration_us":96799,"encode_duration_us":17932,"operation_id":"0xfc29","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10007904},{"operation_name":"upsert","last_dispatch_duration_us":31793,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7373885,"total_server_duration_us":280125,"encode_duration_us":17499,"operation_id":"0xfb51","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10007623},{"operation_name":"upsert","last_dispatch_duration_us":8328668,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9426084,"total_server_duration_us":249978,"encode_duration_us":23395,"operation_id":"0xf969","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10004614},{"operation_name":"upsert","last_dispatch_duration_us":8327440,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9457003,"total_server_duration_us":247627,"encode_duration_us":39042,"operation_id":"0xf973","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10001784},{"operation_name":"upsert","last_dispatch_duration_us":8512362,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9309268,"total_server_duration_us":382769,"encode_duration_us":19562,"operation_id":"0xf963","timeout_ms":10000,"last_server_duration_us":35815,"total_duration_us":10000616},{"operation_name":"upsert","last_dispatch_duration_us":9121324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9697283,"total_server_duration_us":9135568,"encode_duration_us":31044,"operation_id":"0xf96a","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":9779845},{"operation_name":"upsert","last_dispatch_duration_us":9205395,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9205395,"total_server_duration_us":9066042,"encode_duration_us":18294,"operation_id":"0xfcd8","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":9210457},{"operation_name":"get","last_dispatch_duration_us":8894707,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8894707,"total_server_duration_us":7803,"operation_id":"0xfa90","timeout_ms":30000,"last_server_duration_us":7803,"total_duration_us":8896835},{"operation_name":"upsert","last_dispatch_duration_us":2426001,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4594426,"total_server_duration_us":954085,"encode_duration_us":30514,"operation_id":"0xfc63","timeout_ms":10000,"last_server_duration_us":556058,"total_duration_us":4641199}],"total_count":30}} INFO [2022-03-13 17:17:28,596] jepsen worker 86 - jepsen.util 3686 :info :write [28 33] :RequestTimeoutException INFO [2022-03-13 17:17:28,597] jepsen worker 86 - jepsen.util 3776 :invoke :write [28 37] INFO [2022-03-13 17:17:28,616] jepsen worker 29 - jepsen.util 4259 :ok :read [9 26] INFO [2022-03-13 17:17:28,617] jepsen worker 29 - jepsen.util 4259 :invoke :write [9 3] INFO [2022-03-13 17:17:28,646] jepsen worker 67 - jepsen.util 3127 :info :write [22 17] :RequestTimeoutException INFO [2022-03-13 17:17:28,646] jepsen worker 67 - jepsen.util 3217 :invoke :read [22 nil] INFO [2022-03-13 17:17:28,671] jepsen worker 43 - jepsen.util 673 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,671] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,671] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,671] jepsen worker 13 - jepsen.util 733 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,672] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,673] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,673] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,673] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:28,674] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:17:28,675] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:28,675] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:28,676] jepsen worker 71 - jepsen.util 611 :invoke :write [23 4] INFO [2022-03-13 17:17:28,676] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:28,677] jepsen worker 43 - jepsen.util 673 :invoke :write [14 29] WARN [2022-03-13 17:17:28,682] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9041666,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13751827,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9309615,"encode_duration_us":22254,"operation_id":"0xf7aa","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10002178},{"operation_name":"upsert","last_dispatch_duration_us":9063738,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9348378,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9178624,"encode_duration_us":24104,"operation_id":"0xf85b","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10003930}],"total_count":2}} INFO [2022-03-13 17:17:28,686] jepsen worker 83 - jepsen.util 3503 :info :write [27 39] :RequestTimeoutException INFO [2022-03-13 17:17:28,687] jepsen worker 83 - jepsen.util 3593 :invoke :read [27 nil] INFO [2022-03-13 17:17:28,710] jepsen worker 44 - jepsen.util 674 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,711] jepsen worker 44 - jepsen.util 674 :invoke :write [14 42] INFO [2022-03-13 17:17:28,874] jepsen worker 7 - jepsen.util 637 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,875] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:28,876] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,876] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,876] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,877] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,877] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,877] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,877] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,877] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,877] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:28,878] jepsen worker 80 - jepsen.util 800 :invoke :write [26 12] INFO [2022-03-13 17:17:28,879] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:17:28,880] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:28,881] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:28,881] jepsen worker 34 - jepsen.util 934 :invoke :write [11 38] INFO [2022-03-13 17:17:28,882] jepsen worker 35 - jepsen.util 935 :invoke :write [11 48] INFO [2022-03-13 17:17:28,883] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:17:28,920] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:28,921] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:29,009] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7192982456140351}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5117647058823529}} INFO [2022-03-13 17:17:29,009] jepsen worker nemesis - couchbase.util Rebalance check count: 556 INFO [2022-03-13 17:17:29,009] jepsen worker nemesis - couchbase.util Rebalance stuck count: 443 INFO [2022-03-13 17:17:29,072] jepsen worker 24 - jepsen.util 654 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,072] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:29,276] jepsen worker 61 - jepsen.util 781 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,277] jepsen worker 61 - jepsen.util 781 :invoke :write [20 32] INFO [2022-03-13 17:17:29,292] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,292] jepsen worker 71 - jepsen.util 611 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,293] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:17:29,293] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:29,295] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,296] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:17:29,342] jepsen worker 42 - jepsen.util 402 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,343] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:29,414] jepsen worker 39 - jepsen.util 4089 :info :write [13 33] :RequestTimeoutException INFO [2022-03-13 17:17:29,414] jepsen worker 39 - jepsen.util 4179 :invoke :write [13 35] INFO [2022-03-13 17:17:29,484] jepsen worker 44 - jepsen.util 674 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,484] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,484] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,484] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,485] jepsen worker 44 - jepsen.util 674 :invoke :write [14 32] INFO [2022-03-13 17:17:29,485] jepsen worker 51 - jepsen.util 771 :invoke :write [17 13] INFO [2022-03-13 17:17:29,486] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:17:29,487] jepsen worker 33 - jepsen.util 753 :invoke :write [11 20] INFO [2022-03-13 17:17:29,490] jepsen worker 79 - jepsen.util 1159 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,491] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 19] INFO [2022-03-13 17:17:29,522] jepsen worker 80 - jepsen.util 800 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,523] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:29,692] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,692] jepsen worker 69 - jepsen.util 789 :invoke :write [23 6] INFO [2022-03-13 17:17:29,692] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,707] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:29,761] jepsen worker 8 - jepsen.util 728 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,761] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,762] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:29,762] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,762] jepsen worker 62 - jepsen.util 512 :invoke :write [20 9] INFO [2022-03-13 17:17:29,765] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:29,767] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,768] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:17:29,916] jepsen worker 23 - jepsen.util 563 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,916] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,916] jepsen worker 78 - jepsen.util 708 :invoke :write [26 30] INFO [2022-03-13 17:17:29,917] jepsen worker 23 - jepsen.util 563 :invoke :write [7 4] INFO [2022-03-13 17:17:29,947] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,947] jepsen worker 69 - jepsen.util 789 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,954] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:29,955] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:29,956] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,957] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,957] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:29,957] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:17:29,959] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,960] jepsen worker 69 - jepsen.util 789 :invoke :write [23 23] INFO [2022-03-13 17:17:29,964] jepsen worker 15 - jepsen.util 3795 :ok :read [5 43] INFO [2022-03-13 17:17:29,967] jepsen worker 15 - jepsen.util 3795 :invoke :write [5 32] INFO [2022-03-13 17:17:29,977] jepsen worker 34 - jepsen.util 934 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:29,977] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:29,982] jepsen worker 17 - jepsen.util 3977 :ok :read [5 43] INFO [2022-03-13 17:17:29,983] jepsen worker 17 - jepsen.util 3977 :invoke :write [5 39] INFO [2022-03-13 17:17:30,012] jepsen worker 57 - jepsen.util 3927 :ok :read [19 16] INFO [2022-03-13 17:17:30,012] jepsen worker 57 - jepsen.util 3927 :invoke :read [19 nil] INFO [2022-03-13 17:17:30,019] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7251461988304093}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5117647058823529}} INFO [2022-03-13 17:17:30,019] jepsen worker nemesis - couchbase.util Rebalance check count: 557 INFO [2022-03-13 17:17:30,019] jepsen worker nemesis - couchbase.util Rebalance stuck count: 443 INFO [2022-03-13 17:17:30,058] jepsen worker 0 - jepsen.util 3780 :ok :read [0 41] INFO [2022-03-13 17:17:30,058] jepsen worker 0 - jepsen.util 3780 :invoke :read [0 nil] INFO [2022-03-13 17:17:30,100] jepsen worker 6 - jepsen.util 636 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,100] jepsen worker 6 - jepsen.util 636 :invoke :write [2 8] INFO [2022-03-13 17:17:30,146] jepsen worker 35 - jepsen.util 935 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,146] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:17:30,319] jepsen worker 77 - jepsen.util 707 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,320] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:30,800] jepsen worker 2 - jepsen.util 4142 :info :write [0 15] :SyncWriteAmbiguous INFO [2022-03-13 17:17:30,801] jepsen worker 2 - jepsen.util 4232 :invoke :read [0 nil] INFO [2022-03-13 17:17:30,838] jepsen worker 41 - jepsen.util 3821 :info :write [13 1] :RequestTimeoutException INFO [2022-03-13 17:17:30,839] jepsen worker 41 - jepsen.util 3911 :invoke :write [13 41] INFO [2022-03-13 17:17:30,857] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,857] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:30,858] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,859] jepsen worker 25 - jepsen.util 565 :invoke :write [8 10] INFO [2022-03-13 17:17:30,861] jepsen worker 52 - jepsen.util 682 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,861] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,862] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:17:30,862] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,862] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,862] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,863] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:30,863] jepsen worker 13 - jepsen.util 733 :invoke :write [4 14] INFO [2022-03-13 17:17:30,865] jepsen worker 76 - jepsen.util 796 :invoke :write [25 47] INFO [2022-03-13 17:17:30,865] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:30,866] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,866] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,867] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,867] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:30,867] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:30,868] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:30,890] jepsen worker 59 - jepsen.util 3929 :info :write [19 31] :RequestTimeoutException INFO [2022-03-13 17:17:30,891] jepsen worker 59 - jepsen.util 4019 :invoke :read [19 nil] INFO [2022-03-13 17:17:30,951] jepsen worker 70 - jepsen.util 880 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,952] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:30,989] jepsen worker 69 - jepsen.util 789 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,989] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,990] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:17:30,990] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:30,995] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,995] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:30,996] jepsen worker 71 - jepsen.util 611 :invoke :write [23 5] INFO [2022-03-13 17:17:30,996] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:31,005] jepsen worker 33 - jepsen.util 753 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,005] jepsen worker 33 - jepsen.util 753 :invoke :write [11 25] INFO [2022-03-13 17:17:31,011] jepsen worker 30 - jepsen.util 3720 :info :write [10 24] :RequestTimeoutException INFO [2022-03-13 17:17:31,011] jepsen worker 30 - jepsen.util 3810 :invoke :read [10 nil] INFO [2022-03-13 17:17:31,020] jepsen worker 61 - jepsen.util 781 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,021] jepsen worker 61 - jepsen.util 781 :invoke :write [20 23] INFO [2022-03-13 17:17:31,025] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.736842105263158}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5235294117647059}} INFO [2022-03-13 17:17:31,025] jepsen worker nemesis - couchbase.util Rebalance check count: 558 INFO [2022-03-13 17:17:31,025] jepsen worker nemesis - couchbase.util Rebalance stuck count: 443 INFO [2022-03-13 17:17:31,092] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,092] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:31,099] jepsen worker 44 - jepsen.util 674 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,100] jepsen worker 44 - jepsen.util 674 :invoke :write [14 35] INFO [2022-03-13 17:17:31,156] jepsen worker 12 - jepsen.util 912 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,156] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:31,156] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,156] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,157] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:31,157] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:31,158] jepsen worker 8 - jepsen.util 728 :invoke :write [2 4] INFO [2022-03-13 17:17:31,159] jepsen worker 42 - jepsen.util 402 :invoke :write [14 26] INFO [2022-03-13 17:17:31,173] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,173] jepsen worker 71 - jepsen.util 611 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,174] jepsen worker 71 - jepsen.util 611 :invoke :write [23 43] INFO [2022-03-13 17:17:31,174] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:31,175] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,178] jepsen worker 69 - jepsen.util 789 :invoke :write [23 9] INFO [2022-03-13 17:17:31,180] jepsen worker 51 - jepsen.util 771 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,180] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,180] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,180] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,181] jepsen worker 51 - jepsen.util 771 :invoke :write [17 13] INFO [2022-03-13 17:17:31,182] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:31,182] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:31,183] jepsen worker 24 - jepsen.util 654 :invoke :write [8 23] INFO [2022-03-13 17:17:31,273] jepsen worker 79 - jepsen.util 1159 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,274] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:31,291] jepsen worker 71 - jepsen.util 611 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,292] jepsen worker 71 - jepsen.util 611 :invoke :write [23 4] INFO [2022-03-13 17:17:31,296] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,296] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,297] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:31,297] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:31,476] jepsen worker 69 - jepsen.util 789 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,476] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:17:31,656] jepsen worker 32 - jepsen.util 4082 :info :write [10 19] :RequestTimeoutException INFO [2022-03-13 17:17:31,657] jepsen worker 32 - jepsen.util 4172 :invoke :write [10 14] INFO [2022-03-13 17:17:31,681] jepsen worker 71 - jepsen.util 611 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,682] jepsen worker 71 - jepsen.util 611 :invoke :write [23 6] INFO [2022-03-13 17:17:31,685] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,686] jepsen worker 21 - jepsen.util 831 :invoke :write [7 47] INFO [2022-03-13 17:17:31,708] jepsen worker 62 - jepsen.util 512 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,709] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:17:31,757] jepsen worker 1 - jepsen.util 3421 :info :write [0 19] :RequestTimeoutException INFO [2022-03-13 17:17:31,757] jepsen worker 1 - jepsen.util 3511 :invoke :read [0 nil] INFO [2022-03-13 17:17:31,762] jepsen worker 25 - jepsen.util 565 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,763] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:31,785] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,786] jepsen worker 69 - jepsen.util 789 :invoke :write [23 8] INFO [2022-03-13 17:17:31,788] jepsen worker 78 - jepsen.util 708 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,789] jepsen worker 78 - jepsen.util 708 :invoke :write [26 25] INFO [2022-03-13 17:17:31,808] jepsen worker 16 - jepsen.util 3706 :info :write [5 10] :RequestTimeoutException INFO [2022-03-13 17:17:31,815] jepsen worker 16 - jepsen.util 3796 :invoke :write [5 44] INFO [2022-03-13 17:17:31,894] jepsen worker 31 - jepsen.util 3991 :info :write [10 1] :RequestTimeoutException INFO [2022-03-13 17:17:31,895] jepsen worker 31 - jepsen.util 4081 :invoke :write [10 39] INFO [2022-03-13 17:17:31,913] jepsen worker 71 - jepsen.util 611 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,913] jepsen worker 71 - jepsen.util 611 :invoke :write [23 43] INFO [2022-03-13 17:17:31,950] jepsen worker 69 - jepsen.util 789 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,950] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:17:31,973] jepsen worker 33 - jepsen.util 753 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,974] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:31,981] jepsen worker 71 - jepsen.util 611 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:31,982] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:17:32,030] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.736842105263158}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5235294117647059}} INFO [2022-03-13 17:17:32,030] jepsen worker nemesis - couchbase.util Rebalance check count: 559 INFO [2022-03-13 17:17:32,030] jepsen worker nemesis - couchbase.util Rebalance stuck count: 444 INFO [2022-03-13 17:17:32,104] jepsen worker 20 - jepsen.util 4070 :info :write [6 22] :RequestTimeoutException INFO [2022-03-13 17:17:32,104] jepsen worker 56 - jepsen.util 3926 :info :write [18 13] :RequestTimeoutException INFO [2022-03-13 17:17:32,105] jepsen worker 56 - jepsen.util 4016 :invoke :read [18 nil] INFO [2022-03-13 17:17:32,105] jepsen worker 20 - jepsen.util 4160 :invoke :read [6 nil] INFO [2022-03-13 17:17:32,113] jepsen worker 54 - jepsen.util 3474 :info :write [18 11] :RequestTimeoutException INFO [2022-03-13 17:17:32,114] jepsen worker 54 - jepsen.util 3564 :invoke :read [18 nil] INFO [2022-03-13 17:17:32,135] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,135] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:17:32,728] jepsen worker 6 - jepsen.util 636 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,728] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,728] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,728] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,729] jepsen worker 70 - jepsen.util 880 :invoke :write [23 15] INFO [2022-03-13 17:17:32,729] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:32,730] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:32,732] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,733] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] INFO [2022-03-13 17:17:32,734] jepsen worker 7 - jepsen.util 637 :invoke :write [2 3] INFO [2022-03-13 17:17:32,744] jepsen worker 44 - jepsen.util 674 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,744] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,744] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,745] jepsen worker 44 - jepsen.util 674 :invoke :write [14 31] INFO [2022-03-13 17:17:32,746] jepsen worker 34 - jepsen.util 934 :invoke :write [11 14] INFO [2022-03-13 17:17:32,751] jepsen worker 80 - jepsen.util 800 :invoke :write [26 48] INFO [2022-03-13 17:17:32,801] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,802] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:17:32,805] jepsen worker 23 - jepsen.util 563 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,805] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,805] jepsen worker 23 - jepsen.util 563 :invoke :write [7 22] INFO [2022-03-13 17:17:32,806] jepsen worker 77 - jepsen.util 707 :invoke :write [25 20] INFO [2022-03-13 17:17:32,850] jepsen worker 43 - jepsen.util 673 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,851] jepsen worker 43 - jepsen.util 673 :invoke :write [14 34] INFO [2022-03-13 17:17:32,866] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,867] jepsen worker 71 - jepsen.util 611 :invoke :write [23 14] INFO [2022-03-13 17:17:32,872] jepsen worker 51 - jepsen.util 771 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,873] jepsen worker 51 - jepsen.util 771 :invoke :write [17 15] INFO [2022-03-13 17:17:32,896] jepsen worker 8 - jepsen.util 728 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,896] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:32,951] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,952] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:17:32,970] jepsen worker 61 - jepsen.util 781 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,971] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,971] jepsen worker 61 - jepsen.util 781 :invoke :write [20 20] INFO [2022-03-13 17:17:32,977] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 45] INFO [2022-03-13 17:17:32,992] jepsen worker 24 - jepsen.util 654 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,992] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:32,993] jepsen worker 24 - jepsen.util 654 :invoke :write [8 9] INFO [2022-03-13 17:17:32,993] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:17:33,035] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.736842105263158}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5235294117647059}} INFO [2022-03-13 17:17:33,035] jepsen worker nemesis - couchbase.util Rebalance check count: 560 INFO [2022-03-13 17:17:33,035] jepsen worker nemesis - couchbase.util Rebalance stuck count: 445 INFO [2022-03-13 17:17:33,155] jepsen worker 71 - jepsen.util 611 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,156] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:33,158] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,158] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,159] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,158] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,159] jepsen worker 53 - jepsen.util 593 :invoke :write [17 34] INFO [2022-03-13 17:17:33,160] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:33,160] jepsen worker 60 - jepsen.util 690 :invoke :write [20 17] INFO [2022-03-13 17:17:33,161] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:17:33,206] jepsen worker 13 - jepsen.util 733 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,207] jepsen worker 13 - jepsen.util 733 :invoke :write [4 20] INFO [2022-03-13 17:17:33,214] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,214] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,215] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:33,215] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:17:33,220] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,221] jepsen worker 69 - jepsen.util 789 :invoke :write [23 49] INFO [2022-03-13 17:17:33,226] jepsen worker 42 - jepsen.util 402 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,227] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:17:33,230] jepsen worker 21 - jepsen.util 831 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,231] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,231] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,231] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:33,231] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,232] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:33,232] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:33,233] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:33,363] jepsen worker 78 - jepsen.util 708 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,364] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:33,381] jepsen worker 34 - jepsen.util 934 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,382] jepsen worker 34 - jepsen.util 934 :invoke :write [11 11] INFO [2022-03-13 17:17:33,424] jepsen worker 44 - jepsen.util 674 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,424] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:33,564] jepsen worker 40 - jepsen.util 4000 :info :write [13 48] :RequestTimeoutException INFO [2022-03-13 17:17:33,564] jepsen worker 40 - jepsen.util 4090 :invoke :write [13 5] INFO [2022-03-13 17:17:33,571] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,571] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:17:33,585] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,585] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,585] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,586] jepsen worker 80 - jepsen.util 800 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,586] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:33,587] jepsen worker 6 - jepsen.util 636 :invoke :write [2 21] INFO [2022-03-13 17:17:33,589] jepsen worker 80 - jepsen.util 800 :invoke :write [26 43] INFO [2022-03-13 17:17:33,589] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:33,661] jepsen worker 51 - jepsen.util 771 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,661] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:33,665] jepsen worker 23 - jepsen.util 563 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,665] jepsen worker 23 - jepsen.util 563 :invoke :write [7 26] INFO [2022-03-13 17:17:33,786] jepsen worker 69 - jepsen.util 789 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,786] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:33,825] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,825] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] INFO [2022-03-13 17:17:33,834] jepsen worker 70 - jepsen.util 880 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,835] jepsen worker 70 - jepsen.util 880 :invoke :write [23 30] INFO [2022-03-13 17:17:33,855] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,859] jepsen worker 52 - jepsen.util 682 :invoke :write [17 30] INFO [2022-03-13 17:17:33,980] jepsen worker 76 - jepsen.util 796 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,980] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:33,981] jepsen worker 76 - jepsen.util 796 :invoke :write [25 3] INFO [2022-03-13 17:17:33,982] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:34,021] jepsen worker 77 - jepsen.util 707 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,022] jepsen worker 77 - jepsen.util 707 :invoke :write [25 48] INFO [2022-03-13 17:17:34,028] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,028] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,029] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:17:34,031] jepsen worker 69 - jepsen.util 789 :invoke :write [23 6] INFO [2022-03-13 17:17:34,032] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:34,032] jepsen worker 24 - jepsen.util 654 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,034] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:34,035] jepsen worker 44 - jepsen.util 674 :invoke :write [14 39] INFO [2022-03-13 17:17:34,041] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7485380116959064}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5352941176470588}} INFO [2022-03-13 17:17:34,041] jepsen worker nemesis - couchbase.util Rebalance check count: 561 INFO [2022-03-13 17:17:34,042] jepsen worker nemesis - couchbase.util Rebalance stuck count: 445 INFO [2022-03-13 17:17:34,095] jepsen worker 7 - jepsen.util 637 :fail :write [2 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,095] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,096] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:34,096] jepsen worker 7 - jepsen.util 637 :invoke :write [2 4] INFO [2022-03-13 17:17:34,137] jepsen worker 61 - jepsen.util 781 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,137] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:34,151] jepsen worker 69 - jepsen.util 789 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,152] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:17:34,230] jepsen worker 34 - jepsen.util 934 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,231] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,231] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,231] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,231] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,231] jepsen worker 34 - jepsen.util 934 :invoke :write [11 20] INFO [2022-03-13 17:17:34,232] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:34,232] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:17:34,233] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:17:34,233] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:34,267] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,267] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,268] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,268] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:17:34,268] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:34,269] jepsen worker 62 - jepsen.util 512 :invoke :write [20 38] INFO [2022-03-13 17:17:34,432] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,433] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:17:34,464] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,465] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:34,498] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,498] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,498] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:34,499] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:34,499] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,500] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:34,500] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,501] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,501] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:34,501] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:17:34,502] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,503] jepsen worker 71 - jepsen.util 611 :invoke :write [23 32] INFO [2022-03-13 17:17:34,507] jepsen worker 79 - jepsen.util 1159 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,508] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,508] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 30] INFO [2022-03-13 17:17:34,509] jepsen worker 78 - jepsen.util 708 :invoke :write [26 16] INFO [2022-03-13 17:17:34,523] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,523] jepsen worker 80 - jepsen.util 800 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,524] jepsen worker 80 - jepsen.util 800 :invoke :write [26 31] INFO [2022-03-13 17:17:34,524] jepsen worker 51 - jepsen.util 771 :invoke :write [17 12] INFO [2022-03-13 17:17:34,563] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,563] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,563] jepsen worker 25 - jepsen.util 565 :invoke :write [8 30] INFO [2022-03-13 17:17:34,564] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:34,661] jepsen worker 88 - jepsen.util 2248 :info :write [29 47] :RequestTimeoutException INFO [2022-03-13 17:17:34,661] jepsen worker 89 - jepsen.util 2699 :info :write [29 23] :RequestTimeoutException INFO [2022-03-13 17:17:34,661] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:34,662] jepsen worker 88 - jepsen.util 2338 :invoke :write [29 27] INFO [2022-03-13 17:17:34,666] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,666] jepsen worker 69 - jepsen.util 789 :invoke :write [23 2] INFO [2022-03-13 17:17:34,690] jepsen worker 36 - jepsen.util 2376 :info :write [12 26] :RequestTimeoutException INFO [2022-03-13 17:17:34,700] jepsen worker 47 - jepsen.util 3737 :info :write [15 46] :SyncWriteAmbiguous INFO [2022-03-13 17:17:34,771] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:17:34,771] jepsen worker 47 - jepsen.util 3827 :invoke :write [15 41] INFO [2022-03-13 17:17:34,797] jepsen worker 38 - jepsen.util 2648 :info :write [12 25] :RequestTimeoutException INFO [2022-03-13 17:17:34,798] jepsen worker 38 - jepsen.util 2738 :invoke :write [12 39] INFO [2022-03-13 17:17:34,826] jepsen worker 9 - jepsen.util 3429 :ok :read [3 25] INFO [2022-03-13 17:17:34,827] jepsen worker 9 - jepsen.util 3429 :invoke :write [3 7] INFO [2022-03-13 17:17:34,832] jepsen worker 65 - jepsen.util 4295 :ok :read [21 :nil] INFO [2022-03-13 17:17:34,833] jepsen worker 65 - jepsen.util 4295 :invoke :write [21 9] INFO [2022-03-13 17:17:34,860] jepsen worker 6 - jepsen.util 636 :fail :write [2 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,860] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:34,874] jepsen worker 71 - jepsen.util 611 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,875] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:17:34,883] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,884] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,884] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:17:34,884] jepsen worker 44 - jepsen.util 674 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,885] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:34,885] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:34,915] jepsen worker 34 - jepsen.util 934 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:34,916] jepsen worker 34 - jepsen.util 934 :invoke :write [11 4] INFO [2022-03-13 17:17:34,920] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:34,920] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:34,960] jepsen worker 27 - jepsen.util 3987 :info :write [9 6] :RequestTimeoutException INFO [2022-03-13 17:17:34,961] jepsen worker 27 - jepsen.util 4077 :invoke :read [9 nil] INFO [2022-03-13 17:17:34,976] jepsen worker 85 - jepsen.util 3595 :info :write [28 0] :RequestTimeoutException INFO [2022-03-13 17:17:34,977] jepsen worker 85 - jepsen.util 3685 :invoke :read [28 nil] INFO [2022-03-13 17:17:35,047] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7485380116959064}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5352941176470588}} INFO [2022-03-13 17:17:35,047] jepsen worker nemesis - couchbase.util Rebalance check count: 562 INFO [2022-03-13 17:17:35,047] jepsen worker nemesis - couchbase.util Rebalance stuck count: 446 INFO [2022-03-13 17:17:35,079] jepsen worker 23 - jepsen.util 563 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,080] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:35,088] jepsen worker 69 - jepsen.util 789 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,088] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,127] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,127] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:35,154] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,154] jepsen worker 70 - jepsen.util 880 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,155] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:35,156] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,278] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,279] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:17:35,279] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,280] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,285] jepsen worker 80 - jepsen.util 800 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,286] jepsen worker 80 - jepsen.util 800 :invoke :write [26 39] INFO [2022-03-13 17:17:35,290] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,291] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:17:35,328] jepsen worker 28 - jepsen.util 3898 :info :write [9 3] :RequestTimeoutException INFO [2022-03-13 17:17:35,330] jepsen worker 28 - jepsen.util 3988 :invoke :read [9 nil] INFO [2022-03-13 17:17:35,332] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,332] jepsen worker 77 - jepsen.util 707 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,333] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:35,333] jepsen worker 61 - jepsen.util 781 :invoke :write [20 29] INFO [2022-03-13 17:17:35,341] jepsen worker 18 - jepsen.util 3978 :ok :read [6 16] INFO [2022-03-13 17:17:35,342] jepsen worker 18 - jepsen.util 3978 :invoke :write [6 13] INFO [2022-03-13 17:17:35,365] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,366] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,377] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,377] jepsen worker 51 - jepsen.util 771 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,378] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:35,379] jepsen worker 51 - jepsen.util 771 :invoke :write [17 12] INFO [2022-03-13 17:17:35,394] jepsen worker 3 - jepsen.util 2163 :info :write [1 22] :RequestTimeoutException INFO [2022-03-13 17:17:35,395] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:35,396] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,397] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:35,398] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,399] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:35,404] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,405] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:17:35,456] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:35,457] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:35,474] jepsen worker 27 - jepsen.util 4077 :ok :read [9 26] INFO [2022-03-13 17:17:35,474] jepsen worker 27 - jepsen.util 4077 :invoke :write [9 24] INFO [2022-03-13 17:17:35,496] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,496] jepsen worker 7 - jepsen.util 637 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,496] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:35,497] jepsen worker 12 - jepsen.util 912 :invoke :write [4 0] INFO [2022-03-13 17:17:35,512] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,512] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,513] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,513] jepsen worker 71 - jepsen.util 611 :invoke :write [23 6] INFO [2022-03-13 17:17:35,514] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,515] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:17:35,576] jepsen worker 63 - jepsen.util 4023 :info :write [21 26] :RequestTimeoutException INFO [2022-03-13 17:17:35,577] jepsen worker 63 - jepsen.util 4113 :invoke :read [21 nil] INFO [2022-03-13 17:17:35,634] jepsen worker 4 - jepsen.util 2434 :info :write [1 11] :SyncWriteAmbiguous INFO [2022-03-13 17:17:35,635] jepsen worker 4 - jepsen.util 2524 :invoke :read [1 nil] INFO [2022-03-13 17:17:35,639] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:35,640] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:35,659] jepsen worker 37 - jepsen.util 2287 :info :write [12 43] :RequestTimeoutException INFO [2022-03-13 17:17:35,660] jepsen worker 37 - jepsen.util 2377 :invoke :write [12 41] INFO [2022-03-13 17:17:35,692] jepsen worker 78 - jepsen.util 708 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,693] jepsen worker 78 - jepsen.util 708 :invoke :write [26 37] INFO [2022-03-13 17:17:35,709] jepsen worker 25 - jepsen.util 565 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,709] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,709] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:35,710] jepsen worker 44 - jepsen.util 674 :invoke :write [14 0] INFO [2022-03-13 17:17:35,711] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:35,711] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:35,732] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,733] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,760] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,761] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:35,762] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,763] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:17:35,801] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,801] jepsen worker 71 - jepsen.util 611 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,802] jepsen worker 69 - jepsen.util 789 :invoke :write [23 42] INFO [2022-03-13 17:17:35,803] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,805] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,805] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:35,806] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,807] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:17:35,822] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,823] jepsen worker 42 - jepsen.util 402 :invoke :write [14 15] INFO [2022-03-13 17:17:35,853] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:35,854] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:35,871] jepsen worker 4 - jepsen.util 2524 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,871] jepsen worker 4 - jepsen.util 2524 :invoke :read [1 nil] INFO [2022-03-13 17:17:35,873] jepsen worker 4 - jepsen.util 2524 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,873] jepsen worker 4 - jepsen.util 2524 :invoke :write [1 27] INFO [2022-03-13 17:17:35,920] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,921] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:17:35,956] jepsen worker 66 - jepsen.util 3666 :info :write [22 49] :SyncWriteAmbiguous INFO [2022-03-13 17:17:35,956] jepsen worker 79 - jepsen.util 1159 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,957] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,957] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,957] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:35,958] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:35,959] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:35,959] jepsen worker 66 - jepsen.util 3756 :invoke :read [22 nil] INFO [2022-03-13 17:17:35,977] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:35,978] jepsen worker 45 - jepsen.util 4095 :invoke :read [15 nil] INFO [2022-03-13 17:17:35,978] jepsen worker 69 - jepsen.util 789 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:35,979] jepsen worker 69 - jepsen.util 789 :invoke :write [23 14] INFO [2022-03-13 17:17:35,985] jepsen worker 2 - jepsen.util 4232 :ok :read [0 41] INFO [2022-03-13 17:17:35,985] jepsen worker 2 - jepsen.util 4232 :invoke :read [0 nil] INFO [2022-03-13 17:17:35,986] jepsen worker 87 - jepsen.util 2787 :info :write [29 16] :RequestTimeoutException INFO [2022-03-13 17:17:35,987] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:17:36,008] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,008] jepsen worker 43 - jepsen.util 673 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,009] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:36,010] jepsen worker 43 - jepsen.util 673 :invoke :write [14 12] INFO [2022-03-13 17:17:36,043] jepsen worker 85 - jepsen.util 3685 :ok :read [28 44] INFO [2022-03-13 17:17:36,043] jepsen worker 85 - jepsen.util 3685 :invoke :read [28 nil] INFO [2022-03-13 17:17:36,048] jepsen worker 66 - jepsen.util 3756 :ok :read [22 30] INFO [2022-03-13 17:17:36,049] jepsen worker 66 - jepsen.util 3756 :invoke :write [22 9] INFO [2022-03-13 17:17:36,050] jepsen worker 11 - jepsen.util 3791 :info :write [3 7] :RequestTimeoutException INFO [2022-03-13 17:17:36,051] jepsen worker 11 - jepsen.util 3881 :invoke :write [3 43] INFO [2022-03-13 17:17:36,053] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7485380116959064}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5352941176470588}} INFO [2022-03-13 17:17:36,053] jepsen worker nemesis - couchbase.util Rebalance check count: 563 INFO [2022-03-13 17:17:36,053] jepsen worker nemesis - couchbase.util Rebalance stuck count: 447 INFO [2022-03-13 17:17:36,078] jepsen worker 45 - jepsen.util 4095 :ok :read [15 48] INFO [2022-03-13 17:17:36,079] jepsen worker 45 - jepsen.util 4095 :invoke :write [15 49] INFO [2022-03-13 17:17:36,083] jepsen worker 2 - jepsen.util 4232 :ok :read [0 41] INFO [2022-03-13 17:17:36,084] jepsen worker 2 - jepsen.util 4232 :invoke :read [0 nil] INFO [2022-03-13 17:17:36,106] jepsen worker 85 - jepsen.util 3685 :ok :read [28 44] INFO [2022-03-13 17:17:36,107] jepsen worker 85 - jepsen.util 3685 :invoke :read [28 nil] INFO [2022-03-13 17:17:36,128] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,128] jepsen worker 34 - jepsen.util 934 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,128] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:36,129] jepsen worker 34 - jepsen.util 934 :invoke :write [11 32] INFO [2022-03-13 17:17:36,135] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,135] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:17:36,139] jepsen worker 62 - jepsen.util 512 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,140] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,140] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,140] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,140] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,141] jepsen worker 8 - jepsen.util 728 :invoke :write [2 42] INFO [2022-03-13 17:17:36,142] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:36,143] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:36,143] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:36,144] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:36,179] jepsen worker 2 - jepsen.util 4232 :ok :read [0 41] INFO [2022-03-13 17:17:36,179] jepsen worker 2 - jepsen.util 4232 :invoke :write [0 18] INFO [2022-03-13 17:17:36,205] jepsen worker 61 - jepsen.util 781 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,206] jepsen worker 61 - jepsen.util 781 :invoke :write [20 32] INFO [2022-03-13 17:17:36,207] jepsen worker 85 - jepsen.util 3685 :ok :read [28 44] INFO [2022-03-13 17:17:36,207] jepsen worker 85 - jepsen.util 3685 :invoke :write [28 1] INFO [2022-03-13 17:17:36,267] jepsen worker 5 - jepsen.util 2615 :info :write [1 5] :RequestTimeoutException INFO [2022-03-13 17:17:36,268] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:36,343] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,343] jepsen worker 13 - jepsen.util 733 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,344] jepsen worker 13 - jepsen.util 733 :invoke :write [4 10] INFO [2022-03-13 17:17:36,344] jepsen worker 75 - jepsen.util 525 :invoke :write [25 17] INFO [2022-03-13 17:17:36,348] jepsen worker 69 - jepsen.util 789 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,348] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:36,356] jepsen worker 80 - jepsen.util 800 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,357] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:36,390] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,390] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,391] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:36,391] jepsen worker 24 - jepsen.util 654 :invoke :write [8 13] INFO [2022-03-13 17:17:36,450] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,450] jepsen worker 51 - jepsen.util 771 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,450] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,451] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:36,452] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:17:36,453] jepsen worker 51 - jepsen.util 771 :invoke :write [17 34] INFO [2022-03-13 17:17:36,455] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,456] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,456] jepsen worker 71 - jepsen.util 611 :invoke :write [23 32] INFO [2022-03-13 17:17:36,457] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:36,458] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,459] jepsen worker 69 - jepsen.util 789 :invoke :write [23 48] INFO [2022-03-13 17:17:36,461] jepsen worker 12 - jepsen.util 912 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,462] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:36,470] jepsen worker 19 - jepsen.util 3709 :info :write [6 12] :RequestTimeoutException INFO [2022-03-13 17:17:36,471] jepsen worker 19 - jepsen.util 3799 :invoke :read [6 nil] INFO [2022-03-13 17:17:36,517] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,517] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,517] jepsen worker 78 - jepsen.util 708 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,518] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,518] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:36,519] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:36,519] jepsen worker 78 - jepsen.util 708 :invoke :write [26 33] INFO [2022-03-13 17:17:36,520] jepsen worker 23 - jepsen.util 563 :invoke :write [7 18] INFO [2022-03-13 17:17:36,556] jepsen worker 44 - jepsen.util 674 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,556] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:36,568] jepsen worker 69 - jepsen.util 789 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:36,568] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:17:36,827] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:17:36,827] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] WARN [2022-03-13 17:17:36,875] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12438697,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12438697,"total_server_duration_us":27,"operation_id":"0xfc30","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":12440925},{"operation_name":"get","last_dispatch_duration_us":10876582,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10876582,"total_server_duration_us":11,"operation_id":"0xfcce","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":10886002},{"operation_name":"upsert","last_dispatch_duration_us":39323,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1792305,"total_server_duration_us":338251,"encode_duration_us":18078,"operation_id":"0xff67","timeout_ms":10000,"last_server_duration_us":22971,"total_duration_us":10010031},{"operation_name":"upsert","last_dispatch_duration_us":8912700,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":10957903,"total_server_duration_us":402927,"encode_duration_us":22718,"operation_id":"0xfeea","timeout_ms":10000,"last_server_duration_us":62284,"total_duration_us":10007519},{"operation_name":"upsert","last_dispatch_duration_us":9619407,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":10943036,"total_server_duration_us":9225958,"encode_duration_us":32031,"operation_id":"0xfeeb","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10006636},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17956,"operation_id":"0xfd7a","timeout_ms":10000,"total_duration_us":10006446},{"operation_name":"get","last_dispatch_duration_us":9473192,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9473192,"total_server_duration_us":15,"operation_id":"0xfd43","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":9474561},{"operation_name":"get","last_dispatch_duration_us":8262921,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8262921,"total_server_duration_us":27,"operation_id":"0xfddc","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":8265893},{"operation_name":"upsert","last_dispatch_duration_us":447276,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":447276,"total_server_duration_us":163566,"encode_duration_us":17250,"operation_id":"0xff75","timeout_ms":10000,"last_server_duration_us":163566,"total_duration_us":2557710},{"operation_name":"upsert","last_dispatch_duration_us":2251947,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2251947,"total_server_duration_us":523089,"encode_duration_us":25548,"operation_id":"0xff0f","timeout_ms":10000,"last_server_duration_us":523089,"total_duration_us":2257262}],"total_count":96}} WARN [2022-03-13 17:17:36,876] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9044332,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9986444,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9291352,"encode_duration_us":26284,"operation_id":"0xfbc3","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10004158},{"operation_name":"upsert","last_dispatch_duration_us":9089608,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10188529,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9699908,"encode_duration_us":19714,"operation_id":"0xf947","timeout_ms":10000,"last_server_duration_us":9077736,"total_duration_us":10005181},{"operation_name":"upsert","last_dispatch_duration_us":9042382,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10013183,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9529408,"encode_duration_us":21289,"operation_id":"0xfa27","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10004239},{"operation_name":"upsert","last_dispatch_duration_us":16941558,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17507258,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9284102,"encode_duration_us":18129,"operation_id":"0xfa67","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10010124},{"operation_name":"upsert","last_dispatch_duration_us":9619407,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":10943036,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":9225958,"encode_duration_us":32031,"operation_id":"0xfeeb","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10006636}],"total_count":5}} INFO [2022-03-13 17:17:37,008] jepsen worker 5 - jepsen.util 2705 :ok :read [1 42] INFO [2022-03-13 17:17:37,009] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:17:37,038] jepsen worker 72 - jepsen.util 3942 :info :write [24 30] :RequestTimeoutException INFO [2022-03-13 17:17:37,039] jepsen worker 72 - jepsen.util 4032 :invoke :read [24 nil] INFO [2022-03-13 17:17:37,059] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7485380116959064}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5352941176470588}} INFO [2022-03-13 17:17:37,059] jepsen worker nemesis - couchbase.util Rebalance check count: 564 INFO [2022-03-13 17:17:37,059] jepsen worker nemesis - couchbase.util Rebalance stuck count: 448 INFO [2022-03-13 17:17:37,067] jepsen worker 72 - jepsen.util 4032 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,068] jepsen worker 72 - jepsen.util 4032 :invoke :read [24 nil] INFO [2022-03-13 17:17:37,069] jepsen worker 72 - jepsen.util 4032 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,070] jepsen worker 42 - jepsen.util 402 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,070] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,070] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,070] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,070] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,071] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:17:37,072] jepsen worker 62 - jepsen.util 512 :invoke :write [20 44] INFO [2022-03-13 17:17:37,072] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:37,073] jepsen worker 7 - jepsen.util 637 :invoke :write [2 43] INFO [2022-03-13 17:17:37,078] jepsen worker 72 - jepsen.util 4032 :invoke :read [24 nil] INFO [2022-03-13 17:17:37,080] jepsen worker 72 - jepsen.util 4032 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,080] jepsen worker 72 - jepsen.util 4032 :invoke :read [24 nil] INFO [2022-03-13 17:17:37,081] jepsen worker 72 - jepsen.util 4032 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,082] jepsen worker 72 - jepsen.util 4032 :invoke :write [24 20] INFO [2022-03-13 17:17:37,086] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:37,101] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,102] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:17:37,133] jepsen worker 64 - jepsen.util 3934 :info :write [21 31] :RequestTimeoutException INFO [2022-03-13 17:17:37,134] jepsen worker 64 - jepsen.util 4024 :invoke :write [21 29] INFO [2022-03-13 17:17:37,162] jepsen worker 61 - jepsen.util 781 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,164] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:37,165] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,165] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,166] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:17:37,166] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:37,173] jepsen worker 71 - jepsen.util 611 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,173] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:37,178] jepsen worker 34 - jepsen.util 934 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,178] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,178] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,178] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,179] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:37,179] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:37,180] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:17:37,180] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:37,193] jepsen worker 73 - jepsen.util 4033 :info :write [24 33] :RequestTimeoutException INFO [2022-03-13 17:17:37,194] jepsen worker 73 - jepsen.util 4123 :invoke :write [24 32] INFO [2022-03-13 17:17:37,244] jepsen worker 24 - jepsen.util 654 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,244] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,245] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:37,245] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:37,260] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,260] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,261] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:37,261] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:37,263] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,263] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,263] jepsen worker 10 - jepsen.util 3970 :info :write [3 5] :RequestTimeoutException INFO [2022-03-13 17:17:37,264] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:17:37,265] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:17:37,265] jepsen worker 10 - jepsen.util 4060 :invoke :write [3 14] INFO [2022-03-13 17:17:37,276] jepsen worker 8 - jepsen.util 728 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,276] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,277] jepsen worker 8 - jepsen.util 728 :invoke :write [2 20] INFO [2022-03-13 17:17:37,277] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:37,278] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,278] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,278] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:37,279] jepsen worker 6 - jepsen.util 636 :invoke :write [2 6] INFO [2022-03-13 17:17:37,337] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,338] jepsen worker 71 - jepsen.util 611 :invoke :write [23 11] INFO [2022-03-13 17:17:37,344] jepsen worker 46 - jepsen.util 4006 :info :write [15 49] :RequestTimeoutException INFO [2022-03-13 17:17:37,345] jepsen worker 46 - jepsen.util 4096 :invoke :read [15 nil] INFO [2022-03-13 17:17:37,348] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,349] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:17:37,376] jepsen worker 23 - jepsen.util 563 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,377] jepsen worker 23 - jepsen.util 563 :invoke :write [7 36] INFO [2022-03-13 17:17:37,506] jepsen worker 54 - jepsen.util 3564 :ok :read [18 21] INFO [2022-03-13 17:17:37,507] jepsen worker 54 - jepsen.util 3564 :invoke :write [18 9] INFO [2022-03-13 17:17:37,518] jepsen worker 19 - jepsen.util 3799 :ok :read [6 16] INFO [2022-03-13 17:17:37,519] jepsen worker 19 - jepsen.util 3799 :invoke :read [6 nil] INFO [2022-03-13 17:17:37,535] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,536] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:37,560] jepsen worker 60 - jepsen.util 690 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,561] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:37,567] jepsen worker 51 - jepsen.util 771 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,568] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:37,568] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:37,569] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:37,582] jepsen worker 19 - jepsen.util 3799 :ok :read [6 16] INFO [2022-03-13 17:17:37,583] jepsen worker 19 - jepsen.util 3799 :invoke :write [6 23] INFO [2022-03-13 17:17:37,621] jepsen worker 7 - jepsen.util 637 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,621] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:37,622] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,623] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:17:37,624] jepsen worker 71 - jepsen.util 611 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,624] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:17:37,752] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,752] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,752] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,752] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,753] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:37,754] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:17:37,755] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:37,755] jepsen worker 76 - jepsen.util 796 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,755] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:17:37,756] jepsen worker 76 - jepsen.util 796 :invoke :write [25 49] INFO [2022-03-13 17:17:37,767] jepsen worker 78 - jepsen.util 708 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,768] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,768] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,768] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:37,768] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 45] INFO [2022-03-13 17:17:37,769] jepsen worker 78 - jepsen.util 708 :invoke :write [26 35] INFO [2022-03-13 17:17:37,821] jepsen worker 58 - jepsen.util 3568 :info :write [19 6] :RequestTimeoutException INFO [2022-03-13 17:17:37,821] jepsen worker 58 - jepsen.util 3658 :invoke :read [19 nil] INFO [2022-03-13 17:17:37,840] jepsen worker 62 - jepsen.util 512 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,841] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:37,868] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,869] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:37,907] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,907] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,907] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,908] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:37,909] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:17:37,909] jepsen worker 24 - jepsen.util 654 :invoke :write [8 36] INFO [2022-03-13 17:17:37,935] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,936] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:37,961] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,961] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:37,962] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:17:37,963] jepsen worker 71 - jepsen.util 611 :invoke :write [23 14] INFO [2022-03-13 17:17:38,016] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,016] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,016] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,017] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,017] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:38,017] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,017] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,018] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,018] jepsen worker 42 - jepsen.util 402 :invoke :write [14 42] INFO [2022-03-13 17:17:38,019] jepsen worker 34 - jepsen.util 934 :invoke :write [11 49] INFO [2022-03-13 17:17:38,021] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:38,022] jepsen worker 12 - jepsen.util 912 :invoke :write [4 37] INFO [2022-03-13 17:17:38,022] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:17:38,024] jepsen worker 80 - jepsen.util 800 :invoke :write [26 42] INFO [2022-03-13 17:17:38,025] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:38,025] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:38,064] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.760233918128655}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5470588235294118}} INFO [2022-03-13 17:17:38,064] jepsen worker nemesis - couchbase.util Rebalance check count: 565 INFO [2022-03-13 17:17:38,064] jepsen worker nemesis - couchbase.util Rebalance stuck count: 448 INFO [2022-03-13 17:17:38,075] jepsen worker 29 - jepsen.util 4259 :info :write [9 3] :SyncWriteAmbiguous INFO [2022-03-13 17:17:38,076] jepsen worker 30 - jepsen.util 3810 :ok :read [10 :nil] INFO [2022-03-13 17:17:38,077] jepsen worker 30 - jepsen.util 3810 :invoke :write [10 18] INFO [2022-03-13 17:17:38,078] jepsen worker 29 - jepsen.util 4349 :invoke :write [9 11] INFO [2022-03-13 17:17:38,099] jepsen worker 20 - jepsen.util 4160 :ok :read [6 16] INFO [2022-03-13 17:17:38,100] jepsen worker 20 - jepsen.util 4160 :invoke :write [6 42] INFO [2022-03-13 17:17:38,111] jepsen worker 63 - jepsen.util 4113 :ok :read [21 :nil] INFO [2022-03-13 17:17:38,112] jepsen worker 63 - jepsen.util 4113 :invoke :read [21 nil] INFO [2022-03-13 17:17:38,150] jepsen worker 63 - jepsen.util 4113 :ok :read [21 :nil] INFO [2022-03-13 17:17:38,150] jepsen worker 63 - jepsen.util 4113 :invoke :read [21 nil] INFO [2022-03-13 17:17:38,153] jepsen worker 46 - jepsen.util 4096 :ok :read [15 48] INFO [2022-03-13 17:17:38,154] jepsen worker 46 - jepsen.util 4096 :invoke :write [15 39] INFO [2022-03-13 17:17:38,171] jepsen worker 49 - jepsen.util 3559 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:17:38,173] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 33] INFO [2022-03-13 17:17:38,227] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,228] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:17:38,242] jepsen worker 71 - jepsen.util 611 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,244] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:17:38,255] jepsen worker 23 - jepsen.util 563 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,256] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:38,258] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,258] jepsen worker 61 - jepsen.util 781 :invoke :write [20 23] INFO [2022-03-13 17:17:38,272] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,272] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,273] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:38,273] jepsen worker 33 - jepsen.util 753 :invoke :write [11 10] INFO [2022-03-13 17:17:38,279] jepsen worker 63 - jepsen.util 4113 :ok :read [21 :nil] INFO [2022-03-13 17:17:38,280] jepsen worker 63 - jepsen.util 4113 :invoke :write [21 1] WARN [2022-03-13 17:17:38,290] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10077251,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10077251,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9066042,"encode_duration_us":17398,"operation_id":"0xfd74","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10009939},{"operation_name":"upsert","last_dispatch_duration_us":17338857,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17792598,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9112163,"encode_duration_us":20021,"operation_id":"0xfc15","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10010863}],"total_count":2}} WARN [2022-03-13 17:17:38,297] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9241453,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11343722,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9441048,"encode_duration_us":31411,"operation_id":"0xfdf2","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10006069},{"operation_name":"upsert","last_dispatch_duration_us":9248710,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10267143,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9217906,"encode_duration_us":18921,"operation_id":"0xff9e","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10008605}],"total_count":2}} WARN [2022-03-13 17:17:38,395] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":6614,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":15823978,"total_server_duration_us":415182,"operation_id":"0xf91d","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":30002816},{"operation_name":"upsert","last_dispatch_duration_us":8074100,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9561377,"total_server_duration_us":339687,"encode_duration_us":19009,"operation_id":"0xff08","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10009529},{"operation_name":"upsert","last_dispatch_duration_us":7874817,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8622545,"total_server_duration_us":102973,"encode_duration_us":21454,"operation_id":"0xfd89","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10006003},{"operation_name":"upsert","last_dispatch_duration_us":8790610,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":10447940,"total_server_duration_us":811812,"encode_duration_us":19297,"operation_id":"0xfef3","timeout_ms":10000,"last_server_duration_us":51121,"total_duration_us":10005716},{"operation_name":"upsert","last_dispatch_duration_us":165324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":540039,"total_server_duration_us":139548,"encode_duration_us":18690,"operation_id":"0xff59","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10001832},{"operation_name":"upsert","last_dispatch_duration_us":9438900,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9966454,"total_server_duration_us":9442952,"encode_duration_us":20925,"operation_id":"0xff41","timeout_ms":10000,"last_server_duration_us":9305434,"total_duration_us":10000823},{"operation_name":"upsert","last_dispatch_duration_us":9425203,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9735487,"total_server_duration_us":9168812,"encode_duration_us":21133,"operation_id":"0xff09","timeout_ms":10000,"last_server_duration_us":9073483,"total_duration_us":9746062},{"operation_name":"upsert","last_dispatch_duration_us":9057926,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9057926,"total_server_duration_us":9041612,"encode_duration_us":16828,"operation_id":"0xff90","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9062496},{"operation_name":"get","last_dispatch_duration_us":7650135,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7650135,"total_server_duration_us":7803,"operation_id":"0xffa0","timeout_ms":30000,"last_server_duration_us":7803,"total_duration_us":7652030},{"operation_name":"get","last_dispatch_duration_us":5163348,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5163348,"total_server_duration_us":43,"operation_id":"0x100a1","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5166036}],"total_count":30}} INFO [2022-03-13 17:17:38,399] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,400] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:38,401] jepsen worker 6 - jepsen.util 636 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,401] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:38,425] jepsen worker 79 - jepsen.util 1159 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,425] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 28] WARN [2022-03-13 17:17:38,486] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":23067,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":14186707,"total_server_duration_us":1783,"operation_id":"0xf8a7","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":30007473},{"operation_name":"upsert","last_dispatch_duration_us":66314,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1817927,"total_server_duration_us":706797,"encode_duration_us":17532,"operation_id":"0xff56","timeout_ms":10000,"last_server_duration_us":28676,"total_duration_us":10008486},{"operation_name":"upsert","last_dispatch_duration_us":118519,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":4002496,"total_server_duration_us":218161,"encode_duration_us":17383,"operation_id":"0xfdd8","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10007171},{"operation_name":"upsert","last_dispatch_duration_us":8898368,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10242769,"total_server_duration_us":311059,"encode_duration_us":23670,"operation_id":"0xfcbb","timeout_ms":10000,"last_server_duration_us":25976,"total_duration_us":10006867},{"operation_name":"upsert","last_dispatch_duration_us":49060,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":2654104,"total_server_duration_us":894259,"encode_duration_us":18381,"operation_id":"0xfefa","timeout_ms":10000,"last_server_duration_us":36729,"total_duration_us":10005644},{"operation_name":"upsert","last_dispatch_duration_us":9037842,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10314039,"total_server_duration_us":9289906,"encode_duration_us":29322,"operation_id":"0xfcac","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10004347},{"operation_name":"upsert","last_dispatch_duration_us":8994059,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9148948,"total_server_duration_us":59259,"encode_duration_us":18309,"operation_id":"0xff31","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10003632},{"operation_name":"upsert","last_dispatch_duration_us":31935,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1337703,"total_server_duration_us":254049,"encode_duration_us":17960,"operation_id":"0xfcbc","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10002646},{"operation_name":"upsert","last_dispatch_duration_us":8492887,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9899813,"total_server_duration_us":326375,"encode_duration_us":23638,"operation_id":"0xfc9f","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10001368},{"operation_name":"get","last_dispatch_duration_us":8964850,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":8964850,"total_server_duration_us":38,"operation_id":"0xfd12","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":8965084}],"total_count":19}} WARN [2022-03-13 17:17:38,487] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9243624,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17402928,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9281723,"encode_duration_us":18202,"operation_id":"0xfc1f","timeout_ms":10000,"last_server_duration_us":9078800,"total_duration_us":10002568},{"operation_name":"upsert","last_dispatch_duration_us":9037842,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10314039,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9289906,"encode_duration_us":29322,"operation_id":"0xfcac","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10004347},{"operation_name":"upsert","last_dispatch_duration_us":13713997,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":13713997,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":205285,"encode_duration_us":64963,"operation_id":"0xfb6d","timeout_ms":10000,"last_server_duration_us":205285,"total_duration_us":10009396},{"operation_name":"upsert","last_dispatch_duration_us":9018386,"last_local_id":"A312307A00000005/00000000B6F00358","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":18416258,"last_local_socket":"172.23.104.79:37438","total_server_duration_us":531171,"encode_duration_us":19320,"operation_id":"0xf9da","timeout_ms":10000,"last_server_duration_us":173123,"total_duration_us":10009764},{"operation_name":"upsert","last_dispatch_duration_us":9057502,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12855430,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9321783,"encode_duration_us":21716,"operation_id":"0xfbbf","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10006034}],"total_count":5}} WARN [2022-03-13 17:17:38,491] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15683579,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":15683579,"total_server_duration_us":468,"operation_id":"0xfcd7","timeout_ms":30000,"last_server_duration_us":468,"total_duration_us":15683892},{"operation_name":"upsert","last_dispatch_duration_us":509206,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8436174,"total_server_duration_us":46634,"encode_duration_us":17659,"operation_id":"0xff9c","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10010497},{"operation_name":"upsert","last_dispatch_duration_us":10077251,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10077251,"total_server_duration_us":9066042,"encode_duration_us":17398,"operation_id":"0xfd74","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10009939},{"operation_name":"upsert","last_dispatch_duration_us":512350,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10061803,"total_server_duration_us":39465,"encode_duration_us":16670,"operation_id":"0xff16","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10009075},{"operation_name":"upsert","last_dispatch_duration_us":43903,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":496852,"total_server_duration_us":204937,"encode_duration_us":16991,"operation_id":"0xfde3","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10008863},{"operation_name":"upsert","last_dispatch_duration_us":141951,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":1777987,"total_server_duration_us":874132,"encode_duration_us":18929,"operation_id":"0xff62","timeout_ms":10000,"last_server_duration_us":41123,"total_duration_us":10000635},{"operation_name":"get","last_dispatch_duration_us":8963464,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8963464,"total_server_duration_us":9628,"operation_id":"0xff55","timeout_ms":30000,"last_server_duration_us":9628,"total_duration_us":8968072},{"operation_name":"get","last_dispatch_duration_us":6959344,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":6959344,"total_server_duration_us":13756,"operation_id":"0xfff8","timeout_ms":30000,"last_server_duration_us":13756,"total_duration_us":6961425},{"operation_name":"upsert","last_dispatch_duration_us":2843753,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2843753,"total_server_duration_us":73821,"encode_duration_us":38234,"operation_id":"0x10071","timeout_ms":10000,"last_server_duration_us":73821,"total_duration_us":2849411},{"operation_name":"get","last_dispatch_duration_us":2484574,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2484574,"total_server_duration_us":38,"operation_id":"0x1007c","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":2484793}],"total_count":49}} INFO [2022-03-13 17:17:38,569] jepsen worker 28 - jepsen.util 3988 :ok :read [9 26] INFO [2022-03-13 17:17:38,570] jepsen worker 28 - jepsen.util 3988 :invoke :write [9 0] INFO [2022-03-13 17:17:38,573] jepsen worker 81 - jepsen.util 4491 :info :write [27 10] :RequestTimeoutException INFO [2022-03-13 17:17:38,574] jepsen worker 81 - jepsen.util 4581 :invoke :read [27 nil] WARN [2022-03-13 17:17:38,596] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9438900,"last_local_id":"A312307A00000003/00000000B2EF4A50","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9966454,"last_local_socket":"172.23.104.79:33492","total_server_duration_us":9442952,"encode_duration_us":20925,"operation_id":"0xff41","timeout_ms":10000,"last_server_duration_us":9305434,"total_duration_us":10000823}],"total_count":1}} WARN [2022-03-13 17:17:38,596] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":32927,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":643643,"total_server_duration_us":305143,"encode_duration_us":19382,"operation_id":"0xfddb","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10003821},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":21184,"operation_id":"0xff0b","timeout_ms":10000,"total_duration_us":10002964},{"operation_name":"upsert","last_dispatch_duration_us":9502425,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":9739777,"total_server_duration_us":9358615,"encode_duration_us":19212,"operation_id":"0xff53","timeout_ms":10000,"last_server_duration_us":9266781,"total_duration_us":9753621},{"operation_name":"upsert","last_dispatch_duration_us":9104258,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9395115,"total_server_duration_us":9122561,"encode_duration_us":17605,"operation_id":"0x1001c","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":9438768},{"operation_name":"upsert","last_dispatch_duration_us":9078838,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9194267,"total_server_duration_us":9072118,"encode_duration_us":33818,"operation_id":"0xfdd0","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9210071},{"operation_name":"upsert","last_dispatch_duration_us":9192003,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9192003,"total_server_duration_us":9058604,"encode_duration_us":18764,"operation_id":"0xfce0","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":9197283},{"operation_name":"get","last_dispatch_duration_us":7060283,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7060283,"total_server_duration_us":32,"operation_id":"0x100b1","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":7063496},{"operation_name":"get","last_dispatch_duration_us":5976796,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":5976796,"total_server_duration_us":19,"operation_id":"0x100fb","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":5978921},{"operation_name":"upsert","last_dispatch_duration_us":3748496,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3748496,"total_server_duration_us":19496,"encode_duration_us":21277,"operation_id":"0x10178","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":3752377},{"operation_name":"upsert","last_dispatch_duration_us":3082037,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3082037,"total_server_duration_us":350553,"encode_duration_us":17350,"operation_id":"0xffdf","timeout_ms":10000,"last_server_duration_us":350553,"total_duration_us":3192678}],"total_count":22}} WARN [2022-03-13 17:17:38,598] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8558710,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11157051,"total_server_duration_us":630182,"encode_duration_us":17021,"operation_id":"0xfde6","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10010060},{"operation_name":"upsert","last_dispatch_duration_us":8358860,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10625714,"total_server_duration_us":588568,"encode_duration_us":18317,"operation_id":"0xfdf0","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10009232},{"operation_name":"upsert","last_dispatch_duration_us":8596834,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10353970,"total_server_duration_us":300066,"encode_duration_us":16955,"operation_id":"0xffa4","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10009208},{"operation_name":"upsert","last_dispatch_duration_us":8529838,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11004215,"total_server_duration_us":605924,"encode_duration_us":18364,"operation_id":"0xfdf1","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10009087},{"operation_name":"upsert","last_dispatch_duration_us":36096,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":4942409,"total_server_duration_us":338471,"encode_duration_us":20320,"operation_id":"0xfe70","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10008685},{"operation_name":"upsert","last_dispatch_duration_us":8476757,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10451259,"total_server_duration_us":261970,"encode_duration_us":18042,"operation_id":"0xffae","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10008617},{"operation_name":"upsert","last_dispatch_duration_us":9248710,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10267143,"total_server_duration_us":9217906,"encode_duration_us":18921,"operation_id":"0xff9e","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10008605},{"operation_name":"upsert","last_dispatch_duration_us":9241453,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11343722,"total_server_duration_us":9441048,"encode_duration_us":31411,"operation_id":"0xfdf2","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10006069},{"operation_name":"upsert","last_dispatch_duration_us":124523,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":1830493,"total_server_duration_us":344023,"encode_duration_us":19700,"operation_id":"0xffa1","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10004124},{"operation_name":"upsert","last_dispatch_duration_us":90537,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2197773,"total_server_duration_us":771643,"encode_duration_us":19222,"operation_id":"0xff1b","timeout_ms":10000,"last_server_duration_us":28860,"total_duration_us":10003348}],"total_count":62}} INFO [2022-03-13 17:17:38,612] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,612] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:17:38,612] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,613] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,613] jepsen worker 71 - jepsen.util 611 :invoke :write [23 37] INFO [2022-03-13 17:17:38,614] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:17:38,617] jepsen worker 8 - jepsen.util 728 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,618] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,618] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,619] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:38,620] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,621] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:38,624] jepsen worker 7 - jepsen.util 637 :invoke :write [2 48] INFO [2022-03-13 17:17:38,627] jepsen worker 86 - jepsen.util 3776 :info :write [28 37] :RequestTimeoutException INFO [2022-03-13 17:17:38,628] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:38,628] jepsen worker 86 - jepsen.util 3866 :invoke :read [28 nil] INFO [2022-03-13 17:17:38,645] jepsen worker 78 - jepsen.util 708 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,645] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,645] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,646] jepsen worker 23 - jepsen.util 563 :invoke :write [7 47] INFO [2022-03-13 17:17:38,646] jepsen worker 78 - jepsen.util 708 :invoke :write [26 39] INFO [2022-03-13 17:17:38,647] jepsen worker 77 - jepsen.util 707 :invoke :write [25 3] INFO [2022-03-13 17:17:38,682] jepsen worker 42 - jepsen.util 402 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,684] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:38,809] jepsen worker 24 - jepsen.util 654 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,809] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:38,810] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,810] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:38,810] jepsen worker 51 - jepsen.util 771 :invoke :write [17 23] INFO [2022-03-13 17:17:38,811] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:38,811] jepsen worker 61 - jepsen.util 781 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,812] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:38,819] jepsen worker 71 - jepsen.util 611 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,820] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:17:38,878] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:38,878] jepsen worker 69 - jepsen.util 789 :invoke :write [23 22] INFO [2022-03-13 17:17:39,017] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,017] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,017] jepsen worker 12 - jepsen.util 912 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,017] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,017] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,017] jepsen worker 8 - jepsen.util 728 :invoke :write [2 32] INFO [2022-03-13 17:17:39,019] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:39,019] jepsen worker 70 - jepsen.util 880 :invoke :write [23 0] INFO [2022-03-13 17:17:39,019] jepsen worker 12 - jepsen.util 912 :invoke :write [4 11] INFO [2022-03-13 17:17:39,020] jepsen worker 62 - jepsen.util 512 :invoke :write [20 45] INFO [2022-03-13 17:17:39,023] jepsen worker 34 - jepsen.util 934 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,023] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:39,072] jepsen worker 83 - jepsen.util 3593 :ok :read [27 4] INFO [2022-03-13 17:17:39,072] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,073] jepsen worker 71 - jepsen.util 611 :invoke :write [23 19] INFO [2022-03-13 17:17:39,073] jepsen worker 83 - jepsen.util 3593 :invoke :write [27 5] INFO [2022-03-13 17:17:39,075] jepsen worker 79 - jepsen.util 1159 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,076] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 49] INFO [2022-03-13 17:17:39,080] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.760233918128655}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5470588235294118}} INFO [2022-03-13 17:17:39,080] jepsen worker nemesis - couchbase.util Rebalance check count: 566 INFO [2022-03-13 17:17:39,080] jepsen worker nemesis - couchbase.util Rebalance stuck count: 449 INFO [2022-03-13 17:17:39,132] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,132] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:17:39,140] jepsen worker 69 - jepsen.util 789 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,140] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:39,167] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:39,167] jepsen worker 7 - jepsen.util 637 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,167] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,168] jepsen worker 7 - jepsen.util 637 :invoke :write [2 39] INFO [2022-03-13 17:17:39,169] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:17:39,172] jepsen worker 6 - jepsen.util 636 :invoke :write [2 41] INFO [2022-03-13 17:17:39,230] jepsen worker 71 - jepsen.util 611 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,230] jepsen worker 78 - jepsen.util 708 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,230] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,231] jepsen worker 78 - jepsen.util 708 :invoke :write [26 44] INFO [2022-03-13 17:17:39,234] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:17:39,234] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:17:39,239] jepsen worker 80 - jepsen.util 800 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,239] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:39,287] jepsen worker 8 - jepsen.util 728 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,287] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:39,438] jepsen worker 39 - jepsen.util 4179 :info :write [13 35] :RequestTimeoutException INFO [2022-03-13 17:17:39,439] jepsen worker 39 - jepsen.util 4269 :invoke :read [13 nil] INFO [2022-03-13 17:17:39,440] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,441] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:39,442] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,443] jepsen worker 21 - jepsen.util 831 :invoke :write [7 15] INFO [2022-03-13 17:17:39,486] jepsen worker 77 - jepsen.util 707 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,487] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:39,636] jepsen worker 84 - jepsen.util 4314 :ok :read [28 44] INFO [2022-03-13 17:17:39,637] jepsen worker 84 - jepsen.util 4314 :invoke :write [28 12] INFO [2022-03-13 17:17:39,639] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,640] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,640] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:39,641] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:17:39,641] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,642] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:17:39,644] jepsen worker 82 - jepsen.util 4132 :ok :read [27 4] INFO [2022-03-13 17:17:39,645] jepsen worker 82 - jepsen.util 4132 :invoke :write [27 24] INFO [2022-03-13 17:17:39,646] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,646] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,646] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:39,647] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:39,647] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:39,647] jepsen worker 44 - jepsen.util 674 :invoke :write [14 24] INFO [2022-03-13 17:17:39,681] jepsen worker 62 - jepsen.util 512 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,682] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:39,718] jepsen worker 23 - jepsen.util 563 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,718] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,719] jepsen worker 23 - jepsen.util 563 :invoke :write [7 18] INFO [2022-03-13 17:17:39,719] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:39,844] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,844] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:17:39,857] jepsen worker 59 - jepsen.util 4019 :ok :read [19 16] INFO [2022-03-13 17:17:39,857] jepsen worker 59 - jepsen.util 4019 :invoke :read [19 nil] INFO [2022-03-13 17:17:39,860] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,861] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:39,864] jepsen worker 51 - jepsen.util 771 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,865] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:39,875] jepsen worker 39 - jepsen.util 4269 :ok :read [13 39] INFO [2022-03-13 17:17:39,876] jepsen worker 39 - jepsen.util 4269 :invoke :read [13 nil] INFO [2022-03-13 17:17:39,883] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,883] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:17:39,916] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:39,916] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:39,951] jepsen worker 41 - jepsen.util 3911 :info :write [13 41] :SyncWriteAmbiguous INFO [2022-03-13 17:17:39,951] jepsen worker 41 - jepsen.util 4001 :invoke :write [13 10] INFO [2022-03-13 17:17:40,011] jepsen worker 15 - jepsen.util 3795 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:17:40,011] jepsen worker 15 - jepsen.util 3885 :invoke :write [5 39] INFO [2022-03-13 17:17:40,020] jepsen worker 17 - jepsen.util 3977 :info :write [5 39] :RequestTimeoutException INFO [2022-03-13 17:17:40,021] jepsen worker 17 - jepsen.util 4067 :invoke :read [5 nil] INFO [2022-03-13 17:17:40,069] jepsen worker 68 - jepsen.util 4028 :ok :read [22 30] INFO [2022-03-13 17:17:40,069] jepsen worker 68 - jepsen.util 4028 :invoke :read [22 nil] INFO [2022-03-13 17:17:40,072] jepsen worker 59 - jepsen.util 4019 :ok :read [19 16] INFO [2022-03-13 17:17:40,073] jepsen worker 59 - jepsen.util 4019 :invoke :write [19 16] INFO [2022-03-13 17:17:40,090] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.760233918128655}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5470588235294118}} INFO [2022-03-13 17:17:40,090] jepsen worker nemesis - couchbase.util Rebalance check count: 567 INFO [2022-03-13 17:17:40,090] jepsen worker nemesis - couchbase.util Rebalance stuck count: 450 INFO [2022-03-13 17:17:40,096] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,096] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,096] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,097] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,097] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,097] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:17:40,098] jepsen worker 80 - jepsen.util 800 :invoke :write [26 15] INFO [2022-03-13 17:17:40,098] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:40,097] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,106] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:40,108] jepsen worker 53 - jepsen.util 593 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,108] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,108] jepsen worker 70 - jepsen.util 880 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,109] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,110] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:40,111] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,111] jepsen worker 70 - jepsen.util 880 :invoke :write [23 1] INFO [2022-03-13 17:17:40,112] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:17:40,130] jepsen worker 39 - jepsen.util 4269 :ok :read [13 39] INFO [2022-03-13 17:17:40,131] jepsen worker 39 - jepsen.util 4269 :invoke :write [13 22] INFO [2022-03-13 17:17:40,135] jepsen worker 67 - jepsen.util 3217 :ok :read [22 30] INFO [2022-03-13 17:17:40,136] jepsen worker 67 - jepsen.util 3217 :invoke :read [22 nil] INFO [2022-03-13 17:17:40,202] jepsen worker 33 - jepsen.util 753 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,202] jepsen worker 33 - jepsen.util 753 :invoke :write [11 41] INFO [2022-03-13 17:17:40,210] jepsen worker 79 - jepsen.util 1159 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,211] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:40,236] jepsen worker 1 - jepsen.util 3511 :ok :read [0 41] INFO [2022-03-13 17:17:40,236] jepsen worker 1 - jepsen.util 3511 :invoke :read [0 nil] INFO [2022-03-13 17:17:40,247] jepsen worker 68 - jepsen.util 4028 :ok :read [22 30] INFO [2022-03-13 17:17:40,247] jepsen worker 68 - jepsen.util 4028 :invoke :write [22 0] INFO [2022-03-13 17:17:40,259] jepsen worker 67 - jepsen.util 3217 :ok :read [22 30] INFO [2022-03-13 17:17:40,259] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,259] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,260] jepsen worker 67 - jepsen.util 3217 :invoke :write [22 9] INFO [2022-03-13 17:17:40,294] jepsen worker 1 - jepsen.util 3511 :ok :read [0 41] INFO [2022-03-13 17:17:40,295] jepsen worker 1 - jepsen.util 3511 :invoke :write [0 40] INFO [2022-03-13 17:17:40,301] jepsen worker 17 - jepsen.util 4067 :ok :read [5 43] INFO [2022-03-13 17:17:40,302] jepsen worker 17 - jepsen.util 4067 :invoke :write [5 30] INFO [2022-03-13 17:17:40,308] jepsen worker 13 - jepsen.util 733 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,308] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:40,323] jepsen worker 21 - jepsen.util 831 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,323] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,323] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,324] jepsen worker 24 - jepsen.util 654 :invoke :write [8 46] INFO [2022-03-13 17:17:40,325] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:40,325] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:40,331] jepsen worker 12 - jepsen.util 912 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,332] jepsen worker 12 - jepsen.util 912 :invoke :write [4 28] INFO [2022-03-13 17:17:40,355] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,355] jepsen worker 78 - jepsen.util 708 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,355] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,355] jepsen worker 77 - jepsen.util 707 :invoke :write [25 35] INFO [2022-03-13 17:17:40,356] jepsen worker 78 - jepsen.util 708 :invoke :write [26 47] INFO [2022-03-13 17:17:40,361] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:40,547] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,548] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:17:40,623] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,623] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:17:40,702] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,702] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:17:40,707] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,708] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:17:40,739] jepsen worker 7 - jepsen.util 637 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,740] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:40,748] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,748] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,748] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,749] jepsen worker 69 - jepsen.util 789 :invoke :write [23 14] INFO [2022-03-13 17:17:40,750] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,751] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:17:40,762] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,763] jepsen worker 23 - jepsen.util 563 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,763] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,764] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 42] INFO [2022-03-13 17:17:40,767] jepsen worker 61 - jepsen.util 781 :invoke :write [20 25] INFO [2022-03-13 17:17:40,768] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] INFO [2022-03-13 17:17:40,798] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,799] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:17:40,799] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,800] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:17:40,809] jepsen worker 44 - jepsen.util 674 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,810] jepsen worker 44 - jepsen.util 674 :invoke :write [14 31] INFO [2022-03-13 17:17:40,810] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,811] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,811] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,811] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,812] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,812] jepsen worker 42 - jepsen.util 402 :invoke :write [14 3] INFO [2022-03-13 17:17:40,813] jepsen worker 51 - jepsen.util 771 :invoke :write [17 33] INFO [2022-03-13 17:17:40,815] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:40,815] jepsen worker 34 - jepsen.util 934 :invoke :write [11 26] INFO [2022-03-13 17:17:40,815] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:17:40,822] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,823] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:17:40,849] jepsen worker 77 - jepsen.util 707 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,850] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:17:40,859] jepsen worker 69 - jepsen.util 789 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,860] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,906] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,906] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:40,907] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,908] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:40,908] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,909] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:40,910] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,911] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:40,912] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,912] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:40,914] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,914] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:40,915] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,916] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:17:40,922] jepsen worker 80 - jepsen.util 800 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,922] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:40,924] jepsen worker 6 - jepsen.util 636 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,924] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,924] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,925] jepsen worker 6 - jepsen.util 636 :invoke :write [2 13] INFO [2022-03-13 17:17:40,925] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:40,926] jepsen worker 8 - jepsen.util 728 :invoke :write [2 0] INFO [2022-03-13 17:17:40,931] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,931] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 24] INFO [2022-03-13 17:17:40,939] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,940] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:40,942] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,942] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:17:40,944] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,945] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,947] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,948] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,949] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,949] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,950] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:17:40,950] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:40,953] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,954] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:17:40,999] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,999] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:40,999] jepsen worker 33 - jepsen.util 753 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,000] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:17:41,001] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:17:41,001] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:41,027] jepsen worker 36 - jepsen.util 2466 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,028] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:17:41,050] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,051] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:41,086] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,086] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:17:41,096] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.760233918128655}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5470588235294118}} INFO [2022-03-13 17:17:41,096] jepsen worker nemesis - couchbase.util Rebalance check count: 568 INFO [2022-03-13 17:17:41,096] jepsen worker nemesis - couchbase.util Rebalance stuck count: 451 INFO [2022-03-13 17:17:41,102] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,103] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:17:41,148] jepsen worker 78 - jepsen.util 708 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,148] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:41,160] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,161] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 28] INFO [2022-03-13 17:17:41,162] jepsen worker 49 - jepsen.util 3559 :fail :write [16 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,162] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,162] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 10] INFO [2022-03-13 17:17:41,163] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:17:41,179] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,180] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,181] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,182] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,183] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,184] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:17:41,200] jepsen worker 70 - jepsen.util 880 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,201] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:41,203] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,204] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,215] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,216] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:41,235] jepsen worker 79 - jepsen.util 1159 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,236] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 16] INFO [2022-03-13 17:17:41,289] jepsen worker 49 - jepsen.util 3559 :fail :write [16 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,289] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 24] INFO [2022-03-13 17:17:41,310] jepsen worker 36 - jepsen.util 2466 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,310] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,310] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:41,311] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,312] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,312] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,313] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:41,314] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:17:41,315] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,316] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:17:41,318] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,319] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,325] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,326] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:17:41,340] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,340] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:41,379] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,379] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,379] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:41,380] jepsen worker 71 - jepsen.util 611 :invoke :write [23 35] INFO [2022-03-13 17:17:41,387] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,387] jepsen worker 24 - jepsen.util 654 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,387] jepsen worker 24 - jepsen.util 654 :invoke :write [8 28] INFO [2022-03-13 17:17:41,388] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:41,390] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,390] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:41,392] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,392] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:17:41,433] jepsen worker 49 - jepsen.util 3559 :fail :write [16 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,433] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,434] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:41,435] jepsen worker 49 - jepsen.util 3559 :invoke :read [16 nil] INFO [2022-03-13 17:17:41,439] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,439] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,440] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:41,440] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,444] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,445] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:17:41,448] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,448] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,450] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,451] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,452] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,453] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,454] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,454] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:17:41,492] jepsen worker 12 - jepsen.util 912 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,493] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:41,495] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,495] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:41,522] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,523] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:41,524] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,524] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,524] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,525] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,525] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:17:41,525] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,525] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:41,526] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,527] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,527] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:17:41,530] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,531] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:17:41,574] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,575] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:17:41,583] jepsen worker 51 - jepsen.util 771 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,584] jepsen worker 51 - jepsen.util 771 :invoke :write [17 45] INFO [2022-03-13 17:17:41,621] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,621] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:41,672] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,672] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:17:41,681] jepsen worker 32 - jepsen.util 4172 :info :write [10 14] :RequestTimeoutException INFO [2022-03-13 17:17:41,682] jepsen worker 32 - jepsen.util 4262 :invoke :write [10 4] INFO [2022-03-13 17:17:41,693] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,693] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,693] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:17:41,694] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 40] INFO [2022-03-13 17:17:41,714] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,715] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:17:41,732] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,732] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:17:41,810] jepsen worker 3 - jepsen.util 2253 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,810] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,823] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,823] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,824] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,825] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:17:41,834] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,835] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:17:41,840] jepsen worker 16 - jepsen.util 3796 :info :write [5 44] :RequestTimeoutException INFO [2022-03-13 17:17:41,841] jepsen worker 16 - jepsen.util 3886 :invoke :write [5 13] INFO [2022-03-13 17:17:41,876] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,876] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,877] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,877] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:41,879] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,879] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,880] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:41,880] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:17:41,881] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,881] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:17:41,922] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,924] jepsen worker 31 - jepsen.util 4081 :info :write [10 39] :RequestTimeoutException INFO [2022-03-13 17:17:41,924] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:17:41,925] jepsen worker 31 - jepsen.util 4171 :invoke :write [10 10] INFO [2022-03-13 17:17:41,933] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:41,933] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:17:42,020] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,021] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:17:42,034] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,034] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,043] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,043] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,046] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,047] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,051] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,052] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:17:42,056] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,057] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,060] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,061] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:17:42,096] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,097] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:17:42,101] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7719298245614035}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5588235294117647}} INFO [2022-03-13 17:17:42,101] jepsen worker nemesis - couchbase.util Rebalance check count: 569 INFO [2022-03-13 17:17:42,102] jepsen worker nemesis - couchbase.util Rebalance stuck count: 451 INFO [2022-03-13 17:17:42,160] jepsen worker 71 - jepsen.util 611 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,160] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:17:42,163] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,163] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,166] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,166] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:17:42,174] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,174] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,178] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,179] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,181] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,182] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:17:42,184] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,189] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:17:42,234] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,235] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:17:42,250] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,251] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,252] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,253] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:17:42,265] jepsen worker 61 - jepsen.util 781 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,266] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:42,269] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,270] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:17:42,286] jepsen worker 57 - jepsen.util 3927 :ok :read [19 6] INFO [2022-03-13 17:17:42,287] jepsen worker 57 - jepsen.util 3927 :invoke :write [19 37] INFO [2022-03-13 17:17:42,289] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,290] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,295] jepsen worker 0 - jepsen.util 3780 :ok :read [0 41] INFO [2022-03-13 17:17:42,295] jepsen worker 0 - jepsen.util 3780 :invoke :write [0 30] INFO [2022-03-13 17:17:42,313] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,314] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:17:42,344] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,345] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:17:42,352] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,352] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:17:42,356] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,356] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,357] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:42,357] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:17:42,358] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,359] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:17:42,360] jepsen worker 58 - jepsen.util 3658 :ok :read [19 6] INFO [2022-03-13 17:17:42,361] jepsen worker 58 - jepsen.util 3658 :invoke :read [19 nil] INFO [2022-03-13 17:17:42,418] jepsen worker 75 - jepsen.util 525 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,418] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:42,442] jepsen worker 42 - jepsen.util 402 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,442] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,443] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:17:42,443] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:42,444] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,444] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:42,449] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,450] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,461] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,461] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,462] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,463] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:17:42,464] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,464] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,467] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,468] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,469] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,470] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:17:42,500] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,501] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:42,508] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,508] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,509] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:17:42,509] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:17:42,513] jepsen worker 52 - jepsen.util 682 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,513] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:42,513] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,514] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:42,515] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,516] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,522] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,523] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 47] INFO [2022-03-13 17:17:42,526] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,527] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:17:42,548] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,548] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,552] jepsen worker 58 - jepsen.util 3658 :ok :read [19 6] INFO [2022-03-13 17:17:42,553] jepsen worker 58 - jepsen.util 3658 :invoke :write [19 10] INFO [2022-03-13 17:17:42,554] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,555] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,557] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,557] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,562] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,562] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,563] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,564] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:17:42,587] jepsen worker 3 - jepsen.util 2253 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,588] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,596] jepsen worker 76 - jepsen.util 796 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,597] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:42,611] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,612] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,614] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,614] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:17:42,615] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,616] jepsen worker 13 - jepsen.util 733 :invoke :write [4 22] INFO [2022-03-13 17:17:42,616] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,616] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,617] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:42,618] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,618] jepsen worker 75 - jepsen.util 525 :invoke :write [25 13] INFO [2022-03-13 17:17:42,619] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:42,621] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,622] jepsen worker 76 - jepsen.util 796 :invoke :write [25 41] INFO [2022-03-13 17:17:42,628] jepsen worker 6 - jepsen.util 636 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,629] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:42,639] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,640] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:17:42,642] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,643] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:17:42,655] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,656] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:42,695] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,695] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,696] jepsen worker 78 - jepsen.util 708 :invoke :write [26 17] INFO [2022-03-13 17:17:42,696] jepsen worker 77 - jepsen.util 707 :invoke :write [25 19] INFO [2022-03-13 17:17:42,709] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,710] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,715] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,716] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,717] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,718] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:42,720] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,720] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,720] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,721] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:17:42,722] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:17:42,722] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:42,724] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,725] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:42,726] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,727] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:42,728] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,729] jepsen worker 69 - jepsen.util 789 :invoke :write [23 11] INFO [2022-03-13 17:17:42,731] jepsen worker 43 - jepsen.util 673 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,731] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,731] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:42,732] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,732] jepsen worker 60 - jepsen.util 690 :invoke :write [20 14] INFO [2022-03-13 17:17:42,732] jepsen worker 53 - jepsen.util 593 :invoke :write [17 26] INFO [2022-03-13 17:17:42,762] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,763] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,764] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,765] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,766] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,767] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:17:42,787] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,788] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:17:42,825] jepsen worker 13 - jepsen.util 733 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,825] jepsen worker 8 - jepsen.util 728 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,826] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,826] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,826] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,827] jepsen worker 7 - jepsen.util 637 :invoke :write [2 33] INFO [2022-03-13 17:17:42,827] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:42,828] jepsen worker 12 - jepsen.util 912 :invoke :write [4 37] INFO [2022-03-13 17:17:42,828] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:42,830] jepsen worker 42 - jepsen.util 402 :invoke :write [14 31] INFO [2022-03-13 17:17:42,831] jepsen worker 8 - jepsen.util 728 :invoke :write [2 34] INFO [2022-03-13 17:17:42,832] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,833] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,834] jepsen worker 13 - jepsen.util 733 :invoke :write [4 35] INFO [2022-03-13 17:17:42,842] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,843] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:17:42,851] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,852] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:17:42,862] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,862] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,863] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,863] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:17:42,864] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,865] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:17:42,869] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,870] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,871] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,872] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,873] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,874] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,876] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,877] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:17:42,896] jepsen worker 79 - jepsen.util 1159 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,897] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 1] INFO [2022-03-13 17:17:42,900] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,901] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:42,901] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,902] jepsen worker 61 - jepsen.util 781 :invoke :write [20 8] INFO [2022-03-13 17:17:42,915] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,915] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,915] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,915] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:42,916] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:42,917] jepsen worker 52 - jepsen.util 682 :invoke :write [17 38] INFO [2022-03-13 17:17:42,917] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:42,918] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:42,950] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,950] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:17:42,973] jepsen worker 69 - jepsen.util 789 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,975] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:17:42,993] jepsen worker 75 - jepsen.util 525 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,994] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,995] jepsen worker 75 - jepsen.util 525 :invoke :write [25 21] INFO [2022-03-13 17:17:42,995] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:42,996] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:42,997] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:17:43,000] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,001] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:17:43,051] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,051] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:43,053] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,053] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,054] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:43,055] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:43,082] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,082] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:43,084] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,084] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:17:43,107] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,107] jepsen worker 77 - jepsen.util 707 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,108] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:43,108] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7719298245614035}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5588235294117647}} INFO [2022-03-13 17:17:43,108] jepsen worker nemesis - couchbase.util Rebalance check count: 570 INFO [2022-03-13 17:17:43,108] jepsen worker nemesis - couchbase.util Rebalance stuck count: 452 INFO [2022-03-13 17:17:43,108] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:43,134] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,135] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,139] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,140] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,141] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,142] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:17:43,146] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,147] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:17:43,179] jepsen worker 53 - jepsen.util 593 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,180] jepsen worker 53 - jepsen.util 593 :invoke :write [17 7] INFO [2022-03-13 17:17:43,193] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,193] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:43,241] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,241] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:43,243] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,243] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:17:43,248] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,249] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,250] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,250] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,252] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,252] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:17:43,286] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,287] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,287] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:17:43,289] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:17:43,332] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,332] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,332] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,333] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:43,336] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,336] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,336] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:43,337] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:17:43,337] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,338] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:43,339] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,339] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:17:43,349] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:17:43,350] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:17:43,394] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,394] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,438] jepsen worker 76 - jepsen.util 796 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,439] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:43,451] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,451] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:43,453] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,454] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:17:43,461] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:43,461] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,461] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,461] jepsen worker 60 - jepsen.util 690 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,462] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:43,463] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:43,464] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:43,464] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:43,471] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,471] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,473] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,474] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,475] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,476] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:17:43,490] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,490] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,491] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:17:43,492] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:43,497] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,498] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:17:43,509] jepsen worker 13 - jepsen.util 733 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,509] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:43,528] jepsen worker 7 - jepsen.util 637 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,529] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:43,562] jepsen worker 44 - jepsen.util 674 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,562] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,562] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,563] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,564] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:43,564] jepsen worker 69 - jepsen.util 789 :invoke :write [23 49] INFO [2022-03-13 17:17:43,565] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:43,565] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:43,566] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,567] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:43,568] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,569] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:17:43,570] jepsen worker 52 - jepsen.util 682 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,570] jepsen worker 52 - jepsen.util 682 :invoke :write [17 15] INFO [2022-03-13 17:17:43,587] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,588] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:17:43,601] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,602] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,602] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:17:43,604] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:17:43,607] jepsen worker 40 - jepsen.util 4090 :info :write [13 5] :RequestTimeoutException INFO [2022-03-13 17:17:43,609] jepsen worker 40 - jepsen.util 4180 :invoke :read [13 nil] INFO [2022-03-13 17:17:43,656] jepsen worker 78 - jepsen.util 708 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,656] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,656] jepsen worker 78 - jepsen.util 708 :invoke :write [26 10] INFO [2022-03-13 17:17:43,661] jepsen worker 40 - jepsen.util 4180 :ok :read [13 39] INFO [2022-03-13 17:17:43,661] jepsen worker 40 - jepsen.util 4180 :invoke :write [13 37] INFO [2022-03-13 17:17:43,663] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:17:43,692] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,693] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:43,705] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,705] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:17:43,723] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,723] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,725] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:43,726] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:17:43,730] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,730] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:17:43,742] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,743] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:17:43,751] jepsen worker 74 - jepsen.util 4034 :ok :read [24 24] INFO [2022-03-13 17:17:43,752] jepsen worker 74 - jepsen.util 4034 :invoke :read [24 nil] INFO [2022-03-13 17:17:43,761] jepsen worker 56 - jepsen.util 4016 :ok :read [18 21] INFO [2022-03-13 17:17:43,761] jepsen worker 56 - jepsen.util 4016 :invoke :read [18 nil] INFO [2022-03-13 17:17:43,773] jepsen worker 42 - jepsen.util 402 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,774] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:43,779] jepsen worker 69 - jepsen.util 789 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,779] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,780] jepsen worker 69 - jepsen.util 789 :invoke :write [23 12] INFO [2022-03-13 17:17:43,781] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:43,786] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,787] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:17:43,788] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,789] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,812] jepsen worker 79 - jepsen.util 1159 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,812] jepsen worker 55 - jepsen.util 3655 :ok :read [18 21] INFO [2022-03-13 17:17:43,813] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 0] INFO [2022-03-13 17:17:43,813] jepsen worker 34 - jepsen.util 934 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,813] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,813] jepsen worker 55 - jepsen.util 3655 :invoke :write [18 20] INFO [2022-03-13 17:17:43,814] jepsen worker 34 - jepsen.util 934 :invoke :write [11 3] INFO [2022-03-13 17:17:43,815] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:43,830] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,831] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:17:43,876] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,876] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:17:43,881] jepsen worker 8 - jepsen.util 728 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,882] jepsen worker 8 - jepsen.util 728 :invoke :write [2 45] INFO [2022-03-13 17:17:43,887] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,887] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,888] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:17:43,888] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:43,890] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,891] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:43,892] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,893] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:43,895] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,896] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:43,897] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,898] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:17:43,926] jepsen worker 69 - jepsen.util 789 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,927] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:43,931] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,931] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,931] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:43,932] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:17:43,933] jepsen worker 61 - jepsen.util 781 :fail :write [20 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,933] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,934] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:43,935] jepsen worker 77 - jepsen.util 707 :invoke :write [25 46] INFO [2022-03-13 17:17:43,961] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,961] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,962] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,963] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,964] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,965] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,966] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,967] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:43,968] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,969] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:17:43,980] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,980] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:17:43,994] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:43,995] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 25] INFO [2022-03-13 17:17:44,014] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,014] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,016] jepsen worker 69 - jepsen.util 789 :invoke :write [23 43] INFO [2022-03-13 17:17:44,016] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:44,040] jepsen worker 24 - jepsen.util 654 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,040] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:44,044] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,045] jepsen worker 71 - jepsen.util 611 :invoke :write [23 39] INFO [2022-03-13 17:17:44,054] jepsen worker 12 - jepsen.util 912 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,054] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,054] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,054] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,054] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,054] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:44,055] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:44,056] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:44,056] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:17:44,057] jepsen worker 70 - jepsen.util 880 :invoke :write [23 22] INFO [2022-03-13 17:17:44,057] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:44,058] jepsen worker 6 - jepsen.util 636 :invoke :write [2 0] INFO [2022-03-13 17:17:44,142] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,146] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,146] jepsen worker 3 - jepsen.util 2253 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,147] jepsen worker 69 - jepsen.util 789 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,148] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 35] INFO [2022-03-13 17:17:44,148] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,148] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 16] INFO [2022-03-13 17:17:44,148] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:17:44,149] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,150] jepsen worker 69 - jepsen.util 789 :invoke :write [23 11] INFO [2022-03-13 17:17:44,151] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,152] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:17:44,153] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7836257309941521}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5705882352941176}} INFO [2022-03-13 17:17:44,153] jepsen worker nemesis - couchbase.util Rebalance check count: 571 INFO [2022-03-13 17:17:44,153] jepsen worker nemesis - couchbase.util Rebalance stuck count: 452 INFO [2022-03-13 17:17:44,255] jepsen worker 36 - jepsen.util 2466 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,256] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:44,270] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,270] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:17:44,279] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,279] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:44,280] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,281] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:17:44,311] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,311] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:44,321] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,321] jepsen worker 75 - jepsen.util 525 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,321] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,322] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:44,323] jepsen worker 13 - jepsen.util 733 :invoke :write [4 20] INFO [2022-03-13 17:17:44,323] jepsen worker 76 - jepsen.util 796 :invoke :write [25 37] INFO [2022-03-13 17:17:44,343] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,344] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:44,350] jepsen worker 3 - jepsen.util 2253 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,350] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,351] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:44,351] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,352] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,353] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:17:44,353] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,353] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:17:44,378] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,379] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:17:44,379] jepsen worker 71 - jepsen.util 611 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,380] jepsen worker 71 - jepsen.util 611 :invoke :write [23 19] INFO [2022-03-13 17:17:44,395] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,396] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,397] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,398] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:17:44,399] jepsen worker 8 - jepsen.util 728 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,399] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,400] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,400] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,400] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:44,401] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:44,401] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:44,402] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:44,440] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,440] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:44,440] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,441] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:44,442] jepsen worker 80 - jepsen.util 800 :invoke :write [26 30] INFO [2022-03-13 17:17:44,442] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,443] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:44,443] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,443] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:17:44,443] jepsen worker 33 - jepsen.util 753 :invoke :write [11 36] INFO [2022-03-13 17:17:44,444] jepsen worker 78 - jepsen.util 708 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,444] jepsen worker 78 - jepsen.util 708 :invoke :write [26 43] INFO [2022-03-13 17:17:44,461] jepsen worker 69 - jepsen.util 789 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,461] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:44,477] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,478] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:17:44,500] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,500] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,505] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,505] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:17:44,543] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,543] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,543] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:44,544] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:17:44,545] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,545] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:17:44,595] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,596] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:17:44,604] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,604] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:44,608] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,608] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:17:44,614] jepsen worker 34 - jepsen.util 934 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,614] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,615] jepsen worker 34 - jepsen.util 934 :invoke :write [11 15] INFO [2022-03-13 17:17:44,616] jepsen worker 14 - jepsen.util 374 :invoke :write [4 40] INFO [2022-03-13 17:17:44,619] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,619] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:17:44,620] jepsen worker 71 - jepsen.util 611 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,622] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:44,624] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,625] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:17:44,633] jepsen worker 70 - jepsen.util 880 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,634] jepsen worker 70 - jepsen.util 880 :invoke :write [23 15] INFO [2022-03-13 17:17:44,645] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,646] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,665] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,665] jepsen worker 79 - jepsen.util 1159 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,666] jepsen worker 61 - jepsen.util 781 :invoke :write [20 49] INFO [2022-03-13 17:17:44,666] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 26] INFO [2022-03-13 17:17:44,677] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,678] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:17:44,683] jepsen worker 88 - jepsen.util 2338 :info :write [29 27] :RequestTimeoutException INFO [2022-03-13 17:17:44,684] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:44,689] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,689] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:17:44,729] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,729] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,747] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,748] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,774] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:17:44,748] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,775] jepsen worker 69 - jepsen.util 789 :invoke :write [23 8] INFO [2022-03-13 17:17:44,775] jepsen worker 24 - jepsen.util 654 :invoke :write [8 49] INFO [2022-03-13 17:17:44,748] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,775] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:17:44,783] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,784] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:44,784] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,770] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,774] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:17:44,785] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:17:44,790] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,793] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,796] jepsen worker 47 - jepsen.util 3827 :info :write [15 41] :RequestTimeoutException INFO [2022-03-13 17:17:44,797] jepsen worker 47 - jepsen.util 3917 :invoke :read [15 nil] INFO [2022-03-13 17:17:44,802] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,803] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,804] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,805] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,806] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,806] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,807] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,808] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:17:44,821] jepsen worker 38 - jepsen.util 2738 :info :write [12 39] :RequestTimeoutException INFO [2022-03-13 17:17:44,822] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:44,850] jepsen worker 9 - jepsen.util 3429 :info :write [3 7] :RequestTimeoutException INFO [2022-03-13 17:17:44,851] jepsen worker 9 - jepsen.util 3519 :invoke :write [3 35] INFO [2022-03-13 17:17:44,858] jepsen worker 65 - jepsen.util 4295 :info :write [21 9] :RequestTimeoutException INFO [2022-03-13 17:17:44,858] jepsen worker 65 - jepsen.util 4385 :invoke :read [21 nil] INFO [2022-03-13 17:17:44,863] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,863] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:44,865] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,865] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:17:44,867] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,868] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:17:44,871] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,872] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:44,928] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,928] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,928] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:17:44,929] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:44,931] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,931] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:44,933] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,933] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:17:44,936] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,937] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:17:44,944] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:44,945] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:45,015] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,015] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,016] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,016] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 35] INFO [2022-03-13 17:17:45,017] jepsen worker 77 - jepsen.util 707 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,017] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,017] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,018] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:17:45,019] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:45,019] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,021] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,021] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,021] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 12] INFO [2022-03-13 17:17:45,022] jepsen worker 53 - jepsen.util 593 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,022] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,022] jepsen worker 51 - jepsen.util 771 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,022] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,023] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:45,023] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:45,023] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,023] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:45,023] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,023] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,024] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:17:45,025] jepsen worker 44 - jepsen.util 674 :invoke :write [14 11] INFO [2022-03-13 17:17:45,027] jepsen worker 60 - jepsen.util 690 :invoke :write [20 22] INFO [2022-03-13 17:17:45,027] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:45,027] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:45,028] jepsen worker 51 - jepsen.util 771 :invoke :write [17 49] INFO [2022-03-13 17:17:45,029] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:45,029] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:45,030] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:17:45,039] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,039] jepsen worker 69 - jepsen.util 789 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,043] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:17:45,043] jepsen worker 69 - jepsen.util 789 :invoke :write [23 16] INFO [2022-03-13 17:17:45,079] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,080] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:17:45,084] jepsen worker 80 - jepsen.util 800 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,084] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:45,110] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,110] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,110] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:17:45,111] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:17:45,122] jepsen worker 3 - jepsen.util 2253 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,123] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:17:45,148] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,148] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,156] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,156] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,156] jepsen worker 6 - jepsen.util 636 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,156] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,156] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,156] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:45,157] jepsen worker 8 - jepsen.util 728 :invoke :write [2 18] INFO [2022-03-13 17:17:45,157] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:45,158] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7836257309941521}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5705882352941176}} INFO [2022-03-13 17:17:45,159] jepsen worker nemesis - couchbase.util Rebalance check count: 572 INFO [2022-03-13 17:17:45,159] jepsen worker nemesis - couchbase.util Rebalance stuck count: 453 INFO [2022-03-13 17:17:45,159] jepsen worker 7 - jepsen.util 637 :invoke :write [2 29] INFO [2022-03-13 17:17:45,162] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:45,165] jepsen worker 66 - jepsen.util 3756 :info :write [22 9] :SyncWriteAmbiguous INFO [2022-03-13 17:17:45,167] jepsen worker 66 - jepsen.util 3846 :invoke :write [22 2] INFO [2022-03-13 17:17:45,183] jepsen worker 36 - jepsen.util 2466 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,184] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 4] INFO [2022-03-13 17:17:45,186] jepsen worker 37 - jepsen.util 2377 :info :write [12 41] :SyncWriteAmbiguous INFO [2022-03-13 17:17:45,187] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 5] INFO [2022-03-13 17:17:45,190] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,191] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,207] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,208] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,208] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,209] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,209] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,210] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,210] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:17:45,211] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:45,213] jepsen worker 69 - jepsen.util 789 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,214] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,214] jepsen worker 33 - jepsen.util 753 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,215] jepsen worker 33 - jepsen.util 753 :invoke :write [11 7] INFO [2022-03-13 17:17:45,216] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,217] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,225] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,225] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,227] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,227] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,227] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:17:45,227] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,229] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,229] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,232] jepsen worker 45 - jepsen.util 4095 :info :write [15 49] :SyncWriteAmbiguous INFO [2022-03-13 17:17:45,232] jepsen worker 78 - jepsen.util 708 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,233] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,234] jepsen worker 45 - jepsen.util 4185 :invoke :write [15 27] INFO [2022-03-13 17:17:45,235] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:45,235] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:17:45,270] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,271] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:17:45,293] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,293] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,293] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:17:45,294] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:17:45,307] jepsen worker 34 - jepsen.util 934 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,308] jepsen worker 34 - jepsen.util 934 :invoke :write [11 9] INFO [2022-03-13 17:17:45,312] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,312] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 7] INFO [2022-03-13 17:17:45,344] jepsen worker 36 - jepsen.util 2466 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,344] jepsen worker 70 - jepsen.util 880 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,345] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:45,345] jepsen worker 70 - jepsen.util 880 :invoke :write [23 39] INFO [2022-03-13 17:17:45,362] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,362] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:17:45,368] jepsen worker 18 - jepsen.util 3978 :info :write [6 13] :RequestTimeoutException INFO [2022-03-13 17:17:45,369] jepsen worker 18 - jepsen.util 4068 :invoke :read [6 nil] INFO [2022-03-13 17:17:45,379] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,380] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:45,409] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,409] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,410] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,410] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:45,414] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,414] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,415] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:17:45,415] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:17:45,423] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,424] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,425] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,425] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,426] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,427] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,427] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,428] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:45,428] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,429] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:17:45,429] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,430] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:17:45,435] jepsen worker 14 - jepsen.util 374 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,435] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:45,436] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,437] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:17:45,448] jepsen worker 61 - jepsen.util 781 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,450] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:45,484] jepsen worker 87 - jepsen.util 2877 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,484] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:45,485] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,486] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:17:45,493] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,498] jepsen worker 27 - jepsen.util 4077 :info :write [9 24] :RequestTimeoutException INFO [2022-03-13 17:17:45,498] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,499] jepsen worker 27 - jepsen.util 4167 :invoke :write [9 0] INFO [2022-03-13 17:17:45,508] jepsen worker 47 - jepsen.util 3917 :ok :read [15 48] INFO [2022-03-13 17:17:45,509] jepsen worker 47 - jepsen.util 3917 :invoke :write [15 9] INFO [2022-03-13 17:17:45,524] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,525] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,527] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,527] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:17:45,535] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,535] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,546] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,547] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:45,556] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,557] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:45,579] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,579] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,580] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:17:45,581] jepsen worker 69 - jepsen.util 789 :invoke :write [23 30] INFO [2022-03-13 17:17:45,591] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,592] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] INFO [2022-03-13 17:17:45,597] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,597] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,597] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:17:45,597] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 8] INFO [2022-03-13 17:17:45,635] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,636] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,638] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,638] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:17:45,651] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,652] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:17:45,653] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,654] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:45,655] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,655] jepsen worker 24 - jepsen.util 654 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,656] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:17:45,656] jepsen worker 24 - jepsen.util 654 :invoke :write [8 8] INFO [2022-03-13 17:17:45,664] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,664] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,664] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,664] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,664] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:45,665] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:17:45,665] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:45,666] jepsen worker 12 - jepsen.util 912 :invoke :write [4 23] INFO [2022-03-13 17:17:45,689] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,689] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:17:45,716] jepsen worker 81 - jepsen.util 4581 :ok :read [27 4] INFO [2022-03-13 17:17:45,718] jepsen worker 81 - jepsen.util 4581 :invoke :read [27 nil] INFO [2022-03-13 17:17:45,726] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,726] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 6] INFO [2022-03-13 17:17:45,734] jepsen worker 79 - jepsen.util 1159 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,734] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,735] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:45,735] jepsen worker 23 - jepsen.util 563 :invoke :write [7 11] INFO [2022-03-13 17:17:45,739] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,740] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:45,763] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,764] jepsen worker 35 - jepsen.util 935 :invoke :write [11 31] INFO [2022-03-13 17:17:45,789] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,790] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 18] INFO [2022-03-13 17:17:45,791] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,793] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:45,794] jepsen worker 81 - jepsen.util 4581 :ok :read [27 4] INFO [2022-03-13 17:17:45,794] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,796] jepsen worker 81 - jepsen.util 4581 :invoke :read [27 nil] INFO [2022-03-13 17:17:45,797] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:17:45,799] jepsen worker 4 - jepsen.util 2524 :info :write [1 27] :SyncWriteAmbiguous INFO [2022-03-13 17:17:45,800] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:17:45,804] jepsen worker 69 - jepsen.util 789 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,804] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,804] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:17:45,805] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:17:45,815] jepsen worker 3 - jepsen.util 2253 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,816] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:17:45,868] jepsen worker 81 - jepsen.util 4581 :ok :read [27 4] INFO [2022-03-13 17:17:45,868] jepsen worker 81 - jepsen.util 4581 :invoke :read [27 nil] INFO [2022-03-13 17:17:45,890] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,890] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,890] jepsen worker 51 - jepsen.util 771 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,891] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,891] jepsen worker 80 - jepsen.util 800 :invoke :write [26 25] INFO [2022-03-13 17:17:45,892] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:17:45,892] jepsen worker 51 - jepsen.util 771 :invoke :write [17 18] INFO [2022-03-13 17:17:45,892] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:45,906] jepsen worker 8 - jepsen.util 728 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,907] jepsen worker 8 - jepsen.util 728 :invoke :write [2 12] INFO [2022-03-13 17:17:45,909] jepsen worker 81 - jepsen.util 4581 :ok :read [27 4] INFO [2022-03-13 17:17:45,910] jepsen worker 81 - jepsen.util 4581 :invoke :write [27 11] INFO [2022-03-13 17:17:45,933] jepsen worker 34 - jepsen.util 934 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,933] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:45,934] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,935] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:45,949] jepsen worker 36 - jepsen.util 2466 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,950] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:17:45,988] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,989] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:45,989] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:45,989] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,006] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,007] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:17:46,015] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,015] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,015] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,015] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,015] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:46,016] jepsen worker 78 - jepsen.util 708 :invoke :write [26 10] INFO [2022-03-13 17:17:46,017] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:46,017] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:46,018] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,019] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,034] jepsen worker 38 - jepsen.util 2828 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,036] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,050] jepsen worker 76 - jepsen.util 796 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,051] jepsen worker 76 - jepsen.util 796 :invoke :write [25 26] INFO [2022-03-13 17:17:46,060] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,061] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:17:46,086] jepsen worker 11 - jepsen.util 3881 :info :write [3 43] :RequestTimeoutException INFO [2022-03-13 17:17:46,087] jepsen worker 11 - jepsen.util 3971 :invoke :write [3 13] INFO [2022-03-13 17:17:46,092] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,092] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,092] jepsen worker 33 - jepsen.util 753 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,092] jepsen worker 33 - jepsen.util 753 :invoke :write [11 48] INFO [2022-03-13 17:17:46,093] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:17:46,101] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,102] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,103] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,104] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:17:46,106] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:17:46,131] jepsen worker 37 - jepsen.util 2467 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,131] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,133] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,134] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 38] INFO [2022-03-13 17:17:46,136] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,137] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,138] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,139] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:17:46,152] jepsen worker 52 - jepsen.util 682 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,152] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:46,153] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,153] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,153] jepsen worker 43 - jepsen.util 673 :invoke :write [14 28] INFO [2022-03-13 17:17:46,154] jepsen worker 53 - jepsen.util 593 :invoke :write [17 44] INFO [2022-03-13 17:17:46,154] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:46,155] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:46,164] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7953216374269005}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5823529411764705}} INFO [2022-03-13 17:17:46,165] jepsen worker nemesis - couchbase.util Rebalance check count: 573 INFO [2022-03-13 17:17:46,165] jepsen worker nemesis - couchbase.util Rebalance stuck count: 453 INFO [2022-03-13 17:17:46,166] jepsen worker 44 - jepsen.util 674 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,168] jepsen worker 44 - jepsen.util 674 :invoke :write [14 1] INFO [2022-03-13 17:17:46,170] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,170] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,170] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,171] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,171] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,172] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:17:46,173] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,174] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,175] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,175] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,176] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,177] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,178] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,178] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,188] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,189] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,189] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,189] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 9] INFO [2022-03-13 17:17:46,190] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,191] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,192] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,192] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,193] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,194] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:17:46,204] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,205] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,209] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,210] jepsen worker 2 - jepsen.util 4232 :info :write [0 18] :RequestTimeoutException INFO [2022-03-13 17:17:46,210] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,211] jepsen worker 2 - jepsen.util 4322 :invoke :read [0 nil] INFO [2022-03-13 17:17:46,212] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,212] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:17:46,250] jepsen worker 85 - jepsen.util 3685 :info :write [28 1] :RequestTimeoutException INFO [2022-03-13 17:17:46,250] jepsen worker 85 - jepsen.util 3775 :invoke :read [28 nil] INFO [2022-03-13 17:17:46,255] jepsen worker 24 - jepsen.util 654 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,256] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:46,267] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,268] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,268] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:17:46,269] jepsen worker 71 - jepsen.util 611 :invoke :write [23 39] INFO [2022-03-13 17:17:46,270] jepsen worker 7 - jepsen.util 637 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,270] jepsen worker 88 - jepsen.util 2428 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,272] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,272] jepsen worker 7 - jepsen.util 637 :invoke :write [2 20] INFO [2022-03-13 17:17:46,277] jepsen worker 89 - jepsen.util 2789 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,277] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:17:46,303] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,304] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:17:46,317] jepsen worker 23 - jepsen.util 563 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,318] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,318] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,318] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,319] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:46,319] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 39] INFO [2022-03-13 17:17:46,320] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:46,321] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:46,329] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,329] jepsen worker 35 - jepsen.util 935 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,330] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:46,330] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:46,377] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,378] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,389] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,389] jepsen worker 69 - jepsen.util 789 :invoke :write [23 0] INFO [2022-03-13 17:17:46,391] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,391] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,392] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,392] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 29] INFO [2022-03-13 17:17:46,393] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,393] jepsen worker 13 - jepsen.util 733 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,394] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:46,394] jepsen worker 13 - jepsen.util 733 :invoke :write [4 43] INFO [2022-03-13 17:17:46,404] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,404] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,405] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,406] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,407] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,407] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:17:46,458] jepsen worker 51 - jepsen.util 771 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,459] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:46,460] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,460] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,460] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:17:46,461] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:17:46,463] jepsen worker 70 - jepsen.util 880 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,463] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,463] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,464] jepsen worker 70 - jepsen.util 880 :invoke :write [23 42] INFO [2022-03-13 17:17:46,464] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,464] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,465] jepsen worker 62 - jepsen.util 512 :invoke :write [20 10] INFO [2022-03-13 17:17:46,465] jepsen worker 6 - jepsen.util 636 :invoke :write [2 31] INFO [2022-03-13 17:17:46,466] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 13] INFO [2022-03-13 17:17:46,474] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,485] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,485] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,485] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,486] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,500] jepsen worker 60 - jepsen.util 690 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,500] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:46,507] jepsen worker 71 - jepsen.util 611 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,508] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,513] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,514] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,514] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,513] jepsen worker 78 - jepsen.util 708 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,514] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:46,515] jepsen worker 61 - jepsen.util 781 :invoke :write [20 35] INFO [2022-03-13 17:17:46,515] jepsen worker 23 - jepsen.util 563 :invoke :write [7 8] INFO [2022-03-13 17:17:46,516] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:46,531] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,531] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,532] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,533] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,534] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,540] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,542] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,543] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:17:46,561] jepsen worker 3 - jepsen.util 2253 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,561] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,561] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,561] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,562] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,562] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:17:46,565] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,565] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:46,566] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,567] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:17:46,568] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,569] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:17:46,575] jepsen worker 80 - jepsen.util 800 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,576] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,576] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:46,580] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,624] jepsen worker 76 - jepsen.util 796 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,624] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,624] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,625] jepsen worker 42 - jepsen.util 402 :invoke :write [14 21] INFO [2022-03-13 17:17:46,625] jepsen worker 76 - jepsen.util 796 :invoke :write [25 4] INFO [2022-03-13 17:17:46,626] jepsen worker 75 - jepsen.util 525 :invoke :write [25 1] INFO [2022-03-13 17:17:46,631] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,631] jepsen worker 69 - jepsen.util 789 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,631] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,631] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:17:46,632] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,632] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,633] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,634] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:17:46,643] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,643] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,643] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,644] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,645] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,645] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,646] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,647] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:46,648] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,648] jepsen worker 71 - jepsen.util 611 :invoke :write [23 32] INFO [2022-03-13 17:17:46,649] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,649] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,648] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,649] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,650] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:46,651] jepsen worker 52 - jepsen.util 682 :invoke :write [17 0] INFO [2022-03-13 17:17:46,651] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:46,652] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:17:46,661] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,662] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:17:46,671] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,672] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:17:46,701] jepsen worker 19 - jepsen.util 3799 :info :write [6 23] :SyncWriteAmbiguous INFO [2022-03-13 17:17:46,702] jepsen worker 19 - jepsen.util 3889 :invoke :write [6 36] INFO [2022-03-13 17:17:46,708] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,708] jepsen worker 37 - jepsen.util 2467 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,708] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,709] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 10] INFO [2022-03-13 17:17:46,718] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,718] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,719] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 25] INFO [2022-03-13 17:17:46,719] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:17:46,748] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,749] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:46,809] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,810] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:17:46,813] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,814] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:46,820] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,821] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] WARN [2022-03-13 17:17:46,850] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8952909,"last_local_id":"A312307A00000001/00000000E3CF0611","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":11109998,"last_local_socket":"172.23.104.79:33508","total_server_duration_us":273860,"encode_duration_us":17472,"operation_id":"0x101c1","timeout_ms":10000,"last_server_duration_us":51828,"total_duration_us":10004313},{"operation_name":"upsert","last_dispatch_duration_us":14379372,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14379372,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":3182097,"encode_duration_us":17507,"operation_id":"0xffe7","timeout_ms":10000,"last_server_duration_us":3182097,"total_duration_us":10005309},{"operation_name":"upsert","last_dispatch_duration_us":18114402,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18114402,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9044797,"encode_duration_us":17956,"operation_id":"0xfd7a","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10006446},{"operation_name":"upsert","last_dispatch_duration_us":9055183,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10847489,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9381986,"encode_duration_us":18078,"operation_id":"0xff67","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10010031}],"total_count":4}} WARN [2022-03-13 17:17:46,850] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12251894,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12251894,"total_server_duration_us":27,"operation_id":"0x10078","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":12256554},{"operation_name":"get","last_dispatch_duration_us":12216490,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12216490,"total_server_duration_us":15,"operation_id":"0x10079","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":12219466},{"operation_name":"get","last_dispatch_duration_us":10364463,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10364463,"total_server_duration_us":100,"operation_id":"0x10022","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":10366889},{"operation_name":"upsert","last_dispatch_duration_us":12224678,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12224678,"total_server_duration_us":19730,"encode_duration_us":33600,"operation_id":"0x10076","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10009365},{"operation_name":"upsert","last_dispatch_duration_us":12242685,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":12242685,"total_server_duration_us":19108,"encode_duration_us":26816,"operation_id":"0x10077","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10009234},{"operation_name":"upsert","last_dispatch_duration_us":14379372,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":14379372,"total_server_duration_us":3182097,"encode_duration_us":17507,"operation_id":"0xffe7","timeout_ms":10000,"last_server_duration_us":3182097,"total_duration_us":10005309},{"operation_name":"upsert","last_dispatch_duration_us":8952909,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":11109998,"total_server_duration_us":273860,"encode_duration_us":17472,"operation_id":"0x101c1","timeout_ms":10000,"last_server_duration_us":51828,"total_duration_us":10004313},{"operation_name":"upsert","last_dispatch_duration_us":39621,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6332704,"total_server_duration_us":276111,"encode_duration_us":19515,"operation_id":"0x100e6","timeout_ms":10000,"last_server_duration_us":26596,"total_duration_us":10003502},{"operation_name":"upsert","last_dispatch_duration_us":9602216,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9893327,"total_server_duration_us":9141515,"encode_duration_us":18487,"operation_id":"0x1021e","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":9905607},{"operation_name":"upsert","last_dispatch_duration_us":9092212,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9092212,"total_server_duration_us":9025695,"encode_duration_us":18660,"operation_id":"0x102b8","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9097345}],"total_count":104}} INFO [2022-03-13 17:17:46,886] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,886] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:17:46,924] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,924] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:17:46,946] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,947] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,947] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,947] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 18] INFO [2022-03-13 17:17:46,949] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,950] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,951] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,952] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,953] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,954] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:46,955] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:46,956] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:17:47,049] jepsen worker 37 - jepsen.util 2467 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,050] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:17:47,089] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,089] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,090] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:17:47,090] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:17:47,102] jepsen worker 38 - jepsen.util 2828 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,103] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:47,108] jepsen worker 72 - jepsen.util 4032 :info :write [24 20] :RequestTimeoutException INFO [2022-03-13 17:17:47,108] jepsen worker 72 - jepsen.util 4122 :invoke :read [24 nil] INFO [2022-03-13 17:17:47,159] jepsen worker 64 - jepsen.util 4024 :info :write [21 29] :RequestTimeoutException INFO [2022-03-13 17:17:47,160] jepsen worker 64 - jepsen.util 4114 :invoke :write [21 16] INFO [2022-03-13 17:17:47,170] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7953216374269005}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5823529411764705}} INFO [2022-03-13 17:17:47,170] jepsen worker nemesis - couchbase.util Rebalance check count: 574 INFO [2022-03-13 17:17:47,170] jepsen worker nemesis - couchbase.util Rebalance stuck count: 454 INFO [2022-03-13 17:17:47,171] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,171] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:17:47,217] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,218] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:17:47,223] jepsen worker 73 - jepsen.util 4123 :info :write [24 32] :RequestTimeoutException INFO [2022-03-13 17:17:47,224] jepsen worker 73 - jepsen.util 4213 :invoke :write [24 34] INFO [2022-03-13 17:17:47,228] jepsen worker 54 - jepsen.util 3564 :info :write [18 9] :SyncWriteAmbiguous INFO [2022-03-13 17:17:47,229] jepsen worker 54 - jepsen.util 3654 :invoke :read [18 nil] INFO [2022-03-13 17:17:47,261] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,261] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,267] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:17:47,268] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 40] INFO [2022-03-13 17:17:47,287] jepsen worker 54 - jepsen.util 3654 :ok :read [18 21] INFO [2022-03-13 17:17:47,288] jepsen worker 54 - jepsen.util 3654 :invoke :write [18 46] INFO [2022-03-13 17:17:47,290] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,291] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:17:47,296] jepsen worker 10 - jepsen.util 4060 :info :write [3 14] :RequestTimeoutException INFO [2022-03-13 17:17:47,296] jepsen worker 10 - jepsen.util 4150 :invoke :write [3 11] INFO [2022-03-13 17:17:47,300] jepsen worker 13 - jepsen.util 733 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,301] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:47,305] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,305] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:47,316] jepsen worker 79 - jepsen.util 1159 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,317] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:47,319] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,319] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,320] jepsen worker 78 - jepsen.util 708 :invoke :write [26 6] INFO [2022-03-13 17:17:47,324] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:17:47,325] jepsen worker 43 - jepsen.util 673 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,326] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:47,384] jepsen worker 38 - jepsen.util 2828 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,384] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:17:47,386] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,387] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,387] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:17:47,395] jepsen worker 71 - jepsen.util 611 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,395] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:47,397] jepsen worker 76 - jepsen.util 796 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,397] jepsen worker 76 - jepsen.util 796 :invoke :write [25 34] INFO [2022-03-13 17:17:47,398] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:47,403] jepsen worker 88 - jepsen.util 2428 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,404] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:17:47,494] jepsen worker 53 - jepsen.util 593 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,494] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,494] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,494] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,495] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:47,495] jepsen worker 53 - jepsen.util 593 :invoke :write [17 45] INFO [2022-03-13 17:17:47,496] jepsen worker 22 - jepsen.util 652 :invoke :write [7 9] INFO [2022-03-13 17:17:47,497] jepsen worker 26 - jepsen.util 566 :invoke :write [8 14] INFO [2022-03-13 17:17:47,513] jepsen worker 12 - jepsen.util 912 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,514] jepsen worker 12 - jepsen.util 912 :invoke :write [4 32] INFO [2022-03-13 17:17:47,525] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,525] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,526] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:47,526] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:47,541] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,542] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:17:47,544] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,545] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:47,557] jepsen worker 33 - jepsen.util 753 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,557] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,558] jepsen worker 51 - jepsen.util 771 :invoke :write [17 3] INFO [2022-03-13 17:17:47,559] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:47,577] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,577] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,577] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:17:47,578] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:47,579] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,580] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:47,586] jepsen worker 63 - jepsen.util 4113 :info :write [21 1] :SyncWriteAmbiguous INFO [2022-03-13 17:17:47,587] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,587] jepsen worker 63 - jepsen.util 4203 :invoke :read [21 nil] INFO [2022-03-13 17:17:47,588] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:47,590] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,591] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:17:47,641] jepsen worker 23 - jepsen.util 563 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,641] jepsen worker 23 - jepsen.util 563 :invoke :write [7 43] INFO [2022-03-13 17:17:47,661] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,661] jepsen worker 75 - jepsen.util 525 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,661] jepsen worker 75 - jepsen.util 525 :invoke :write [25 12] INFO [2022-03-13 17:17:47,663] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,663] jepsen worker 13 - jepsen.util 733 :invoke :write [4 38] INFO [2022-03-13 17:17:47,665] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 11] INFO [2022-03-13 17:17:47,689] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,689] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,690] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:47,690] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:47,691] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,692] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:47,692] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,693] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,693] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:17:47,693] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,693] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:17:47,695] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:17:47,695] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,696] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:47,697] jepsen worker 44 - jepsen.util 674 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,697] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,697] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,697] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,697] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,697] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,697] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,698] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,698] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,698] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:47,699] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:17:47,699] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,699] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,700] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,701] jepsen worker 80 - jepsen.util 800 :invoke :write [26 40] INFO [2022-03-13 17:17:47,702] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:17:47,702] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:47,703] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:47,704] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,704] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:17:47,705] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:17:47,705] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:47,706] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:47,706] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:17:47,707] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,707] jepsen worker 33 - jepsen.util 753 :invoke :write [11 41] INFO [2022-03-13 17:17:47,707] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:47,708] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,710] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:17:47,710] jepsen worker 44 - jepsen.util 674 :invoke :write [14 8] INFO [2022-03-13 17:17:47,723] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,724] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:47,777] jepsen worker 52 - jepsen.util 682 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,778] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:47,778] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,778] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:47,779] jepsen worker 43 - jepsen.util 673 :invoke :write [14 16] INFO [2022-03-13 17:17:47,779] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:47,804] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,804] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:17:47,811] jepsen worker 8 - jepsen.util 728 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,812] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:47,815] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,815] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:47,819] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,820] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:47,826] jepsen worker 51 - jepsen.util 771 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,826] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,827] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,827] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:47,828] jepsen worker 24 - jepsen.util 654 :invoke :write [8 49] INFO [2022-03-13 17:17:47,828] jepsen worker 34 - jepsen.util 934 :invoke :write [11 21] INFO [2022-03-13 17:17:47,830] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,835] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:17:47,846] jepsen worker 28 - jepsen.util 3988 :info :write [9 0] :SyncWriteAmbiguous INFO [2022-03-13 17:17:47,847] jepsen worker 28 - jepsen.util 4078 :invoke :write [9 12] INFO [2022-03-13 17:17:47,859] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,860] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,860] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:47,860] jepsen worker 69 - jepsen.util 789 :invoke :write [23 39] INFO [2022-03-13 17:17:47,866] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,867] jepsen worker 71 - jepsen.util 611 :invoke :write [23 38] INFO [2022-03-13 17:17:47,878] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,878] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,878] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:17:47,879] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:17:47,884] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,885] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 45] INFO [2022-03-13 17:17:47,886] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,887] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:47,888] jepsen worker 61 - jepsen.util 781 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,889] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:47,906] jepsen worker 76 - jepsen.util 796 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,907] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:47,912] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,912] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:47,928] jepsen worker 80 - jepsen.util 800 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,928] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:47,972] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,973] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:47,977] jepsen worker 53 - jepsen.util 593 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,977] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:47,987] jepsen worker 69 - jepsen.util 789 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:47,988] jepsen worker 69 - jepsen.util 789 :invoke :write [23 23] INFO [2022-03-13 17:17:48,003] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,003] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,021] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,021] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,021] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,022] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,022] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:17:48,023] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:48,023] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,024] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,024] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,025] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:48,026] jepsen worker 7 - jepsen.util 637 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,027] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,027] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:48,028] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,028] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,029] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:17:48,030] jepsen worker 33 - jepsen.util 753 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,030] jepsen worker 33 - jepsen.util 753 :invoke :write [11 4] INFO [2022-03-13 17:17:48,032] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,033] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,034] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,035] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:17:48,075] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,075] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,076] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,076] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,077] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,077] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 2] INFO [2022-03-13 17:17:48,078] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,078] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,079] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,080] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:17:48,096] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,097] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:17:48,097] jepsen worker 29 - jepsen.util 4349 :info :write [9 11] :RequestTimeoutException INFO [2022-03-13 17:17:48,098] jepsen worker 29 - jepsen.util 4439 :invoke :write [9 45] INFO [2022-03-13 17:17:48,104] jepsen worker 37 - jepsen.util 2467 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,104] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,105] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,106] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:17:48,106] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,106] jepsen worker 30 - jepsen.util 3810 :info :write [10 18] :RequestTimeoutException INFO [2022-03-13 17:17:48,107] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:17:48,108] jepsen worker 30 - jepsen.util 3900 :invoke :write [10 47] INFO [2022-03-13 17:17:48,127] jepsen worker 20 - jepsen.util 4160 :info :write [6 42] :RequestTimeoutException INFO [2022-03-13 17:17:48,128] jepsen worker 20 - jepsen.util 4250 :invoke :write [6 0] INFO [2022-03-13 17:17:48,168] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,169] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:17:48,172] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,173] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,175] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.7953216374269005}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5823529411764705}} INFO [2022-03-13 17:17:48,175] jepsen worker nemesis - couchbase.util Rebalance check count: 575 INFO [2022-03-13 17:17:48,175] jepsen worker nemesis - couchbase.util Rebalance stuck count: 455 INFO [2022-03-13 17:17:48,186] jepsen worker 46 - jepsen.util 4096 :info :write [15 39] :RequestTimeoutException INFO [2022-03-13 17:17:48,187] jepsen worker 46 - jepsen.util 4186 :invoke :read [15 nil] INFO [2022-03-13 17:17:48,200] jepsen worker 88 - jepsen.util 2428 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,200] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,201] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,201] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,206] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,206] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,206] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:17:48,207] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 36] WARN [2022-03-13 17:17:48,254] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17362476,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17859329,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9276294,"encode_duration_us":16991,"operation_id":"0xfde3","timeout_ms":10000,"last_server_duration_us":9071357,"total_duration_us":10008863},{"operation_name":"upsert","last_dispatch_duration_us":9191733,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17627907,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9099927,"encode_duration_us":17659,"operation_id":"0xff9c","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10010497}],"total_count":2}} INFO [2022-03-13 17:17:48,273] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,273] jepsen worker 89 - jepsen.util 2789 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,273] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,273] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,276] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,276] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,277] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:17:48,277] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,278] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,278] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,279] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,280] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:17:48,292] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,292] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,297] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,297] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 15] INFO [2022-03-13 17:17:48,303] jepsen worker 71 - jepsen.util 611 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,304] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:48,320] jepsen worker 78 - jepsen.util 708 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,320] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:48,353] jepsen worker 13 - jepsen.util 733 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,353] jepsen worker 13 - jepsen.util 733 :invoke :write [4 41] INFO [2022-03-13 17:17:48,364] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,365] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:17:48,378] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,378] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] INFO [2022-03-13 17:17:48,382] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,383] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:17:48,403] jepsen worker 70 - jepsen.util 880 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,404] jepsen worker 70 - jepsen.util 880 :invoke :write [23 42] INFO [2022-03-13 17:17:48,414] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,415] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:17:48,421] jepsen worker 69 - jepsen.util 789 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,421] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,421] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:48,422] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:48,423] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,423] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,423] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:48,424] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:17:48,432] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,432] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,432] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:48,436] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:17:48,438] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,438] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:48,439] jepsen worker 26 - jepsen.util 566 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,440] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,440] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:48,441] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:48,442] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,442] jepsen worker 69 - jepsen.util 789 :invoke :write [23 43] WARN [2022-03-13 17:17:48,448] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9307404,"last_local_id":"A312307A00000004/0000000095638474","last_remote_socket":"172.23.107.227:11210","total_dispatch_duration_us":9886917,"last_local_socket":"172.23.104.79:33498","total_server_duration_us":9496297,"encode_duration_us":17443,"operation_id":"0x101b7","timeout_ms":10000,"last_server_duration_us":9119248,"total_duration_us":10003049}],"total_count":1}} INFO [2022-03-13 17:17:48,452] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,452] jepsen worker 77 - jepsen.util 707 :invoke :write [25 16] WARN [2022-03-13 17:17:48,458] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":8359259,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":15512254,"total_server_duration_us":4582,"operation_id":"0xfabc","timeout_ms":30000,"last_server_duration_us":2352,"total_duration_us":30008168},{"operation_name":"upsert","last_dispatch_duration_us":7724816,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8796767,"total_server_duration_us":279968,"encode_duration_us":17767,"operation_id":"0x101d8","timeout_ms":10000,"last_server_duration_us":23983,"total_duration_us":10008119},{"operation_name":"upsert","last_dispatch_duration_us":10754162,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10754162,"total_server_duration_us":35011,"encode_duration_us":16471,"operation_id":"0x1003d","timeout_ms":10000,"last_server_duration_us":35011,"total_duration_us":10006224},{"operation_name":"upsert","last_dispatch_duration_us":40293,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":279134,"total_server_duration_us":161992,"encode_duration_us":19369,"operation_id":"0x1027e","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10005911},{"operation_name":"upsert","last_dispatch_duration_us":98543,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":6909595,"total_server_duration_us":129780,"encode_duration_us":36322,"operation_id":"0x10162","timeout_ms":10000,"last_server_duration_us":36017,"total_duration_us":10005904},{"operation_name":"upsert","last_dispatch_duration_us":7772859,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9280473,"total_server_duration_us":288750,"encode_duration_us":18805,"operation_id":"0x101c3","timeout_ms":10000,"last_server_duration_us":24411,"total_duration_us":10005453},{"operation_name":"upsert","last_dispatch_duration_us":71906,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7387454,"total_server_duration_us":40653,"encode_duration_us":17117,"operation_id":"0x102e1","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10001879},{"operation_name":"upsert","last_dispatch_duration_us":7745047,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8737810,"total_server_duration_us":289410,"encode_duration_us":21834,"operation_id":"0x101e8","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10001788},{"operation_name":"upsert","last_dispatch_duration_us":9292192,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":9490409,"total_server_duration_us":9206195,"encode_duration_us":17934,"operation_id":"0x10209","timeout_ms":10000,"last_server_duration_us":9139501,"total_duration_us":9506098},{"operation_name":"upsert","last_dispatch_duration_us":9128799,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9128799,"total_server_duration_us":9058604,"encode_duration_us":18624,"operation_id":"0x1022e","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":9132551}],"total_count":60}} INFO [2022-03-13 17:17:48,516] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,517] jepsen worker 35 - jepsen.util 935 :invoke :write [11 9] INFO [2022-03-13 17:17:48,530] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,530] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,530] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:48,531] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] WARN [2022-03-13 17:17:48,548] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17933176,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19270880,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9305217,"encode_duration_us":17960,"operation_id":"0xfcbc","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":10002646},{"operation_name":"upsert","last_dispatch_duration_us":18021281,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":22023777,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9255527,"encode_duration_us":17383,"operation_id":"0xfdd8","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10007171}],"total_count":2}} WARN [2022-03-13 17:17:48,548] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16270070,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":16270070,"total_server_duration_us":5813,"operation_id":"0xffb5","timeout_ms":30000,"last_server_duration_us":5813,"total_duration_us":16273349},{"operation_name":"get","last_dispatch_duration_us":14721303,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14721303,"total_server_duration_us":32,"operation_id":"0xff06","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":14723734},{"operation_name":"get","last_dispatch_duration_us":13620518,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":13620518,"total_server_duration_us":19,"operation_id":"0xff57","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":13622879},{"operation_name":"get","last_dispatch_duration_us":11637321,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":11637321,"total_server_duration_us":8557,"operation_id":"0x100fa","timeout_ms":30000,"last_server_duration_us":8557,"total_duration_us":11641335},{"operation_name":"upsert","last_dispatch_duration_us":11038856,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11038856,"total_server_duration_us":23729,"encode_duration_us":17609,"operation_id":"0x10016","timeout_ms":10000,"last_server_duration_us":23729,"total_duration_us":10012164},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":25648,"operation_id":"0x1022c","timeout_ms":10000,"total_duration_us":10008610},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":21810,"operation_id":"0x10293","timeout_ms":10000,"total_duration_us":10008176},{"operation_name":"get","last_dispatch_duration_us":8946621,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8946621,"total_server_duration_us":174904,"operation_id":"0x100ac","timeout_ms":30000,"last_server_duration_us":174904,"total_duration_us":8949411},{"operation_name":"upsert","last_dispatch_duration_us":52194,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1197671,"total_server_duration_us":601734,"encode_duration_us":18902,"operation_id":"0x101bd","timeout_ms":10000,"last_server_duration_us":41660,"total_duration_us":5910874},{"operation_name":"upsert","last_dispatch_duration_us":140302,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1462925,"total_server_duration_us":559872,"encode_duration_us":17126,"operation_id":"0x101e0","timeout_ms":10000,"last_server_duration_us":45167,"total_duration_us":5679628}],"total_count":16}} WARN [2022-03-13 17:17:48,548] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15275171,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":15275171,"total_server_duration_us":43,"operation_id":"0x10006","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":15278117},{"operation_name":"upsert","last_dispatch_duration_us":50546,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9663152,"total_server_duration_us":224806,"encode_duration_us":31901,"operation_id":"0x1023a","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10009799},{"operation_name":"upsert","last_dispatch_duration_us":8673804,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":13244132,"total_server_duration_us":205930,"encode_duration_us":16903,"operation_id":"0x100f4","timeout_ms":10000,"last_server_duration_us":23390,"total_duration_us":10007812},{"operation_name":"upsert","last_dispatch_duration_us":6760380,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":6760380,"total_server_duration_us":21001,"encode_duration_us":18406,"operation_id":"0x10281","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10006569},{"operation_name":"upsert","last_dispatch_duration_us":49944,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9644088,"total_server_duration_us":223749,"encode_duration_us":24138,"operation_id":"0x10238","timeout_ms":10000,"last_server_duration_us":24841,"total_duration_us":10005623},{"operation_name":"upsert","last_dispatch_duration_us":8982515,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8982515,"total_server_duration_us":34215,"encode_duration_us":19036,"operation_id":"0x101c2","timeout_ms":10000,"last_server_duration_us":34215,"total_duration_us":10003385},{"operation_name":"upsert","last_dispatch_duration_us":9084423,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9084423,"total_server_duration_us":9041612,"encode_duration_us":19842,"operation_id":"0x1022b","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9093739},{"operation_name":"upsert","last_dispatch_duration_us":9080777,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":9080777,"total_server_duration_us":113463,"encode_duration_us":18496,"operation_id":"0x1013a","timeout_ms":10000,"last_server_duration_us":113463,"total_duration_us":9089669},{"operation_name":"upsert","last_dispatch_duration_us":8626914,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":8626914,"total_server_duration_us":58501,"encode_duration_us":18074,"operation_id":"0x10176","timeout_ms":10000,"last_server_duration_us":58501,"total_duration_us":8635293},{"operation_name":"get","last_dispatch_duration_us":7123758,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7123758,"total_server_duration_us":394005,"operation_id":"0x10304","timeout_ms":30000,"last_server_duration_us":394005,"total_duration_us":7126004}],"total_count":40}} INFO [2022-03-13 17:17:48,570] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,571] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,586] jepsen worker 37 - jepsen.util 2467 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,586] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 1] INFO [2022-03-13 17:17:48,600] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,601] jepsen worker 71 - jepsen.util 611 :invoke :write [23 5] INFO [2022-03-13 17:17:48,607] jepsen worker 62 - jepsen.util 512 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,608] jepsen worker 62 - jepsen.util 512 :invoke :write [20 43] WARN [2022-03-13 17:17:48,649] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17122646,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17662685,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9182222,"encode_duration_us":18690,"operation_id":"0xff59","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10001832}],"total_count":1}} WARN [2022-03-13 17:17:48,649] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12418898,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12418898,"total_server_duration_us":56,"operation_id":"0xffdb","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":12422132},{"operation_name":"get","last_dispatch_duration_us":11461462,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":11461462,"total_server_duration_us":32,"operation_id":"0x1001d","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":11467256},{"operation_name":"upsert","last_dispatch_duration_us":8981634,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9477868,"total_server_duration_us":83610,"encode_duration_us":18161,"operation_id":"0x102e2","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10010864},{"operation_name":"upsert","last_dispatch_duration_us":8981430,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9446721,"total_server_duration_us":72190,"encode_duration_us":25497,"operation_id":"0x102e6","timeout_ms":10000,"last_server_duration_us":27945,"total_duration_us":10006414},{"operation_name":"upsert","last_dispatch_duration_us":8940959,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12853685,"total_server_duration_us":114178,"encode_duration_us":17879,"operation_id":"0x101bc","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10005964},{"operation_name":"upsert","last_dispatch_duration_us":280128,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":651695,"total_server_duration_us":60493,"encode_duration_us":20648,"operation_id":"0x102e4","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10005627},{"operation_name":"upsert","last_dispatch_duration_us":9278276,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9278276,"total_server_duration_us":9024634,"encode_duration_us":20500,"operation_id":"0x102ee","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":9282160},{"operation_name":"upsert","last_dispatch_duration_us":6045701,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6045701,"total_server_duration_us":120988,"encode_duration_us":21723,"operation_id":"0x1023f","timeout_ms":10000,"last_server_duration_us":120988,"total_duration_us":6050948},{"operation_name":"upsert","last_dispatch_duration_us":4816521,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4816521,"total_server_duration_us":84429,"encode_duration_us":18450,"operation_id":"0x102c6","timeout_ms":10000,"last_server_duration_us":84429,"total_duration_us":4820552},{"operation_name":"upsert","last_dispatch_duration_us":3939351,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3939351,"total_server_duration_us":64473,"encode_duration_us":18088,"operation_id":"0x1023e","timeout_ms":10000,"last_server_duration_us":64473,"total_duration_us":3944950}],"total_count":25}} INFO [2022-03-13 17:17:48,649] jepsen worker 75 - jepsen.util 525 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,649] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:17:48,649] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15848262,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16491906,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9351002,"encode_duration_us":19382,"operation_id":"0xfddb","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10003821},{"operation_name":"upsert","last_dispatch_duration_us":15929450,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":23303336,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9310063,"encode_duration_us":17499,"operation_id":"0xfb51","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10007623}],"total_count":2}} INFO [2022-03-13 17:17:48,650] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:48,650] jepsen worker 75 - jepsen.util 525 :invoke :write [25 9] INFO [2022-03-13 17:17:48,655] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,656] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,656] jepsen worker 5 - jepsen.util 2705 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,656] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:17:48,659] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:17:48,660] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:17:48,673] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,673] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,673] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,674] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:17:48,675] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:17:48,675] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:48,682] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,683] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:48,686] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,687] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:17:48,716] jepsen worker 69 - jepsen.util 789 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,717] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:17:48,719] jepsen worker 23 - jepsen.util 563 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,719] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,720] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,720] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,720] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:48,721] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:48,722] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:48,722] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] WARN [2022-03-13 17:17:48,749] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8106003,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8277351,"total_server_duration_us":68999,"encode_duration_us":18018,"operation_id":"0x100f5","timeout_ms":10000,"last_server_duration_us":24669,"total_duration_us":10010341},{"operation_name":"upsert","last_dispatch_duration_us":8354594,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8927247,"total_server_duration_us":382647,"encode_duration_us":17610,"operation_id":"0x1000e","timeout_ms":10000,"last_server_duration_us":60128,"total_duration_us":10008045},{"operation_name":"upsert","last_dispatch_duration_us":8289668,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8548410,"total_server_duration_us":60851,"encode_duration_us":20582,"operation_id":"0x1028a","timeout_ms":10000,"last_server_duration_us":19809,"total_duration_us":10007537},{"operation_name":"upsert","last_dispatch_duration_us":9307404,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9886917,"total_server_duration_us":9496297,"encode_duration_us":17443,"operation_id":"0x101b7","timeout_ms":10000,"last_server_duration_us":9119248,"total_duration_us":10003049},{"operation_name":"upsert","last_dispatch_duration_us":8544950,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":9273682,"total_server_duration_us":151317,"encode_duration_us":21938,"operation_id":"0x10228","timeout_ms":10000,"last_server_duration_us":77290,"total_duration_us":9302060},{"operation_name":"upsert","last_dispatch_duration_us":9145317,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9237515,"total_server_duration_us":9063354,"encode_duration_us":18088,"operation_id":"0x1030b","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":9255846},{"operation_name":"upsert","last_dispatch_duration_us":9083712,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9083712,"total_server_duration_us":9067105,"encode_duration_us":20686,"operation_id":"0x100a3","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":9089372},{"operation_name":"get","last_dispatch_duration_us":98526,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":98526,"total_server_duration_us":6827,"operation_id":"0x101d6","timeout_ms":30000,"last_server_duration_us":6827,"total_duration_us":3222917},{"operation_name":"upsert","last_dispatch_duration_us":1915208,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1915208,"total_server_duration_us":44833,"encode_duration_us":17041,"operation_id":"0x106cd","timeout_ms":10000,"last_server_duration_us":44833,"total_duration_us":1920997},{"operation_name":"upsert","last_dispatch_duration_us":1879363,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1879363,"total_server_duration_us":80113,"encode_duration_us":19399,"operation_id":"0x10681","timeout_ms":10000,"last_server_duration_us":80113,"total_duration_us":1884246}],"total_count":64}} INFO [2022-03-13 17:17:48,754] jepsen worker 22 - jepsen.util 652 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,754] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,755] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,755] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:48,756] jepsen worker 60 - jepsen.util 690 :invoke :write [20 36] INFO [2022-03-13 17:17:48,757] jepsen worker 52 - jepsen.util 682 :invoke :write [17 49] INFO [2022-03-13 17:17:48,759] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,760] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:48,766] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,767] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,772] jepsen worker 71 - jepsen.util 611 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,773] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,773] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:48,774] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:17:48,793] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,794] jepsen worker 14 - jepsen.util 374 :invoke :write [4 8] INFO [2022-03-13 17:17:48,821] jepsen worker 13 - jepsen.util 733 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,821] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,822] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:48,822] jepsen worker 13 - jepsen.util 733 :invoke :write [4 39] INFO [2022-03-13 17:17:48,886] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,887] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:48,898] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,898] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,900] jepsen worker 82 - jepsen.util 4132 :info :write [27 24] :SyncWriteAmbiguous INFO [2022-03-13 17:17:48,901] jepsen worker 82 - jepsen.util 4222 :invoke :read [27 nil] INFO [2022-03-13 17:17:48,923] jepsen worker 82 - jepsen.util 4222 :ok :read [27 4] INFO [2022-03-13 17:17:48,924] jepsen worker 82 - jepsen.util 4222 :invoke :read [27 nil] INFO [2022-03-13 17:17:48,945] jepsen worker 82 - jepsen.util 4222 :ok :read [27 4] INFO [2022-03-13 17:17:48,946] jepsen worker 82 - jepsen.util 4222 :invoke :read [27 nil] INFO [2022-03-13 17:17:48,949] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,950] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,950] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:17:48,950] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,952] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,952] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,953] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,954] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,955] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,955] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,956] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,957] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:48,957] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,958] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:17:48,974] jepsen worker 82 - jepsen.util 4222 :ok :read [27 4] INFO [2022-03-13 17:17:48,974] jepsen worker 82 - jepsen.util 4222 :invoke :write [27 48] INFO [2022-03-13 17:17:48,987] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:48,987] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:17:49,019] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,019] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,027] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,028] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:17:49,055] jepsen worker 12 - jepsen.util 912 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,055] jepsen worker 12 - jepsen.util 912 :invoke :write [4 27] INFO [2022-03-13 17:17:49,067] jepsen worker 86 - jepsen.util 3866 :ok :read [28 44] INFO [2022-03-13 17:17:49,068] jepsen worker 86 - jepsen.util 3866 :invoke :read [28 nil] INFO [2022-03-13 17:17:49,071] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,071] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,072] jepsen worker 71 - jepsen.util 611 :invoke :write [23 39] INFO [2022-03-13 17:17:49,072] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:49,074] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,074] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:17:49,101] jepsen worker 83 - jepsen.util 3593 :info :write [27 5] :RequestTimeoutException INFO [2022-03-13 17:17:49,101] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,103] jepsen worker 83 - jepsen.util 3683 :invoke :read [27 nil] INFO [2022-03-13 17:17:49,104] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:17:49,112] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,112] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,112] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,113] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:17:49,114] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,114] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:17:49,115] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,116] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,117] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,117] jepsen worker 86 - jepsen.util 3866 :ok :read [28 44] INFO [2022-03-13 17:17:49,118] jepsen worker 86 - jepsen.util 3866 :invoke :write [28 11] INFO [2022-03-13 17:17:49,119] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:17:49,127] jepsen worker 34 - jepsen.util 934 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,127] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,128] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:49,129] jepsen worker 80 - jepsen.util 800 :invoke :write [26 34] INFO [2022-03-13 17:17:49,134] jepsen worker 43 - jepsen.util 673 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 77 - jepsen.util 707 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:49,135] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,135] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,136] jepsen worker 53 - jepsen.util 593 :invoke :write [17 16] INFO [2022-03-13 17:17:49,137] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:49,138] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:49,139] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:49,140] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:49,141] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:49,142] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:17:49,145] jepsen worker 37 - jepsen.util 2467 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,146] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 38] INFO [2022-03-13 17:17:49,148] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,148] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,149] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:49,149] jepsen worker 77 - jepsen.util 707 :invoke :write [25 39] INFO [2022-03-13 17:17:49,149] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 40] INFO [2022-03-13 17:17:49,150] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,151] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 31] INFO [2022-03-13 17:17:49,154] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,154] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,155] jepsen worker 23 - jepsen.util 563 :invoke :write [7 47] INFO [2022-03-13 17:17:49,160] jepsen worker 84 - jepsen.util 4314 :info :write [28 12] :SyncWriteAmbiguous INFO [2022-03-13 17:17:49,168] jepsen worker 78 - jepsen.util 708 :invoke :write [26 9] INFO [2022-03-13 17:17:49,168] jepsen worker 84 - jepsen.util 4404 :invoke :read [28 nil] INFO [2022-03-13 17:17:49,173] jepsen worker 83 - jepsen.util 3683 :ok :read [27 4] INFO [2022-03-13 17:17:49,174] jepsen worker 83 - jepsen.util 3683 :invoke :write [27 10] INFO [2022-03-13 17:17:49,182] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8070175438596492}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5941176470588235}} INFO [2022-03-13 17:17:49,182] jepsen worker nemesis - couchbase.util Rebalance check count: 576 INFO [2022-03-13 17:17:49,182] jepsen worker nemesis - couchbase.util Rebalance stuck count: 455 INFO [2022-03-13 17:17:49,201] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,202] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:49,220] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,220] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:49,247] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,248] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,249] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,250] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:17:49,251] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,251] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,254] jepsen worker 42 - jepsen.util 402 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,255] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,255] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:49,255] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,256] jepsen worker 7 - jepsen.util 637 :invoke :write [2 24] INFO [2022-03-13 17:17:49,256] jepsen worker 8 - jepsen.util 728 :invoke :write [2 40] INFO [2022-03-13 17:17:49,271] jepsen worker 75 - jepsen.util 525 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,271] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,272] jepsen worker 75 - jepsen.util 525 :invoke :write [25 41] INFO [2022-03-13 17:17:49,273] jepsen worker 76 - jepsen.util 796 :invoke :write [25 25] INFO [2022-03-13 17:17:49,283] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,283] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:49,297] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,297] jepsen worker 71 - jepsen.util 611 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,298] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:49,298] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:17:49,299] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,300] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:49,301] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,302] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:17:49,329] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,330] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:17:49,330] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,330] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:17:49,338] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,339] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,348] jepsen worker 52 - jepsen.util 682 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,348] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,348] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:49,349] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:17:49,379] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,379] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,379] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,381] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:17:49,381] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,382] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:17:49,382] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,383] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:17:49,397] jepsen worker 24 - jepsen.util 654 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,398] jepsen worker 24 - jepsen.util 654 :invoke :write [8 27] INFO [2022-03-13 17:17:49,420] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,420] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:17:49,436] jepsen worker 1 - jepsen.util 3511 :info :write [0 40] :SyncWriteAmbiguous INFO [2022-03-13 17:17:49,436] jepsen worker 1 - jepsen.util 3601 :invoke :write [0 12] INFO [2022-03-13 17:17:49,451] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,452] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:17:49,473] jepsen worker 6 - jepsen.util 636 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,473] jepsen worker 77 - jepsen.util 707 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,473] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:49,474] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:49,480] jepsen worker 13 - jepsen.util 733 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,480] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:49,483] jepsen worker 88 - jepsen.util 2428 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,484] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 5] INFO [2022-03-13 17:17:49,490] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,491] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:17:49,492] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,493] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:17:49,497] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,499] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:17:49,509] jepsen worker 17 - jepsen.util 4067 :info :write [5 30] :SyncWriteAmbiguous INFO [2022-03-13 17:17:49,509] jepsen worker 17 - jepsen.util 4157 :invoke :write [5 24] INFO [2022-03-13 17:17:49,548] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,548] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:17:49,560] jepsen worker 60 - jepsen.util 690 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,560] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,560] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,560] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:49,560] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:49,561] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:49,561] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:49,562] jepsen worker 43 - jepsen.util 673 :invoke :write [14 3] INFO [2022-03-13 17:17:49,562] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,562] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:17:49,564] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,564] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:49,594] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,594] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:17:49,598] jepsen worker 85 - jepsen.util 3775 :ok :read [28 44] INFO [2022-03-13 17:17:49,599] jepsen worker 85 - jepsen.util 3775 :invoke :read [28 nil] INFO [2022-03-13 17:17:49,604] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,605] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:17:49,640] jepsen worker 85 - jepsen.util 3775 :ok :read [28 44] INFO [2022-03-13 17:17:49,641] jepsen worker 85 - jepsen.util 3775 :invoke :read [28 nil] INFO [2022-03-13 17:17:49,663] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,663] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,668] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,668] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,669] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:17:49,669] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:17:49,673] jepsen worker 85 - jepsen.util 3775 :ok :read [28 44] INFO [2022-03-13 17:17:49,673] jepsen worker 85 - jepsen.util 3775 :invoke :read [28 nil] INFO [2022-03-13 17:17:49,681] jepsen worker 35 - jepsen.util 935 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,682] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:49,687] jepsen worker 37 - jepsen.util 2467 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,688] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,704] jepsen worker 85 - jepsen.util 3775 :ok :read [28 44] INFO [2022-03-13 17:17:49,705] jepsen worker 85 - jepsen.util 3775 :invoke :write [28 44] INFO [2022-03-13 17:17:49,731] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,731] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:17:49,736] jepsen worker 76 - jepsen.util 796 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,745] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,747] jepsen worker 70 - jepsen.util 880 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,784] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:49,747] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,785] jepsen worker 70 - jepsen.util 880 :invoke :write [23 23] INFO [2022-03-13 17:17:49,785] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:17:49,785] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:49,786] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,788] jepsen worker 44 - jepsen.util 674 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,788] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,788] jepsen worker 44 - jepsen.util 674 :invoke :write [14 14] INFO [2022-03-13 17:17:49,790] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,791] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,793] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,794] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,795] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,796] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,796] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:17:49,798] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:17:49,799] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,800] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,816] jepsen worker 53 - jepsen.util 593 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,816] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,816] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,816] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,818] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:49,820] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:49,820] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,821] jepsen worker 26 - jepsen.util 566 :invoke :write [8 31] INFO [2022-03-13 17:17:49,822] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:49,823] jepsen worker 52 - jepsen.util 682 :invoke :write [17 8] INFO [2022-03-13 17:17:49,854] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,854] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,855] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:49,856] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,878] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,878] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,878] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:17:49,879] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,880] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,880] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:17:49,895] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,895] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:49,904] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,905] jepsen worker 71 - jepsen.util 611 :invoke :write [23 38] INFO [2022-03-13 17:17:49,913] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,913] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,913] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,913] jepsen worker 88 - jepsen.util 2428 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,914] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:17:49,914] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:17:49,919] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 27] INFO [2022-03-13 17:17:49,923] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 32] INFO [2022-03-13 17:17:49,933] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,933] jepsen worker 14 - jepsen.util 374 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,933] jepsen worker 21 - jepsen.util 831 :invoke :write [7 2] INFO [2022-03-13 17:17:49,934] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:49,949] jepsen worker 79 - jepsen.util 1159 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,949] jepsen worker 62 - jepsen.util 512 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,949] jepsen worker 62 - jepsen.util 512 :invoke :write [20 35] INFO [2022-03-13 17:17:49,950] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:49,976] jepsen worker 41 - jepsen.util 4001 :info :write [13 10] :RequestTimeoutException INFO [2022-03-13 17:17:49,976] jepsen worker 41 - jepsen.util 4091 :invoke :read [13 nil] INFO [2022-03-13 17:17:49,989] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:49,990] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:49,999] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,000] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:17:50,014] jepsen worker 78 - jepsen.util 708 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,014] jepsen worker 12 - jepsen.util 912 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,015] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:50,015] jepsen worker 12 - jepsen.util 912 :invoke :write [4 2] INFO [2022-03-13 17:17:50,017] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:50,017] jepsen worker 78 - jepsen.util 708 :invoke :write [26 5] INFO [2022-03-13 17:17:50,030] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,031] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,031] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:50,032] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,035] jepsen worker 41 - jepsen.util 4091 :ok :read [13 39] INFO [2022-03-13 17:17:50,036] jepsen worker 75 - jepsen.util 525 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,036] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,036] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,037] jepsen worker 41 - jepsen.util 4091 :invoke :read [13 nil] INFO [2022-03-13 17:17:50,037] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:50,038] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:50,038] jepsen worker 76 - jepsen.util 796 :invoke :write [25 35] INFO [2022-03-13 17:17:50,040] jepsen worker 15 - jepsen.util 3885 :info :write [5 39] :RequestTimeoutException INFO [2022-03-13 17:17:50,040] jepsen worker 15 - jepsen.util 3975 :invoke :read [5 nil] INFO [2022-03-13 17:17:50,042] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,043] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:17:50,051] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,051] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,052] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:17:50,053] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,054] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,055] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,056] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,061] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:17:50,082] jepsen worker 80 - jepsen.util 800 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,083] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:50,099] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,100] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,100] jepsen worker 41 - jepsen.util 4091 :ok :read [13 39] INFO [2022-03-13 17:17:50,100] jepsen worker 41 - jepsen.util 4091 :invoke :read [13 nil] INFO [2022-03-13 17:17:50,101] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,102] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,106] jepsen worker 59 - jepsen.util 4019 :info :write [19 16] :RequestTimeoutException INFO [2022-03-13 17:17:50,107] jepsen worker 59 - jepsen.util 4109 :invoke :read [19 nil] INFO [2022-03-13 17:17:50,111] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,111] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,111] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,112] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:17:50,120] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,121] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 7] INFO [2022-03-13 17:17:50,139] jepsen worker 59 - jepsen.util 4109 :ok :read [19 6] INFO [2022-03-13 17:17:50,140] jepsen worker 59 - jepsen.util 4109 :invoke :write [19 41] INFO [2022-03-13 17:17:50,156] jepsen worker 39 - jepsen.util 4269 :info :write [13 22] :RequestTimeoutException INFO [2022-03-13 17:17:50,157] jepsen worker 39 - jepsen.util 4359 :invoke :write [13 16] INFO [2022-03-13 17:17:50,168] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,169] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,182] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,182] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,183] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,183] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:17:50,189] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8070175438596492}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.5941176470588235}} INFO [2022-03-13 17:17:50,189] jepsen worker nemesis - couchbase.util Rebalance check count: 577 INFO [2022-03-13 17:17:50,189] jepsen worker nemesis - couchbase.util Rebalance stuck count: 456 INFO [2022-03-13 17:17:50,194] jepsen worker 88 - jepsen.util 2428 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,195] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 29] INFO [2022-03-13 17:17:50,238] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,238] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,239] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,240] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,241] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,241] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,241] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,242] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:17:50,242] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,243] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:17:50,264] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,264] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,264] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 42] INFO [2022-03-13 17:17:50,265] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,274] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,275] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:17:50,276] jepsen worker 68 - jepsen.util 4028 :info :write [22 0] :RequestTimeoutException INFO [2022-03-13 17:17:50,276] jepsen worker 68 - jepsen.util 4118 :invoke :write [22 49] INFO [2022-03-13 17:17:50,286] jepsen worker 67 - jepsen.util 3217 :info :write [22 9] :RequestTimeoutException INFO [2022-03-13 17:17:50,287] jepsen worker 67 - jepsen.util 3307 :invoke :write [22 37] INFO [2022-03-13 17:17:50,316] jepsen worker 37 - jepsen.util 2467 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,317] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:17:50,351] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,351] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,352] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,352] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:17:50,356] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,357] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,364] jepsen worker 87 - jepsen.util 2877 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,365] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,366] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,366] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:17:50,407] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,407] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,408] jepsen worker 43 - jepsen.util 673 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,408] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,408] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,408] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,409] jepsen worker 75 - jepsen.util 525 :invoke :write [25 2] INFO [2022-03-13 17:17:50,410] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:50,411] jepsen worker 43 - jepsen.util 673 :invoke :write [14 12] INFO [2022-03-13 17:17:50,411] jepsen worker 13 - jepsen.util 733 :invoke :write [4 3] INFO [2022-03-13 17:17:50,413] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:50,414] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:50,415] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,415] jepsen worker 3 - jepsen.util 2253 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,416] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:17:50,416] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,429] jepsen worker 8 - jepsen.util 728 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,430] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,430] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,429] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,429] jepsen worker 23 - jepsen.util 563 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,431] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:50,432] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:50,433] jepsen worker 8 - jepsen.util 728 :invoke :write [2 12] INFO [2022-03-13 17:17:50,434] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:50,435] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:50,441] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,442] jepsen worker 71 - jepsen.util 611 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,442] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:17:50,442] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:50,445] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,446] jepsen worker 88 - jepsen.util 2428 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,446] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:50,447] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 25] INFO [2022-03-13 17:17:50,448] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,448] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,449] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,449] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,450] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,451] jepsen worker 71 - jepsen.util 611 :invoke :write [23 39] INFO [2022-03-13 17:17:50,453] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,454] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:17:50,454] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,455] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:17:50,491] jepsen worker 7 - jepsen.util 637 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,492] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,492] jepsen worker 41 - jepsen.util 4091 :ok :read [13 39] INFO [2022-03-13 17:17:50,493] jepsen worker 7 - jepsen.util 637 :invoke :write [2 22] INFO [2022-03-13 17:17:50,494] jepsen worker 41 - jepsen.util 4091 :invoke :write [13 37] INFO [2022-03-13 17:17:50,496] jepsen worker 6 - jepsen.util 636 :invoke :write [2 30] INFO [2022-03-13 17:17:50,499] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,499] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,500] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:17:50,507] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:17:50,519] jepsen worker 78 - jepsen.util 708 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,519] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,519] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,519] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,519] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,520] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:50,521] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:17:50,522] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,522] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:50,523] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,523] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:50,524] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:50,525] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,525] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,526] jepsen worker 78 - jepsen.util 708 :invoke :write [26 7] INFO [2022-03-13 17:17:50,528] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 26] INFO [2022-03-13 17:17:50,528] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,530] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:50,530] jepsen worker 61 - jepsen.util 781 :invoke :write [20 25] INFO [2022-03-13 17:17:50,532] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:50,534] jepsen worker 24 - jepsen.util 654 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,534] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,535] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:50,536] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,541] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,541] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:50,548] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,548] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:17:50,565] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,566] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:17:50,566] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,567] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:17:50,610] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,611] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,623] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,623] jepsen worker 26 - jepsen.util 566 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,623] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:50,624] jepsen worker 26 - jepsen.util 566 :invoke :write [8 27] INFO [2022-03-13 17:17:50,643] jepsen worker 70 - jepsen.util 880 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,643] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:50,682] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,682] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,684] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,684] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,692] jepsen worker 76 - jepsen.util 796 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,693] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:50,705] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,705] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,706] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,707] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,722] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,722] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,722] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:17:50,723] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:17:50,771] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,771] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,772] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,772] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:17:50,785] jepsen worker 62 - jepsen.util 512 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,785] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:50,786] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,786] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,787] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:50,788] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:17:50,788] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:17:50,792] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:50,799] jepsen worker 71 - jepsen.util 611 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,799] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,800] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:50,800] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:17:50,804] jepsen worker 52 - jepsen.util 682 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,804] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,804] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,804] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,805] jepsen worker 52 - jepsen.util 682 :invoke :write [17 47] INFO [2022-03-13 17:17:50,805] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:50,806] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:50,806] jepsen worker 60 - jepsen.util 690 :invoke :write [20 11] INFO [2022-03-13 17:17:50,820] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,820] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,820] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:50,821] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,823] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,823] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,823] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,824] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:17:50,829] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,829] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,830] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,831] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,831] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,832] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,832] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:50,832] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 24] INFO [2022-03-13 17:17:50,833] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,833] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:17:50,849] jepsen worker 61 - jepsen.util 781 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,849] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:50,865] jepsen worker 75 - jepsen.util 525 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,865] jepsen worker 75 - jepsen.util 525 :invoke :write [25 43] INFO [2022-03-13 17:17:50,866] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,868] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,887] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,887] jepsen worker 44 - jepsen.util 674 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,888] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:50,888] jepsen worker 44 - jepsen.util 674 :invoke :write [14 43] INFO [2022-03-13 17:17:50,889] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,890] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:17:50,938] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,938] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,939] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,939] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,942] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,942] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,942] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,943] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:17:50,944] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,944] jepsen worker 88 - jepsen.util 2428 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,944] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,945] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:50,946] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,947] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,947] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,948] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,948] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,948] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 24] INFO [2022-03-13 17:17:50,948] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:17:50,949] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:50,950] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,950] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 11] INFO [2022-03-13 17:17:50,962] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,963] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,963] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,963] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:17:50,964] jepsen worker 69 - jepsen.util 789 :invoke :write [23 46] INFO [2022-03-13 17:17:50,964] jepsen worker 71 - jepsen.util 611 :invoke :write [23 43] INFO [2022-03-13 17:17:50,970] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,971] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:17:50,981] jepsen worker 21 - jepsen.util 831 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,981] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:17:50,981] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:50,982] jepsen worker 80 - jepsen.util 800 :invoke :write [26 27] INFO [2022-03-13 17:17:51,038] jepsen worker 36 - jepsen.util 2466 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,038] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:17:51,048] jepsen worker 12 - jepsen.util 912 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,048] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,049] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:51,049] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:17:51,065] jepsen worker 69 - jepsen.util 789 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,065] jepsen worker 69 - jepsen.util 789 :invoke :write [23 39] INFO [2022-03-13 17:17:51,081] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,081] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,084] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,086] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,097] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,097] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,099] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,100] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,103] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,103] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,107] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,108] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,110] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,111] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,113] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,114] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:17:51,127] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,127] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,164] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,164] jepsen worker 13 - jepsen.util 733 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,164] jepsen worker 43 - jepsen.util 673 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,164] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:51,165] jepsen worker 13 - jepsen.util 733 :invoke :write [4 11] INFO [2022-03-13 17:17:51,165] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:51,171] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,171] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,172] jepsen worker 33 - jepsen.util 753 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,172] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,173] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:51,173] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:51,174] jepsen worker 34 - jepsen.util 934 :invoke :write [11 18] INFO [2022-03-13 17:17:51,174] jepsen worker 33 - jepsen.util 753 :invoke :write [11 36] INFO [2022-03-13 17:17:51,195] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8187134502923976}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6058823529411765}} INFO [2022-03-13 17:17:51,195] jepsen worker nemesis - couchbase.util Rebalance check count: 578 INFO [2022-03-13 17:17:51,195] jepsen worker nemesis - couchbase.util Rebalance stuck count: 456 INFO [2022-03-13 17:17:51,196] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,196] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,197] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 2] INFO [2022-03-13 17:17:51,198] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,200] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,200] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,201] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,202] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,206] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,207] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:17:51,208] jepsen worker 88 - jepsen.util 2428 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,209] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:17:51,235] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,236] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,242] jepsen worker 71 - jepsen.util 611 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,242] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:17:51,275] jepsen worker 79 - jepsen.util 1159 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,275] jepsen worker 8 - jepsen.util 728 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,276] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 11] INFO [2022-03-13 17:17:51,276] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:51,286] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,287] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:51,299] jepsen worker 89 - jepsen.util 2789 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,299] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:17:51,344] jepsen worker 26 - jepsen.util 566 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,344] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,344] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,344] jepsen worker 75 - jepsen.util 525 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,344] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,345] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:51,346] jepsen worker 76 - jepsen.util 796 :invoke :write [25 39] INFO [2022-03-13 17:17:51,346] jepsen worker 53 - jepsen.util 593 :invoke :write [17 11] INFO [2022-03-13 17:17:51,346] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:51,347] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:51,350] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,350] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:17:51,369] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,370] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:17:51,378] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,378] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,378] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,379] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,379] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,380] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,387] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,387] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,388] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,388] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,388] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,388] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,388] jepsen worker 69 - jepsen.util 789 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,388] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,389] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:51,389] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,390] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,390] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:51,391] jepsen worker 14 - jepsen.util 374 :invoke :write [4 30] INFO [2022-03-13 17:17:51,391] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:17:51,391] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:17:51,392] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:17:51,425] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,425] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,453] jepsen worker 37 - jepsen.util 2467 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,454] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,456] jepsen worker 6 - jepsen.util 636 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,456] jepsen worker 78 - jepsen.util 708 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,457] jepsen worker 78 - jepsen.util 708 :invoke :write [26 6] INFO [2022-03-13 17:17:51,457] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:17:51,462] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,462] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,463] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:51,463] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:51,464] jepsen worker 44 - jepsen.util 674 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,464] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:51,467] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,467] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,467] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:51,468] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:17:51,468] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,469] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:51,470] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,471] jepsen worker 71 - jepsen.util 611 :invoke :write [23 37] INFO [2022-03-13 17:17:51,472] jepsen worker 52 - jepsen.util 682 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,472] jepsen worker 52 - jepsen.util 682 :invoke :write [17 31] INFO [2022-03-13 17:17:51,473] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,474] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:17:51,510] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,511] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,512] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,512] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,525] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,525] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 2] INFO [2022-03-13 17:17:51,527] jepsen worker 57 - jepsen.util 3927 :info :write [19 37] :SyncWriteAmbiguous INFO [2022-03-13 17:17:51,528] jepsen worker 57 - jepsen.util 4017 :invoke :write [19 35] INFO [2022-03-13 17:17:51,532] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,532] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,532] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,533] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 5] INFO [2022-03-13 17:17:51,533] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,534] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,536] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,537] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,537] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,538] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,538] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,539] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:17:51,539] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,540] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,544] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,544] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:51,548] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,549] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:17:51,552] jepsen worker 13 - jepsen.util 733 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,552] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,553] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:17:51,554] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:51,576] jepsen worker 7 - jepsen.util 637 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,576] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,576] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,576] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,576] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,576] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:51,576] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,577] jepsen worker 7 - jepsen.util 637 :invoke :write [2 37] INFO [2022-03-13 17:17:51,577] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:51,578] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:51,579] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:51,579] jepsen worker 62 - jepsen.util 512 :invoke :write [20 47] INFO [2022-03-13 17:17:51,580] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:17:51,581] jepsen worker 42 - jepsen.util 402 :invoke :write [14 43] INFO [2022-03-13 17:17:51,612] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,613] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,616] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,616] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:17:51,619] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,620] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:17:51,636] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,636] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,637] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:17:51,637] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,638] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,641] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:51,650] jepsen worker 71 - jepsen.util 611 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,650] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,650] jepsen worker 80 - jepsen.util 800 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,651] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,651] jepsen worker 71 - jepsen.util 611 :invoke :write [23 21] INFO [2022-03-13 17:17:51,652] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:51,652] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:51,653] jepsen worker 80 - jepsen.util 800 :invoke :write [26 28] INFO [2022-03-13 17:17:51,703] jepsen worker 3 - jepsen.util 2253 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,703] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:17:51,707] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,707] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,707] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:51,708] jepsen worker 60 - jepsen.util 690 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,708] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:17:51,708] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:51,708] jepsen worker 60 - jepsen.util 690 :invoke :write [20 18] INFO [2022-03-13 17:17:51,715] jepsen worker 32 - jepsen.util 4262 :info :write [10 4] :RequestTimeoutException INFO [2022-03-13 17:17:51,716] jepsen worker 32 - jepsen.util 4352 :invoke :write [10 16] INFO [2022-03-13 17:17:51,720] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:51,743] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,744] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:17:51,746] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,746] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,747] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,747] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:17:51,748] jepsen worker 34 - jepsen.util 934 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,749] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:51,752] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,752] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,753] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,753] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,754] jepsen worker 70 - jepsen.util 880 :invoke :write [23 17] INFO [2022-03-13 17:17:51,754] jepsen worker 76 - jepsen.util 796 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,755] jepsen worker 8 - jepsen.util 728 :invoke :write [2 7] INFO [2022-03-13 17:17:51,756] jepsen worker 12 - jepsen.util 912 :invoke :write [4 6] INFO [2022-03-13 17:17:51,758] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:17:51,758] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,758] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:51,759] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:51,762] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,763] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,764] jepsen worker 37 - jepsen.util 2467 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,764] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,769] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:17:51,770] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:17:51,798] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,798] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,799] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,800] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:17:51,805] jepsen worker 71 - jepsen.util 611 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,806] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:17:51,827] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,828] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,835] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,835] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,835] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,835] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:51,864] jepsen worker 16 - jepsen.util 3886 :info :write [5 13] :RequestTimeoutException INFO [2022-03-13 17:17:51,864] jepsen worker 16 - jepsen.util 3976 :invoke :write [5 32] INFO [2022-03-13 17:17:51,871] jepsen worker 33 - jepsen.util 753 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,872] jepsen worker 33 - jepsen.util 753 :invoke :write [11 0] INFO [2022-03-13 17:17:51,891] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,891] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:17:51,909] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,909] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,909] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,910] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:17:51,910] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:51,911] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 22] INFO [2022-03-13 17:17:51,912] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,912] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:17:51,927] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,927] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,928] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:51,929] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 12] INFO [2022-03-13 17:17:51,944] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,945] jepsen worker 23 - jepsen.util 563 :invoke :write [7 11] INFO [2022-03-13 17:17:51,946] jepsen worker 31 - jepsen.util 4171 :info :write [10 10] :RequestTimeoutException INFO [2022-03-13 17:17:51,947] jepsen worker 31 - jepsen.util 4261 :invoke :read [10 nil] INFO [2022-03-13 17:17:51,952] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,953] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:51,964] jepsen worker 14 - jepsen.util 374 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,965] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:51,965] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,965] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,965] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,966] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,966] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,966] jepsen worker 14 - jepsen.util 374 :invoke :write [4 2] INFO [2022-03-13 17:17:51,967] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:17:51,967] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:51,968] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:51,968] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:51,969] jepsen worker 44 - jepsen.util 674 :invoke :write [14 43] INFO [2022-03-13 17:17:51,969] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:17:51,973] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,973] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:17:51,979] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:51,979] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,011] jepsen worker 3 - jepsen.util 2253 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,012] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:17:52,023] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,023] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,024] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:17:52,024] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,027] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,028] jepsen worker 80 - jepsen.util 800 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,029] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:52,029] jepsen worker 80 - jepsen.util 800 :invoke :write [26 49] INFO [2022-03-13 17:17:52,035] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,035] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:17:52,106] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,106] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,106] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:52,107] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:52,107] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,108] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:17:52,108] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,109] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:17:52,111] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,112] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:52,113] jepsen worker 53 - jepsen.util 593 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,113] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,113] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,114] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:52,115] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:17:52,115] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:52,116] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,116] jepsen worker 13 - jepsen.util 733 :invoke :write [4 42] INFO [2022-03-13 17:17:52,133] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,134] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:17:52,136] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,136] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,136] jepsen worker 37 - jepsen.util 2467 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,137] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,137] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,138] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,139] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,139] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,142] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,142] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,142] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,143] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 40] INFO [2022-03-13 17:17:52,143] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,144] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,148] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,148] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,149] jepsen worker 33 - jepsen.util 753 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,149] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,149] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,149] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,149] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,150] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,150] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:52,150] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,150] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,151] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:52,151] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,152] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:52,153] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,153] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:52,176] jepsen worker 79 - jepsen.util 1159 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,176] jepsen worker 42 - jepsen.util 402 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,177] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:52,178] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,178] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:17:52,179] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:17:52,200] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8187134502923976}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6058823529411765}} INFO [2022-03-13 17:17:52,201] jepsen worker nemesis - couchbase.util Rebalance check count: 579 INFO [2022-03-13 17:17:52,201] jepsen worker nemesis - couchbase.util Rebalance stuck count: 457 INFO [2022-03-13 17:17:52,217] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,217] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,217] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,218] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,219] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,219] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,222] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,222] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,223] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:17:52,223] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,224] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,225] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:17:52,234] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,235] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:52,244] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,244] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,254] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,255] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:52,280] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,281] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,305] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,305] jepsen worker 37 - jepsen.util 2467 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,306] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 32] INFO [2022-03-13 17:17:52,306] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:17:52,311] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,311] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,311] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:17:52,312] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,318] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,319] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,320] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,320] jepsen worker 0 - jepsen.util 3780 :info :write [0 30] :RequestTimeoutException INFO [2022-03-13 17:17:52,321] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,321] jepsen worker 0 - jepsen.util 3870 :invoke :write [0 21] INFO [2022-03-13 17:17:52,322] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,323] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 24] INFO [2022-03-13 17:17:52,366] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,367] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:17:52,367] jepsen worker 52 - jepsen.util 682 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,367] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,368] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,368] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:52,369] jepsen worker 76 - jepsen.util 796 :invoke :write [25 16] INFO [2022-03-13 17:17:52,370] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:52,370] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,371] jepsen worker 26 - jepsen.util 566 :invoke :write [8 40] INFO [2022-03-13 17:17:52,371] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:52,382] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:52,390] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,390] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,391] jepsen worker 69 - jepsen.util 789 :invoke :write [23 22] INFO [2022-03-13 17:17:52,391] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:52,399] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,399] jepsen worker 71 - jepsen.util 611 :invoke :write [23 41] INFO [2022-03-13 17:17:52,411] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,411] jepsen worker 78 - jepsen.util 708 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,411] jepsen worker 7 - jepsen.util 637 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,412] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:52,413] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:52,413] jepsen worker 61 - jepsen.util 781 :invoke :write [20 40] INFO [2022-03-13 17:17:52,417] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,418] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:17:52,428] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,428] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,430] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,430] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,430] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:17:52,431] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:17:52,431] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,432] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,460] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,461] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:17:52,506] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,506] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,506] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,506] jepsen worker 38 - jepsen.util 2828 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,506] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:17:52,507] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:17:52,507] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 6] INFO [2022-03-13 17:17:52,511] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,511] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:52,514] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,514] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:17:52,530] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 30] INFO [2022-03-13 17:17:52,568] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,568] jepsen worker 62 - jepsen.util 512 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,568] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:52,569] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:17:52,581] jepsen worker 58 - jepsen.util 3658 :info :write [19 10] :RequestTimeoutException INFO [2022-03-13 17:17:52,581] jepsen worker 58 - jepsen.util 3748 :invoke :read [19 nil] INFO [2022-03-13 17:17:52,590] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,591] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:52,613] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,614] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,641] jepsen worker 37 - jepsen.util 2467 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,641] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,658] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,658] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,659] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:17:52,659] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,661] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,662] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,663] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,664] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,665] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,665] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,666] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,667] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:17:52,710] jepsen worker 36 - jepsen.util 2466 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,710] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:17:52,764] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,765] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:17:52,774] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,774] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,776] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:17:52,776] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,778] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,778] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,779] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,780] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,781] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,781] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:17:52,784] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,784] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,787] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,787] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:17:52,847] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,848] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:17:52,859] jepsen worker 88 - jepsen.util 2428 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,860] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 3] INFO [2022-03-13 17:17:52,860] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,861] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 26] INFO [2022-03-13 17:17:52,876] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,876] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,877] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,878] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:52,932] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,932] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:17:52,957] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,957] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,958] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,959] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:17:52,959] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,960] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 28] INFO [2022-03-13 17:17:52,960] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,962] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,963] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,963] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:52,964] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,965] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:17:52,976] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:52,977] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 37] INFO [2022-03-13 17:17:53,043] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,043] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,043] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,044] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,045] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,045] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,045] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,046] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:17:53,046] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,047] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,047] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,048] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,051] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,051] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:17:53,083] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,083] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:17:53,112] jepsen worker 37 - jepsen.util 2467 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,112] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,113] jepsen worker 38 - jepsen.util 2828 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,113] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,113] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,114] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,118] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,118] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,119] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,120] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,121] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,121] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,122] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,123] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:17:53,163] jepsen worker 69 - jepsen.util 789 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,164] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:53,167] jepsen worker 14 - jepsen.util 374 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,168] jepsen worker 14 - jepsen.util 374 :invoke :write [4 11] INFO [2022-03-13 17:17:53,179] jepsen worker 36 - jepsen.util 2466 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,179] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:17:53,199] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,199] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,199] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,200] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:17:53,200] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,202] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:53,203] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,203] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:17:53,207] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8187134502923976}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6058823529411765}} INFO [2022-03-13 17:17:53,207] jepsen worker nemesis - couchbase.util Rebalance check count: 580 INFO [2022-03-13 17:17:53,207] jepsen worker nemesis - couchbase.util Rebalance stuck count: 458 INFO [2022-03-13 17:17:53,235] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,235] jepsen worker 88 - jepsen.util 2428 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,236] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,236] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 27] INFO [2022-03-13 17:17:53,239] jepsen worker 70 - jepsen.util 880 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,239] jepsen worker 70 - jepsen.util 880 :invoke :write [23 2] INFO [2022-03-13 17:17:53,252] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,252] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:17:53,259] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,259] jepsen worker 13 - jepsen.util 733 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,260] jepsen worker 23 - jepsen.util 563 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,260] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:53,260] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,260] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,260] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:53,260] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:53,261] jepsen worker 78 - jepsen.util 708 :invoke :write [26 43] INFO [2022-03-13 17:17:53,261] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:53,299] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,299] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,300] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,300] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,300] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:17:53,301] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,307] jepsen worker 44 - jepsen.util 674 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,308] jepsen worker 44 - jepsen.util 674 :invoke :write [14 27] INFO [2022-03-13 17:17:53,316] jepsen worker 71 - jepsen.util 611 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,316] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,317] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:17:53,317] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:53,318] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,319] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:17:53,344] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,344] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:17:53,378] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,378] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,379] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,380] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:17:53,380] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,380] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,380] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,381] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,381] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,382] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,382] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,383] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:17:53,383] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,387] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:17:53,388] jepsen worker 60 - jepsen.util 690 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,388] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,388] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:53,388] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,389] jepsen worker 53 - jepsen.util 593 :invoke :write [17 44] INFO [2022-03-13 17:17:53,389] jepsen worker 60 - jepsen.util 690 :invoke :write [20 30] INFO [2022-03-13 17:17:53,390] jepsen worker 52 - jepsen.util 682 :invoke :write [17 21] INFO [2022-03-13 17:17:53,390] jepsen worker 43 - jepsen.util 673 :invoke :write [14 29] INFO [2022-03-13 17:17:53,403] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,403] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,403] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,403] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,404] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,404] jepsen worker 24 - jepsen.util 654 :invoke :write [8 43] INFO [2022-03-13 17:17:53,404] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,405] jepsen worker 80 - jepsen.util 800 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,414] jepsen worker 8 - jepsen.util 728 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,415] jepsen worker 61 - jepsen.util 781 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,415] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,415] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,415] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,416] jepsen worker 23 - jepsen.util 563 :invoke :write [7 6] INFO [2022-03-13 17:17:53,417] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:53,418] jepsen worker 34 - jepsen.util 934 :invoke :write [11 16] INFO [2022-03-13 17:17:53,420] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:53,421] jepsen worker 77 - jepsen.util 707 :invoke :write [25 41] INFO [2022-03-13 17:17:53,421] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:53,422] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:53,423] jepsen worker 8 - jepsen.util 728 :invoke :write [2 16] INFO [2022-03-13 17:17:53,426] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:53,426] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:17:53,430] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:17:53,431] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,433] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,444] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,444] jepsen worker 88 - jepsen.util 2428 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,444] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 35] INFO [2022-03-13 17:17:53,445] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 17] INFO [2022-03-13 17:17:53,450] jepsen worker 76 - jepsen.util 796 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,450] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,450] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,451] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:53,451] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,452] jepsen worker 76 - jepsen.util 796 :invoke :write [25 27] INFO [2022-03-13 17:17:53,452] jepsen worker 75 - jepsen.util 525 :invoke :write [25 49] INFO [2022-03-13 17:17:53,453] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,453] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:17:53,453] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:53,466] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,467] jepsen worker 13 - jepsen.util 733 :invoke :write [4 3] INFO [2022-03-13 17:17:53,481] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,482] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:17:53,503] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,503] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:17:53,531] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,532] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:17:53,548] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,548] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,555] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,555] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:17:53,585] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,585] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,589] jepsen worker 12 - jepsen.util 912 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,589] jepsen worker 12 - jepsen.util 912 :invoke :write [4 43] INFO [2022-03-13 17:17:53,589] jepsen worker 26 - jepsen.util 566 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,590] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,590] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:53,591] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:17:53,595] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,596] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:53,610] jepsen worker 14 - jepsen.util 374 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,611] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:53,627] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,628] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,628] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,628] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,629] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,629] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:17:53,638] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,639] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,641] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,641] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:17:53,675] jepsen worker 75 - jepsen.util 525 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,676] jepsen worker 75 - jepsen.util 525 :invoke :write [25 27] INFO [2022-03-13 17:17:53,676] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,677] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,678] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,678] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,679] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,679] jepsen worker 78 - jepsen.util 708 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,679] jepsen worker 61 - jepsen.util 781 :invoke :write [20 30] INFO [2022-03-13 17:17:53,680] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,681] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,681] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,682] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:17:53,682] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,683] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,684] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,686] jepsen worker 40 - jepsen.util 4180 :info :write [13 37] :RequestTimeoutException INFO [2022-03-13 17:17:53,686] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,687] jepsen worker 40 - jepsen.util 4270 :invoke :write [13 8] INFO [2022-03-13 17:17:53,688] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 1] INFO [2022-03-13 17:17:53,691] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,692] jepsen worker 78 - jepsen.util 708 :invoke :write [26 31] INFO [2022-03-13 17:17:53,694] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,696] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:17:53,702] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,703] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:17:53,715] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,715] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,730] jepsen worker 44 - jepsen.util 674 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,730] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,731] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,731] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,731] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,732] jepsen worker 51 - jepsen.util 771 :invoke :write [17 7] INFO [2022-03-13 17:17:53,732] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,735] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:17:53,735] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] INFO [2022-03-13 17:17:53,738] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:53,738] jepsen worker 44 - jepsen.util 674 :invoke :write [14 18] INFO [2022-03-13 17:17:53,747] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:17:53,749] jepsen worker 88 - jepsen.util 2428 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,749] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,777] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,777] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,777] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:53,778] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,784] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,784] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,785] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,786] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:53,786] jepsen worker 6 - jepsen.util 636 :invoke :write [2 11] INFO [2022-03-13 17:17:53,787] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:53,790] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,790] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:17:53,816] jepsen worker 52 - jepsen.util 682 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,818] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:53,836] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,836] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,842] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,843] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:17:53,847] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,847] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:53,850] jepsen worker 55 - jepsen.util 3655 :info :write [18 20] :RequestTimeoutException INFO [2022-03-13 17:17:53,850] jepsen worker 55 - jepsen.util 3745 :invoke :write [18 46] INFO [2022-03-13 17:17:53,865] jepsen worker 23 - jepsen.util 563 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,865] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:53,866] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,866] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,866] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,867] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:17:53,867] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,868] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:17:53,869] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,870] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:53,872] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,873] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 49] INFO [2022-03-13 17:17:53,875] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,876] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:17:53,880] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,881] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:53,911] jepsen worker 76 - jepsen.util 796 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,911] jepsen worker 76 - jepsen.util 796 :invoke :write [25 13] INFO [2022-03-13 17:17:53,938] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,939] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:53,958] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,958] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:53,961] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,962] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:17:53,970] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,970] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:17:53,983] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:53,989] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,000] jepsen worker 60 - jepsen.util 690 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,000] jepsen worker 60 - jepsen.util 690 :invoke :write [20 36] INFO [2022-03-13 17:17:54,012] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,012] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,013] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:17:54,013] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,014] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,015] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:17:54,023] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,024] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:54,047] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,047] jepsen worker 37 - jepsen.util 2467 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,050] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,050] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 32] INFO [2022-03-13 17:17:54,056] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,056] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 43] INFO [2022-03-13 17:17:54,061] jepsen worker 89 - jepsen.util 2789 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,062] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,070] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,071] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 36] INFO [2022-03-13 17:17:54,075] jepsen worker 34 - jepsen.util 934 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,075] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:54,132] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,132] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,132] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,133] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:17:54,133] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,133] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:17:54,134] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,134] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:17:54,147] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,148] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:17:54,163] jepsen worker 13 - jepsen.util 733 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,163] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:54,165] jepsen worker 77 - jepsen.util 707 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,166] jepsen worker 77 - jepsen.util 707 :invoke :write [25 49] INFO [2022-03-13 17:17:54,178] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,179] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,184] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,184] jepsen worker 4 - jepsen.util 2614 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,185] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,185] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,202] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,203] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:54,212] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8304093567251463}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6176470588235294}} INFO [2022-03-13 17:17:54,212] jepsen worker nemesis - couchbase.util Rebalance check count: 581 INFO [2022-03-13 17:17:54,212] jepsen worker nemesis - couchbase.util Rebalance stuck count: 458 INFO [2022-03-13 17:17:54,213] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,214] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:17:54,233] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,234] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:17:54,258] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,258] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,258] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,259] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,259] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,260] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:17:54,261] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,261] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,261] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:17:54,262] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,263] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,264] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,265] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,265] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:17:54,267] jepsen worker 43 - jepsen.util 673 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,268] jepsen worker 43 - jepsen.util 673 :invoke :write [14 2] INFO [2022-03-13 17:17:54,284] jepsen worker 66 - jepsen.util 3846 :info :write [22 2] :SyncWriteAmbiguous INFO [2022-03-13 17:17:54,285] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,285] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,286] jepsen worker 66 - jepsen.util 3936 :invoke :read [22 nil] INFO [2022-03-13 17:17:54,295] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,296] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,296] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,296] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,301] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,302] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,302] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,303] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,306] jepsen worker 37 - jepsen.util 2467 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,307] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 37] INFO [2022-03-13 17:17:54,309] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,309] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,309] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,310] jepsen worker 71 - jepsen.util 611 :invoke :write [23 23] INFO [2022-03-13 17:17:54,311] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,311] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:17:54,319] jepsen worker 24 - jepsen.util 654 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,320] jepsen worker 24 - jepsen.util 654 :invoke :write [8 7] INFO [2022-03-13 17:17:54,370] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,370] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,371] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,371] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:17:54,372] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,372] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,373] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,374] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,375] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,375] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,376] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:17:54,376] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,377] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,377] jepsen worker 75 - jepsen.util 525 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,378] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,379] jepsen worker 75 - jepsen.util 525 :invoke :write [25 40] INFO [2022-03-13 17:17:54,381] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,382] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:17:54,395] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,396] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:17:54,414] jepsen worker 51 - jepsen.util 771 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,414] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,415] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:54,415] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:54,428] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,429] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:17:54,456] jepsen worker 45 - jepsen.util 4185 :info :write [15 27] :SyncWriteAmbiguous INFO [2022-03-13 17:17:54,457] jepsen worker 70 - jepsen.util 880 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,457] jepsen worker 61 - jepsen.util 781 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,457] jepsen worker 45 - jepsen.util 4275 :invoke :read [15 nil] INFO [2022-03-13 17:17:54,458] jepsen worker 70 - jepsen.util 880 :invoke :write [23 46] INFO [2022-03-13 17:17:54,459] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:54,465] jepsen worker 88 - jepsen.util 2428 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,465] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 18] INFO [2022-03-13 17:17:54,477] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,478] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:17:54,496] jepsen worker 53 - jepsen.util 593 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,496] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,496] jepsen worker 53 - jepsen.util 593 :invoke :write [17 4] INFO [2022-03-13 17:17:54,497] jepsen worker 26 - jepsen.util 566 :invoke :write [8 26] INFO [2022-03-13 17:17:54,520] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,520] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:17:54,531] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,531] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:17:54,533] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,534] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:17:54,544] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,544] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,597] jepsen worker 71 - jepsen.util 611 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,597] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,609] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,610] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:17:54,610] jepsen worker 8 - jepsen.util 728 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,610] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:54,626] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,627] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,633] jepsen worker 37 - jepsen.util 2467 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,634] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,653] jepsen worker 76 - jepsen.util 796 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,653] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,654] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:54,654] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:54,669] jepsen worker 44 - jepsen.util 674 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,669] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,669] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:54,669] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,669] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,669] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,670] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:54,670] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,670] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:54,671] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:54,671] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:54,672] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:54,679] jepsen worker 78 - jepsen.util 708 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,680] jepsen worker 78 - jepsen.util 708 :invoke :write [26 43] INFO [2022-03-13 17:17:54,691] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,691] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,691] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:17:54,692] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:17:54,692] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,693] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,702] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,703] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,707] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,708] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,710] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,711] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:17:54,725] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,725] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,725] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,726] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,728] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,729] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 6] INFO [2022-03-13 17:17:54,729] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,729] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,730] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:17:54,730] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:54,740] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,740] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,753] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:17:54,753] jepsen worker 52 - jepsen.util 682 :invoke :write [17 27] INFO [2022-03-13 17:17:54,758] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,758] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,759] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,758] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,758] jepsen worker 24 - jepsen.util 654 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,759] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,761] jepsen worker 51 - jepsen.util 771 :invoke :write [17 35] INFO [2022-03-13 17:17:54,761] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:17:54,762] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,762] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:54,763] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:17:54,765] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,767] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,769] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,769] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,770] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:54,771] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,771] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:17:54,810] jepsen worker 12 - jepsen.util 912 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,810] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,810] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:54,810] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,810] jepsen worker 88 - jepsen.util 2428 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,812] jepsen worker 42 - jepsen.util 402 :invoke :write [14 44] INFO [2022-03-13 17:17:54,813] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:54,813] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,815] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,815] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:54,818] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,819] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,824] jepsen worker 4 - jepsen.util 2614 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,824] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:17:54,849] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,849] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:54,849] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,849] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,849] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,849] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,849] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,850] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,850] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,851] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:54,852] jepsen worker 44 - jepsen.util 674 :invoke :write [14 18] INFO [2022-03-13 17:17:54,852] jepsen worker 80 - jepsen.util 800 :invoke :write [26 49] INFO [2022-03-13 17:17:54,853] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:54,854] jepsen worker 24 - jepsen.util 654 :invoke :write [8 0] INFO [2022-03-13 17:17:54,855] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:17:54,855] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:54,856] jepsen worker 33 - jepsen.util 753 :invoke :write [11 49] INFO [2022-03-13 17:17:54,856] jepsen worker 34 - jepsen.util 934 :invoke :write [11 40] INFO [2022-03-13 17:17:54,863] jepsen worker 75 - jepsen.util 525 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,866] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,866] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,866] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,867] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:54,867] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:17:54,867] jepsen worker 75 - jepsen.util 525 :invoke :write [25 20] INFO [2022-03-13 17:17:54,868] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,868] jepsen worker 76 - jepsen.util 796 :invoke :write [25 14] INFO [2022-03-13 17:17:54,873] jepsen worker 13 - jepsen.util 733 :invoke :write [4 13] INFO [2022-03-13 17:17:54,878] jepsen worker 9 - jepsen.util 3519 :info :write [3 35] :RequestTimeoutException INFO [2022-03-13 17:17:54,879] jepsen worker 9 - jepsen.util 3609 :invoke :read [3 nil] INFO [2022-03-13 17:17:54,880] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,880] jepsen worker 79 - jepsen.util 1159 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,881] jepsen worker 23 - jepsen.util 563 :invoke :write [7 26] INFO [2022-03-13 17:17:54,882] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 15] INFO [2022-03-13 17:17:54,885] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,886] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:17:54,890] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,890] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:17:54,898] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,899] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:17:54,908] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,908] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,909] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,909] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:54,911] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,912] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:54,912] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,913] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 31] INFO [2022-03-13 17:17:54,943] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,943] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:17:54,948] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,948] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,949] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,949] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:17:54,962] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,963] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,965] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,965] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,965] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 32] INFO [2022-03-13 17:17:54,966] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:17:54,968] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,968] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:54,981] jepsen worker 51 - jepsen.util 771 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,981] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,981] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,981] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:54,981] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,981] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,983] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:17:54,983] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:54,984] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:54,985] jepsen worker 21 - jepsen.util 831 :invoke :write [7 29] INFO [2022-03-13 17:17:54,991] jepsen worker 60 - jepsen.util 690 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:54,992] jepsen worker 60 - jepsen.util 690 :invoke :write [20 42] INFO [2022-03-13 17:17:55,034] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,034] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:55,071] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,071] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,071] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,071] jepsen worker 6 - jepsen.util 636 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,072] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,072] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,073] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:55,074] jepsen worker 6 - jepsen.util 636 :invoke :write [2 41] INFO [2022-03-13 17:17:55,078] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:17:55,078] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:17:55,092] jepsen worker 80 - jepsen.util 800 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,093] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,094] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,094] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:55,096] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,096] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,105] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,105] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,122] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,123] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:17:55,135] jepsen worker 88 - jepsen.util 2428 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,135] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,135] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,136] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:55,137] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,137] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:17:55,140] jepsen worker 77 - jepsen.util 707 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,140] jepsen worker 77 - jepsen.util 707 :invoke :write [25 28] INFO [2022-03-13 17:17:55,140] jepsen worker 43 - jepsen.util 673 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,141] jepsen worker 43 - jepsen.util 673 :invoke :write [14 2] INFO [2022-03-13 17:17:55,142] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,142] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:55,158] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,158] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 4] INFO [2022-03-13 17:17:55,205] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,206] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,216] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,216] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,217] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:17:55,217] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:17:55,220] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8304093567251463}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6176470588235294}} INFO [2022-03-13 17:17:55,220] jepsen worker nemesis - couchbase.util Rebalance check count: 582 INFO [2022-03-13 17:17:55,220] jepsen worker nemesis - couchbase.util Rebalance stuck count: 459 INFO [2022-03-13 17:17:55,229] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,230] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,235] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,236] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:17:55,249] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,250] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 37] INFO [2022-03-13 17:17:55,289] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,289] jepsen worker 37 - jepsen.util 2467 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,290] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,290] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,291] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,292] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,292] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,292] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,292] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:17:55,293] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:17:55,294] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,294] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 20] INFO [2022-03-13 17:17:55,306] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,307] jepsen worker 69 - jepsen.util 789 :invoke :write [23 22] INFO [2022-03-13 17:17:55,344] jepsen worker 89 - jepsen.util 2789 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,346] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:17:55,350] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,351] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:17:55,378] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,378] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:55,388] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,389] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:55,402] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,403] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:17:55,411] jepsen worker 53 - jepsen.util 593 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,412] jepsen worker 53 - jepsen.util 593 :invoke :write [17 38] INFO [2022-03-13 17:17:55,418] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,418] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,419] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,419] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:55,420] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:55,420] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:55,421] jepsen worker 78 - jepsen.util 708 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,421] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:55,439] jepsen worker 38 - jepsen.util 2828 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,440] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,444] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,444] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,469] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,469] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,475] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,476] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,476] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,476] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:55,476] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,477] jepsen worker 14 - jepsen.util 374 :invoke :write [4 11] INFO [2022-03-13 17:17:55,477] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:55,478] jepsen worker 25 - jepsen.util 565 :invoke :write [8 2] INFO [2022-03-13 17:17:55,482] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,482] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:17:55,510] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,510] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,511] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,511] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,511] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,512] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,512] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,512] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:55,513] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,513] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,513] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:17:55,513] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,513] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,514] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,514] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,515] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 33] INFO [2022-03-13 17:17:55,516] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,516] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,517] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,517] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:17:55,517] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,518] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,523] jepsen worker 27 - jepsen.util 4167 :info :write [9 0] :RequestTimeoutException INFO [2022-03-13 17:17:55,524] jepsen worker 27 - jepsen.util 4257 :invoke :read [9 nil] INFO [2022-03-13 17:17:55,526] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,526] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,527] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:17:55,527] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,528] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,529] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,530] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,531] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:17:55,539] jepsen worker 47 - jepsen.util 3917 :info :write [15 9] :RequestTimeoutException INFO [2022-03-13 17:17:55,540] jepsen worker 47 - jepsen.util 4007 :invoke :read [15 nil] INFO [2022-03-13 17:17:55,588] jepsen worker 88 - jepsen.util 2428 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,589] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,628] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,629] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:17:55,649] jepsen worker 4 - jepsen.util 2614 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,649] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:17:55,662] jepsen worker 75 - jepsen.util 525 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,664] jepsen worker 75 - jepsen.util 525 :invoke :write [25 15] INFO [2022-03-13 17:17:55,666] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,668] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,679] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,679] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:55,682] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,683] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:17:55,717] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,718] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,743] jepsen worker 69 - jepsen.util 789 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,743] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,745] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,745] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:17:55,745] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,747] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,757] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,757] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:17:55,767] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,768] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,777] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,777] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,778] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 26] INFO [2022-03-13 17:17:55,779] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 34] INFO [2022-03-13 17:17:55,787] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,787] jepsen worker 42 - jepsen.util 402 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,788] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:55,789] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:55,801] jepsen worker 33 - jepsen.util 753 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,802] jepsen worker 33 - jepsen.util 753 :invoke :write [11 28] INFO [2022-03-13 17:17:55,814] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,814] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:17:55,827] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,827] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,828] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,828] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,829] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,829] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,830] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,830] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:55,831] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,831] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,832] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:17:55,833] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,834] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,835] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:55,836] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,836] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:17:55,870] jepsen worker 23 - jepsen.util 563 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,870] jepsen worker 23 - jepsen.util 563 :invoke :write [7 49] INFO [2022-03-13 17:17:55,880] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,881] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:17:55,904] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,905] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:17:55,932] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,932] jepsen worker 21 - jepsen.util 831 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,932] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:55,933] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:17:55,940] jepsen worker 81 - jepsen.util 4581 :info :write [27 11] :RequestTimeoutException INFO [2022-03-13 17:17:55,940] jepsen worker 81 - jepsen.util 4671 :invoke :read [27 nil] INFO [2022-03-13 17:17:55,950] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,950] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,951] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:55,951] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:17:55,952] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,953] jepsen worker 69 - jepsen.util 789 :invoke :write [23 45] INFO [2022-03-13 17:17:55,955] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,955] jepsen worker 6 - jepsen.util 636 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,956] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,956] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:55,956] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:55,957] jepsen worker 7 - jepsen.util 637 :invoke :write [2 45] INFO [2022-03-13 17:17:55,962] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,962] jepsen worker 8 - jepsen.util 728 :invoke :write [2 12] INFO [2022-03-13 17:17:55,990] jepsen worker 88 - jepsen.util 2428 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,991] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:17:55,996] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:55,997] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:17:56,007] jepsen worker 26 - jepsen.util 566 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,007] jepsen worker 13 - jepsen.util 733 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,008] jepsen worker 26 - jepsen.util 566 :invoke :write [8 42] INFO [2022-03-13 17:17:56,020] jepsen worker 13 - jepsen.util 733 :invoke :write [4 1] INFO [2022-03-13 17:17:56,043] jepsen worker 79 - jepsen.util 1159 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,043] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,044] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 37] INFO [2022-03-13 17:17:56,044] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:56,100] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,100] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,100] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:56,101] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:17:56,115] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,116] jepsen worker 11 - jepsen.util 3971 :info :write [3 13] :RequestTimeoutException INFO [2022-03-13 17:17:56,116] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:17:56,117] jepsen worker 11 - jepsen.util 4061 :invoke :read [3 nil] INFO [2022-03-13 17:17:56,163] jepsen worker 70 - jepsen.util 880 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,163] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,163] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:56,163] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,163] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,164] jepsen worker 70 - jepsen.util 880 :invoke :write [23 43] INFO [2022-03-13 17:17:56,166] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:56,168] jepsen worker 6 - jepsen.util 636 :invoke :write [2 29] INFO [2022-03-13 17:17:56,168] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:56,168] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:17:56,186] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,187] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:17:56,199] jepsen worker 69 - jepsen.util 789 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,200] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,203] jepsen worker 77 - jepsen.util 707 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,203] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,204] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:56,205] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:56,225] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8421052631578947}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6294117647058823}} INFO [2022-03-13 17:17:56,225] jepsen worker nemesis - couchbase.util Rebalance check count: 583 INFO [2022-03-13 17:17:56,225] jepsen worker nemesis - couchbase.util Rebalance stuck count: 459 INFO [2022-03-13 17:17:56,249] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,249] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,250] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,250] jepsen worker 76 - jepsen.util 796 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,251] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:17:56,251] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:56,267] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,268] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,291] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,292] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:17:56,294] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,294] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,295] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,295] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,295] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,296] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 18] INFO [2022-03-13 17:17:56,300] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,300] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,301] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,301] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:17:56,303] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,303] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:17:56,322] jepsen worker 23 - jepsen.util 563 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,322] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,322] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:17:56,323] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:56,329] jepsen worker 37 - jepsen.util 2467 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,330] jepsen worker 7 - jepsen.util 637 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,331] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 18] INFO [2022-03-13 17:17:56,332] jepsen worker 7 - jepsen.util 637 :invoke :write [2 16] INFO [2022-03-13 17:17:56,363] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,364] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:17:56,371] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,371] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,372] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,373] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:56,382] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,383] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,383] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:17:56,387] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:56,388] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,389] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:17:56,397] jepsen worker 54 - jepsen.util 3654 :info :write [18 46] :SyncWriteAmbiguous INFO [2022-03-13 17:17:56,398] jepsen worker 54 - jepsen.util 3744 :invoke :write [18 22] INFO [2022-03-13 17:17:56,400] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,401] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,419] jepsen worker 52 - jepsen.util 682 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,419] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:56,422] jepsen worker 8 - jepsen.util 728 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,422] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,422] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,422] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:17:56,422] jepsen worker 38 - jepsen.util 2828 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,422] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:56,423] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:56,423] jepsen worker 12 - jepsen.util 912 :invoke :write [4 38] INFO [2022-03-13 17:17:56,424] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:56,424] jepsen worker 42 - jepsen.util 402 :invoke :write [14 42] INFO [2022-03-13 17:17:56,438] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,439] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 45] INFO [2022-03-13 17:17:56,458] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,458] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,459] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,459] jepsen worker 79 - jepsen.util 1159 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,460] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:56,461] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:56,461] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:56,462] jepsen worker 78 - jepsen.util 708 :invoke :write [26 23] INFO [2022-03-13 17:17:56,459] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,463] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:56,463] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,463] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:56,473] jepsen worker 27 - jepsen.util 4257 :ok :read [9 26] INFO [2022-03-13 17:17:56,473] jepsen worker 27 - jepsen.util 4257 :invoke :read [9 nil] INFO [2022-03-13 17:17:56,487] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,487] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,488] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,489] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:56,492] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,492] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,492] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:17:56,493] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:17:56,507] jepsen worker 27 - jepsen.util 4257 :ok :read [9 26] INFO [2022-03-13 17:17:56,508] jepsen worker 27 - jepsen.util 4257 :invoke :read [9 nil] INFO [2022-03-13 17:17:56,513] jepsen worker 37 - jepsen.util 2467 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,513] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 23] INFO [2022-03-13 17:17:56,515] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,516] jepsen worker 75 - jepsen.util 525 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,516] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:17:56,516] jepsen worker 75 - jepsen.util 525 :invoke :write [25 27] INFO [2022-03-13 17:17:56,518] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,519] jepsen worker 71 - jepsen.util 611 :invoke :write [23 21] INFO [2022-03-13 17:17:56,550] jepsen worker 70 - jepsen.util 880 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,550] jepsen worker 70 - jepsen.util 880 :invoke :write [23 19] INFO [2022-03-13 17:17:56,553] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,553] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,554] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,554] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:17:56,560] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 4] INFO [2022-03-13 17:17:56,560] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:56,561] jepsen worker 27 - jepsen.util 4257 :ok :read [9 26] INFO [2022-03-13 17:17:56,562] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,562] jepsen worker 27 - jepsen.util 4257 :invoke :write [9 12] INFO [2022-03-13 17:17:56,563] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:17:56,585] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,586] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:17:56,613] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,613] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,613] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,614] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:56,615] jepsen worker 61 - jepsen.util 781 :invoke :write [20 38] INFO [2022-03-13 17:17:56,616] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:56,617] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,618] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:17:56,645] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,645] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,666] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,666] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,666] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:17:56,667] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:56,671] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,671] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:17:56,705] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,706] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:17:56,730] jepsen worker 19 - jepsen.util 3889 :info :write [6 36] :RequestTimeoutException INFO [2022-03-13 17:17:56,731] jepsen worker 19 - jepsen.util 3979 :invoke :read [6 nil] INFO [2022-03-13 17:17:56,731] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,732] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,760] jepsen worker 88 - jepsen.util 2428 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,761] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 23] INFO [2022-03-13 17:17:56,777] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,777] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,778] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,778] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:17:56,781] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,782] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:17:56,791] jepsen worker 6 - jepsen.util 636 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,792] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:56,813] jepsen worker 60 - jepsen.util 690 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,813] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:56,836] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,836] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:17:56,851] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,851] jepsen worker 78 - jepsen.util 708 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,852] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:56,852] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,852] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,853] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:56,853] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 47] INFO [2022-03-13 17:17:56,854] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:17:56,860] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,861] jepsen worker 13 - jepsen.util 733 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,861] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,862] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,862] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:56,863] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:56,863] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,864] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,880] jepsen worker 71 - jepsen.util 611 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,880] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,880] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,881] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,882] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,883] jepsen worker 71 - jepsen.util 611 :invoke :write [23 4] INFO [2022-03-13 17:17:56,883] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,886] jepsen worker 44 - jepsen.util 674 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,887] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,887] jepsen worker 44 - jepsen.util 674 :invoke :write [14 16] INFO [2022-03-13 17:17:56,889] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,890] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,891] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,892] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:56,892] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,893] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,893] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:17:56,894] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:17:56,927] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,927] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,928] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,928] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:17:56,934] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,935] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:17:56,937] jepsen worker 4 - jepsen.util 2614 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,938] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:17:56,964] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,966] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:56,969] jepsen worker 71 - jepsen.util 611 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,970] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] WARN [2022-03-13 17:17:56,975] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7499849,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8397471,"total_server_duration_us":237797,"encode_duration_us":17917,"operation_id":"0x10497","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10009350},{"operation_name":"upsert","last_dispatch_duration_us":7726998,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10948031,"total_server_duration_us":243689,"encode_duration_us":19840,"operation_id":"0x10379","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10007960},{"operation_name":"upsert","last_dispatch_duration_us":76925,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":992266,"total_server_duration_us":209792,"encode_duration_us":18705,"operation_id":"0x10467","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10005373},{"operation_name":"upsert","last_dispatch_duration_us":7759231,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11856579,"total_server_duration_us":243927,"encode_duration_us":21965,"operation_id":"0x10332","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10005138},{"operation_name":"upsert","last_dispatch_duration_us":9119307,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9569680,"total_server_duration_us":9218594,"encode_duration_us":17596,"operation_id":"0x102a7","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":9702272},{"operation_name":"upsert","last_dispatch_duration_us":9131335,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9202359,"total_server_duration_us":9049957,"encode_duration_us":17598,"operation_id":"0x10466","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":9220982},{"operation_name":"upsert","last_dispatch_duration_us":9067257,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9067257,"total_server_duration_us":9053293,"encode_duration_us":35660,"operation_id":"0x1073b","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9071395},{"operation_name":"upsert","last_dispatch_duration_us":1007754,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1007754,"total_server_duration_us":153302,"encode_duration_us":19947,"operation_id":"0x107b8","timeout_ms":10000,"last_server_duration_us":153302,"total_duration_us":3120413},{"operation_name":"get","last_dispatch_duration_us":1107022,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1107022,"total_server_duration_us":49,"operation_id":"0x1079c","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":2700335},{"operation_name":"get","last_dispatch_duration_us":960575,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":960575,"total_server_duration_us":43,"operation_id":"0x10846","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":2042881}],"total_count":87}} INFO [2022-03-13 17:17:56,986] jepsen worker 19 - jepsen.util 3979 :ok :read [6 16] INFO [2022-03-13 17:17:56,987] jepsen worker 19 - jepsen.util 3979 :invoke :read [6 nil] INFO [2022-03-13 17:17:56,993] jepsen worker 37 - jepsen.util 2467 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:56,994] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,027] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,028] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:17:57,031] jepsen worker 7 - jepsen.util 637 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,031] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,031] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,032] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,032] jepsen worker 61 - jepsen.util 781 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,032] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,033] jepsen worker 77 - jepsen.util 707 :invoke :write [25 11] INFO [2022-03-13 17:17:57,034] jepsen worker 62 - jepsen.util 512 :invoke :write [20 32] INFO [2022-03-13 17:17:57,037] jepsen worker 19 - jepsen.util 3979 :ok :read [6 16] INFO [2022-03-13 17:17:57,040] jepsen worker 19 - jepsen.util 3979 :invoke :write [6 12] INFO [2022-03-13 17:17:57,041] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:57,042] jepsen worker 7 - jepsen.util 637 :invoke :write [2 27] INFO [2022-03-13 17:17:57,043] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:57,044] jepsen worker 75 - jepsen.util 525 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,044] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,044] jepsen worker 75 - jepsen.util 525 :invoke :write [25 15] INFO [2022-03-13 17:17:57,044] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,045] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:57,045] jepsen worker 13 - jepsen.util 733 :invoke :write [4 5] INFO [2022-03-13 17:17:57,046] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:57,049] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,050] jepsen worker 43 - jepsen.util 673 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,051] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,051] jepsen worker 43 - jepsen.util 673 :invoke :write [14 1] INFO [2022-03-13 17:17:57,052] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,053] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:57,054] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,054] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,057] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,057] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,057] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,059] jepsen worker 72 - jepsen.util 4122 :ok :read [24 24] INFO [2022-03-13 17:17:57,061] jepsen worker 76 - jepsen.util 796 :invoke :write [25 3] INFO [2022-03-13 17:17:57,061] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,062] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:17:57,063] jepsen worker 72 - jepsen.util 4122 :invoke :write [24 10] INFO [2022-03-13 17:17:57,063] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,064] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,066] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:17:57,071] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,073] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:17:57,073] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,075] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,075] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,077] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,078] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,079] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,080] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,081] jepsen worker 9 - jepsen.util 3609 :ok :read [3 25] INFO [2022-03-13 17:17:57,082] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,082] jepsen worker 9 - jepsen.util 3609 :invoke :read [3 nil] INFO [2022-03-13 17:17:57,083] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:17:57,097] jepsen worker 88 - jepsen.util 2428 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,098] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,102] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,103] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,112] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,113] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:17:57,150] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,151] jepsen worker 53 - jepsen.util 593 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,151] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:17:57,152] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:57,179] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,180] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:17:57,182] jepsen worker 12 - jepsen.util 912 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,183] jepsen worker 12 - jepsen.util 912 :invoke :write [4 17] INFO [2022-03-13 17:17:57,186] jepsen worker 64 - jepsen.util 4114 :info :write [21 16] :RequestTimeoutException INFO [2022-03-13 17:17:57,187] jepsen worker 64 - jepsen.util 4204 :invoke :read [21 nil] INFO [2022-03-13 17:17:57,206] jepsen worker 9 - jepsen.util 3609 :ok :read [3 25] INFO [2022-03-13 17:17:57,206] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,207] jepsen worker 9 - jepsen.util 3609 :invoke :read [3 nil] INFO [2022-03-13 17:17:57,207] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,219] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,220] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:57,231] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8421052631578947}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6294117647058823}} INFO [2022-03-13 17:17:57,231] jepsen worker nemesis - couchbase.util Rebalance check count: 584 INFO [2022-03-13 17:17:57,231] jepsen worker nemesis - couchbase.util Rebalance stuck count: 460 INFO [2022-03-13 17:17:57,242] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,242] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,242] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,242] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,243] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:17:57,244] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,245] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:17:57,246] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,246] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,247] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,247] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,248] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,248] jepsen worker 73 - jepsen.util 4213 :info :write [24 34] :RequestTimeoutException INFO [2022-03-13 17:17:57,248] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,249] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,249] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,249] jepsen worker 73 - jepsen.util 4303 :invoke :write [24 22] INFO [2022-03-13 17:17:57,250] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,250] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 27] INFO [2022-03-13 17:17:57,251] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 26] INFO [2022-03-13 17:17:57,262] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,263] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:17:57,273] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,273] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,274] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,274] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,275] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:17:57,276] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:17:57,276] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,276] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,278] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,279] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,280] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,281] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,282] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,283] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,284] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,285] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:17:57,302] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,303] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:57,306] jepsen worker 26 - jepsen.util 566 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,307] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:57,309] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,310] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:57,310] jepsen worker 9 - jepsen.util 3609 :ok :read [3 25] INFO [2022-03-13 17:17:57,311] jepsen worker 9 - jepsen.util 3609 :invoke :write [3 14] INFO [2022-03-13 17:17:57,328] jepsen worker 10 - jepsen.util 4150 :info :write [3 11] :RequestTimeoutException INFO [2022-03-13 17:17:57,328] jepsen worker 10 - jepsen.util 4240 :invoke :write [3 7] INFO [2022-03-13 17:17:57,337] jepsen worker 13 - jepsen.util 733 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,338] jepsen worker 13 - jepsen.util 733 :invoke :write [4 39] INFO [2022-03-13 17:17:57,346] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,347] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:17:57,361] jepsen worker 37 - jepsen.util 2467 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,362] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,387] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,387] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,387] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 36] INFO [2022-03-13 17:17:57,387] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,388] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:17:57,388] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:17:57,398] jepsen worker 42 - jepsen.util 402 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,399] jepsen worker 42 - jepsen.util 402 :invoke :write [14 39] INFO [2022-03-13 17:17:57,412] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,413] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,425] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,426] jepsen worker 71 - jepsen.util 611 :invoke :write [23 45] INFO [2022-03-13 17:17:57,439] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,439] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,440] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:57,441] jepsen worker 22 - jepsen.util 652 :invoke :write [7 34] INFO [2022-03-13 17:17:57,442] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,444] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,444] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,444] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,444] jepsen worker 79 - jepsen.util 1159 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,445] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:57,445] jepsen worker 78 - jepsen.util 708 :invoke :write [26 33] INFO [2022-03-13 17:17:57,446] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 3] INFO [2022-03-13 17:17:57,465] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,466] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 15] INFO [2022-03-13 17:17:57,469] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,470] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,507] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,508] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,520] jepsen worker 88 - jepsen.util 2428 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,520] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,520] jepsen worker 4 - jepsen.util 2614 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,521] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,521] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,520] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,523] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:17:57,524] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,525] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 1] INFO [2022-03-13 17:17:57,525] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,525] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:57,526] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:17:57,526] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,526] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:17:57,529] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,530] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:17:57,536] jepsen worker 76 - jepsen.util 796 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,537] jepsen worker 76 - jepsen.util 796 :invoke :write [25 17] INFO [2022-03-13 17:17:57,562] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,562] jepsen worker 70 - jepsen.util 880 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,563] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,563] jepsen worker 6 - jepsen.util 636 :invoke :write [2 16] INFO [2022-03-13 17:17:57,564] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,566] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:17:57,597] jepsen worker 71 - jepsen.util 611 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,597] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,599] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,599] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:17:57,610] jepsen worker 5 - jepsen.util 2705 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,610] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:17:57,611] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,612] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,613] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,614] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,615] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,616] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,617] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,619] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,623] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,624] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:57,632] jepsen worker 77 - jepsen.util 707 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,632] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:17:57,634] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,635] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:17:57,682] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,682] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,683] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:17:57,684] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:17:57,706] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,706] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,718] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,718] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,718] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,719] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:17:57,720] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:57,720] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,721] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,722] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,722] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:57,723] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:17:57,723] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,724] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:17:57,732] jepsen worker 7 - jepsen.util 637 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,732] jepsen worker 7 - jepsen.util 637 :invoke :write [2 25] INFO [2022-03-13 17:17:57,743] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,743] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,744] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,744] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,746] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,746] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,747] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,748] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:17:57,748] jepsen worker 75 - jepsen.util 525 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,748] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,748] jepsen worker 43 - jepsen.util 673 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,748] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,749] jepsen worker 43 - jepsen.util 673 :invoke :write [14 42] INFO [2022-03-13 17:17:57,749] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,749] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,749] jepsen worker 26 - jepsen.util 566 :invoke :write [8 48] INFO [2022-03-13 17:17:57,750] jepsen worker 52 - jepsen.util 682 :invoke :write [17 28] INFO [2022-03-13 17:17:57,751] jepsen worker 53 - jepsen.util 593 :invoke :write [17 25] INFO [2022-03-13 17:17:57,751] jepsen worker 75 - jepsen.util 525 :invoke :write [25 29] INFO [2022-03-13 17:17:57,755] jepsen worker 60 - jepsen.util 690 :invoke :write [20 37] INFO [2022-03-13 17:17:57,798] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,800] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:17:57,809] jepsen worker 24 - jepsen.util 654 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,810] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,810] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:57,810] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,811] jepsen worker 23 - jepsen.util 563 :invoke :write [7 3] INFO [2022-03-13 17:17:57,811] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:57,818] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,818] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,819] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:17:57,819] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,821] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,822] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:17:57,827] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,827] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:57,833] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,833] jepsen worker 88 - jepsen.util 2428 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,833] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 45] INFO [2022-03-13 17:17:57,834] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:57,839] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,839] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 20] INFO [2022-03-13 17:17:57,879] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,880] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:17:57,884] jepsen worker 28 - jepsen.util 4078 :info :write [9 12] :RequestTimeoutException INFO [2022-03-13 17:17:57,885] jepsen worker 28 - jepsen.util 4168 :invoke :read [9 nil] INFO [2022-03-13 17:17:57,901] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,901] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:17:57,921] jepsen worker 62 - jepsen.util 512 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,921] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:17:57,923] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,923] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,924] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:17:57,924] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:17:57,938] jepsen worker 13 - jepsen.util 733 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,938] jepsen worker 22 - jepsen.util 652 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,938] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:17:57,939] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:57,940] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,940] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:57,957] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,957] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:17:57,966] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,967] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:17:57,982] jepsen worker 78 - jepsen.util 708 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:57,982] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:58,026] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,026] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 35] INFO [2022-03-13 17:17:58,044] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,044] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,044] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:17:58,045] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,047] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,047] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:17:58,053] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,053] jepsen worker 69 - jepsen.util 789 :invoke :write [23 11] INFO [2022-03-13 17:17:58,063] jepsen worker 37 - jepsen.util 2467 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,066] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 32] INFO [2022-03-13 17:17:58,068] jepsen worker 82 - jepsen.util 4222 :info :write [27 48] :SyncWriteAmbiguous INFO [2022-03-13 17:17:58,070] jepsen worker 43 - jepsen.util 673 :fail :write [14 42] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,070] jepsen worker 82 - jepsen.util 4312 :invoke :read [27 nil] INFO [2022-03-13 17:17:58,072] jepsen worker 43 - jepsen.util 673 :invoke :write [14 4] INFO [2022-03-13 17:17:58,075] jepsen worker 12 - jepsen.util 912 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,076] jepsen worker 12 - jepsen.util 912 :invoke :write [4 22] INFO [2022-03-13 17:17:58,111] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,113] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,114] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,115] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,119] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,120] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,126] jepsen worker 29 - jepsen.util 4439 :info :write [9 45] :RequestTimeoutException INFO [2022-03-13 17:17:58,127] jepsen worker 29 - jepsen.util 4529 :invoke :read [9 nil] INFO [2022-03-13 17:17:58,127] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,128] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,129] jepsen worker 15 - jepsen.util 3975 :ok :read [5 43] INFO [2022-03-13 17:17:58,129] jepsen worker 15 - jepsen.util 3975 :invoke :write [5 21] INFO [2022-03-13 17:17:58,136] jepsen worker 30 - jepsen.util 3900 :info :write [10 47] :RequestTimeoutException INFO [2022-03-13 17:17:58,137] jepsen worker 30 - jepsen.util 3990 :invoke :write [10 31] INFO [2022-03-13 17:17:58,139] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,139] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,142] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,143] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 4] INFO [2022-03-13 17:17:58,144] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,145] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,151] jepsen worker 20 - jepsen.util 4250 :info :write [6 0] :RequestTimeoutException INFO [2022-03-13 17:17:58,151] jepsen worker 20 - jepsen.util 4340 :invoke :read [6 nil] INFO [2022-03-13 17:17:58,156] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,157] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,161] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,161] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:17:58,163] jepsen worker 79 - jepsen.util 1159 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,164] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 3] INFO [2022-03-13 17:17:58,170] jepsen worker 76 - jepsen.util 796 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,171] jepsen worker 76 - jepsen.util 796 :invoke :write [25 44] INFO [2022-03-13 17:17:58,177] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,178] jepsen worker 29 - jepsen.util 4529 :ok :read [9 26] INFO [2022-03-13 17:17:58,183] jepsen worker 29 - jepsen.util 4529 :invoke :write [9 44] INFO [2022-03-13 17:17:58,183] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,184] jepsen worker 69 - jepsen.util 789 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,184] jepsen worker 69 - jepsen.util 789 :invoke :write [23 4] INFO [2022-03-13 17:17:58,187] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,187] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:17:58,221] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,222] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:17:58,228] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,228] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,236] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8538011695906433}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6352941176470588}} INFO [2022-03-13 17:17:58,236] jepsen worker nemesis - couchbase.util Rebalance check count: 585 INFO [2022-03-13 17:17:58,237] jepsen worker nemesis - couchbase.util Rebalance stuck count: 460 INFO [2022-03-13 17:17:58,248] jepsen worker 86 - jepsen.util 3866 :info :write [28 11] :SyncWriteAmbiguous INFO [2022-03-13 17:17:58,249] jepsen worker 86 - jepsen.util 3956 :invoke :write [28 49] INFO [2022-03-13 17:17:58,279] jepsen worker 88 - jepsen.util 2428 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,279] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 39] INFO [2022-03-13 17:17:58,287] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,287] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,287] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:17:58,288] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,289] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,290] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:17:58,291] jepsen worker 42 - jepsen.util 402 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,291] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,291] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,292] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,292] jepsen worker 8 - jepsen.util 728 :invoke :write [2 45] WARN [2022-03-13 17:17:58,300] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16752733,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17031867,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9188748,"encode_duration_us":19369,"operation_id":"0x1027e","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10005911}],"total_count":1}} INFO [2022-03-13 17:17:58,307] jepsen worker 42 - jepsen.util 402 :invoke :write [14 44] INFO [2022-03-13 17:17:58,308] jepsen worker 84 - jepsen.util 4404 :ok :read [28 44] INFO [2022-03-13 17:17:58,308] jepsen worker 84 - jepsen.util 4404 :invoke :write [28 0] INFO [2022-03-13 17:17:58,312] jepsen worker 3 - jepsen.util 2253 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,313] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:17:58,323] jepsen worker 69 - jepsen.util 789 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,323] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:17:58,386] jepsen worker 36 - jepsen.util 2466 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,386] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:17:58,386] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11483077,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11483077,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9049045,"encode_duration_us":18189,"operation_id":"0x1063f","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005733},{"operation_name":"upsert","last_dispatch_duration_us":17350789,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":22293198,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9404513,"encode_duration_us":20320,"operation_id":"0xfe70","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10008685}],"total_count":2}} INFO [2022-03-13 17:17:58,386] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,387] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 6] INFO [2022-03-13 17:17:58,393] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,394] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,396] jepsen worker 60 - jepsen.util 690 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,396] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:58,412] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,412] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,414] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,414] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,415] jepsen worker 61 - jepsen.util 781 :invoke :write [20 41] INFO [2022-03-13 17:17:58,415] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:17:58,427] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,428] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,439] jepsen worker 37 - jepsen.util 2467 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,439] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,456] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,456] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,456] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,457] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,465] jepsen worker 34 - jepsen.util 934 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,466] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:17:58,471] jepsen worker 75 - jepsen.util 525 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,472] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] WARN [2022-03-13 17:17:58,472] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11276345,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11276345,"total_server_duration_us":9045859,"encode_duration_us":17003,"operation_id":"0x10439","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10010781},{"operation_name":"upsert","last_dispatch_duration_us":69555,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":643032,"total_server_duration_us":363707,"encode_duration_us":21489,"operation_id":"0x10682","timeout_ms":10000,"last_server_duration_us":49024,"total_duration_us":10007424},{"operation_name":"upsert","last_dispatch_duration_us":9198461,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9198461,"total_server_duration_us":20441,"encode_duration_us":29381,"operation_id":"0x10554","timeout_ms":10000,"last_server_duration_us":20441,"total_duration_us":10006238},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":21243,"operation_id":"0x10714","timeout_ms":10000,"total_duration_us":10005724},{"operation_name":"upsert","last_dispatch_duration_us":9057662,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9057662,"total_server_duration_us":9046921,"encode_duration_us":18265,"operation_id":"0x10839","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9063048},{"operation_name":"get","last_dispatch_duration_us":8067763,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8067763,"total_server_duration_us":92,"operation_id":"0x108e0","timeout_ms":30000,"last_server_duration_us":92,"total_duration_us":8070869},{"operation_name":"upsert","last_dispatch_duration_us":1699202,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1699202,"total_server_duration_us":44058,"encode_duration_us":20671,"operation_id":"0x10c3e","timeout_ms":10000,"last_server_duration_us":44058,"total_duration_us":1881064},{"operation_name":"upsert","last_dispatch_duration_us":1748915,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1748915,"total_server_duration_us":105184,"encode_duration_us":21422,"operation_id":"0x10c2f","timeout_ms":10000,"last_server_duration_us":105184,"total_duration_us":1797932},{"operation_name":"upsert","last_dispatch_duration_us":1147043,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1147043,"total_server_duration_us":78555,"encode_duration_us":16762,"operation_id":"0x10c58","timeout_ms":10000,"last_server_duration_us":78555,"total_duration_us":1721400},{"operation_name":"upsert","last_dispatch_duration_us":1652055,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1652055,"total_server_duration_us":81971,"encode_duration_us":20650,"operation_id":"0x10a0e","timeout_ms":10000,"last_server_duration_us":81971,"total_duration_us":1658497}],"total_count":63}} INFO [2022-03-13 17:17:58,476] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,477] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,477] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,478] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:17:58,486] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,487] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] WARN [2022-03-13 17:17:58,500] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":17956,"operation_id":"0x10641","timeout_ms":10000,"total_duration_us":10011957},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":21654,"operation_id":"0x1073a","timeout_ms":10000,"total_duration_us":10008678},{"operation_name":"upsert","last_dispatch_duration_us":9183095,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9183095,"total_server_duration_us":9029938,"encode_duration_us":31832,"operation_id":"0x10618","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":9187710},{"operation_name":"upsert","last_dispatch_duration_us":9165403,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9165403,"total_server_duration_us":9054355,"encode_duration_us":22641,"operation_id":"0x1038f","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9181852},{"operation_name":"upsert","last_dispatch_duration_us":9113045,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9113045,"total_server_duration_us":9090500,"encode_duration_us":20792,"operation_id":"0x1038e","timeout_ms":10000,"last_server_duration_us":9090500,"total_duration_us":9118489},{"operation_name":"get","last_dispatch_duration_us":3328290,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":3328290,"total_server_duration_us":49,"operation_id":"0x106b0","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":3330292},{"operation_name":"upsert","last_dispatch_duration_us":1533780,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1533780,"total_server_duration_us":630351,"encode_duration_us":24416,"operation_id":"0x10a07","timeout_ms":10000,"last_server_duration_us":630351,"total_duration_us":1537546},{"operation_name":"upsert","last_dispatch_duration_us":1290135,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1290135,"total_server_duration_us":19340,"encode_duration_us":19032,"operation_id":"0x10a34","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":1295216},{"operation_name":"upsert","last_dispatch_duration_us":1157271,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1157271,"total_server_duration_us":316133,"encode_duration_us":46312,"operation_id":"0x1085c","timeout_ms":10000,"last_server_duration_us":316133,"total_duration_us":1227792},{"operation_name":"upsert","last_dispatch_duration_us":1140356,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1140356,"total_server_duration_us":64603,"encode_duration_us":43044,"operation_id":"0x10b52","timeout_ms":10000,"last_server_duration_us":64603,"total_duration_us":1148931}],"total_count":48}} INFO [2022-03-13 17:17:58,514] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,514] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,517] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:17:58,517] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:17:58,518] jepsen worker 4 - jepsen.util 2614 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,518] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,519] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,519] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,520] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,521] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,527] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,527] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,527] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,528] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,528] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,529] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:17:58,529] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,529] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,529] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:17:58,530] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:17:58,541] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,541] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,544] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,544] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,544] jepsen worker 88 - jepsen.util 2428 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,545] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,546] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,546] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,547] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,547] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,547] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,548] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,548] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,549] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:17:58,550] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,555] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,557] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,560] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 30] INFO [2022-03-13 17:17:58,561] jepsen worker 14 - jepsen.util 374 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,562] jepsen worker 14 - jepsen.util 374 :invoke :write [4 40] INFO [2022-03-13 17:17:58,569] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,570] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,570] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,571] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:17:58,571] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,572] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] WARN [2022-03-13 17:17:58,573] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16409388,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17514938,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9278664,"encode_duration_us":18030,"operation_id":"0x10386","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10006585},{"operation_name":"upsert","last_dispatch_duration_us":16763337,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16763337,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9053293,"encode_duration_us":25648,"operation_id":"0x1022c","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10008610}],"total_count":2}} WARN [2022-03-13 17:17:58,573] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":16901,"operation_id":"0x1072f","timeout_ms":10000,"total_duration_us":10008573},{"operation_name":"upsert","last_dispatch_duration_us":8599613,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17096868,"total_server_duration_us":312194,"encode_duration_us":25037,"operation_id":"0x1037c","timeout_ms":10000,"last_server_duration_us":34612,"total_duration_us":10007830},{"operation_name":"upsert","last_dispatch_duration_us":16409388,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17514938,"total_server_duration_us":9278664,"encode_duration_us":18030,"operation_id":"0x10386","timeout_ms":10000,"last_server_duration_us":9055417,"total_duration_us":10006585},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":23596,"operation_id":"0x107c2","timeout_ms":10000,"total_duration_us":10003314},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":19240,"operation_id":"0x1044f","timeout_ms":10000,"total_duration_us":10000853},{"operation_name":"upsert","last_dispatch_duration_us":9216976,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9470789,"total_server_duration_us":9051425,"encode_duration_us":31339,"operation_id":"0x10364","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":9487906},{"operation_name":"upsert","last_dispatch_duration_us":9231210,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9231210,"total_server_duration_us":9036305,"encode_duration_us":18725,"operation_id":"0x10363","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":9235233},{"operation_name":"get","last_dispatch_duration_us":9119040,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9119040,"total_server_duration_us":43,"operation_id":"0x10853","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9122394},{"operation_name":"upsert","last_dispatch_duration_us":889295,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":889295,"total_server_duration_us":97793,"encode_duration_us":21910,"operation_id":"0x10a7c","timeout_ms":10000,"last_server_duration_us":97793,"total_duration_us":894172},{"operation_name":"upsert","last_dispatch_duration_us":746493,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":746493,"total_server_duration_us":725911,"encode_duration_us":19678,"operation_id":"0x10a79","timeout_ms":10000,"last_server_duration_us":725911,"total_duration_us":752498}],"total_count":14}} WARN [2022-03-13 17:17:58,577] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":11276345,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11276345,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9045859,"encode_duration_us":17003,"operation_id":"0x10439","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10010781}],"total_count":1}} WARN [2022-03-13 17:17:58,579] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":50959,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9390395,"total_server_duration_us":227806,"encode_duration_us":17419,"operation_id":"0x107c5","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10010199},{"operation_name":"upsert","last_dispatch_duration_us":10685324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10685324,"total_server_duration_us":34017,"encode_duration_us":18759,"operation_id":"0x10692","timeout_ms":10000,"last_server_duration_us":34017,"total_duration_us":10009589},{"operation_name":"upsert","last_dispatch_duration_us":8470805,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10013545,"total_server_duration_us":455022,"encode_duration_us":19270,"operation_id":"0x1038b","timeout_ms":10000,"last_server_duration_us":32937,"total_duration_us":10008749},{"operation_name":"upsert","last_dispatch_duration_us":8521552,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10140822,"total_server_duration_us":440150,"encode_duration_us":18634,"operation_id":"0x10377","timeout_ms":10000,"last_server_duration_us":30445,"total_duration_us":10004734},{"operation_name":"upsert","last_dispatch_duration_us":8513740,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9074635,"total_server_duration_us":313560,"encode_duration_us":20389,"operation_id":"0x1053b","timeout_ms":10000,"last_server_duration_us":32161,"total_duration_us":10004222},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":19507,"operation_id":"0x10733","timeout_ms":10000,"total_duration_us":10001253},{"operation_name":"upsert","last_dispatch_duration_us":35460,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1369023,"total_server_duration_us":338856,"encode_duration_us":24960,"operation_id":"0x1038c","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10001022},{"operation_name":"upsert","last_dispatch_duration_us":9068481,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9068481,"total_server_duration_us":9051168,"encode_duration_us":28118,"operation_id":"0x10608","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9073201},{"operation_name":"get","last_dispatch_duration_us":52854,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":52854,"total_server_duration_us":9341,"operation_id":"0x10c13","timeout_ms":30000,"last_server_duration_us":9341,"total_duration_us":2161785},{"operation_name":"upsert","last_dispatch_duration_us":1038959,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1349298,"total_server_duration_us":132487,"encode_duration_us":17401,"operation_id":"0x10c15","timeout_ms":10000,"last_server_duration_us":72862,"total_duration_us":1365038}],"total_count":34}} INFO [2022-03-13 17:17:58,580] jepsen worker 26 - jepsen.util 566 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,581] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:58,581] jepsen worker 6 - jepsen.util 636 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,583] jepsen worker 6 - jepsen.util 636 :invoke :write [2 31] INFO [2022-03-13 17:17:58,636] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,637] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:17:58,644] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,644] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,654] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,655] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:58,657] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,658] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,671] jepsen worker 23 - jepsen.util 563 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,671] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,672] jepsen worker 23 - jepsen.util 563 :invoke :write [7 49] INFO [2022-03-13 17:17:58,672] jepsen worker 33 - jepsen.util 753 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,672] jepsen worker 78 - jepsen.util 708 :invoke :write [26 18] INFO [2022-03-13 17:17:58,673] jepsen worker 33 - jepsen.util 753 :invoke :write [11 1] WARN [2022-03-13 17:17:58,679] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17937755,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18589450,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9110599,"encode_duration_us":20648,"operation_id":"0x102e4","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10005627}],"total_count":1}} WARN [2022-03-13 17:17:58,687] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10417334,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10417334,"total_server_duration_us":8123,"operation_id":"0x10315","timeout_ms":30000,"last_server_duration_us":8123,"total_duration_us":10419881},{"operation_name":"upsert","last_dispatch_duration_us":9191061,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":12262586,"total_server_duration_us":42770,"encode_duration_us":17378,"operation_id":"0x105e0","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10009578},{"operation_name":"upsert","last_dispatch_duration_us":11483077,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11483077,"total_server_duration_us":9049045,"encode_duration_us":18189,"operation_id":"0x1063f","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10005733},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":29690,"operation_id":"0x107a1","timeout_ms":10000,"total_duration_us":10005689},{"operation_name":"upsert","last_dispatch_duration_us":9094682,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16631242,"total_server_duration_us":9212030,"encode_duration_us":18791,"operation_id":"0x10448","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10002977},{"operation_name":"get","last_dispatch_duration_us":9923647,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9923647,"total_server_duration_us":62,"operation_id":"0x1072e","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":9926905},{"operation_name":"upsert","last_dispatch_duration_us":9093192,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9093192,"total_server_duration_us":9041612,"encode_duration_us":26025,"operation_id":"0x1084f","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9102353},{"operation_name":"upsert","last_dispatch_duration_us":1385694,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1385694,"total_server_duration_us":76730,"encode_duration_us":21872,"operation_id":"0x106d5","timeout_ms":10000,"last_server_duration_us":76730,"total_duration_us":2985047},{"operation_name":"upsert","last_dispatch_duration_us":1512804,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1512804,"total_server_duration_us":73273,"encode_duration_us":19590,"operation_id":"0x10704","timeout_ms":10000,"last_server_duration_us":73273,"total_duration_us":2609412},{"operation_name":"upsert","last_dispatch_duration_us":2114772,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2114772,"total_server_duration_us":127328,"encode_duration_us":18665,"operation_id":"0x106d1","timeout_ms":10000,"last_server_duration_us":127328,"total_duration_us":2123301}],"total_count":74}} INFO [2022-03-13 17:17:58,696] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,697] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:17:58,699] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,700] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,770] jepsen worker 44 - jepsen.util 674 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,770] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,770] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,770] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,771] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,772] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:17:58,772] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,773] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,774] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:17:58,775] jepsen worker 44 - jepsen.util 674 :invoke :write [14 2] INFO [2022-03-13 17:17:58,782] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,784] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,786] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,786] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:17:58,787] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,788] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,809] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,810] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,810] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:17:58,811] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:58,812] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,813] jepsen worker 71 - jepsen.util 611 :invoke :write [23 35] INFO [2022-03-13 17:17:58,830] jepsen worker 76 - jepsen.util 796 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,830] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,831] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,831] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:58,831] jepsen worker 53 - jepsen.util 593 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,833] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,833] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:58,834] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,835] jepsen worker 76 - jepsen.util 796 :invoke :write [25 48] INFO [2022-03-13 17:17:58,835] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:58,837] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,839] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:17:58,839] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:58,840] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:58,853] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,853] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,853] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:58,854] jepsen worker 75 - jepsen.util 525 :invoke :write [25 31] INFO [2022-03-13 17:17:58,855] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,856] jepsen worker 13 - jepsen.util 733 :invoke :write [4 12] INFO [2022-03-13 17:17:58,860] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,861] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,862] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,862] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,862] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,863] jepsen worker 80 - jepsen.util 800 :invoke :write [26 26] INFO [2022-03-13 17:17:58,863] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,863] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,864] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,864] jepsen worker 34 - jepsen.util 934 :invoke :write [11 40] INFO [2022-03-13 17:17:58,865] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,866] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:17:58,867] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,867] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:17:58,868] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,868] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,869] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,870] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:17:58,870] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,871] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:17:58,882] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,883] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:58,885] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,886] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:17:58,890] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,891] jepsen worker 7 - jepsen.util 637 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,891] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,891] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,891] jepsen worker 79 - jepsen.util 1159 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,892] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:17:58,893] jepsen worker 77 - jepsen.util 707 :invoke :write [25 46] INFO [2022-03-13 17:17:58,894] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:17:58,895] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:58,896] jepsen worker 62 - jepsen.util 512 :invoke :write [20 16] INFO [2022-03-13 17:17:58,922] jepsen worker 71 - jepsen.util 611 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,923] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:17:58,947] jepsen worker 52 - jepsen.util 682 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,947] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,948] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:17:58,948] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:58,961] jepsen worker 14 - jepsen.util 374 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,961] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,961] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,962] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:58,962] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:58,963] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:58,971] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,971] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:58,980] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,981] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:58,981] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:17:58,981] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,025] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,026] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,046] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,047] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,055] jepsen worker 12 - jepsen.util 912 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,055] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,056] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,057] jepsen worker 12 - jepsen.util 912 :invoke :write [4 28] INFO [2022-03-13 17:17:59,057] jepsen worker 25 - jepsen.util 565 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,058] jepsen worker 25 - jepsen.util 565 :invoke :write [8 3] INFO [2022-03-13 17:17:59,059] jepsen worker 76 - jepsen.util 796 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,060] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:59,072] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,072] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,073] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,073] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:17:59,074] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,075] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:17:59,085] jepsen worker 37 - jepsen.util 2467 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,086] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,087] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,087] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,088] jepsen worker 71 - jepsen.util 611 :invoke :write [23 19] INFO [2022-03-13 17:17:59,088] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,088] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,089] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 4] INFO [2022-03-13 17:17:59,089] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,089] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,090] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,090] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 7] INFO [2022-03-13 17:17:59,091] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,092] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:17:59,107] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,107] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,107] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,108] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,108] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,109] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 48] INFO [2022-03-13 17:17:59,114] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,114] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,116] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:17:59,116] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,118] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,118] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:17:59,147] jepsen worker 33 - jepsen.util 753 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,147] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:17:59,148] jepsen worker 61 - jepsen.util 781 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,149] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:59,153] jepsen worker 43 - jepsen.util 673 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,154] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,154] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,154] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,155] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,155] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:17:59,155] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:17:59,155] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,155] jepsen worker 53 - jepsen.util 593 :invoke :write [17 7] INFO [2022-03-13 17:17:59,156] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,156] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:17:59,156] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:17:59,156] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:17:59,157] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:17:59,157] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,158] jepsen worker 60 - jepsen.util 690 :invoke :write [20 47] INFO [2022-03-13 17:17:59,158] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,158] jepsen worker 43 - jepsen.util 673 :invoke :write [14 35] INFO [2022-03-13 17:17:59,159] jepsen worker 52 - jepsen.util 682 :invoke :write [17 30] INFO [2022-03-13 17:17:59,163] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,163] jepsen worker 26 - jepsen.util 566 :invoke :write [8 30] INFO [2022-03-13 17:17:59,164] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:17:59,194] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,195] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,196] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,197] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,204] jepsen worker 83 - jepsen.util 3683 :info :write [27 10] :RequestTimeoutException INFO [2022-03-13 17:17:59,205] jepsen worker 83 - jepsen.util 3773 :invoke :read [27 nil] INFO [2022-03-13 17:17:59,211] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,211] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,231] jepsen worker 37 - jepsen.util 2467 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,231] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,231] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,232] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:17:59,233] jepsen worker 75 - jepsen.util 525 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,234] jepsen worker 75 - jepsen.util 525 :invoke :write [25 5] INFO [2022-03-13 17:17:59,236] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,236] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:17:59,242] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8538011695906433}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6352941176470588}} INFO [2022-03-13 17:17:59,242] jepsen worker nemesis - couchbase.util Rebalance check count: 586 INFO [2022-03-13 17:17:59,242] jepsen worker nemesis - couchbase.util Rebalance stuck count: 461 INFO [2022-03-13 17:17:59,243] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,244] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,263] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,266] jepsen worker 8 - jepsen.util 728 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,267] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,268] jepsen worker 8 - jepsen.util 728 :invoke :write [2 7] INFO [2022-03-13 17:17:59,276] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,276] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,276] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:17:59,277] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,278] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,279] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,280] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,280] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,281] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,282] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,283] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,283] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:17:59,295] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,295] jepsen worker 88 - jepsen.util 2428 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,296] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 8] INFO [2022-03-13 17:17:59,297] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,339] jepsen worker 23 - jepsen.util 563 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,339] jepsen worker 4 - jepsen.util 2614 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,340] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,341] jepsen worker 23 - jepsen.util 563 :invoke :write [7 38] INFO [2022-03-13 17:17:59,343] jepsen worker 44 - jepsen.util 674 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,344] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:17:59,354] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,355] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,355] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,355] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,355] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:17:59,356] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,363] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,364] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,365] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,366] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,367] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,367] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,369] jepsen worker 13 - jepsen.util 733 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,369] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,369] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,369] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:59,370] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:17:59,370] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:17:59,377] jepsen worker 71 - jepsen.util 611 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,378] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,378] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:17:59,378] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:17:59,432] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,433] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,434] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,435] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,435] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,436] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:17:59,436] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 40] INFO [2022-03-13 17:17:59,437] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,440] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,441] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,441] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,442] jepsen worker 80 - jepsen.util 800 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,442] jepsen worker 22 - jepsen.util 652 :invoke :write [7 33] INFO [2022-03-13 17:17:59,443] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,443] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,444] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:17:59,450] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,451] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,451] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,452] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:17:59,454] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:17:59,458] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:59,467] jepsen worker 1 - jepsen.util 3601 :info :write [0 12] :RequestTimeoutException INFO [2022-03-13 17:17:59,467] jepsen worker 1 - jepsen.util 3691 :invoke :read [0 nil] INFO [2022-03-13 17:17:59,512] jepsen worker 37 - jepsen.util 2467 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,512] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,513] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:17:59,514] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 19] INFO [2022-03-13 17:17:59,534] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,534] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,540] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,540] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,550] jepsen worker 17 - jepsen.util 4157 :info :write [5 24] :RequestTimeoutException INFO [2022-03-13 17:17:59,551] jepsen worker 17 - jepsen.util 4247 :invoke :write [5 35] INFO [2022-03-13 17:17:59,599] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,599] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,599] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,600] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,600] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:17:59,600] jepsen worker 4 - jepsen.util 2614 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,600] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,601] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,601] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,602] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,602] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:17:59,602] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:17:59,629] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,629] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,630] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:17:59,634] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,638] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,639] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:17:59,639] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,640] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,641] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,642] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,644] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,644] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:17:59,655] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,656] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,658] jepsen worker 42 - jepsen.util 402 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,659] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:17:59,705] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,705] jepsen worker 34 - jepsen.util 934 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,705] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,705] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,706] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,706] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:17:59,707] jepsen worker 34 - jepsen.util 934 :invoke :write [11 19] INFO [2022-03-13 17:17:59,708] jepsen worker 24 - jepsen.util 654 :invoke :write [8 22] INFO [2022-03-13 17:17:59,708] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:17:59,708] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:59,711] jepsen worker 78 - jepsen.util 708 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,711] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,712] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:17:59,713] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:17:59,721] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,721] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 35] INFO [2022-03-13 17:17:59,728] jepsen worker 85 - jepsen.util 3775 :info :write [28 44] :RequestTimeoutException INFO [2022-03-13 17:17:59,729] jepsen worker 85 - jepsen.util 3865 :invoke :read [28 nil] INFO [2022-03-13 17:17:59,733] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,734] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:17:59,736] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,765] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:17:59,761] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,761] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,766] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,766] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:17:59,768] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,769] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,771] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,772] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,775] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,776] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:17:59,780] jepsen worker 75 - jepsen.util 525 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,780] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,780] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,781] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:59,782] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:59,782] jepsen worker 76 - jepsen.util 796 :invoke :write [25 31] INFO [2022-03-13 17:17:59,822] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,823] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:17:59,835] jepsen worker 53 - jepsen.util 593 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,836] jepsen worker 53 - jepsen.util 593 :invoke :write [17 10] INFO [2022-03-13 17:17:59,837] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,837] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,838] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:17:59,839] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:17:59,839] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,840] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,844] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,845] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:17:59,851] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,851] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:17:59,864] jepsen worker 25 - jepsen.util 565 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,864] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,865] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:17:59,865] jepsen worker 33 - jepsen.util 753 :invoke :write [11 21] INFO [2022-03-13 17:17:59,890] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,890] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:17:59,890] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,891] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,939] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,940] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:17:59,949] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:17:59,949] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,949] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,949] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,950] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,950] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,950] jepsen worker 44 - jepsen.util 674 :invoke :write [14 26] INFO [2022-03-13 17:17:59,951] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:17:59,952] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:17:59,953] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:17:59,953] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:17:59,954] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:17:59,964] jepsen worker 60 - jepsen.util 690 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,965] jepsen worker 60 - jepsen.util 690 :invoke :write [20 19] INFO [2022-03-13 17:17:59,966] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,966] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,968] jepsen worker 37 - jepsen.util 2467 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,972] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:17:59,973] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:17:59,973] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:17:59,974] jepsen worker 77 - jepsen.util 707 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,974] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,974] jepsen worker 6 - jepsen.util 636 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,974] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,974] jepsen worker 77 - jepsen.util 707 :invoke :write [25 20] INFO [2022-03-13 17:17:59,975] jepsen worker 7 - jepsen.util 637 :invoke :write [2 13] INFO [2022-03-13 17:17:59,975] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:17:59,976] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:17:59,990] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:17:59,990] jepsen worker 69 - jepsen.util 789 :invoke :write [23 14] INFO [2022-03-13 17:18:00,014] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,014] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:00,018] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,018] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,018] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,019] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:18:00,019] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:18:00,020] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:00,021] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,022] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:18:00,035] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,035] jepsen worker 76 - jepsen.util 796 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,035] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,036] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:00,036] jepsen worker 76 - jepsen.util 796 :invoke :write [25 35] INFO [2022-03-13 17:18:00,036] jepsen worker 75 - jepsen.util 525 :invoke :write [25 10] INFO [2022-03-13 17:18:00,041] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,042] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:18:00,044] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,044] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:00,046] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,047] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:18:00,075] jepsen worker 24 - jepsen.util 654 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,076] jepsen worker 24 - jepsen.util 654 :invoke :write [8 32] INFO [2022-03-13 17:18:00,087] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,088] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,111] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,111] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,112] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:00,113] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:18:00,115] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,115] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:18:00,119] jepsen worker 62 - jepsen.util 512 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,120] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,121] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:00,124] jepsen worker 70 - jepsen.util 880 :invoke :write [23 21] INFO [2022-03-13 17:18:00,141] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,142] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:18:00,154] jepsen worker 88 - jepsen.util 2428 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,155] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,155] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 10] INFO [2022-03-13 17:18:00,157] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:00,163] jepsen worker 43 - jepsen.util 673 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,164] jepsen worker 59 - jepsen.util 4109 :info :write [19 41] :RequestTimeoutException INFO [2022-03-13 17:18:00,164] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:00,165] jepsen worker 59 - jepsen.util 4199 :invoke :read [19 nil] INFO [2022-03-13 17:18:00,176] jepsen worker 39 - jepsen.util 4359 :info :write [13 16] :RequestTimeoutException INFO [2022-03-13 17:18:00,177] jepsen worker 39 - jepsen.util 4449 :invoke :read [13 nil] INFO [2022-03-13 17:18:00,196] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,196] jepsen worker 34 - jepsen.util 934 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,196] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:00,197] jepsen worker 34 - jepsen.util 934 :invoke :write [11 18] INFO [2022-03-13 17:18:00,198] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,199] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,199] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:00,200] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:00,200] jepsen worker 23 - jepsen.util 563 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,200] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,200] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,200] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,200] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,201] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 4] INFO [2022-03-13 17:18:00,201] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:00,202] jepsen worker 61 - jepsen.util 781 :invoke :write [20 46] INFO [2022-03-13 17:18:00,202] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:00,203] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:18:00,206] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,206] jepsen worker 69 - jepsen.util 789 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,207] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,208] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:18:00,209] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,209] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:18:00,210] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,211] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:18:00,221] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,221] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:00,259] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8654970760233918}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6411764705882352}} INFO [2022-03-13 17:18:00,259] jepsen worker nemesis - couchbase.util Rebalance check count: 587 INFO [2022-03-13 17:18:00,259] jepsen worker nemesis - couchbase.util Rebalance stuck count: 461 INFO [2022-03-13 17:18:00,269] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,270] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:18:00,275] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,276] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:18:00,296] jepsen worker 68 - jepsen.util 4118 :info :write [22 49] :RequestTimeoutException INFO [2022-03-13 17:18:00,297] jepsen worker 68 - jepsen.util 4208 :invoke :write [22 3] INFO [2022-03-13 17:18:00,310] jepsen worker 67 - jepsen.util 3307 :info :write [22 37] :RequestTimeoutException INFO [2022-03-13 17:18:00,311] jepsen worker 67 - jepsen.util 3397 :invoke :read [22 nil] INFO [2022-03-13 17:18:00,312] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,312] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,313] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:18:00,314] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:00,318] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,318] jepsen worker 37 - jepsen.util 2467 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,319] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 23] INFO [2022-03-13 17:18:00,321] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:00,323] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,323] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 42] INFO [2022-03-13 17:18:00,331] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,331] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,331] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,332] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:00,334] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:00,331] jepsen worker 33 - jepsen.util 753 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,335] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:00,337] jepsen worker 80 - jepsen.util 800 :invoke :write [26 44] INFO [2022-03-13 17:18:00,342] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,343] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,353] jepsen worker 76 - jepsen.util 796 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,353] jepsen worker 76 - jepsen.util 796 :invoke :write [25 27] INFO [2022-03-13 17:18:00,369] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,370] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:00,376] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,377] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:18:00,400] jepsen worker 12 - jepsen.util 912 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,401] jepsen worker 12 - jepsen.util 912 :invoke :write [4 4] INFO [2022-03-13 17:18:00,421] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,422] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:18:00,435] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,435] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,436] jepsen worker 44 - jepsen.util 674 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,436] jepsen worker 52 - jepsen.util 682 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,436] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:18:00,437] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:00,437] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:00,438] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,439] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,439] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:18:00,447] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,448] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:18:00,468] jepsen worker 77 - jepsen.util 707 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,468] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,470] jepsen worker 77 - jepsen.util 707 :invoke :write [25 45] INFO [2022-03-13 17:18:00,470] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,470] jepsen worker 78 - jepsen.util 708 :invoke :write [26 47] INFO [2022-03-13 17:18:00,472] jepsen worker 23 - jepsen.util 563 :invoke :write [7 31] INFO [2022-03-13 17:18:00,495] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,496] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,496] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:18:00,497] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:18:00,518] jepsen worker 37 - jepsen.util 2467 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,519] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 2] INFO [2022-03-13 17:18:00,527] jepsen worker 41 - jepsen.util 4091 :info :write [13 37] :RequestTimeoutException INFO [2022-03-13 17:18:00,528] jepsen worker 41 - jepsen.util 4181 :invoke :read [13 nil] INFO [2022-03-13 17:18:00,538] jepsen worker 75 - jepsen.util 525 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,539] jepsen worker 75 - jepsen.util 525 :invoke :write [25 43] INFO [2022-03-13 17:18:00,542] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,543] jepsen worker 14 - jepsen.util 374 :invoke :write [4 32] INFO [2022-03-13 17:18:00,552] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,553] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:00,598] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,599] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,605] jepsen worker 57 - jepsen.util 4017 :info :write [19 35] :SyncWriteAmbiguous INFO [2022-03-13 17:18:00,606] jepsen worker 57 - jepsen.util 4107 :invoke :write [19 9] INFO [2022-03-13 17:18:00,613] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,614] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:18:00,616] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,617] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:18:00,628] jepsen worker 26 - jepsen.util 566 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,628] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:00,632] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:00,632] jepsen worker 8 - jepsen.util 728 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,632] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,633] jepsen worker 42 - jepsen.util 402 :invoke :write [14 20] INFO [2022-03-13 17:18:00,633] jepsen worker 8 - jepsen.util 728 :invoke :write [2 24] INFO [2022-03-13 17:18:00,634] jepsen worker 6 - jepsen.util 636 :invoke :write [2 23] INFO [2022-03-13 17:18:00,651] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,652] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:00,669] jepsen worker 2 - jepsen.util 4322 :ok :read [0 41] INFO [2022-03-13 17:18:00,670] jepsen worker 2 - jepsen.util 4322 :invoke :write [0 13] INFO [2022-03-13 17:18:00,688] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,688] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,689] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:00,689] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:00,692] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,692] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,692] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,692] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,699] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,700] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:18:00,720] jepsen worker 79 - jepsen.util 1159 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,720] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 32] INFO [2022-03-13 17:18:00,728] jepsen worker 24 - jepsen.util 654 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,728] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,729] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:18:00,730] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:00,731] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,731] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,731] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:00,732] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:00,732] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,733] jepsen worker 69 - jepsen.util 789 :invoke :write [23 16] INFO [2022-03-13 17:18:00,739] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,740] jepsen worker 71 - jepsen.util 611 :invoke :write [23 19] INFO [2022-03-13 17:18:00,747] jepsen worker 88 - jepsen.util 2428 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,748] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 47] INFO [2022-03-13 17:18:00,785] jepsen worker 58 - jepsen.util 3748 :ok :read [19 6] INFO [2022-03-13 17:18:00,786] jepsen worker 58 - jepsen.util 3748 :invoke :write [19 47] INFO [2022-03-13 17:18:00,788] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,788] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,789] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,789] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:18:00,790] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:18:00,790] jepsen worker 22 - jepsen.util 652 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,791] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:00,794] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:18:00,810] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,811] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:00,824] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,825] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:00,833] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,834] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:18:00,841] jepsen worker 34 - jepsen.util 934 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,842] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:00,860] jepsen worker 69 - jepsen.util 789 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,860] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:18:00,883] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,883] jepsen worker 61 - jepsen.util 781 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,883] jepsen worker 7 - jepsen.util 637 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,883] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:00,884] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:00,884] jepsen worker 7 - jepsen.util 637 :invoke :write [2 45] INFO [2022-03-13 17:18:00,907] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,907] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,907] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:00,908] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:18:00,910] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,911] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:18:00,912] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,912] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:00,912] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,913] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:18:00,914] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,914] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:00,915] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,916] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:00,917] jepsen worker 76 - jepsen.util 796 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,918] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:00,983] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,983] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:00,992] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,993] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:00,996] jepsen worker 37 - jepsen.util 2467 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,997] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:00,997] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,997] jepsen worker 80 - jepsen.util 800 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,998] jepsen worker 51 - jepsen.util 771 :invoke :write [17 0] INFO [2022-03-13 17:18:00,998] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:00,998] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,998] jepsen worker 80 - jepsen.util 800 :invoke :write [26 4] INFO [2022-03-13 17:18:00,998] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:00,999] jepsen worker 33 - jepsen.util 753 :invoke :write [11 30] INFO [2022-03-13 17:18:00,999] jepsen worker 71 - jepsen.util 611 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,000] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:01,000] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:01,002] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:18:01,057] jepsen worker 70 - jepsen.util 880 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,057] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:01,059] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,060] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:18:01,062] jepsen worker 53 - jepsen.util 593 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,063] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:01,069] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,069] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,070] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:18:01,071] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:18:01,083] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,083] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,083] jepsen worker 14 - jepsen.util 374 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,084] jepsen worker 24 - jepsen.util 654 :invoke :write [8 20] INFO [2022-03-13 17:18:01,085] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:01,085] jepsen worker 35 - jepsen.util 935 :invoke :write [11 33] INFO [2022-03-13 17:18:01,095] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,095] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,096] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:18:01,096] jepsen worker 69 - jepsen.util 789 :invoke :write [23 8] INFO [2022-03-13 17:18:01,100] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,101] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,106] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,107] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,120] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,121] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:18:01,137] jepsen worker 77 - jepsen.util 707 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,138] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,164] jepsen worker 88 - jepsen.util 2428 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,165] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,167] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,167] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,167] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,167] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:01,168] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,169] jepsen worker 34 - jepsen.util 934 :invoke :write [11 21] INFO [2022-03-13 17:18:01,169] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:18:01,169] jepsen worker 44 - jepsen.util 674 :invoke :write [14 27] INFO [2022-03-13 17:18:01,194] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,195] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,195] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,195] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,196] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:18:01,196] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,198] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,198] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,198] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,199] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:18:01,200] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,200] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:18:01,204] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,205] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,209] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,209] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,209] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,210] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,210] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,210] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,211] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,211] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,211] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,211] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,211] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,212] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,212] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,212] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:18:01,212] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,212] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,213] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:18:01,214] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 3] INFO [2022-03-13 17:18:01,264] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8654970760233918}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6411764705882352}} INFO [2022-03-13 17:18:01,264] jepsen worker nemesis - couchbase.util Rebalance check count: 588 INFO [2022-03-13 17:18:01,264] jepsen worker nemesis - couchbase.util Rebalance stuck count: 462 INFO [2022-03-13 17:18:01,280] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,280] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,281] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:18:01,281] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,309] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,309] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,313] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,313] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,390] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,391] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:18:01,391] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,392] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,392] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,393] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:18:01,393] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,394] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,395] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,396] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:18:01,440] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,440] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,441] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,441] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:18:01,468] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,468] jepsen worker 37 - jepsen.util 2467 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,468] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,469] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,469] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,470] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,472] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,472] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,472] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,472] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,473] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,473] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 23] INFO [2022-03-13 17:18:01,475] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,476] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,482] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,482] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,483] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,483] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:18:01,485] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,485] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:18:01,491] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,492] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:18:01,534] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,535] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,535] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,535] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,536] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,537] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,540] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,541] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,542] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,543] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 15] INFO [2022-03-13 17:18:01,575] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,576] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:18:01,606] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,606] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,606] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,607] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,607] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,608] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:18:01,608] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,608] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,609] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,609] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,610] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,610] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:18:01,610] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,611] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 44] INFO [2022-03-13 17:18:01,682] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,682] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,689] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,690] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:18:01,703] jepsen worker 88 - jepsen.util 2428 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,703] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,704] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 31] INFO [2022-03-13 17:18:01,704] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:18:01,740] jepsen worker 32 - jepsen.util 4352 :info :write [10 16] :RequestTimeoutException INFO [2022-03-13 17:18:01,741] jepsen worker 32 - jepsen.util 4442 :invoke :read [10 nil] INFO [2022-03-13 17:18:01,753] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,754] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,765] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,765] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,766] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,767] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:01,767] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,767] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,768] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,769] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,770] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,770] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:18:01,815] jepsen worker 4 - jepsen.util 2614 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,816] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,816] jepsen worker 5 - jepsen.util 2705 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,817] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:18:01,855] jepsen worker 75 - jepsen.util 525 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,856] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,856] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,856] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:01,856] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,857] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,864] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,865] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:01,883] jepsen worker 16 - jepsen.util 3976 :info :write [5 32] :RequestTimeoutException INFO [2022-03-13 17:18:01,884] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,884] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,884] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:01,885] jepsen worker 16 - jepsen.util 4066 :invoke :read [5 nil] INFO [2022-03-13 17:18:01,885] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 30] INFO [2022-03-13 17:18:01,913] jepsen worker 51 - jepsen.util 771 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,914] jepsen worker 51 - jepsen.util 771 :invoke :write [17 31] INFO [2022-03-13 17:18:01,921] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,921] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,922] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,922] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,923] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,923] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:18:01,925] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,925] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,925] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:18:01,926] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,927] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,928] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:01,929] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,930] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:18:01,936] jepsen worker 12 - jepsen.util 912 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,937] jepsen worker 12 - jepsen.util 912 :invoke :write [4 33] INFO [2022-03-13 17:18:01,951] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,951] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,951] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,951] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,951] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,951] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:01,951] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,951] jepsen worker 60 - jepsen.util 690 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,951] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,952] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:01,954] jepsen worker 26 - jepsen.util 566 :invoke :write [8 3] INFO [2022-03-13 17:18:01,954] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,955] jepsen worker 13 - jepsen.util 733 :invoke :write [4 13] INFO [2022-03-13 17:18:01,955] jepsen worker 53 - jepsen.util 593 :invoke :write [17 26] INFO [2022-03-13 17:18:01,956] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:01,957] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:01,959] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,959] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:18:01,959] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,959] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:01,960] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:01,960] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,960] jepsen worker 76 - jepsen.util 796 :invoke :write [25 35] INFO [2022-03-13 17:18:01,961] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,962] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,963] jepsen worker 43 - jepsen.util 673 :invoke :write [14 30] INFO [2022-03-13 17:18:01,966] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,966] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,967] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,968] jepsen worker 52 - jepsen.util 682 :invoke :write [17 44] INFO [2022-03-13 17:18:01,969] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,970] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,971] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,972] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:01,973] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,975] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,976] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,976] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:18:01,977] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:01,978] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:01,983] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,985] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:01,987] jepsen worker 69 - jepsen.util 789 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,988] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,988] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:18:01,990] jepsen worker 75 - jepsen.util 525 :invoke :write [25 41] INFO [2022-03-13 17:18:01,991] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,994] jepsen worker 65 - jepsen.util 4385 :ok :read [21 :nil] INFO [2022-03-13 17:18:01,996] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:01,994] jepsen worker 23 - jepsen.util 563 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,996] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:01,997] jepsen worker 65 - jepsen.util 4385 :invoke :write [21 26] INFO [2022-03-13 17:18:01,998] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:01,998] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,000] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,000] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:18:02,011] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,012] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,016] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,017] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,017] jepsen worker 74 - jepsen.util 4034 :ok :read [24 24] INFO [2022-03-13 17:18:02,017] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,019] jepsen worker 74 - jepsen.util 4034 :invoke :read [24 nil] INFO [2022-03-13 17:18:02,019] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:18:02,022] jepsen worker 56 - jepsen.util 4016 :ok :read [18 21] INFO [2022-03-13 17:18:02,022] jepsen worker 56 - jepsen.util 4016 :invoke :write [18 7] INFO [2022-03-13 17:18:02,039] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,039] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,040] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,041] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,042] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,042] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,043] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,043] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,044] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,045] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:18:02,053] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,053] jepsen worker 88 - jepsen.util 2428 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,053] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:18:02,054] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,066] jepsen worker 31 - jepsen.util 4261 :ok :read [10 :nil] INFO [2022-03-13 17:18:02,067] jepsen worker 31 - jepsen.util 4261 :invoke :read [10 nil] INFO [2022-03-13 17:18:02,110] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,110] jepsen worker 45 - jepsen.util 4275 :ok :read [15 48] INFO [2022-03-13 17:18:02,111] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:18:02,112] jepsen worker 45 - jepsen.util 4275 :invoke :write [15 27] INFO [2022-03-13 17:18:02,119] jepsen worker 18 - jepsen.util 4068 :ok :read [6 16] INFO [2022-03-13 17:18:02,119] jepsen worker 18 - jepsen.util 4068 :invoke :read [6 nil] INFO [2022-03-13 17:18:02,123] jepsen worker 42 - jepsen.util 402 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,124] jepsen worker 42 - jepsen.util 402 :invoke :write [14 11] INFO [2022-03-13 17:18:02,125] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,125] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:18:02,129] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,130] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,133] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,133] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:18:02,135] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,135] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,136] jepsen worker 60 - jepsen.util 690 :invoke :write [20 26] INFO [2022-03-13 17:18:02,136] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:18:02,147] jepsen worker 63 - jepsen.util 4203 :ok :read [21 :nil] INFO [2022-03-13 17:18:02,148] jepsen worker 63 - jepsen.util 4203 :invoke :write [21 28] INFO [2022-03-13 17:18:02,168] jepsen worker 80 - jepsen.util 800 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,168] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:02,187] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,187] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:18:02,193] jepsen worker 78 - jepsen.util 708 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,193] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:02,229] jepsen worker 76 - jepsen.util 796 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,229] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,230] jepsen worker 76 - jepsen.util 796 :invoke :write [25 30] INFO [2022-03-13 17:18:02,230] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,231] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:18:02,231] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:18:02,248] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,248] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,249] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,250] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:02,251] jepsen worker 33 - jepsen.util 753 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,252] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:02,254] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,254] jepsen worker 14 - jepsen.util 374 :invoke :write [4 9] INFO [2022-03-13 17:18:02,278] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8771929824561404}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6470588235294117}} INFO [2022-03-13 17:18:02,278] jepsen worker nemesis - couchbase.util Rebalance check count: 589 INFO [2022-03-13 17:18:02,278] jepsen worker nemesis - couchbase.util Rebalance stuck count: 462 INFO [2022-03-13 17:18:02,319] jepsen worker 8 - jepsen.util 728 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,320] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:02,322] jepsen worker 37 - jepsen.util 2467 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,322] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,322] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,323] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 12] INFO [2022-03-13 17:18:02,330] jepsen worker 53 - jepsen.util 593 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,331] jepsen worker 53 - jepsen.util 593 :invoke :write [17 10] INFO [2022-03-13 17:18:02,347] jepsen worker 0 - jepsen.util 3870 :info :write [0 21] :RequestTimeoutException INFO [2022-03-13 17:18:02,348] jepsen worker 0 - jepsen.util 3960 :invoke :write [0 41] INFO [2022-03-13 17:18:02,351] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,355] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,367] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,367] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,371] jepsen worker 18 - jepsen.util 4068 :ok :read [6 16] INFO [2022-03-13 17:18:02,372] jepsen worker 18 - jepsen.util 4068 :invoke :write [6 16] INFO [2022-03-13 17:18:02,397] jepsen worker 24 - jepsen.util 654 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,399] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,399] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,399] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,399] jepsen worker 79 - jepsen.util 1159 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,399] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:02,399] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,400] jepsen worker 61 - jepsen.util 781 :invoke :write [20 40] INFO [2022-03-13 17:18:02,400] jepsen worker 77 - jepsen.util 707 :invoke :write [25 29] INFO [2022-03-13 17:18:02,402] jepsen worker 78 - jepsen.util 708 :invoke :write [26 32] INFO [2022-03-13 17:18:02,402] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:02,403] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:02,405] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,405] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,406] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 32] INFO [2022-03-13 17:18:02,407] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:18:02,430] jepsen worker 13 - jepsen.util 733 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,431] jepsen worker 13 - jepsen.util 733 :invoke :write [4 12] INFO [2022-03-13 17:18:02,432] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,433] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,433] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 23] INFO [2022-03-13 17:18:02,434] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:18:02,445] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,445] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,445] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,446] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,447] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:18:02,447] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,448] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:18:02,448] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,450] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,450] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:18:02,470] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,470] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,470] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,471] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:18:02,472] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,472] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,479] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,480] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,481] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,482] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,483] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,484] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:18:02,492] jepsen worker 6 - jepsen.util 636 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,492] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,493] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:02,493] jepsen worker 62 - jepsen.util 512 :invoke :write [20 16] INFO [2022-03-13 17:18:02,518] jepsen worker 35 - jepsen.util 935 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,519] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:02,526] jepsen worker 26 - jepsen.util 566 :fail :write [8 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,526] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,528] jepsen worker 26 - jepsen.util 566 :invoke :write [8 1] INFO [2022-03-13 17:18:02,529] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:18:02,529] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,530] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:18:02,539] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,539] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:18:02,579] jepsen worker 37 - jepsen.util 2467 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,580] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,600] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,601] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:02,619] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,619] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,623] jepsen worker 75 - jepsen.util 525 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,623] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:02,631] jepsen worker 51 - jepsen.util 771 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,631] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,631] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:02,632] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:02,636] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,636] jepsen worker 7 - jepsen.util 637 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,637] jepsen worker 7 - jepsen.util 637 :invoke :write [2 45] INFO [2022-03-13 17:18:02,638] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,649] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,650] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:18:02,674] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,675] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,676] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,677] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,679] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,680] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:18:02,691] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,692] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:02,716] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,716] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,716] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,716] jepsen worker 77 - jepsen.util 707 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,717] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:18:02,717] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:02,723] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,724] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:02,726] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,727] jepsen worker 69 - jepsen.util 789 :invoke :write [23 2] INFO [2022-03-13 17:18:02,727] jepsen worker 52 - jepsen.util 682 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,728] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:02,735] jepsen worker 88 - jepsen.util 2428 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,735] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,737] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,737] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,737] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,738] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:02,752] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,753] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,755] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,755] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,757] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,757] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,757] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:02,758] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:18:02,759] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,759] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,759] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,760] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:02,760] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,761] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:18:02,767] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,767] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:18:02,774] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,774] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,774] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,774] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 33] INFO [2022-03-13 17:18:02,775] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 15] INFO [2022-03-13 17:18:02,776] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 45] INFO [2022-03-13 17:18:02,817] jepsen worker 76 - jepsen.util 796 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,818] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:02,829] jepsen worker 12 - jepsen.util 912 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,830] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:02,838] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,839] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:18:02,845] jepsen worker 14 - jepsen.util 374 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,845] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,845] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,845] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,845] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,845] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,845] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,846] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:02,847] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:02,848] jepsen worker 51 - jepsen.util 771 :invoke :write [17 11] INFO [2022-03-13 17:18:02,848] jepsen worker 24 - jepsen.util 654 :invoke :write [8 47] INFO [2022-03-13 17:18:02,849] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:18:02,849] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:02,849] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,850] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,858] jepsen worker 33 - jepsen.util 753 :invoke :write [11 13] INFO [2022-03-13 17:18:02,866] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,866] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:18:02,867] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,868] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,871] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,871] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,871] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:18:02,872] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,875] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,876] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,880] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,881] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:18:02,890] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,891] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,900] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,901] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,902] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,903] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:02,910] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,911] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:18:02,929] jepsen worker 78 - jepsen.util 708 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,930] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:02,951] jepsen worker 43 - jepsen.util 673 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,952] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:02,955] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,956] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:02,973] jepsen worker 34 - jepsen.util 934 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,974] jepsen worker 34 - jepsen.util 934 :invoke :write [11 47] INFO [2022-03-13 17:18:02,975] jepsen worker 69 - jepsen.util 789 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,976] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:02,988] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:02,988] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:18:03,038] jepsen worker 13 - jepsen.util 733 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,038] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,038] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,038] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:18:03,039] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:03,039] jepsen worker 76 - jepsen.util 796 :invoke :write [25 45] INFO [2022-03-13 17:18:03,052] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,052] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:03,054] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,054] jepsen worker 75 - jepsen.util 525 :invoke :write [25 22] INFO [2022-03-13 17:18:03,056] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,057] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,058] jepsen worker 42 - jepsen.util 402 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,058] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,058] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,059] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:03,059] jepsen worker 6 - jepsen.util 636 :invoke :write [2 45] INFO [2022-03-13 17:18:03,060] jepsen worker 8 - jepsen.util 728 :invoke :write [2 42] INFO [2022-03-13 17:18:03,062] jepsen worker 5 - jepsen.util 2705 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,062] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,063] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,063] jepsen worker 88 - jepsen.util 2428 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,064] jepsen worker 71 - jepsen.util 611 :invoke :write [23 22] INFO [2022-03-13 17:18:03,064] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,065] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,066] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:18:03,099] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,099] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,101] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,102] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:03,117] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,117] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,117] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,118] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:18:03,118] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,119] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:18:03,125] jepsen worker 61 - jepsen.util 781 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,126] jepsen worker 61 - jepsen.util 781 :invoke :write [20 25] INFO [2022-03-13 17:18:03,126] jepsen worker 60 - jepsen.util 690 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,127] jepsen worker 60 - jepsen.util 690 :invoke :write [20 48] INFO [2022-03-13 17:18:03,128] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,128] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,130] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,130] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:18:03,186] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,186] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,187] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:18:03,187] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,192] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,192] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,194] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,194] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,195] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,196] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:18:03,196] jepsen worker 37 - jepsen.util 2467 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,196] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,197] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,197] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:18:03,217] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,218] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:18:03,273] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,274] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,274] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:18:03,274] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,275] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,276] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 42] INFO [2022-03-13 17:18:03,283] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8771929824561404}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6470588235294117}} INFO [2022-03-13 17:18:03,283] jepsen worker nemesis - couchbase.util Rebalance check count: 590 INFO [2022-03-13 17:18:03,283] jepsen worker nemesis - couchbase.util Rebalance stuck count: 463 INFO [2022-03-13 17:18:03,299] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,300] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,345] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,345] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:18:03,350] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,351] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,355] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,355] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:18:03,386] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,386] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,387] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,387] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,388] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,389] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,390] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:03,391] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,392] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,392] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,393] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,394] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:18:03,412] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,413] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:18:03,484] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,484] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,484] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:18:03,485] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 42] INFO [2022-03-13 17:18:03,491] jepsen worker 3 - jepsen.util 2253 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,492] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:18:03,492] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,492] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 10] INFO [2022-03-13 17:18:03,539] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,539] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,541] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,542] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:18:03,602] jepsen worker 46 - jepsen.util 4186 :ok :read [15 48] INFO [2022-03-13 17:18:03,603] jepsen worker 46 - jepsen.util 4186 :invoke :write [15 4] INFO [2022-03-13 17:18:03,608] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,608] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,609] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,610] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,613] jepsen worker 47 - jepsen.util 4007 :ok :read [15 48] INFO [2022-03-13 17:18:03,613] jepsen worker 47 - jepsen.util 4007 :invoke :read [15 nil] INFO [2022-03-13 17:18:03,615] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,616] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,617] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,618] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:18:03,623] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,624] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:18:03,655] jepsen worker 32 - jepsen.util 4442 :ok :read [10 :nil] INFO [2022-03-13 17:18:03,656] jepsen worker 32 - jepsen.util 4442 :invoke :write [10 20] INFO [2022-03-13 17:18:03,669] jepsen worker 20 - jepsen.util 4340 :ok :read [6 16] INFO [2022-03-13 17:18:03,669] jepsen worker 20 - jepsen.util 4340 :invoke :read [6 nil] INFO [2022-03-13 17:18:03,691] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,692] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,705] jepsen worker 47 - jepsen.util 4007 :ok :read [15 48] INFO [2022-03-13 17:18:03,706] jepsen worker 47 - jepsen.util 4007 :invoke :write [15 16] INFO [2022-03-13 17:18:03,710] jepsen worker 38 - jepsen.util 2828 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,710] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:03,711] jepsen worker 37 - jepsen.util 2467 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,712] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,728] jepsen worker 40 - jepsen.util 4270 :info :write [13 8] :RequestTimeoutException INFO [2022-03-13 17:18:03,729] jepsen worker 40 - jepsen.util 4360 :invoke :read [13 nil] INFO [2022-03-13 17:18:03,754] jepsen worker 20 - jepsen.util 4340 :ok :read [6 16] INFO [2022-03-13 17:18:03,755] jepsen worker 20 - jepsen.util 4340 :invoke :write [6 31] INFO [2022-03-13 17:18:03,781] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,782] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:18:03,784] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,784] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,784] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:18:03,785] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,786] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,787] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:18:03,795] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,795] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:18:03,811] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,811] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:18:03,880] jepsen worker 55 - jepsen.util 3745 :info :write [18 46] :RequestTimeoutException INFO [2022-03-13 17:18:03,881] jepsen worker 55 - jepsen.util 3835 :invoke :write [18 21] INFO [2022-03-13 17:18:03,898] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,899] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,904] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,904] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,905] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,906] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:03,912] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,913] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:18:03,961] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,961] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:03,980] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,981] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:18:03,985] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,985] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,985] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,986] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:18:03,986] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,987] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,988] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,989] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 12] INFO [2022-03-13 17:18:03,992] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,993] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:03,993] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:03,994] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:18:04,004] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,004] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:18:04,040] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,040] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,076] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,076] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,103] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,103] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,107] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,107] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,112] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,113] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,119] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,119] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,121] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,122] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:18:04,128] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,129] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:18:04,156] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,156] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,156] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,157] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:18:04,158] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:18:04,158] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:18:04,160] jepsen worker 44 - jepsen.util 674 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,160] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,160] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,160] jepsen worker 44 - jepsen.util 674 :invoke :write [14 12] INFO [2022-03-13 17:18:04,160] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,161] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:04,162] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:04,162] jepsen worker 80 - jepsen.util 800 :invoke :write [26 38] INFO [2022-03-13 17:18:04,163] jepsen worker 76 - jepsen.util 796 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,164] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:04,200] jepsen worker 71 - jepsen.util 611 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,200] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,201] jepsen worker 69 - jepsen.util 789 :invoke :write [23 45] INFO [2022-03-13 17:18:04,202] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:18:04,212] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,212] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,244] jepsen worker 53 - jepsen.util 593 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,245] jepsen worker 53 - jepsen.util 593 :invoke :write [17 10] INFO [2022-03-13 17:18:04,247] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,248] jepsen worker 62 - jepsen.util 512 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,248] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:04,248] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:04,249] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,249] jepsen worker 70 - jepsen.util 880 :invoke :write [23 34] INFO [2022-03-13 17:18:04,273] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,274] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,274] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,275] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,276] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:18:04,279] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:18:04,279] jepsen worker 24 - jepsen.util 654 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,280] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:04,288] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8771929824561404}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6470588235294117}} INFO [2022-03-13 17:18:04,288] jepsen worker nemesis - couchbase.util Rebalance check count: 591 INFO [2022-03-13 17:18:04,288] jepsen worker nemesis - couchbase.util Rebalance stuck count: 464 INFO [2022-03-13 17:18:04,294] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,294] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,295] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:18:04,298] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:18:04,302] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,303] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,336] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,337] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,360] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,361] jepsen worker 13 - jepsen.util 733 :invoke :write [4 5] INFO [2022-03-13 17:18:04,391] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:04,391] jepsen worker 7 - jepsen.util 637 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,391] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,392] jepsen worker 12 - jepsen.util 912 :invoke :write [4 15] INFO [2022-03-13 17:18:04,392] jepsen worker 42 - jepsen.util 402 :invoke :write [14 10] INFO [2022-03-13 17:18:04,393] jepsen worker 7 - jepsen.util 637 :invoke :write [2 17] INFO [2022-03-13 17:18:04,396] jepsen worker 69 - jepsen.util 789 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,396] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,396] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,397] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,403] jepsen worker 51 - jepsen.util 771 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,403] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:04,407] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,407] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,407] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:18:04,408] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,410] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,411] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,412] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,412] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,412] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,413] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,413] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:18:04,414] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,415] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,415] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,416] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,417] jepsen worker 26 - jepsen.util 566 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,417] jepsen worker 69 - jepsen.util 789 :invoke :write [23 46] INFO [2022-03-13 17:18:04,418] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,418] jepsen worker 88 - jepsen.util 2428 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,421] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,421] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:04,421] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,422] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,422] jepsen worker 52 - jepsen.util 682 :invoke :write [17 42] INFO [2022-03-13 17:18:04,423] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:18:04,423] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,423] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,424] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 1] INFO [2022-03-13 17:18:04,451] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,452] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:04,490] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,490] jepsen worker 79 - jepsen.util 1159 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,490] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,491] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 49] INFO [2022-03-13 17:18:04,491] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:04,491] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:04,503] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,504] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,505] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,505] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:18:04,540] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,541] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,549] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,549] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:04,582] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,582] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 7] INFO [2022-03-13 17:18:04,621] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,623] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,624] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,624] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,626] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,626] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,627] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,627] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,628] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,628] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:18:04,629] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,629] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,632] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,632] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,632] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:04,633] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,635] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,635] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:18:04,664] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,664] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,704] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,705] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,721] jepsen worker 3 - jepsen.util 2253 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,722] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,724] jepsen worker 75 - jepsen.util 525 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,725] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:04,726] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,727] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:04,730] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,730] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,731] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,731] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:18:04,748] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,748] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,751] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,752] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,755] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,755] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 35] INFO [2022-03-13 17:18:04,794] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,796] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,805] jepsen worker 34 - jepsen.util 934 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,805] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,806] jepsen worker 34 - jepsen.util 934 :invoke :write [11 42] INFO [2022-03-13 17:18:04,806] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:18:04,814] jepsen worker 8 - jepsen.util 728 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,815] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:04,833] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,833] jepsen worker 4 - jepsen.util 2614 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,833] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,833] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,834] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:18:04,835] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:18:04,836] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,836] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,838] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,838] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:18:04,855] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,855] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:04,860] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:04,860] jepsen worker 43 - jepsen.util 673 :invoke :write [14 4] INFO [2022-03-13 17:18:04,867] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,868] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,869] jepsen worker 69 - jepsen.util 789 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,869] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,870] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:18:04,871] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:04,874] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,875] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:18:04,878] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,878] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,879] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,880] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:04,882] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,883] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,890] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,891] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,891] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,891] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,892] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,893] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:18:04,893] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,894] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:18:04,896] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,896] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,898] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,899] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,900] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,901] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:04,902] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,903] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:18:04,923] jepsen worker 61 - jepsen.util 781 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,923] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,923] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,923] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:18:04,924] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,924] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:04,924] jepsen worker 78 - jepsen.util 708 :invoke :write [26 1] INFO [2022-03-13 17:18:04,930] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:04,960] jepsen worker 33 - jepsen.util 753 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,961] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:04,976] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,976] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:04,987] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,987] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:18:04,993] jepsen worker 13 - jepsen.util 733 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,993] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,993] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,993] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:04,994] jepsen worker 75 - jepsen.util 525 :invoke :write [25 15] INFO [2022-03-13 17:18:04,995] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:04,995] jepsen worker 76 - jepsen.util 796 :invoke :write [25 0] INFO [2022-03-13 17:18:04,996] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:18:05,020] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,021] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,048] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,048] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:18:05,064] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,064] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,065] jepsen worker 60 - jepsen.util 690 :fail :write [20 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,065] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,066] jepsen worker 60 - jepsen.util 690 :invoke :write [20 6] INFO [2022-03-13 17:18:05,067] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:05,075] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,076] jepsen worker 6 - jepsen.util 636 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,076] jepsen worker 62 - jepsen.util 512 :invoke :write [20 44] INFO [2022-03-13 17:18:05,078] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,078] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,078] jepsen worker 80 - jepsen.util 800 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,078] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,079] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,079] jepsen worker 14 - jepsen.util 374 :invoke :write [4 33] INFO [2022-03-13 17:18:05,080] jepsen worker 24 - jepsen.util 654 :invoke :write [8 20] INFO [2022-03-13 17:18:05,080] jepsen worker 21 - jepsen.util 831 :invoke :write [7 19] INFO [2022-03-13 17:18:05,080] jepsen worker 6 - jepsen.util 636 :invoke :write [2 37] INFO [2022-03-13 17:18:05,083] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:05,084] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 25] INFO [2022-03-13 17:18:05,134] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,134] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,135] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,135] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,135] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,136] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:18:05,136] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,136] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:18:05,138] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:18:05,139] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,145] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,145] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,147] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,147] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,147] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,147] jepsen worker 79 - jepsen.util 1159 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,148] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:05,149] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,149] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:05,150] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:18:05,152] jepsen worker 23 - jepsen.util 563 :invoke :write [7 27] INFO [2022-03-13 17:18:05,162] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:05,198] jepsen worker 53 - jepsen.util 593 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,198] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,198] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,198] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,199] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:05,199] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,200] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 29] INFO [2022-03-13 17:18:05,200] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:05,201] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,201] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:18:05,220] jepsen worker 44 - jepsen.util 674 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,220] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,220] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,221] jepsen worker 44 - jepsen.util 674 :invoke :write [14 28] INFO [2022-03-13 17:18:05,221] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,222] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:18:05,223] jepsen worker 51 - jepsen.util 771 :invoke :write [17 17] INFO [2022-03-13 17:18:05,228] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,232] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,233] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,234] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,236] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:18:05,274] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,274] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,294] jepsen worker 75 - jepsen.util 525 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,294] jepsen worker 75 - jepsen.util 525 :invoke :write [25 22] INFO [2022-03-13 17:18:05,294] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8771929824561404}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6470588235294117}} INFO [2022-03-13 17:18:05,294] jepsen worker nemesis - couchbase.util Rebalance check count: 592 INFO [2022-03-13 17:18:05,294] jepsen worker nemesis - couchbase.util Rebalance stuck count: 465 INFO [2022-03-13 17:18:05,315] jepsen worker 37 - jepsen.util 2467 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,315] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,316] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,316] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:18:05,328] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,330] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:18:05,369] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,370] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,370] jepsen worker 69 - jepsen.util 789 :invoke :write [23 6] INFO [2022-03-13 17:18:05,372] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:18:05,378] jepsen worker 34 - jepsen.util 934 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,378] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,379] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:18:05,379] jepsen worker 34 - jepsen.util 934 :invoke :write [11 14] INFO [2022-03-13 17:18:05,395] jepsen worker 52 - jepsen.util 682 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,395] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,396] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:05,403] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:05,437] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,439] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:18:05,442] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,442] jepsen worker 70 - jepsen.util 880 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,443] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:18:05,444] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,467] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,468] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,487] jepsen worker 88 - jepsen.util 2428 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,487] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,488] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 47] INFO [2022-03-13 17:18:05,488] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,490] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,490] jepsen worker 76 - jepsen.util 796 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,490] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:05,491] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:05,512] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,512] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,536] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,536] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:18:05,561] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,561] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,561] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,562] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:05,562] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:05,563] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:05,575] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,575] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,576] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,576] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,578] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,579] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,582] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,582] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:18:05,596] jepsen worker 3 - jepsen.util 2253 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,596] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:05,602] jepsen worker 69 - jepsen.util 789 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,602] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,603] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,603] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:18:05,607] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,608] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,609] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,610] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,611] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,612] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,613] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,613] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:18:05,622] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,623] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,647] jepsen worker 75 - jepsen.util 525 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,647] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,647] jepsen worker 43 - jepsen.util 673 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,647] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,648] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:05,648] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:05,649] jepsen worker 42 - jepsen.util 402 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,649] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:05,650] jepsen worker 13 - jepsen.util 733 :invoke :write [4 27] INFO [2022-03-13 17:18:05,650] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:05,701] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,702] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,713] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,714] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:05,717] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,718] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:05,720] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,721] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:05,722] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,723] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:18:05,726] jepsen worker 14 - jepsen.util 374 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,726] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:05,729] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,729] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,729] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:18:05,731] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,732] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:18:05,733] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,735] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:18:05,736] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,737] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:18:05,743] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,743] jepsen worker 78 - jepsen.util 708 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,743] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,746] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,747] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,750] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:05,751] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:18:05,752] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:05,752] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 34] INFO [2022-03-13 17:18:05,763] jepsen worker 88 - jepsen.util 2428 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,763] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,765] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 37] INFO [2022-03-13 17:18:05,765] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,767] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,768] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 14] INFO [2022-03-13 17:18:05,794] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,794] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,795] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:18:05,795] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:18:05,803] jepsen worker 21 - jepsen.util 831 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,805] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:18:05,821] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,821] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,822] jepsen worker 76 - jepsen.util 796 :invoke :write [25 47] INFO [2022-03-13 17:18:05,829] jepsen worker 75 - jepsen.util 525 :invoke :write [25 36] INFO [2022-03-13 17:18:05,881] jepsen worker 64 - jepsen.util 4204 :ok :read [21 :nil] INFO [2022-03-13 17:18:05,882] jepsen worker 64 - jepsen.util 4204 :invoke :read [21 nil] INFO [2022-03-13 17:18:05,892] jepsen worker 12 - jepsen.util 912 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,893] jepsen worker 12 - jepsen.util 912 :invoke :write [4 30] INFO [2022-03-13 17:18:05,894] jepsen worker 44 - jepsen.util 674 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,895] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:05,897] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,899] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:18:05,901] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,902] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,910] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,911] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:05,935] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,936] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:05,960] jepsen worker 23 - jepsen.util 563 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,961] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:05,971] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,972] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:18:05,974] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,974] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,974] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,975] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:05,976] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,976] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,976] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,978] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:18:05,978] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:18:05,980] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:05,987] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,987] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:05,989] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:05,989] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:06,031] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,031] jepsen worker 7 - jepsen.util 637 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,032] jepsen worker 7 - jepsen.util 637 :invoke :write [2 26] INFO [2022-03-13 17:18:06,032] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:06,055] jepsen worker 88 - jepsen.util 2428 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,057] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 14] INFO [2022-03-13 17:18:06,058] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,058] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,059] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,059] jepsen worker 71 - jepsen.util 611 :invoke :write [23 23] INFO [2022-03-13 17:18:06,060] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,060] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,061] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,062] jepsen worker 69 - jepsen.util 789 :invoke :write [23 41] INFO [2022-03-13 17:18:06,069] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,069] jepsen worker 34 - jepsen.util 934 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,069] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,069] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,070] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:06,070] jepsen worker 80 - jepsen.util 800 :invoke :write [26 36] INFO [2022-03-13 17:18:06,071] jepsen worker 33 - jepsen.util 753 :invoke :write [11 28] INFO [2022-03-13 17:18:06,071] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:06,072] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,073] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,073] jepsen worker 61 - jepsen.util 781 :invoke :write [20 20] INFO [2022-03-13 17:18:06,073] jepsen worker 78 - jepsen.util 708 :invoke :write [26 49] INFO [2022-03-13 17:18:06,078] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,079] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,079] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,079] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,079] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:06,080] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,081] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:18:06,081] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,085] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,085] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,085] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,086] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,086] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 6] INFO [2022-03-13 17:18:06,096] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:18:06,150] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,150] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,150] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,151] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:06,153] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,153] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,154] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:18:06,154] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 27] INFO [2022-03-13 17:18:06,160] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,161] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:06,168] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,168] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,168] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:18:06,169] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:18:06,169] jepsen worker 11 - jepsen.util 4061 :ok :read [3 25] INFO [2022-03-13 17:18:06,169] jepsen worker 11 - jepsen.util 4061 :invoke :read [3 nil] INFO [2022-03-13 17:18:06,238] jepsen worker 37 - jepsen.util 2467 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,238] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,239] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 2] INFO [2022-03-13 17:18:06,240] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,260] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,261] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:18:06,291] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,291] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:18:06,299] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:06,299] jepsen worker nemesis - couchbase.util Rebalance check count: 593 INFO [2022-03-13 17:18:06,299] jepsen worker nemesis - couchbase.util Rebalance stuck count: 465 INFO [2022-03-13 17:18:06,305] jepsen worker 89 - jepsen.util 2789 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,305] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,306] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:06,306] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,307] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,307] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,308] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:18:06,308] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,309] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,310] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:18:06,344] jepsen worker 4 - jepsen.util 2614 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,344] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,347] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,348] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:18:06,414] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,415] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:18:06,416] jepsen worker 88 - jepsen.util 2428 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,417] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:06,420] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,421] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:18:06,426] jepsen worker 54 - jepsen.util 3744 :info :write [18 22] :RequestTimeoutException INFO [2022-03-13 17:18:06,426] jepsen worker 54 - jepsen.util 3834 :invoke :write [18 34] INFO [2022-03-13 17:18:06,488] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,488] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:18:06,515] jepsen worker 71 - jepsen.util 611 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,516] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:18:06,525] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,525] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,526] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:18:06,526] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:18:06,536] jepsen worker 24 - jepsen.util 654 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,536] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,537] jepsen worker 14 - jepsen.util 374 :invoke :write [4 20] INFO [2022-03-13 17:18:06,539] jepsen worker 24 - jepsen.util 654 :invoke :write [8 20] INFO [2022-03-13 17:18:06,558] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,559] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:18:06,571] jepsen worker 62 - jepsen.util 512 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,572] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,571] jepsen worker 79 - jepsen.util 1159 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,572] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,572] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:06,573] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:06,574] jepsen worker 23 - jepsen.util 563 :invoke :write [7 22] INFO [2022-03-13 17:18:06,574] jepsen worker 62 - jepsen.util 512 :invoke :write [20 34] INFO [2022-03-13 17:18:06,576] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,577] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,578] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,579] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,585] jepsen worker 69 - jepsen.util 789 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,586] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,591] jepsen worker 27 - jepsen.util 4257 :info :write [9 12] :RequestTimeoutException INFO [2022-03-13 17:18:06,591] jepsen worker 27 - jepsen.util 4347 :invoke :write [9 43] INFO [2022-03-13 17:18:06,618] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,619] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:18:06,626] jepsen worker 51 - jepsen.util 771 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,627] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:06,630] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,631] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,652] jepsen worker 37 - jepsen.util 2467 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,652] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,653] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,653] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:18:06,669] jepsen worker 6 - jepsen.util 636 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,669] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:06,671] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:06,671] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:06,677] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,678] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,678] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,679] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,687] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,687] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,688] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:18:06,690] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,691] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,692] jepsen worker 71 - jepsen.util 611 :invoke :write [23 6] INFO [2022-03-13 17:18:06,695] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,695] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,696] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:18:06,696] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:18:06,708] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,708] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,708] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,709] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:18:06,709] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,710] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:18:06,714] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,714] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,715] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 19] INFO [2022-03-13 17:18:06,718] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:06,718] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,718] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:06,719] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:06,722] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,722] jepsen worker 61 - jepsen.util 781 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,723] jepsen worker 61 - jepsen.util 781 :invoke :write [20 10] INFO [2022-03-13 17:18:06,780] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,780] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,781] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 28] INFO [2022-03-13 17:18:06,781] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,781] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:18:06,781] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,807] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,808] jepsen worker 69 - jepsen.util 789 :invoke :write [23 14] INFO [2022-03-13 17:18:06,816] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,816] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,816] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,817] jepsen worker 35 - jepsen.util 935 :invoke :write [11 48] INFO [2022-03-13 17:18:06,817] jepsen worker 25 - jepsen.util 565 :invoke :write [8 14] INFO [2022-03-13 17:18:06,817] jepsen worker 34 - jepsen.util 934 :invoke :write [11 11] INFO [2022-03-13 17:18:06,825] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,825] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,825] jepsen worker 12 - jepsen.util 912 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,827] jepsen worker 8 - jepsen.util 728 :invoke :write [2 42] INFO [2022-03-13 17:18:06,828] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:06,829] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,837] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:18:06,838] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] INFO [2022-03-13 17:18:06,841] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,841] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:18:06,845] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,846] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:18:06,891] jepsen worker 78 - jepsen.util 708 :fail :write [26 49] :ServerOutOfMemoryException WARN [2022-03-13 17:18:06,891] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9129961,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17593621,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9243287,"encode_duration_us":17779,"operation_id":"0x107c6","timeout_ms":10000,"last_server_duration_us":9113921,"total_duration_us":10003870},{"operation_name":"upsert","last_dispatch_duration_us":16705450,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17697716,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9243975,"encode_duration_us":18705,"operation_id":"0x10467","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10005373}],"total_count":2}} INFO [2022-03-13 17:18:06,892] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:06,895] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,895] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,895] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:06,895] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:06,898] jepsen worker 71 - jepsen.util 611 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,898] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:06,931] jepsen worker 80 - jepsen.util 800 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,932] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:06,939] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,939] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,940] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:06,941] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:18:06,943] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,943] jepsen worker 37 - jepsen.util 2467 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,943] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:06,944] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:06,945] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,946] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:18:06,983] jepsen worker 69 - jepsen.util 789 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,983] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,984] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:18:06,985] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] WARN [2022-03-13 17:18:06,992] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14437651,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":14437651,"total_server_duration_us":27,"operation_id":"0x106ad","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":14441449},{"operation_name":"upsert","last_dispatch_duration_us":11134848,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11134848,"total_server_duration_us":20760,"encode_duration_us":30717,"operation_id":"0x10892","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10009412},{"operation_name":"upsert","last_dispatch_duration_us":9075228,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9256330,"total_server_duration_us":85127,"encode_duration_us":18909,"operation_id":"0x10d1b","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10009233},{"operation_name":"upsert","last_dispatch_duration_us":9129961,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":17593621,"total_server_duration_us":9243287,"encode_duration_us":17779,"operation_id":"0x107c6","timeout_ms":10000,"last_server_duration_us":9113921,"total_duration_us":10003870},{"operation_name":"upsert","last_dispatch_duration_us":8807858,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8807858,"total_server_duration_us":22471,"encode_duration_us":20428,"operation_id":"0x10900","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10001908},{"operation_name":"get","last_dispatch_duration_us":8692689,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8692689,"total_server_duration_us":534,"operation_id":"0x10d75","timeout_ms":30000,"last_server_duration_us":534,"total_duration_us":8693039},{"operation_name":"get","last_dispatch_duration_us":8176436,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8176436,"total_server_duration_us":84,"operation_id":"0x10aa3","timeout_ms":30000,"last_server_duration_us":84,"total_duration_us":8177871},{"operation_name":"upsert","last_dispatch_duration_us":3100582,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3540159,"total_server_duration_us":151035,"encode_duration_us":18856,"operation_id":"0x10c11","timeout_ms":10000,"last_server_duration_us":79262,"total_duration_us":3590044},{"operation_name":"upsert","last_dispatch_duration_us":422136,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":422136,"total_server_duration_us":62029,"encode_duration_us":25326,"operation_id":"0x10c65","timeout_ms":10000,"last_server_duration_us":62029,"total_duration_us":3554162},{"operation_name":"get","last_dispatch_duration_us":358871,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":358871,"total_server_duration_us":38,"operation_id":"0x10c5c","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":3484439}],"total_count":88}} INFO [2022-03-13 17:18:06,996] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:06,997] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:18:06,999] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,000] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:07,009] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,009] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,010] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,010] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,014] jepsen worker 7 - jepsen.util 637 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,015] jepsen worker 7 - jepsen.util 637 :invoke :write [2 15] INFO [2022-03-13 17:18:07,022] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,023] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:18:07,034] jepsen worker 14 - jepsen.util 374 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,035] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:07,066] jepsen worker 23 - jepsen.util 563 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,067] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:07,076] jepsen worker 19 - jepsen.util 3979 :info :write [6 12] :RequestTimeoutException INFO [2022-03-13 17:18:07,076] jepsen worker 19 - jepsen.util 4069 :invoke :write [6 11] INFO [2022-03-13 17:18:07,083] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,084] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:18:07,093] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,093] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,093] jepsen worker 88 - jepsen.util 2428 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,093] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,094] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,094] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,093] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,094] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:07,095] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,096] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,096] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:07,097] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:07,097] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:07,098] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:18:07,103] jepsen worker 72 - jepsen.util 4122 :info :write [24 10] :RequestTimeoutException INFO [2022-03-13 17:18:07,104] jepsen worker 72 - jepsen.util 4212 :invoke :read [24 nil] INFO [2022-03-13 17:18:07,106] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,106] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,107] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,107] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,107] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:18:07,108] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 26] INFO [2022-03-13 17:18:07,109] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:07,109] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,109] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:07,110] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,111] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,111] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,111] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:18:07,112] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:18:07,113] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,113] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,118] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,119] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,125] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,126] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,142] jepsen worker 24 - jepsen.util 654 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,143] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,144] jepsen worker 51 - jepsen.util 771 :invoke :write [17 42] INFO [2022-03-13 17:18:07,145] jepsen worker 24 - jepsen.util 654 :invoke :write [8 46] INFO [2022-03-13 17:18:07,149] jepsen worker 81 - jepsen.util 4671 :ok :read [27 4] INFO [2022-03-13 17:18:07,151] jepsen worker 81 - jepsen.util 4671 :invoke :read [27 nil] INFO [2022-03-13 17:18:07,181] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,181] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,205] jepsen worker 33 - jepsen.util 753 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,206] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:07,206] jepsen worker 33 - jepsen.util 753 :invoke :write [11 13] INFO [2022-03-13 17:18:07,207] jepsen worker 44 - jepsen.util 674 :invoke :write [14 35] INFO [2022-03-13 17:18:07,210] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,211] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:07,217] jepsen worker 39 - jepsen.util 4449 :ok :read [13 39] INFO [2022-03-13 17:18:07,217] jepsen worker 39 - jepsen.util 4449 :invoke :read [13 nil] INFO [2022-03-13 17:18:07,235] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,236] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:18:07,253] jepsen worker 62 - jepsen.util 512 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,253] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,253] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,253] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,253] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,253] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,253] jepsen worker 61 - jepsen.util 781 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,254] jepsen worker 62 - jepsen.util 512 :invoke :write [20 20] INFO [2022-03-13 17:18:07,254] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,253] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:07,254] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,255] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:07,256] jepsen worker 16 - jepsen.util 4066 :ok :read [5 43] INFO [2022-03-13 17:18:07,256] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:07,257] jepsen worker 78 - jepsen.util 708 :invoke :write [26 42] INFO [2022-03-13 17:18:07,258] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:07,258] jepsen worker 12 - jepsen.util 912 :invoke :write [4 20] INFO [2022-03-13 17:18:07,260] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:07,260] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,261] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,261] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:07,261] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:07,262] jepsen worker 16 - jepsen.util 4066 :invoke :write [5 15] INFO [2022-03-13 17:18:07,263] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] INFO [2022-03-13 17:18:07,264] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,264] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,264] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,265] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,265] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,265] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 45] INFO [2022-03-13 17:18:07,265] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,266] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,266] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:07,266] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:07,267] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:18:07,267] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,267] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,268] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,268] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,268] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 46] INFO [2022-03-13 17:18:07,269] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:07,274] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:07,274] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,275] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:07,275] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,276] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:18:07,276] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,277] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,277] jepsen worker 73 - jepsen.util 4303 :info :write [24 22] :RequestTimeoutException INFO [2022-03-13 17:18:07,278] jepsen worker 73 - jepsen.util 4393 :invoke :write [24 16] INFO [2022-03-13 17:18:07,278] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,279] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,279] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:18:07,279] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,281] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,281] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,282] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,283] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,284] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,284] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,285] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,286] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:18:07,299] jepsen worker 34 - jepsen.util 934 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,299] jepsen worker 34 - jepsen.util 934 :invoke :write [11 39] INFO [2022-03-13 17:18:07,301] jepsen worker 8 - jepsen.util 728 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,302] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:07,306] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:07,306] jepsen worker nemesis - couchbase.util Rebalance check count: 594 INFO [2022-03-13 17:18:07,306] jepsen worker nemesis - couchbase.util Rebalance stuck count: 466 INFO [2022-03-13 17:18:07,309] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,309] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,310] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:07,310] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:07,310] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,311] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:18:07,322] jepsen worker 81 - jepsen.util 4671 :ok :read [27 4] INFO [2022-03-13 17:18:07,323] jepsen worker 81 - jepsen.util 4671 :invoke :read [27 nil] INFO [2022-03-13 17:18:07,337] jepsen worker 9 - jepsen.util 3609 :info :write [3 14] :RequestTimeoutException INFO [2022-03-13 17:18:07,338] jepsen worker 9 - jepsen.util 3699 :invoke :write [3 34] INFO [2022-03-13 17:18:07,350] jepsen worker 10 - jepsen.util 4240 :info :write [3 7] :RequestTimeoutException INFO [2022-03-13 17:18:07,351] jepsen worker 10 - jepsen.util 4330 :invoke :write [3 9] INFO [2022-03-13 17:18:07,370] jepsen worker 39 - jepsen.util 4449 :ok :read [13 39] INFO [2022-03-13 17:18:07,370] jepsen worker 39 - jepsen.util 4449 :invoke :write [13 28] INFO [2022-03-13 17:18:07,384] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:07,384] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,385] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,385] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:07,386] jepsen worker 7 - jepsen.util 637 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,387] jepsen worker 6 - jepsen.util 636 :invoke :write [2 19] INFO [2022-03-13 17:18:07,387] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,387] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,388] jepsen worker 7 - jepsen.util 637 :invoke :write [2 9] INFO [2022-03-13 17:18:07,389] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 10] INFO [2022-03-13 17:18:07,391] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,392] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:18:07,396] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,396] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,403] jepsen worker 25 - jepsen.util 565 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,404] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:18:07,409] jepsen worker 82 - jepsen.util 4312 :ok :read [27 4] INFO [2022-03-13 17:18:07,410] jepsen worker 82 - jepsen.util 4312 :invoke :write [27 4] INFO [2022-03-13 17:18:07,411] jepsen worker 37 - jepsen.util 2467 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,419] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 31] INFO [2022-03-13 17:18:07,447] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,447] jepsen worker 81 - jepsen.util 4671 :ok :read [27 4] INFO [2022-03-13 17:18:07,447] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,448] jepsen worker 81 - jepsen.util 4671 :invoke :write [27 3] INFO [2022-03-13 17:18:07,462] jepsen worker 1 - jepsen.util 3691 :ok :read [0 41] INFO [2022-03-13 17:18:07,463] jepsen worker 1 - jepsen.util 3691 :invoke :read [0 nil] INFO [2022-03-13 17:18:07,473] jepsen worker 67 - jepsen.util 3397 :ok :read [22 30] INFO [2022-03-13 17:18:07,474] jepsen worker 67 - jepsen.util 3397 :invoke :read [22 nil] INFO [2022-03-13 17:18:07,479] jepsen worker 40 - jepsen.util 4360 :ok :read [13 39] INFO [2022-03-13 17:18:07,479] jepsen worker 40 - jepsen.util 4360 :invoke :read [13 nil] INFO [2022-03-13 17:18:07,507] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,508] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:18:07,525] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,525] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:07,544] jepsen worker 79 - jepsen.util 1159 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,544] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,544] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,545] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 21] INFO [2022-03-13 17:18:07,545] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:07,546] jepsen worker 77 - jepsen.util 707 :invoke :write [25 10] INFO [2022-03-13 17:18:07,557] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,557] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:18:07,566] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,567] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,567] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,567] jepsen worker 71 - jepsen.util 611 :invoke :write [23 42] INFO [2022-03-13 17:18:07,570] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,571] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,573] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,574] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:07,576] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,577] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:18:07,581] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,581] jepsen worker 35 - jepsen.util 935 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,581] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,582] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:07,583] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:18:07,583] jepsen worker 80 - jepsen.util 800 :invoke :write [26 19] INFO [2022-03-13 17:18:07,587] jepsen worker 1 - jepsen.util 3691 :ok :read [0 41] INFO [2022-03-13 17:18:07,588] jepsen worker 1 - jepsen.util 3691 :invoke :write [0 31] INFO [2022-03-13 17:18:07,595] jepsen worker 67 - jepsen.util 3397 :ok :read [22 30] INFO [2022-03-13 17:18:07,596] jepsen worker 67 - jepsen.util 3397 :invoke :read [22 nil] INFO [2022-03-13 17:18:07,605] jepsen worker 12 - jepsen.util 912 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,605] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,606] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:07,607] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:07,612] jepsen worker 40 - jepsen.util 4360 :ok :read [13 39] INFO [2022-03-13 17:18:07,613] jepsen worker 40 - jepsen.util 4360 :invoke :read [13 nil] INFO [2022-03-13 17:18:07,638] jepsen worker 38 - jepsen.util 2828 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,638] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,639] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:07,640] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:07,644] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,645] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:18:07,673] jepsen worker 51 - jepsen.util 771 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,673] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:07,676] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,676] jepsen worker 4 - jepsen.util 2614 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,677] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:18:07,677] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:18:07,688] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,690] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:18:07,715] jepsen worker 78 - jepsen.util 708 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,716] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:07,716] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,717] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 44] INFO [2022-03-13 17:18:07,753] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,753] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,753] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,753] jepsen worker 60 - jepsen.util 690 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,753] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,754] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:07,754] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:07,755] jepsen worker 52 - jepsen.util 682 :invoke :write [17 42] INFO [2022-03-13 17:18:07,756] jepsen worker 60 - jepsen.util 690 :invoke :write [20 13] INFO [2022-03-13 17:18:07,757] jepsen worker 53 - jepsen.util 593 :invoke :write [17 17] INFO [2022-03-13 17:18:07,758] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:07,759] jepsen worker 43 - jepsen.util 673 :invoke :write [14 43] INFO [2022-03-13 17:18:07,760] jepsen worker 66 - jepsen.util 3936 :ok :read [22 30] INFO [2022-03-13 17:18:07,761] jepsen worker 66 - jepsen.util 3936 :invoke :read [22 nil] INFO [2022-03-13 17:18:07,770] jepsen worker 24 - jepsen.util 654 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,770] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:07,786] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,787] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:07,789] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,790] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,792] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,793] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,794] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,795] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,799] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,799] jepsen worker 71 - jepsen.util 611 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,800] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,800] jepsen worker 71 - jepsen.util 611 :invoke :write [23 41] INFO [2022-03-13 17:18:07,809] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,811] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,813] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,813] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:18:07,814] jepsen worker 83 - jepsen.util 3773 :ok :read [27 4] INFO [2022-03-13 17:18:07,815] jepsen worker 83 - jepsen.util 3773 :invoke :write [27 18] INFO [2022-03-13 17:18:07,822] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,823] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,828] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,828] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:07,830] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,830] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:18:07,858] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,859] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:18:07,859] jepsen worker 62 - jepsen.util 512 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,860] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:07,860] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,860] jepsen worker 62 - jepsen.util 512 :invoke :write [20 34] INFO [2022-03-13 17:18:07,861] jepsen worker 70 - jepsen.util 880 :invoke :write [23 45] INFO [2022-03-13 17:18:07,861] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:07,891] jepsen worker 66 - jepsen.util 3936 :ok :read [22 30] INFO [2022-03-13 17:18:07,891] jepsen worker 66 - jepsen.util 3936 :invoke :write [22 48] INFO [2022-03-13 17:18:07,897] jepsen worker 44 - jepsen.util 674 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,898] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:07,906] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,907] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:18:07,908] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,908] jepsen worker 13 - jepsen.util 733 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,909] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,909] jepsen worker 13 - jepsen.util 733 :invoke :write [4 40] INFO [2022-03-13 17:18:07,911] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:07,912] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,913] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,913] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:18:07,914] jepsen worker 26 - jepsen.util 566 :invoke :write [8 5] INFO [2022-03-13 17:18:07,920] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,921] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:18:07,922] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:18:07,924] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,924] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:18:07,945] jepsen worker 71 - jepsen.util 611 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,947] jepsen worker 71 - jepsen.util 611 :invoke :write [23 42] INFO [2022-03-13 17:18:07,976] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,976] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:07,978] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:07,979] jepsen worker 61 - jepsen.util 781 :invoke :write [20 28] INFO [2022-03-13 17:18:08,001] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,002] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 14] INFO [2022-03-13 17:18:08,009] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,010] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:18:08,017] jepsen worker 33 - jepsen.util 753 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,017] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,017] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,020] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:08,021] jepsen worker 33 - jepsen.util 753 :invoke :write [11 42] INFO [2022-03-13 17:18:08,022] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:08,029] jepsen worker 37 - jepsen.util 2467 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,030] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 7] INFO [2022-03-13 17:18:08,049] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,050] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:08,063] jepsen worker 7 - jepsen.util 637 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,064] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:08,092] jepsen worker 71 - jepsen.util 611 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,092] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:18:08,105] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,106] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:18:08,109] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,110] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:08,112] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,112] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,114] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:18:08,115] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 24] INFO [2022-03-13 17:18:08,115] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,116] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:18:08,120] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,121] jepsen worker 35 - jepsen.util 935 :invoke :write [11 28] INFO [2022-03-13 17:18:08,137] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,138] jepsen worker 69 - jepsen.util 789 :invoke :write [23 30] INFO [2022-03-13 17:18:08,160] jepsen worker 15 - jepsen.util 3975 :info :write [5 21] :RequestTimeoutException INFO [2022-03-13 17:18:08,160] jepsen worker 30 - jepsen.util 3990 :info :write [10 31] :RequestTimeoutException INFO [2022-03-13 17:18:08,160] jepsen worker 15 - jepsen.util 4065 :invoke :read [5 nil] INFO [2022-03-13 17:18:08,161] jepsen worker 30 - jepsen.util 4080 :invoke :read [10 nil] INFO [2022-03-13 17:18:08,175] jepsen worker 79 - jepsen.util 1159 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,176] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 46] INFO [2022-03-13 17:18:08,193] jepsen worker 38 - jepsen.util 2828 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,193] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:08,202] jepsen worker 88 - jepsen.util 2428 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,202] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:08,205] jepsen worker 29 - jepsen.util 4529 :info :write [9 44] :RequestTimeoutException INFO [2022-03-13 17:18:08,206] jepsen worker 29 - jepsen.util 4619 :invoke :write [9 11] INFO [2022-03-13 17:18:08,216] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,217] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,221] jepsen worker 6 - jepsen.util 636 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,221] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,221] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,222] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:08,221] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,222] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,222] jepsen worker 6 - jepsen.util 636 :invoke :write [2 15] INFO [2022-03-13 17:18:08,223] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:08,224] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:18:08,224] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:08,225] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:18:08,225] jepsen worker 12 - jepsen.util 912 :invoke :write [4 19] INFO [2022-03-13 17:18:08,242] jepsen worker 80 - jepsen.util 800 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,243] jepsen worker 80 - jepsen.util 800 :invoke :write [26 33] INFO [2022-03-13 17:18:08,252] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,253] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:18:08,261] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,261] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:08,270] jepsen worker 86 - jepsen.util 3956 :info :write [28 49] :RequestTimeoutException INFO [2022-03-13 17:18:08,271] jepsen worker 86 - jepsen.util 4046 :invoke :read [28 nil] INFO [2022-03-13 17:18:08,275] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,276] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:18:08,294] jepsen worker 77 - jepsen.util 707 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,294] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,294] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,294] jepsen worker 77 - jepsen.util 707 :invoke :write [25 38] INFO [2022-03-13 17:18:08,295] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:08,295] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:18:08,311] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:08,312] jepsen worker nemesis - couchbase.util Rebalance check count: 595 INFO [2022-03-13 17:18:08,312] jepsen worker nemesis - couchbase.util Rebalance stuck count: 467 INFO [2022-03-13 17:18:08,316] jepsen worker 4 - jepsen.util 2614 :fail :write [1 14] :ServerOutOfMemoryException WARN [2022-03-13 17:18:08,316] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18006813,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18006813,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9039489,"encode_duration_us":17956,"operation_id":"0x10641","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10011957}],"total_count":1}} INFO [2022-03-13 17:18:08,316] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,338] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,338] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:18:08,343] jepsen worker 69 - jepsen.util 789 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,343] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:08,346] jepsen worker 84 - jepsen.util 4404 :info :write [28 0] :RequestTimeoutException INFO [2022-03-13 17:18:08,347] jepsen worker 84 - jepsen.util 4494 :invoke :read [28 nil] INFO [2022-03-13 17:18:08,356] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,356] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,357] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:08,357] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:08,358] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,359] jepsen worker 51 - jepsen.util 771 :invoke :write [17 36] INFO [2022-03-13 17:18:08,359] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:08,360] jepsen worker 44 - jepsen.util 674 :invoke :write [14 44] INFO [2022-03-13 17:18:08,371] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,371] jepsen worker 37 - jepsen.util 2467 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,371] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,373] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 49] INFO [2022-03-13 17:18:08,374] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:18:08,374] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:08,380] jepsen worker 62 - jepsen.util 512 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,380] jepsen worker 62 - jepsen.util 512 :invoke :write [20 47] INFO [2022-03-13 17:18:08,398] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,398] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,398] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,399] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,400] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,401] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:08,403] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,403] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,403] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,404] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:18:08,404] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,404] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,405] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:18:08,406] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,407] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,408] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,409] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 36] INFO [2022-03-13 17:18:08,410] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 40] INFO [2022-03-13 17:18:08,422] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,426] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:18:08,443] jepsen worker 34 - jepsen.util 934 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,443] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,443] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,444] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:08,445] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:08,445] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:08,463] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,463] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,463] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:18:08,466] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:08,471] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,472] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:18:08,484] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,485] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] WARN [2022-03-13 17:18:08,497] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17193288,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17193288,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9034183,"encode_duration_us":17134,"operation_id":"0x108f3","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10000665},{"operation_name":"upsert","last_dispatch_duration_us":24613466,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24613466,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9045859,"encode_duration_us":21810,"operation_id":"0x10293","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10008176}],"total_count":2}} WARN [2022-03-13 17:18:08,497] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16731466,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":16731466,"total_server_duration_us":49,"operation_id":"0x10620","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":16733513},{"operation_name":"get","last_dispatch_duration_us":14558533,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":14558533,"total_server_duration_us":1161,"operation_id":"0x1076a","timeout_ms":30000,"last_server_duration_us":1161,"total_duration_us":14558915},{"operation_name":"get","last_dispatch_duration_us":11188340,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11188340,"total_server_duration_us":38,"operation_id":"0x10cb5","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":11192375},{"operation_name":"upsert","last_dispatch_duration_us":29311,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8971208,"total_server_duration_us":275757,"encode_duration_us":20489,"operation_id":"0x10b6d","timeout_ms":10000,"last_server_duration_us":18723,"total_duration_us":10008484},{"operation_name":"upsert","last_dispatch_duration_us":10442019,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10442019,"total_server_duration_us":20841,"encode_duration_us":20227,"operation_id":"0x10a0f","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10003915},{"operation_name":"upsert","last_dispatch_duration_us":165778,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9274429,"total_server_duration_us":89191,"encode_duration_us":17261,"operation_id":"0x10e28","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10003431},{"operation_name":"upsert","last_dispatch_duration_us":114118,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9296174,"total_server_duration_us":94307,"encode_duration_us":28275,"operation_id":"0x10e1d","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10001465},{"operation_name":"upsert","last_dispatch_duration_us":4989487,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":4989487,"total_server_duration_us":23559,"encode_duration_us":19528,"operation_id":"0x10d92","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10001360},{"operation_name":"get","last_dispatch_duration_us":7022097,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7022097,"total_server_duration_us":27,"operation_id":"0x10f84","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":7024027},{"operation_name":"get","last_dispatch_duration_us":5350616,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5350616,"total_server_duration_us":19,"operation_id":"0x1108b","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":5355151}],"total_count":52}} INFO [2022-03-13 17:18:08,507] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,508] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] WARN [2022-03-13 17:18:08,511] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15848444,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15848444,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9028877,"encode_duration_us":16929,"operation_id":"0x10a29","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10001283},{"operation_name":"upsert","last_dispatch_duration_us":18211408,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18211408,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9039489,"encode_duration_us":29690,"operation_id":"0x107a1","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10005689},{"operation_name":"upsert","last_dispatch_duration_us":9094682,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16631242,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9212030,"encode_duration_us":18791,"operation_id":"0x10448","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10002977}],"total_count":3}} INFO [2022-03-13 17:18:08,515] jepsen worker 33 - jepsen.util 753 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,516] jepsen worker 33 - jepsen.util 753 :invoke :write [11 4] WARN [2022-03-13 17:18:08,516] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15394574,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":15394574,"total_server_duration_us":56,"operation_id":"0x107c9","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":15396593},{"operation_name":"upsert","last_dispatch_duration_us":78047,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7339793,"total_server_duration_us":277853,"encode_duration_us":17650,"operation_id":"0x10d80","timeout_ms":10000,"last_server_duration_us":55061,"total_duration_us":10009784},{"operation_name":"upsert","last_dispatch_duration_us":32338,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":866779,"total_server_duration_us":585172,"encode_duration_us":17297,"operation_id":"0x108b0","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10005181},{"operation_name":"upsert","last_dispatch_duration_us":7019651,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":7019651,"total_server_duration_us":25625,"encode_duration_us":37607,"operation_id":"0x10b51","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10003002},{"operation_name":"upsert","last_dispatch_duration_us":7964827,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8486923,"total_server_duration_us":289027,"encode_duration_us":30837,"operation_id":"0x1092e","timeout_ms":10000,"last_server_duration_us":38274,"total_duration_us":10001649},{"operation_name":"upsert","last_dispatch_duration_us":9048312,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9048312,"total_server_duration_us":9030999,"encode_duration_us":22873,"operation_id":"0x109e6","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":9052184},{"operation_name":"get","last_dispatch_duration_us":8052842,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8052842,"total_server_duration_us":23,"operation_id":"0x10c80","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":8054912},{"operation_name":"get","last_dispatch_duration_us":5500995,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5500995,"total_server_duration_us":6628,"operation_id":"0x10e1a","timeout_ms":30000,"last_server_duration_us":6628,"total_duration_us":5503322},{"operation_name":"upsert","last_dispatch_duration_us":1978451,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1978451,"total_server_duration_us":97174,"encode_duration_us":26539,"operation_id":"0x110e2","timeout_ms":10000,"last_server_duration_us":97174,"total_duration_us":2055970},{"operation_name":"get","last_dispatch_duration_us":1913193,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":1913193,"total_server_duration_us":49,"operation_id":"0x11077","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":1913516}],"total_count":51}} INFO [2022-03-13 17:18:08,519] jepsen worker 61 - jepsen.util 781 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,520] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:08,522] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,523] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:18:08,560] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,561] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:18:08,579] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,579] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] WARN [2022-03-13 17:18:08,598] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":177731,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":5651816,"total_server_duration_us":3883,"operation_id":"0x10274","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":30008309},{"operation_name":"get","last_dispatch_duration_us":18229076,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":18229076,"total_server_duration_us":11739,"operation_id":"0x10544","timeout_ms":30000,"last_server_duration_us":11739,"total_duration_us":18231834},{"operation_name":"get","last_dispatch_duration_us":18223641,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":18223641,"total_server_duration_us":7962,"operation_id":"0x10542","timeout_ms":30000,"last_server_duration_us":7962,"total_duration_us":18227167},{"operation_name":"get","last_dispatch_duration_us":17134955,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":17134955,"total_server_duration_us":1331,"operation_id":"0x105de","timeout_ms":30000,"last_server_duration_us":1331,"total_duration_us":17135367},{"operation_name":"get","last_dispatch_duration_us":10117633,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10117633,"total_server_duration_us":43,"operation_id":"0x10a31","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":10117990},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17753,"operation_id":"0x10ce7","timeout_ms":10000,"total_duration_us":10009295},{"operation_name":"upsert","last_dispatch_duration_us":111361,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9235028,"total_server_duration_us":69724,"encode_duration_us":33003,"operation_id":"0x10e30","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10003930},{"operation_name":"upsert","last_dispatch_duration_us":17193288,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":17193288,"total_server_duration_us":9034183,"encode_duration_us":17134,"operation_id":"0x108f3","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10000665},{"operation_name":"get","last_dispatch_duration_us":9313106,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9313106,"total_server_duration_us":32,"operation_id":"0x10e0d","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9316587},{"operation_name":"get","last_dispatch_duration_us":7974789,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7974789,"total_server_duration_us":69,"operation_id":"0x10f05","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":7977751}],"total_count":17}} WARN [2022-03-13 17:18:08,598] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18160446,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":24920826,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9044574,"encode_duration_us":18406,"operation_id":"0x10281","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10006569},{"operation_name":"upsert","last_dispatch_duration_us":17360613,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18003645,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9408504,"encode_duration_us":21489,"operation_id":"0x10682","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10007424}],"total_count":2}} WARN [2022-03-13 17:18:08,598] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":18338,"operation_id":"0x10877","timeout_ms":10000,"total_duration_us":10010875},{"operation_name":"upsert","last_dispatch_duration_us":9018551,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9018551,"total_server_duration_us":19340,"encode_duration_us":17608,"operation_id":"0x10a70","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10007200},{"operation_name":"upsert","last_dispatch_duration_us":7348279,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8011378,"total_server_duration_us":252548,"encode_duration_us":19320,"operation_id":"0x10d91","timeout_ms":10000,"last_server_duration_us":25362,"total_duration_us":10005889},{"operation_name":"upsert","last_dispatch_duration_us":81571,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":731228,"total_server_duration_us":241469,"encode_duration_us":32898,"operation_id":"0x10d62","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10002815},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":17074,"operation_id":"0x108ff","timeout_ms":10000,"total_duration_us":10001912},{"operation_name":"upsert","last_dispatch_duration_us":7365260,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7677845,"total_server_duration_us":201101,"encode_duration_us":20904,"operation_id":"0x10e23","timeout_ms":10000,"last_server_duration_us":30350,"total_duration_us":10001528},{"operation_name":"upsert","last_dispatch_duration_us":2235293,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2235293,"total_server_duration_us":60759,"encode_duration_us":19297,"operation_id":"0x112d2","timeout_ms":10000,"last_server_duration_us":60759,"total_duration_us":2238829},{"operation_name":"upsert","last_dispatch_duration_us":1646546,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1646546,"total_server_duration_us":189015,"encode_duration_us":19282,"operation_id":"0x11160","timeout_ms":10000,"last_server_duration_us":189015,"total_duration_us":1649930},{"operation_name":"upsert","last_dispatch_duration_us":1295339,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1295339,"total_server_duration_us":50301,"encode_duration_us":20552,"operation_id":"0x1115d","timeout_ms":10000,"last_server_duration_us":50301,"total_duration_us":1300906},{"operation_name":"upsert","last_dispatch_duration_us":1294566,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1294566,"total_server_duration_us":722283,"encode_duration_us":17938,"operation_id":"0x10fc6","timeout_ms":10000,"last_server_duration_us":722283,"total_duration_us":1298334}],"total_count":24}} INFO [2022-03-13 17:18:08,611] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,612] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:18:08,626] jepsen worker 35 - jepsen.util 935 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,626] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:18:08,647] jepsen worker 70 - jepsen.util 880 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,648] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,648] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,648] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:08,648] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:08,649] jepsen worker 7 - jepsen.util 637 :invoke :write [2 25] INFO [2022-03-13 17:18:08,650] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:08,650] jepsen worker 8 - jepsen.util 728 :invoke :write [2 26] INFO [2022-03-13 17:18:08,675] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,676] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,682] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,683] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:18:08,692] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,693] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] WARN [2022-03-13 17:18:08,699] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17515996,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18885019,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9380468,"encode_duration_us":24960,"operation_id":"0x1038c","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10001022},{"operation_name":"upsert","last_dispatch_duration_us":18204886,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27682755,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9110366,"encode_duration_us":18161,"operation_id":"0x102e2","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10010864},{"operation_name":"upsert","last_dispatch_duration_us":18122707,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18122707,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9057542,"encode_duration_us":19507,"operation_id":"0x10733","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10001253}],"total_count":3}} INFO [2022-03-13 17:18:08,708] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,708] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,708] jepsen worker 4 - jepsen.util 2614 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,708] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,709] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,709] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,710] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:18:08,710] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,710] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] WARN [2022-03-13 17:18:08,712] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13457040,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13457040,"total_server_duration_us":49,"operation_id":"0x10b98","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":13459601},{"operation_name":"get","last_dispatch_duration_us":10035511,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10035511,"total_server_duration_us":41445,"operation_id":"0x10cc5","timeout_ms":30000,"last_server_duration_us":41445,"total_duration_us":10037585},{"operation_name":"upsert","last_dispatch_duration_us":7647221,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8367774,"total_server_duration_us":280580,"encode_duration_us":20107,"operation_id":"0x1085b","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10009233},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":33674,"operation_id":"0x10d6d","timeout_ms":10000,"total_duration_us":10006340},{"operation_name":"upsert","last_dispatch_duration_us":9104801,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":17583144,"total_server_duration_us":94718,"encode_duration_us":18621,"operation_id":"0x108f1","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":10004990},{"operation_name":"upsert","last_dispatch_duration_us":15848444,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":15848444,"total_server_duration_us":9028877,"encode_duration_us":16929,"operation_id":"0x10a29","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10001283},{"operation_name":"get","last_dispatch_duration_us":8588716,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8588716,"total_server_duration_us":62,"operation_id":"0x10edd","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":8591069},{"operation_name":"upsert","last_dispatch_duration_us":1989213,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1989213,"total_server_duration_us":83848,"encode_duration_us":21777,"operation_id":"0x11162","timeout_ms":10000,"last_server_duration_us":83848,"total_duration_us":1993459},{"operation_name":"upsert","last_dispatch_duration_us":1722664,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1722664,"total_server_duration_us":69080,"encode_duration_us":26386,"operation_id":"0x10fdc","timeout_ms":10000,"last_server_duration_us":69080,"total_duration_us":1829946},{"operation_name":"upsert","last_dispatch_duration_us":1732511,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1732511,"total_server_duration_us":260754,"encode_duration_us":17970,"operation_id":"0x11161","timeout_ms":10000,"last_server_duration_us":260754,"total_duration_us":1736456}],"total_count":64}} INFO [2022-03-13 17:18:08,714] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,719] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,719] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:18:08,728] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,728] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,728] jepsen worker 80 - jepsen.util 800 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,728] jepsen worker 80 - jepsen.util 800 :invoke :write [26 16] INFO [2022-03-13 17:18:08,729] jepsen worker 24 - jepsen.util 654 :invoke :write [8 26] INFO [2022-03-13 17:18:08,729] jepsen worker 25 - jepsen.util 565 :invoke :write [8 15] INFO [2022-03-13 17:18:08,764] jepsen worker 79 - jepsen.util 1159 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,764] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,764] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,765] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:08,766] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 33] INFO [2022-03-13 17:18:08,766] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:08,770] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,770] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:18:08,771] jepsen worker 88 - jepsen.util 2428 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,771] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:08,796] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,797] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,808] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,809] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:18:08,826] jepsen worker 51 - jepsen.util 771 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,826] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,826] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,827] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:08,827] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:08,828] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:18:08,833] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,834] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:08,835] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,836] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,836] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:18:08,837] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:18:08,853] jepsen worker 6 - jepsen.util 636 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,854] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:08,857] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,857] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:08,894] jepsen worker 37 - jepsen.util 2467 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,894] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,895] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:08,896] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:18:08,899] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,899] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,899] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 33] INFO [2022-03-13 17:18:08,900] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:08,924] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,924] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,924] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,925] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,925] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:08,925] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:18:08,926] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,926] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:18:08,929] jepsen worker 44 - jepsen.util 674 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,929] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,930] jepsen worker 44 - jepsen.util 674 :invoke :write [14 38] INFO [2022-03-13 17:18:08,931] jepsen worker 77 - jepsen.util 707 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,931] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,931] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,931] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:08,932] jepsen worker 77 - jepsen.util 707 :invoke :write [25 16] INFO [2022-03-13 17:18:08,934] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:08,934] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:08,934] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,935] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:18:08,944] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,944] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:18:08,947] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,948] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:08,993] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,993] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,994] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:08,994] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:18:08,998] jepsen worker 12 - jepsen.util 912 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,999] jepsen worker 33 - jepsen.util 753 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:08,999] jepsen worker 33 - jepsen.util 753 :invoke :write [11 6] INFO [2022-03-13 17:18:09,000] jepsen worker 12 - jepsen.util 912 :invoke :write [4 25] INFO [2022-03-13 17:18:09,000] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,001] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:18:09,028] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,029] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,029] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:09,030] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:18:09,033] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,033] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:09,036] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,037] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:18:09,041] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,042] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:09,067] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,068] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,068] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,068] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:09,069] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:09,069] jepsen worker 78 - jepsen.util 708 :invoke :write [26 24] INFO [2022-03-13 17:18:09,079] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,080] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:09,082] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,083] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 13] INFO [2022-03-13 17:18:09,134] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,134] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:09,152] jepsen worker 24 - jepsen.util 654 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,152] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:09,156] jepsen worker 62 - jepsen.util 512 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,156] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,156] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:09,156] jepsen worker 62 - jepsen.util 512 :invoke :write [20 26] INFO [2022-03-13 17:18:09,158] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:09,159] jepsen worker 42 - jepsen.util 402 :invoke :write [14 1] INFO [2022-03-13 17:18:09,210] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,210] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,211] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:09,211] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:18:09,231] jepsen worker 79 - jepsen.util 1159 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,232] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 5] INFO [2022-03-13 17:18:09,235] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,236] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:09,251] jepsen worker 25 - jepsen.util 565 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,252] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:09,254] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,255] jepsen worker 51 - jepsen.util 771 :invoke :write [17 25] INFO [2022-03-13 17:18:09,268] jepsen worker 7 - jepsen.util 637 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,269] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:09,316] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:09,316] jepsen worker nemesis - couchbase.util Rebalance check count: 596 INFO [2022-03-13 17:18:09,317] jepsen worker nemesis - couchbase.util Rebalance stuck count: 468 INFO [2022-03-13 17:18:09,443] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,444] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,444] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 37] INFO [2022-03-13 17:18:09,445] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,489] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,490] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,491] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,492] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,493] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,493] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,496] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,497] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:09,510] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,511] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,512] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,512] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,513] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,514] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,527] jepsen worker 38 - jepsen.util 2828 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,527] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,528] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:18:09,528] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:09,529] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,530] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,530] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,531] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:18:09,537] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,538] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,538] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,538] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:18:09,538] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:18:09,539] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:09,548] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,549] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:09,550] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,551] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:18:09,561] jepsen worker 88 - jepsen.util 2428 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,562] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 11] INFO [2022-03-13 17:18:09,577] jepsen worker 17 - jepsen.util 4247 :info :write [5 35] :RequestTimeoutException INFO [2022-03-13 17:18:09,577] jepsen worker 17 - jepsen.util 4337 :invoke :write [5 3] INFO [2022-03-13 17:18:09,622] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,623] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:09,622] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,623] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:09,624] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,625] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,626] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:18:09,626] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:18:09,655] jepsen worker 37 - jepsen.util 2467 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,656] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:09,670] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,671] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:18:09,685] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,685] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:18:09,741] jepsen worker 59 - jepsen.util 4199 :ok :read [19 6] INFO [2022-03-13 17:18:09,741] jepsen worker 59 - jepsen.util 4199 :invoke :read [19 nil] INFO [2022-03-13 17:18:09,750] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,768] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,768] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,768] jepsen worker 88 - jepsen.util 2428 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,829] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 16] INFO [2022-03-13 17:18:09,786] jepsen worker 85 - jepsen.util 3865 :ok :read [28 44] INFO [2022-03-13 17:18:09,787] jepsen worker 15 - jepsen.util 4065 :ok :read [5 43] INFO [2022-03-13 17:18:09,829] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:18:09,829] jepsen worker 85 - jepsen.util 3865 :invoke :read [28 nil] INFO [2022-03-13 17:18:09,829] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:09,829] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:09,831] jepsen worker 15 - jepsen.util 4065 :invoke :read [5 nil] INFO [2022-03-13 17:18:09,844] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,845] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:18:09,870] jepsen worker 59 - jepsen.util 4199 :ok :read [19 6] INFO [2022-03-13 17:18:09,871] jepsen worker 59 - jepsen.util 4199 :invoke :write [19 12] INFO [2022-03-13 17:18:09,874] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,875] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:18:09,875] jepsen worker 15 - jepsen.util 4065 :ok :read [5 43] INFO [2022-03-13 17:18:09,876] jepsen worker 15 - jepsen.util 4065 :invoke :read [5 nil] INFO [2022-03-13 17:18:09,904] jepsen worker 15 - jepsen.util 4065 :ok :read [5 43] INFO [2022-03-13 17:18:09,905] jepsen worker 15 - jepsen.util 4065 :invoke :write [5 27] INFO [2022-03-13 17:18:09,914] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,914] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,915] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:18:09,916] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:09,917] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,917] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,917] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,918] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:09,919] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:09,920] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,920] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,920] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:18:09,921] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 19] INFO [2022-03-13 17:18:09,922] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:18:09,951] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:09,951] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,009] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,010] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:18:10,011] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,011] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,012] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,012] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,013] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,013] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,014] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,014] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:18:10,015] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,015] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,016] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,017] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,019] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,020] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:18:10,088] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,088] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:18:10,089] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,089] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:10,093] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,094] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:10,095] jepsen worker 77 - jepsen.util 707 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,095] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,096] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,096] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:10,097] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:10,097] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:10,098] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,098] jepsen worker 71 - jepsen.util 611 :invoke :write [23 40] INFO [2022-03-13 17:18:10,102] jepsen worker 3 - jepsen.util 2253 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,102] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,107] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,108] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,110] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,110] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,112] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,112] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,113] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,114] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,116] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,117] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,119] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,119] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:18:10,139] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,140] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,197] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,197] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,200] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,200] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:18:10,211] jepsen worker 80 - jepsen.util 800 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,211] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,211] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,211] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,212] jepsen worker 80 - jepsen.util 800 :invoke :write [26 5] INFO [2022-03-13 17:18:10,212] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:10,213] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:18:10,213] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:10,242] jepsen worker 89 - jepsen.util 2789 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,243] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,243] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:18:10,243] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,245] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,245] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:18:10,255] jepsen worker 8 - jepsen.util 728 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,256] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,256] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:10,257] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,257] jepsen worker 6 - jepsen.util 636 :invoke :write [2 17] INFO [2022-03-13 17:18:10,257] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:18:10,267] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,267] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,267] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,268] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,269] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:18:10,269] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,274] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,274] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,275] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:18:10,276] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,277] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,277] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 7] INFO [2022-03-13 17:18:10,286] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,287] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,317] jepsen worker 44 - jepsen.util 674 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,317] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:10,323] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:10,323] jepsen worker nemesis - couchbase.util Rebalance check count: 597 INFO [2022-03-13 17:18:10,323] jepsen worker nemesis - couchbase.util Rebalance stuck count: 469 INFO [2022-03-13 17:18:10,324] jepsen worker 68 - jepsen.util 4208 :info :write [22 3] :RequestTimeoutException INFO [2022-03-13 17:18:10,325] jepsen worker 68 - jepsen.util 4298 :invoke :write [22 18] INFO [2022-03-13 17:18:10,338] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,338] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,347] jepsen worker 71 - jepsen.util 611 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,347] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] INFO [2022-03-13 17:18:10,354] jepsen worker 78 - jepsen.util 708 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,355] jepsen worker 78 - jepsen.util 708 :invoke :write [26 42] INFO [2022-03-13 17:18:10,397] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,397] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,399] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,399] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,400] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,400] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,401] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:18:10,403] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:18:10,436] jepsen worker 33 - jepsen.util 753 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,436] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,436] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,436] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,437] jepsen worker 33 - jepsen.util 753 :invoke :write [11 11] INFO [2022-03-13 17:18:10,438] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:18:10,439] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:10,440] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:10,455] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,456] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:18:10,461] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,462] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,467] jepsen worker 12 - jepsen.util 912 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,467] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,468] jepsen worker 12 - jepsen.util 912 :invoke :write [4 26] INFO [2022-03-13 17:18:10,475] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,476] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:18:10,482] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:10,506] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,507] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:18:10,539] jepsen worker 51 - jepsen.util 771 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,539] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,539] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,540] jepsen worker 34 - jepsen.util 934 :invoke :write [11 47] INFO [2022-03-13 17:18:10,541] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:10,542] jepsen worker 51 - jepsen.util 771 :invoke :write [17 48] INFO [2022-03-13 17:18:10,550] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,551] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:10,557] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,557] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,558] jepsen worker 79 - jepsen.util 1159 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,558] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,558] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,559] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:10,560] jepsen worker 61 - jepsen.util 781 :invoke :write [20 49] INFO [2022-03-13 17:18:10,561] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:10,561] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,562] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:10,594] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,595] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,607] jepsen worker 37 - jepsen.util 2467 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,607] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,608] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,608] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,613] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,613] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,614] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,616] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:18:10,619] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,619] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,620] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,620] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:18:10,622] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,623] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:18:10,629] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,630] jepsen worker 71 - jepsen.util 611 :invoke :write [23 42] INFO [2022-03-13 17:18:10,630] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,637] jepsen worker 62 - jepsen.util 512 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,637] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:18:10,638] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:10,640] jepsen worker 57 - jepsen.util 4107 :info :write [19 9] :RequestTimeoutException INFO [2022-03-13 17:18:10,641] jepsen worker 57 - jepsen.util 4197 :invoke :write [19 47] INFO [2022-03-13 17:18:10,690] jepsen worker 2 - jepsen.util 4322 :info :write [0 13] :RequestTimeoutException INFO [2022-03-13 17:18:10,691] jepsen worker 2 - jepsen.util 4412 :invoke :write [0 23] INFO [2022-03-13 17:18:10,697] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,697] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:10,698] jepsen worker 24 - jepsen.util 654 :invoke :write [8 37] INFO [2022-03-13 17:18:10,697] jepsen worker 80 - jepsen.util 800 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,698] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,701] jepsen worker 44 - jepsen.util 674 :invoke :write [14 19] INFO [2022-03-13 17:18:10,703] jepsen worker 80 - jepsen.util 800 :invoke :write [26 28] INFO [2022-03-13 17:18:10,706] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,707] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,707] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:18:10,709] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,711] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,711] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 10] INFO [2022-03-13 17:18:10,712] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 37] INFO [2022-03-13 17:18:10,720] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,721] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:18:10,721] jepsen worker 78 - jepsen.util 708 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,722] jepsen worker 78 - jepsen.util 708 :invoke :write [26 12] INFO [2022-03-13 17:18:10,723] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,723] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,724] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,724] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 16] INFO [2022-03-13 17:18:10,724] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:10,727] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,729] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:10,734] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,734] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,734] jepsen worker 4 - jepsen.util 2614 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,734] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,735] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,737] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,738] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,738] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,739] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,739] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,739] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:18:10,741] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,742] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,743] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,744] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,745] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:18:10,747] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:18:10,768] jepsen worker 71 - jepsen.util 611 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,768] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:10,800] jepsen worker 42 - jepsen.util 402 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,800] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,800] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,800] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,800] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,801] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,802] jepsen worker 7 - jepsen.util 637 :invoke :write [2 26] INFO [2022-03-13 17:18:10,802] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:10,803] jepsen worker 42 - jepsen.util 402 :invoke :write [14 32] INFO [2022-03-13 17:18:10,804] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,804] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:10,804] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:10,805] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,805] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,806] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:18:10,807] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:18:10,816] jepsen worker 33 - jepsen.util 753 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,816] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:10,878] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,878] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,878] jepsen worker 61 - jepsen.util 781 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,878] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,879] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:10,879] jepsen worker 58 - jepsen.util 3748 :info :write [19 47] :RequestTimeoutException INFO [2022-03-13 17:18:10,881] jepsen worker 23 - jepsen.util 563 :invoke :write [7 11] INFO [2022-03-13 17:18:10,881] jepsen worker 58 - jepsen.util 3838 :invoke :read [19 nil] INFO [2022-03-13 17:18:10,882] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:10,882] jepsen worker 61 - jepsen.util 781 :invoke :write [20 23] INFO [2022-03-13 17:18:10,888] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,888] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,888] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,889] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:18:10,889] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:10,890] jepsen worker 71 - jepsen.util 611 :invoke :write [23 40] INFO [2022-03-13 17:18:10,944] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,944] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,945] jepsen worker 21 - jepsen.util 831 :invoke :write [7 44] INFO [2022-03-13 17:18:10,946] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:10,973] jepsen worker 6 - jepsen.util 636 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,973] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,974] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:10,975] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:10,980] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,981] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:10,981] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:10,981] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:18:11,021] jepsen worker 71 - jepsen.util 611 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,022] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:18:11,025] jepsen worker 53 - jepsen.util 593 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,025] jepsen worker 76 - jepsen.util 796 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,025] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:11,026] jepsen worker 76 - jepsen.util 796 :invoke :write [25 11] INFO [2022-03-13 17:18:11,026] jepsen worker 34 - jepsen.util 934 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,027] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:11,053] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,053] jepsen worker 88 - jepsen.util 2428 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,053] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,054] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,055] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,056] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,058] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,058] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:18:11,074] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,075] jepsen worker 78 - jepsen.util 708 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,075] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,076] jepsen worker 78 - jepsen.util 708 :invoke :write [26 10] INFO [2022-03-13 17:18:11,080] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,081] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,110] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,111] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,111] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:18:11,112] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:11,135] jepsen worker 51 - jepsen.util 771 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,136] jepsen worker 51 - jepsen.util 771 :invoke :write [17 2] INFO [2022-03-13 17:18:11,140] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,140] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,141] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,141] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:18:11,142] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,142] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,143] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,144] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,145] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,145] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:18:11,149] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,149] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,149] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:11,150] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:11,157] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,158] jepsen worker 62 - jepsen.util 512 :invoke :write [20 45] INFO [2022-03-13 17:18:11,158] jepsen worker 12 - jepsen.util 912 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,159] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,159] jepsen worker 12 - jepsen.util 912 :invoke :write [4 4] INFO [2022-03-13 17:18:11,161] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:11,182] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,182] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,182] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,182] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:18:11,183] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:11,184] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 2] INFO [2022-03-13 17:18:11,185] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,185] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:18:11,219] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,219] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,219] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:11,220] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,222] jepsen worker 37 - jepsen.util 2467 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,222] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 0] INFO [2022-03-13 17:18:11,223] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,223] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,223] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:18:11,224] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,224] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,225] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 1] INFO [2022-03-13 17:18:11,225] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,225] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,227] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,227] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:11,228] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,229] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:18:11,231] jepsen worker 74 - jepsen.util 4034 :ok :read [24 24] INFO [2022-03-13 17:18:11,232] jepsen worker 74 - jepsen.util 4034 :invoke :read [24 nil] INFO [2022-03-13 17:18:11,232] jepsen worker 31 - jepsen.util 4261 :ok :read [10 :nil] INFO [2022-03-13 17:18:11,233] jepsen worker 31 - jepsen.util 4261 :invoke :read [10 nil] INFO [2022-03-13 17:18:11,245] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,245] jepsen worker 69 - jepsen.util 789 :invoke :write [23 46] INFO [2022-03-13 17:18:11,250] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,251] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:11,254] jepsen worker 79 - jepsen.util 1159 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,254] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,254] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:11,255] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 49] INFO [2022-03-13 17:18:11,297] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,297] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:18:11,313] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,313] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:11,317] jepsen worker 44 - jepsen.util 674 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,318] jepsen worker 44 - jepsen.util 674 :invoke :write [14 24] INFO [2022-03-13 17:18:11,331] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:11,331] jepsen worker nemesis - couchbase.util Rebalance check count: 598 INFO [2022-03-13 17:18:11,331] jepsen worker nemesis - couchbase.util Rebalance stuck count: 470 INFO [2022-03-13 17:18:11,343] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,344] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,352] jepsen worker 7 - jepsen.util 637 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,353] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:11,378] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,379] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:18:11,386] jepsen worker 24 - jepsen.util 654 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,387] jepsen worker 24 - jepsen.util 654 :invoke :write [8 13] INFO [2022-03-13 17:18:11,417] jepsen worker 63 - jepsen.util 4203 :info :write [21 28] :SyncWriteAmbiguous INFO [2022-03-13 17:18:11,418] jepsen worker 63 - jepsen.util 4293 :invoke :write [21 38] INFO [2022-03-13 17:18:11,436] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,436] jepsen worker 69 - jepsen.util 789 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,436] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:18:11,437] jepsen worker 69 - jepsen.util 789 :invoke :write [23 30] INFO [2022-03-13 17:18:11,440] jepsen worker 36 - jepsen.util 2466 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,442] jepsen worker 49 - jepsen.util 3559 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:18:11,442] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:18:11,443] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 29] INFO [2022-03-13 17:18:11,451] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,451] jepsen worker 23 - jepsen.util 563 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,451] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:11,452] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:11,533] jepsen worker 42 - jepsen.util 402 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,533] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,533] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,533] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,534] jepsen worker 6 - jepsen.util 636 :invoke :write [2 47] INFO [2022-03-13 17:18:11,536] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:11,536] jepsen worker 70 - jepsen.util 880 :invoke :write [23 2] INFO [2022-03-13 17:18:11,537] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:11,543] jepsen worker 80 - jepsen.util 800 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,543] jepsen worker 69 - jepsen.util 789 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,544] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:11,545] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,546] jepsen worker 30 - jepsen.util 4080 :ok :read [10 :nil] INFO [2022-03-13 17:18:11,546] jepsen worker 30 - jepsen.util 4080 :invoke :write [10 21] INFO [2022-03-13 17:18:11,609] jepsen worker 61 - jepsen.util 781 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,610] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:11,617] jepsen worker 51 - jepsen.util 771 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,618] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:11,637] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,637] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,638] jepsen worker 71 - jepsen.util 611 :invoke :write [23 41] INFO [2022-03-13 17:18:11,638] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,640] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,640] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:11,641] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,642] jepsen worker 69 - jepsen.util 789 :invoke :write [23 16] INFO [2022-03-13 17:18:11,674] jepsen worker 62 - jepsen.util 512 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,675] jepsen worker 62 - jepsen.util 512 :invoke :write [20 44] INFO [2022-03-13 17:18:11,694] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,694] jepsen worker 21 - jepsen.util 831 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,695] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:18:11,695] jepsen worker 14 - jepsen.util 374 :invoke :write [4 35] INFO [2022-03-13 17:18:11,740] jepsen worker 71 - jepsen.util 611 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,740] jepsen worker 71 - jepsen.util 611 :invoke :write [23 5] INFO [2022-03-13 17:18:11,757] jepsen worker 78 - jepsen.util 708 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,758] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,758] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:11,758] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:11,762] jepsen worker 44 - jepsen.util 674 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,763] jepsen worker 44 - jepsen.util 674 :invoke :write [14 24] INFO [2022-03-13 17:18:11,804] jepsen worker 69 - jepsen.util 789 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,805] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:18:11,948] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,948] jepsen worker 12 - jepsen.util 912 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,948] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:11,948] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,948] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:11,949] jepsen worker 7 - jepsen.util 637 :invoke :write [2 25] INFO [2022-03-13 17:18:11,950] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:11,955] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:11,960] jepsen worker 24 - jepsen.util 654 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,960] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,961] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,961] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,961] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,961] jepsen worker 24 - jepsen.util 654 :invoke :write [8 43] INFO [2022-03-13 17:18:11,961] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,961] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:11,962] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:11,963] jepsen worker 51 - jepsen.util 771 :invoke :write [17 11] INFO [2022-03-13 17:18:11,963] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:18:11,974] jepsen worker 80 - jepsen.util 800 :invoke :write [26 34] INFO [2022-03-13 17:18:11,976] jepsen worker 71 - jepsen.util 611 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:11,976] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:12,020] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,020] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,021] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:18:12,021] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:12,023] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,024] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:12,025] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,025] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:18:12,036] jepsen worker 65 - jepsen.util 4385 :info :write [21 26] :RequestTimeoutException INFO [2022-03-13 17:18:12,036] jepsen worker 65 - jepsen.util 4475 :invoke :read [21 nil] INFO [2022-03-13 17:18:12,056] jepsen worker 56 - jepsen.util 4016 :info :write [18 7] :RequestTimeoutException INFO [2022-03-13 17:18:12,057] jepsen worker 56 - jepsen.util 4106 :invoke :read [18 nil] INFO [2022-03-13 17:18:12,057] jepsen worker 79 - jepsen.util 1159 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,057] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,057] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,058] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,058] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 17] INFO [2022-03-13 17:18:12,058] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,059] jepsen worker 61 - jepsen.util 781 :invoke :write [20 12] INFO [2022-03-13 17:18:12,059] jepsen worker 77 - jepsen.util 707 :invoke :write [25 47] INFO [2022-03-13 17:18:12,060] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:12,065] jepsen worker 78 - jepsen.util 708 :invoke :write [26 47] INFO [2022-03-13 17:18:12,070] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,071] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:12,121] jepsen worker 6 - jepsen.util 636 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,121] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:12,135] jepsen worker 45 - jepsen.util 4275 :info :write [15 27] :RequestTimeoutException INFO [2022-03-13 17:18:12,136] jepsen worker 45 - jepsen.util 4365 :invoke :read [15 nil] INFO [2022-03-13 17:18:12,152] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,153] jepsen worker 23 - jepsen.util 563 :invoke :write [7 30] INFO [2022-03-13 17:18:12,155] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,155] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:18:12,166] jepsen worker 45 - jepsen.util 4365 :ok :read [15 48] INFO [2022-03-13 17:18:12,167] jepsen worker 45 - jepsen.util 4365 :invoke :read [15 nil] INFO [2022-03-13 17:18:12,168] jepsen worker 14 - jepsen.util 374 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,169] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:18:12,193] jepsen worker 45 - jepsen.util 4365 :ok :read [15 48] INFO [2022-03-13 17:18:12,194] jepsen worker 45 - jepsen.util 4365 :invoke :write [15 10] INFO [2022-03-13 17:18:12,205] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,206] jepsen worker 69 - jepsen.util 789 :invoke :write [23 43] INFO [2022-03-13 17:18:12,284] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,285] jepsen worker 71 - jepsen.util 611 :invoke :write [23 9] INFO [2022-03-13 17:18:12,288] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,288] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,288] jepsen worker 44 - jepsen.util 674 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,288] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,289] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:12,290] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:12,290] jepsen worker 33 - jepsen.util 753 :invoke :write [11 29] INFO [2022-03-13 17:18:12,291] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:18:12,306] jepsen worker 70 - jepsen.util 880 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,306] jepsen worker 70 - jepsen.util 880 :invoke :write [23 0] INFO [2022-03-13 17:18:12,336] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:12,336] jepsen worker nemesis - couchbase.util Rebalance check count: 599 INFO [2022-03-13 17:18:12,336] jepsen worker nemesis - couchbase.util Rebalance stuck count: 471 INFO [2022-03-13 17:18:12,356] jepsen worker 79 - jepsen.util 1159 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,356] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 25] INFO [2022-03-13 17:18:12,374] jepsen worker 0 - jepsen.util 3960 :info :write [0 41] :RequestTimeoutException INFO [2022-03-13 17:18:12,374] jepsen worker 0 - jepsen.util 4050 :invoke :read [0 nil] INFO [2022-03-13 17:18:12,379] jepsen worker 24 - jepsen.util 654 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,380] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:12,387] jepsen worker 69 - jepsen.util 789 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,388] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:12,400] jepsen worker 18 - jepsen.util 4068 :info :write [6 16] :RequestTimeoutException INFO [2022-03-13 17:18:12,400] jepsen worker 18 - jepsen.util 4158 :invoke :read [6 nil] INFO [2022-03-13 17:18:12,414] jepsen worker 51 - jepsen.util 771 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,415] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:12,434] jepsen worker 0 - jepsen.util 4050 :ok :read [0 41] INFO [2022-03-13 17:18:12,434] jepsen worker 0 - jepsen.util 4050 :invoke :write [0 25] INFO [2022-03-13 17:18:12,489] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,490] jepsen worker 71 - jepsen.util 611 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,490] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:18:12,490] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,490] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:18:12,491] jepsen worker 69 - jepsen.util 789 :invoke :write [23 6] INFO [2022-03-13 17:18:12,535] jepsen worker 41 - jepsen.util 4181 :ok :read [13 39] INFO [2022-03-13 17:18:12,536] jepsen worker 41 - jepsen.util 4181 :invoke :read [13 nil] INFO [2022-03-13 17:18:12,542] jepsen worker 84 - jepsen.util 4494 :ok :read [28 44] INFO [2022-03-13 17:18:12,542] jepsen worker 84 - jepsen.util 4494 :invoke :write [28 8] INFO [2022-03-13 17:18:12,543] jepsen worker 62 - jepsen.util 512 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,544] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,544] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:12,544] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,544] jepsen worker 61 - jepsen.util 781 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,545] jepsen worker 62 - jepsen.util 512 :invoke :write [20 22] INFO [2022-03-13 17:18:12,546] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:12,547] jepsen worker 42 - jepsen.util 402 :invoke :write [14 43] INFO [2022-03-13 17:18:12,547] jepsen worker 12 - jepsen.util 912 :invoke :write [4 11] INFO [2022-03-13 17:18:12,550] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:12,562] jepsen worker 41 - jepsen.util 4181 :ok :read [13 39] INFO [2022-03-13 17:18:12,562] jepsen worker 41 - jepsen.util 4181 :invoke :read [13 nil] INFO [2022-03-13 17:18:12,575] jepsen worker 80 - jepsen.util 800 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,575] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,576] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:12,576] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:12,594] jepsen worker 41 - jepsen.util 4181 :ok :read [13 39] INFO [2022-03-13 17:18:12,595] jepsen worker 41 - jepsen.util 4181 :invoke :write [13 1] INFO [2022-03-13 17:18:12,610] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,610] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:12,670] jepsen worker 69 - jepsen.util 789 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,670] jepsen worker 7 - jepsen.util 637 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,671] jepsen worker 77 - jepsen.util 707 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,671] jepsen worker 69 - jepsen.util 789 :invoke :write [23 39] INFO [2022-03-13 17:18:12,671] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,671] jepsen worker 77 - jepsen.util 707 :invoke :write [25 3] INFO [2022-03-13 17:18:12,672] jepsen worker 6 - jepsen.util 636 :invoke :write [2 10] INFO [2022-03-13 17:18:12,672] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:12,673] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,673] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:18:12,683] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,683] jepsen worker 34 - jepsen.util 934 :invoke :write [11 8] INFO [2022-03-13 17:18:12,684] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,684] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:12,685] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:12,685] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:12,737] jepsen worker 33 - jepsen.util 753 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,738] jepsen worker 33 - jepsen.util 753 :invoke :write [11 31] INFO [2022-03-13 17:18:12,785] jepsen worker 69 - jepsen.util 789 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,786] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:18:12,815] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,815] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,815] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,816] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:12,816] jepsen worker 24 - jepsen.util 654 :invoke :write [8 9] INFO [2022-03-13 17:18:12,816] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:12,847] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,848] jepsen worker 70 - jepsen.util 880 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,849] jepsen worker 78 - jepsen.util 708 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,849] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:18:12,849] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:12,849] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:12,851] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,851] jepsen worker 8 - jepsen.util 728 :invoke :write [2 4] INFO [2022-03-13 17:18:12,913] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,914] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:12,936] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,937] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,937] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,937] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:12,937] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:12,937] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:18:12,938] jepsen worker 44 - jepsen.util 674 :invoke :write [14 15] INFO [2022-03-13 17:18:12,939] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:18:12,939] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:12,940] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:12,944] jepsen worker 46 - jepsen.util 4186 :info :write [15 4] :SyncWriteAmbiguous INFO [2022-03-13 17:18:12,945] jepsen worker 46 - jepsen.util 4276 :invoke :read [15 nil] INFO [2022-03-13 17:18:13,012] jepsen worker 62 - jepsen.util 512 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,012] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:13,023] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,023] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,024] jepsen worker 69 - jepsen.util 789 :invoke :write [23 30] INFO [2022-03-13 17:18:13,024] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:13,080] jepsen worker 34 - jepsen.util 934 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,081] jepsen worker 34 - jepsen.util 934 :invoke :write [11 24] INFO [2022-03-13 17:18:13,085] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,086] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:18:13,098] jepsen worker 23 - jepsen.util 563 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,098] jepsen worker 23 - jepsen.util 563 :invoke :write [7 4] INFO [2022-03-13 17:18:13,112] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,113] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:18:13,117] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,118] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:18:13,130] jepsen worker 60 - jepsen.util 690 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,130] jepsen worker 75 - jepsen.util 525 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,131] jepsen worker 60 - jepsen.util 690 :invoke :write [20 32] INFO [2022-03-13 17:18:13,131] jepsen worker 75 - jepsen.util 525 :invoke :write [25 32] INFO [2022-03-13 17:18:13,142] jepsen worker 33 - jepsen.util 753 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,142] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,143] jepsen worker 33 - jepsen.util 753 :invoke :write [11 29] INFO [2022-03-13 17:18:13,144] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,144] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:13,144] jepsen worker 35 - jepsen.util 935 :invoke :write [11 2] INFO [2022-03-13 17:18:13,179] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,180] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:18:13,193] jepsen worker 56 - jepsen.util 4106 :ok :read [18 21] INFO [2022-03-13 17:18:13,193] jepsen worker 69 - jepsen.util 789 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,194] jepsen worker 56 - jepsen.util 4106 :invoke :write [18 28] INFO [2022-03-13 17:18:13,194] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:18:13,202] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,202] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,203] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,203] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,227] jepsen worker 88 - jepsen.util 2428 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,227] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 5] INFO [2022-03-13 17:18:13,250] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,250] jepsen worker 24 - jepsen.util 654 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,251] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,252] jepsen worker 80 - jepsen.util 800 :invoke :write [26 35] INFO [2022-03-13 17:18:13,253] jepsen worker 24 - jepsen.util 654 :invoke :write [8 2] INFO [2022-03-13 17:18:13,255] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:13,266] jepsen worker 49 - jepsen.util 3559 :fail :write [16 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,267] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 24] INFO [2022-03-13 17:18:13,276] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,277] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:18:13,305] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,305] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,305] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,306] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:18:13,306] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,307] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,307] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,308] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,308] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,309] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:18:13,309] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,310] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,311] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,311] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 12] INFO [2022-03-13 17:18:13,318] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,319] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,327] jepsen worker 42 - jepsen.util 402 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,327] jepsen worker 79 - jepsen.util 1159 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,328] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,328] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:13,329] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:13,329] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:13,340] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,340] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:18:13,342] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:13,342] jepsen worker nemesis - couchbase.util Rebalance check count: 600 INFO [2022-03-13 17:18:13,342] jepsen worker nemesis - couchbase.util Rebalance stuck count: 472 INFO [2022-03-13 17:18:13,343] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,343] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:13,359] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:18:13,360] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 9] INFO [2022-03-13 17:18:13,378] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,378] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,379] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:13,380] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:13,403] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,405] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:18:13,439] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,439] jepsen worker 69 - jepsen.util 789 :invoke :write [23 11] INFO [2022-03-13 17:18:13,454] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,455] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:13,455] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,456] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:18:13,483] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,483] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:13,489] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,489] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:13,491] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,491] jepsen worker 12 - jepsen.util 912 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,491] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,491] jepsen worker 77 - jepsen.util 707 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,492] jepsen worker 7 - jepsen.util 637 :invoke :write [2 47] INFO [2022-03-13 17:18:13,492] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:13,493] jepsen worker 12 - jepsen.util 912 :invoke :write [4 40] INFO [2022-03-13 17:18:13,494] jepsen worker 78 - jepsen.util 708 :invoke :write [26 20] INFO [2022-03-13 17:18:13,512] jepsen worker 49 - jepsen.util 3559 :fail :write [16 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,512] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,513] jepsen worker 49 - jepsen.util 3559 :invoke :read [16 nil] INFO [2022-03-13 17:18:13,514] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:13,515] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,516] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:13,540] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,540] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,541] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:18:13,541] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:13,542] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,543] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:18:13,543] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,543] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:13,567] jepsen worker 44 - jepsen.util 674 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,569] jepsen worker 44 - jepsen.util 674 :invoke :write [14 43] INFO [2022-03-13 17:18:13,631] jepsen worker 69 - jepsen.util 789 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,631] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,631] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:13,631] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:18:13,634] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,635] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:18:13,640] jepsen worker 37 - jepsen.util 2467 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,640] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:13,651] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,652] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:13,653] jepsen worker 6 - jepsen.util 636 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,653] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,653] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,653] jepsen worker 23 - jepsen.util 563 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,654] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,654] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:13,655] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:13,656] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:13,656] jepsen worker 61 - jepsen.util 781 :invoke :write [20 15] INFO [2022-03-13 17:18:13,657] jepsen worker 6 - jepsen.util 636 :invoke :write [2 0] INFO [2022-03-13 17:18:13,657] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,658] jepsen worker 77 - jepsen.util 707 :invoke :write [25 24] INFO [2022-03-13 17:18:13,661] jepsen worker 34 - jepsen.util 934 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,661] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,661] jepsen worker 51 - jepsen.util 771 :invoke :write [17 21] INFO [2022-03-13 17:18:13,662] jepsen worker 34 - jepsen.util 934 :invoke :write [11 0] INFO [2022-03-13 17:18:13,678] jepsen worker 32 - jepsen.util 4442 :info :write [10 20] :RequestTimeoutException INFO [2022-03-13 17:18:13,678] jepsen worker 32 - jepsen.util 4532 :invoke :read [10 nil] INFO [2022-03-13 17:18:13,691] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,692] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:13,696] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,697] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:18:13,728] jepsen worker 47 - jepsen.util 4007 :info :write [15 16] :RequestTimeoutException INFO [2022-03-13 17:18:13,728] jepsen worker 47 - jepsen.util 4097 :invoke :write [15 5] INFO [2022-03-13 17:18:13,736] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,736] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,737] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:18:13,738] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:18:13,778] jepsen worker 20 - jepsen.util 4340 :info :write [6 31] :RequestTimeoutException INFO [2022-03-13 17:18:13,778] jepsen worker 20 - jepsen.util 4430 :invoke :write [6 4] INFO [2022-03-13 17:18:13,848] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,849] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:13,865] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,865] jepsen worker 88 - jepsen.util 2428 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,866] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:13,866] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 37] INFO [2022-03-13 17:18:13,885] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,886] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:13,890] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:13,891] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:13,906] jepsen worker 55 - jepsen.util 3835 :info :write [18 21] :RequestTimeoutException INFO [2022-03-13 17:18:13,906] jepsen worker 55 - jepsen.util 3925 :invoke :read [18 nil] INFO [2022-03-13 17:18:14,011] jepsen worker 33 - jepsen.util 753 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,011] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:18:14,046] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,047] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:14,049] jepsen worker 3 - jepsen.util 2253 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,049] jepsen worker 48 - jepsen.util 3558 :fail :write [16 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,049] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:14,050] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:18:14,057] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,057] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:18:14,085] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,086] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:18:14,098] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:14,098] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,098] jepsen worker 8 - jepsen.util 728 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,099] jepsen worker 42 - jepsen.util 402 :invoke :write [14 14] INFO [2022-03-13 17:18:14,099] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:14,100] jepsen worker 62 - jepsen.util 512 :invoke :write [20 15] INFO [2022-03-13 17:18:14,129] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,129] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:14,130] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,131] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:18:14,134] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,134] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,135] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:18:14,135] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:14,136] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,137] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:18:14,155] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,155] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:14,158] jepsen worker 35 - jepsen.util 935 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,159] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:14,162] jepsen worker 88 - jepsen.util 2428 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,162] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,163] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:18:14,163] jepsen worker 78 - jepsen.util 708 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,163] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,163] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 40] INFO [2022-03-13 17:18:14,164] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,164] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,165] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:14,166] jepsen worker 78 - jepsen.util 708 :invoke :write [26 30] INFO [2022-03-13 17:18:14,168] jepsen worker 13 - jepsen.util 733 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,168] jepsen worker 13 - jepsen.util 733 :invoke :write [4 21] INFO [2022-03-13 17:18:14,170] jepsen worker 52 - jepsen.util 682 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,173] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:14,183] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,184] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,223] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,224] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:14,225] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,226] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:18:14,248] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,249] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:14,255] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,256] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,256] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,257] jepsen worker 14 - jepsen.util 374 :invoke :write [4 39] INFO [2022-03-13 17:18:14,258] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:18:14,258] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,275] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,275] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:14,276] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,277] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,277] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:18:14,285] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:18:14,295] jepsen worker 43 - jepsen.util 673 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,296] jepsen worker 43 - jepsen.util 673 :invoke :write [14 43] INFO [2022-03-13 17:18:14,311] jepsen worker 12 - jepsen.util 912 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,312] jepsen worker 12 - jepsen.util 912 :invoke :write [4 8] INFO [2022-03-13 17:18:14,346] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:14,347] jepsen worker nemesis - couchbase.util Rebalance check count: 601 INFO [2022-03-13 17:18:14,347] jepsen worker nemesis - couchbase.util Rebalance stuck count: 473 INFO [2022-03-13 17:18:14,362] jepsen worker 24 - jepsen.util 654 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,363] jepsen worker 24 - jepsen.util 654 :invoke :write [8 26] INFO [2022-03-13 17:18:14,366] jepsen worker 76 - jepsen.util 796 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,366] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:14,380] jepsen worker 77 - jepsen.util 707 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,381] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,381] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:18:14,381] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,468] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,468] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,468] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,469] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,472] jepsen worker 80 - jepsen.util 800 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,473] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:14,475] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,475] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,476] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:18:14,476] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,477] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,478] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:18:14,536] jepsen worker 75 - jepsen.util 525 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,536] jepsen worker 26 - jepsen.util 566 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,537] jepsen worker 75 - jepsen.util 525 :invoke :write [25 29] INFO [2022-03-13 17:18:14,537] jepsen worker 26 - jepsen.util 566 :invoke :write [8 24] INFO [2022-03-13 17:18:14,581] jepsen worker 51 - jepsen.util 771 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,582] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,582] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,582] jepsen worker 7 - jepsen.util 637 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,582] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,582] jepsen worker 61 - jepsen.util 781 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,583] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:14,583] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,584] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:14,585] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:14,585] jepsen worker 70 - jepsen.util 880 :invoke :write [23 16] INFO [2022-03-13 17:18:14,586] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:14,591] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,591] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,677] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,677] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,677] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,678] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:18:14,683] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,684] jepsen worker 71 - jepsen.util 611 :invoke :write [23 45] INFO [2022-03-13 17:18:14,694] jepsen worker 34 - jepsen.util 934 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,694] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,695] jepsen worker 34 - jepsen.util 934 :invoke :write [11 47] INFO [2022-03-13 17:18:14,695] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:14,722] jepsen worker 13 - jepsen.util 733 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,723] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,723] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,723] jepsen worker 13 - jepsen.util 733 :invoke :write [4 13] INFO [2022-03-13 17:18:14,727] jepsen worker 53 - jepsen.util 593 :invoke :write [17 20] INFO [2022-03-13 17:18:14,727] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,728] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,729] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:14,759] jepsen worker 6 - jepsen.util 636 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,759] jepsen worker 78 - jepsen.util 708 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,759] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,759] jepsen worker 6 - jepsen.util 636 :invoke :write [2 20] INFO [2022-03-13 17:18:14,860] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:14,814] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,821] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,860] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,860] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:14,860] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:14,877] jepsen worker 42 - jepsen.util 402 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,878] jepsen worker 42 - jepsen.util 402 :invoke :write [14 19] INFO [2022-03-13 17:18:14,901] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,901] jepsen worker 71 - jepsen.util 611 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,901] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,902] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:18:14,920] jepsen worker 14 - jepsen.util 374 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,920] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,921] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:18:14,922] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,924] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,924] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:14,925] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,926] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:14,928] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,928] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,928] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,928] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,928] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,929] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,930] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:14,930] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,930] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:14,931] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,931] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,931] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,932] jepsen worker 61 - jepsen.util 781 :invoke :write [20 22] INFO [2022-03-13 17:18:14,932] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,933] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,933] jepsen worker 78 - jepsen.util 708 :invoke :write [26 25] INFO [2022-03-13 17:18:14,934] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,934] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 16] INFO [2022-03-13 17:18:14,935] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:14,936] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:14,937] jepsen worker 60 - jepsen.util 690 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,938] jepsen worker 75 - jepsen.util 525 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,938] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,939] jepsen worker 75 - jepsen.util 525 :invoke :write [25 2] INFO [2022-03-13 17:18:14,940] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,940] jepsen worker 60 - jepsen.util 690 :invoke :write [20 9] INFO [2022-03-13 17:18:14,941] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:14,945] jepsen worker 52 - jepsen.util 682 :invoke :write [17 19] INFO [2022-03-13 17:18:14,959] jepsen worker 62 - jepsen.util 512 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,959] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:14,961] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,961] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:14,962] jepsen worker 64 - jepsen.util 4204 :ok :read [21 :nil] INFO [2022-03-13 17:18:14,962] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:18:14,963] jepsen worker 64 - jepsen.util 4204 :invoke :read [21 nil] INFO [2022-03-13 17:18:14,976] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:15,029] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,029] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,030] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,034] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:18:15,043] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,044] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,045] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,045] jepsen worker 69 - jepsen.util 789 :invoke :write [23 22] INFO [2022-03-13 17:18:15,054] jepsen worker 12 - jepsen.util 912 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,056] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,056] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,057] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:15,057] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,058] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:15,059] jepsen worker 80 - jepsen.util 800 :invoke :write [26 12] INFO [2022-03-13 17:18:15,059] jepsen worker 7 - jepsen.util 637 :invoke :write [2 20] INFO [2022-03-13 17:18:15,096] jepsen worker 43 - jepsen.util 673 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,097] jepsen worker 43 - jepsen.util 673 :invoke :write [14 44] INFO [2022-03-13 17:18:15,111] jepsen worker 61 - jepsen.util 781 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,111] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:15,146] jepsen worker 24 - jepsen.util 654 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,146] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,146] jepsen worker 24 - jepsen.util 654 :invoke :write [8 22] INFO [2022-03-13 17:18:15,147] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:15,151] jepsen worker 13 - jepsen.util 733 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,151] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,151] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:15,152] jepsen worker 76 - jepsen.util 796 :invoke :write [25 26] INFO [2022-03-13 17:18:15,181] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,182] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,211] jepsen worker 70 - jepsen.util 880 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,212] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,212] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,212] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:15,216] jepsen worker 28 - jepsen.util 4168 :ok :read [9 26] INFO [2022-03-13 17:18:15,216] jepsen worker 28 - jepsen.util 4168 :invoke :write [9 20] INFO [2022-03-13 17:18:15,221] jepsen worker 11 - jepsen.util 4061 :ok :read [3 25] INFO [2022-03-13 17:18:15,221] jepsen worker 11 - jepsen.util 4061 :invoke :read [3 nil] INFO [2022-03-13 17:18:15,259] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,259] jepsen worker 44 - jepsen.util 674 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,260] jepsen worker 44 - jepsen.util 674 :invoke :write [14 39] INFO [2022-03-13 17:18:15,260] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:15,263] jepsen worker 72 - jepsen.util 4212 :ok :read [24 24] INFO [2022-03-13 17:18:15,263] jepsen worker 72 - jepsen.util 4212 :invoke :write [24 30] INFO [2022-03-13 17:18:15,307] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,308] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,308] jepsen worker 69 - jepsen.util 789 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,308] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:18:15,309] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:18:15,311] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,311] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,312] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,313] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:18:15,314] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,315] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,316] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,316] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,317] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,318] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,319] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,321] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,321] jepsen worker 79 - jepsen.util 1159 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,322] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:15,322] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,322] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,322] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:18:15,323] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,323] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,324] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,325] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,325] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:18:15,326] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,338] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,338] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,338] jepsen worker 26 - jepsen.util 566 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,338] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,338] jepsen worker 75 - jepsen.util 525 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,340] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:15,341] jepsen worker 26 - jepsen.util 566 :invoke :write [8 14] INFO [2022-03-13 17:18:15,344] jepsen worker 55 - jepsen.util 3925 :ok :read [18 21] INFO [2022-03-13 17:18:15,346] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:18:15,347] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,347] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:15,347] jepsen worker 55 - jepsen.util 3925 :invoke :read [18 nil] INFO [2022-03-13 17:18:15,353] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:15,353] jepsen worker nemesis - couchbase.util Rebalance check count: 602 INFO [2022-03-13 17:18:15,353] jepsen worker nemesis - couchbase.util Rebalance stuck count: 474 INFO [2022-03-13 17:18:15,363] jepsen worker 11 - jepsen.util 4061 :ok :read [3 25] INFO [2022-03-13 17:18:15,364] jepsen worker 11 - jepsen.util 4061 :invoke :read [3 nil] INFO [2022-03-13 17:18:15,371] jepsen worker 88 - jepsen.util 2428 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,373] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 26] INFO [2022-03-13 17:18:15,382] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,383] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,384] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:15,384] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:18:15,394] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,395] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:15,396] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,397] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:15,398] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,399] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:15,400] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,400] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:15,401] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,402] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:18:15,455] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,455] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,456] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,456] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:18:15,460] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,460] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:15,500] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,500] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,522] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,523] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:15,529] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,530] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,530] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,530] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:18:15,593] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,593] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:18:15,594] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,595] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:18:15,597] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,597] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,597] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:18:15,598] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,600] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,600] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,601] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,602] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:18:15,606] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,607] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,608] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,609] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,610] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,610] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,611] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,611] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,612] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,613] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:18:15,675] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,676] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:18:15,685] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,686] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:15,722] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,722] jepsen worker 88 - jepsen.util 2428 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,722] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:18:15,723] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,757] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,757] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,758] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,759] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,762] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,762] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:18:15,763] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,764] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:18:15,777] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,778] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:15,826] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,826] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,828] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,828] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,829] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,830] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:18:15,865] jepsen worker 34 - jepsen.util 934 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,865] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,866] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,866] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:15,867] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:15,868] jepsen worker 33 - jepsen.util 753 :invoke :write [11 0] INFO [2022-03-13 17:18:15,869] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,870] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:18:15,875] jepsen worker 6 - jepsen.util 636 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,876] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:15,876] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,877] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,886] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,886] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,887] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,888] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:18:15,895] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,896] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 37] INFO [2022-03-13 17:18:15,937] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,937] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,938] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:15,939] jepsen worker 25 - jepsen.util 565 :invoke :write [8 29] INFO [2022-03-13 17:18:15,941] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,942] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:18:15,955] jepsen worker 76 - jepsen.util 796 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,955] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,955] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,955] jepsen worker 53 - jepsen.util 593 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,956] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:15,957] jepsen worker 53 - jepsen.util 593 :invoke :write [17 18] INFO [2022-03-13 17:18:15,958] jepsen worker 76 - jepsen.util 796 :invoke :write [25 23] INFO [2022-03-13 17:18:15,959] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:15,964] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,964] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,965] jepsen worker 75 - jepsen.util 525 :invoke :write [25 0] INFO [2022-03-13 17:18:15,966] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:15,967] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,968] jepsen worker 13 - jepsen.util 733 :invoke :write [4 48] INFO [2022-03-13 17:18:15,970] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,970] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:15,973] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,974] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:15,978] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,978] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,978] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,978] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:18:15,979] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:18:15,988] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:15,990] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,991] jepsen worker 78 - jepsen.util 708 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,992] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:15,992] jepsen worker 78 - jepsen.util 708 :invoke :write [26 3] INFO [2022-03-13 17:18:15,994] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:18:16,000] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:16,024] jepsen worker 11 - jepsen.util 4061 :ok :read [3 25] INFO [2022-03-13 17:18:16,025] jepsen worker 11 - jepsen.util 4061 :invoke :read [3 nil] INFO [2022-03-13 17:18:16,031] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,031] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 46] INFO [2022-03-13 17:18:16,033] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,033] jepsen worker 80 - jepsen.util 800 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,033] jepsen worker 51 - jepsen.util 771 :invoke :write [17 21] INFO [2022-03-13 17:18:16,034] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:16,047] jepsen worker 46 - jepsen.util 4276 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,048] jepsen worker 46 - jepsen.util 4276 :invoke :write [15 47] INFO [2022-03-13 17:18:16,065] jepsen worker 52 - jepsen.util 682 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,065] jepsen worker 52 - jepsen.util 682 :invoke :write [17 35] INFO [2022-03-13 17:18:16,089] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,090] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,090] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,091] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:18:16,098] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,099] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,099] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,099] jepsen worker 55 - jepsen.util 3925 :ok :read [18 21] INFO [2022-03-13 17:18:16,099] jepsen worker 69 - jepsen.util 789 :invoke :write [23 22] INFO [2022-03-13 17:18:16,101] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:16,101] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,101] jepsen worker 55 - jepsen.util 3925 :invoke :write [18 20] INFO [2022-03-13 17:18:16,102] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,102] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,103] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:16,104] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,104] jepsen worker 42 - jepsen.util 402 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,105] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,105] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,105] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,105] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,105] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:16,107] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:18:16,107] jepsen worker 62 - jepsen.util 512 :invoke :write [20 36] INFO [2022-03-13 17:18:16,108] jepsen worker 12 - jepsen.util 912 :invoke :write [4 44] INFO [2022-03-13 17:18:16,108] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,110] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,111] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:18:16,119] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,120] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 40] INFO [2022-03-13 17:18:16,125] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,134] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,139] jepsen worker 24 - jepsen.util 654 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,139] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,139] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:16,140] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:18:16,178] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,178] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,179] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,180] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,180] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,180] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:16,181] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,181] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 28] INFO [2022-03-13 17:18:16,183] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:18:16,183] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:16,194] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,195] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:18:16,217] jepsen worker 76 - jepsen.util 796 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,218] jepsen worker 76 - jepsen.util 796 :invoke :write [25 36] INFO [2022-03-13 17:18:16,234] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,234] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,234] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,235] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,236] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,236] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,242] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,242] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,242] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,243] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,243] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:18:16,244] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:18:16,245] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,245] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,246] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,247] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,248] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,249] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,250] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,251] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,251] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,252] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 27] INFO [2022-03-13 17:18:16,252] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,253] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:18:16,255] jepsen worker 88 - jepsen.util 2428 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,256] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 18] INFO [2022-03-13 17:18:16,281] jepsen worker 44 - jepsen.util 674 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,283] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,283] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:16,284] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,284] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:16,284] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:16,285] jepsen worker 87 - jepsen.util 2877 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,285] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:18:16,324] jepsen worker 60 - jepsen.util 690 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,325] jepsen worker 60 - jepsen.util 690 :invoke :write [20 0] INFO [2022-03-13 17:18:16,339] jepsen worker 7 - jepsen.util 637 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,339] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,339] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:16,339] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,339] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,340] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,340] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:16,340] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,341] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:16,342] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:16,343] jepsen worker 70 - jepsen.util 880 :invoke :write [23 44] INFO [2022-03-13 17:18:16,343] jepsen worker 6 - jepsen.util 636 :invoke :write [2 26] INFO [2022-03-13 17:18:16,346] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:16,346] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:18:16,349] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,349] jepsen worker 33 - jepsen.util 753 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,350] jepsen worker 14 - jepsen.util 374 :invoke :write [4 43] INFO [2022-03-13 17:18:16,351] jepsen worker 33 - jepsen.util 753 :invoke :write [11 45] INFO [2022-03-13 17:18:16,359] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:16,359] jepsen worker nemesis - couchbase.util Rebalance check count: 603 INFO [2022-03-13 17:18:16,359] jepsen worker nemesis - couchbase.util Rebalance stuck count: 475 INFO [2022-03-13 17:18:16,362] jepsen worker 37 - jepsen.util 2467 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,363] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,369] jepsen worker 89 - jepsen.util 2789 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,370] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:16,376] jepsen worker 69 - jepsen.util 789 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,376] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,377] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:16,377] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,386] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,386] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,388] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,389] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:18:16,393] jepsen worker 75 - jepsen.util 525 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,394] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:16,399] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,400] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,400] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:18:16,400] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,401] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,402] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:18:16,404] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,404] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:18:16,407] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,407] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,412] jepsen worker 25 - jepsen.util 565 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,413] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:16,413] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,414] jepsen worker 80 - jepsen.util 800 :invoke :write [26 47] INFO [2022-03-13 17:18:16,421] jepsen worker 78 - jepsen.util 708 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,421] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,422] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,423] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:16,423] jepsen worker 61 - jepsen.util 781 :invoke :write [20 5] INFO [2022-03-13 17:18:16,423] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:16,437] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,437] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:18:16,451] jepsen worker 54 - jepsen.util 3834 :info :write [18 34] :RequestTimeoutException INFO [2022-03-13 17:18:16,452] jepsen worker 54 - jepsen.util 3924 :invoke :read [18 nil] INFO [2022-03-13 17:18:16,462] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,463] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:16,465] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,465] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:16,466] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,467] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:18:16,474] jepsen worker 43 - jepsen.util 673 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,475] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:16,476] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,477] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,478] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,479] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,500] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,500] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:18:16,505] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,505] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,506] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,506] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,507] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:18:16,507] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,518] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,518] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,520] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:18:16,520] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,522] jepsen worker 88 - jepsen.util 2428 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,523] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 4] INFO [2022-03-13 17:18:16,522] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,528] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:18:16,544] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,544] jepsen worker 51 - jepsen.util 771 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,545] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:16,545] jepsen worker 51 - jepsen.util 771 :invoke :write [17 14] INFO [2022-03-13 17:18:16,552] jepsen worker 62 - jepsen.util 512 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,553] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:16,555] jepsen worker 82 - jepsen.util 4312 :info :write [27 4] :SyncWriteAmbiguous INFO [2022-03-13 17:18:16,555] jepsen worker 82 - jepsen.util 4402 :invoke :write [27 35] INFO [2022-03-13 17:18:16,577] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,578] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:18:16,584] jepsen worker 67 - jepsen.util 3397 :ok :read [22 30] INFO [2022-03-13 17:18:16,585] jepsen worker 67 - jepsen.util 3397 :invoke :write [22 1] INFO [2022-03-13 17:18:16,589] jepsen worker 39 - jepsen.util 4449 :info :write [13 28] :SyncWriteAmbiguous INFO [2022-03-13 17:18:16,590] jepsen worker 39 - jepsen.util 4539 :invoke :read [13 nil] INFO [2022-03-13 17:18:16,603] jepsen worker 13 - jepsen.util 733 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,611] jepsen worker 13 - jepsen.util 733 :invoke :write [4 9] INFO [2022-03-13 17:18:16,618] jepsen worker 27 - jepsen.util 4347 :info :write [9 43] :RequestTimeoutException INFO [2022-03-13 17:18:16,619] jepsen worker 27 - jepsen.util 4437 :invoke :read [9 nil] INFO [2022-03-13 17:18:16,620] jepsen worker 40 - jepsen.util 4360 :ok :read [13 39] INFO [2022-03-13 17:18:16,620] jepsen worker 40 - jepsen.util 4360 :invoke :write [13 29] INFO [2022-03-13 17:18:16,631] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,637] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:16,648] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:16,648] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,648] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,649] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,649] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:18:16,650] jepsen worker 21 - jepsen.util 831 :invoke :write [7 30] INFO [2022-03-13 17:18:16,651] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:16,651] jepsen worker 34 - jepsen.util 934 :invoke :write [11 27] INFO [2022-03-13 17:18:16,664] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,664] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 7] INFO [2022-03-13 17:18:16,683] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,684] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:16,691] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,692] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,701] jepsen worker 86 - jepsen.util 4046 :ok :read [28 44] INFO [2022-03-13 17:18:16,702] jepsen worker 86 - jepsen.util 4046 :invoke :read [28 nil] INFO [2022-03-13 17:18:16,704] jepsen worker 58 - jepsen.util 3838 :ok :read [19 6] INFO [2022-03-13 17:18:16,705] jepsen worker 58 - jepsen.util 3838 :invoke :read [19 nil] INFO [2022-03-13 17:18:16,707] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,708] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:16,731] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,732] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:18:16,738] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,739] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:18:16,751] jepsen worker 12 - jepsen.util 912 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,751] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:16,752] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,751] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,752] jepsen worker 12 - jepsen.util 912 :invoke :write [4 45] INFO [2022-03-13 17:18:16,753] jepsen worker 42 - jepsen.util 402 :invoke :write [14 10] INFO [2022-03-13 17:18:16,754] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:16,754] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:16,761] jepsen worker 14 - jepsen.util 374 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,762] jepsen worker 14 - jepsen.util 374 :invoke :write [4 5] INFO [2022-03-13 17:18:16,787] jepsen worker 86 - jepsen.util 4046 :ok :read [28 44] INFO [2022-03-13 17:18:16,788] jepsen worker 86 - jepsen.util 4046 :invoke :read [28 nil] INFO [2022-03-13 17:18:16,806] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,807] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:18:16,808] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,809] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:18:16,809] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,809] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,809] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,810] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,815] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,815] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,816] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:18:16,817] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:16,824] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,825] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:18:16,827] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,827] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 37] INFO [2022-03-13 17:18:16,830] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,831] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:16,833] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,833] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,833] jepsen worker 76 - jepsen.util 796 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,834] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:18:16,835] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:16,836] jepsen worker 76 - jepsen.util 796 :invoke :write [25 20] INFO [2022-03-13 17:18:16,844] jepsen worker 86 - jepsen.util 4046 :ok :read [28 44] INFO [2022-03-13 17:18:16,845] jepsen worker 86 - jepsen.util 4046 :invoke :read [28 nil] INFO [2022-03-13 17:18:16,847] jepsen worker 58 - jepsen.util 3838 :ok :read [19 6] INFO [2022-03-13 17:18:16,848] jepsen worker 58 - jepsen.util 3838 :invoke :read [19 nil] INFO [2022-03-13 17:18:16,865] jepsen worker 79 - jepsen.util 1159 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,865] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,865] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,866] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 6] INFO [2022-03-13 17:18:16,867] jepsen worker 77 - jepsen.util 707 :invoke :write [25 36] INFO [2022-03-13 17:18:16,867] jepsen worker 78 - jepsen.util 708 :invoke :write [26 31] INFO [2022-03-13 17:18:16,890] jepsen worker 26 - jepsen.util 566 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,892] jepsen worker 26 - jepsen.util 566 :invoke :write [8 13] INFO [2022-03-13 17:18:16,892] jepsen worker 33 - jepsen.util 753 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,892] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,893] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:16,894] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:16,906] jepsen worker 86 - jepsen.util 4046 :ok :read [28 44] INFO [2022-03-13 17:18:16,906] jepsen worker 86 - jepsen.util 4046 :invoke :write [28 7] INFO [2022-03-13 17:18:16,908] jepsen worker 36 - jepsen.util 2466 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,909] jepsen worker 88 - jepsen.util 2428 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,909] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,910] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] WARN [2022-03-13 17:18:16,910] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16794219,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16794219,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9037366,"encode_duration_us":19821,"operation_id":"0x10e1b","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10002699},{"operation_name":"upsert","last_dispatch_duration_us":17577362,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25974833,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9263492,"encode_duration_us":17917,"operation_id":"0x10497","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009350}],"total_count":2}} INFO [2022-03-13 17:18:16,919] jepsen worker 58 - jepsen.util 3838 :ok :read [19 6] INFO [2022-03-13 17:18:16,920] jepsen worker 58 - jepsen.util 3838 :invoke :write [19 49] INFO [2022-03-13 17:18:16,926] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,927] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:16,969] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,970] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,971] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,972] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:16,973] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,973] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:16,978] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,978] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:18:16,982] jepsen worker 80 - jepsen.util 800 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,982] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:16,983] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:16,983] jepsen worker 80 - jepsen.util 800 :invoke :write [26 44] WARN [2022-03-13 17:18:17,011] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":31769,"operation_id":"0x10ff5","timeout_ms":10000,"total_duration_us":10060991},{"operation_name":"get","last_dispatch_duration_us":10032754,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10032754,"total_server_duration_us":32,"operation_id":"0x10f40","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":10034977},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":25255,"operation_id":"0x10fd6","timeout_ms":10000,"total_duration_us":10007762},{"operation_name":"upsert","last_dispatch_duration_us":8578114,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8578114,"total_server_duration_us":20841,"encode_duration_us":17120,"operation_id":"0x1132a","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10007354},{"operation_name":"upsert","last_dispatch_duration_us":16794219,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16794219,"total_server_duration_us":9037366,"encode_duration_us":19821,"operation_id":"0x10e1b","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10002699},{"operation_name":"upsert","last_dispatch_duration_us":9134381,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9134381,"total_server_duration_us":22389,"encode_duration_us":17413,"operation_id":"0x10fdd","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10001966},{"operation_name":"get","last_dispatch_duration_us":9079152,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9079152,"total_server_duration_us":4741,"operation_id":"0x112f0","timeout_ms":30000,"last_server_duration_us":4741,"total_duration_us":9079405},{"operation_name":"upsert","last_dispatch_duration_us":2374830,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":2374830,"total_server_duration_us":161835,"encode_duration_us":34049,"operation_id":"0x11625","timeout_ms":10000,"last_server_duration_us":161835,"total_duration_us":2383159},{"operation_name":"upsert","last_dispatch_duration_us":1968676,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":1968676,"total_server_duration_us":85449,"encode_duration_us":26009,"operation_id":"0x11621","timeout_ms":10000,"last_server_duration_us":85449,"total_duration_us":1975547},{"operation_name":"upsert","last_dispatch_duration_us":584483,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":584483,"total_server_duration_us":66957,"encode_duration_us":18060,"operation_id":"0x11737","timeout_ms":10000,"last_server_duration_us":66957,"total_duration_us":1671159}],"total_count":101}} INFO [2022-03-13 17:18:17,016] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,016] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,016] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,017] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,019] jepsen worker 70 - jepsen.util 880 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,019] jepsen worker 70 - jepsen.util 880 :invoke :write [23 32] INFO [2022-03-13 17:18:17,025] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,025] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:18:17,025] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,025] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,026] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 10] INFO [2022-03-13 17:18:17,026] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,028] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,028] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,029] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,030] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,031] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,032] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:18:17,039] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,040] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 1] INFO [2022-03-13 17:18:17,052] jepsen worker 13 - jepsen.util 733 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,052] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,052] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,053] jepsen worker 4 - jepsen.util 2614 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,053] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,054] jepsen worker 13 - jepsen.util 733 :invoke :write [4 19] INFO [2022-03-13 17:18:17,055] jepsen worker 75 - jepsen.util 525 :invoke :write [25 11] INFO [2022-03-13 17:18:17,064] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,095] jepsen worker 51 - jepsen.util 771 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,095] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:17,096] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:17,096] jepsen worker 61 - jepsen.util 781 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,096] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:17,097] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:17,097] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,098] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:17,100] jepsen worker 19 - jepsen.util 4069 :info :write [6 11] :RequestTimeoutException INFO [2022-03-13 17:18:17,100] jepsen worker 19 - jepsen.util 4159 :invoke :read [6 nil] INFO [2022-03-13 17:18:17,103] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,104] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:18:17,122] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,123] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,123] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,124] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:18:17,125] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:18:17,126] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,126] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,126] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:18:17,128] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,128] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,130] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,131] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,132] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,132] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:18:17,139] jepsen worker 53 - jepsen.util 593 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,139] jepsen worker 53 - jepsen.util 593 :invoke :write [17 20] INFO [2022-03-13 17:18:17,172] jepsen worker 38 - jepsen.util 2828 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,173] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,194] jepsen worker 21 - jepsen.util 831 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,195] jepsen worker 21 - jepsen.util 831 :invoke :write [7 14] INFO [2022-03-13 17:18:17,203] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,204] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 30] INFO [2022-03-13 17:18:17,228] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,229] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:18:17,230] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,231] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:18:17,242] jepsen worker 6 - jepsen.util 636 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,242] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,243] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,243] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,243] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:17,243] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:17,244] jepsen worker 7 - jepsen.util 637 :invoke :write [2 42] INFO [2022-03-13 17:18:17,244] jepsen worker 8 - jepsen.util 728 :invoke :write [2 1] INFO [2022-03-13 17:18:17,253] jepsen worker 76 - jepsen.util 796 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,254] jepsen worker 76 - jepsen.util 796 :invoke :write [25 44] INFO [2022-03-13 17:18:17,264] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,266] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,268] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,268] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,269] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,270] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:18:17,271] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,271] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,277] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,278] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,279] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,279] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,280] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:18:17,281] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,282] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,283] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,284] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,284] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:18:17,289] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,290] jepsen worker 16 - jepsen.util 4066 :info :write [5 15] :RequestTimeoutException INFO [2022-03-13 17:18:17,290] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:17,291] jepsen worker 16 - jepsen.util 4156 :invoke :read [5 nil] INFO [2022-03-13 17:18:17,324] jepsen worker 52 - jepsen.util 682 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,324] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:17,334] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,335] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,336] jepsen worker 79 - jepsen.util 1159 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,336] jepsen worker 73 - jepsen.util 4393 :info :write [24 16] :RequestTimeoutException INFO [2022-03-13 17:18:17,336] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:17,337] jepsen worker 73 - jepsen.util 4483 :invoke :read [24 nil] INFO [2022-03-13 17:18:17,337] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,337] jepsen worker 69 - jepsen.util 789 :invoke :write [23 42] INFO [2022-03-13 17:18:17,359] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,359] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:18:17,365] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:17,365] jepsen worker nemesis - couchbase.util Rebalance check count: 604 INFO [2022-03-13 17:18:17,365] jepsen worker nemesis - couchbase.util Rebalance stuck count: 476 INFO [2022-03-13 17:18:17,366] jepsen worker 9 - jepsen.util 3699 :info :write [3 34] :RequestTimeoutException INFO [2022-03-13 17:18:17,367] jepsen worker 9 - jepsen.util 3789 :invoke :write [3 13] INFO [2022-03-13 17:18:17,381] jepsen worker 14 - jepsen.util 374 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,382] jepsen worker 14 - jepsen.util 374 :invoke :write [4 0] INFO [2022-03-13 17:18:17,390] jepsen worker 10 - jepsen.util 4330 :info :write [3 9] :RequestTimeoutException INFO [2022-03-13 17:18:17,390] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,391] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,392] jepsen worker 10 - jepsen.util 4420 :invoke :write [3 1] INFO [2022-03-13 17:18:17,423] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,424] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:18:17,432] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,433] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:17,439] jepsen worker 75 - jepsen.util 525 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,439] jepsen worker 75 - jepsen.util 525 :invoke :write [25 19] INFO [2022-03-13 17:18:17,465] jepsen worker 34 - jepsen.util 934 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,465] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,465] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,466] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,466] jepsen worker 33 - jepsen.util 753 :invoke :write [11 23] INFO [2022-03-13 17:18:17,467] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:18:17,468] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:17,470] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,470] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,470] jepsen worker 24 - jepsen.util 654 :invoke :write [8 36] INFO [2022-03-13 17:18:17,470] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,470] jepsen worker 81 - jepsen.util 4671 :info :write [27 3] :RequestTimeoutException INFO [2022-03-13 17:18:17,471] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,472] jepsen worker 81 - jepsen.util 4761 :invoke :write [27 41] INFO [2022-03-13 17:18:17,475] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,475] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,475] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:18:17,476] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,477] jepsen worker 42 - jepsen.util 402 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,478] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:17,478] jepsen worker 37 - jepsen.util 2467 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,479] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,480] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,480] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:18:17,481] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,482] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,483] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,484] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:18:17,526] jepsen worker 60 - jepsen.util 690 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,526] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:17,526] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,527] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:17,528] jepsen worker 22 - jepsen.util 652 :invoke :write [7 40] INFO [2022-03-13 17:18:17,528] jepsen worker 60 - jepsen.util 690 :invoke :write [20 32] INFO [2022-03-13 17:18:17,538] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,538] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,539] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 37] INFO [2022-03-13 17:18:17,541] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:18:17,581] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,582] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,584] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,585] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:18:17,596] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,597] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:18:17,626] jepsen worker 1 - jepsen.util 3691 :info :write [0 31] :RequestTimeoutException INFO [2022-03-13 17:18:17,627] jepsen worker 1 - jepsen.util 3781 :invoke :write [0 33] INFO [2022-03-13 17:18:17,637] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,638] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:18:17,675] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,676] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:18:17,696] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,696] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,757] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,757] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,758] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:18:17,758] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:18:17,766] jepsen worker 88 - jepsen.util 2428 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,767] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:18:17,769] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,770] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,831] jepsen worker 69 - jepsen.util 789 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,832] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,832] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:18:17,833] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:17,834] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:17,834] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,834] jepsen worker 80 - jepsen.util 800 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,834] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,835] jepsen worker 51 - jepsen.util 771 :invoke :write [17 28] INFO [2022-03-13 17:18:17,837] jepsen worker 71 - jepsen.util 611 :invoke :write [23 9] INFO [2022-03-13 17:18:17,837] jepsen worker 80 - jepsen.util 800 :invoke :write [26 25] INFO [2022-03-13 17:18:17,838] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:17,839] jepsen worker 78 - jepsen.util 708 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,840] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:17,843] jepsen worker 83 - jepsen.util 3773 :info :write [27 18] :RequestTimeoutException INFO [2022-03-13 17:18:17,844] jepsen worker 83 - jepsen.util 3863 :invoke :write [27 49] INFO [2022-03-13 17:18:17,863] jepsen worker 4 - jepsen.util 2614 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,863] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,869] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,870] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:17,898] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,898] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,902] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,903] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,906] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,907] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:18:17,913] jepsen worker 66 - jepsen.util 3936 :info :write [22 48] :RequestTimeoutException INFO [2022-03-13 17:18:17,914] jepsen worker 66 - jepsen.util 4026 :invoke :write [22 45] INFO [2022-03-13 17:18:17,916] jepsen worker 13 - jepsen.util 733 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,916] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:17,943] jepsen worker 21 - jepsen.util 831 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,944] jepsen worker 21 - jepsen.util 831 :invoke :write [7 40] INFO [2022-03-13 17:18:17,944] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,945] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:17,946] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,946] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,946] jepsen worker 71 - jepsen.util 611 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,947] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 9] INFO [2022-03-13 17:18:17,947] jepsen worker 71 - jepsen.util 611 :invoke :write [23 48] INFO [2022-03-13 17:18:17,948] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 36] INFO [2022-03-13 17:18:17,967] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,968] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,977] jepsen worker 12 - jepsen.util 912 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,977] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:17,981] jepsen worker 26 - jepsen.util 566 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,981] jepsen worker 26 - jepsen.util 566 :invoke :write [8 13] INFO [2022-03-13 17:18:17,986] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,986] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,986] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,987] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 4] INFO [2022-03-13 17:18:17,988] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:17,988] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:17,999] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,999] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:17,999] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,000] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,001] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,002] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,002] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:18:18,003] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,003] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,004] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:18:18,004] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,005] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:18:18,031] jepsen worker 14 - jepsen.util 374 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,031] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,032] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:18,032] jepsen worker 34 - jepsen.util 934 :invoke :write [11 32] INFO [2022-03-13 17:18:18,049] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,049] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:18,063] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,064] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,064] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,064] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,063] jepsen worker 77 - jepsen.util 707 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,064] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:18,065] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:18,065] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,066] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,066] jepsen worker 38 - jepsen.util 2828 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,067] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:18:18,067] jepsen worker 61 - jepsen.util 781 :invoke :write [20 41] INFO [2022-03-13 17:18:18,068] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 17] INFO [2022-03-13 17:18:18,070] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:18,071] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:18,074] jepsen worker 76 - jepsen.util 796 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,075] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:18,075] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:18,075] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,076] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:18,077] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,078] jepsen worker 77 - jepsen.util 707 :invoke :write [25 27] INFO [2022-03-13 17:18:18,078] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,079] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:18,080] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,080] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:18,091] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,092] jepsen worker 78 - jepsen.util 708 :invoke :write [26 36] INFO [2022-03-13 17:18:18,105] jepsen worker 4 - jepsen.util 2614 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,105] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,106] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,107] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 20] INFO [2022-03-13 17:18:18,107] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,108] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 36] INFO [2022-03-13 17:18:18,146] jepsen worker 24 - jepsen.util 654 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,147] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:18,148] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,148] jepsen worker 53 - jepsen.util 593 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,148] jepsen worker 52 - jepsen.util 682 :invoke :write [17 0] INFO [2022-03-13 17:18:18,150] jepsen worker 53 - jepsen.util 593 :invoke :write [17 23] INFO [2022-03-13 17:18:18,151] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:18,151] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:18,159] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,160] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,161] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,161] jepsen worker 71 - jepsen.util 611 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,162] jepsen worker 69 - jepsen.util 789 :invoke :write [23 39] INFO [2022-03-13 17:18:18,162] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,162] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:18:18,163] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,164] jepsen worker 89 - jepsen.util 2789 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,165] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,169] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,169] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,171] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,172] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,174] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,174] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,176] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,177] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,178] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,179] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:18:18,187] jepsen worker 70 - jepsen.util 880 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,187] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,187] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,187] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:18,187] jepsen worker 62 - jepsen.util 512 :invoke :write [20 43] INFO [2022-03-13 17:18:18,188] jepsen worker 6 - jepsen.util 636 :invoke :write [2 47] INFO [2022-03-13 17:18:18,196] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,196] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,234] jepsen worker 29 - jepsen.util 4619 :info :write [9 11] :RequestTimeoutException INFO [2022-03-13 17:18:18,234] jepsen worker 29 - jepsen.util 4709 :invoke :write [9 44] INFO [2022-03-13 17:18:18,246] jepsen worker 75 - jepsen.util 525 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,246] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,246] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,246] jepsen worker 75 - jepsen.util 525 :invoke :write [25 18] INFO [2022-03-13 17:18:18,247] jepsen worker 13 - jepsen.util 733 :invoke :write [4 17] INFO [2022-03-13 17:18:18,247] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,248] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:18,254] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:18,262] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,262] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,262] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,262] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,263] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:18:18,263] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,265] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,266] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,266] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:18:18,271] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,271] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:18:18,271] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:18:18,296] jepsen worker 88 - jepsen.util 2428 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,297] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,297] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,297] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,297] jepsen worker 69 - jepsen.util 789 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,299] jepsen worker 69 - jepsen.util 789 :invoke :write [23 49] INFO [2022-03-13 17:18:18,300] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:18:18,300] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,302] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,303] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:18:18,314] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,315] jepsen worker 22 - jepsen.util 652 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,315] jepsen worker 76 - jepsen.util 796 :invoke :write [25 15] INFO [2022-03-13 17:18:18,316] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:18,328] jepsen worker 79 - jepsen.util 1159 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,329] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] WARN [2022-03-13 17:18:18,332] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16957344,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17824123,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9633155,"encode_duration_us":17297,"operation_id":"0x108b0","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10005181}],"total_count":1}} INFO [2022-03-13 17:18:18,351] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:18,351] jepsen worker 33 - jepsen.util 753 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,351] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:18,352] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:18,365] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,366] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:18:18,370] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:18,370] jepsen worker nemesis - couchbase.util Rebalance check count: 605 INFO [2022-03-13 17:18:18,370] jepsen worker nemesis - couchbase.util Rebalance stuck count: 477 INFO [2022-03-13 17:18:18,374] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,375] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:18:18,386] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,387] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,392] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,393] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:18:18,399] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,399] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,423] jepsen worker 7 - jepsen.util 637 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,424] jepsen worker 7 - jepsen.util 637 :invoke :write [2 40] INFO [2022-03-13 17:18:18,439] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,439] jepsen worker 37 - jepsen.util 2467 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,440] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 18] INFO [2022-03-13 17:18:18,441] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,447] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,447] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,456] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,457] jepsen worker 80 - jepsen.util 800 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,457] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:18,457] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:18,468] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,468] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,468] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:18:18,469] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,470] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,471] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:18:18,476] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,476] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,476] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:18:18,477] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,503] jepsen worker 69 - jepsen.util 789 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,503] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] WARN [2022-03-13 17:18:18,510] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":98755,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":3830959,"total_server_duration_us":6881,"operation_id":"0x10406","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":30005234},{"operation_name":"upsert","last_dispatch_duration_us":36708,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8527959,"total_server_duration_us":275403,"encode_duration_us":20666,"operation_id":"0x110da","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10006625},{"operation_name":"upsert","last_dispatch_duration_us":43966,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":5030392,"total_server_duration_us":313441,"encode_duration_us":19763,"operation_id":"0x11398","timeout_ms":10000,"last_server_duration_us":32937,"total_duration_us":10003112},{"operation_name":"upsert","last_dispatch_duration_us":167615,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":167615,"total_server_duration_us":23559,"encode_duration_us":19603,"operation_id":"0x113e6","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10002037},{"operation_name":"upsert","last_dispatch_duration_us":9097159,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9458501,"total_server_duration_us":117126,"encode_duration_us":24687,"operation_id":"0x113c5","timeout_ms":10000,"last_server_duration_us":34911,"total_duration_us":10001899},{"operation_name":"upsert","last_dispatch_duration_us":9242786,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9242786,"total_server_duration_us":9053293,"encode_duration_us":17847,"operation_id":"0x110c7","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9246790},{"operation_name":"upsert","last_dispatch_duration_us":9186375,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9186375,"total_server_duration_us":9062854,"encode_duration_us":24590,"operation_id":"0x113db","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":9191738},{"operation_name":"upsert","last_dispatch_duration_us":6746643,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":6746643,"total_server_duration_us":64473,"encode_duration_us":32571,"operation_id":"0x11447","timeout_ms":10000,"last_server_duration_us":64473,"total_duration_us":6776164},{"operation_name":"upsert","last_dispatch_duration_us":6564709,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":6564709,"total_server_duration_us":46852,"encode_duration_us":22217,"operation_id":"0x11445","timeout_ms":10000,"last_server_duration_us":46852,"total_duration_us":6599122},{"operation_name":"upsert","last_dispatch_duration_us":6327507,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":6327507,"total_server_duration_us":47420,"encode_duration_us":29341,"operation_id":"0x1142e","timeout_ms":10000,"last_server_duration_us":47420,"total_duration_us":6506439}],"total_count":41}} WARN [2022-03-13 17:18:18,518] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":14734237,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14734237,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9049045,"encode_duration_us":17753,"operation_id":"0x10ce7","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10009295}],"total_count":1}} WARN [2022-03-13 17:18:18,518] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9157690,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9157690,"total_server_duration_us":21405,"encode_duration_us":27880,"operation_id":"0x110b3","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10010369},{"operation_name":"upsert","last_dispatch_duration_us":9127948,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9127948,"total_server_duration_us":23306,"encode_duration_us":18631,"operation_id":"0x110c2","timeout_ms":10000,"last_server_duration_us":23306,"total_duration_us":10004351},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":53326,"operation_id":"0x113d4","timeout_ms":10000,"total_duration_us":10003958},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":31742,"operation_id":"0x110ba","timeout_ms":10000,"total_duration_us":10001746},{"operation_name":"upsert","last_dispatch_duration_us":62625,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9604374,"total_server_duration_us":240225,"encode_duration_us":30423,"operation_id":"0x11419","timeout_ms":10000,"last_server_duration_us":23138,"total_duration_us":10001578},{"operation_name":"get","last_dispatch_duration_us":9186360,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9186360,"total_server_duration_us":36729,"operation_id":"0x110b0","timeout_ms":30000,"last_server_duration_us":36729,"total_duration_us":9189813},{"operation_name":"get","last_dispatch_duration_us":9164412,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9164412,"total_server_duration_us":69,"operation_id":"0x110bd","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":9164771},{"operation_name":"upsert","last_dispatch_duration_us":9115381,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9115381,"total_server_duration_us":9053293,"encode_duration_us":21693,"operation_id":"0x113e2","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9120713},{"operation_name":"get","last_dispatch_duration_us":8987775,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8987775,"total_server_duration_us":15,"operation_id":"0x11412","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":8990710},{"operation_name":"get","last_dispatch_duration_us":8967720,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8967720,"total_server_duration_us":49,"operation_id":"0x1140b","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8971485}],"total_count":25}} INFO [2022-03-13 17:18:18,523] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,523] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:18:18,527] jepsen worker 60 - jepsen.util 690 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,527] jepsen worker 75 - jepsen.util 525 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,528] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:18,528] jepsen worker 60 - jepsen.util 690 :invoke :write [20 4] WARN [2022-03-13 17:18:18,534] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18057552,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18057552,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9050106,"encode_duration_us":33674,"operation_id":"0x10d6d","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10006340}],"total_count":1}} INFO [2022-03-13 17:18:18,540] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,540] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:18,554] jepsen worker 21 - jepsen.util 831 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,554] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,555] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:18:18,555] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:18:18,568] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,569] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,576] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,577] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:18:18,580] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,582] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,600] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,600] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,600] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:18:18,601] jepsen worker 71 - jepsen.util 611 :invoke :write [23 37] INFO [2022-03-13 17:18:18,610] jepsen worker 88 - jepsen.util 2428 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,610] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,611] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 17] WARN [2022-03-13 17:18:18,611] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9118080,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":14107567,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9065171,"encode_duration_us":19528,"operation_id":"0x10d92","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10001360}],"total_count":1}} INFO [2022-03-13 17:18:18,611] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,616] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,617] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 13] WARN [2022-03-13 17:18:18,617] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11986622,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":11986622,"total_server_duration_us":49,"operation_id":"0x10fbd","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":11988822},{"operation_name":"upsert","last_dispatch_duration_us":11186776,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11186776,"total_server_duration_us":22804,"encode_duration_us":18128,"operation_id":"0x111c4","timeout_ms":10000,"last_server_duration_us":22804,"total_duration_us":10006020},{"operation_name":"upsert","last_dispatch_duration_us":12935148,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12935148,"total_server_duration_us":31108,"encode_duration_us":19271,"operation_id":"0x10f1e","timeout_ms":10000,"last_server_duration_us":31108,"total_duration_us":10005442},{"operation_name":"upsert","last_dispatch_duration_us":486508,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8654928,"total_server_duration_us":111838,"encode_duration_us":21952,"operation_id":"0x113d7","timeout_ms":10000,"last_server_duration_us":28310,"total_duration_us":10005118},{"operation_name":"upsert","last_dispatch_duration_us":7273300,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":7273300,"total_server_duration_us":44944,"encode_duration_us":22829,"operation_id":"0x112ef","timeout_ms":10000,"last_server_duration_us":44944,"total_duration_us":7277182},{"operation_name":"upsert","last_dispatch_duration_us":6210817,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":6210817,"total_server_duration_us":72862,"encode_duration_us":41848,"operation_id":"0x1144a","timeout_ms":10000,"last_server_duration_us":72862,"total_duration_us":6215847},{"operation_name":"upsert","last_dispatch_duration_us":5176924,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":5176924,"total_server_duration_us":61774,"encode_duration_us":21259,"operation_id":"0x112ee","timeout_ms":10000,"last_server_duration_us":61774,"total_duration_us":5180825},{"operation_name":"get","last_dispatch_duration_us":4175160,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4175160,"total_server_duration_us":32,"operation_id":"0x11475","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":4177554},{"operation_name":"upsert","last_dispatch_duration_us":3312751,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3312751,"total_server_duration_us":68547,"encode_duration_us":22723,"operation_id":"0x11600","timeout_ms":10000,"last_server_duration_us":68547,"total_duration_us":3316528},{"operation_name":"get","last_dispatch_duration_us":480279,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":480279,"total_server_duration_us":467243,"operation_id":"0x116e2","timeout_ms":30000,"last_server_duration_us":467243,"total_duration_us":3101032}],"total_count":23}} INFO [2022-03-13 17:18:18,631] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:18,631] jepsen worker 8 - jepsen.util 728 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,632] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,632] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,632] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:18,633] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:18,634] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:18,634] jepsen worker 12 - jepsen.util 912 :invoke :write [4 16] WARN [2022-03-13 17:18:18,635] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":1588626,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":5454995,"total_server_duration_us":27776,"operation_id":"0x104ff","timeout_ms":30000,"last_server_duration_us":2712,"total_duration_us":30006992},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":21222,"operation_id":"0x11351","timeout_ms":10000,"total_duration_us":10004694},{"operation_name":"upsert","last_dispatch_duration_us":7196325,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8171904,"total_server_duration_us":265104,"encode_duration_us":17317,"operation_id":"0x111b0","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10003657},{"operation_name":"upsert","last_dispatch_duration_us":7129801,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8160427,"total_server_duration_us":276059,"encode_duration_us":18078,"operation_id":"0x111b3","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10003128},{"operation_name":"upsert","last_dispatch_duration_us":7095544,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8133906,"total_server_duration_us":288622,"encode_duration_us":19674,"operation_id":"0x111b6","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10002644},{"operation_name":"get","last_dispatch_duration_us":9560924,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9560924,"total_server_duration_us":49,"operation_id":"0x10f82","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9561898},{"operation_name":"upsert","last_dispatch_duration_us":9314159,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9314159,"total_server_duration_us":9281805,"encode_duration_us":19741,"operation_id":"0x111ac","timeout_ms":10000,"last_server_duration_us":9281805,"total_duration_us":9319356},{"operation_name":"get","last_dispatch_duration_us":1602732,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1602732,"total_server_duration_us":43,"operation_id":"0x1145d","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":1604794},{"operation_name":"upsert","last_dispatch_duration_us":1301634,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1301634,"total_server_duration_us":46513,"encode_duration_us":17986,"operation_id":"0x11502","timeout_ms":10000,"last_server_duration_us":46513,"total_duration_us":1306781},{"operation_name":"upsert","last_dispatch_duration_us":1264075,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1264075,"total_server_duration_us":75478,"encode_duration_us":16844,"operation_id":"0x114f8","timeout_ms":10000,"last_server_duration_us":75478,"total_duration_us":1267480}],"total_count":54}} INFO [2022-03-13 17:18:18,640] jepsen worker 34 - jepsen.util 934 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,640] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,640] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:18,640] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,641] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,641] jepsen worker 34 - jepsen.util 934 :invoke :write [11 28] INFO [2022-03-13 17:18:18,641] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:18,642] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:18,642] jepsen worker 25 - jepsen.util 565 :invoke :write [8 0] INFO [2022-03-13 17:18:18,646] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,646] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,646] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:18,647] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:18:18,647] jepsen worker 24 - jepsen.util 654 :invoke :write [8 9] INFO [2022-03-13 17:18:18,650] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,651] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,653] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,654] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,655] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:18:18,655] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:18:18,695] jepsen worker 71 - jepsen.util 611 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,695] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] WARN [2022-03-13 17:18:18,717] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18028196,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18028196,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9049045,"encode_duration_us":17074,"operation_id":"0x108ff","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10001912},{"operation_name":"upsert","last_dispatch_duration_us":22974065,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":22974065,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9047983,"encode_duration_us":18338,"operation_id":"0x10877","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10010875},{"operation_name":"upsert","last_dispatch_duration_us":16112686,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16843915,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9270346,"encode_duration_us":32898,"operation_id":"0x10d62","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10002815}],"total_count":3}} INFO [2022-03-13 17:18:18,731] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,731] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,731] jepsen worker 51 - jepsen.util 771 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,732] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:18,733] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:18,733] jepsen worker 35 - jepsen.util 935 :invoke :write [11 3] WARN [2022-03-13 17:18:18,735] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":17313770,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":17313770,"total_server_duration_us":6579,"operation_id":"0x10dfe","timeout_ms":30000,"last_server_duration_us":6579,"total_duration_us":17315881},{"operation_name":"upsert","last_dispatch_duration_us":70185,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8550850,"total_server_duration_us":292390,"encode_duration_us":18526,"operation_id":"0x10f96","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10007947},{"operation_name":"upsert","last_dispatch_duration_us":31657,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":1288817,"total_server_duration_us":361634,"encode_duration_us":20298,"operation_id":"0x1143b","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10007528},{"operation_name":"upsert","last_dispatch_duration_us":254739,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7302982,"total_server_duration_us":44927,"encode_duration_us":20137,"operation_id":"0x11466","timeout_ms":10000,"last_server_duration_us":25275,"total_duration_us":10006248},{"operation_name":"upsert","last_dispatch_duration_us":41241,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":6720474,"total_server_duration_us":364249,"encode_duration_us":19442,"operation_id":"0x110d9","timeout_ms":10000,"last_server_duration_us":21977,"total_duration_us":10005625},{"operation_name":"upsert","last_dispatch_duration_us":30244,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":1269513,"total_server_duration_us":370228,"encode_duration_us":19941,"operation_id":"0x11443","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10001591},{"operation_name":"get","last_dispatch_duration_us":9030037,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9030037,"total_server_duration_us":30162,"operation_id":"0x11319","timeout_ms":30000,"last_server_duration_us":30162,"total_duration_us":9037650},{"operation_name":"get","last_dispatch_duration_us":8140256,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8140256,"total_server_duration_us":12055,"operation_id":"0x1139d","timeout_ms":30000,"last_server_duration_us":12055,"total_duration_us":8142786},{"operation_name":"upsert","last_dispatch_duration_us":2036850,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2036850,"total_server_duration_us":1754165,"encode_duration_us":17594,"operation_id":"0x11605","timeout_ms":10000,"last_server_duration_us":1754165,"total_duration_us":2040766},{"operation_name":"upsert","last_dispatch_duration_us":1617852,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1617852,"total_server_duration_us":49371,"encode_duration_us":17023,"operation_id":"0x114fe","timeout_ms":10000,"last_server_duration_us":49371,"total_duration_us":1623436}],"total_count":83}} INFO [2022-03-13 17:18:18,736] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:18,736] jepsen worker 13 - jepsen.util 733 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,736] jepsen worker 26 - jepsen.util 566 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,737] jepsen worker 26 - jepsen.util 566 :invoke :write [8 34] INFO [2022-03-13 17:18:18,737] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:18,738] jepsen worker 43 - jepsen.util 673 :invoke :write [14 31] INFO [2022-03-13 17:18:18,744] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,744] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,747] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,747] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:18:18,749] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,750] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,759] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,760] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:18:18,769] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,770] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:18:18,798] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,798] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,799] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:18,799] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:18:18,803] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,803] jepsen worker 69 - jepsen.util 789 :invoke :write [23 9] INFO [2022-03-13 17:18:18,816] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,816] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,817] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,817] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:18,818] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,818] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:18:18,831] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,832] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,835] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,836] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:18:18,838] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,838] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:18:18,886] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,886] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,886] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:18:18,887] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 22] INFO [2022-03-13 17:18:18,904] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,905] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,905] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:18,905] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,905] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:18:18,906] jepsen worker 44 - jepsen.util 674 :invoke :write [14 26] INFO [2022-03-13 17:18:18,911] jepsen worker 14 - jepsen.util 374 :invoke :write [4 18] INFO [2022-03-13 17:18:18,912] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:18,920] jepsen worker 76 - jepsen.util 796 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,920] jepsen worker 52 - jepsen.util 682 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,921] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,921] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,922] jepsen worker 69 - jepsen.util 789 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,922] jepsen worker 88 - jepsen.util 2428 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,924] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:18,925] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:18:18,926] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:18,927] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:18,928] jepsen worker 76 - jepsen.util 796 :invoke :write [25 38] INFO [2022-03-13 17:18:18,929] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:18,936] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,936] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,937] jepsen worker 13 - jepsen.util 733 :invoke :write [4 1] INFO [2022-03-13 17:18:18,938] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:18,969] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,970] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 1] INFO [2022-03-13 17:18:18,974] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,975] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:18,988] jepsen worker 34 - jepsen.util 934 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:18,989] jepsen worker 34 - jepsen.util 934 :invoke :write [11 18] INFO [2022-03-13 17:18:19,003] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,003] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:18:19,006] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,007] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,011] jepsen worker 15 - jepsen.util 4065 :info :write [5 27] :SyncWriteAmbiguous INFO [2022-03-13 17:18:19,011] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,012] jepsen worker 15 - jepsen.util 4155 :invoke :write [5 27] INFO [2022-03-13 17:18:19,013] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,014] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,014] jepsen worker 53 - jepsen.util 593 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,015] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,015] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:19,015] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:19,016] jepsen worker 53 - jepsen.util 593 :invoke :write [17 24] INFO [2022-03-13 17:18:19,022] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,022] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:18:19,061] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,061] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,068] jepsen worker 24 - jepsen.util 654 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,068] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:19,071] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,072] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,073] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,074] jepsen worker 37 - jepsen.util 2467 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,074] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:18:19,075] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,076] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,077] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:18:19,147] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,147] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,147] jepsen worker 3 - jepsen.util 2253 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,148] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:18:19,148] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:19,149] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 40] INFO [2022-03-13 17:18:19,152] jepsen worker 38 - jepsen.util 2828 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,153] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,153] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,153] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,154] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,155] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:19,156] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,156] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,157] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:18:19,159] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 15] INFO [2022-03-13 17:18:19,162] jepsen worker 60 - jepsen.util 690 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,162] jepsen worker 76 - jepsen.util 796 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,162] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:19,163] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:19,168] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,168] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,169] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:18:19,170] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,177] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,177] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,187] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,189] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,189] jepsen worker 25 - jepsen.util 565 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,189] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,190] jepsen worker 80 - jepsen.util 800 :invoke :write [26 45] INFO [2022-03-13 17:18:19,190] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,191] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:19,192] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,192] jepsen worker 51 - jepsen.util 771 :invoke :write [17 18] INFO [2022-03-13 17:18:19,192] jepsen worker 69 - jepsen.util 789 :invoke :write [23 16] INFO [2022-03-13 17:18:19,236] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,237] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:18:19,260] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,260] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,261] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 26] INFO [2022-03-13 17:18:19,261] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:18:19,279] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,279] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,279] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,280] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,280] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,281] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,281] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,282] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:18:19,282] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,283] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,284] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,285] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,286] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,287] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,288] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,289] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,290] jepsen worker 35 - jepsen.util 935 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,290] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,291] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,291] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,291] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:19,292] jepsen worker 3 - jepsen.util 2253 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,292] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,293] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,293] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:19,294] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:19,295] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:18:19,295] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:18:19,297] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,298] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,316] jepsen worker 26 - jepsen.util 566 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,317] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,317] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,317] jepsen worker 13 - jepsen.util 733 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,317] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:19,318] jepsen worker 13 - jepsen.util 733 :invoke :write [4 35] INFO [2022-03-13 17:18:19,319] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:18:19,319] jepsen worker 75 - jepsen.util 525 :invoke :write [25 9] INFO [2022-03-13 17:18:19,325] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,326] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:18:19,360] jepsen worker 69 - jepsen.util 789 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,360] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,360] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,360] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:18:19,361] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,361] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,363] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,363] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:18:19,369] jepsen worker 43 - jepsen.util 673 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,369] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,369] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,370] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:18:19,373] jepsen worker 52 - jepsen.util 682 :invoke :write [17 21] INFO [2022-03-13 17:18:19,374] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,375] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:18:19,376] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:19,382] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:19,382] jepsen worker nemesis - couchbase.util Rebalance check count: 606 INFO [2022-03-13 17:18:19,382] jepsen worker nemesis - couchbase.util Rebalance stuck count: 478 INFO [2022-03-13 17:18:19,387] jepsen worker 44 - jepsen.util 674 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,392] jepsen worker 44 - jepsen.util 674 :invoke :write [14 32] INFO [2022-03-13 17:18:19,425] jepsen worker 4 - jepsen.util 2614 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,426] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,426] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,426] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 7] INFO [2022-03-13 17:18:19,427] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,427] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:19,437] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,438] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,438] jepsen worker 37 - jepsen.util 2467 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,439] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,439] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,443] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:19,444] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:18:19,445] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 31] INFO [2022-03-13 17:18:19,462] jepsen worker 53 - jepsen.util 593 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,462] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,462] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,463] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:19,463] jepsen worker 53 - jepsen.util 593 :invoke :write [17 1] INFO [2022-03-13 17:18:19,464] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,465] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:18:19,465] jepsen worker 26 - jepsen.util 566 :invoke :write [8 26] INFO [2022-03-13 17:18:19,465] jepsen worker 43 - jepsen.util 673 :invoke :write [14 20] INFO [2022-03-13 17:18:19,469] jepsen worker 60 - jepsen.util 690 :invoke :write [20 22] INFO [2022-03-13 17:18:19,488] jepsen worker 14 - jepsen.util 374 :fail :write [4 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,489] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:19,527] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,527] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,528] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,529] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:18:19,533] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,534] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:18:19,537] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,538] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:18:19,543] jepsen worker 34 - jepsen.util 934 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,543] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,543] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,543] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:19,544] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:19,544] jepsen worker 34 - jepsen.util 934 :invoke :write [11 17] INFO [2022-03-13 17:18:19,552] jepsen worker 75 - jepsen.util 525 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,553] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:19,569] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,569] jepsen worker 4 - jepsen.util 2614 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,571] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 8] INFO [2022-03-13 17:18:19,571] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,572] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:18:19,573] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,598] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,599] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 19] INFO [2022-03-13 17:18:19,601] jepsen worker 17 - jepsen.util 4337 :info :write [5 3] :RequestTimeoutException INFO [2022-03-13 17:18:19,602] jepsen worker 17 - jepsen.util 4427 :invoke :read [5 nil] INFO [2022-03-13 17:18:19,634] jepsen worker 51 - jepsen.util 771 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,635] jepsen worker 51 - jepsen.util 771 :invoke :write [17 4] INFO [2022-03-13 17:18:19,637] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,638] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:18:19,655] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,658] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,660] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,661] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,663] jepsen worker 61 - jepsen.util 781 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,663] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:19,664] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,664] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,664] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,665] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:18:19,666] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,666] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,667] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,668] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:18:19,670] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,671] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:18:19,672] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,673] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:18:19,705] jepsen worker 3 - jepsen.util 2253 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,707] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 25] INFO [2022-03-13 17:18:19,707] jepsen worker 80 - jepsen.util 800 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,707] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:19,708] jepsen worker 17 - jepsen.util 4427 :ok :read [5 43] INFO [2022-03-13 17:18:19,709] jepsen worker 17 - jepsen.util 4427 :invoke :write [5 7] INFO [2022-03-13 17:18:19,745] jepsen worker 37 - jepsen.util 2467 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,746] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,757] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,757] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:18:19,765] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,766] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,773] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,774] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,774] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,774] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,774] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,779] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:19,779] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:19,779] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:18:19,779] jepsen worker 33 - jepsen.util 753 :invoke :write [11 35] INFO [2022-03-13 17:18:19,779] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,779] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:18:19,780] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:18:19,814] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,815] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:19,816] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,817] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:18:19,840] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,840] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,844] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,844] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,844] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,845] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,848] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,848] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,848] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,849] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:19,850] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,850] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,851] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,852] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:18:19,852] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,853] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:18:19,856] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,857] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:18:19,882] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,882] jepsen worker 44 - jepsen.util 674 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,882] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,882] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,883] jepsen worker 44 - jepsen.util 674 :invoke :read [14 nil] INFO [2022-03-13 17:18:19,883] jepsen worker 14 - jepsen.util 374 :invoke :write [4 9] INFO [2022-03-13 17:18:19,884] jepsen worker 24 - jepsen.util 654 :invoke :write [8 9] INFO [2022-03-13 17:18:19,884] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:18:19,898] jepsen worker 59 - jepsen.util 4199 :info :write [19 12] :RequestTimeoutException INFO [2022-03-13 17:18:19,898] jepsen worker 59 - jepsen.util 4289 :invoke :read [19 nil] INFO [2022-03-13 17:18:19,903] jepsen worker 3 - jepsen.util 2253 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,904] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:19,919] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,919] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,920] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,920] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,925] jepsen worker 52 - jepsen.util 682 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,925] jepsen worker 13 - jepsen.util 733 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,925] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,927] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,928] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:19,929] jepsen worker 52 - jepsen.util 682 :invoke :write [17 8] INFO [2022-03-13 17:18:19,929] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,930] jepsen worker 76 - jepsen.util 796 :invoke :write [25 16] INFO [2022-03-13 17:18:19,931] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:18:19,931] jepsen worker 75 - jepsen.util 525 :invoke :write [25 33] INFO [2022-03-13 17:18:19,937] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,938] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,938] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,939] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,943] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,944] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,944] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,945] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,945] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,946] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,946] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,947] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,947] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,947] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,948] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,949] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,949] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:18:19,949] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,955] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,956] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:19,957] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,957] jepsen worker 34 - jepsen.util 934 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,957] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:18:19,958] jepsen worker 34 - jepsen.util 934 :invoke :write [11 30] INFO [2022-03-13 17:18:19,962] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,962] jepsen worker 88 - jepsen.util 2428 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,963] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:19,964] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 18] INFO [2022-03-13 17:18:19,966] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,967] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 30] INFO [2022-03-13 17:18:19,975] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,976] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:18:19,998] jepsen worker 51 - jepsen.util 771 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,998] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:19,999] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:19,999] jepsen worker 80 - jepsen.util 800 :invoke :write [26 9] INFO [2022-03-13 17:18:20,013] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,013] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,019] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,020] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,021] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,022] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:18:20,034] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,035] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:20,053] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,053] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,054] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:18:20,055] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,056] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,057] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,059] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,060] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,061] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,061] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,062] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,062] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 1] INFO [2022-03-13 17:18:20,063] jepsen worker 53 - jepsen.util 593 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,064] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:18:20,064] jepsen worker 53 - jepsen.util 593 :invoke :write [17 15] INFO [2022-03-13 17:18:20,065] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,079] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,080] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:20,099] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,099] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:20,105] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,106] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,115] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,115] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:18:20,135] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,135] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,135] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,136] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:18:20,175] jepsen worker 33 - jepsen.util 753 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,175] jepsen worker 44 - jepsen.util 674 :ok :read [14 :nil] INFO [2022-03-13 17:18:20,176] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:20,177] jepsen worker 44 - jepsen.util 674 :invoke :write [14 3] INFO [2022-03-13 17:18:20,188] jepsen worker 88 - jepsen.util 2428 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,188] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,211] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,211] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,242] jepsen worker 26 - jepsen.util 566 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,242] jepsen worker 26 - jepsen.util 566 :invoke :write [8 31] INFO [2022-03-13 17:18:20,245] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,245] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,245] jepsen worker 71 - jepsen.util 611 :invoke :write [23 41] INFO [2022-03-13 17:18:20,246] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:20,288] jepsen worker 24 - jepsen.util 654 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,289] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:20,289] jepsen worker 3 - jepsen.util 2253 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,290] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,290] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,290] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:18:20,299] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,300] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,300] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,301] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,302] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,303] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,303] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,303] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:18:20,304] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,304] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,305] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,306] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:18:20,320] jepsen worker 6 - jepsen.util 636 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,321] jepsen worker 6 - jepsen.util 636 :invoke :write [2 9] INFO [2022-03-13 17:18:20,327] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,327] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:18:20,340] jepsen worker 75 - jepsen.util 525 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,340] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:20,358] jepsen worker 68 - jepsen.util 4298 :info :write [22 18] :RequestTimeoutException INFO [2022-03-13 17:18:20,359] jepsen worker 68 - jepsen.util 4388 :invoke :read [22 nil] INFO [2022-03-13 17:18:20,360] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,360] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:20,389] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:20,389] jepsen worker nemesis - couchbase.util Rebalance check count: 607 INFO [2022-03-13 17:18:20,389] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,389] jepsen worker nemesis - couchbase.util Rebalance stuck count: 479 INFO [2022-03-13 17:18:20,389] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,390] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 16] INFO [2022-03-13 17:18:20,391] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:18:20,395] jepsen worker 37 - jepsen.util 2467 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,396] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,396] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,397] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 5] INFO [2022-03-13 17:18:20,403] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,403] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,422] jepsen worker 71 - jepsen.util 611 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,423] jepsen worker 78 - jepsen.util 708 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,423] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,423] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:18:20,424] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:20,424] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:20,425] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,426] jepsen worker 71 - jepsen.util 611 :invoke :write [23 32] INFO [2022-03-13 17:18:20,449] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,449] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,450] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,450] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 19] INFO [2022-03-13 17:18:20,451] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,452] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,453] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,453] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:18:20,458] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,458] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,458] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:18:20,459] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 49] INFO [2022-03-13 17:18:20,465] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,466] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,494] jepsen worker 3 - jepsen.util 2253 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,495] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,554] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,555] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:18:20,563] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,563] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,563] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,564] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:18:20,564] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,565] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,567] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,567] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,568] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:18:20,568] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:18:20,569] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,570] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:18:20,627] jepsen worker 88 - jepsen.util 2428 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,627] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,632] jepsen worker 89 - jepsen.util 2789 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,632] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:18:20,661] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,662] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,670] jepsen worker 57 - jepsen.util 4197 :info :write [19 47] :RequestTimeoutException INFO [2022-03-13 17:18:20,670] jepsen worker 57 - jepsen.util 4287 :invoke :write [19 48] INFO [2022-03-13 17:18:20,703] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,704] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:18:20,714] jepsen worker 2 - jepsen.util 4412 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:18:20,714] jepsen worker 2 - jepsen.util 4502 :invoke :write [0 9] INFO [2022-03-13 17:18:20,733] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,733] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,742] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,743] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:18:20,772] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,772] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:18:20,799] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,799] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,799] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,800] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:18:20,800] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,800] jepsen worker 37 - jepsen.util 2467 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,801] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:18:20,801] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,802] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 0] INFO [2022-03-13 17:18:20,803] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:18:20,823] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,823] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:18:20,825] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,826] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,872] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,873] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:18:20,881] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,881] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:18:20,918] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,919] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:18:20,936] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,937] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,940] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,941] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:20,969] jepsen worker 37 - jepsen.util 2467 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,969] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,970] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,970] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,978] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,978] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:20,978] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,979] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:20,979] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,980] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 5] INFO [2022-03-13 17:18:20,980] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,981] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 3] INFO [2022-03-13 17:18:20,996] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:20,997] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:18:21,034] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,035] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,035] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,035] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:18:21,036] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:21,037] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:18:21,040] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,041] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:21,042] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,043] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:21,044] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,045] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:18:21,055] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,056] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,060] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,061] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,068] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,069] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,071] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,072] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,074] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,074] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:18:21,102] jepsen worker 88 - jepsen.util 2428 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,103] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:21,291] jepsen worker 43 - jepsen.util 673 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,292] jepsen worker 43 - jepsen.util 673 :invoke :write [14 12] INFO [2022-03-13 17:18:21,321] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,322] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,362] jepsen worker 71 - jepsen.util 611 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,362] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,363] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,363] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:18:21,367] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,367] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:18:21,393] jepsen worker 62 - jepsen.util 512 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,394] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:18:21,395] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:21,395] jepsen worker nemesis - couchbase.util Rebalance check count: 608 INFO [2022-03-13 17:18:21,395] jepsen worker nemesis - couchbase.util Rebalance stuck count: 480 INFO [2022-03-13 17:18:21,401] jepsen worker 76 - jepsen.util 796 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,402] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,446] jepsen worker 63 - jepsen.util 4293 :info :write [21 38] :RequestTimeoutException INFO [2022-03-13 17:18:21,447] jepsen worker 63 - jepsen.util 4383 :invoke :write [21 25] INFO [2022-03-13 17:18:21,484] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,485] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:21,513] jepsen worker 0 - jepsen.util 4050 :info :write [0 25] :SyncWriteAmbiguous INFO [2022-03-13 17:18:21,513] jepsen worker 0 - jepsen.util 4140 :invoke :read [0 nil] INFO [2022-03-13 17:18:21,521] jepsen worker 77 - jepsen.util 707 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,521] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,521] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,521] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,522] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,522] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:21,523] jepsen worker 61 - jepsen.util 781 :invoke :write [20 35] INFO [2022-03-13 17:18:21,523] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:21,524] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:21,524] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:18:21,533] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,533] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,533] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,534] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 48] INFO [2022-03-13 17:18:21,534] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:21,542] jepsen worker 78 - jepsen.util 708 :invoke :write [26 41] INFO [2022-03-13 17:18:21,570] jepsen worker 30 - jepsen.util 4080 :info :write [10 21] :RequestTimeoutException INFO [2022-03-13 17:18:21,571] jepsen worker 30 - jepsen.util 4170 :invoke :write [10 4] INFO [2022-03-13 17:18:21,577] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,578] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,582] jepsen worker 7 - jepsen.util 637 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,582] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,582] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:21,582] jepsen worker 7 - jepsen.util 637 :invoke :write [2 41] INFO [2022-03-13 17:18:21,582] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,583] jepsen worker 8 - jepsen.util 728 :invoke :write [2 43] INFO [2022-03-13 17:18:21,583] jepsen worker 70 - jepsen.util 880 :invoke :write [23 23] INFO [2022-03-13 17:18:21,584] jepsen worker 42 - jepsen.util 402 :invoke :write [14 13] INFO [2022-03-13 17:18:21,597] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,598] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:21,603] jepsen worker 60 - jepsen.util 690 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,603] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,603] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,603] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,604] jepsen worker 13 - jepsen.util 733 :invoke :write [4 24] INFO [2022-03-13 17:18:21,605] jepsen worker 60 - jepsen.util 690 :invoke :write [20 45] INFO [2022-03-13 17:18:21,606] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:18:21,606] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,606] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,607] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,615] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,615] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,616] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,617] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,620] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,620] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,620] jepsen worker 76 - jepsen.util 796 :invoke :write [25 32] INFO [2022-03-13 17:18:21,621] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,623] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,623] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,641] jepsen worker 84 - jepsen.util 4494 :info :write [28 8] :SyncWriteAmbiguous INFO [2022-03-13 17:18:21,642] jepsen worker 84 - jepsen.util 4584 :invoke :read [28 nil] INFO [2022-03-13 17:18:21,673] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,673] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,675] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:18:21,676] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,677] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,678] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,679] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,679] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,681] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,681] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,682] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,683] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,684] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,685] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:18:21,711] jepsen worker 52 - jepsen.util 682 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,711] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:21,717] jepsen worker 12 - jepsen.util 912 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,717] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:21,718] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,719] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,719] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:18:21,719] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,719] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,720] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,720] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,721] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:18:21,735] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,735] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,735] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 33] INFO [2022-03-13 17:18:21,736] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,738] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,738] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,740] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,740] jepsen worker 31 - jepsen.util 4261 :ok :read [10 :nil] INFO [2022-03-13 17:18:21,740] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,741] jepsen worker 31 - jepsen.util 4261 :invoke :write [10 9] INFO [2022-03-13 17:18:21,742] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,743] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:18:21,755] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,756] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:21,775] jepsen worker 45 - jepsen.util 4365 :info :write [15 10] :SyncWriteAmbiguous INFO [2022-03-13 17:18:21,775] jepsen worker 45 - jepsen.util 4455 :invoke :write [15 30] INFO [2022-03-13 17:18:21,786] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,786] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,789] jepsen worker 14 - jepsen.util 374 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,790] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 4] INFO [2022-03-13 17:18:21,791] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,791] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:21,792] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:18:21,792] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:21,799] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,799] jepsen worker 61 - jepsen.util 781 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,800] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:18:21,802] jepsen worker 61 - jepsen.util 781 :invoke :write [20 14] INFO [2022-03-13 17:18:21,808] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,809] jepsen worker 71 - jepsen.util 611 :invoke :write [23 42] INFO [2022-03-13 17:18:21,814] jepsen worker 54 - jepsen.util 3924 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,815] jepsen worker 54 - jepsen.util 3924 :invoke :write [18 1] INFO [2022-03-13 17:18:21,821] jepsen worker 18 - jepsen.util 4158 :ok :read [6 16] INFO [2022-03-13 17:18:21,821] jepsen worker 18 - jepsen.util 4158 :invoke :read [6 nil] INFO [2022-03-13 17:18:21,834] jepsen worker 74 - jepsen.util 4034 :ok :read [24 24] INFO [2022-03-13 17:18:21,835] jepsen worker 74 - jepsen.util 4034 :invoke :read [24 nil] INFO [2022-03-13 17:18:21,837] jepsen worker 13 - jepsen.util 733 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,837] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,839] jepsen worker 75 - jepsen.util 525 :invoke :write [25 37] INFO [2022-03-13 17:18:21,840] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:21,870] jepsen worker 37 - jepsen.util 2467 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,870] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,870] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:21,871] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:21,874] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,875] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:21,891] jepsen worker 73 - jepsen.util 4483 :ok :read [24 24] INFO [2022-03-13 17:18:21,893] jepsen worker 73 - jepsen.util 4483 :invoke :write [24 8] INFO [2022-03-13 17:18:21,900] jepsen worker 89 - jepsen.util 2789 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,900] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:18:21,911] jepsen worker 53 - jepsen.util 593 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,912] jepsen worker 53 - jepsen.util 593 :invoke :write [17 39] INFO [2022-03-13 17:18:21,916] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,917] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:21,941] jepsen worker 18 - jepsen.util 4158 :ok :read [6 16] INFO [2022-03-13 17:18:21,942] jepsen worker 18 - jepsen.util 4158 :invoke :read [6 nil] INFO [2022-03-13 17:18:21,960] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,960] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,960] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,961] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 37] INFO [2022-03-13 17:18:21,961] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:18:21,962] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:21,967] jepsen worker 74 - jepsen.util 4034 :ok :read [24 24] INFO [2022-03-13 17:18:21,968] jepsen worker 74 - jepsen.util 4034 :invoke :write [24 12] INFO [2022-03-13 17:18:21,968] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,969] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:21,974] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,974] jepsen worker 6 - jepsen.util 636 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,974] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 44] INFO [2022-03-13 17:18:21,975] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:21,980] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,981] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:21,992] jepsen worker 76 - jepsen.util 796 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,992] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:21,993] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:21,993] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:22,036] jepsen worker 71 - jepsen.util 611 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,036] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,037] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:18:22,037] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:18:22,039] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,040] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,040] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:18:22,042] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:22,043] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,044] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:18:22,051] jepsen worker 18 - jepsen.util 4158 :ok :read [6 16] INFO [2022-03-13 17:18:22,052] jepsen worker 18 - jepsen.util 4158 :invoke :write [6 7] INFO [2022-03-13 17:18:22,064] jepsen worker 26 - jepsen.util 566 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,070] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:22,082] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,082] jepsen worker 4 - jepsen.util 2614 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,082] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:18:22,082] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,083] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,084] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:18:22,086] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,087] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,088] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,089] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:18:22,109] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,109] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,110] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,111] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:18:22,111] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,115] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:18:22,180] jepsen worker 75 - jepsen.util 525 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,180] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,180] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,181] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:22,182] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:22,183] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:22,183] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,184] jepsen worker 88 - jepsen.util 2428 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,184] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,185] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 26] INFO [2022-03-13 17:18:22,187] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,188] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,190] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,190] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,191] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:18:22,192] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,192] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,192] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,192] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,193] jepsen worker 76 - jepsen.util 796 :invoke :write [25 21] INFO [2022-03-13 17:18:22,194] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:22,194] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,194] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:22,195] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:18:22,199] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,200] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,241] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,241] jepsen worker 27 - jepsen.util 4437 :ok :read [9 26] INFO [2022-03-13 17:18:22,242] jepsen worker 62 - jepsen.util 512 :invoke :write [20 18] INFO [2022-03-13 17:18:22,242] jepsen worker 27 - jepsen.util 4437 :invoke :write [9 30] INFO [2022-03-13 17:18:22,260] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,261] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:18:22,287] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,287] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,287] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,288] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,289] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,289] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,290] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:18:22,290] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,291] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,292] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:18:22,294] jepsen worker 37 - jepsen.util 2467 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,295] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:18:22,297] jepsen worker 43 - jepsen.util 673 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,298] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,298] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,298] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:22,299] jepsen worker 13 - jepsen.util 733 :invoke :write [4 39] INFO [2022-03-13 17:18:22,299] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:22,307] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,307] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,307] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,315] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:18:22,322] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,323] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:18:22,326] jepsen worker 79 - jepsen.util 1159 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,327] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:22,363] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,363] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,364] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:18:22,364] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:18:22,387] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,387] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,391] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,391] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,393] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,394] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:18:22,404] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:22,404] jepsen worker nemesis - couchbase.util Rebalance check count: 609 INFO [2022-03-13 17:18:22,404] jepsen worker nemesis - couchbase.util Rebalance stuck count: 481 INFO [2022-03-13 17:18:22,414] jepsen worker 60 - jepsen.util 690 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,414] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,416] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:18:22,416] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:22,421] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,422] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,428] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,429] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 36] INFO [2022-03-13 17:18:22,443] jepsen worker 7 - jepsen.util 637 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,444] jepsen worker 7 - jepsen.util 637 :invoke :write [2 13] INFO [2022-03-13 17:18:22,476] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,476] jepsen worker 76 - jepsen.util 796 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,476] jepsen worker 75 - jepsen.util 525 :invoke :write [25 6] INFO [2022-03-13 17:18:22,477] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:18:22,499] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,500] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,500] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,502] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,513] jepsen worker 78 - jepsen.util 708 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,513] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:22,519] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,519] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,558] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,558] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,559] jepsen worker 52 - jepsen.util 682 :invoke :write [17 33] INFO [2022-03-13 17:18:22,559] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:18:22,560] jepsen worker 88 - jepsen.util 2428 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,561] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,561] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,562] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 29] INFO [2022-03-13 17:18:22,563] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,563] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:18:22,571] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,572] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:18:22,572] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,573] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:18:22,585] jepsen worker 13 - jepsen.util 733 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,586] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:22,592] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,593] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,593] jepsen worker 42 - jepsen.util 402 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,593] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,593] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,594] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:18:22,595] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:18:22,602] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,607] jepsen worker 42 - jepsen.util 402 :invoke :write [14 26] INFO [2022-03-13 17:18:22,609] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,610] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,610] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:18:22,616] jepsen worker 41 - jepsen.util 4181 :info :write [13 1] :RequestTimeoutException INFO [2022-03-13 17:18:22,617] jepsen worker 41 - jepsen.util 4271 :invoke :read [13 nil] INFO [2022-03-13 17:18:22,623] jepsen worker 38 - jepsen.util 2828 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,623] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:18:22,627] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,627] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:18:22,629] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,630] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:18:22,657] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,658] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,658] jepsen worker 61 - jepsen.util 781 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,658] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,659] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:22,666] jepsen worker 61 - jepsen.util 781 :invoke :write [20 30] INFO [2022-03-13 17:18:22,676] jepsen worker 41 - jepsen.util 4271 :ok :read [13 39] INFO [2022-03-13 17:18:22,677] jepsen worker 41 - jepsen.util 4271 :invoke :read [13 nil] INFO [2022-03-13 17:18:22,680] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,681] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,702] jepsen worker 41 - jepsen.util 4271 :ok :read [13 39] INFO [2022-03-13 17:18:22,702] jepsen worker 41 - jepsen.util 4271 :invoke :write [13 24] INFO [2022-03-13 17:18:22,704] jepsen worker 53 - jepsen.util 593 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,704] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,704] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,704] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:22,705] jepsen worker 53 - jepsen.util 593 :invoke :write [17 39] INFO [2022-03-13 17:18:22,705] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:22,706] jepsen worker 26 - jepsen.util 566 :invoke :write [8 25] INFO [2022-03-13 17:18:22,706] jepsen worker 60 - jepsen.util 690 :invoke :write [20 46] INFO [2022-03-13 17:18:22,743] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,744] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,744] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,745] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,746] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,747] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,747] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:18:22,747] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,750] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,750] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:18:22,759] jepsen worker 75 - jepsen.util 525 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,760] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,760] jepsen worker 75 - jepsen.util 525 :invoke :write [25 46] INFO [2022-03-13 17:18:22,760] jepsen worker 8 - jepsen.util 728 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,761] jepsen worker 12 - jepsen.util 912 :invoke :write [4 9] INFO [2022-03-13 17:18:22,761] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:22,807] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,807] jepsen worker 88 - jepsen.util 2428 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,808] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,808] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 44] INFO [2022-03-13 17:18:22,815] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,815] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,816] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,816] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,819] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,819] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,820] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:18:22,821] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,821] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,821] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:22,822] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:22,823] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:22,848] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,848] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,849] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,849] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,849] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,850] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,851] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,851] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,851] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,852] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:18:22,852] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,853] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:18:22,853] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,854] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:18:22,855] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,856] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,857] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,858] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,859] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,860] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,861] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,862] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,864] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,864] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,866] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,867] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,868] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,868] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,870] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,871] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,872] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,872] jepsen worker 71 - jepsen.util 611 :invoke :write [23 4] INFO [2022-03-13 17:18:22,874] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,875] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,876] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,877] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,878] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,878] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,878] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 17] INFO [2022-03-13 17:18:22,879] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,880] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,881] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:22,883] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,884] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] INFO [2022-03-13 17:18:22,908] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,908] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,909] jepsen worker 52 - jepsen.util 682 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,909] jepsen worker 13 - jepsen.util 733 :invoke :write [4 2] INFO [2022-03-13 17:18:22,910] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:22,910] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:22,928] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,928] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,928] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,928] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,929] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:22,929] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,929] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,930] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,930] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,930] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,931] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,931] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,932] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:18:22,932] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 14] INFO [2022-03-13 17:18:22,933] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,934] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,935] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,936] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:22,937] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,938] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:18:22,938] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,939] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:18:22,950] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,950] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:22,961] jepsen worker 37 - jepsen.util 2467 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,961] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,962] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 7] INFO [2022-03-13 17:18:22,962] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:18:22,969] jepsen worker 70 - jepsen.util 880 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,969] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:22,969] jepsen worker 70 - jepsen.util 880 :invoke :write [23 47] INFO [2022-03-13 17:18:22,973] jepsen worker 6 - jepsen.util 636 :invoke :write [2 4] INFO [2022-03-13 17:18:23,003] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,004] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:23,004] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:18:23,004] jepsen worker 43 - jepsen.util 673 :invoke :write [14 28] INFO [2022-03-13 17:18:23,009] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,009] jepsen worker 71 - jepsen.util 611 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,009] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:18:23,010] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,013] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,014] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,015] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,016] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,018] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,018] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,019] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,020] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,020] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:18:23,021] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:18:23,028] jepsen worker 75 - jepsen.util 525 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,028] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,029] jepsen worker 75 - jepsen.util 525 :invoke :write [25 30] INFO [2022-03-13 17:18:23,030] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:18:23,048] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,049] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,049] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,048] jepsen worker 61 - jepsen.util 781 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,049] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:23,050] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:23,051] jepsen worker 78 - jepsen.util 708 :invoke :write [26 28] INFO [2022-03-13 17:18:23,055] jepsen worker 61 - jepsen.util 781 :invoke :write [20 0] INFO [2022-03-13 17:18:23,068] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,069] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:18:23,105] jepsen worker 4 - jepsen.util 2614 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,105] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,124] jepsen worker 26 - jepsen.util 566 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,125] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:23,133] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,133] jepsen worker 71 - jepsen.util 611 :invoke :write [23 40] INFO [2022-03-13 17:18:23,143] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,144] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:18:23,152] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,152] jepsen worker 88 - jepsen.util 2428 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,153] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:18:23,153] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:18:23,159] jepsen worker 62 - jepsen.util 512 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,160] jepsen worker 62 - jepsen.util 512 :invoke :write [20 1] INFO [2022-03-13 17:18:23,175] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,177] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:18:23,179] jepsen worker 13 - jepsen.util 733 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,181] jepsen worker 13 - jepsen.util 733 :invoke :write [4 9] INFO [2022-03-13 17:18:23,209] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,210] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:18:23,218] jepsen worker 56 - jepsen.util 4106 :info :write [18 28] :RequestTimeoutException INFO [2022-03-13 17:18:23,219] jepsen worker 56 - jepsen.util 4196 :invoke :read [18 nil] INFO [2022-03-13 17:18:23,227] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,228] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:18:23,239] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,240] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,241] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,241] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,241] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,243] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:23,243] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:23,244] jepsen worker 23 - jepsen.util 563 :invoke :write [7 15] INFO [2022-03-13 17:18:23,272] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,272] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,272] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:18:23,273] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:18:23,300] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,300] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,301] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,301] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,302] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,303] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,303] jepsen worker 71 - jepsen.util 611 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,304] jepsen worker 71 - jepsen.util 611 :invoke :write [23 24] INFO [2022-03-13 17:18:23,309] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,309] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,310] jepsen worker 37 - jepsen.util 2467 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,310] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,311] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,311] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,311] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,311] jepsen worker 53 - jepsen.util 593 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,312] jepsen worker 53 - jepsen.util 593 :invoke :write [17 11] INFO [2022-03-13 17:18:23,312] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,313] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:18:23,313] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:18:23,318] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,318] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,373] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,374] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:23,374] jepsen worker 7 - jepsen.util 637 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,375] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:23,377] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,377] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,380] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,380] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,406] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,407] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:18:23,410] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:23,410] jepsen worker nemesis - couchbase.util Rebalance check count: 610 INFO [2022-03-13 17:18:23,410] jepsen worker nemesis - couchbase.util Rebalance stuck count: 482 INFO [2022-03-13 17:18:23,429] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,430] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,445] jepsen worker 78 - jepsen.util 708 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,445] jepsen worker 78 - jepsen.util 708 :invoke :write [26 11] INFO [2022-03-13 17:18:23,451] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,451] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,451] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,452] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,452] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:18:23,453] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,453] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,453] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,454] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,454] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,455] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,455] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,456] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,456] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,457] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 40] INFO [2022-03-13 17:18:23,457] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:18:23,464] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,464] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,465] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 18] INFO [2022-03-13 17:18:23,466] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:18:23,494] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,494] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:18:23,501] jepsen worker 71 - jepsen.util 611 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,501] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,502] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,503] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:23,515] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,516] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,517] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,518] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:18:23,542] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,542] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,543] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:18:23,543] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,548] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,549] jepsen worker 25 - jepsen.util 565 :invoke :write [8 10] INFO [2022-03-13 17:18:23,550] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,551] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,552] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,552] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:18:23,564] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,564] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,573] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,576] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:18:23,578] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,578] jepsen worker 60 - jepsen.util 690 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,579] jepsen worker 52 - jepsen.util 682 :invoke :write [17 40] INFO [2022-03-13 17:18:23,579] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,580] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:23,581] jepsen worker 8 - jepsen.util 728 :invoke :write [2 47] INFO [2022-03-13 17:18:23,581] jepsen worker 42 - jepsen.util 402 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,581] jepsen worker 75 - jepsen.util 525 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,583] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:23,583] jepsen worker 42 - jepsen.util 402 :invoke :write [14 12] INFO [2022-03-13 17:18:23,594] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,595] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:18:23,641] jepsen worker 61 - jepsen.util 781 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,641] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,642] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,642] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,642] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:23,643] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:23,644] jepsen worker 61 - jepsen.util 781 :invoke :write [20 24] INFO [2022-03-13 17:18:23,645] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,649] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,649] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:18:23,658] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,659] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,679] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,681] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:23,681] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,681] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,719] jepsen worker 13 - jepsen.util 733 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,719] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,719] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,720] jepsen worker 76 - jepsen.util 796 :invoke :write [25 2] INFO [2022-03-13 17:18:23,720] jepsen worker 75 - jepsen.util 525 :invoke :write [25 49] INFO [2022-03-13 17:18:23,721] jepsen worker 13 - jepsen.util 733 :invoke :write [4 12] INFO [2022-03-13 17:18:23,758] jepsen worker 47 - jepsen.util 4097 :info :write [15 5] :RequestTimeoutException INFO [2022-03-13 17:18:23,759] jepsen worker 47 - jepsen.util 4187 :invoke :read [15 nil] INFO [2022-03-13 17:18:23,763] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,764] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:18:23,776] jepsen worker 43 - jepsen.util 673 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,777] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,777] jepsen worker 43 - jepsen.util 673 :invoke :write [14 37] INFO [2022-03-13 17:18:23,778] jepsen worker 26 - jepsen.util 566 :invoke :write [8 26] INFO [2022-03-13 17:18:23,779] jepsen worker 12 - jepsen.util 912 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,779] jepsen worker 12 - jepsen.util 912 :invoke :write [4 35] INFO [2022-03-13 17:18:23,784] jepsen worker 4 - jepsen.util 2614 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,784] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,784] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,785] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:18:23,786] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:18:23,786] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,789] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,789] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,790] jepsen worker 69 - jepsen.util 789 :invoke :write [23 4] INFO [2022-03-13 17:18:23,794] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,795] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:23,795] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,795] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,796] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,797] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,797] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,799] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,799] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,799] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,799] jepsen worker 20 - jepsen.util 4430 :info :write [6 4] :RequestTimeoutException INFO [2022-03-13 17:18:23,800] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:18:23,802] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:18:23,803] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:18:23,803] jepsen worker 20 - jepsen.util 4520 :invoke :write [6 23] INFO [2022-03-13 17:18:23,840] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,840] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,856] jepsen worker 53 - jepsen.util 593 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,857] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:23,858] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,859] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:23,860] jepsen worker 23 - jepsen.util 563 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,861] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:23,909] jepsen worker 69 - jepsen.util 789 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,910] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:18:23,916] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,916] jepsen worker 37 - jepsen.util 2467 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,916] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,917] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,924] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,925] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:23,927] jepsen worker 75 - jepsen.util 525 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,928] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:23,948] jepsen worker 70 - jepsen.util 880 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,948] jepsen worker 70 - jepsen.util 880 :invoke :write [23 32] INFO [2022-03-13 17:18:23,952] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,953] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,953] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,954] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 10] INFO [2022-03-13 17:18:23,956] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,956] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:23,957] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,958] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 18] INFO [2022-03-13 17:18:23,964] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,964] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,964] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,964] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:18:23,965] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:18:23,965] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:23,967] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:23,967] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:18:24,003] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,003] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,003] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:24,004] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:18:24,006] jepsen worker 78 - jepsen.util 708 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,007] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,007] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,007] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:24,007] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,008] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:24,030] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,031] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:24,048] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,048] jepsen worker 52 - jepsen.util 682 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,048] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:18:24,049] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:24,060] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,060] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:18:24,072] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,072] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,093] jepsen worker 65 - jepsen.util 4475 :ok :read [21 :nil] INFO [2022-03-13 17:18:24,093] jepsen worker 65 - jepsen.util 4475 :invoke :write [21 11] INFO [2022-03-13 17:18:24,111] jepsen worker 6 - jepsen.util 636 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,111] jepsen worker 6 - jepsen.util 636 :invoke :write [2 35] INFO [2022-03-13 17:18:24,115] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,115] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,115] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,115] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:24,119] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:24,119] jepsen worker 13 - jepsen.util 733 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,120] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:18:24,120] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:24,121] jepsen worker 89 - jepsen.util 2789 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,122] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:18:24,126] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,126] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,127] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:24,127] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:18:24,130] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,131] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:18:24,155] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,155] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,156] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 15] INFO [2022-03-13 17:18:24,156] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:24,157] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,158] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:24,160] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,161] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:18:24,181] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,181] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,182] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,182] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,184] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,184] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,188] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,188] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,188] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,188] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:18:24,189] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,190] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:18:24,197] jepsen worker 61 - jepsen.util 781 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,197] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,197] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,197] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,197] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,198] jepsen worker 78 - jepsen.util 708 :invoke :write [26 39] INFO [2022-03-13 17:18:24,199] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:24,199] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,199] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:18:24,200] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,200] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,203] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:24,204] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 32] INFO [2022-03-13 17:18:24,204] jepsen worker 77 - jepsen.util 707 :invoke :write [25 21] INFO [2022-03-13 17:18:24,207] jepsen worker 38 - jepsen.util 2828 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,208] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,209] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:24,209] jepsen worker 61 - jepsen.util 781 :invoke :write [20 7] INFO [2022-03-13 17:18:24,213] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,214] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:24,220] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,221] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:18:24,226] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,227] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:24,235] jepsen worker 43 - jepsen.util 673 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,235] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,235] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,235] jepsen worker 76 - jepsen.util 796 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,236] jepsen worker 43 - jepsen.util 673 :invoke :write [14 5] INFO [2022-03-13 17:18:24,237] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:24,237] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,237] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,238] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,238] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,238] jepsen worker 13 - jepsen.util 733 :invoke :write [4 22] INFO [2022-03-13 17:18:24,238] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,239] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,239] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,240] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,240] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,240] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:18:24,241] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,242] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,248] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,259] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,260] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,260] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,260] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,261] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:18:24,262] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:24,263] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,263] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,264] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 21] INFO [2022-03-13 17:18:24,266] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:24,270] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,270] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,305] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,305] jepsen worker 62 - jepsen.util 512 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,306] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:24,306] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:24,327] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,328] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,328] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:24,329] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:24,338] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,338] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:24,344] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,344] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,344] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:18:24,345] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:18:24,357] jepsen worker 26 - jepsen.util 566 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,357] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,357] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,358] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,358] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:24,358] jepsen worker 60 - jepsen.util 690 :invoke :write [20 48] INFO [2022-03-13 17:18:24,359] jepsen worker 52 - jepsen.util 682 :invoke :write [17 15] INFO [2022-03-13 17:18:24,359] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:24,369] jepsen worker 37 - jepsen.util 2467 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,370] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:18:24,399] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,399] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,400] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:18:24,400] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:24,404] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,405] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:18:24,408] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,408] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,409] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:18:24,409] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:18:24,415] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:24,415] jepsen worker nemesis - couchbase.util Rebalance check count: 611 INFO [2022-03-13 17:18:24,415] jepsen worker nemesis - couchbase.util Rebalance stuck count: 483 INFO [2022-03-13 17:18:24,433] jepsen worker 8 - jepsen.util 728 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,434] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:24,460] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,460] jepsen worker 13 - jepsen.util 733 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,461] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,462] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:24,481] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,482] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:18:24,508] jepsen worker 72 - jepsen.util 4212 :info :write [24 30] :SyncWriteAmbiguous INFO [2022-03-13 17:18:24,509] jepsen worker 72 - jepsen.util 4302 :invoke :read [24 nil] INFO [2022-03-13 17:18:24,518] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,519] jepsen worker 22 - jepsen.util 652 :invoke :write [7 37] INFO [2022-03-13 17:18:24,565] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,565] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,566] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,566] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:18:24,570] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,571] jepsen worker 69 - jepsen.util 789 :invoke :write [23 49] INFO [2022-03-13 17:18:24,583] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,583] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,584] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,584] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,584] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,585] jepsen worker 13 - jepsen.util 733 :invoke :write [4 46] INFO [2022-03-13 17:18:24,589] jepsen worker 42 - jepsen.util 402 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,589] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,590] jepsen worker 42 - jepsen.util 402 :invoke :write [14 37] INFO [2022-03-13 17:18:24,590] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,624] jepsen worker 78 - jepsen.util 708 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,625] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:24,652] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,652] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,652] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,652] jepsen worker 43 - jepsen.util 673 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,653] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,653] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:24,654] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:24,655] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,655] jepsen worker 76 - jepsen.util 796 :invoke :write [25 29] INFO [2022-03-13 17:18:24,656] jepsen worker 53 - jepsen.util 593 :invoke :write [17 32] INFO [2022-03-13 17:18:24,663] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,663] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,757] jepsen worker 69 - jepsen.util 789 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,757] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,757] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:24,758] jepsen worker 71 - jepsen.util 611 :invoke :write [23 39] INFO [2022-03-13 17:18:24,758] jepsen worker 12 - jepsen.util 912 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,758] jepsen worker 77 - jepsen.util 707 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,759] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:24,759] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:24,816] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,816] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:18:24,849] jepsen worker 13 - jepsen.util 733 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,849] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,849] jepsen worker 52 - jepsen.util 682 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,879] jepsen worker 75 - jepsen.util 525 :invoke :write [25 6] INFO [2022-03-13 17:18:24,879] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:24,879] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:24,884] jepsen worker 32 - jepsen.util 4532 :ok :read [10 :nil] INFO [2022-03-13 17:18:24,885] jepsen worker 32 - jepsen.util 4532 :invoke :read [10 nil] INFO [2022-03-13 17:18:24,925] jepsen worker 11 - jepsen.util 4061 :ok :read [3 25] INFO [2022-03-13 17:18:24,926] jepsen worker 11 - jepsen.util 4061 :invoke :read [3 nil] INFO [2022-03-13 17:18:24,928] jepsen worker 64 - jepsen.util 4204 :ok :read [21 :nil] INFO [2022-03-13 17:18:24,928] jepsen worker 64 - jepsen.util 4204 :invoke :read [21 nil] INFO [2022-03-13 17:18:24,934] jepsen worker 72 - jepsen.util 4302 :ok :read [24 24] INFO [2022-03-13 17:18:24,935] jepsen worker 72 - jepsen.util 4302 :invoke :read [24 nil] INFO [2022-03-13 17:18:24,992] jepsen worker 71 - jepsen.util 611 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:24,993] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:18:25,013] jepsen worker 11 - jepsen.util 4061 :ok :read [3 25] INFO [2022-03-13 17:18:25,014] jepsen worker 11 - jepsen.util 4061 :invoke :write [3 15] INFO [2022-03-13 17:18:25,018] jepsen worker 79 - jepsen.util 1159 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,018] jepsen worker 70 - jepsen.util 880 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,019] jepsen worker 70 - jepsen.util 880 :invoke :write [23 6] INFO [2022-03-13 17:18:25,020] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 13] INFO [2022-03-13 17:18:25,031] jepsen worker 88 - jepsen.util 2428 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,032] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 30] INFO [2022-03-13 17:18:25,038] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,038] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,039] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,039] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,040] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:18:25,041] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 47] INFO [2022-03-13 17:18:25,050] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,051] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:18:25,055] jepsen worker 72 - jepsen.util 4302 :ok :read [24 24] INFO [2022-03-13 17:18:25,056] jepsen worker 72 - jepsen.util 4302 :invoke :write [24 49] INFO [2022-03-13 17:18:25,063] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,064] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,110] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,111] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,112] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,112] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,112] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,113] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,113] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,114] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,117] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,117] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,118] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:18:25,119] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:18:25,174] jepsen worker 3 - jepsen.util 2253 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,175] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,200] jepsen worker 6 - jepsen.util 636 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,200] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,200] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,200] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,200] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:25,201] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:25,201] jepsen worker 62 - jepsen.util 512 :invoke :write [20 19] INFO [2022-03-13 17:18:25,202] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:25,221] jepsen worker 61 - jepsen.util 781 :fail :write [20 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,221] jepsen worker 61 - jepsen.util 781 :invoke :write [20 9] INFO [2022-03-13 17:18:25,239] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,240] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 4] INFO [2022-03-13 17:18:25,244] jepsen worker 28 - jepsen.util 4168 :info :write [9 20] :RequestTimeoutException INFO [2022-03-13 17:18:25,249] jepsen worker 28 - jepsen.util 4258 :invoke :write [9 14] INFO [2022-03-13 17:18:25,253] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,253] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:18:25,339] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,339] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:18:25,349] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,349] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,349] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,349] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,350] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,350] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 40] INFO [2022-03-13 17:18:25,351] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:18:25,360] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,361] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,362] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:18:25,395] jepsen worker 76 - jepsen.util 796 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,395] jepsen worker 60 - jepsen.util 690 :fail :write [20 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,396] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,396] jepsen worker 60 - jepsen.util 690 :invoke :write [20 39] INFO [2022-03-13 17:18:25,396] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:25,397] jepsen worker 76 - jepsen.util 796 :invoke :write [25 35] INFO [2022-03-13 17:18:25,401] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,401] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:18:25,421] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:25,422] jepsen worker nemesis - couchbase.util Rebalance check count: 612 INFO [2022-03-13 17:18:25,422] jepsen worker nemesis - couchbase.util Rebalance stuck count: 484 INFO [2022-03-13 17:18:25,422] jepsen worker 89 - jepsen.util 2789 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,423] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,433] jepsen worker 42 - jepsen.util 402 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,433] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,434] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:18:25,434] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:25,441] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,441] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,442] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:18:25,442] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:18:25,444] jepsen worker 88 - jepsen.util 2428 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,444] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,445] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,445] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,446] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,447] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,447] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:18:25,447] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 33] INFO [2022-03-13 17:18:25,509] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,509] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,509] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:18:25,510] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,514] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,515] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,520] jepsen worker 3 - jepsen.util 2253 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,520] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,521] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,522] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:18:25,576] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,576] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,604] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,604] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,604] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,605] jepsen worker 77 - jepsen.util 707 :invoke :write [25 45] INFO [2022-03-13 17:18:25,605] jepsen worker 78 - jepsen.util 708 :invoke :write [26 32] INFO [2022-03-13 17:18:25,605] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:25,606] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,607] jepsen worker 71 - jepsen.util 611 :invoke :write [23 35] INFO [2022-03-13 17:18:25,616] jepsen worker 34 - jepsen.util 934 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,616] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,616] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,616] jepsen worker 22 - jepsen.util 652 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,616] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:25,617] jepsen worker 26 - jepsen.util 566 :invoke :write [8 46] INFO [2022-03-13 17:18:25,618] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:25,618] jepsen worker 43 - jepsen.util 673 :invoke :write [14 23] INFO [2022-03-13 17:18:25,619] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:25,619] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:18:25,626] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,626] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,626] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,627] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 15] INFO [2022-03-13 17:18:25,628] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,630] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:18:25,660] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,660] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,660] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,661] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:18:25,661] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:18:25,662] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,663] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,663] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,664] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,665] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:18:25,676] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,676] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:25,679] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,680] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,695] jepsen worker 67 - jepsen.util 3397 :info :write [22 1] :SyncWriteAmbiguous INFO [2022-03-13 17:18:25,696] jepsen worker 67 - jepsen.util 3487 :invoke :write [22 43] INFO [2022-03-13 17:18:25,711] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,712] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,750] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,750] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,759] jepsen worker 40 - jepsen.util 4360 :info :write [13 29] :SyncWriteAmbiguous INFO [2022-03-13 17:18:25,760] jepsen worker 40 - jepsen.util 4450 :invoke :write [13 2] INFO [2022-03-13 17:18:25,763] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,763] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,764] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,764] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:18:25,765] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,765] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:18:25,785] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,786] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,808] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,808] jepsen worker 75 - jepsen.util 525 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,808] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:18:25,809] jepsen worker 75 - jepsen.util 525 :invoke :write [25 46] INFO [2022-03-13 17:18:25,818] jepsen worker 88 - jepsen.util 2428 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,818] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,819] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 13] INFO [2022-03-13 17:18:25,820] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,836] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,836] jepsen worker 70 - jepsen.util 880 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,836] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,836] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:25,837] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:25,837] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:25,850] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,850] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,850] jepsen worker 53 - jepsen.util 593 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,851] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,852] jepsen worker 53 - jepsen.util 593 :invoke :write [17 11] INFO [2022-03-13 17:18:25,853] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,853] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,854] jepsen worker 52 - jepsen.util 682 :invoke :write [17 27] INFO [2022-03-13 17:18:25,856] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,860] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,860] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,861] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:25,868] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,869] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,869] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:18:25,870] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,871] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,872] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,873] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,873] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,875] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,876] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:18:25,878] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,878] jepsen worker 59 - jepsen.util 4289 :ok :read [19 6] INFO [2022-03-13 17:18:25,879] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:18:25,886] jepsen worker 59 - jepsen.util 4289 :invoke :read [19 nil] INFO [2022-03-13 17:18:25,909] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,909] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,910] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,910] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:18:25,911] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,912] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,913] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,913] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:25,914] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,915] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:18:25,923] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,923] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:18:25,925] jepsen worker 5 - jepsen.util 2705 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,926] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:25,926] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,928] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 19] INFO [2022-03-13 17:18:25,931] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,932] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:18:25,934] jepsen worker 59 - jepsen.util 4289 :ok :read [19 6] INFO [2022-03-13 17:18:25,935] jepsen worker 59 - jepsen.util 4289 :invoke :write [19 49] INFO [2022-03-13 17:18:25,947] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:25,948] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,010] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,010] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:18:26,011] jepsen worker 76 - jepsen.util 796 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,012] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:26,023] jepsen worker 37 - jepsen.util 2467 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,023] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,028] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,028] jepsen worker 79 - jepsen.util 1159 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,028] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,029] jepsen worker 7 - jepsen.util 637 :invoke :write [2 26] INFO [2022-03-13 17:18:26,029] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:18:26,035] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:26,037] jepsen worker 71 - jepsen.util 611 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,038] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,038] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:26,039] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:26,040] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,040] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,041] jepsen worker 71 - jepsen.util 611 :invoke :write [23 23] INFO [2022-03-13 17:18:26,041] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:18:26,058] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,059] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:18:26,076] jepsen worker 46 - jepsen.util 4276 :info :write [15 47] :RequestTimeoutException INFO [2022-03-13 17:18:26,077] jepsen worker 46 - jepsen.util 4366 :invoke :read [15 nil] INFO [2022-03-13 17:18:26,115] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,115] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,128] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,128] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,129] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,129] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:18:26,134] jepsen worker 55 - jepsen.util 3925 :info :write [18 20] :RequestTimeoutException INFO [2022-03-13 17:18:26,135] jepsen worker 55 - jepsen.util 4015 :invoke :read [18 nil] INFO [2022-03-13 17:18:26,156] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,156] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,157] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,157] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,158] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,159] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,159] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:26,159] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,160] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,161] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 10] INFO [2022-03-13 17:18:26,177] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,179] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:18:26,180] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,180] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,181] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:18:26,181] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,215] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,216] jepsen worker 60 - jepsen.util 690 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,216] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:18:26,216] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,217] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:18:26,217] jepsen worker 60 - jepsen.util 690 :invoke :write [20 44] INFO [2022-03-13 17:18:26,255] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,255] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,255] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,256] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,272] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,273] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:18:26,282] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,283] jepsen worker 88 - jepsen.util 2428 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,283] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,283] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,284] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,284] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:18:26,285] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,285] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,287] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,288] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 0] INFO [2022-03-13 17:18:26,298] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,299] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,308] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,309] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:18:26,342] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,342] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,342] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,343] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,343] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,344] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,344] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,344] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,344] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,345] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,345] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,346] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:18:26,346] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,346] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,346] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,347] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,347] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,348] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:18:26,348] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,349] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,350] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,350] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,351] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,352] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:18:26,353] jepsen worker 89 - jepsen.util 2789 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,354] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,357] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,358] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:18:26,427] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:26,427] jepsen worker nemesis - couchbase.util Rebalance check count: 613 INFO [2022-03-13 17:18:26,427] jepsen worker nemesis - couchbase.util Rebalance stuck count: 485 INFO [2022-03-13 17:18:26,428] jepsen worker 26 - jepsen.util 566 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,428] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,428] jepsen worker 75 - jepsen.util 525 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,429] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:26,430] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:26,430] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,430] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:26,431] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,436] jepsen worker 71 - jepsen.util 611 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,436] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:18:26,457] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,457] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,458] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:18:26,458] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,459] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,460] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:18:26,486] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,487] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,509] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,510] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:18:26,545] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,546] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,546] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,546] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,547] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:18:26,547] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,551] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,551] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,552] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:18:26,552] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:18:26,558] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,558] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,558] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,559] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,559] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,560] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,560] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,560] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,561] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:18:26,561] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,562] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,562] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,563] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,564] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:18:26,576] jepsen worker 82 - jepsen.util 4402 :info :write [27 35] :RequestTimeoutException INFO [2022-03-13 17:18:26,577] jepsen worker 82 - jepsen.util 4492 :invoke :write [27 7] INFO [2022-03-13 17:18:26,632] jepsen worker 43 - jepsen.util 673 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,632] jepsen worker 43 - jepsen.util 673 :invoke :write [14 24] INFO [2022-03-13 17:18:26,637] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,637] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,639] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,640] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,641] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,642] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:18:26,643] jepsen worker 61 - jepsen.util 781 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,644] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,645] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:26,646] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:26,648] jepsen worker 62 - jepsen.util 512 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,648] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,649] jepsen worker 12 - jepsen.util 912 :invoke :write [4 36] INFO [2022-03-13 17:18:26,649] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:26,652] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,653] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:18:26,655] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,659] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,663] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:18:26,664] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,692] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,693] jepsen worker 71 - jepsen.util 611 :invoke :write [23 42] INFO [2022-03-13 17:18:26,706] jepsen worker 37 - jepsen.util 2467 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,707] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,730] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,731] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,739] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,739] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,789] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,789] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,789] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,790] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:18:26,792] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,793] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,799] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,800] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,801] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,801] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:18:26,805] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,806] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:18:26,824] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,824] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,824] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,824] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:18:26,827] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:18:26,827] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,828] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,828] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:26,829] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,830] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:18:26,861] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,861] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,861] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,862] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,862] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,862] jepsen worker 13 - jepsen.util 733 :invoke :write [4 29] INFO [2022-03-13 17:18:26,863] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,864] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:26,864] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,865] jepsen worker 75 - jepsen.util 525 :invoke :write [25 26] INFO [2022-03-13 17:18:26,867] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,867] jepsen worker 76 - jepsen.util 796 :invoke :write [25 0] INFO [2022-03-13 17:18:26,887] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,887] jepsen worker 69 - jepsen.util 789 :invoke :write [23 12] INFO [2022-03-13 17:18:26,894] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,894] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,894] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,895] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 1] INFO [2022-03-13 17:18:26,896] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,898] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,898] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:26,898] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:26,899] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,900] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:26,900] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:18:26,904] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 3] WARN [2022-03-13 17:18:26,931] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16697464,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16697464,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9053293,"encode_duration_us":37449,"operation_id":"0x113d9","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10000724},{"operation_name":"upsert","last_dispatch_duration_us":18173100,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18173100,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9039489,"encode_duration_us":25255,"operation_id":"0x10fd6","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10007762}],"total_count":2}} INFO [2022-03-13 17:18:26,936] jepsen worker 86 - jepsen.util 4046 :info :write [28 7] :RequestTimeoutException INFO [2022-03-13 17:18:26,936] jepsen worker 86 - jepsen.util 4136 :invoke :write [28 16] INFO [2022-03-13 17:18:26,946] jepsen worker 58 - jepsen.util 3838 :info :write [19 49] :RequestTimeoutException INFO [2022-03-13 17:18:26,947] jepsen worker 58 - jepsen.util 3928 :invoke :write [19 15] WARN [2022-03-13 17:18:27,031] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12055480,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":12055480,"total_server_duration_us":657,"operation_id":"0x1167c","timeout_ms":30000,"last_server_duration_us":657,"total_duration_us":12055832},{"operation_name":"get","last_dispatch_duration_us":11204572,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11204572,"total_server_duration_us":791020,"operation_id":"0x11747","timeout_ms":30000,"last_server_duration_us":791020,"total_duration_us":11204866},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":18219,"operation_id":"0x11531","timeout_ms":10000,"total_duration_us":10004678},{"operation_name":"upsert","last_dispatch_duration_us":16697464,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16697464,"total_server_duration_us":9053293,"encode_duration_us":37449,"operation_id":"0x113d9","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10000724},{"operation_name":"get","last_dispatch_duration_us":9964214,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9964214,"total_server_duration_us":43838,"operation_id":"0x117d9","timeout_ms":30000,"last_server_duration_us":43838,"total_duration_us":9964524},{"operation_name":"upsert","last_dispatch_duration_us":5561014,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":5561014,"total_server_duration_us":125946,"encode_duration_us":48038,"operation_id":"0x11b38","timeout_ms":10000,"last_server_duration_us":125946,"total_duration_us":5608535},{"operation_name":"upsert","last_dispatch_duration_us":3626656,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3626656,"total_server_duration_us":78555,"encode_duration_us":19455,"operation_id":"0x11b21","timeout_ms":10000,"last_server_duration_us":78555,"total_duration_us":3643801},{"operation_name":"upsert","last_dispatch_duration_us":1871351,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1871351,"total_server_duration_us":64084,"encode_duration_us":18124,"operation_id":"0x11b1f","timeout_ms":10000,"last_server_duration_us":64084,"total_duration_us":1887131},{"operation_name":"upsert","last_dispatch_duration_us":956693,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":956693,"total_server_duration_us":50886,"encode_duration_us":17856,"operation_id":"0x11ddd","timeout_ms":10000,"last_server_duration_us":50886,"total_duration_us":960545},{"operation_name":"upsert","last_dispatch_duration_us":859513,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":859513,"total_server_duration_us":59751,"encode_duration_us":22299,"operation_id":"0x11bb8","timeout_ms":10000,"last_server_duration_us":59751,"total_duration_us":865308}],"total_count":42}} INFO [2022-03-13 17:18:27,064] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,064] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,064] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,064] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,065] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:27,066] jepsen worker 6 - jepsen.util 636 :invoke :write [2 26] INFO [2022-03-13 17:18:27,066] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,067] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:27,111] jepsen worker 71 - jepsen.util 611 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,111] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,116] jepsen worker 78 - jepsen.util 708 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,116] jepsen worker 78 - jepsen.util 708 :invoke :write [26 3] INFO [2022-03-13 17:18:27,265] jepsen worker 53 - jepsen.util 593 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,265] jepsen worker 53 - jepsen.util 593 :invoke :write [17 1] INFO [2022-03-13 17:18:27,317] jepsen worker 13 - jepsen.util 733 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,317] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:27,396] jepsen worker 9 - jepsen.util 3789 :info :write [3 13] :RequestTimeoutException INFO [2022-03-13 17:18:27,396] jepsen worker 9 - jepsen.util 3879 :invoke :write [3 42] INFO [2022-03-13 17:18:27,421] jepsen worker 10 - jepsen.util 4420 :info :write [3 1] :RequestTimeoutException INFO [2022-03-13 17:18:27,421] jepsen worker 10 - jepsen.util 4510 :invoke :write [3 46] INFO [2022-03-13 17:18:27,433] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:27,433] jepsen worker nemesis - couchbase.util Rebalance check count: 614 INFO [2022-03-13 17:18:27,433] jepsen worker nemesis - couchbase.util Rebalance stuck count: 486 INFO [2022-03-13 17:18:27,476] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,476] jepsen worker 69 - jepsen.util 789 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,477] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,477] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,479] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,479] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,479] jepsen worker 7 - jepsen.util 637 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,479] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:18:27,480] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,480] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:27,481] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,481] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,482] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,482] jepsen worker 62 - jepsen.util 512 :invoke :write [20 17] INFO [2022-03-13 17:18:27,483] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,484] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,485] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,485] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:18:27,488] jepsen worker 52 - jepsen.util 682 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,489] jepsen worker 52 - jepsen.util 682 :invoke :write [17 6] INFO [2022-03-13 17:18:27,518] jepsen worker 81 - jepsen.util 4761 :info :write [27 41] :RequestTimeoutException INFO [2022-03-13 17:18:27,519] jepsen worker 81 - jepsen.util 4851 :invoke :read [27 nil] INFO [2022-03-13 17:18:27,540] jepsen worker 81 - jepsen.util 4851 :ok :read [27 4] INFO [2022-03-13 17:18:27,541] jepsen worker 81 - jepsen.util 4851 :invoke :read [27 nil] INFO [2022-03-13 17:18:27,563] jepsen worker 81 - jepsen.util 4851 :ok :read [27 4] INFO [2022-03-13 17:18:27,564] jepsen worker 81 - jepsen.util 4851 :invoke :write [27 11] INFO [2022-03-13 17:18:27,650] jepsen worker 1 - jepsen.util 3781 :info :write [0 33] :RequestTimeoutException INFO [2022-03-13 17:18:27,651] jepsen worker 1 - jepsen.util 3871 :invoke :read [0 nil] INFO [2022-03-13 17:18:27,694] jepsen worker 77 - jepsen.util 707 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,694] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,695] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:27,695] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 11] INFO [2022-03-13 17:18:27,696] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,696] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,697] jepsen worker 23 - jepsen.util 563 :invoke :write [7 40] INFO [2022-03-13 17:18:27,697] jepsen worker 61 - jepsen.util 781 :invoke :write [20 43] INFO [2022-03-13 17:18:27,752] jepsen worker 75 - jepsen.util 525 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,752] jepsen worker 75 - jepsen.util 525 :invoke :write [25 46] INFO [2022-03-13 17:18:27,874] jepsen worker 83 - jepsen.util 3863 :info :write [27 49] :RequestTimeoutException INFO [2022-03-13 17:18:27,874] jepsen worker 83 - jepsen.util 3953 :invoke :write [27 47] INFO [2022-03-13 17:18:27,891] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,892] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,892] jepsen worker 80 - jepsen.util 800 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,892] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:27,893] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:27,893] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:18:27,935] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:27,935] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:27,946] jepsen worker 66 - jepsen.util 4026 :info :write [22 45] :RequestTimeoutException INFO [2022-03-13 17:18:27,947] jepsen worker 66 - jepsen.util 4116 :invoke :read [22 nil] INFO [2022-03-13 17:18:27,956] jepsen worker 85 - jepsen.util 3865 :ok :read [28 44] INFO [2022-03-13 17:18:27,956] jepsen worker 85 - jepsen.util 3865 :invoke :read [28 nil] INFO [2022-03-13 17:18:27,962] jepsen worker 39 - jepsen.util 4539 :ok :read [13 39] INFO [2022-03-13 17:18:27,962] jepsen worker 39 - jepsen.util 4539 :invoke :write [13 9] INFO [2022-03-13 17:18:28,026] jepsen worker 85 - jepsen.util 3865 :ok :read [28 44] INFO [2022-03-13 17:18:28,027] jepsen worker 85 - jepsen.util 3865 :invoke :read [28 nil] INFO [2022-03-13 17:18:28,107] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:28,107] jepsen worker 12 - jepsen.util 912 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,107] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,107] jepsen worker 70 - jepsen.util 880 :invoke :write [23 15] INFO [2022-03-13 17:18:28,108] jepsen worker 42 - jepsen.util 402 :invoke :write [14 3] INFO [2022-03-13 17:18:28,108] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:28,115] jepsen worker 76 - jepsen.util 796 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,115] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,115] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:28,116] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:28,117] jepsen worker 85 - jepsen.util 3865 :ok :read [28 44] INFO [2022-03-13 17:18:28,118] jepsen worker 85 - jepsen.util 3865 :invoke :write [28 31] INFO [2022-03-13 17:18:28,256] jepsen worker 29 - jepsen.util 4709 :info :write [9 44] :RequestTimeoutException INFO [2022-03-13 17:18:28,257] jepsen worker 29 - jepsen.util 4799 :invoke :write [9 37] INFO [2022-03-13 17:18:28,301] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,301] jepsen worker 69 - jepsen.util 789 :invoke :write [23 0] INFO [2022-03-13 17:18:28,302] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,302] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:28,303] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,304] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:28,305] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,305] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,305] jepsen worker 78 - jepsen.util 708 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,306] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,306] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:28,306] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:28,307] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:28,307] jepsen worker 78 - jepsen.util 708 :invoke :write [26 4] INFO [2022-03-13 17:18:28,315] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,315] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:18:28,327] jepsen worker 60 - jepsen.util 690 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,327] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,328] jepsen worker 60 - jepsen.util 690 :invoke :write [20 42] INFO [2022-03-13 17:18:28,329] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] WARN [2022-03-13 17:18:28,350] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15473465,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15473465,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9023573,"encode_duration_us":21222,"operation_id":"0x11351","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":10004694}],"total_count":1}} INFO [2022-03-13 17:18:28,404] jepsen worker 69 - jepsen.util 789 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,404] jepsen worker 69 - jepsen.util 789 :invoke :write [23 2] INFO [2022-03-13 17:18:28,438] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:28,438] jepsen worker nemesis - couchbase.util Rebalance check count: 615 INFO [2022-03-13 17:18:28,438] jepsen worker nemesis - couchbase.util Rebalance stuck count: 487 INFO [2022-03-13 17:18:28,504] jepsen worker 75 - jepsen.util 525 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,504] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,504] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,504] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:28,505] jepsen worker 76 - jepsen.util 796 :invoke :write [25 16] INFO [2022-03-13 17:18:28,506] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,506] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:18:28,507] jepsen worker 75 - jepsen.util 525 :invoke :write [25 8] INFO [2022-03-13 17:18:28,527] jepsen worker 43 - jepsen.util 673 :fail :write [14 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,527] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] WARN [2022-03-13 17:18:28,532] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":33182,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":634991,"total_server_duration_us":297159,"encode_duration_us":18280,"operation_id":"0x1164e","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10004684},{"operation_name":"upsert","last_dispatch_duration_us":30516,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8651687,"total_server_duration_us":280849,"encode_duration_us":18256,"operation_id":"0x1199c","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10003989},{"operation_name":"upsert","last_dispatch_duration_us":15098453,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":15098453,"total_server_duration_us":27312,"encode_duration_us":25357,"operation_id":"0x115c8","timeout_ms":10000,"last_server_duration_us":27312,"total_duration_us":10002823},{"operation_name":"upsert","last_dispatch_duration_us":7091304,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":7442506,"total_server_duration_us":258732,"encode_duration_us":18291,"operation_id":"0x11757","timeout_ms":10000,"last_server_duration_us":47990,"total_duration_us":10002159},{"operation_name":"upsert","last_dispatch_duration_us":9094048,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9422788,"total_server_duration_us":9199624,"encode_duration_us":20569,"operation_id":"0x1168c","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9557117},{"operation_name":"get","last_dispatch_duration_us":5963877,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":5963877,"total_server_duration_us":62,"operation_id":"0x11b1d","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":5965202},{"operation_name":"upsert","last_dispatch_duration_us":2032358,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2032358,"total_server_duration_us":63439,"encode_duration_us":25892,"operation_id":"0x11ace","timeout_ms":10000,"last_server_duration_us":63439,"total_duration_us":2107057},{"operation_name":"upsert","last_dispatch_duration_us":2080259,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2080259,"total_server_duration_us":22389,"encode_duration_us":21915,"operation_id":"0x11ed3","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":2087532},{"operation_name":"get","last_dispatch_duration_us":1877029,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1877029,"total_server_duration_us":43,"operation_id":"0x11eec","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":1897965},{"operation_name":"get","last_dispatch_duration_us":1802608,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1802608,"total_server_duration_us":391,"operation_id":"0x11b04","timeout_ms":30000,"last_server_duration_us":391,"total_duration_us":1823067}],"total_count":57}} WARN [2022-03-13 17:18:28,542] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18173232,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18173232,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9027816,"encode_duration_us":31742,"operation_id":"0x110ba","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10001746}],"total_count":1}} WARN [2022-03-13 17:18:28,542] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10570623,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10570623,"total_server_duration_us":1481667,"operation_id":"0x1161e","timeout_ms":30000,"last_server_duration_us":1481667,"total_duration_us":10573217},{"operation_name":"get","last_dispatch_duration_us":10506668,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10506668,"total_server_duration_us":1428947,"operation_id":"0x1161f","timeout_ms":30000,"last_server_duration_us":1428947,"total_duration_us":10506963},{"operation_name":"upsert","last_dispatch_duration_us":8215474,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8768516,"total_server_duration_us":221595,"encode_duration_us":20184,"operation_id":"0x1191f","timeout_ms":10000,"last_server_duration_us":30162,"total_duration_us":10008644},{"operation_name":"upsert","last_dispatch_duration_us":41129,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8973533,"total_server_duration_us":249128,"encode_duration_us":18834,"operation_id":"0x11923","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10006560},{"operation_name":"upsert","last_dispatch_duration_us":87141,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8501675,"total_server_duration_us":198347,"encode_duration_us":25245,"operation_id":"0x119b8","timeout_ms":10000,"last_server_duration_us":41445,"total_duration_us":10006045},{"operation_name":"get","last_dispatch_duration_us":9389089,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9389089,"total_server_duration_us":1432339,"operation_id":"0x11698","timeout_ms":30000,"last_server_duration_us":1432339,"total_duration_us":9393172},{"operation_name":"upsert","last_dispatch_duration_us":9111226,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9111226,"total_server_duration_us":9040551,"encode_duration_us":19564,"operation_id":"0x118f2","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9118072},{"operation_name":"upsert","last_dispatch_duration_us":9080191,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9080191,"total_server_duration_us":9046921,"encode_duration_us":25178,"operation_id":"0x118e8","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":9084202},{"operation_name":"get","last_dispatch_duration_us":5352369,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5352369,"total_server_duration_us":43,"operation_id":"0x118ba","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5352712},{"operation_name":"get","last_dispatch_duration_us":4533094,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":4533094,"total_server_duration_us":27,"operation_id":"0x1195e","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":4539289}],"total_count":22}} WARN [2022-03-13 17:18:28,550] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":34497,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":2191789,"total_server_duration_us":262938,"encode_duration_us":35074,"operation_id":"0x11979","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10010180},{"operation_name":"upsert","last_dispatch_duration_us":41930,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8676132,"total_server_duration_us":115483,"encode_duration_us":19421,"operation_id":"0x1174e","timeout_ms":10000,"last_server_duration_us":27132,"total_duration_us":10009664},{"operation_name":"upsert","last_dispatch_duration_us":48543,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9094626,"total_server_duration_us":263364,"encode_duration_us":22702,"operation_id":"0x1154f","timeout_ms":10000,"last_server_duration_us":32355,"total_duration_us":10003584},{"operation_name":"upsert","last_dispatch_duration_us":36031,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9126846,"total_server_duration_us":270208,"encode_duration_us":30310,"operation_id":"0x11585","timeout_ms":10000,"last_server_duration_us":21813,"total_duration_us":10002073},{"operation_name":"upsert","last_dispatch_duration_us":67284,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8958635,"total_server_duration_us":220820,"encode_duration_us":21859,"operation_id":"0x116f7","timeout_ms":10000,"last_server_duration_us":34116,"total_duration_us":10001385},{"operation_name":"upsert","last_dispatch_duration_us":9076224,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9076224,"total_server_duration_us":9047983,"encode_duration_us":23503,"operation_id":"0x11553","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":9080665},{"operation_name":"get","last_dispatch_duration_us":5598449,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5598449,"total_server_duration_us":7541,"operation_id":"0x118ef","timeout_ms":30000,"last_server_duration_us":7541,"total_duration_us":5603198},{"operation_name":"upsert","last_dispatch_duration_us":3399932,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3399932,"total_server_duration_us":65384,"encode_duration_us":28586,"operation_id":"0x119d9","timeout_ms":10000,"last_server_duration_us":65384,"total_duration_us":3414304},{"operation_name":"get","last_dispatch_duration_us":3190925,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3190925,"total_server_duration_us":1754,"operation_id":"0x11a01","timeout_ms":30000,"last_server_duration_us":1754,"total_duration_us":3191151},{"operation_name":"get","last_dispatch_duration_us":2980354,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2980354,"total_server_duration_us":1185,"operation_id":"0x11a29","timeout_ms":30000,"last_server_duration_us":1185,"total_duration_us":2980546}],"total_count":39}} WARN [2022-03-13 17:18:28,631] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9138167,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12934259,"total_server_duration_us":9064898,"encode_duration_us":16698,"operation_id":"0x1162f","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10010394},{"operation_name":"upsert","last_dispatch_duration_us":8340883,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8340883,"total_server_duration_us":20045,"encode_duration_us":19585,"operation_id":"0x11873","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10007687},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":21693,"operation_id":"0x11961","timeout_ms":10000,"total_duration_us":10006619},{"operation_name":"upsert","last_dispatch_duration_us":37883,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9640546,"total_server_duration_us":238128,"encode_duration_us":18046,"operation_id":"0x116c1","timeout_ms":10000,"last_server_duration_us":21568,"total_duration_us":10002830},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":18720,"operation_id":"0x119f1","timeout_ms":10000,"total_duration_us":10002053},{"operation_name":"upsert","last_dispatch_duration_us":33536,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5971852,"total_server_duration_us":339605,"encode_duration_us":17723,"operation_id":"0x118df","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10002031},{"operation_name":"upsert","last_dispatch_duration_us":9067075,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9067075,"total_server_duration_us":9051168,"encode_duration_us":24137,"operation_id":"0x116b7","timeout_ms":10000,"last_server_duration_us":9051168,"total_duration_us":9072946},{"operation_name":"upsert","last_dispatch_duration_us":1631032,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1631032,"total_server_duration_us":70556,"encode_duration_us":34357,"operation_id":"0x11b31","timeout_ms":10000,"last_server_duration_us":70556,"total_duration_us":1636752},{"operation_name":"upsert","last_dispatch_duration_us":1433657,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1433657,"total_server_duration_us":107914,"encode_duration_us":32142,"operation_id":"0x11b2f","timeout_ms":10000,"last_server_duration_us":107914,"total_duration_us":1438115},{"operation_name":"get","last_dispatch_duration_us":1261285,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1261285,"total_server_duration_us":56,"operation_id":"0x11b5d","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":1261525}],"total_count":26}} WARN [2022-03-13 17:18:28,633] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18072607,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18240222,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9058803,"encode_duration_us":19603,"operation_id":"0x113e6","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10002037}],"total_count":1}} INFO [2022-03-13 17:18:28,711] jepsen worker 6 - jepsen.util 636 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,711] jepsen worker 79 - jepsen.util 1159 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,711] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,712] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:28,712] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:28,712] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:18:28,720] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,721] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] WARN [2022-03-13 17:18:28,754] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":141146,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9976004,"total_server_duration_us":199447,"encode_duration_us":16609,"operation_id":"0x117f1","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10010684},{"operation_name":"upsert","last_dispatch_duration_us":30118,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9289482,"total_server_duration_us":182872,"encode_duration_us":21446,"operation_id":"0x1187c","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10010543},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":20254,"operation_id":"0x119b1","timeout_ms":10000,"total_duration_us":10008323},{"operation_name":"upsert","last_dispatch_duration_us":31980,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":489429,"total_server_duration_us":258158,"encode_duration_us":18729,"operation_id":"0x115c9","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10003431},{"operation_name":"upsert","last_dispatch_duration_us":9220800,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9220800,"total_server_duration_us":9035244,"encode_duration_us":17757,"operation_id":"0x117f4","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":9224996},{"operation_name":"upsert","last_dispatch_duration_us":9054623,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9054623,"total_server_duration_us":9041612,"encode_duration_us":18621,"operation_id":"0x1169a","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":9058372},{"operation_name":"get","last_dispatch_duration_us":8882428,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8882428,"total_server_duration_us":357242,"operation_id":"0x1186f","timeout_ms":30000,"last_server_duration_us":357242,"total_duration_us":8884817},{"operation_name":"upsert","last_dispatch_duration_us":3180176,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3180176,"total_server_duration_us":957112,"encode_duration_us":18896,"operation_id":"0x119ef","timeout_ms":10000,"last_server_duration_us":957112,"total_duration_us":3185987},{"operation_name":"upsert","last_dispatch_duration_us":3136308,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3136308,"total_server_duration_us":41875,"encode_duration_us":17238,"operation_id":"0x11a0b","timeout_ms":10000,"last_server_duration_us":41875,"total_duration_us":3139818},{"operation_name":"upsert","last_dispatch_duration_us":3039999,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3039999,"total_server_duration_us":80968,"encode_duration_us":36835,"operation_id":"0x11a4c","timeout_ms":10000,"last_server_duration_us":80968,"total_duration_us":3045216}],"total_count":62}} INFO [2022-03-13 17:18:28,771] jepsen worker 53 - jepsen.util 593 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,772] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:28,797] jepsen worker 17 - jepsen.util 4427 :info :write [5 7] :SyncWriteAmbiguous INFO [2022-03-13 17:18:28,798] jepsen worker 17 - jepsen.util 4517 :invoke :read [5 nil] WARN [2022-03-13 17:18:28,832] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9138167,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12934259,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9064898,"encode_duration_us":16698,"operation_id":"0x1162f","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":10010394}],"total_count":1}} INFO [2022-03-13 17:18:28,912] jepsen worker 62 - jepsen.util 512 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,912] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:28,912] jepsen worker 12 - jepsen.util 912 :invoke :write [4 38] INFO [2022-03-13 17:18:28,913] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:29,040] jepsen worker 15 - jepsen.util 4155 :info :write [5 27] :RequestTimeoutException INFO [2022-03-13 17:18:29,041] jepsen worker 15 - jepsen.util 4245 :invoke :read [5 nil] INFO [2022-03-13 17:18:29,116] jepsen worker 69 - jepsen.util 789 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,116] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:18:29,117] jepsen worker 52 - jepsen.util 682 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,117] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,118] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:29,118] jepsen worker 26 - jepsen.util 566 :invoke :write [8 5] INFO [2022-03-13 17:18:29,122] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,123] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:18:29,126] jepsen worker 23 - jepsen.util 563 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,127] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:18:29,165] jepsen worker 76 - jepsen.util 796 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,166] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:29,186] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,187] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:29,193] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,194] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:29,258] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,259] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:18:29,265] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,265] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,267] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,267] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:18:29,268] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,269] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,270] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,271] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,272] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,273] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,276] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,277] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,279] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,280] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:18:29,283] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,284] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,350] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,350] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:18:29,368] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,369] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,369] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,369] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:18:29,370] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 35] INFO [2022-03-13 17:18:29,370] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,372] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,373] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,374] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,374] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,376] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,376] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,377] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,378] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:18:29,443] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:29,443] jepsen worker nemesis - couchbase.util Rebalance check count: 616 INFO [2022-03-13 17:18:29,443] jepsen worker nemesis - couchbase.util Rebalance stuck count: 488 INFO [2022-03-13 17:18:29,463] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,464] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:18:29,475] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,475] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,476] jepsen worker 88 - jepsen.util 2428 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,477] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,477] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:29,478] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:29,483] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,484] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:18:29,530] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,530] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,531] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,531] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:18:29,537] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,538] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,542] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,543] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,544] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,544] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:18:29,559] jepsen worker 3 - jepsen.util 2253 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,560] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:18:29,619] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,620] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:18:29,628] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,628] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,663] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,664] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:29,709] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,710] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:29,735] jepsen worker 37 - jepsen.util 2467 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,736] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,736] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 44] INFO [2022-03-13 17:18:29,736] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 8] INFO [2022-03-13 17:18:29,743] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,743] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,743] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:18:29,744] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:18:29,764] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,765] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:18:29,791] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,791] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:29,793] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,793] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:29,794] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,795] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:18:29,808] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,808] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,808] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,809] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:18:29,809] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:18:29,810] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:18:29,826] jepsen worker 37 - jepsen.util 2467 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,826] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,853] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,864] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,916] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:18:29,916] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:29,922] jepsen worker 88 - jepsen.util 2428 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,922] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 29] INFO [2022-03-13 17:18:29,927] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,928] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 18] INFO [2022-03-13 17:18:29,934] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,934] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:18:29,952] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,952] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,952] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:29,953] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:18:29,954] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:18:29,963] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:29,991] jepsen worker 0 - jepsen.util 4140 :ok :read [0 41] INFO [2022-03-13 17:18:29,992] jepsen worker 0 - jepsen.util 4140 :invoke :write [0 37] INFO [2022-03-13 17:18:29,992] jepsen worker 17 - jepsen.util 4517 :ok :read [5 43] INFO [2022-03-13 17:18:29,993] jepsen worker 17 - jepsen.util 4517 :invoke :read [5 nil] INFO [2022-03-13 17:18:30,016] jepsen worker 17 - jepsen.util 4517 :ok :read [5 43] INFO [2022-03-13 17:18:30,017] jepsen worker 17 - jepsen.util 4517 :invoke :write [5 19] INFO [2022-03-13 17:18:30,045] jepsen worker 88 - jepsen.util 2428 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,045] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,045] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,046] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:18:30,047] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,047] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:18:30,050] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,051] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,052] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,054] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 0] INFO [2022-03-13 17:18:30,054] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,055] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:18:30,066] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,067] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,098] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:30,098] jepsen worker 60 - jepsen.util 690 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,098] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,098] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,099] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:30,100] jepsen worker 43 - jepsen.util 673 :invoke :write [14 26] INFO [2022-03-13 17:18:30,100] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:30,101] jepsen worker 53 - jepsen.util 593 :invoke :write [17 31] INFO [2022-03-13 17:18:30,115] jepsen worker 42 - jepsen.util 402 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,115] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:30,146] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,147] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:30,151] jepsen worker 38 - jepsen.util 2828 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,151] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:18:30,151] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,152] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,158] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,158] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,159] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,160] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,161] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,162] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,163] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,163] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:18:30,168] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,168] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:30,172] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,173] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,174] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,175] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,177] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,178] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,192] jepsen worker 61 - jepsen.util 781 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,192] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,192] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:30,193] jepsen worker 61 - jepsen.util 781 :invoke :write [20 11] INFO [2022-03-13 17:18:30,201] jepsen worker 44 - jepsen.util 674 :info :write [14 3] :RequestTimeoutException INFO [2022-03-13 17:18:30,202] jepsen worker 44 - jepsen.util 764 :invoke :write [14 40] INFO [2022-03-13 17:18:30,234] jepsen worker 26 - jepsen.util 566 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,235] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:30,248] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,248] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,254] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,254] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,255] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:30,256] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,256] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,256] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:30,256] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,257] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,258] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,258] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,261] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,262] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:18:30,362] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,362] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,363] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:18:30,363] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,364] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,365] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,367] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,368] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:18:30,398] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,399] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:18:30,427] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,428] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,442] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,442] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,443] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,443] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,443] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:18:30,446] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,448] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:30,448] jepsen worker nemesis - couchbase.util Rebalance check count: 617 INFO [2022-03-13 17:18:30,448] jepsen worker nemesis - couchbase.util Rebalance stuck count: 489 INFO [2022-03-13 17:18:30,449] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,449] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,449] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:18:30,450] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 6] INFO [2022-03-13 17:18:30,454] jepsen worker 75 - jepsen.util 525 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,455] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,455] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,455] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,456] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,456] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:18:30,456] jepsen worker 75 - jepsen.util 525 :invoke :write [25 16] INFO [2022-03-13 17:18:30,457] jepsen worker 13 - jepsen.util 733 :invoke :write [4 47] INFO [2022-03-13 17:18:30,460] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,460] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:18:30,462] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:18:30,462] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:18:30,471] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,471] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,472] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:30,473] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,475] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,475] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:18:30,502] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,502] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,503] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,504] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:30,504] jepsen worker 60 - jepsen.util 690 :invoke :write [20 43] INFO [2022-03-13 17:18:30,505] jepsen worker 52 - jepsen.util 682 :invoke :write [17 1] INFO [2022-03-13 17:18:30,527] jepsen worker 37 - jepsen.util 2467 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,528] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,556] jepsen worker 36 - jepsen.util 2466 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,557] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:18:30,566] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,567] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,611] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,611] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,612] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,612] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,613] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,613] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,614] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:18:30,618] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,623] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,627] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,628] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,629] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:18:30,635] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,636] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,637] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,637] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 2] INFO [2022-03-13 17:18:30,637] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,638] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,639] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,640] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:18:30,648] jepsen worker 16 - jepsen.util 4156 :ok :read [5 43] INFO [2022-03-13 17:18:30,649] jepsen worker 16 - jepsen.util 4156 :invoke :write [5 36] INFO [2022-03-13 17:18:30,654] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,655] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,663] jepsen worker 70 - jepsen.util 880 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,663] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,663] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,664] jepsen worker 7 - jepsen.util 637 :invoke :write [2 24] INFO [2022-03-13 17:18:30,665] jepsen worker 70 - jepsen.util 880 :invoke :write [23 8] INFO [2022-03-13 17:18:30,665] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:30,678] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,678] jepsen worker 71 - jepsen.util 611 :invoke :write [23 38] INFO [2022-03-13 17:18:30,679] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,680] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,680] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:18:30,681] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:18:30,696] jepsen worker 57 - jepsen.util 4287 :info :write [19 48] :RequestTimeoutException INFO [2022-03-13 17:18:30,697] jepsen worker 57 - jepsen.util 4377 :invoke :read [19 nil] INFO [2022-03-13 17:18:30,727] jepsen worker 68 - jepsen.util 4388 :ok :read [22 30] INFO [2022-03-13 17:18:30,728] jepsen worker 68 - jepsen.util 4388 :invoke :read [22 nil] INFO [2022-03-13 17:18:30,736] jepsen worker 2 - jepsen.util 4502 :info :write [0 9] :RequestTimeoutException INFO [2022-03-13 17:18:30,737] jepsen worker 2 - jepsen.util 4592 :invoke :read [0 nil] INFO [2022-03-13 17:18:30,737] jepsen worker 84 - jepsen.util 4584 :ok :read [28 44] INFO [2022-03-13 17:18:30,738] jepsen worker 84 - jepsen.util 4584 :invoke :read [28 nil] INFO [2022-03-13 17:18:30,765] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,766] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,775] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,776] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,802] jepsen worker 1 - jepsen.util 3871 :ok :read [0 41] INFO [2022-03-13 17:18:30,802] jepsen worker 1 - jepsen.util 3871 :invoke :read [0 nil] INFO [2022-03-13 17:18:30,843] jepsen worker 68 - jepsen.util 4388 :ok :read [22 30] INFO [2022-03-13 17:18:30,843] jepsen worker 68 - jepsen.util 4388 :invoke :write [22 1] INFO [2022-03-13 17:18:30,856] jepsen worker 84 - jepsen.util 4584 :ok :read [28 44] INFO [2022-03-13 17:18:30,857] jepsen worker 84 - jepsen.util 4584 :invoke :read [28 nil] INFO [2022-03-13 17:18:30,859] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,860] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:18:30,873] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,873] jepsen worker 78 - jepsen.util 708 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,874] jepsen worker 13 - jepsen.util 733 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,874] jepsen worker 53 - jepsen.util 593 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,875] jepsen worker 78 - jepsen.util 708 :invoke :write [26 17] INFO [2022-03-13 17:18:30,875] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:30,876] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:30,877] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:30,878] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,879] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:18:30,880] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,881] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,884] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,885] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,886] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,886] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,887] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:30,888] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,888] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,888] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,888] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:30,889] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,889] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:18:30,889] jepsen worker 89 - jepsen.util 2789 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,890] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:18:30,890] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:30,891] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:18:30,895] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,896] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:18:30,898] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:18:30,901] jepsen worker 71 - jepsen.util 611 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,903] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:18:30,908] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,910] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:18:30,917] jepsen worker 45 - jepsen.util 4455 :info :write [15 30] :SyncWriteAmbiguous INFO [2022-03-13 17:18:30,922] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,925] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 35] INFO [2022-03-13 17:18:30,926] jepsen worker 45 - jepsen.util 4545 :invoke :write [15 5] INFO [2022-03-13 17:18:30,927] jepsen worker 1 - jepsen.util 3871 :ok :read [0 41] INFO [2022-03-13 17:18:30,932] jepsen worker 1 - jepsen.util 3871 :invoke :read [0 nil] INFO [2022-03-13 17:18:30,948] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,951] jepsen worker 54 - jepsen.util 3924 :info :write [18 1] :SyncWriteAmbiguous INFO [2022-03-13 17:18:30,951] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:30,952] jepsen worker 54 - jepsen.util 4014 :invoke :read [18 nil] INFO [2022-03-13 17:18:30,991] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:30,992] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:18:30,993] jepsen worker 84 - jepsen.util 4584 :ok :read [28 44] INFO [2022-03-13 17:18:30,994] jepsen worker 84 - jepsen.util 4584 :invoke :write [28 26] INFO [2022-03-13 17:18:31,023] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,023] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:18:31,027] jepsen worker 19 - jepsen.util 4159 :ok :read [6 16] INFO [2022-03-13 17:18:31,028] jepsen worker 19 - jepsen.util 4159 :invoke :read [6 nil] INFO [2022-03-13 17:18:31,055] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,055] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:31,068] jepsen worker 1 - jepsen.util 3871 :ok :read [0 41] INFO [2022-03-13 17:18:31,069] jepsen worker 1 - jepsen.util 3871 :invoke :write [0 6] INFO [2022-03-13 17:18:31,072] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,073] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:18:31,098] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,098] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:31,129] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,130] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,131] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,132] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,140] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,141] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,185] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,185] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,185] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,186] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:18:31,187] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,187] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,188] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,189] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,190] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,190] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,190] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,190] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:18:31,191] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:18:31,191] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:18:31,214] jepsen worker 88 - jepsen.util 2428 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,214] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,214] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,215] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:18:31,215] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:18:31,215] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 44] INFO [2022-03-13 17:18:31,271] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,271] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,272] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:31,273] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:18:31,282] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,282] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,282] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,283] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,283] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,284] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:31,330] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,331] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,335] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,335] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,337] jepsen worker 27 - jepsen.util 4437 :info :write [9 30] :SyncWriteAmbiguous INFO [2022-03-13 17:18:31,338] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,338] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,338] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,338] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,339] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,339] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,339] jepsen worker 27 - jepsen.util 4527 :invoke :read [9 nil] INFO [2022-03-13 17:18:31,340] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,340] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:18:31,341] jepsen worker 14 - jepsen.util 374 :invoke :write [4 46] INFO [2022-03-13 17:18:31,342] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:18:31,343] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:18:31,344] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:31,346] jepsen worker 24 - jepsen.util 654 :invoke :write [8 42] INFO [2022-03-13 17:18:31,347] jepsen worker 51 - jepsen.util 771 :invoke :write [17 40] INFO [2022-03-13 17:18:31,364] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,366] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:31,367] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,368] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,368] jepsen worker 6 - jepsen.util 636 :invoke :write [2 14] INFO [2022-03-13 17:18:31,369] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:31,370] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,375] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:31,396] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,396] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,396] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,396] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:31,397] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:18:31,397] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,397] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,398] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:18:31,398] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,399] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,400] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,401] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,402] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,402] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,403] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,404] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,405] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,405] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,407] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,407] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,408] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,409] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,410] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,410] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,411] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:18:31,411] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,412] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,413] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,414] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,415] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:18:31,428] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,429] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:18:31,453] jepsen worker 47 - jepsen.util 4187 :ok :read [15 48] INFO [2022-03-13 17:18:31,454] jepsen worker 47 - jepsen.util 4187 :invoke :read [15 nil] INFO [2022-03-13 17:18:31,455] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:31,455] jepsen worker nemesis - couchbase.util Rebalance check count: 618 INFO [2022-03-13 17:18:31,455] jepsen worker nemesis - couchbase.util Rebalance stuck count: 490 INFO [2022-03-13 17:18:31,457] jepsen worker 54 - jepsen.util 4014 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,458] jepsen worker 54 - jepsen.util 4014 :invoke :read [18 nil] INFO [2022-03-13 17:18:31,460] jepsen worker 46 - jepsen.util 4366 :ok :read [15 48] INFO [2022-03-13 17:18:31,460] jepsen worker 46 - jepsen.util 4366 :invoke :read [15 nil] INFO [2022-03-13 17:18:31,467] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,468] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:18:31,471] jepsen worker 63 - jepsen.util 4383 :info :write [21 25] :RequestTimeoutException INFO [2022-03-13 17:18:31,471] jepsen worker 63 - jepsen.util 4473 :invoke :read [21 nil] INFO [2022-03-13 17:18:31,486] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,487] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,487] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,486] jepsen worker 43 - jepsen.util 673 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,487] jepsen worker 47 - jepsen.util 4187 :ok :read [15 48] INFO [2022-03-13 17:18:31,487] jepsen worker 26 - jepsen.util 566 :invoke :write [8 10] INFO [2022-03-13 17:18:31,489] jepsen worker 47 - jepsen.util 4187 :invoke :read [15 nil] INFO [2022-03-13 17:18:31,489] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:31,489] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:18:31,494] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,494] jepsen worker 43 - jepsen.util 673 :invoke :write [14 41] INFO [2022-03-13 17:18:31,495] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,496] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,503] jepsen worker 54 - jepsen.util 4014 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,504] jepsen worker 54 - jepsen.util 4014 :invoke :write [18 23] INFO [2022-03-13 17:18:31,506] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 22] INFO [2022-03-13 17:18:31,508] jepsen worker 25 - jepsen.util 565 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,509] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,509] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:31,510] jepsen worker 80 - jepsen.util 800 :invoke :write [26 15] INFO [2022-03-13 17:18:31,514] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,516] jepsen worker 34 - jepsen.util 934 :invoke :write [11 12] INFO [2022-03-13 17:18:31,535] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,536] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:18:31,553] jepsen worker 46 - jepsen.util 4366 :ok :read [15 48] INFO [2022-03-13 17:18:31,555] jepsen worker 46 - jepsen.util 4366 :invoke :read [15 nil] INFO [2022-03-13 17:18:31,561] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,562] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:31,565] jepsen worker 47 - jepsen.util 4187 :ok :read [15 48] INFO [2022-03-13 17:18:31,566] jepsen worker 47 - jepsen.util 4187 :invoke :read [15 nil] INFO [2022-03-13 17:18:31,597] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,598] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,599] jepsen worker 30 - jepsen.util 4170 :info :write [10 4] :RequestTimeoutException INFO [2022-03-13 17:18:31,600] jepsen worker 30 - jepsen.util 4260 :invoke :write [10 27] INFO [2022-03-13 17:18:31,635] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,635] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,635] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,635] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,636] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,636] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:18:31,637] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:18:31,638] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 6] INFO [2022-03-13 17:18:31,642] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,643] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,644] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,644] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:18:31,671] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,671] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,672] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,672] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:31,678] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,678] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,679] jepsen worker 47 - jepsen.util 4187 :ok :read [15 48] INFO [2022-03-13 17:18:31,679] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:18:31,680] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:18:31,681] jepsen worker 47 - jepsen.util 4187 :invoke :write [15 45] INFO [2022-03-13 17:18:31,681] jepsen worker 46 - jepsen.util 4366 :ok :read [15 48] INFO [2022-03-13 17:18:31,682] jepsen worker 46 - jepsen.util 4366 :invoke :read [15 nil] INFO [2022-03-13 17:18:31,688] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,688] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:18:31,691] jepsen worker 60 - jepsen.util 690 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,691] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,691] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:31,691] jepsen worker 12 - jepsen.util 912 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,692] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,692] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:31,693] jepsen worker 75 - jepsen.util 525 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,693] jepsen worker 42 - jepsen.util 402 :invoke :write [14 38] INFO [2022-03-13 17:18:31,694] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:31,695] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:31,695] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,696] jepsen worker 13 - jepsen.util 733 :invoke :write [4 45] INFO [2022-03-13 17:18:31,697] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:31,697] jepsen worker 12 - jepsen.util 912 :invoke :write [4 13] INFO [2022-03-13 17:18:31,698] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,700] jepsen worker 75 - jepsen.util 525 :invoke :write [25 29] INFO [2022-03-13 17:18:31,702] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,704] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:31,712] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,712] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,714] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:31,714] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:31,725] jepsen worker 88 - jepsen.util 2428 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,726] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,744] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,744] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,767] jepsen worker 31 - jepsen.util 4261 :info :write [10 9] :RequestTimeoutException INFO [2022-03-13 17:18:31,768] jepsen worker 31 - jepsen.util 4351 :invoke :write [10 14] INFO [2022-03-13 17:18:31,775] jepsen worker 52 - jepsen.util 682 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,775] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,776] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,778] jepsen worker 76 - jepsen.util 796 :invoke :write [25 29] INFO [2022-03-13 17:18:31,779] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:31,782] jepsen worker 53 - jepsen.util 593 :invoke :write [17 3] INFO [2022-03-13 17:18:31,799] jepsen worker 46 - jepsen.util 4366 :ok :read [15 48] INFO [2022-03-13 17:18:31,799] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,800] jepsen worker 46 - jepsen.util 4366 :invoke :read [15 nil] INFO [2022-03-13 17:18:31,800] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,801] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,802] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:18:31,851] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,851] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:31,877] jepsen worker 46 - jepsen.util 4366 :ok :read [15 48] INFO [2022-03-13 17:18:31,878] jepsen worker 46 - jepsen.util 4366 :invoke :write [15 20] INFO [2022-03-13 17:18:31,882] jepsen worker 36 - jepsen.util 2466 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,883] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:18:31,897] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,898] jepsen worker 61 - jepsen.util 781 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,898] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:31,899] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 13] INFO [2022-03-13 17:18:31,900] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,900] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:31,905] jepsen worker 14 - jepsen.util 374 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,906] jepsen worker 14 - jepsen.util 374 :invoke :write [4 4] INFO [2022-03-13 17:18:31,916] jepsen worker 73 - jepsen.util 4483 :info :write [24 8] :RequestTimeoutException INFO [2022-03-13 17:18:31,916] jepsen worker 73 - jepsen.util 4573 :invoke :read [24 nil] INFO [2022-03-13 17:18:31,917] jepsen worker 37 - jepsen.util 2467 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,917] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,918] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 2] INFO [2022-03-13 17:18:31,918] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,928] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,929] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,930] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:31,938] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,939] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,940] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,941] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,942] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:18:31,969] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,969] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,969] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:18:31,970] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:31,989] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,989] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:31,990] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:31,991] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,006] jepsen worker 74 - jepsen.util 4034 :info :write [24 12] :RequestTimeoutException INFO [2022-03-13 17:18:32,007] jepsen worker 74 - jepsen.util 4124 :invoke :read [24 nil] INFO [2022-03-13 17:18:32,030] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,030] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,075] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,075] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,075] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,075] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:18:32,076] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,076] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,077] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,077] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,078] jepsen worker 74 - jepsen.util 4124 :ok :read [24 24] INFO [2022-03-13 17:18:32,078] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,079] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,079] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,079] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,079] jepsen worker 74 - jepsen.util 4124 :invoke :read [24 nil] INFO [2022-03-13 17:18:32,080] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,081] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:18:32,081] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:18:32,083] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,086] jepsen worker 18 - jepsen.util 4158 :info :write [6 7] :RequestTimeoutException INFO [2022-03-13 17:18:32,087] jepsen worker 18 - jepsen.util 4248 :invoke :read [6 nil] INFO [2022-03-13 17:18:32,092] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,093] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:18:32,100] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,100] jepsen worker 51 - jepsen.util 771 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,102] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:32,104] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:18:32,110] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,110] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,111] jepsen worker 44 - jepsen.util 764 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,111] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,111] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,111] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,111] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,113] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,113] jepsen worker 44 - jepsen.util 764 :invoke :write [14 47] INFO [2022-03-13 17:18:32,115] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,115] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:32,115] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:32,116] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,116] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:32,117] jepsen worker 71 - jepsen.util 611 :invoke :write [23 24] INFO [2022-03-13 17:18:32,117] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,119] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,120] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:18:32,129] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,130] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:18:32,145] jepsen worker 37 - jepsen.util 2467 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,145] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,146] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,146] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 36] INFO [2022-03-13 17:18:32,150] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,151] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,153] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,153] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,155] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,156] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 43] INFO [2022-03-13 17:18:32,161] jepsen worker 74 - jepsen.util 4124 :ok :read [24 24] INFO [2022-03-13 17:18:32,162] jepsen worker 74 - jepsen.util 4124 :invoke :write [24 1] INFO [2022-03-13 17:18:32,179] jepsen worker 78 - jepsen.util 708 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,179] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,179] jepsen worker 75 - jepsen.util 525 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,179] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,179] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,179] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,180] jepsen worker 78 - jepsen.util 708 :invoke :write [26 1] INFO [2022-03-13 17:18:32,180] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:32,181] jepsen worker 61 - jepsen.util 781 :invoke :write [20 39] INFO [2022-03-13 17:18:32,182] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:18:32,182] jepsen worker 75 - jepsen.util 525 :invoke :write [25 37] INFO [2022-03-13 17:18:32,183] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:18:32,209] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,210] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,227] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,228] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,235] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,236] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,240] jepsen worker 80 - jepsen.util 800 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,241] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:32,246] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,247] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,298] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,298] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,299] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,299] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,300] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,300] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,301] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,302] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,306] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,306] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,306] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,307] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:18:32,308] jepsen worker 71 - jepsen.util 611 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,308] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,308] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:18:32,308] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,309] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,309] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:32,310] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:18:32,310] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:18:32,315] jepsen worker 70 - jepsen.util 880 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,315] jepsen worker 70 - jepsen.util 880 :invoke :write [23 26] INFO [2022-03-13 17:18:32,356] jepsen worker 13 - jepsen.util 733 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,357] jepsen worker 26 - jepsen.util 566 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,358] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:32,358] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:32,365] jepsen worker 88 - jepsen.util 2428 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,366] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,390] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,390] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,390] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,391] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,392] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,392] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,393] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,394] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:18:32,394] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,395] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,395] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 40] INFO [2022-03-13 17:18:32,396] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:18:32,417] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,419] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:18:32,446] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,446] jepsen worker 37 - jepsen.util 2467 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,447] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 25] INFO [2022-03-13 17:18:32,448] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 44] INFO [2022-03-13 17:18:32,461] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:32,461] jepsen worker nemesis - couchbase.util Rebalance check count: 619 INFO [2022-03-13 17:18:32,461] jepsen worker nemesis - couchbase.util Rebalance stuck count: 491 INFO [2022-03-13 17:18:32,491] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,491] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,492] jepsen worker 34 - jepsen.util 934 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,493] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:32,493] jepsen worker 71 - jepsen.util 611 :invoke :write [23 6] INFO [2022-03-13 17:18:32,494] jepsen worker 34 - jepsen.util 934 :invoke :write [11 11] INFO [2022-03-13 17:18:32,494] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,498] jepsen worker 79 - jepsen.util 1159 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,498] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,498] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:32,499] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:32,499] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,499] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:32,500] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:18:32,534] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,534] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,536] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:18:32,536] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:18:32,547] jepsen worker 4 - jepsen.util 2614 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,548] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:18:32,585] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,585] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:32,591] jepsen worker 88 - jepsen.util 2428 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,592] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:18:32,654] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,655] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:18:32,667] jepsen worker 76 - jepsen.util 796 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,667] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,667] jepsen worker 43 - jepsen.util 673 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,668] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,669] jepsen worker 52 - jepsen.util 682 :invoke :write [17 39] INFO [2022-03-13 17:18:32,670] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,670] jepsen worker 43 - jepsen.util 673 :invoke :write [14 25] INFO [2022-03-13 17:18:32,671] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:32,672] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,672] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,672] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,672] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,672] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:32,673] jepsen worker 33 - jepsen.util 753 :invoke :write [11 12] INFO [2022-03-13 17:18:32,674] jepsen worker 21 - jepsen.util 831 :invoke :write [7 11] INFO [2022-03-13 17:18:32,675] jepsen worker 35 - jepsen.util 935 :invoke :write [11 12] INFO [2022-03-13 17:18:32,675] jepsen worker 71 - jepsen.util 611 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,675] jepsen worker 71 - jepsen.util 611 :invoke :write [23 45] INFO [2022-03-13 17:18:32,702] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,703] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:32,711] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,711] jepsen worker 37 - jepsen.util 2467 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,712] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:18:32,713] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,726] jepsen worker 41 - jepsen.util 4271 :info :write [13 24] :RequestTimeoutException INFO [2022-03-13 17:18:32,728] jepsen worker 41 - jepsen.util 4361 :invoke :write [13 24] INFO [2022-03-13 17:18:32,739] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,740] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:18:32,747] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,748] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,748] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:18:32,753] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:18:32,775] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,775] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:32,828] jepsen worker 14 - jepsen.util 374 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,829] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:32,831] jepsen worker 71 - jepsen.util 611 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,832] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:18:32,845] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,846] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,846] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:18:32,847] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,864] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,865] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,865] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,866] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,866] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:18:32,867] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,868] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,868] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,873] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,873] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,900] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,900] jepsen worker 7 - jepsen.util 637 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,900] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,900] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:32,901] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:18:32,901] jepsen worker 62 - jepsen.util 512 :invoke :write [20 34] INFO [2022-03-13 17:18:32,909] jepsen worker 24 - jepsen.util 654 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,910] jepsen worker 24 - jepsen.util 654 :invoke :write [8 19] INFO [2022-03-13 17:18:32,948] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,948] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,948] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:18:32,949] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,954] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,955] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,955] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:18:32,955] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:18:32,961] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,962] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,963] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,963] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,964] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,964] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,965] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:32,965] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,966] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,966] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,966] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,967] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,967] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 9] INFO [2022-03-13 17:18:32,968] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:32,968] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:32,968] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:33,000] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,000] jepsen worker 53 - jepsen.util 593 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,000] jepsen worker 75 - jepsen.util 525 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,000] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,001] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,001] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,002] jepsen worker 75 - jepsen.util 525 :invoke :write [25 1] INFO [2022-03-13 17:18:33,002] jepsen worker 76 - jepsen.util 796 :invoke :write [25 19] INFO [2022-03-13 17:18:33,003] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:33,005] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:33,013] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,014] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:18:33,017] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,018] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,018] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 31] INFO [2022-03-13 17:18:33,019] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,022] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:18:33,025] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,025] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,026] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 30] INFO [2022-03-13 17:18:33,027] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,029] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:18:33,049] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,050] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:18:33,089] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,089] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,089] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,090] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:33,091] jepsen worker 80 - jepsen.util 800 :invoke :write [26 10] INFO [2022-03-13 17:18:33,092] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:33,103] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,104] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,104] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,104] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,115] jepsen worker 37 - jepsen.util 2467 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,116] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:33,135] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,135] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:33,163] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,164] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,164] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,164] jepsen worker 36 - jepsen.util 2466 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,164] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,165] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:33,176] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,176] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,176] jepsen worker 71 - jepsen.util 611 :invoke :write [23 23] INFO [2022-03-13 17:18:33,177] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,178] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,179] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,180] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,181] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,181] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,182] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,182] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,182] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:33,182] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,183] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:18:33,183] jepsen worker 60 - jepsen.util 690 :invoke :write [20 16] INFO [2022-03-13 17:18:33,184] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:33,196] jepsen worker 34 - jepsen.util 934 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,197] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:33,379] jepsen worker 6 - jepsen.util 636 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,380] jepsen worker 78 - jepsen.util 708 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,380] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:33,381] jepsen worker 78 - jepsen.util 708 :invoke :write [26 10] INFO [2022-03-13 17:18:33,405] jepsen worker 71 - jepsen.util 611 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,406] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:18:33,418] jepsen worker 33 - jepsen.util 753 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,418] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:33,452] jepsen worker 43 - jepsen.util 673 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,453] jepsen worker 43 - jepsen.util 673 :invoke :write [14 10] INFO [2022-03-13 17:18:33,466] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:33,466] jepsen worker nemesis - couchbase.util Rebalance check count: 620 INFO [2022-03-13 17:18:33,466] jepsen worker nemesis - couchbase.util Rebalance stuck count: 492 INFO [2022-03-13 17:18:33,563] jepsen worker 42 - jepsen.util 402 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,564] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:33,571] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,572] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,584] jepsen worker 35 - jepsen.util 935 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,585] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:18:33,597] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,597] jepsen worker 61 - jepsen.util 781 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,597] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,598] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:33,599] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:18:33,599] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:33,654] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,655] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,654] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,658] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,658] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,659] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:18:33,660] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,660] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:33,660] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,661] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,661] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,662] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:33,663] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,663] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,664] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:18:33,664] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,668] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,668] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,669] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:33,670] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:18:33,670] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,671] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:33,672] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,672] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,673] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:33,673] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:18:33,674] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,675] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:33,676] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,677] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:33,678] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,679] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:18:33,707] jepsen worker 88 - jepsen.util 2428 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,707] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,707] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 35] INFO [2022-03-13 17:18:33,708] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:33,767] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,767] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,768] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:18:33,768] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,768] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 49] INFO [2022-03-13 17:18:33,774] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:18:33,776] jepsen worker 75 - jepsen.util 525 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,777] jepsen worker 75 - jepsen.util 525 :invoke :write [25 20] INFO [2022-03-13 17:18:33,778] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,780] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:18:33,784] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,784] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,785] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,785] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,786] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:18:33,787] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,788] jepsen worker 12 - jepsen.util 912 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,801] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,802] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:33,802] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,802] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:33,803] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:18:33,804] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,804] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:18:33,816] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,817] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:33,830] jepsen worker 20 - jepsen.util 4520 :info :write [6 23] :RequestTimeoutException INFO [2022-03-13 17:18:33,831] jepsen worker 20 - jepsen.util 4610 :invoke :read [6 nil] INFO [2022-03-13 17:18:33,849] jepsen worker 37 - jepsen.util 2467 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,850] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:33,858] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,858] jepsen worker 21 - jepsen.util 831 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,858] jepsen worker 21 - jepsen.util 831 :invoke :write [7 14] INFO [2022-03-13 17:18:33,859] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:33,863] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,864] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:33,876] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,877] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:18:33,921] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,922] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:33,930] jepsen worker 52 - jepsen.util 682 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,930] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,930] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,930] jepsen worker 52 - jepsen.util 682 :invoke :write [17 1] INFO [2022-03-13 17:18:33,930] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,931] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:33,931] jepsen worker 26 - jepsen.util 566 :invoke :write [8 35] INFO [2022-03-13 17:18:33,931] jepsen worker 53 - jepsen.util 593 :invoke :write [17 3] INFO [2022-03-13 17:18:33,938] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,939] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:18:33,952] jepsen worker 76 - jepsen.util 796 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:33,953] jepsen worker 76 - jepsen.util 796 :invoke :write [25 34] INFO [2022-03-13 17:18:33,999] jepsen worker 60 - jepsen.util 690 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,000] jepsen worker 60 - jepsen.util 690 :invoke :write [20 28] INFO [2022-03-13 17:18:34,013] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,013] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,013] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,013] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,013] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,014] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,047] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,048] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,048] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:18:34,048] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,051] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,051] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,053] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,053] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:18:34,071] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,071] jepsen worker 88 - jepsen.util 2428 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,071] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,072] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,072] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,073] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,074] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,074] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:18:34,081] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,081] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,081] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,081] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,083] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,083] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,084] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 31] INFO [2022-03-13 17:18:34,084] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,085] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,085] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:34,087] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,087] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,089] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,090] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,091] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,092] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,092] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 33] INFO [2022-03-13 17:18:34,095] jepsen worker 24 - jepsen.util 654 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,107] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:34,107] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:34,110] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,111] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,111] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,112] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,120] jepsen worker 65 - jepsen.util 4475 :info :write [21 11] :RequestTimeoutException INFO [2022-03-13 17:18:34,121] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,121] jepsen worker 44 - jepsen.util 764 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,121] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,121] jepsen worker 78 - jepsen.util 708 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,121] jepsen worker 65 - jepsen.util 4565 :invoke :read [21 nil] INFO [2022-03-13 17:18:34,122] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,123] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,123] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,123] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,123] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:34,124] jepsen worker 65 - jepsen.util 4565 :ok :read [21 :nil] INFO [2022-03-13 17:18:34,124] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:34,125] jepsen worker 65 - jepsen.util 4565 :invoke :read [21 nil] INFO [2022-03-13 17:18:34,126] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:18:34,127] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:34,127] jepsen worker 65 - jepsen.util 4565 :ok :read [21 :nil] INFO [2022-03-13 17:18:34,128] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:18:34,128] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:34,129] jepsen worker 65 - jepsen.util 4565 :invoke :read [21 nil] INFO [2022-03-13 17:18:34,131] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:18:34,131] jepsen worker 65 - jepsen.util 4565 :ok :read [21 :nil] INFO [2022-03-13 17:18:34,132] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:18:34,133] jepsen worker 65 - jepsen.util 4565 :invoke :read [21 nil] INFO [2022-03-13 17:18:34,134] jepsen worker 65 - jepsen.util 4565 :ok :read [21 :nil] INFO [2022-03-13 17:18:34,135] jepsen worker 65 - jepsen.util 4565 :invoke :read [21 nil] INFO [2022-03-13 17:18:34,137] jepsen worker 65 - jepsen.util 4565 :ok :read [21 :nil] INFO [2022-03-13 17:18:34,137] jepsen worker 65 - jepsen.util 4565 :invoke :write [21 24] INFO [2022-03-13 17:18:34,163] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,163] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,167] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:18:34,168] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:18:34,176] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,176] jepsen worker 80 - jepsen.util 800 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,176] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,176] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,176] jepsen worker 80 - jepsen.util 800 :invoke :write [26 33] INFO [2022-03-13 17:18:34,177] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,177] jepsen worker 51 - jepsen.util 771 :invoke :write [17 25] INFO [2022-03-13 17:18:34,178] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:18:34,181] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:34,181] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:34,233] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,233] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:18:34,244] jepsen worker 37 - jepsen.util 2467 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,244] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,267] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,267] jepsen worker 43 - jepsen.util 673 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,267] jepsen worker 75 - jepsen.util 525 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,267] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:34,268] jepsen worker 43 - jepsen.util 673 :invoke :write [14 41] INFO [2022-03-13 17:18:34,268] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:34,269] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,270] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:18:34,276] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,276] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,277] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,278] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,279] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,279] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,279] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:18:34,283] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:18:34,314] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,315] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:18:34,321] jepsen worker 52 - jepsen.util 682 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,322] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:34,324] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,325] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,332] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,333] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,348] jepsen worker 72 - jepsen.util 4302 :info :write [24 49] :SyncWriteAmbiguous INFO [2022-03-13 17:18:34,348] jepsen worker 72 - jepsen.util 4392 :invoke :write [24 47] INFO [2022-03-13 17:18:34,366] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,366] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,368] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,369] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:18:34,394] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,395] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,408] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,408] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,410] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:18:34,411] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:18:34,412] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,412] jepsen worker 62 - jepsen.util 512 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,412] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,412] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,412] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,412] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,412] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,413] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,416] jepsen worker 62 - jepsen.util 512 :invoke :write [20 24] INFO [2022-03-13 17:18:34,417] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:18:34,418] jepsen worker 61 - jepsen.util 781 :invoke :write [20 6] INFO [2022-03-13 17:18:34,419] jepsen worker 78 - jepsen.util 708 :invoke :write [26 20] INFO [2022-03-13 17:18:34,420] jepsen worker 76 - jepsen.util 796 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,420] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,420] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,420] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:18:34,427] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:34,428] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:34,428] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:18:34,429] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 34] INFO [2022-03-13 17:18:34,429] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:34,429] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:34,430] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,430] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,432] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:18:34,433] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,433] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,434] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,434] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,435] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 41] INFO [2022-03-13 17:18:34,442] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,444] jepsen worker 75 - jepsen.util 525 :invoke :write [25 26] INFO [2022-03-13 17:18:34,446] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,452] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,453] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,455] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:18:34,466] jepsen worker 88 - jepsen.util 2428 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,466] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,467] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,466] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,467] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,467] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:18:34,468] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 3] INFO [2022-03-13 17:18:34,468] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,468] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,469] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,470] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,470] jepsen worker 70 - jepsen.util 880 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,471] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:34,471] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,471] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,471] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,471] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:18:34,471] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,472] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:18:34,472] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,472] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,472] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:34,472] jepsen worker nemesis - couchbase.util Rebalance check count: 621 INFO [2022-03-13 17:18:34,472] jepsen worker nemesis - couchbase.util Rebalance stuck count: 493 INFO [2022-03-13 17:18:34,473] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:34,474] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:18:34,474] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,475] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:34,475] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,475] jepsen worker 6 - jepsen.util 636 :invoke :write [2 8] INFO [2022-03-13 17:18:34,477] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,478] jepsen worker 44 - jepsen.util 764 :invoke :write [14 5] INFO [2022-03-13 17:18:34,480] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,481] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,481] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,482] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,483] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:18:34,485] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,486] jepsen worker 77 - jepsen.util 707 :invoke :write [25 16] INFO [2022-03-13 17:18:34,493] jepsen worker 26 - jepsen.util 566 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,494] jepsen worker 26 - jepsen.util 566 :invoke :write [8 36] INFO [2022-03-13 17:18:34,537] jepsen worker 21 - jepsen.util 831 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,537] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,537] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,537] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,538] jepsen worker 38 - jepsen.util 2828 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,538] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:34,538] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,539] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,539] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:18:34,539] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:34,540] jepsen worker 34 - jepsen.util 934 :invoke :write [11 15] INFO [2022-03-13 17:18:34,541] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,542] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:18:34,542] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,565] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,565] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,585] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,585] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,585] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,586] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,587] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,587] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,589] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,589] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:18:34,590] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,590] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,591] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,592] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,593] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,593] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:18:34,608] jepsen worker 53 - jepsen.util 593 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,609] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:34,610] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,610] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:18:34,635] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,636] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,637] jepsen worker 56 - jepsen.util 4196 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,638] jepsen worker 56 - jepsen.util 4196 :invoke :read [18 nil] INFO [2022-03-13 17:18:34,644] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,645] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:18:34,646] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,646] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:18:34,669] jepsen worker 37 - jepsen.util 2467 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,669] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,669] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 9] INFO [2022-03-13 17:18:34,675] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,684] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,684] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,684] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,685] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:18:34,685] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 33] INFO [2022-03-13 17:18:34,687] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,687] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,688] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,688] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,689] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:34,690] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,690] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,690] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,691] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:18:34,692] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,692] jepsen worker 32 - jepsen.util 4532 :ok :read [10 :nil] INFO [2022-03-13 17:18:34,692] jepsen worker 64 - jepsen.util 4204 :ok :read [21 :nil] INFO [2022-03-13 17:18:34,692] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 2] INFO [2022-03-13 17:18:34,693] jepsen worker 64 - jepsen.util 4204 :invoke :write [21 16] INFO [2022-03-13 17:18:34,694] jepsen worker 32 - jepsen.util 4532 :invoke :write [10 20] INFO [2022-03-13 17:18:34,728] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,728] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:34,742] jepsen worker 51 - jepsen.util 771 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,743] jepsen worker 51 - jepsen.util 771 :invoke :write [17 20] INFO [2022-03-13 17:18:34,758] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,758] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,759] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:18:34,759] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,760] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,761] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,764] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,765] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:18:34,770] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,770] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,774] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,775] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:34,780] jepsen worker 67 - jepsen.util 3487 :info :write [22 43] :SyncWriteAmbiguous INFO [2022-03-13 17:18:34,782] jepsen worker 67 - jepsen.util 3577 :invoke :write [22 8] INFO [2022-03-13 17:18:34,805] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,806] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:18:34,900] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,901] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:34,912] jepsen worker 61 - jepsen.util 781 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,912] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,917] jepsen worker 56 - jepsen.util 4196 :ok :read [18 21] INFO [2022-03-13 17:18:34,962] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:34,968] jepsen worker 8 - jepsen.util 728 :invoke :write [2 4] INFO [2022-03-13 17:18:34,968] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:18:34,969] jepsen worker 56 - jepsen.util 4196 :invoke :write [18 21] INFO [2022-03-13 17:18:34,971] jepsen worker 61 - jepsen.util 781 :invoke :write [20 40] INFO [2022-03-13 17:18:35,040] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,040] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,040] jepsen worker 69 - jepsen.util 789 :invoke :write [23 41] INFO [2022-03-13 17:18:35,041] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:35,044] jepsen worker 11 - jepsen.util 4061 :info :write [3 15] :RequestTimeoutException INFO [2022-03-13 17:18:35,044] jepsen worker 11 - jepsen.util 4151 :invoke :write [3 41] INFO [2022-03-13 17:18:35,072] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,074] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:18:35,076] jepsen worker 80 - jepsen.util 800 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,078] jepsen worker 80 - jepsen.util 800 :invoke :write [26 18] INFO [2022-03-13 17:18:35,080] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,080] jepsen worker 60 - jepsen.util 690 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,081] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:35,082] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:35,086] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,087] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:35,100] jepsen worker 66 - jepsen.util 4116 :ok :read [22 30] INFO [2022-03-13 17:18:35,101] jepsen worker 66 - jepsen.util 4116 :invoke :read [22 nil] INFO [2022-03-13 17:18:35,133] jepsen worker 2 - jepsen.util 4592 :ok :read [0 41] INFO [2022-03-13 17:18:35,133] jepsen worker 2 - jepsen.util 4592 :invoke :read [0 nil] INFO [2022-03-13 17:18:35,148] jepsen worker 66 - jepsen.util 4116 :ok :read [22 30] INFO [2022-03-13 17:18:35,148] jepsen worker 66 - jepsen.util 4116 :invoke :write [22 42] INFO [2022-03-13 17:18:35,156] jepsen worker 2 - jepsen.util 4592 :ok :read [0 41] INFO [2022-03-13 17:18:35,157] jepsen worker 2 - jepsen.util 4592 :invoke :read [0 nil] INFO [2022-03-13 17:18:35,160] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,161] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:35,189] jepsen worker 2 - jepsen.util 4592 :ok :read [0 41] INFO [2022-03-13 17:18:35,190] jepsen worker 2 - jepsen.util 4592 :invoke :read [0 nil] INFO [2022-03-13 17:18:35,266] jepsen worker 2 - jepsen.util 4592 :ok :read [0 41] INFO [2022-03-13 17:18:35,266] jepsen worker 69 - jepsen.util 789 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,266] jepsen worker 2 - jepsen.util 4592 :invoke :read [0 nil] INFO [2022-03-13 17:18:35,267] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:35,271] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,271] jepsen worker 62 - jepsen.util 512 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,272] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,272] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,272] jepsen worker 78 - jepsen.util 708 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,273] jepsen worker 7 - jepsen.util 637 :invoke :write [2 18] INFO [2022-03-13 17:18:35,273] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:35,274] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:18:35,275] jepsen worker 70 - jepsen.util 880 :invoke :write [23 26] INFO [2022-03-13 17:18:35,276] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:35,278] jepsen worker 28 - jepsen.util 4258 :info :write [9 14] :RequestTimeoutException INFO [2022-03-13 17:18:35,279] jepsen worker 28 - jepsen.util 4348 :invoke :write [9 12] INFO [2022-03-13 17:18:35,311] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,311] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,312] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:35,312] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:35,377] jepsen worker 2 - jepsen.util 4592 :ok :read [0 41] INFO [2022-03-13 17:18:35,378] jepsen worker 2 - jepsen.util 4592 :invoke :read [0 nil] INFO [2022-03-13 17:18:35,449] jepsen worker 57 - jepsen.util 4377 :ok :read [19 6] INFO [2022-03-13 17:18:35,450] jepsen worker 57 - jepsen.util 4377 :invoke :write [19 39] INFO [2022-03-13 17:18:35,456] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,456] jepsen worker 43 - jepsen.util 673 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,456] jepsen worker 75 - jepsen.util 525 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,456] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:35,457] jepsen worker 52 - jepsen.util 682 :invoke :write [17 44] INFO [2022-03-13 17:18:35,457] jepsen worker 43 - jepsen.util 673 :invoke :write [14 3] INFO [2022-03-13 17:18:35,471] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,471] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,472] jepsen worker 71 - jepsen.util 611 :invoke :write [23 11] INFO [2022-03-13 17:18:35,473] jepsen worker 69 - jepsen.util 789 :invoke :write [23 8] INFO [2022-03-13 17:18:35,476] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,477] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:18:35,479] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:35,479] jepsen worker nemesis - couchbase.util Rebalance check count: 622 INFO [2022-03-13 17:18:35,479] jepsen worker nemesis - couchbase.util Rebalance stuck count: 494 INFO [2022-03-13 17:18:35,531] jepsen worker 2 - jepsen.util 4592 :ok :read [0 41] INFO [2022-03-13 17:18:35,532] jepsen worker 2 - jepsen.util 4592 :invoke :write [0 4] INFO [2022-03-13 17:18:35,655] jepsen worker 34 - jepsen.util 934 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,656] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:35,660] jepsen worker 71 - jepsen.util 611 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,661] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:35,691] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,691] jepsen worker 79 - jepsen.util 1159 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,692] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:35,692] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:35,748] jepsen worker 73 - jepsen.util 4573 :ok :read [24 24] INFO [2022-03-13 17:18:35,748] jepsen worker 73 - jepsen.util 4573 :invoke :read [24 nil] INFO [2022-03-13 17:18:35,780] jepsen worker 40 - jepsen.util 4450 :info :write [13 2] :RequestTimeoutException INFO [2022-03-13 17:18:35,781] jepsen worker 40 - jepsen.util 4540 :invoke :write [13 13] INFO [2022-03-13 17:18:35,810] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,810] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,811] jepsen worker 37 - jepsen.util 2467 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,811] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,811] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,812] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:35,812] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,813] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 49] INFO [2022-03-13 17:18:35,818] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,819] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,820] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,821] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,826] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,826] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,827] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,827] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,828] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:35,828] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,829] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,829] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,830] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,830] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,831] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,831] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:18:35,832] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,832] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:18:35,837] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,837] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,838] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,839] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:18:35,853] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,853] jepsen worker 75 - jepsen.util 525 :invoke :write [25 22] INFO [2022-03-13 17:18:35,854] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,855] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:35,856] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,857] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:35,858] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,858] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,859] jepsen worker 3 - jepsen.util 2253 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,859] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:35,860] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:18:35,861] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:35,865] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,866] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,866] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,867] jepsen worker 60 - jepsen.util 690 :invoke :write [20 40] INFO [2022-03-13 17:18:35,868] jepsen worker 22 - jepsen.util 652 :invoke :write [7 26] INFO [2022-03-13 17:18:35,868] jepsen worker 26 - jepsen.util 566 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,869] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:35,869] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:35,884] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,885] jepsen worker 25 - jepsen.util 565 :invoke :write [8 18] INFO [2022-03-13 17:18:35,887] jepsen worker 52 - jepsen.util 682 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,889] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:35,890] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,891] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,891] jepsen worker 76 - jepsen.util 796 :invoke :write [25 22] INFO [2022-03-13 17:18:35,892] jepsen worker 13 - jepsen.util 733 :invoke :write [4 9] INFO [2022-03-13 17:18:35,928] jepsen worker 43 - jepsen.util 673 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,929] jepsen worker 43 - jepsen.util 673 :invoke :write [14 32] INFO [2022-03-13 17:18:35,929] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,930] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,930] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,931] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:35,932] jepsen worker 52 - jepsen.util 682 :invoke :write [17 33] INFO [2022-03-13 17:18:35,933] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:35,940] jepsen worker 88 - jepsen.util 2428 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,942] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:35,942] jepsen worker 69 - jepsen.util 789 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,943] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,943] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:18:35,943] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:18:35,948] jepsen worker 73 - jepsen.util 4573 :ok :read [24 24] INFO [2022-03-13 17:18:35,949] jepsen worker 73 - jepsen.util 4573 :invoke :write [24 7] INFO [2022-03-13 17:18:35,955] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,956] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:18:35,970] jepsen worker 59 - jepsen.util 4289 :info :write [19 49] :RequestTimeoutException INFO [2022-03-13 17:18:35,971] jepsen worker 59 - jepsen.util 4379 :invoke :read [19 nil] INFO [2022-03-13 17:18:35,982] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:35,983] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,021] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,022] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,023] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,024] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,025] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,026] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:18:36,045] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,046] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:18:36,085] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,086] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,102] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,103] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:18:36,151] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,151] jepsen worker 37 - jepsen.util 2467 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,152] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,152] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:18:36,166] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,166] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,166] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,167] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,167] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:18:36,167] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,169] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,170] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,170] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,170] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,171] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:18:36,172] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:18:36,222] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,222] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,223] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,223] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:18:36,224] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,225] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:18:36,240] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,240] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,241] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,242] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:18:36,242] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,243] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,244] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,245] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,246] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,247] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:18:36,270] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,271] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,304] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,305] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:18:36,329] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,330] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:18:36,371] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,372] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:18:36,404] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,404] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,405] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:18:36,405] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,412] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,413] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 43] INFO [2022-03-13 17:18:36,473] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,474] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:18:36,485] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:36,485] jepsen worker nemesis - couchbase.util Rebalance check count: 623 INFO [2022-03-13 17:18:36,485] jepsen worker nemesis - couchbase.util Rebalance stuck count: 495 INFO [2022-03-13 17:18:36,486] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,487] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:18:36,493] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,493] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,494] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,494] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:18:36,539] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,539] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:18:36,562] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,563] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:18:36,597] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,597] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,597] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:18:36,598] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,604] jepsen worker 82 - jepsen.util 4492 :info :write [27 7] :RequestTimeoutException INFO [2022-03-13 17:18:36,605] jepsen worker 82 - jepsen.util 4582 :invoke :write [27 39] INFO [2022-03-13 17:18:36,616] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,617] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,623] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,623] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 28] INFO [2022-03-13 17:18:36,667] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,668] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:18:36,669] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,670] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:18:36,697] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,697] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,697] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:18:36,698] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,701] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,702] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:18:36,704] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,704] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:18:36,718] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,719] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,729] jepsen worker 37 - jepsen.util 2467 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,729] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:18:36,809] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,810] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:18:36,843] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,844] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,846] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 10] INFO [2022-03-13 17:18:36,847] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:18:36,848] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,848] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] INFO [2022-03-13 17:18:36,905] jepsen worker 88 - jepsen.util 2428 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,905] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:18:36,928] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,928] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,928] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,928] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,929] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,929] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,931] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,931] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,934] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,934] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,934] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,935] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:18:36,935] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:18:36,937] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,938] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,939] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,940] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,940] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:36,941] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,942] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 48] WARN [2022-03-13 17:18:36,946] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":12552132,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":12552132,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9056479,"encode_duration_us":19790,"operation_id":"0x11bcd","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":10003496},{"operation_name":"upsert","last_dispatch_duration_us":27036453,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27036453,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9039489,"encode_duration_us":31769,"operation_id":"0x10ff5","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10060991},{"operation_name":"upsert","last_dispatch_duration_us":11073098,"last_local_id":"A312307A00000001/00000000A19F4D82","last_remote_socket":"172.23.107.228:11210","total_dispatch_duration_us":11073098,"last_local_socket":"172.23.104.79:37462","total_server_duration_us":54818,"encode_duration_us":17702,"operation_id":"0x11b4f","timeout_ms":10000,"last_server_duration_us":54818,"total_duration_us":10005776}],"total_count":3}} INFO [2022-03-13 17:18:36,955] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,956] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,957] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,958] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,966] jepsen worker 58 - jepsen.util 3928 :info :write [19 15] :RequestTimeoutException INFO [2022-03-13 17:18:36,966] jepsen worker 86 - jepsen.util 4136 :info :write [28 16] :RequestTimeoutException INFO [2022-03-13 17:18:36,967] jepsen worker 58 - jepsen.util 4018 :invoke :write [19 18] INFO [2022-03-13 17:18:36,967] jepsen worker 86 - jepsen.util 4226 :invoke :write [28 46] INFO [2022-03-13 17:18:36,970] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,971] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,975] jepsen worker 38 - jepsen.util 2828 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,975] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,976] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,976] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,977] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,977] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,978] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:36,978] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,999] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:36,999] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:36,999] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,000] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:18:37,004] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,004] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,004] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,005] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:18:37,005] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,006] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:18:37,006] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,007] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] WARN [2022-03-13 17:18:37,046] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18103233,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":18103233,"total_server_duration_us":32,"operation_id":"0x11546","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":18110517},{"operation_name":"get","last_dispatch_duration_us":11417260,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11417260,"total_server_duration_us":614025,"operation_id":"0x11d0f","timeout_ms":30000,"last_server_duration_us":614025,"total_duration_us":11417551},{"operation_name":"get","last_dispatch_duration_us":11354918,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11354918,"total_server_duration_us":19,"operation_id":"0x118e6","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":11357810},{"operation_name":"get","last_dispatch_duration_us":9747425,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9747425,"total_server_duration_us":23,"operation_id":"0x11b37","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":11338744},{"operation_name":"get","last_dispatch_duration_us":9638027,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9638027,"total_server_duration_us":15,"operation_id":"0x11b48","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":11237922},{"operation_name":"get","last_dispatch_duration_us":9568291,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9568291,"total_server_duration_us":15,"operation_id":"0x11b4d","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":11161464},{"operation_name":"get","last_dispatch_duration_us":9967936,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":9967936,"total_server_duration_us":3310,"operation_id":"0x11b54","timeout_ms":30000,"last_server_duration_us":3310,"total_duration_us":11048589},{"operation_name":"upsert","last_dispatch_duration_us":9764248,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":17305843,"total_server_duration_us":44800,"encode_duration_us":17332,"operation_id":"0x11964","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10010280},{"operation_name":"upsert","last_dispatch_duration_us":10575940,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10575940,"total_server_duration_us":36119,"encode_duration_us":17771,"operation_id":"0x11da2","timeout_ms":10000,"last_server_duration_us":36119,"total_duration_us":10008369},{"operation_name":"upsert","last_dispatch_duration_us":11073098,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":11073098,"total_server_duration_us":54818,"encode_duration_us":17702,"operation_id":"0x11b4f","timeout_ms":10000,"last_server_duration_us":54818,"total_duration_us":10005776}],"total_count":65}} INFO [2022-03-13 17:18:37,047] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,047] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,048] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,049] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:18:37,078] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,079] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:18:37,100] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,101] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:18:37,144] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,145] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,179] jepsen worker 4 - jepsen.util 2614 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,180] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:18:37,181] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,181] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:18:37,225] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,225] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,233] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,234] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,234] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,235] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:18:37,281] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,282] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 18] INFO [2022-03-13 17:18:37,285] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,285] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:18:37,286] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,286] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:18:37,292] jepsen worker 15 - jepsen.util 4245 :ok :read [5 43] INFO [2022-03-13 17:18:37,292] jepsen worker 15 - jepsen.util 4245 :invoke :read [5 nil] INFO [2022-03-13 17:18:37,299] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,299] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,299] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,300] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:18:37,301] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,306] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:18:37,320] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,321] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:18:37,326] jepsen worker 15 - jepsen.util 4245 :ok :read [5 43] INFO [2022-03-13 17:18:37,327] jepsen worker 15 - jepsen.util 4245 :invoke :write [5 3] INFO [2022-03-13 17:18:37,384] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,384] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,385] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:18:37,386] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,400] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,401] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,420] jepsen worker 9 - jepsen.util 3879 :info :write [3 42] :RequestTimeoutException INFO [2022-03-13 17:18:37,420] jepsen worker 9 - jepsen.util 3969 :invoke :read [3 nil] INFO [2022-03-13 17:18:37,432] jepsen worker 37 - jepsen.util 2467 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,432] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 7] INFO [2022-03-13 17:18:37,448] jepsen worker 10 - jepsen.util 4510 :info :write [3 46] :RequestTimeoutException INFO [2022-03-13 17:18:37,449] jepsen worker 10 - jepsen.util 4600 :invoke :write [3 5] INFO [2022-03-13 17:18:37,462] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,463] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,463] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,463] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:18:37,464] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,465] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:18:37,468] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,468] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 24] INFO [2022-03-13 17:18:37,495] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:37,495] jepsen worker nemesis - couchbase.util Rebalance check count: 624 INFO [2022-03-13 17:18:37,495] jepsen worker nemesis - couchbase.util Rebalance stuck count: 496 INFO [2022-03-13 17:18:37,543] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,543] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,544] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:18:37,544] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 31] INFO [2022-03-13 17:18:37,554] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,554] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,557] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,557] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,558] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,558] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,589] jepsen worker 81 - jepsen.util 4851 :info :write [27 11] :RequestTimeoutException INFO [2022-03-13 17:18:37,590] jepsen worker 81 - jepsen.util 4941 :invoke :read [27 nil] INFO [2022-03-13 17:18:37,599] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,600] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:18:37,622] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,622] jepsen worker 38 - jepsen.util 2828 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,623] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,624] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:18:37,624] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,625] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,626] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,626] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,628] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,628] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:18:37,633] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,633] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,633] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,634] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:18:37,634] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,635] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:18:37,638] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,638] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:18:37,652] jepsen worker 37 - jepsen.util 2467 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,653] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:18:37,674] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,675] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:18:37,747] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,747] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 41] INFO [2022-03-13 17:18:37,750] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,750] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,777] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,778] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,828] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,828] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:18:37,832] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,833] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:18:37,836] jepsen worker 88 - jepsen.util 2428 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,836] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,836] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 47] INFO [2022-03-13 17:18:37,837] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,843] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,844] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:37,845] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,846] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:18:37,884] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,884] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,885] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,886] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,887] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,888] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,896] jepsen worker 83 - jepsen.util 3953 :info :write [27 47] :RequestTimeoutException INFO [2022-03-13 17:18:37,897] jepsen worker 83 - jepsen.util 4043 :invoke :read [27 nil] INFO [2022-03-13 17:18:37,922] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,922] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,923] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:37,923] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,929] jepsen worker 38 - jepsen.util 2828 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,929] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:37,948] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,948] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,949] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:18:37,951] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 12] INFO [2022-03-13 17:18:37,990] jepsen worker 39 - jepsen.util 4539 :info :write [13 9] :RequestTimeoutException INFO [2022-03-13 17:18:37,991] jepsen worker 39 - jepsen.util 4629 :invoke :read [13 nil] INFO [2022-03-13 17:18:37,995] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,995] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:37,996] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:37,998] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:18:38,009] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,009] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,010] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,011] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:18:38,011] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:18:38,012] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:18:38,101] jepsen worker 83 - jepsen.util 4043 :ok :read [27 4] INFO [2022-03-13 17:18:38,102] jepsen worker 83 - jepsen.util 4043 :invoke :read [27 nil] INFO [2022-03-13 17:18:38,112] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,113] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,119] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,120] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,120] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,121] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,121] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:18:38,122] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,122] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,122] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,123] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,123] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:18:38,139] jepsen worker 83 - jepsen.util 4043 :ok :read [27 4] INFO [2022-03-13 17:18:38,139] jepsen worker 83 - jepsen.util 4043 :invoke :read [27 nil] INFO [2022-03-13 17:18:38,141] jepsen worker 85 - jepsen.util 3865 :info :write [28 31] :RequestTimeoutException INFO [2022-03-13 17:18:38,141] jepsen worker 85 - jepsen.util 3955 :invoke :write [28 1] INFO [2022-03-13 17:18:38,180] jepsen worker 88 - jepsen.util 2428 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,181] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 39] INFO [2022-03-13 17:18:38,187] jepsen worker 83 - jepsen.util 4043 :ok :read [27 4] INFO [2022-03-13 17:18:38,187] jepsen worker 83 - jepsen.util 4043 :invoke :read [27 nil] INFO [2022-03-13 17:18:38,209] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,210] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,215] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,216] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:38,231] jepsen worker 83 - jepsen.util 4043 :ok :read [27 4] INFO [2022-03-13 17:18:38,231] jepsen worker 83 - jepsen.util 4043 :invoke :read [27 nil] INFO [2022-03-13 17:18:38,257] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,257] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:38,263] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,263] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:38,264] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,265] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:18:38,281] jepsen worker 29 - jepsen.util 4799 :info :write [9 37] :RequestTimeoutException INFO [2022-03-13 17:18:38,282] jepsen worker 29 - jepsen.util 4889 :invoke :read [9 nil] INFO [2022-03-13 17:18:38,283] jepsen worker 83 - jepsen.util 4043 :ok :read [27 4] INFO [2022-03-13 17:18:38,284] jepsen worker 83 - jepsen.util 4043 :invoke :write [27 5] INFO [2022-03-13 17:18:38,284] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,284] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,284] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,285] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:18:38,286] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,289] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,289] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,290] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,290] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:18:38,294] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,294] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,295] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:18:38,296] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,297] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,298] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,298] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,299] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,300] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,301] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:18:38,302] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 27] INFO [2022-03-13 17:18:38,319] jepsen worker 37 - jepsen.util 2467 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,320] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,383] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,383] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:18:38,394] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,395] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,451] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,452] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,477] jepsen worker 89 - jepsen.util 2789 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,477] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,478] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,479] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:18:38,499] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,500] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:18:38,502] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:38,502] jepsen worker nemesis - couchbase.util Rebalance check count: 625 INFO [2022-03-13 17:18:38,502] jepsen worker nemesis - couchbase.util Rebalance stuck count: 497 INFO [2022-03-13 17:18:38,507] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,508] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:18:38,528] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,529] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:18:38,552] jepsen worker 88 - jepsen.util 2428 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,552] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:18:38,552] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13905789,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":13905789,"total_server_duration_us":51238,"operation_id":"0x1193c","timeout_ms":30000,"last_server_duration_us":51238,"total_duration_us":13908381},{"operation_name":"upsert","last_dispatch_duration_us":9040570,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9616471,"total_server_duration_us":9286452,"encode_duration_us":26400,"operation_id":"0x11eb8","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10008398},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":18431,"operation_id":"0x11d81","timeout_ms":10000,"total_duration_us":10007355},{"operation_name":"upsert","last_dispatch_duration_us":8719581,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9063218,"total_server_duration_us":228161,"encode_duration_us":18366,"operation_id":"0x11f7e","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10006693},{"operation_name":"upsert","last_dispatch_duration_us":9473041,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9473041,"total_server_duration_us":53015,"encode_duration_us":24984,"operation_id":"0x11bf1","timeout_ms":10000,"last_server_duration_us":53015,"total_duration_us":10003251},{"operation_name":"get","last_dispatch_duration_us":8233462,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8233462,"total_server_duration_us":49,"operation_id":"0x11fdd","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8235749},{"operation_name":"upsert","last_dispatch_duration_us":2441401,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":2441401,"total_server_duration_us":51592,"encode_duration_us":17371,"operation_id":"0x11f3a","timeout_ms":10000,"last_server_duration_us":51592,"total_duration_us":2445162},{"operation_name":"upsert","last_dispatch_duration_us":2372802,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":2372802,"total_server_duration_us":2351281,"encode_duration_us":19208,"operation_id":"0x11f39","timeout_ms":10000,"last_server_duration_us":2351281,"total_duration_us":2376777},{"operation_name":"upsert","last_dispatch_duration_us":1862241,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1862241,"total_server_duration_us":18417,"encode_duration_us":25728,"operation_id":"0x11f26","timeout_ms":10000,"last_server_duration_us":18417,"total_duration_us":1868379},{"operation_name":"upsert","last_dispatch_duration_us":1747182,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1747182,"total_server_duration_us":907831,"encode_duration_us":18047,"operation_id":"0x11faa","timeout_ms":10000,"last_server_duration_us":907831,"total_duration_us":1750936}],"total_count":56}} INFO [2022-03-13 17:18:38,553] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,553] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,554] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,554] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,555] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 24] INFO [2022-03-13 17:18:38,559] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] WARN [2022-03-13 17:18:38,559] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9122115,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17890631,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9275950,"encode_duration_us":20184,"operation_id":"0x1191f","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10008644}],"total_count":1}} WARN [2022-03-13 17:18:38,560] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":72914,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":12001431,"total_server_duration_us":2799,"operation_id":"0x1137b","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":30007022},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17614,"operation_id":"0x11e3f","timeout_ms":10000,"total_duration_us":10010502},{"operation_name":"upsert","last_dispatch_duration_us":40258,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9556031,"total_server_duration_us":216287,"encode_duration_us":18132,"operation_id":"0x11f4e","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10010374},{"operation_name":"upsert","last_dispatch_duration_us":8422748,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9068987,"total_server_duration_us":170217,"encode_duration_us":29264,"operation_id":"0x11c3f","timeout_ms":10000,"last_server_duration_us":41983,"total_duration_us":10008619},{"operation_name":"upsert","last_dispatch_duration_us":8588152,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14394697,"total_server_duration_us":190108,"encode_duration_us":17041,"operation_id":"0x11b9b","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10008104},{"operation_name":"upsert","last_dispatch_duration_us":108461,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":734987,"total_server_duration_us":181045,"encode_duration_us":19445,"operation_id":"0x11c1a","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10005345},{"operation_name":"upsert","last_dispatch_duration_us":8399839,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9156381,"total_server_duration_us":183941,"encode_duration_us":19251,"operation_id":"0x11c32","timeout_ms":10000,"last_server_duration_us":27945,"total_duration_us":10002976},{"operation_name":"upsert","last_dispatch_duration_us":30904,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8733991,"total_server_duration_us":225294,"encode_duration_us":18524,"operation_id":"0x11f93","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10002419},{"operation_name":"upsert","last_dispatch_duration_us":8406169,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8926287,"total_server_duration_us":133122,"encode_duration_us":31841,"operation_id":"0x11c48","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":10002123},{"operation_name":"upsert","last_dispatch_duration_us":9092605,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9092605,"total_server_duration_us":9026756,"encode_duration_us":29687,"operation_id":"0x11c2d","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":9102336}],"total_count":22}} WARN [2022-03-13 17:18:38,565] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":31923,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":4635177,"total_server_duration_us":278898,"encode_duration_us":17082,"operation_id":"0x11f59","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10008923},{"operation_name":"upsert","last_dispatch_duration_us":9067295,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9067295,"total_server_duration_us":9037366,"encode_duration_us":20434,"operation_id":"0x11c5e","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":9072460},{"operation_name":"upsert","last_dispatch_duration_us":9047836,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9047836,"total_server_duration_us":9033121,"encode_duration_us":31871,"operation_id":"0x11b01","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":9053880},{"operation_name":"get","last_dispatch_duration_us":8450421,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8450421,"total_server_duration_us":1134278,"operation_id":"0x11bdb","timeout_ms":30000,"last_server_duration_us":1134278,"total_duration_us":8453044},{"operation_name":"get","last_dispatch_duration_us":7663978,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7663978,"total_server_duration_us":16339,"operation_id":"0x11d6f","timeout_ms":30000,"last_server_duration_us":16339,"total_duration_us":7666203},{"operation_name":"get","last_dispatch_duration_us":5353593,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":5353593,"total_server_duration_us":38,"operation_id":"0x11ec3","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":5356388},{"operation_name":"upsert","last_dispatch_duration_us":2543287,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2543287,"total_server_duration_us":208052,"encode_duration_us":16986,"operation_id":"0x11fa7","timeout_ms":10000,"last_server_duration_us":208052,"total_duration_us":2548200},{"operation_name":"upsert","last_dispatch_duration_us":2468222,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2468222,"total_server_duration_us":67617,"encode_duration_us":17915,"operation_id":"0x11f89","timeout_ms":10000,"last_server_duration_us":67617,"total_duration_us":2476156},{"operation_name":"upsert","last_dispatch_duration_us":2214022,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2214022,"total_server_duration_us":82835,"encode_duration_us":18231,"operation_id":"0x11fe1","timeout_ms":10000,"last_server_duration_us":82835,"total_duration_us":2218124},{"operation_name":"get","last_dispatch_duration_us":2160594,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2160594,"total_server_duration_us":38,"operation_id":"0x11fcf","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":2160826}],"total_count":35}} INFO [2022-03-13 17:18:38,576] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,577] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] WARN [2022-03-13 17:18:38,595] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17699967,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18189396,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9282792,"encode_duration_us":18729,"operation_id":"0x115c9","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10003431}],"total_count":1}} INFO [2022-03-13 17:18:38,623] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,623] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,624] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:18:38,624] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:18:38,626] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,627] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,628] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,628] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,629] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,630] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] WARN [2022-03-13 17:18:38,652] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7027194,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9157291,"total_server_duration_us":442914,"encode_duration_us":16804,"operation_id":"0x11b9d","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10004345},{"operation_name":"upsert","last_dispatch_duration_us":9084565,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9641343,"total_server_duration_us":9335888,"encode_duration_us":21503,"operation_id":"0x11cb4","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10002382},{"operation_name":"upsert","last_dispatch_duration_us":60552,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8033449,"total_server_duration_us":267728,"encode_duration_us":16979,"operation_id":"0x11f4f","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10001438},{"operation_name":"upsert","last_dispatch_duration_us":9057873,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9057873,"total_server_duration_us":9044797,"encode_duration_us":20076,"operation_id":"0x11e8d","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9061684},{"operation_name":"get","last_dispatch_duration_us":7131183,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7131183,"total_server_duration_us":49,"operation_id":"0x11f96","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":7135314},{"operation_name":"get","last_dispatch_duration_us":4377942,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4377942,"total_server_duration_us":62,"operation_id":"0x120af","timeout_ms":30000,"last_server_duration_us":62,"total_duration_us":4380228},{"operation_name":"upsert","last_dispatch_duration_us":2428177,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":2428177,"total_server_duration_us":2350682,"encode_duration_us":18272,"operation_id":"0x11f3b","timeout_ms":10000,"last_server_duration_us":2350682,"total_duration_us":2433586},{"operation_name":"upsert","last_dispatch_duration_us":1917252,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1917252,"total_server_duration_us":206773,"encode_duration_us":16998,"operation_id":"0x11fbb","timeout_ms":10000,"last_server_duration_us":206773,"total_duration_us":1930050},{"operation_name":"upsert","last_dispatch_duration_us":1611423,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1611423,"total_server_duration_us":37447,"encode_duration_us":17058,"operation_id":"0x11fba","timeout_ms":10000,"last_server_duration_us":37447,"total_duration_us":1621290},{"operation_name":"get","last_dispatch_duration_us":1287839,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1287839,"total_server_duration_us":49,"operation_id":"0x11fe3","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":1288246}],"total_count":27}} WARN [2022-03-13 17:18:38,652] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16131644,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":21162037,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9349746,"encode_duration_us":19763,"operation_id":"0x11398","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10003112},{"operation_name":"upsert","last_dispatch_duration_us":16083475,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16718466,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9324975,"encode_duration_us":18280,"operation_id":"0x1164e","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10004684},{"operation_name":"upsert","last_dispatch_duration_us":9040570,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9616471,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9286452,"encode_duration_us":26400,"operation_id":"0x11eb8","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10008398}],"total_count":3}} INFO [2022-03-13 17:18:38,654] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,655] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,655] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:38,656] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:38,661] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,662] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,662] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:18:38,662] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:18:38,668] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,669] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:38,671] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,671] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:38,734] jepsen worker 37 - jepsen.util 2467 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,735] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 32] INFO [2022-03-13 17:18:38,741] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,741] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] WARN [2022-03-13 17:18:38,753] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9060531,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15032383,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9378033,"encode_duration_us":17723,"operation_id":"0x118df","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10002031},{"operation_name":"upsert","last_dispatch_duration_us":17376370,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26031298,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9156635,"encode_duration_us":21952,"operation_id":"0x113d7","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10005118},{"operation_name":"upsert","last_dispatch_duration_us":9084565,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9641343,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9335888,"encode_duration_us":21503,"operation_id":"0x11cb4","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10002382}],"total_count":3}} INFO [2022-03-13 17:18:38,773] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,774] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] WARN [2022-03-13 17:18:38,795] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13338416,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13338416,"total_server_duration_us":43,"operation_id":"0x11958","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":13340434},{"operation_name":"get","last_dispatch_duration_us":10349974,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10349974,"total_server_duration_us":32065,"operation_id":"0x11b60","timeout_ms":30000,"last_server_duration_us":32065,"total_duration_us":10352467},{"operation_name":"upsert","last_dispatch_duration_us":43152,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":5555031,"total_server_duration_us":294717,"encode_duration_us":17512,"operation_id":"0x11f1a","timeout_ms":10000,"last_server_duration_us":28584,"total_duration_us":10009190},{"operation_name":"upsert","last_dispatch_duration_us":52438,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":450999,"total_server_duration_us":129880,"encode_duration_us":20692,"operation_id":"0x11e29","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10008370},{"operation_name":"upsert","last_dispatch_duration_us":9158651,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9253740,"total_server_duration_us":9050936,"encode_duration_us":25791,"operation_id":"0x11e32","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":9264389},{"operation_name":"get","last_dispatch_duration_us":9077985,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9077985,"total_server_duration_us":43,"operation_id":"0x11c07","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9080322},{"operation_name":"get","last_dispatch_duration_us":3134123,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":3134123,"total_server_duration_us":43,"operation_id":"0x11f85","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":3136364},{"operation_name":"upsert","last_dispatch_duration_us":2756870,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2756870,"total_server_duration_us":297999,"encode_duration_us":18015,"operation_id":"0x11fdc","timeout_ms":10000,"last_server_duration_us":297999,"total_duration_us":2760349},{"operation_name":"upsert","last_dispatch_duration_us":2622033,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2622033,"total_server_duration_us":326662,"encode_duration_us":27601,"operation_id":"0x11fd2","timeout_ms":10000,"last_server_duration_us":326662,"total_duration_us":2626152},{"operation_name":"upsert","last_dispatch_duration_us":2555289,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000004/0000000095638474","last_local_socket":"172.23.104.79:33498","total_dispatch_duration_us":2555289,"total_server_duration_us":62668,"encode_duration_us":17240,"operation_id":"0x11f4c","timeout_ms":10000,"last_server_duration_us":62668,"total_duration_us":2558631}],"total_count":43}} INFO [2022-03-13 17:18:38,817] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,818] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:18:38,875] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,876] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:18:38,878] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,879] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:38,978] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,978] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,978] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,979] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 14] INFO [2022-03-13 17:18:38,980] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:38,980] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:18:38,992] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:38,993] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:18:38,999] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,000] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:18:39,054] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,055] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 19] INFO [2022-03-13 17:18:39,064] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,064] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,064] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,065] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:18:39,108] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,109] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:18:39,121] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,122] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:18:39,143] jepsen worker 37 - jepsen.util 2467 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,144] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 10] INFO [2022-03-13 17:18:39,226] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,226] jepsen worker 4 - jepsen.util 2614 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,227] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,227] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,234] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,235] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,280] jepsen worker 5 - jepsen.util 2705 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,280] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:18:39,322] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,322] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,322] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,322] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:18:39,323] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 40] INFO [2022-03-13 17:18:39,323] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,324] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,325] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,326] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,327] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:18:39,331] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,331] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,332] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:18:39,332] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:18:39,343] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,345] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,399] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,400] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,401] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,402] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,423] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,424] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,425] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,426] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:18:39,429] jepsen worker 4 - jepsen.util 2614 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,429] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,430] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:18:39,431] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:39,514] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:39,514] jepsen worker nemesis - couchbase.util Rebalance check count: 626 INFO [2022-03-13 17:18:39,514] jepsen worker nemesis - couchbase.util Rebalance stuck count: 498 INFO [2022-03-13 17:18:39,515] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,515] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,515] jepsen worker 37 - jepsen.util 2467 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,516] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,516] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 18] INFO [2022-03-13 17:18:39,517] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:18:39,518] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,522] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 9] INFO [2022-03-13 17:18:39,522] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,522] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,523] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,524] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,537] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,538] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:18:39,564] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,565] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,593] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,593] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,593] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:18:39,594] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 27] INFO [2022-03-13 17:18:39,645] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,645] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,649] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,649] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:18:39,695] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,695] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,696] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 25] INFO [2022-03-13 17:18:39,697] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,699] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:18:39,700] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:18:39,708] jepsen worker 0 - jepsen.util 4140 :info :write [0 37] :SyncWriteAmbiguous INFO [2022-03-13 17:18:39,709] jepsen worker 0 - jepsen.util 4230 :invoke :write [0 45] INFO [2022-03-13 17:18:39,723] jepsen worker 89 - jepsen.util 2789 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,723] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,758] jepsen worker 80 - jepsen.util 800 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,758] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,759] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,759] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,759] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:39,760] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:18:39,760] jepsen worker 24 - jepsen.util 654 :invoke :write [8 45] INFO [2022-03-13 17:18:39,761] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:39,776] jepsen worker 88 - jepsen.util 2428 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,777] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,793] jepsen worker 3 - jepsen.util 2253 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,794] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:18:39,815] jepsen worker 51 - jepsen.util 771 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,815] jepsen worker 51 - jepsen.util 771 :invoke :write [17 21] INFO [2022-03-13 17:18:39,845] jepsen worker 39 - jepsen.util 4629 :ok :read [13 39] INFO [2022-03-13 17:18:39,845] jepsen worker 39 - jepsen.util 4629 :invoke :read [13 nil] INFO [2022-03-13 17:18:39,850] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,850] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,851] jepsen worker 75 - jepsen.util 525 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,851] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,852] jepsen worker 75 - jepsen.util 525 :invoke :write [25 17] INFO [2022-03-13 17:18:39,855] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,857] jepsen worker 44 - jepsen.util 764 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,857] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,858] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:18:39,858] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,859] jepsen worker 44 - jepsen.util 764 :invoke :write [14 3] INFO [2022-03-13 17:18:39,859] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,862] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,862] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:18:39,879] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,883] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,883] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:18:39,884] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:39,887] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,888] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,888] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,888] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,892] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,892] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,892] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,893] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,895] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,895] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,895] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,896] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:18:39,896] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,897] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:39,898] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,899] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:18:39,923] jepsen worker 39 - jepsen.util 4629 :ok :read [13 39] INFO [2022-03-13 17:18:39,945] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,945] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,622] jepsen worker 39 - jepsen.util 4629 :invoke :read [13 nil] INFO [2022-03-13 17:18:39,945] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,622] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:18:40,622] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:39,947] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,622] jepsen worker 34 - jepsen.util 934 :invoke :write [11 43] INFO [2022-03-13 17:18:39,971] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,622] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:39,978] jepsen worker 22 - jepsen.util 652 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,622] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:40,622] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:39,997] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:39,997] jepsen worker 37 - jepsen.util 2467 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,624] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 22] INFO [2022-03-13 17:18:40,624] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 38] INFO [2022-03-13 17:18:40,625] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,625] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,626] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,626] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:18:40,628] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:18:40,631] jepsen worker 43 - jepsen.util 673 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,001] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,047] jepsen worker 17 - jepsen.util 4517 :info :write [5 19] :RequestTimeoutException INFO [2022-03-13 17:18:40,051] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,051] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,064] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,632] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 6] INFO [2022-03-13 17:18:40,084] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,632] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:18:40,632] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:40,108] jepsen worker 25 - jepsen.util 565 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,632] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:40,108] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,632] jepsen worker 80 - jepsen.util 800 :invoke :write [26 40] INFO [2022-03-13 17:18:40,108] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,137] jepsen worker 24 - jepsen.util 654 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,214] jepsen worker 19 - jepsen.util 4159 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,638] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:40,228] jepsen worker 55 - jepsen.util 4015 :ok :read [18 21] INFO [2022-03-13 17:18:40,638] jepsen worker 19 - jepsen.util 4159 :invoke :write [6 0] INFO [2022-03-13 17:18:40,264] jepsen worker 13 - jepsen.util 733 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,638] jepsen worker 55 - jepsen.util 4015 :invoke :write [18 40] INFO [2022-03-13 17:18:40,638] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:18:40,639] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,271] jepsen worker 77 - jepsen.util 707 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,640] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:40,272] jepsen worker 6 - jepsen.util 636 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,296] jepsen worker 18 - jepsen.util 4248 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,642] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:40,297] jepsen worker 27 - jepsen.util 4527 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,642] jepsen worker 18 - jepsen.util 4248 :invoke :write [6 6] INFO [2022-03-13 17:18:40,329] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,642] jepsen worker 27 - jepsen.util 4527 :invoke :write [9 22] INFO [2022-03-13 17:18:40,642] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:18:40,329] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,642] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:40,334] jepsen worker 60 - jepsen.util 690 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,646] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:40,650] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,335] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,335] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,448] jepsen worker 51 - jepsen.util 771 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,650] jepsen worker 26 - jepsen.util 566 :invoke :write [8 31] INFO [2022-03-13 17:18:40,650] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:40,461] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,461] jepsen worker 8 - jepsen.util 728 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,461] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,462] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,650] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:40,462] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,462] jepsen worker 61 - jepsen.util 781 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,469] jepsen worker 76 - jepsen.util 796 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,519] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:40,651] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:18:40,651] jepsen worker 76 - jepsen.util 796 :invoke :write [25 29] INFO [2022-03-13 17:18:40,651] jepsen worker nemesis - couchbase.util Rebalance check count: 627 INFO [2022-03-13 17:18:40,651] jepsen worker nemesis - couchbase.util Rebalance stuck count: 499 INFO [2022-03-13 17:18:40,597] jepsen worker 20 - jepsen.util 4610 :ok :read [6 16] INFO [2022-03-13 17:18:40,631] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:18:40,631] jepsen worker 17 - jepsen.util 4607 :invoke :read [5 nil] INFO [2022-03-13 17:18:40,651] jepsen worker 20 - jepsen.util 4610 :invoke :write [6 31] INFO [2022-03-13 17:18:40,652] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,632] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:40,638] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:40,642] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,642] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:40,650] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,650] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:40,650] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:40,658] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,650] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:40,650] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:40,651] jepsen worker 62 - jepsen.util 512 :invoke :write [20 17] INFO [2022-03-13 17:18:40,659] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,667] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,670] jepsen worker 43 - jepsen.util 673 :invoke :write [14 14] INFO [2022-03-13 17:18:40,671] jepsen worker 51 - jepsen.util 771 :invoke :write [17 5] INFO [2022-03-13 17:18:40,672] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:40,673] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,675] jepsen worker 75 - jepsen.util 525 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,676] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:40,677] jepsen worker 25 - jepsen.util 565 :invoke :write [8 23] INFO [2022-03-13 17:18:40,678] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:18:40,679] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:40,680] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:18:40,681] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:18:40,681] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:40,684] jepsen worker 16 - jepsen.util 4156 :info :write [5 36] :RequestTimeoutException INFO [2022-03-13 17:18:40,684] jepsen worker 24 - jepsen.util 654 :invoke :write [8 35] INFO [2022-03-13 17:18:40,685] jepsen worker 16 - jepsen.util 4246 :invoke :write [5 12] INFO [2022-03-13 17:18:40,749] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,750] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:18:40,755] jepsen worker 39 - jepsen.util 4629 :ok :read [13 39] INFO [2022-03-13 17:18:40,755] jepsen worker 39 - jepsen.util 4629 :invoke :write [13 11] INFO [2022-03-13 17:18:40,770] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,771] jepsen worker 33 - jepsen.util 753 :invoke :write [11 9] INFO [2022-03-13 17:18:40,771] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,772] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:40,785] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,785] jepsen worker 52 - jepsen.util 682 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,786] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,787] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:40,788] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,788] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,788] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:18:40,789] jepsen worker 75 - jepsen.util 525 :invoke :write [25 25] INFO [2022-03-13 17:18:40,789] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:40,790] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:40,804] jepsen worker 54 - jepsen.util 4014 :info :write [18 23] :SyncWriteAmbiguous INFO [2022-03-13 17:18:40,805] jepsen worker 54 - jepsen.util 4104 :invoke :read [18 nil] INFO [2022-03-13 17:18:40,812] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,812] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:18:40,821] jepsen worker 4 - jepsen.util 2614 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,822] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 4] INFO [2022-03-13 17:18:40,851] jepsen worker 47 - jepsen.util 4187 :info :write [15 45] :SyncWriteAmbiguous INFO [2022-03-13 17:18:40,852] jepsen worker 47 - jepsen.util 4277 :invoke :read [15 nil] INFO [2022-03-13 17:18:40,863] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,864] jepsen worker 68 - jepsen.util 4388 :info :write [22 1] :RequestTimeoutException INFO [2022-03-13 17:18:40,864] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:18:40,867] jepsen worker 68 - jepsen.util 4478 :invoke :read [22 nil] INFO [2022-03-13 17:18:40,868] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,868] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,868] jepsen worker 70 - jepsen.util 880 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,869] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,869] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,870] jepsen worker 23 - jepsen.util 563 :invoke :write [7 40] INFO [2022-03-13 17:18:40,871] jepsen worker 77 - jepsen.util 707 :invoke :write [25 37] INFO [2022-03-13 17:18:40,872] jepsen worker 78 - jepsen.util 708 :invoke :write [26 4] INFO [2022-03-13 17:18:40,874] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:40,874] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:40,883] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,884] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:40,906] jepsen worker 37 - jepsen.util 2467 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,906] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,907] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:40,908] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:18:40,943] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,944] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:18:40,947] jepsen worker 34 - jepsen.util 934 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,947] jepsen worker 34 - jepsen.util 934 :invoke :write [11 25] INFO [2022-03-13 17:18:40,960] jepsen worker 45 - jepsen.util 4545 :info :write [15 5] :RequestTimeoutException INFO [2022-03-13 17:18:40,961] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,961] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,961] jepsen worker 45 - jepsen.util 4635 :invoke :read [15 nil] INFO [2022-03-13 17:18:40,962] jepsen worker 69 - jepsen.util 789 :invoke :write [23 43] INFO [2022-03-13 17:18:40,963] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:40,964] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,965] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:18:40,978] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,979] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:40,990] jepsen worker 7 - jepsen.util 637 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:40,990] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:40,991] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:40,992] jepsen worker 42 - jepsen.util 402 :invoke :write [14 19] INFO [2022-03-13 17:18:41,021] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,022] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:41,024] jepsen worker 84 - jepsen.util 4584 :info :write [28 26] :RequestTimeoutException INFO [2022-03-13 17:18:41,025] jepsen worker 84 - jepsen.util 4674 :invoke :write [28 40] INFO [2022-03-13 17:18:41,035] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,036] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:41,038] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,039] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,064] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,064] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,064] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:41,065] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,066] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,066] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,066] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:41,067] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,068] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,068] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:41,068] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,069] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,069] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,070] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:41,070] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,071] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,071] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,071] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:18:41,072] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,072] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:18:41,079] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,079] jepsen worker 88 - jepsen.util 2428 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,079] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,080] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 39] INFO [2022-03-13 17:18:41,081] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:18:41,081] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 40] INFO [2022-03-13 17:18:41,089] jepsen worker 54 - jepsen.util 4104 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,090] jepsen worker 54 - jepsen.util 4104 :invoke :write [18 39] INFO [2022-03-13 17:18:41,093] jepsen worker 1 - jepsen.util 3871 :info :write [0 6] :RequestTimeoutException INFO [2022-03-13 17:18:41,094] jepsen worker 1 - jepsen.util 3961 :invoke :read [0 nil] INFO [2022-03-13 17:18:41,123] jepsen worker 4 - jepsen.util 2614 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,124] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:18:41,124] jepsen worker 1 - jepsen.util 3961 :ok :read [0 41] INFO [2022-03-13 17:18:41,125] jepsen worker 1 - jepsen.util 3961 :invoke :write [0 30] INFO [2022-03-13 17:18:41,133] jepsen worker 26 - jepsen.util 566 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,134] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,133] jepsen worker 76 - jepsen.util 796 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,135] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:18:41,135] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:41,136] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,136] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,137] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:41,138] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:41,138] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 1] INFO [2022-03-13 17:18:41,149] jepsen worker 69 - jepsen.util 789 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,150] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:18:41,167] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,169] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:18:41,196] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,196] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,212] jepsen worker 51 - jepsen.util 771 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,212] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:41,219] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,219] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,220] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,220] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:41,275] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,275] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,276] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:41,276] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,276] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,277] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:18:41,277] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,278] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,278] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:41,278] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:18:41,282] jepsen worker 45 - jepsen.util 4635 :ok :read [15 48] INFO [2022-03-13 17:18:41,282] jepsen worker 45 - jepsen.util 4635 :invoke :write [15 7] INFO [2022-03-13 17:18:41,287] jepsen worker 59 - jepsen.util 4379 :ok :read [19 6] INFO [2022-03-13 17:18:41,288] jepsen worker 59 - jepsen.util 4379 :invoke :read [19 nil] INFO [2022-03-13 17:18:41,289] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,290] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 11] INFO [2022-03-13 17:18:41,307] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,307] jepsen worker 37 - jepsen.util 2467 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,308] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:18:41,309] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 42] INFO [2022-03-13 17:18:41,335] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,337] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:18:41,339] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,339] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,339] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,340] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 8] INFO [2022-03-13 17:18:41,347] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,348] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:18:41,352] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,352] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:18:41,358] jepsen worker 47 - jepsen.util 4277 :ok :read [15 48] INFO [2022-03-13 17:18:41,362] jepsen worker 47 - jepsen.util 4277 :invoke :write [15 13] INFO [2022-03-13 17:18:41,368] jepsen worker 81 - jepsen.util 4941 :ok :read [27 4] INFO [2022-03-13 17:18:41,369] jepsen worker 81 - jepsen.util 4941 :invoke :read [27 nil] INFO [2022-03-13 17:18:41,392] jepsen worker 88 - jepsen.util 2428 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,393] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 23] INFO [2022-03-13 17:18:41,404] jepsen worker 59 - jepsen.util 4379 :ok :read [19 6] INFO [2022-03-13 17:18:41,405] jepsen worker 59 - jepsen.util 4379 :invoke :write [19 16] INFO [2022-03-13 17:18:41,421] jepsen worker 81 - jepsen.util 4941 :ok :read [27 4] INFO [2022-03-13 17:18:41,421] jepsen worker 81 - jepsen.util 4941 :invoke :write [27 19] INFO [2022-03-13 17:18:41,439] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,440] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:41,454] jepsen worker 89 - jepsen.util 2789 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,455] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 9] INFO [2022-03-13 17:18:41,500] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,501] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:18:41,510] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,511] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,532] jepsen worker 43 - jepsen.util 673 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,532] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,532] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,532] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,532] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,532] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,533] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:18:41,534] jepsen worker 52 - jepsen.util 682 :invoke :write [17 33] INFO [2022-03-13 17:18:41,535] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:41,536] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:41,536] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,537] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,537] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:41,538] jepsen worker 8 - jepsen.util 728 :invoke :write [2 45] INFO [2022-03-13 17:18:41,538] jepsen worker 44 - jepsen.util 764 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,538] jepsen worker 77 - jepsen.util 707 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,539] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,539] jepsen worker 44 - jepsen.util 764 :invoke :write [14 16] INFO [2022-03-13 17:18:41,540] jepsen worker 6 - jepsen.util 636 :invoke :write [2 38] INFO [2022-03-13 17:18:41,540] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,541] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:41,541] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:41,542] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:41,543] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,544] jepsen worker 12 - jepsen.util 912 :invoke :write [4 23] INFO [2022-03-13 17:18:41,544] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:41,562] jepsen worker 37 - jepsen.util 2467 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,563] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,585] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,585] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,585] jepsen worker 4 - jepsen.util 2614 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,585] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:41,586] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:18:41,586] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:41,589] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,589] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,590] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:18:41,591] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:18:41,616] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,617] jepsen worker 89 - jepsen.util 2789 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,617] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:18:41,618] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:41,626] jepsen worker 30 - jepsen.util 4260 :info :write [10 27] :RequestTimeoutException INFO [2022-03-13 17:18:41,627] jepsen worker 30 - jepsen.util 4350 :invoke :read [10 nil] INFO [2022-03-13 17:18:41,652] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,653] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:41,657] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:41,657] jepsen worker nemesis - couchbase.util Rebalance check count: 628 INFO [2022-03-13 17:18:41,658] jepsen worker nemesis - couchbase.util Rebalance stuck count: 500 INFO [2022-03-13 17:18:41,659] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,660] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:18:41,685] jepsen worker 80 - jepsen.util 800 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,685] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:41,708] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,708] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,709] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:18:41,710] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:18:41,712] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,713] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,739] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,741] jepsen worker 69 - jepsen.util 789 :invoke :write [23 14] INFO [2022-03-13 17:18:41,756] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,756] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,756] jepsen worker 75 - jepsen.util 525 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,756] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,757] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,757] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,757] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:41,757] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:41,758] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:41,759] jepsen worker 43 - jepsen.util 673 :invoke :write [14 26] INFO [2022-03-13 17:18:41,760] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:41,760] jepsen worker 60 - jepsen.util 690 :invoke :write [20 5] INFO [2022-03-13 17:18:41,761] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:41,763] jepsen worker 76 - jepsen.util 796 :invoke :write [25 25] INFO [2022-03-13 17:18:41,767] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,767] jepsen worker 88 - jepsen.util 2428 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,768] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,769] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 48] INFO [2022-03-13 17:18:41,797] jepsen worker 31 - jepsen.util 4351 :info :write [10 14] :RequestTimeoutException INFO [2022-03-13 17:18:41,797] jepsen worker 31 - jepsen.util 4441 :invoke :read [10 nil] INFO [2022-03-13 17:18:41,799] jepsen worker 30 - jepsen.util 4350 :ok :read [10 :nil] INFO [2022-03-13 17:18:41,800] jepsen worker 30 - jepsen.util 4350 :invoke :write [10 24] INFO [2022-03-13 17:18:41,804] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,805] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:18:41,819] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,821] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:18:41,821] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,822] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:18:41,835] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,835] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:41,887] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,887] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,887] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:41,888] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:18:41,897] jepsen worker 46 - jepsen.util 4366 :info :write [15 20] :RequestTimeoutException INFO [2022-03-13 17:18:41,898] jepsen worker 46 - jepsen.util 4456 :invoke :read [15 nil] INFO [2022-03-13 17:18:41,940] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,941] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:18:41,941] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,942] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:41,952] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,952] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,953] jepsen worker 62 - jepsen.util 512 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,952] jepsen worker 23 - jepsen.util 563 :fail :write [7 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,954] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:41,955] jepsen worker 62 - jepsen.util 512 :invoke :write [20 41] INFO [2022-03-13 17:18:41,955] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:41,956] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:41,991] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,991] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:41,991] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:41,992] jepsen worker 75 - jepsen.util 525 :invoke :write [25 27] INFO [2022-03-13 17:18:41,996] jepsen worker 31 - jepsen.util 4441 :ok :read [10 :nil] INFO [2022-03-13 17:18:41,997] jepsen worker 31 - jepsen.util 4441 :invoke :read [10 nil] INFO [2022-03-13 17:18:42,116] jepsen worker 31 - jepsen.util 4441 :ok :read [10 :nil] INFO [2022-03-13 17:18:42,117] jepsen worker 31 - jepsen.util 4441 :invoke :write [10 29] INFO [2022-03-13 17:18:42,159] jepsen worker 25 - jepsen.util 565 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,160] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:42,163] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,163] jepsen worker 69 - jepsen.util 789 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,164] jepsen worker 52 - jepsen.util 682 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,164] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,164] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,164] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:18:42,164] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,165] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:18:42,166] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:42,166] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:42,167] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,167] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:42,167] jepsen worker 26 - jepsen.util 566 :invoke :write [8 19] INFO [2022-03-13 17:18:42,168] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:18:42,197] jepsen worker 74 - jepsen.util 4124 :info :write [24 1] :RequestTimeoutException INFO [2022-03-13 17:18:42,198] jepsen worker 74 - jepsen.util 4214 :invoke :write [24 44] INFO [2022-03-13 17:18:42,253] jepsen worker 42 - jepsen.util 402 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,254] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:18:42,254] jepsen worker 24 - jepsen.util 654 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,255] jepsen worker 24 - jepsen.util 654 :invoke :write [8 31] INFO [2022-03-13 17:18:42,255] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,255] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,255] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,255] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,256] jepsen worker 80 - jepsen.util 800 :invoke :write [26 37] INFO [2022-03-13 17:18:42,256] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:42,257] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:42,259] jepsen worker 78 - jepsen.util 708 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,259] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,260] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:18:42,260] jepsen worker 78 - jepsen.util 708 :invoke :write [26 0] INFO [2022-03-13 17:18:42,265] jepsen worker 51 - jepsen.util 771 :invoke :write [17 13] INFO [2022-03-13 17:18:42,271] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,272] jepsen worker 71 - jepsen.util 611 :invoke :write [23 4] INFO [2022-03-13 17:18:42,328] jepsen worker 33 - jepsen.util 753 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,328] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:42,332] jepsen worker 60 - jepsen.util 690 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,332] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,332] jepsen worker 76 - jepsen.util 796 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,332] jepsen worker 60 - jepsen.util 690 :invoke :write [20 35] INFO [2022-03-13 17:18:42,333] jepsen worker 76 - jepsen.util 796 :invoke :write [25 17] INFO [2022-03-13 17:18:42,333] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:18:42,372] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,373] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:42,376] jepsen worker 8 - jepsen.util 728 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,376] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:42,431] jepsen worker 71 - jepsen.util 611 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,431] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,432] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:42,433] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:42,575] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,576] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:18:42,579] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,579] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,580] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,580] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,580] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,580] jepsen worker 79 - jepsen.util 1159 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,580] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:42,581] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:42,582] jepsen worker 61 - jepsen.util 781 :invoke :write [20 2] INFO [2022-03-13 17:18:42,584] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:18:42,584] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:18:42,585] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,586] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:42,587] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,587] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:42,587] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:18:42,589] jepsen worker 43 - jepsen.util 673 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,589] jepsen worker 75 - jepsen.util 525 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,589] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,589] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,590] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:42,591] jepsen worker 75 - jepsen.util 525 :invoke :write [25 26] INFO [2022-03-13 17:18:42,591] jepsen worker 52 - jepsen.util 682 :invoke :write [17 44] INFO [2022-03-13 17:18:42,595] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:42,662] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:42,663] jepsen worker nemesis - couchbase.util Rebalance check count: 629 INFO [2022-03-13 17:18:42,663] jepsen worker nemesis - couchbase.util Rebalance stuck count: 501 INFO [2022-03-13 17:18:42,763] jepsen worker 41 - jepsen.util 4361 :info :write [13 24] :RequestTimeoutException INFO [2022-03-13 17:18:42,764] jepsen worker 41 - jepsen.util 4451 :invoke :write [13 7] INFO [2022-03-13 17:18:42,780] jepsen worker 76 - jepsen.util 796 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,780] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,781] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:42,782] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:42,787] jepsen worker 34 - jepsen.util 934 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,787] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,787] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,787] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,788] jepsen worker 34 - jepsen.util 934 :invoke :write [11 44] INFO [2022-03-13 17:18:42,788] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:18:42,789] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:42,790] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:18:42,976] jepsen worker 17 - jepsen.util 4607 :ok :read [5 43] INFO [2022-03-13 17:18:42,977] jepsen worker 17 - jepsen.util 4607 :invoke :write [5 9] INFO [2022-03-13 17:18:42,984] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:42,984] jepsen worker 71 - jepsen.util 611 :invoke :write [23 41] INFO [2022-03-13 17:18:43,198] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,199] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:43,203] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,203] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,204] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:18:43,204] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,204] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,205] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,209] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,210] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:18:43,225] jepsen worker 65 - jepsen.util 4565 :info :write [21 24] :SyncWriteAmbiguous INFO [2022-03-13 17:18:43,227] jepsen worker 65 - jepsen.util 4655 :invoke :write [21 32] INFO [2022-03-13 17:18:43,267] jepsen worker 88 - jepsen.util 2428 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,267] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 40] INFO [2022-03-13 17:18:43,278] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,279] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,281] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,282] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:43,285] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,285] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,292] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,293] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,294] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,294] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,295] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 10] INFO [2022-03-13 17:18:43,298] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,346] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,346] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:43,386] jepsen worker 63 - jepsen.util 4473 :ok :read [21 :nil] INFO [2022-03-13 17:18:43,386] jepsen worker 63 - jepsen.util 4473 :invoke :read [21 nil] INFO [2022-03-13 17:18:43,394] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,395] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:18:43,402] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,402] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,402] jepsen worker 89 - jepsen.util 2789 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,402] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,403] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:18:43,407] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:18:43,407] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 8] INFO [2022-03-13 17:18:43,407] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:18:43,458] jepsen worker 29 - jepsen.util 4889 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,459] jepsen worker 46 - jepsen.util 4456 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,459] jepsen worker 29 - jepsen.util 4889 :invoke :read [9 nil] INFO [2022-03-13 17:18:43,460] jepsen worker 46 - jepsen.util 4456 :invoke :read [15 nil] INFO [2022-03-13 17:18:43,461] jepsen worker 63 - jepsen.util 4473 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,461] jepsen worker 63 - jepsen.util 4473 :invoke :write [21 45] INFO [2022-03-13 17:18:43,469] jepsen worker 9 - jepsen.util 3969 :ok :read [3 25] INFO [2022-03-13 17:18:43,469] jepsen worker 9 - jepsen.util 3969 :invoke :write [3 6] INFO [2022-03-13 17:18:43,476] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,476] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:43,477] jepsen worker 29 - jepsen.util 4889 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,477] jepsen worker 46 - jepsen.util 4456 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,477] jepsen worker 29 - jepsen.util 4889 :invoke :write [9 1] INFO [2022-03-13 17:18:43,478] jepsen worker 46 - jepsen.util 4456 :invoke :read [15 nil] INFO [2022-03-13 17:18:43,480] jepsen worker 46 - jepsen.util 4456 :fail :read [15 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,481] jepsen worker 46 - jepsen.util 4456 :invoke :write [15 35] INFO [2022-03-13 17:18:43,503] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,504] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:18:43,523] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,523] jepsen worker 49 - jepsen.util 3559 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:18:43,523] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,524] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 35] INFO [2022-03-13 17:18:43,542] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,543] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,548] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,549] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:18:43,591] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,591] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:18:43,601] jepsen worker 89 - jepsen.util 2789 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,601] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,632] jepsen worker 88 - jepsen.util 2428 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,632] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,633] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,633] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:43,663] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,663] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,664] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:18:43,664] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:18:43,668] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:43,668] jepsen worker nemesis - couchbase.util Rebalance check count: 630 INFO [2022-03-13 17:18:43,668] jepsen worker nemesis - couchbase.util Rebalance stuck count: 502 INFO [2022-03-13 17:18:43,670] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,671] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:43,699] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,700] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:18:43,703] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,704] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,704] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,704] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 14] INFO [2022-03-13 17:18:43,705] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,706] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,707] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,708] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,709] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,710] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:43,755] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,755] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:18:43,776] jepsen worker 49 - jepsen.util 3559 :fail :write [16 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,776] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,776] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,776] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 38] INFO [2022-03-13 17:18:43,777] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:18:43,778] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:18:43,783] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,784] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,784] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,785] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 41] INFO [2022-03-13 17:18:43,790] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,791] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,792] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,792] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,794] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,795] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:43,796] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,797] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:18:43,826] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,826] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:18:43,893] jepsen worker 49 - jepsen.util 3559 :fail :write [16 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,893] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,894] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:18:43,894] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 1] INFO [2022-03-13 17:18:43,958] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,958] jepsen worker 37 - jepsen.util 2467 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,958] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,959] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:18:43,963] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:43,963] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:43,984] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:43,985] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,014] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,014] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:44,015] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,016] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:18:44,031] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,032] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:18:44,059] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:18:44,059] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 35] INFO [2022-03-13 17:18:44,100] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,101] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:44,114] jepsen worker 88 - jepsen.util 2428 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,114] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,115] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:44,115] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 45] INFO [2022-03-13 17:18:44,116] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,116] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 19] INFO [2022-03-13 17:18:44,122] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,123] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,125] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,125] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:44,180] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,181] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:18:44,191] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,192] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:44,197] jepsen worker 49 - jepsen.util 3559 :fail :write [16 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,197] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,197] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,198] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:18:44,199] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:18:44,199] jepsen worker 49 - jepsen.util 3559 :invoke :read [16 nil] INFO [2022-03-13 17:18:44,229] jepsen worker 71 - jepsen.util 611 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,229] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,230] jepsen worker 71 - jepsen.util 611 :invoke :write [23 48] INFO [2022-03-13 17:18:44,231] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:18:44,239] jepsen worker 80 - jepsen.util 800 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,240] jepsen worker 80 - jepsen.util 800 :invoke :write [26 33] INFO [2022-03-13 17:18:44,293] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,294] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:18:44,326] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,326] jepsen worker 48 - jepsen.util 3558 :fail :write [16 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,326] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,327] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,327] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 23] INFO [2022-03-13 17:18:44,328] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:18:44,328] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,329] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:18:44,330] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,335] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,336] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,337] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,337] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,338] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,338] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:18:44,339] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,339] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,340] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 24] INFO [2022-03-13 17:18:44,371] jepsen worker 72 - jepsen.util 4392 :info :write [24 47] :RequestTimeoutException INFO [2022-03-13 17:18:44,371] jepsen worker 6 - jepsen.util 636 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,372] jepsen worker 72 - jepsen.util 4482 :invoke :write [24 26] INFO [2022-03-13 17:18:44,372] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,372] jepsen worker 78 - jepsen.util 708 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,373] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,373] jepsen worker 6 - jepsen.util 636 :invoke :write [2 45] INFO [2022-03-13 17:18:44,373] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:44,374] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:44,374] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:44,392] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,392] jepsen worker 13 - jepsen.util 733 :invoke :write [4 2] INFO [2022-03-13 17:18:44,393] jepsen worker 71 - jepsen.util 611 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,394] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:18:44,401] jepsen worker 88 - jepsen.util 2428 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,405] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 38] INFO [2022-03-13 17:18:44,421] jepsen worker 48 - jepsen.util 3558 :fail :write [16 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,422] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 9] INFO [2022-03-13 17:18:44,449] jepsen worker 68 - jepsen.util 4478 :ok :read [22 30] INFO [2022-03-13 17:18:44,449] jepsen worker 68 - jepsen.util 4478 :invoke :write [22 20] INFO [2022-03-13 17:18:44,460] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,461] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:18:44,476] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,476] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:44,479] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,480] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:18:44,533] jepsen worker 37 - jepsen.util 2467 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,534] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 0] INFO [2022-03-13 17:18:44,542] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,543] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:18:44,583] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,584] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:44,587] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,588] jepsen worker 24 - jepsen.util 654 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,588] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:44,589] jepsen worker 33 - jepsen.util 753 :invoke :write [11 26] INFO [2022-03-13 17:18:44,593] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,593] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:18:44,640] jepsen worker 48 - jepsen.util 3558 :fail :write [16 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,641] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 6] INFO [2022-03-13 17:18:44,657] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,658] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,660] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,661] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 8] INFO [2022-03-13 17:18:44,674] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:44,674] jepsen worker nemesis - couchbase.util Rebalance check count: 631 INFO [2022-03-13 17:18:44,674] jepsen worker nemesis - couchbase.util Rebalance stuck count: 503 INFO [2022-03-13 17:18:44,703] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,704] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:18:44,706] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,707] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,716] jepsen worker 88 - jepsen.util 2428 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,717] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 1] INFO [2022-03-13 17:18:44,720] jepsen worker 32 - jepsen.util 4532 :info :write [10 20] :RequestTimeoutException INFO [2022-03-13 17:18:44,721] jepsen worker 32 - jepsen.util 4622 :invoke :write [10 16] INFO [2022-03-13 17:18:44,731] jepsen worker 64 - jepsen.util 4204 :info :write [21 16] :RequestTimeoutException INFO [2022-03-13 17:18:44,731] jepsen worker 64 - jepsen.util 4294 :invoke :read [21 nil] INFO [2022-03-13 17:18:44,780] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,781] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:44,790] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,790] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,790] jepsen worker 89 - jepsen.util 2789 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,791] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:44,791] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,792] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:18:44,794] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:44,795] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,795] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:18:44,795] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:18:44,801] jepsen worker 36 - jepsen.util 2466 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,802] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:18:44,808] jepsen worker 67 - jepsen.util 3577 :info :write [22 8] :RequestTimeoutException INFO [2022-03-13 17:18:44,808] jepsen worker 67 - jepsen.util 3667 :invoke :read [22 nil] INFO [2022-03-13 17:18:44,810] jepsen worker 26 - jepsen.util 566 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,810] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:44,840] jepsen worker 44 - jepsen.util 764 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,840] jepsen worker 61 - jepsen.util 781 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,841] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:18:44,841] jepsen worker 61 - jepsen.util 781 :invoke :write [20 20] INFO [2022-03-13 17:18:44,850] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,851] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:44,852] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,853] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:44,857] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,857] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,857] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:44,858] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:44,858] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,859] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:18:44,867] jepsen worker 48 - jepsen.util 3558 :fail :write [16 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,867] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 9] INFO [2022-03-13 17:18:44,883] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,883] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:44,927] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:44,927] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:18:44,931] jepsen worker 64 - jepsen.util 4294 :ok :read [21 :nil] INFO [2022-03-13 17:18:44,958] jepsen worker 37 - jepsen.util 2467 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,041] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 1] INFO [2022-03-13 17:18:45,053] jepsen worker 40 - jepsen.util 4540 :info :write [13 13] :SyncWriteAmbiguous INFO [2022-03-13 17:18:44,959] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,000] jepsen worker 56 - jepsen.util 4196 :info :write [18 21] :RequestTimeoutException INFO [2022-03-13 17:18:45,053] jepsen worker 56 - jepsen.util 4286 :invoke :write [18 17] INFO [2022-03-13 17:18:45,009] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,030] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,054] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:18:45,054] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:45,041] jepsen worker 64 - jepsen.util 4294 :invoke :read [21 nil] INFO [2022-03-13 17:18:45,053] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,062] jepsen worker 40 - jepsen.util 4630 :invoke :write [13 29] INFO [2022-03-13 17:18:45,070] jepsen worker 11 - jepsen.util 4151 :info :write [3 41] :RequestTimeoutException INFO [2022-03-13 17:18:45,070] jepsen worker 11 - jepsen.util 4241 :invoke :write [3 33] INFO [2022-03-13 17:18:45,093] jepsen worker 89 - jepsen.util 2789 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,094] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:45,095] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,095] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,095] jepsen worker 48 - jepsen.util 3558 :fail :write [16 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,096] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,097] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:18:45,098] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,106] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,107] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:18:45,107] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,108] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 18] INFO [2022-03-13 17:18:45,143] jepsen worker 88 - jepsen.util 2428 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,143] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,143] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,144] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:18:45,144] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:45,145] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:18:45,147] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,148] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:18:45,177] jepsen worker 66 - jepsen.util 4116 :info :write [22 42] :RequestTimeoutException INFO [2022-03-13 17:18:45,178] jepsen worker 66 - jepsen.util 4206 :invoke :read [22 nil] INFO [2022-03-13 17:18:45,206] jepsen worker 37 - jepsen.util 2467 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,207] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 11] INFO [2022-03-13 17:18:45,213] jepsen worker 51 - jepsen.util 771 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,214] jepsen worker 51 - jepsen.util 771 :invoke :write [17 45] INFO [2022-03-13 17:18:45,219] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,220] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:18:45,222] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,223] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:18:45,256] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,257] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,257] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:18:45,258] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:45,284] jepsen worker 38 - jepsen.util 2828 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,285] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 5] INFO [2022-03-13 17:18:45,304] jepsen worker 28 - jepsen.util 4348 :info :write [9 12] :RequestTimeoutException INFO [2022-03-13 17:18:45,304] jepsen worker 28 - jepsen.util 4438 :invoke :write [9 18] INFO [2022-03-13 17:18:45,334] jepsen worker 64 - jepsen.util 4294 :ok :read [21 :nil] INFO [2022-03-13 17:18:45,335] jepsen worker 64 - jepsen.util 4294 :invoke :read [21 nil] INFO [2022-03-13 17:18:45,345] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,345] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,345] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 2] INFO [2022-03-13 17:18:45,346] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,396] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,397] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:18:45,439] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,439] jepsen worker 75 - jepsen.util 525 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,439] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:45,439] jepsen worker 60 - jepsen.util 690 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,439] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,441] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:45,442] jepsen worker 43 - jepsen.util 673 :invoke :write [14 16] INFO [2022-03-13 17:18:45,442] jepsen worker 76 - jepsen.util 796 :invoke :write [25 9] INFO [2022-03-13 17:18:45,443] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:45,443] jepsen worker 60 - jepsen.util 690 :invoke :write [20 6] INFO [2022-03-13 17:18:45,446] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,447] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:18:45,450] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,451] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,465] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,465] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,465] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,467] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,467] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 30] INFO [2022-03-13 17:18:45,468] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,468] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:45,469] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:18:45,476] jepsen worker 57 - jepsen.util 4377 :info :write [19 39] :RequestTimeoutException INFO [2022-03-13 17:18:45,477] jepsen worker 57 - jepsen.util 4467 :invoke :write [19 7] INFO [2022-03-13 17:18:45,479] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,479] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:45,481] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,482] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:18:45,492] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,493] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 28] INFO [2022-03-13 17:18:45,497] jepsen worker 3 - jepsen.util 2253 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,499] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:18:45,511] jepsen worker 38 - jepsen.util 2828 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,513] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,545] jepsen worker 64 - jepsen.util 4294 :ok :read [21 :nil] INFO [2022-03-13 17:18:45,546] jepsen worker 64 - jepsen.util 4294 :invoke :write [21 24] INFO [2022-03-13 17:18:45,557] jepsen worker 2 - jepsen.util 4592 :info :write [0 4] :RequestTimeoutException INFO [2022-03-13 17:18:45,557] jepsen worker 2 - jepsen.util 4682 :invoke :write [0 5] INFO [2022-03-13 17:18:45,567] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,567] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,579] jepsen worker 37 - jepsen.util 2467 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,579] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,619] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,619] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,619] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:18:45,620] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:45,621] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,622] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:18:45,627] jepsen worker 34 - jepsen.util 934 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,628] jepsen worker 34 - jepsen.util 934 :invoke :write [11 27] INFO [2022-03-13 17:18:45,632] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,632] jepsen worker 12 - jepsen.util 912 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,632] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,632] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,632] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,632] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:45,633] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:45,634] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:45,634] jepsen worker 77 - jepsen.util 707 :invoke :write [25 3] INFO [2022-03-13 17:18:45,635] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:45,645] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,645] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:18:45,680] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:45,680] jepsen worker nemesis - couchbase.util Rebalance check count: 632 INFO [2022-03-13 17:18:45,680] jepsen worker nemesis - couchbase.util Rebalance stuck count: 504 INFO [2022-03-13 17:18:45,823] jepsen worker 36 - jepsen.util 2466 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,823] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,824] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,824] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,825] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,826] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:18:45,833] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,833] jepsen worker 52 - jepsen.util 682 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,833] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,834] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,834] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:18:45,834] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:18:45,835] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,836] jepsen worker 13 - jepsen.util 733 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,837] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,837] jepsen worker 13 - jepsen.util 733 :invoke :write [4 25] INFO [2022-03-13 17:18:45,837] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,838] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,838] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,838] jepsen worker 88 - jepsen.util 2428 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,839] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,840] jepsen worker 26 - jepsen.util 566 :invoke :write [8 28] INFO [2022-03-13 17:18:45,842] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,842] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:45,845] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,846] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 8] INFO [2022-03-13 17:18:45,847] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,848] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:45,848] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,849] jepsen worker 75 - jepsen.util 525 :invoke :write [25 39] INFO [2022-03-13 17:18:45,849] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,850] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,850] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,851] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,851] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:18:45,852] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 36] INFO [2022-03-13 17:18:45,856] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,856] jepsen worker 14 - jepsen.util 374 :invoke :write [4 39] INFO [2022-03-13 17:18:45,863] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,863] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,863] jepsen worker 62 - jepsen.util 512 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,867] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:45,867] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:45,868] jepsen worker 62 - jepsen.util 512 :invoke :write [20 43] INFO [2022-03-13 17:18:45,888] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,889] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,889] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,889] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:45,891] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 33] INFO [2022-03-13 17:18:45,891] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,891] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:18:45,894] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 35] INFO [2022-03-13 17:18:45,903] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,904] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,935] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,935] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,936] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 27] INFO [2022-03-13 17:18:45,936] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,938] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,938] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,961] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,961] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:45,966] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:45,967] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:18:45,990] jepsen worker 73 - jepsen.util 4573 :info :write [24 7] :RequestTimeoutException INFO [2022-03-13 17:18:45,991] jepsen worker 73 - jepsen.util 4663 :invoke :write [24 30] INFO [2022-03-13 17:18:46,016] jepsen worker 4 - jepsen.util 2614 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,017] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:18:46,041] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,041] jepsen worker 89 - jepsen.util 2789 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,042] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:18:46,043] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:18:46,045] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,046] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:18:46,055] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,056] jepsen worker 61 - jepsen.util 781 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,056] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:18:46,057] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,057] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,057] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,058] jepsen worker 61 - jepsen.util 781 :invoke :write [20 14] INFO [2022-03-13 17:18:46,058] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,059] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:46,060] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:18:46,061] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:46,061] jepsen worker 78 - jepsen.util 708 :invoke :write [26 38] INFO [2022-03-13 17:18:46,122] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,123] jepsen worker 77 - jepsen.util 707 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,123] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,123] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:46,123] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,123] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,123] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,124] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:18:46,124] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:46,125] jepsen worker 77 - jepsen.util 707 :invoke :write [25 22] INFO [2022-03-13 17:18:46,126] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 7] INFO [2022-03-13 17:18:46,126] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:18:46,126] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,127] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:18:46,176] jepsen worker 88 - jepsen.util 2428 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,176] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,194] jepsen worker 3 - jepsen.util 2253 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,194] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,208] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,208] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,251] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:18:46,251] jepsen worker 6 - jepsen.util 636 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,252] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,252] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,253] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:46,253] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,254] jepsen worker 44 - jepsen.util 764 :invoke :write [14 6] INFO [2022-03-13 17:18:46,254] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:46,255] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:46,256] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:46,262] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,263] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:46,307] jepsen worker 43 - jepsen.util 673 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,308] jepsen worker 76 - jepsen.util 796 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,308] jepsen worker 43 - jepsen.util 673 :invoke :write [14 6] INFO [2022-03-13 17:18:46,309] jepsen worker 76 - jepsen.util 796 :invoke :write [25 45] INFO [2022-03-13 17:18:46,319] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,320] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:18:46,326] jepsen worker 61 - jepsen.util 781 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,327] jepsen worker 61 - jepsen.util 781 :invoke :write [20 1] INFO [2022-03-13 17:18:46,365] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,365] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,365] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,366] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:18:46,367] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:18:46,367] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,377] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,377] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,377] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:46,378] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,430] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,430] jepsen worker 37 - jepsen.util 2467 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,430] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,431] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,431] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,431] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:18:46,431] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,431] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,432] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:18:46,432] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,460] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,460] jepsen worker 51 - jepsen.util 771 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,462] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:46,463] jepsen worker 51 - jepsen.util 771 :invoke :write [17 33] INFO [2022-03-13 17:18:46,477] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,477] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,478] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:18:46,480] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:18:46,487] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,488] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,488] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 14] INFO [2022-03-13 17:18:46,488] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:18:46,496] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,497] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 40] INFO [2022-03-13 17:18:46,504] jepsen worker 67 - jepsen.util 3667 :ok :read [22 30] INFO [2022-03-13 17:18:46,505] jepsen worker 67 - jepsen.util 3667 :invoke :read [22 nil] INFO [2022-03-13 17:18:46,560] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,561] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:46,563] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,563] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,564] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,564] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 17] INFO [2022-03-13 17:18:46,565] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:18:46,565] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,567] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,567] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,567] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,568] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,569] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,569] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,576] jepsen worker 67 - jepsen.util 3667 :ok :read [22 30] INFO [2022-03-13 17:18:46,576] jepsen worker 67 - jepsen.util 3667 :invoke :read [22 nil] INFO [2022-03-13 17:18:46,591] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,592] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,595] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,595] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,607] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,607] jepsen worker 38 - jepsen.util 2828 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,608] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:18:46,610] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,611] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,612] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,613] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,614] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:18:46,630] jepsen worker 82 - jepsen.util 4582 :info :write [27 39] :RequestTimeoutException INFO [2022-03-13 17:18:46,631] jepsen worker 82 - jepsen.util 4672 :invoke :read [27 nil] INFO [2022-03-13 17:18:46,653] jepsen worker 67 - jepsen.util 3667 :ok :read [22 30] INFO [2022-03-13 17:18:46,653] jepsen worker 67 - jepsen.util 3667 :invoke :write [22 24] INFO [2022-03-13 17:18:46,659] jepsen worker 13 - jepsen.util 733 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,659] jepsen worker 60 - jepsen.util 690 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,660] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,660] jepsen worker 13 - jepsen.util 733 :invoke :write [4 35] INFO [2022-03-13 17:18:46,660] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,661] jepsen worker 60 - jepsen.util 690 :invoke :write [20 18] INFO [2022-03-13 17:18:46,661] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:46,662] jepsen worker 52 - jepsen.util 682 :invoke :write [17 22] INFO [2022-03-13 17:18:46,666] jepsen worker 62 - jepsen.util 512 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,667] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:18:46,680] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,680] jepsen worker 4 - jepsen.util 2614 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,681] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:46,682] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:18:46,687] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:46,687] jepsen worker nemesis - couchbase.util Rebalance check count: 633 INFO [2022-03-13 17:18:46,687] jepsen worker nemesis - couchbase.util Rebalance stuck count: 505 INFO [2022-03-13 17:18:46,694] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,694] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,696] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,696] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,698] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,698] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,700] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,700] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:18:46,739] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,740] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:46,743] jepsen worker 78 - jepsen.util 708 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,744] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,744] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:46,745] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:46,752] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,753] jepsen worker 71 - jepsen.util 611 :invoke :write [23 14] INFO [2022-03-13 17:18:46,763] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,763] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:18:46,768] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,769] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,775] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,775] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,776] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:18:46,776] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,844] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,844] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,848] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,848] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,849] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,849] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:46,851] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,851] jepsen worker 88 - jepsen.util 2428 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,851] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,852] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,853] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,854] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:18:46,854] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,854] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,855] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,855] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,856] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,856] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:18:46,857] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,858] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 24] INFO [2022-03-13 17:18:46,933] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,933] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,933] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,934] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:18:46,934] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:18:46,935] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:18:46,947] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,947] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,948] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:46,948] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:46,949] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,949] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:46,950] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:18:46,950] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:18:46,988] jepsen worker 58 - jepsen.util 4018 :info :write [19 18] :RequestTimeoutException INFO [2022-03-13 17:18:46,989] jepsen worker 58 - jepsen.util 4108 :invoke :write [19 45] INFO [2022-03-13 17:18:46,990] jepsen worker 86 - jepsen.util 4226 :info :write [28 46] :RequestTimeoutException INFO [2022-03-13 17:18:46,991] jepsen worker 86 - jepsen.util 4316 :invoke :read [28 nil] WARN [2022-03-13 17:18:46,999] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9077161,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18672765,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9207928,"encode_duration_us":21587,"operation_id":"0x11a81","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10005602},{"operation_name":"upsert","last_dispatch_duration_us":9226525,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15760776,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9074546,"encode_duration_us":17153,"operation_id":"0x11fa0","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10006289}],"total_count":2}} INFO [2022-03-13 17:18:47,004] jepsen worker 15 - jepsen.util 4245 :info :write [5 3] :SyncWriteAmbiguous INFO [2022-03-13 17:18:47,005] jepsen worker 15 - jepsen.util 4335 :invoke :write [5 19] INFO [2022-03-13 17:18:47,026] jepsen worker 86 - jepsen.util 4316 :ok :read [28 44] INFO [2022-03-13 17:18:47,026] jepsen worker 86 - jepsen.util 4316 :invoke :write [28 6] INFO [2022-03-13 17:18:47,092] jepsen worker 80 - jepsen.util 800 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,092] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:47,095] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,096] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] WARN [2022-03-13 17:18:47,099] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7997094,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8882255,"total_server_duration_us":258214,"encode_duration_us":16958,"operation_id":"0x11f98","timeout_ms":10000,"last_server_duration_us":26418,"total_duration_us":10010496},{"operation_name":"upsert","last_dispatch_duration_us":70218,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9305703,"total_server_duration_us":410841,"encode_duration_us":31482,"operation_id":"0x123b8","timeout_ms":10000,"last_server_duration_us":41445,"total_duration_us":10009333},{"operation_name":"upsert","last_dispatch_duration_us":415781,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1341336,"total_server_duration_us":163086,"encode_duration_us":22100,"operation_id":"0x12345","timeout_ms":10000,"last_server_duration_us":33524,"total_duration_us":10008947},{"operation_name":"upsert","last_dispatch_duration_us":9226525,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":15760776,"total_server_duration_us":9074546,"encode_duration_us":17153,"operation_id":"0x11fa0","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10006289},{"operation_name":"upsert","last_dispatch_duration_us":7901552,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8759828,"total_server_duration_us":249587,"encode_duration_us":17502,"operation_id":"0x11f9f","timeout_ms":10000,"last_server_duration_us":25450,"total_duration_us":10004868},{"operation_name":"upsert","last_dispatch_duration_us":192365,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1254956,"total_server_duration_us":199352,"encode_duration_us":32850,"operation_id":"0x12332","timeout_ms":10000,"last_server_duration_us":43070,"total_duration_us":10004215},{"operation_name":"upsert","last_dispatch_duration_us":8306523,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8675295,"total_server_duration_us":88039,"encode_duration_us":17633,"operation_id":"0x122c9","timeout_ms":10000,"last_server_duration_us":39952,"total_duration_us":10004094},{"operation_name":"upsert","last_dispatch_duration_us":32054,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2160495,"total_server_duration_us":457816,"encode_duration_us":19804,"operation_id":"0x123ff","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10004055},{"operation_name":"upsert","last_dispatch_duration_us":8497438,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":13253167,"total_server_duration_us":187651,"encode_duration_us":29413,"operation_id":"0x120d2","timeout_ms":10000,"last_server_duration_us":52183,"total_duration_us":10003650},{"operation_name":"upsert","last_dispatch_duration_us":8415843,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16673524,"total_server_duration_us":218977,"encode_duration_us":19117,"operation_id":"0x11f58","timeout_ms":10000,"last_server_duration_us":87654,"total_duration_us":10003641}],"total_count":52}} INFO [2022-03-13 17:18:47,102] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,103] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:18:47,133] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,133] jepsen worker 75 - jepsen.util 525 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,133] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:47,134] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:47,147] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,147] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:18:47,198] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,199] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:47,281] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,281] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:47,287] jepsen worker 77 - jepsen.util 707 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,288] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:47,314] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,314] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:47,360] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,360] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,360] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:47,361] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:47,362] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,363] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:18:47,370] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,371] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:18:47,385] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,385] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,386] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:47,386] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:47,476] jepsen worker 10 - jepsen.util 4600 :info :write [3 5] :RequestTimeoutException INFO [2022-03-13 17:18:47,477] jepsen worker 10 - jepsen.util 4690 :invoke :read [3 nil] INFO [2022-03-13 17:18:47,478] jepsen worker 10 - jepsen.util 4690 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,479] jepsen worker 10 - jepsen.util 4690 :invoke :read [3 nil] INFO [2022-03-13 17:18:47,480] jepsen worker 10 - jepsen.util 4690 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,482] jepsen worker 10 - jepsen.util 4690 :invoke :write [3 24] INFO [2022-03-13 17:18:47,483] jepsen worker 34 - jepsen.util 934 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,484] jepsen worker 71 - jepsen.util 611 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,484] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,485] jepsen worker 34 - jepsen.util 934 :invoke :write [11 5] INFO [2022-03-13 17:18:47,485] jepsen worker 69 - jepsen.util 789 :invoke :write [23 0] INFO [2022-03-13 17:18:47,486] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:47,515] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,516] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:18:47,693] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:47,693] jepsen worker nemesis - couchbase.util Rebalance check count: 634 INFO [2022-03-13 17:18:47,693] jepsen worker nemesis - couchbase.util Rebalance stuck count: 506 INFO [2022-03-13 17:18:47,810] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,811] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:47,856] jepsen worker 26 - jepsen.util 566 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,856] jepsen worker 76 - jepsen.util 796 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,856] jepsen worker 26 - jepsen.util 566 :invoke :write [8 48] INFO [2022-03-13 17:18:47,857] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:47,882] jepsen worker 69 - jepsen.util 789 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,883] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:47,884] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,885] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,885] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:47,885] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:18:47,886] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,887] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:18:47,891] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,891] jepsen worker 14 - jepsen.util 374 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,891] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:47,892] jepsen worker 21 - jepsen.util 831 :invoke :write [7 47] INFO [2022-03-13 17:18:47,892] jepsen worker 24 - jepsen.util 654 :invoke :write [8 23] INFO [2022-03-13 17:18:47,893] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:48,170] jepsen worker 85 - jepsen.util 3955 :info :write [28 1] :RequestTimeoutException INFO [2022-03-13 17:18:48,170] jepsen worker 85 - jepsen.util 4045 :invoke :write [28 27] INFO [2022-03-13 17:18:48,240] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,240] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,240] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,240] jepsen worker 79 - jepsen.util 1159 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,240] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,240] jepsen worker 6 - jepsen.util 636 :invoke :write [2 36] INFO [2022-03-13 17:18:48,240] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,241] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 6] INFO [2022-03-13 17:18:48,242] jepsen worker 70 - jepsen.util 880 :invoke :write [23 43] INFO [2022-03-13 17:18:48,243] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:48,243] jepsen worker 7 - jepsen.util 637 :invoke :write [2 45] INFO [2022-03-13 17:18:48,244] jepsen worker 12 - jepsen.util 912 :invoke :write [4 21] INFO [2022-03-13 17:18:48,255] jepsen worker 43 - jepsen.util 673 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,255] jepsen worker 13 - jepsen.util 733 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,255] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,256] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,256] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,256] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:48,257] jepsen worker 53 - jepsen.util 593 :invoke :write [17 26] INFO [2022-03-13 17:18:48,258] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:48,259] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:48,260] jepsen worker 13 - jepsen.util 733 :invoke :write [4 38] INFO [2022-03-13 17:18:48,311] jepsen worker 83 - jepsen.util 4043 :info :write [27 5] :RequestTimeoutException INFO [2022-03-13 17:18:48,311] jepsen worker 83 - jepsen.util 4133 :invoke :read [27 nil] INFO [2022-03-13 17:18:48,436] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,436] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,437] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:48,437] jepsen worker 75 - jepsen.util 525 :invoke :write [25 48] INFO [2022-03-13 17:18:48,443] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,443] jepsen worker 33 - jepsen.util 753 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,444] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,444] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:48,445] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:48,445] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:18:48,511] jepsen worker 44 - jepsen.util 764 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,512] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] WARN [2022-03-13 17:18:48,596] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17390643,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18125631,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9241774,"encode_duration_us":19445,"operation_id":"0x11c1a","timeout_ms":10000,"last_server_duration_us":9060729,"total_duration_us":10005345},{"operation_name":"upsert","last_dispatch_duration_us":9045824,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10309824,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9236347,"encode_duration_us":18141,"operation_id":"0x12362","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10007596}],"total_count":2}} WARN [2022-03-13 17:18:48,609] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9039704,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10522808,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9353914,"encode_duration_us":17292,"operation_id":"0x120bb","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10002568},{"operation_name":"upsert","last_dispatch_duration_us":9047338,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":14602369,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9329961,"encode_duration_us":17512,"operation_id":"0x11f1a","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10009190},{"operation_name":"upsert","last_dispatch_duration_us":17836301,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18287300,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9158757,"encode_duration_us":20692,"operation_id":"0x11e29","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10008370},{"operation_name":"upsert","last_dispatch_duration_us":9041711,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10610668,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9367864,"encode_duration_us":19270,"operation_id":"0x120d4","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10010376}],"total_count":4}} INFO [2022-03-13 17:18:48,639] jepsen worker 60 - jepsen.util 690 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,640] jepsen worker 60 - jepsen.util 690 :invoke :write [20 15] INFO [2022-03-13 17:18:48,645] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,645] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,646] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:48,646] jepsen worker 51 - jepsen.util 771 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,647] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,648] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,648] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:48,648] jepsen worker 51 - jepsen.util 771 :invoke :write [17 19] INFO [2022-03-13 17:18:48,649] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,649] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,650] jepsen worker 80 - jepsen.util 800 :invoke :write [26 32] INFO [2022-03-13 17:18:48,651] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:48,652] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:48,654] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,655] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:48,657] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,658] jepsen worker 69 - jepsen.util 789 :invoke :write [23 14] INFO [2022-03-13 17:18:48,659] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:18:48,662] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,662] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,663] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,663] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:48,664] jepsen worker 23 - jepsen.util 563 :invoke :write [7 14] INFO [2022-03-13 17:18:48,665] jepsen worker 77 - jepsen.util 707 :invoke :write [25 28] INFO [2022-03-13 17:18:48,666] jepsen worker 61 - jepsen.util 781 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,667] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] WARN [2022-03-13 17:18:48,696] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":1148531,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":10957712,"total_server_duration_us":7150,"operation_id":"0x11723","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":30006848},{"operation_name":"get","last_dispatch_duration_us":14076364,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":14076364,"total_server_duration_us":33328,"operation_id":"0x11ec7","timeout_ms":30000,"last_server_duration_us":33328,"total_duration_us":14078469},{"operation_name":"upsert","last_dispatch_duration_us":7325033,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7725155,"total_server_duration_us":258348,"encode_duration_us":18070,"operation_id":"0x124a0","timeout_ms":10000,"last_server_duration_us":23054,"total_duration_us":10009542},{"operation_name":"upsert","last_dispatch_duration_us":205055,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":6829652,"total_server_duration_us":367644,"encode_duration_us":19931,"operation_id":"0x1234a","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10004636},{"operation_name":"upsert","last_dispatch_duration_us":34362,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":803994,"total_server_duration_us":270758,"encode_duration_us":20408,"operation_id":"0x12426","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10000944},{"operation_name":"upsert","last_dispatch_duration_us":9048253,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9446293,"total_server_duration_us":9203912,"encode_duration_us":17248,"operation_id":"0x1243f","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":9657834},{"operation_name":"get","last_dispatch_duration_us":9185306,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9185306,"total_server_duration_us":56,"operation_id":"0x120d5","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9185657},{"operation_name":"get","last_dispatch_duration_us":8955874,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8955874,"total_server_duration_us":126,"operation_id":"0x120ef","timeout_ms":30000,"last_server_duration_us":126,"total_duration_us":8956180},{"operation_name":"get","last_dispatch_duration_us":8208479,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8208479,"total_server_duration_us":49,"operation_id":"0x12180","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8208722},{"operation_name":"upsert","last_dispatch_duration_us":4818336,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4818336,"total_server_duration_us":104546,"encode_duration_us":26480,"operation_id":"0x123b4","timeout_ms":10000,"last_server_duration_us":104546,"total_duration_us":4825926}],"total_count":42}} WARN [2022-03-13 17:18:48,696] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":46258,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":6836193,"total_server_duration_us":2918,"operation_id":"0x11761","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":30007636},{"operation_name":"upsert","last_dispatch_duration_us":50809,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":5743017,"total_server_duration_us":413556,"encode_duration_us":20106,"operation_id":"0x12341","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10004856},{"operation_name":"upsert","last_dispatch_duration_us":7898180,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8549177,"total_server_duration_us":254662,"encode_duration_us":27114,"operation_id":"0x12040","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10002573},{"operation_name":"upsert","last_dispatch_duration_us":37101,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":3698096,"total_server_duration_us":487061,"encode_duration_us":17401,"operation_id":"0x12425","timeout_ms":10000,"last_server_duration_us":23898,"total_duration_us":10002472},{"operation_name":"upsert","last_dispatch_duration_us":7214108,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7726086,"total_server_duration_us":294115,"encode_duration_us":33768,"operation_id":"0x12197","timeout_ms":10000,"last_server_duration_us":44944,"total_duration_us":10000743},{"operation_name":"upsert","last_dispatch_duration_us":196315,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8056946,"total_server_duration_us":373300,"encode_duration_us":18372,"operation_id":"0x12167","timeout_ms":10000,"last_server_duration_us":69214,"total_duration_us":10000625},{"operation_name":"upsert","last_dispatch_duration_us":9084829,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9459533,"total_server_duration_us":9218040,"encode_duration_us":30698,"operation_id":"0x1203f","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":9683249},{"operation_name":"upsert","last_dispatch_duration_us":9097255,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9244722,"total_server_duration_us":9106602,"encode_duration_us":24388,"operation_id":"0x12123","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9271961},{"operation_name":"upsert","last_dispatch_duration_us":9122799,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9122799,"total_server_duration_us":9054355,"encode_duration_us":34440,"operation_id":"0x12151","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9127088},{"operation_name":"upsert","last_dispatch_duration_us":5757061,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":5757061,"total_server_duration_us":172334,"encode_duration_us":22239,"operation_id":"0x122f5","timeout_ms":10000,"last_server_duration_us":172334,"total_duration_us":5762743}],"total_count":36}} WARN [2022-03-13 17:18:48,696] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16339102,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16339102,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9029938,"encode_duration_us":18431,"operation_id":"0x11d81","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10007355}],"total_count":1}} WARN [2022-03-13 17:18:48,697] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":293751,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":610069,"total_server_duration_us":101309,"encode_duration_us":18382,"operation_id":"0x12446","timeout_ms":10000,"last_server_duration_us":72999,"total_duration_us":10008229},{"operation_name":"upsert","last_dispatch_duration_us":9045824,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":10309824,"total_server_duration_us":9236347,"encode_duration_us":18141,"operation_id":"0x12362","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10007596},{"operation_name":"upsert","last_dispatch_duration_us":240804,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9695632,"total_server_duration_us":406893,"encode_duration_us":24033,"operation_id":"0x1213e","timeout_ms":10000,"last_server_duration_us":86035,"total_duration_us":10001365},{"operation_name":"upsert","last_dispatch_duration_us":9247837,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9247837,"total_server_duration_us":9025695,"encode_duration_us":17465,"operation_id":"0x12392","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":9252187},{"operation_name":"get","last_dispatch_duration_us":52802,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":52802,"total_server_duration_us":7180,"operation_id":"0x12281","timeout_ms":30000,"last_server_duration_us":7180,"total_duration_us":6750288},{"operation_name":"upsert","last_dispatch_duration_us":4108467,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":4108467,"total_server_duration_us":141686,"encode_duration_us":27304,"operation_id":"0x123b6","timeout_ms":10000,"last_server_duration_us":141686,"total_duration_us":4112608},{"operation_name":"upsert","last_dispatch_duration_us":3895843,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":3895843,"total_server_duration_us":3880286,"encode_duration_us":18134,"operation_id":"0x123b5","timeout_ms":10000,"last_server_duration_us":3880286,"total_duration_us":3900233},{"operation_name":"get","last_dispatch_duration_us":1669181,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1669181,"total_server_duration_us":56,"operation_id":"0x12788","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":1672047},{"operation_name":"upsert","last_dispatch_duration_us":1570376,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1570376,"total_server_duration_us":1377994,"encode_duration_us":37649,"operation_id":"0x12648","timeout_ms":10000,"last_server_duration_us":1377994,"total_duration_us":1574338},{"operation_name":"upsert","last_dispatch_duration_us":1569432,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1569432,"total_server_duration_us":1198253,"encode_duration_us":32086,"operation_id":"0x126b4","timeout_ms":10000,"last_server_duration_us":1198253,"total_duration_us":1573886}],"total_count":19}} INFO [2022-03-13 17:18:48,700] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:48,700] jepsen worker nemesis - couchbase.util Rebalance check count: 635 INFO [2022-03-13 17:18:48,701] jepsen worker nemesis - couchbase.util Rebalance stuck count: 507 INFO [2022-03-13 17:18:48,735] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,736] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] WARN [2022-03-13 17:18:48,797] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":204393,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7025828,"total_server_duration_us":293638,"encode_duration_us":18589,"operation_id":"0x12354","timeout_ms":10000,"last_server_duration_us":28860,"total_duration_us":10008963},{"operation_name":"upsert","last_dispatch_duration_us":55345,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7093092,"total_server_duration_us":274324,"encode_duration_us":17364,"operation_id":"0x1237a","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10006568},{"operation_name":"upsert","last_dispatch_duration_us":8130788,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10319952,"total_server_duration_us":282064,"encode_duration_us":22162,"operation_id":"0x1215d","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10005913},{"operation_name":"upsert","last_dispatch_duration_us":9053883,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9053883,"total_server_duration_us":9037366,"encode_duration_us":26887,"operation_id":"0x122b3","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":9058587},{"operation_name":"get","last_dispatch_duration_us":5287108,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":5287108,"total_server_duration_us":5164855,"operation_id":"0x123b7","timeout_ms":30000,"last_server_duration_us":5164855,"total_duration_us":5289962},{"operation_name":"upsert","last_dispatch_duration_us":4543381,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4543381,"total_server_duration_us":79829,"encode_duration_us":27977,"operation_id":"0x123ae","timeout_ms":10000,"last_server_duration_us":79829,"total_duration_us":4548811},{"operation_name":"upsert","last_dispatch_duration_us":4345549,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4345549,"total_server_duration_us":257006,"encode_duration_us":21284,"operation_id":"0x123ad","timeout_ms":10000,"last_server_duration_us":257006,"total_duration_us":4350079},{"operation_name":"upsert","last_dispatch_duration_us":3962664,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":3962664,"total_server_duration_us":3890663,"encode_duration_us":28329,"operation_id":"0x123a8","timeout_ms":10000,"last_server_duration_us":3890663,"total_duration_us":3969051},{"operation_name":"get","last_dispatch_duration_us":3758764,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":3758764,"total_server_duration_us":135,"operation_id":"0x12467","timeout_ms":30000,"last_server_duration_us":135,"total_duration_us":3762006},{"operation_name":"upsert","last_dispatch_duration_us":2815967,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2815967,"total_server_duration_us":19887,"encode_duration_us":26908,"operation_id":"0x126b5","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":2820814}],"total_count":27}} WARN [2022-03-13 17:18:48,809] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11913108,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11913108,"total_server_duration_us":452,"operation_id":"0x1210e","timeout_ms":30000,"last_server_duration_us":452,"total_duration_us":11913430},{"operation_name":"upsert","last_dispatch_duration_us":9041711,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10610668,"total_server_duration_us":9367864,"encode_duration_us":19270,"operation_id":"0x120d4","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10010376},{"operation_name":"upsert","last_dispatch_duration_us":8404968,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8975005,"total_server_duration_us":326162,"encode_duration_us":24905,"operation_id":"0x121f0","timeout_ms":10000,"last_server_duration_us":23644,"total_duration_us":10010161},{"operation_name":"upsert","last_dispatch_duration_us":8421091,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10000888,"total_server_duration_us":392174,"encode_duration_us":25204,"operation_id":"0x120a7","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10009189},{"operation_name":"upsert","last_dispatch_duration_us":8586418,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10056898,"total_server_duration_us":358757,"encode_duration_us":17994,"operation_id":"0x120d8","timeout_ms":10000,"last_server_duration_us":27854,"total_duration_us":10006269},{"operation_name":"upsert","last_dispatch_duration_us":137475,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9455582,"total_server_duration_us":260896,"encode_duration_us":18689,"operation_id":"0x1235c","timeout_ms":10000,"last_server_duration_us":33819,"total_duration_us":10004954},{"operation_name":"upsert","last_dispatch_duration_us":9039704,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10522808,"total_server_duration_us":9353914,"encode_duration_us":17292,"operation_id":"0x120bb","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10002568},{"operation_name":"get","last_dispatch_duration_us":6027992,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6027992,"total_server_duration_us":22141,"operation_id":"0x12443","timeout_ms":30000,"last_server_duration_us":22141,"total_duration_us":6032200},{"operation_name":"upsert","last_dispatch_duration_us":5743761,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":5743761,"total_server_duration_us":146985,"encode_duration_us":36102,"operation_id":"0x122fd","timeout_ms":10000,"last_server_duration_us":146985,"total_duration_us":5759342},{"operation_name":"upsert","last_dispatch_duration_us":5684894,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":5684894,"total_server_duration_us":78132,"encode_duration_us":25537,"operation_id":"0x1235d","timeout_ms":10000,"last_server_duration_us":78132,"total_duration_us":5690688}],"total_count":48}} INFO [2022-03-13 17:18:48,844] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,844] jepsen worker 34 - jepsen.util 934 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,844] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:48,845] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:18:48,848] jepsen worker 79 - jepsen.util 1159 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,848] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:48,850] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,850] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,850] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:48,851] jepsen worker 61 - jepsen.util 781 :invoke :write [20 39] INFO [2022-03-13 17:18:48,858] jepsen worker 13 - jepsen.util 733 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,859] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,859] jepsen worker 13 - jepsen.util 733 :invoke :write [4 24] INFO [2022-03-13 17:18:48,859] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,860] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:48,861] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:48,882] jepsen worker 21 - jepsen.util 831 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:48,884] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:18:49,269] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,269] jepsen worker 24 - jepsen.util 654 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,270] jepsen worker 33 - jepsen.util 753 :invoke :write [11 31] INFO [2022-03-13 17:18:49,271] jepsen worker 24 - jepsen.util 654 :invoke :write [8 20] INFO [2022-03-13 17:18:49,283] jepsen worker 69 - jepsen.util 789 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,283] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,284] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:49,284] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:49,285] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,285] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,285] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:49,286] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:49,286] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,287] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:49,287] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,287] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:18:49,288] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,288] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:18:49,378] jepsen worker 52 - jepsen.util 682 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,378] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,378] jepsen worker 52 - jepsen.util 682 :invoke :write [17 48] INFO [2022-03-13 17:18:49,379] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:49,380] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,380] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,381] jepsen worker 8 - jepsen.util 728 :invoke :write [2 10] INFO [2022-03-13 17:18:49,383] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:49,473] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,473] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,473] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:18:49,474] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:49,484] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,484] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:49,513] jepsen worker 51 - jepsen.util 771 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,513] jepsen worker 51 - jepsen.util 771 :invoke :write [17 15] INFO [2022-03-13 17:18:49,695] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,695] jepsen worker 75 - jepsen.util 525 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,696] jepsen worker 80 - jepsen.util 800 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,696] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,697] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:49,697] jepsen worker 76 - jepsen.util 796 :invoke :write [25 26] INFO [2022-03-13 17:18:49,698] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:49,698] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:49,700] jepsen worker 23 - jepsen.util 563 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,700] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:49,707] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:49,707] jepsen worker nemesis - couchbase.util Rebalance check count: 636 INFO [2022-03-13 17:18:49,707] jepsen worker nemesis - couchbase.util Rebalance stuck count: 508 INFO [2022-03-13 17:18:49,736] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,736] jepsen worker 0 - jepsen.util 4230 :info :write [0 45] :RequestTimeoutException INFO [2022-03-13 17:18:49,737] jepsen worker 0 - jepsen.util 4320 :invoke :read [0 nil] INFO [2022-03-13 17:18:49,737] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:49,907] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,908] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:18:49,916] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,916] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,917] jepsen worker 71 - jepsen.util 611 :invoke :write [23 14] INFO [2022-03-13 17:18:49,918] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:49,919] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:49,920] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:18:49,948] jepsen worker 33 - jepsen.util 753 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,016] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:50,110] jepsen worker 26 - jepsen.util 566 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,110] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:50,111] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:50,111] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:50,159] jepsen worker 71 - jepsen.util 611 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,160] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:18:50,203] jepsen worker 1 - jepsen.util 3961 :info :write [0 30] :SyncWriteAmbiguous INFO [2022-03-13 17:18:50,203] jepsen worker 1 - jepsen.util 4051 :invoke :write [0 5] INFO [2022-03-13 17:18:50,305] jepsen worker 24 - jepsen.util 654 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,306] jepsen worker 70 - jepsen.util 880 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,306] jepsen worker 77 - jepsen.util 707 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,306] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,306] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,306] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,307] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:50,307] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:50,308] jepsen worker 13 - jepsen.util 733 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,308] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:50,308] jepsen worker 77 - jepsen.util 707 :invoke :write [25 11] INFO [2022-03-13 17:18:50,309] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:50,310] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 7] INFO [2022-03-13 17:18:50,310] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:18:50,314] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,315] jepsen worker 75 - jepsen.util 525 :invoke :write [25 15] INFO [2022-03-13 17:18:50,532] jepsen worker 53 - jepsen.util 593 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,533] jepsen worker 53 - jepsen.util 593 :invoke :write [17 15] INFO [2022-03-13 17:18:50,696] jepsen worker 20 - jepsen.util 4610 :info :write [6 31] :RequestTimeoutException INFO [2022-03-13 17:18:50,697] jepsen worker 20 - jepsen.util 4700 :invoke :write [6 29] INFO [2022-03-13 17:18:50,710] jepsen worker 16 - jepsen.util 4246 :info :write [5 12] :RequestTimeoutException INFO [2022-03-13 17:18:50,711] jepsen worker 16 - jepsen.util 4336 :invoke :read [5 nil] INFO [2022-03-13 17:18:50,711] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,712] jepsen worker 25 - jepsen.util 565 :invoke :write [8 22] INFO [2022-03-13 17:18:50,713] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:50,713] jepsen worker nemesis - couchbase.util Rebalance check count: 637 INFO [2022-03-13 17:18:50,713] jepsen worker nemesis - couchbase.util Rebalance stuck count: 509 INFO [2022-03-13 17:18:50,718] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,719] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:18:50,719] jepsen worker 18 - jepsen.util 4248 :info :write [6 6] :RequestTimeoutException INFO [2022-03-13 17:18:50,720] jepsen worker 27 - jepsen.util 4527 :info :write [9 22] :RequestTimeoutException INFO [2022-03-13 17:18:50,721] jepsen worker 19 - jepsen.util 4159 :info :write [6 0] :RequestTimeoutException INFO [2022-03-13 17:18:50,721] jepsen worker 27 - jepsen.util 4617 :invoke :read [9 nil] INFO [2022-03-13 17:18:50,722] jepsen worker 18 - jepsen.util 4338 :invoke :write [6 47] INFO [2022-03-13 17:18:50,722] jepsen worker 19 - jepsen.util 4249 :invoke :write [6 23] INFO [2022-03-13 17:18:50,723] jepsen worker 27 - jepsen.util 4617 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,723] jepsen worker 27 - jepsen.util 4617 :invoke :write [9 23] INFO [2022-03-13 17:18:50,732] jepsen worker 7 - jepsen.util 637 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,732] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:50,732] jepsen worker 76 - jepsen.util 796 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,733] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:50,740] jepsen worker 55 - jepsen.util 4015 :info :write [18 40] :RequestTimeoutException INFO [2022-03-13 17:18:50,741] jepsen worker 55 - jepsen.util 4105 :invoke :read [18 nil] INFO [2022-03-13 17:18:50,768] jepsen worker 60 - jepsen.util 690 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,768] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:50,768] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:18:50,769] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:50,778] jepsen worker 39 - jepsen.util 4629 :info :write [13 11] :RequestTimeoutException INFO [2022-03-13 17:18:50,779] jepsen worker 39 - jepsen.util 4719 :invoke :write [13 42] INFO [2022-03-13 17:18:50,816] jepsen worker 45 - jepsen.util 4635 :info :write [15 7] :SyncWriteAmbiguous INFO [2022-03-13 17:18:50,817] jepsen worker 45 - jepsen.util 4725 :invoke :write [15 49] INFO [2022-03-13 17:18:51,056] jepsen worker 84 - jepsen.util 4674 :info :write [28 40] :RequestTimeoutException INFO [2022-03-13 17:18:51,056] jepsen worker 84 - jepsen.util 4764 :invoke :read [28 nil] INFO [2022-03-13 17:18:51,118] jepsen worker 54 - jepsen.util 4104 :info :write [18 39] :RequestTimeoutException INFO [2022-03-13 17:18:51,118] jepsen worker 84 - jepsen.util 4764 :ok :read [28 44] INFO [2022-03-13 17:18:51,118] jepsen worker 54 - jepsen.util 4194 :invoke :read [18 nil] INFO [2022-03-13 17:18:51,119] jepsen worker 84 - jepsen.util 4764 :invoke :write [28 23] INFO [2022-03-13 17:18:51,123] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,124] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:51,143] jepsen worker 61 - jepsen.util 781 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,143] jepsen worker 61 - jepsen.util 781 :invoke :write [20 44] INFO [2022-03-13 17:18:51,146] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,146] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,146] jepsen worker 78 - jepsen.util 708 :invoke :write [26 14] INFO [2022-03-13 17:18:51,147] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:51,164] jepsen worker 52 - jepsen.util 682 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,165] jepsen worker 52 - jepsen.util 682 :invoke :write [17 22] INFO [2022-03-13 17:18:51,167] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,168] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:51,168] jepsen worker 26 - jepsen.util 566 :invoke :write [8 44] INFO [2022-03-13 17:18:51,170] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:51,286] jepsen worker 31 - jepsen.util 4441 :info :write [10 29] :SyncWriteAmbiguous INFO [2022-03-13 17:18:51,287] jepsen worker 31 - jepsen.util 4531 :invoke :read [10 nil] INFO [2022-03-13 17:18:51,390] jepsen worker 47 - jepsen.util 4277 :info :write [15 13] :RequestTimeoutException INFO [2022-03-13 17:18:51,391] jepsen worker 47 - jepsen.util 4367 :invoke :write [15 21] INFO [2022-03-13 17:18:51,426] jepsen worker 59 - jepsen.util 4379 :info :write [19 16] :RequestTimeoutException INFO [2022-03-13 17:18:51,427] jepsen worker 59 - jepsen.util 4469 :invoke :write [19 35] INFO [2022-03-13 17:18:51,434] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,434] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,434] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:51,435] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:18:51,438] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,439] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,439] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:18:51,439] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:18:51,446] jepsen worker 81 - jepsen.util 4941 :info :write [27 19] :RequestTimeoutException INFO [2022-03-13 17:18:51,449] jepsen worker 81 - jepsen.util 5031 :invoke :write [27 25] INFO [2022-03-13 17:18:51,504] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,505] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:51,505] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,505] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:51,540] jepsen worker 88 - jepsen.util 2428 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,540] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,540] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,541] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,541] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,543] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,543] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:51,543] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:51,544] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:18:51,544] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,544] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:51,546] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,546] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:51,546] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,546] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,546] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:18:51,547] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,548] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:18:51,549] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:18:51,549] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,550] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,551] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,579] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,581] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 9] INFO [2022-03-13 17:18:51,592] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,592] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,593] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,594] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,598] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,598] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,599] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,599] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,600] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,601] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:18:51,643] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,645] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:18:51,687] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,688] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:51,693] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,694] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:18:51,719] jepsen worker 31 - jepsen.util 4531 :fail :read [10 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,720] jepsen worker 31 - jepsen.util 4531 :invoke :read [10 nil] INFO [2022-03-13 17:18:51,723] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:51,723] jepsen worker nemesis - couchbase.util Rebalance check count: 638 INFO [2022-03-13 17:18:51,723] jepsen worker nemesis - couchbase.util Rebalance stuck count: 510 INFO [2022-03-13 17:18:51,749] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,749] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,749] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 3] INFO [2022-03-13 17:18:51,750] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 22] INFO [2022-03-13 17:18:51,751] jepsen worker 89 - jepsen.util 2789 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,752] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,753] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,753] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,754] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,755] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,755] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,755] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,756] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 16] INFO [2022-03-13 17:18:51,756] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,756] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:18:51,761] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:18:51,778] jepsen worker 75 - jepsen.util 525 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,778] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,778] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,779] jepsen worker 51 - jepsen.util 771 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,779] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,779] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,779] jepsen worker 75 - jepsen.util 525 :invoke :write [25 12] INFO [2022-03-13 17:18:51,779] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,780] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,780] jepsen worker 13 - jepsen.util 733 :invoke :write [4 31] INFO [2022-03-13 17:18:51,781] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:18:51,782] jepsen worker 76 - jepsen.util 796 :invoke :write [25 24] INFO [2022-03-13 17:18:51,782] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:51,784] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:51,784] jepsen worker 51 - jepsen.util 771 :invoke :write [17 19] INFO [2022-03-13 17:18:51,786] jepsen worker 54 - jepsen.util 4194 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,787] jepsen worker 54 - jepsen.util 4194 :invoke :write [18 17] INFO [2022-03-13 17:18:51,794] jepsen worker 34 - jepsen.util 934 :invoke :write [11 41] INFO [2022-03-13 17:18:51,795] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,797] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:18:51,818] jepsen worker 12 - jepsen.util 912 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,819] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:18:51,819] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,820] jepsen worker 44 - jepsen.util 764 :invoke :write [14 48] INFO [2022-03-13 17:18:51,820] jepsen worker 12 - jepsen.util 912 :invoke :write [4 46] INFO [2022-03-13 17:18:51,825] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:18:51,828] jepsen worker 30 - jepsen.util 4350 :info :write [10 24] :RequestTimeoutException INFO [2022-03-13 17:18:51,829] jepsen worker 30 - jepsen.util 4440 :invoke :write [10 26] INFO [2022-03-13 17:18:51,881] jepsen worker 77 - jepsen.util 707 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,882] jepsen worker 77 - jepsen.util 707 :invoke :write [25 34] INFO [2022-03-13 17:18:51,895] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,896] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:18:51,927] jepsen worker 31 - jepsen.util 4531 :fail :read [10 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,927] jepsen worker 31 - jepsen.util 4531 :invoke :write [10 44] INFO [2022-03-13 17:18:51,947] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,948] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:18:51,952] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,953] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:51,992] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,992] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,993] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,994] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,994] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:51,995] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:51,996] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:18:51,996] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,997] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:51,998] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:51,999] jepsen worker 53 - jepsen.util 593 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:51,999] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,000] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,000] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:52,000] jepsen worker 60 - jepsen.util 690 :invoke :write [20 31] INFO [2022-03-13 17:18:52,001] jepsen worker 53 - jepsen.util 593 :invoke :write [17 11] INFO [2022-03-13 17:18:52,001] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:52,001] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:18:52,050] jepsen worker 6 - jepsen.util 636 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,051] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:52,051] jepsen worker 6 - jepsen.util 636 :invoke :write [2 46] INFO [2022-03-13 17:18:52,052] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:52,064] jepsen worker 88 - jepsen.util 2428 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,064] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 17] INFO [2022-03-13 17:18:52,066] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,066] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:52,068] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,069] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,069] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:18:52,076] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:52,077] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,078] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:52,079] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,080] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:18:52,196] jepsen worker 79 - jepsen.util 1159 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,196] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,196] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:52,197] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:52,226] jepsen worker 74 - jepsen.util 4214 :info :write [24 44] :RequestTimeoutException INFO [2022-03-13 17:18:52,227] jepsen worker 74 - jepsen.util 4304 :invoke :read [24 nil] INFO [2022-03-13 17:18:52,412] jepsen worker 25 - jepsen.util 565 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,412] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,412] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,413] jepsen worker 80 - jepsen.util 800 :invoke :write [26 12] INFO [2022-03-13 17:18:52,413] jepsen worker 25 - jepsen.util 565 :invoke :write [8 15] INFO [2022-03-13 17:18:52,414] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:52,415] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,416] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,417] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,417] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,418] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,419] jepsen worker 71 - jepsen.util 611 :invoke :write [23 19] INFO [2022-03-13 17:18:52,421] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,422] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,424] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,426] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,428] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,428] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,428] jepsen worker 8 - jepsen.util 728 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,428] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,429] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:52,429] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,430] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,430] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,431] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,432] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,432] jepsen worker 7 - jepsen.util 637 :invoke :write [2 7] INFO [2022-03-13 17:18:52,432] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,434] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,440] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,603] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,604] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:18:52,639] jepsen worker 52 - jepsen.util 682 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,640] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:52,641] jepsen worker 13 - jepsen.util 733 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,642] jepsen worker 13 - jepsen.util 733 :invoke :write [4 34] INFO [2022-03-13 17:18:52,643] jepsen worker 34 - jepsen.util 934 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,644] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:52,656] jepsen worker 71 - jepsen.util 611 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,657] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:18:52,708] jepsen worker 74 - jepsen.util 4304 :ok :read [24 24] INFO [2022-03-13 17:18:52,709] jepsen worker 74 - jepsen.util 4304 :invoke :read [24 nil] INFO [2022-03-13 17:18:52,729] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:52,729] jepsen worker nemesis - couchbase.util Rebalance check count: 639 INFO [2022-03-13 17:18:52,729] jepsen worker nemesis - couchbase.util Rebalance stuck count: 511 INFO [2022-03-13 17:18:52,786] jepsen worker 41 - jepsen.util 4451 :info :write [13 7] :RequestTimeoutException INFO [2022-03-13 17:18:52,787] jepsen worker 41 - jepsen.util 4541 :invoke :read [13 nil] INFO [2022-03-13 17:18:52,840] jepsen worker 78 - jepsen.util 708 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,840] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:52,880] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,881] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:52,900] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,900] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,900] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,901] jepsen worker 21 - jepsen.util 831 :invoke :write [7 36] INFO [2022-03-13 17:18:52,902] jepsen worker 24 - jepsen.util 654 :invoke :write [8 40] INFO [2022-03-13 17:18:52,902] jepsen worker 33 - jepsen.util 753 :invoke :write [11 10] INFO [2022-03-13 17:18:52,930] jepsen worker 26 - jepsen.util 566 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,930] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,930] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:18:52,931] jepsen worker 22 - jepsen.util 652 :invoke :write [7 17] INFO [2022-03-13 17:18:52,933] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:52,934] jepsen worker 43 - jepsen.util 673 :invoke :write [14 31] INFO [2022-03-13 17:18:52,949] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:52,950] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:18:53,006] jepsen worker 17 - jepsen.util 4607 :info :write [5 9] :RequestTimeoutException INFO [2022-03-13 17:18:53,007] jepsen worker 17 - jepsen.util 4697 :invoke :write [5 19] INFO [2022-03-13 17:18:53,017] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,018] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:53,018] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,019] jepsen worker 23 - jepsen.util 563 :invoke :write [7 31] INFO [2022-03-13 17:18:53,021] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,021] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:18:53,070] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:18:53,071] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:18:53,251] jepsen worker 65 - jepsen.util 4655 :info :write [21 32] :RequestTimeoutException INFO [2022-03-13 17:18:53,251] jepsen worker 65 - jepsen.util 4745 :invoke :read [21 nil] INFO [2022-03-13 17:18:53,288] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,288] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,288] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,288] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,289] jepsen worker 8 - jepsen.util 728 :invoke :write [2 19] INFO [2022-03-13 17:18:53,289] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:18:53,290] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:53,291] jepsen worker 52 - jepsen.util 682 :invoke :write [17 45] INFO [2022-03-13 17:18:53,366] jepsen worker 63 - jepsen.util 4473 :info :write [21 45] :SyncWriteAmbiguous INFO [2022-03-13 17:18:53,366] jepsen worker 63 - jepsen.util 4563 :invoke :read [21 nil] INFO [2022-03-13 17:18:53,376] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,377] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,377] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:18:53,378] jepsen worker 78 - jepsen.util 708 :invoke :write [26 39] INFO [2022-03-13 17:18:53,430] jepsen worker 74 - jepsen.util 4304 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,431] jepsen worker 74 - jepsen.util 4304 :invoke :read [24 nil] INFO [2022-03-13 17:18:53,433] jepsen worker 65 - jepsen.util 4745 :ok :read [21 :nil] INFO [2022-03-13 17:18:53,433] jepsen worker 65 - jepsen.util 4745 :invoke :read [21 nil] INFO [2022-03-13 17:18:53,434] jepsen worker 65 - jepsen.util 4745 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,435] jepsen worker 65 - jepsen.util 4745 :invoke :write [21 2] INFO [2022-03-13 17:18:53,452] jepsen worker 63 - jepsen.util 4563 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,453] jepsen worker 63 - jepsen.util 4563 :invoke :write [21 36] INFO [2022-03-13 17:18:53,485] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,486] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:18:53,493] jepsen worker 9 - jepsen.util 3969 :info :write [3 6] :RequestTimeoutException INFO [2022-03-13 17:18:53,494] jepsen worker 9 - jepsen.util 4059 :invoke :write [3 48] INFO [2022-03-13 17:18:53,504] jepsen worker 29 - jepsen.util 4889 :info :write [9 1] :RequestTimeoutException INFO [2022-03-13 17:18:53,504] jepsen worker 46 - jepsen.util 4456 :info :write [15 35] :RequestTimeoutException INFO [2022-03-13 17:18:53,505] jepsen worker 29 - jepsen.util 4979 :invoke :read [9 nil] INFO [2022-03-13 17:18:53,505] jepsen worker 46 - jepsen.util 4546 :invoke :write [15 23] INFO [2022-03-13 17:18:53,540] jepsen worker 37 - jepsen.util 2467 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,543] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:53,548] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,548] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:53,585] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,587] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:53,594] jepsen worker 3 - jepsen.util 2253 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,594] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,595] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:53,595] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:18:53,600] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,600] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:18:53,677] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,677] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,678] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:18:53,679] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:53,712] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,712] jepsen worker 88 - jepsen.util 2428 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,712] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 21] INFO [2022-03-13 17:18:53,714] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:53,734] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:53,734] jepsen worker nemesis - couchbase.util Rebalance check count: 640 INFO [2022-03-13 17:18:53,734] jepsen worker nemesis - couchbase.util Rebalance stuck count: 512 INFO [2022-03-13 17:18:53,749] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,749] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,751] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:53,752] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:18:53,752] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,752] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,753] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:18:53,754] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 8] INFO [2022-03-13 17:18:53,818] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,819] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:18:53,841] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,842] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:53,843] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,844] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,844] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:53,845] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:18:53,845] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,846] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:53,847] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,848] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:18:53,878] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,878] jepsen worker 88 - jepsen.util 2428 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,879] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:53,879] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 11] INFO [2022-03-13 17:18:53,880] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,881] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:53,883] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,883] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 27] INFO [2022-03-13 17:18:53,930] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,931] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:18:53,965] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,965] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:53,987] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:53,987] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,007] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,007] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,035] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,036] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,046] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,047] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:18:54,073] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,073] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,073] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,074] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:18:54,075] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,075] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,082] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,082] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:18:54,098] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,099] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 7] INFO [2022-03-13 17:18:54,134] jepsen worker 29 - jepsen.util 4979 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,135] jepsen worker 29 - jepsen.util 4979 :invoke :read [9 nil] INFO [2022-03-13 17:18:54,140] jepsen worker 88 - jepsen.util 2428 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,141] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 19] INFO [2022-03-13 17:18:54,149] jepsen worker 74 - jepsen.util 4304 :ok :read [24 24] INFO [2022-03-13 17:18:54,149] jepsen worker 74 - jepsen.util 4304 :invoke :write [24 27] INFO [2022-03-13 17:18:54,173] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,174] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,181] jepsen worker 3 - jepsen.util 2253 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,181] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,182] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,182] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,219] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,220] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:18:54,236] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,236] jepsen worker 69 - jepsen.util 789 :invoke :write [23 8] INFO [2022-03-13 17:18:54,252] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,252] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,252] jepsen worker 36 - jepsen.util 2466 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,252] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,253] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,254] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:18:54,254] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,254] jepsen worker 23 - jepsen.util 563 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,255] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,255] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,256] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:54,257] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:18:54,305] jepsen worker 44 - jepsen.util 764 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,306] jepsen worker 44 - jepsen.util 764 :invoke :write [14 25] INFO [2022-03-13 17:18:54,318] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,319] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,319] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,320] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:18:54,320] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,321] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:18:54,324] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,324] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,326] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,326] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,327] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,328] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,329] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,330] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,335] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 19] INFO [2022-03-13 17:18:54,336] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:18:54,350] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,352] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:18:54,384] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,384] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,408] jepsen worker 72 - jepsen.util 4482 :info :write [24 26] :RequestTimeoutException INFO [2022-03-13 17:18:54,411] jepsen worker 72 - jepsen.util 4572 :invoke :read [24 nil] INFO [2022-03-13 17:18:54,428] jepsen worker 13 - jepsen.util 733 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,428] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:54,428] jepsen worker 60 - jepsen.util 690 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,429] jepsen worker 60 - jepsen.util 690 :invoke :write [20 20] INFO [2022-03-13 17:18:54,437] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,437] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,438] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 46] INFO [2022-03-13 17:18:54,438] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,446] jepsen worker 37 - jepsen.util 2467 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,447] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,474] jepsen worker 68 - jepsen.util 4478 :info :write [22 20] :RequestTimeoutException INFO [2022-03-13 17:18:54,475] jepsen worker 68 - jepsen.util 4568 :invoke :write [22 42] INFO [2022-03-13 17:18:54,503] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,503] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,503] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,504] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 47] INFO [2022-03-13 17:18:54,526] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,527] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,528] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,528] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,529] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,529] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:18:54,532] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,532] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 49] INFO [2022-03-13 17:18:54,547] jepsen worker 88 - jepsen.util 2428 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,547] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,548] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,548] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:18:54,572] jepsen worker 4 - jepsen.util 2614 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,572] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,572] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,573] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,602] jepsen worker 5 - jepsen.util 2705 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,602] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,603] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,604] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:18:54,604] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,605] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,605] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,606] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:18:54,607] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,607] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:18:54,610] jepsen worker 29 - jepsen.util 4979 :ok :read [9 26] INFO [2022-03-13 17:18:54,611] jepsen worker 29 - jepsen.util 4979 :invoke :read [9 nil] INFO [2022-03-13 17:18:54,615] jepsen worker 55 - jepsen.util 4105 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,615] jepsen worker 72 - jepsen.util 4572 :fail :read [24 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,616] jepsen worker 72 - jepsen.util 4572 :invoke :write [24 5] INFO [2022-03-13 17:18:54,616] jepsen worker 55 - jepsen.util 4105 :invoke :read [18 nil] INFO [2022-03-13 17:18:54,637] jepsen worker 61 - jepsen.util 781 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,637] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,638] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 33] INFO [2022-03-13 17:18:54,638] jepsen worker 61 - jepsen.util 781 :invoke :write [20 4] INFO [2022-03-13 17:18:54,654] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,655] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:18:54,712] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,712] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,712] jepsen worker 14 - jepsen.util 374 :invoke :write [4 44] INFO [2022-03-13 17:18:54,713] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:54,715] jepsen worker 89 - jepsen.util 2789 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,715] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:18:54,740] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:54,740] jepsen worker nemesis - couchbase.util Rebalance check count: 641 INFO [2022-03-13 17:18:54,740] jepsen worker nemesis - couchbase.util Rebalance stuck count: 513 INFO [2022-03-13 17:18:54,747] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,747] jepsen worker 3 - jepsen.util 2253 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,747] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,747] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,748] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,748] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:54,750] jepsen worker 32 - jepsen.util 4622 :info :write [10 16] :RequestTimeoutException INFO [2022-03-13 17:18:54,750] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,750] jepsen worker 32 - jepsen.util 4712 :invoke :read [10 nil] INFO [2022-03-13 17:18:54,751] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:54,751] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,751] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,752] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,752] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:18:54,752] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 11] INFO [2022-03-13 17:18:54,753] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:18:54,803] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,804] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:18:54,808] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,808] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,809] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 27] INFO [2022-03-13 17:18:54,810] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 8] INFO [2022-03-13 17:18:54,871] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,871] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,917] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,918] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:18:54,937] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,938] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:54,938] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:54,939] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:18:55,047] jepsen worker 12 - jepsen.util 912 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,048] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:18:55,051] jepsen worker 75 - jepsen.util 525 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,051] jepsen worker 53 - jepsen.util 593 :fail :write [17 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,052] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,066] jepsen worker 75 - jepsen.util 525 :invoke :write [25 3] INFO [2022-03-13 17:18:55,066] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:55,066] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:55,089] jepsen worker 69 - jepsen.util 789 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,089] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:18:55,090] jepsen worker 40 - jepsen.util 4630 :info :write [13 29] :RequestTimeoutException INFO [2022-03-13 17:18:55,091] jepsen worker 40 - jepsen.util 4720 :invoke :write [13 19] INFO [2022-03-13 17:18:55,096] jepsen worker 56 - jepsen.util 4286 :info :write [18 17] :RequestTimeoutException INFO [2022-03-13 17:18:55,097] jepsen worker 56 - jepsen.util 4376 :invoke :write [18 38] INFO [2022-03-13 17:18:55,098] jepsen worker 11 - jepsen.util 4241 :info :write [3 33] :RequestTimeoutException INFO [2022-03-13 17:18:55,099] jepsen worker 11 - jepsen.util 4331 :invoke :write [3 17] INFO [2022-03-13 17:18:55,334] jepsen worker 28 - jepsen.util 4438 :info :write [9 18] :RequestTimeoutException INFO [2022-03-13 17:18:55,335] jepsen worker 28 - jepsen.util 4528 :invoke :write [9 44] INFO [2022-03-13 17:18:55,458] jepsen worker 77 - jepsen.util 707 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,459] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:18:55,470] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,471] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:18:55,506] jepsen worker 57 - jepsen.util 4467 :info :write [19 7] :RequestTimeoutException INFO [2022-03-13 17:18:55,507] jepsen worker 57 - jepsen.util 4557 :invoke :read [19 nil] INFO [2022-03-13 17:18:55,518] jepsen worker 82 - jepsen.util 4672 :ok :read [27 4] INFO [2022-03-13 17:18:55,519] jepsen worker 82 - jepsen.util 4672 :invoke :read [27 nil] INFO [2022-03-13 17:18:55,522] jepsen worker 83 - jepsen.util 4133 :ok :read [27 4] INFO [2022-03-13 17:18:55,523] jepsen worker 83 - jepsen.util 4133 :invoke :read [27 nil] INFO [2022-03-13 17:18:55,544] jepsen worker 82 - jepsen.util 4672 :ok :read [27 4] INFO [2022-03-13 17:18:55,545] jepsen worker 82 - jepsen.util 4672 :invoke :read [27 nil] INFO [2022-03-13 17:18:55,548] jepsen worker 83 - jepsen.util 4133 :ok :read [27 4] INFO [2022-03-13 17:18:55,548] jepsen worker 83 - jepsen.util 4133 :invoke :read [27 nil] INFO [2022-03-13 17:18:55,569] jepsen worker 82 - jepsen.util 4672 :ok :read [27 4] INFO [2022-03-13 17:18:55,569] jepsen worker 82 - jepsen.util 4672 :invoke :read [27 nil] INFO [2022-03-13 17:18:55,573] jepsen worker 83 - jepsen.util 4133 :ok :read [27 4] INFO [2022-03-13 17:18:55,574] jepsen worker 83 - jepsen.util 4133 :invoke :read [27 nil] INFO [2022-03-13 17:18:55,574] jepsen worker 64 - jepsen.util 4294 :info :write [21 24] :RequestTimeoutException INFO [2022-03-13 17:18:55,575] jepsen worker 64 - jepsen.util 4384 :invoke :read [21 nil] INFO [2022-03-13 17:18:55,586] jepsen worker 2 - jepsen.util 4682 :info :write [0 5] :RequestTimeoutException INFO [2022-03-13 17:18:55,587] jepsen worker 2 - jepsen.util 4772 :invoke :read [0 nil] INFO [2022-03-13 17:18:55,588] jepsen worker 2 - jepsen.util 4772 :fail :read [0 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,589] jepsen worker 2 - jepsen.util 4772 :invoke :write [0 29] INFO [2022-03-13 17:18:55,595] jepsen worker 82 - jepsen.util 4672 :ok :read [27 4] INFO [2022-03-13 17:18:55,596] jepsen worker 82 - jepsen.util 4672 :invoke :write [27 28] INFO [2022-03-13 17:18:55,598] jepsen worker 83 - jepsen.util 4133 :ok :read [27 4] INFO [2022-03-13 17:18:55,598] jepsen worker 83 - jepsen.util 4133 :invoke :write [27 7] INFO [2022-03-13 17:18:55,683] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,683] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,683] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,684] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:18:55,684] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:18:55,686] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:18:55,726] jepsen worker 4 - jepsen.util 2614 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,727] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:55,746] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:55,746] jepsen worker nemesis - couchbase.util Rebalance check count: 642 INFO [2022-03-13 17:18:55,746] jepsen worker nemesis - couchbase.util Rebalance stuck count: 514 INFO [2022-03-13 17:18:55,767] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,767] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:18:55,776] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,776] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:18:55,793] jepsen worker 88 - jepsen.util 2428 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,793] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 23] INFO [2022-03-13 17:18:55,824] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,825] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:55,833] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,833] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:55,834] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,835] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:18:55,863] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,864] jepsen worker 42 - jepsen.util 402 :invoke :write [14 34] INFO [2022-03-13 17:18:55,868] jepsen worker 76 - jepsen.util 796 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,868] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,868] jepsen worker 52 - jepsen.util 682 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,869] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:18:55,869] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:18:55,870] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:55,876] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,877] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:18:55,890] jepsen worker 80 - jepsen.util 800 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,891] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:55,905] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,906] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,906] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:55,907] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:18:55,908] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,908] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,909] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:18:55,910] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:55,910] jepsen worker 37 - jepsen.util 2467 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,911] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:18:55,914] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,914] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:55,915] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,916] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:55,917] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,918] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:55,919] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,920] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:18:55,941] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,941] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,941] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:55,942] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:18:55,981] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:55,982] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:18:56,021] jepsen worker 73 - jepsen.util 4663 :info :write [24 30] :RequestTimeoutException INFO [2022-03-13 17:18:56,021] jepsen worker 73 - jepsen.util 4753 :invoke :write [24 33] INFO [2022-03-13 17:18:56,049] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,050] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,063] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,063] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,063] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,064] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:18:56,064] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,065] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:18:56,067] jepsen worker 78 - jepsen.util 708 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,068] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,068] jepsen worker 78 - jepsen.util 708 :invoke :write [26 41] INFO [2022-03-13 17:18:56,069] jepsen worker 23 - jepsen.util 563 :invoke :write [7 31] INFO [2022-03-13 17:18:56,083] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,083] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,085] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,085] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:18:56,100] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,100] jepsen worker 25 - jepsen.util 565 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,101] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:56,101] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:56,133] jepsen worker 15 - jepsen.util 4335 :info :write [5 19] :SyncWriteAmbiguous INFO [2022-03-13 17:18:56,135] jepsen worker 15 - jepsen.util 4425 :invoke :read [5 nil] INFO [2022-03-13 17:18:56,135] jepsen worker 66 - jepsen.util 4206 :ok :read [22 30] INFO [2022-03-13 17:18:56,136] jepsen worker 66 - jepsen.util 4206 :invoke :write [22 19] INFO [2022-03-13 17:18:56,148] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,148] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,148] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:18:56,149] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,163] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,163] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,163] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,164] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,170] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,170] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,171] jepsen worker 88 - jepsen.util 2428 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,171] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:56,172] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:18:56,172] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,225] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,225] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,226] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:18:56,226] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:18:56,290] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,290] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,297] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,297] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,297] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:18:56,298] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,315] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,316] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:18:56,386] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,386] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,387] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:18:56,387] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,391] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,392] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:18:56,411] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,412] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,418] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,418] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,418] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,419] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:18:56,419] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:18:56,420] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 27] INFO [2022-03-13 17:18:56,502] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,503] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:18:56,504] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,504] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:18:56,506] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,507] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,567] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,567] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,609] jepsen worker 51 - jepsen.util 771 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,609] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:56,631] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,631] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,632] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,632] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:18:56,637] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,638] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,638] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,639] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:18:56,639] jepsen worker 21 - jepsen.util 831 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,639] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:56,640] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,640] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,642] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,643] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,646] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,647] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 10] INFO [2022-03-13 17:18:56,668] jepsen worker 24 - jepsen.util 654 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,668] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:56,676] jepsen worker 67 - jepsen.util 3667 :info :write [22 24] :RequestTimeoutException INFO [2022-03-13 17:18:56,677] jepsen worker 67 - jepsen.util 3757 :invoke :write [22 43] INFO [2022-03-13 17:18:56,703] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,704] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:18:56,705] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,706] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,752] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:56,752] jepsen worker nemesis - couchbase.util Rebalance check count: 643 INFO [2022-03-13 17:18:56,752] jepsen worker nemesis - couchbase.util Rebalance stuck count: 515 INFO [2022-03-13 17:18:56,769] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,770] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 37] INFO [2022-03-13 17:18:56,771] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,771] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,772] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:18:56,773] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:18:56,774] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,774] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,774] jepsen worker 88 - jepsen.util 2428 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,775] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:18:56,775] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,776] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,777] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,777] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,777] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,779] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,779] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:18:56,780] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,781] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,782] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 46] INFO [2022-03-13 17:18:56,786] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,787] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 30] INFO [2022-03-13 17:18:56,813] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,814] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,819] jepsen worker 75 - jepsen.util 525 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,819] jepsen worker 22 - jepsen.util 652 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,819] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,820] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:18:56,820] jepsen worker 22 - jepsen.util 652 :invoke :write [7 45] INFO [2022-03-13 17:18:56,820] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,821] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,821] jepsen worker 62 - jepsen.util 512 :invoke :write [20 4] INFO [2022-03-13 17:18:56,827] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:56,827] jepsen worker 76 - jepsen.util 796 :invoke :write [25 0] INFO [2022-03-13 17:18:56,841] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,841] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,841] jepsen worker 75 - jepsen.util 525 :invoke :write [25 26] INFO [2022-03-13 17:18:56,842] jepsen worker 13 - jepsen.util 733 :invoke :write [4 28] INFO [2022-03-13 17:18:56,884] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,884] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,887] jepsen worker 61 - jepsen.util 781 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,888] jepsen worker 61 - jepsen.util 781 :invoke :write [20 6] INFO [2022-03-13 17:18:56,897] jepsen worker 33 - jepsen.util 753 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,897] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,897] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,897] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,897] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:18:56,897] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,898] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:56,898] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:56,899] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:18:56,899] jepsen worker 34 - jepsen.util 934 :invoke :write [11 5] INFO [2022-03-13 17:18:56,913] jepsen worker 89 - jepsen.util 2789 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,913] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:56,935] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,936] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,937] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,938] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,943] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,943] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:18:56,950] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,950] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,950] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,951] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:56,952] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,952] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,952] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,953] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,953] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:18:56,953] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,953] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:56,954] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 35] INFO [2022-03-13 17:18:56,954] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:56,955] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 48] INFO [2022-03-13 17:18:57,003] jepsen worker 38 - jepsen.util 2828 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,003] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,004] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:18:57,004] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,006] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,006] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:18:57,010] jepsen worker 37 - jepsen.util 2467 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,010] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,020] jepsen worker 58 - jepsen.util 4108 :info :write [19 45] :RequestTimeoutException INFO [2022-03-13 17:18:57,020] jepsen worker 58 - jepsen.util 4198 :invoke :read [19 nil] INFO [2022-03-13 17:18:57,027] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,028] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:18:57,044] jepsen worker 8 - jepsen.util 728 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,045] jepsen worker 8 - jepsen.util 728 :invoke :write [2 20] INFO [2022-03-13 17:18:57,054] jepsen worker 86 - jepsen.util 4316 :info :write [28 6] :RequestTimeoutException INFO [2022-03-13 17:18:57,054] jepsen worker 86 - jepsen.util 4406 :invoke :read [28 nil] INFO [2022-03-13 17:18:57,068] jepsen worker 3 - jepsen.util 2253 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,069] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:18:57,102] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,103] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:18:57,111] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,112] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] WARN [2022-03-13 17:18:57,114] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":66250,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2471543,"total_server_duration_us":787773,"encode_duration_us":18924,"operation_id":"0x1281a","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10009391},{"operation_name":"upsert","last_dispatch_duration_us":6017279,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6788782,"total_server_duration_us":477148,"encode_duration_us":18310,"operation_id":"0x124af","timeout_ms":10000,"last_server_duration_us":33819,"total_duration_us":10007793},{"operation_name":"upsert","last_dispatch_duration_us":49185,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":5899919,"total_server_duration_us":370450,"encode_duration_us":21490,"operation_id":"0x1279d","timeout_ms":10000,"last_server_duration_us":33328,"total_duration_us":10005738},{"operation_name":"upsert","last_dispatch_duration_us":350140,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1789030,"total_server_duration_us":509120,"encode_duration_us":17278,"operation_id":"0x126cc","timeout_ms":10000,"last_server_duration_us":28768,"total_duration_us":10005652},{"operation_name":"upsert","last_dispatch_duration_us":6272626,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6851129,"total_server_duration_us":370443,"encode_duration_us":20316,"operation_id":"0x125a4","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10004794},{"operation_name":"upsert","last_dispatch_duration_us":9051368,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9051368,"total_server_duration_us":9040551,"encode_duration_us":18437,"operation_id":"0x125f5","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":9057067},{"operation_name":"upsert","last_dispatch_duration_us":3703457,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":4779939,"total_server_duration_us":719097,"encode_duration_us":33506,"operation_id":"0x12a16","timeout_ms":10000,"last_server_duration_us":491903,"total_duration_us":4790072},{"operation_name":"upsert","last_dispatch_duration_us":4657193,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":4657193,"total_server_duration_us":4011597,"encode_duration_us":19842,"operation_id":"0x1289b","timeout_ms":10000,"last_server_duration_us":4011597,"total_duration_us":4662135},{"operation_name":"get","last_dispatch_duration_us":4178583,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":4178583,"total_server_duration_us":32,"operation_id":"0x128b2","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":4178772},{"operation_name":"upsert","last_dispatch_duration_us":3961791,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":3961791,"total_server_duration_us":218840,"encode_duration_us":20177,"operation_id":"0x12a76","timeout_ms":10000,"last_server_duration_us":218840,"total_duration_us":3973560}],"total_count":67}} INFO [2022-03-13 17:18:57,161] jepsen worker 4 - jepsen.util 2614 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,161] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 48] INFO [2022-03-13 17:18:57,173] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,174] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,202] jepsen worker 88 - jepsen.util 2428 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,202] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,202] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 22] INFO [2022-03-13 17:18:57,203] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,221] jepsen worker 43 - jepsen.util 673 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,222] jepsen worker 43 - jepsen.util 673 :invoke :write [14 38] INFO [2022-03-13 17:18:57,227] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,227] jepsen worker 14 - jepsen.util 374 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,228] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,228] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,228] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,228] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:57,228] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,228] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,229] jepsen worker 21 - jepsen.util 831 :invoke :write [7 49] INFO [2022-03-13 17:18:57,229] jepsen worker 79 - jepsen.util 1159 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,229] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,230] jepsen worker 25 - jepsen.util 565 :invoke :write [8 17] INFO [2022-03-13 17:18:57,231] jepsen worker 33 - jepsen.util 753 :invoke :write [11 43] INFO [2022-03-13 17:18:57,231] jepsen worker 24 - jepsen.util 654 :invoke :write [8 1] INFO [2022-03-13 17:18:57,232] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:57,236] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 43] INFO [2022-03-13 17:18:57,237] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:57,238] jepsen worker 77 - jepsen.util 707 :invoke :write [25 27] INFO [2022-03-13 17:18:57,245] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,245] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:57,246] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,247] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:18:57,255] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,255] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:18:57,265] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,265] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,266] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:18:57,267] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,270] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,271] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:18:57,274] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,275] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,276] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,277] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 10] INFO [2022-03-13 17:18:57,321] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,321] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,323] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:57,324] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:18:57,372] jepsen worker 88 - jepsen.util 2428 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,373] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,384] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,384] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:18:57,410] jepsen worker 38 - jepsen.util 2828 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,411] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:18:57,432] jepsen worker 7 - jepsen.util 637 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,432] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:57,433] jepsen worker 76 - jepsen.util 796 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,434] jepsen worker 76 - jepsen.util 796 :invoke :write [25 41] INFO [2022-03-13 17:18:57,452] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,452] jepsen worker 34 - jepsen.util 934 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,453] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,453] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,453] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:57,454] jepsen worker 14 - jepsen.util 374 :invoke :write [4 8] INFO [2022-03-13 17:18:57,456] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:57,456] jepsen worker 80 - jepsen.util 800 :invoke :write [26 9] INFO [2022-03-13 17:18:57,460] jepsen worker 23 - jepsen.util 563 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,460] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:18:57,465] jepsen worker 4 - jepsen.util 2614 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,465] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:57,480] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,481] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,481] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,483] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:18:57,506] jepsen worker 10 - jepsen.util 4690 :info :write [3 24] :RequestTimeoutException INFO [2022-03-13 17:18:57,507] jepsen worker 10 - jepsen.util 4780 :invoke :read [3 nil] INFO [2022-03-13 17:18:57,533] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,534] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:18:57,538] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,538] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,540] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,540] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,542] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,542] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,543] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,544] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 40] INFO [2022-03-13 17:18:57,547] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,548] jepsen worker 10 - jepsen.util 4780 :ok :read [3 25] INFO [2022-03-13 17:18:57,548] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,548] jepsen worker 10 - jepsen.util 4780 :invoke :write [3 35] INFO [2022-03-13 17:18:57,549] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,549] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:18:57,560] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,560] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,560] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,561] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,561] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,562] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:18:57,563] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,563] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,563] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:18:57,564] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,565] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,565] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,566] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,567] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:18:57,611] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,611] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,611] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:57,612] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:18:57,633] jepsen worker 38 - jepsen.util 2828 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,634] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,636] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,637] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:57,662] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,662] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:18:57,693] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,694] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,694] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:18:57,695] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:18:57,718] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,718] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,719] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,719] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:18:57,721] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,721] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:18:57,745] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,746] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:57,757] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:57,757] jepsen worker nemesis - couchbase.util Rebalance check count: 644 INFO [2022-03-13 17:18:57,757] jepsen worker nemesis - couchbase.util Rebalance stuck count: 516 INFO [2022-03-13 17:18:57,812] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,812] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,813] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:57,813] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:18:57,828] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,829] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,843] jepsen worker 6 - jepsen.util 636 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,843] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,843] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,844] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:57,844] jepsen worker 12 - jepsen.util 912 :invoke :write [4 41] INFO [2022-03-13 17:18:57,846] jepsen worker 70 - jepsen.util 880 :invoke :write [23 35] INFO [2022-03-13 17:18:57,848] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,848] jepsen worker 60 - jepsen.util 690 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,849] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,849] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,849] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,850] jepsen worker 60 - jepsen.util 690 :invoke :write [20 20] INFO [2022-03-13 17:18:57,850] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:18:57,851] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:57,853] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,853] jepsen worker 52 - jepsen.util 682 :invoke :write [17 3] INFO [2022-03-13 17:18:57,880] jepsen worker 78 - jepsen.util 708 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,881] jepsen worker 78 - jepsen.util 708 :invoke :write [26 22] INFO [2022-03-13 17:18:57,910] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,911] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,911] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:18:57,911] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,912] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:57,912] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:57,914] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,915] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:57,920] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,920] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,920] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,921] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:18:57,922] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:18:57,923] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:57,926] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,927] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:18:57,987] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,988] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,988] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:18:57,989] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:57,990] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:18:58,002] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,005] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,005] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 46] INFO [2022-03-13 17:18:58,009] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,010] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,013] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,013] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:18:58,054] jepsen worker 21 - jepsen.util 831 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,055] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:18:58,061] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,061] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,069] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,069] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,070] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,071] jepsen worker 75 - jepsen.util 525 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,071] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,072] jepsen worker 75 - jepsen.util 525 :invoke :write [25 19] INFO [2022-03-13 17:18:58,073] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,073] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:18:58,100] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,101] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:58,102] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,103] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,126] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,126] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,153] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,153] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,153] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,153] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:18:58,154] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,154] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:18:58,162] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,162] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:18:58,166] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,167] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:18:58,172] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,173] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:58,179] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,183] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,185] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,186] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:18:58,200] jepsen worker 85 - jepsen.util 4045 :info :write [28 27] :RequestTimeoutException INFO [2022-03-13 17:18:58,201] jepsen worker 85 - jepsen.util 4135 :invoke :write [28 25] INFO [2022-03-13 17:18:58,220] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,221] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,250] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,250] jepsen worker 37 - jepsen.util 2467 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,251] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,251] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 28] INFO [2022-03-13 17:18:58,256] jepsen worker 22 - jepsen.util 652 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,257] jepsen worker 22 - jepsen.util 652 :invoke :write [7 23] INFO [2022-03-13 17:18:58,285] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,286] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,292] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,292] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 25] INFO [2022-03-13 17:18:58,306] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,307] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,307] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:18:58,307] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 23] INFO [2022-03-13 17:18:58,361] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,361] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:18:58,371] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,371] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,372] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,372] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:18:58,380] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,380] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,382] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,383] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,383] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,384] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,384] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:18:58,385] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,386] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,387] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,388] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,389] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,390] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,390] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,391] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,392] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:18:58,408] jepsen worker 88 - jepsen.util 2428 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,409] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 36] WARN [2022-03-13 17:18:58,414] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9250764,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16976851,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9367598,"encode_duration_us":33768,"operation_id":"0x12197","timeout_ms":10000,"last_server_duration_us":9073483,"total_duration_us":10000743},{"operation_name":"upsert","last_dispatch_duration_us":9068151,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9830751,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9471109,"encode_duration_us":18534,"operation_id":"0x125b9","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10003751},{"operation_name":"upsert","last_dispatch_duration_us":9051957,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12750054,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9527612,"encode_duration_us":17401,"operation_id":"0x12425","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10002472}],"total_count":3}} INFO [2022-03-13 17:18:58,459] jepsen worker 24 - jepsen.util 654 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,460] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:58,461] jepsen worker 42 - jepsen.util 402 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,461] jepsen worker 61 - jepsen.util 781 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,462] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:18:58,463] jepsen worker 61 - jepsen.util 781 :invoke :write [20 6] INFO [2022-03-13 17:18:58,470] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,470] jepsen worker 3 - jepsen.util 2253 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,471] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,471] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,484] jepsen worker 13 - jepsen.util 733 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,484] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:18:58,486] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,487] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:18:58,504] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,504] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,551] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,551] jepsen worker 33 - jepsen.util 753 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,552] jepsen worker 33 - jepsen.util 753 :invoke :write [11 46] INFO [2022-03-13 17:18:58,555] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:18:58,557] jepsen worker 43 - jepsen.util 673 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,557] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,557] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,558] jepsen worker 43 - jepsen.util 673 :invoke :write [14 39] INFO [2022-03-13 17:18:58,558] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:18:58,559] jepsen worker 26 - jepsen.util 566 :invoke :write [8 9] INFO [2022-03-13 17:18:58,569] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,569] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,569] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,570] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,570] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,571] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:18:58,573] jepsen worker 37 - jepsen.util 2467 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,574] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 26] INFO [2022-03-13 17:18:58,578] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,578] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,579] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,579] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:18:58,582] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,583] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,586] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,587] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,588] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,589] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,590] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,590] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:18:58,591] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,592] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:18:58,596] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,597] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,598] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,599] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,603] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,605] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,606] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,607] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,609] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,610] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] WARN [2022-03-13 17:18:58,617] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10938059,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10938059,"total_server_duration_us":7750,"operation_id":"0x127aa","timeout_ms":30000,"last_server_duration_us":7750,"total_duration_us":10940411},{"operation_name":"upsert","last_dispatch_duration_us":9153271,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9153271,"total_server_duration_us":43948,"encode_duration_us":21197,"operation_id":"0x128a4","timeout_ms":10000,"last_server_duration_us":43948,"total_duration_us":10008670},{"operation_name":"upsert","last_dispatch_duration_us":9446252,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10042799,"total_server_duration_us":9162826,"encode_duration_us":69698,"operation_id":"0x125a7","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10007252},{"operation_name":"upsert","last_dispatch_duration_us":8962276,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10705873,"total_server_duration_us":384546,"encode_duration_us":74118,"operation_id":"0x125a8","timeout_ms":10000,"last_server_duration_us":51946,"total_duration_us":10006990},{"operation_name":"upsert","last_dispatch_duration_us":8914837,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10410303,"total_server_duration_us":253797,"encode_duration_us":94186,"operation_id":"0x125b1","timeout_ms":10000,"last_server_duration_us":49255,"total_duration_us":10006722},{"operation_name":"upsert","last_dispatch_duration_us":50668,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":979319,"total_server_duration_us":249003,"encode_duration_us":22256,"operation_id":"0x1277e","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10005839},{"operation_name":"upsert","last_dispatch_duration_us":8983139,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10099785,"total_server_duration_us":216544,"encode_duration_us":22207,"operation_id":"0x1260e","timeout_ms":10000,"last_server_duration_us":53374,"total_duration_us":10005228},{"operation_name":"upsert","last_dispatch_duration_us":8846779,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10506201,"total_server_duration_us":237340,"encode_duration_us":74159,"operation_id":"0x125aa","timeout_ms":10000,"last_server_duration_us":65645,"total_duration_us":10003020},{"operation_name":"upsert","last_dispatch_duration_us":9143092,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9143092,"total_server_duration_us":9037366,"encode_duration_us":16948,"operation_id":"0x12672","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":9149681},{"operation_name":"upsert","last_dispatch_duration_us":3297492,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":4255782,"total_server_duration_us":336899,"encode_duration_us":19051,"operation_id":"0x12a79","timeout_ms":10000,"last_server_duration_us":318176,"total_duration_us":4267106}],"total_count":46}} WARN [2022-03-13 17:18:58,618] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":98642,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9541270,"total_server_duration_us":399217,"encode_duration_us":17052,"operation_id":"0x1267b","timeout_ms":10000,"last_server_duration_us":66694,"total_duration_us":10010486},{"operation_name":"upsert","last_dispatch_duration_us":97679,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":1789977,"total_server_duration_us":274866,"encode_duration_us":21259,"operation_id":"0x127c9","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10007521},{"operation_name":"upsert","last_dispatch_duration_us":8424120,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":15889079,"total_server_duration_us":290747,"encode_duration_us":21729,"operation_id":"0x12543","timeout_ms":10000,"last_server_duration_us":29415,"total_duration_us":10004720},{"operation_name":"upsert","last_dispatch_duration_us":8537934,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9378806,"total_server_duration_us":255342,"encode_duration_us":20420,"operation_id":"0x12879","timeout_ms":10000,"last_server_duration_us":42417,"total_duration_us":10001497},{"operation_name":"upsert","last_dispatch_duration_us":9249496,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10085732,"total_server_duration_us":289491,"encode_duration_us":42747,"operation_id":"0x125a1","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10001479},{"operation_name":"upsert","last_dispatch_duration_us":9294399,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9495913,"total_server_duration_us":9146816,"encode_duration_us":16853,"operation_id":"0x12604","timeout_ms":10000,"last_server_duration_us":9077736,"total_duration_us":9514674},{"operation_name":"upsert","last_dispatch_duration_us":9103903,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9103903,"total_server_duration_us":9034183,"encode_duration_us":17684,"operation_id":"0x128a5","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":9107520},{"operation_name":"upsert","last_dispatch_duration_us":4454546,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":4454546,"total_server_duration_us":4040946,"encode_duration_us":32305,"operation_id":"0x128a0","timeout_ms":10000,"last_server_duration_us":4040946,"total_duration_us":4466052},{"operation_name":"get","last_dispatch_duration_us":4234046,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":4234046,"total_server_duration_us":56,"operation_id":"0x128ad","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":4234297},{"operation_name":"upsert","last_dispatch_duration_us":4110287,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":4110287,"total_server_duration_us":24069,"encode_duration_us":17386,"operation_id":"0x128b6","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":4115208}],"total_count":30}} INFO [2022-03-13 17:18:58,620] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,621] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:18:58,660] jepsen worker 44 - jepsen.util 764 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,660] jepsen worker 44 - jepsen.util 764 :invoke :write [14 7] INFO [2022-03-13 17:18:58,679] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,680] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:18:58,685] jepsen worker 79 - jepsen.util 1159 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,686] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 40] WARN [2022-03-13 17:18:58,714] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":407649,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":2748905,"total_server_duration_us":802968,"encode_duration_us":31261,"operation_id":"0x1279e","timeout_ms":10000,"last_server_duration_us":35513,"total_duration_us":10010523},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":22085,"operation_id":"0x127df","timeout_ms":10000,"total_duration_us":10005907},{"operation_name":"upsert","last_dispatch_duration_us":151151,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1749685,"total_server_duration_us":550168,"encode_duration_us":17992,"operation_id":"0x128c0","timeout_ms":10000,"last_server_duration_us":46965,"total_duration_us":10005534},{"operation_name":"upsert","last_dispatch_duration_us":9178948,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":11234813,"total_server_duration_us":9200395,"encode_duration_us":19429,"operation_id":"0x12612","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10004568},{"operation_name":"upsert","last_dispatch_duration_us":8777794,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9050887,"total_server_duration_us":81110,"encode_duration_us":17156,"operation_id":"0x126b9","timeout_ms":10000,"last_server_duration_us":22306,"total_duration_us":10004547},{"operation_name":"upsert","last_dispatch_duration_us":8692912,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10957837,"total_server_duration_us":273278,"encode_duration_us":17647,"operation_id":"0x1260f","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10002417},{"operation_name":"upsert","last_dispatch_duration_us":8774838,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9841516,"total_server_duration_us":243039,"encode_duration_us":26428,"operation_id":"0x126c5","timeout_ms":10000,"last_server_duration_us":22471,"total_duration_us":10002060},{"operation_name":"upsert","last_dispatch_duration_us":8324071,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10851625,"total_server_duration_us":260216,"encode_duration_us":28054,"operation_id":"0x125e1","timeout_ms":10000,"last_server_duration_us":20282,"total_duration_us":10001978},{"operation_name":"upsert","last_dispatch_duration_us":4105436,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":4105436,"total_server_duration_us":35011,"encode_duration_us":18304,"operation_id":"0x128c9","timeout_ms":10000,"last_server_duration_us":35011,"total_duration_us":4108846},{"operation_name":"upsert","last_dispatch_duration_us":1937920,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":4017847,"total_server_duration_us":269887,"encode_duration_us":38240,"operation_id":"0x12a17","timeout_ms":10000,"last_server_duration_us":202744,"total_duration_us":4047046}],"total_count":32}} WARN [2022-03-13 17:18:58,715] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":704111,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9758372,"total_server_duration_us":356282,"encode_duration_us":18345,"operation_id":"0x12661","timeout_ms":10000,"last_server_duration_us":58874,"total_duration_us":10008853},{"operation_name":"upsert","last_dispatch_duration_us":703062,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9609887,"total_server_duration_us":210539,"encode_duration_us":26868,"operation_id":"0x1274a","timeout_ms":10000,"last_server_duration_us":45167,"total_duration_us":10008308},{"operation_name":"upsert","last_dispatch_duration_us":8564986,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9476320,"total_server_duration_us":571610,"encode_duration_us":21227,"operation_id":"0x125ef","timeout_ms":10000,"last_server_duration_us":67353,"total_duration_us":10006208},{"operation_name":"upsert","last_dispatch_duration_us":465091,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9520160,"total_server_duration_us":322600,"encode_duration_us":21603,"operation_id":"0x1279f","timeout_ms":10000,"last_server_duration_us":26685,"total_duration_us":10004936},{"operation_name":"upsert","last_dispatch_duration_us":9068151,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9830751,"total_server_duration_us":9471109,"encode_duration_us":18534,"operation_id":"0x125b9","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10003751},{"operation_name":"get","last_dispatch_duration_us":8867800,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8867800,"total_server_duration_us":9284,"operation_id":"0x12875","timeout_ms":30000,"last_server_duration_us":9284,"total_duration_us":8869638},{"operation_name":"get","last_dispatch_duration_us":7193525,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":7193525,"total_server_duration_us":32,"operation_id":"0x128f8","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":7195569},{"operation_name":"upsert","last_dispatch_duration_us":2065086,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3526624,"total_server_duration_us":506098,"encode_duration_us":18101,"operation_id":"0x12a1f","timeout_ms":10000,"last_server_duration_us":408967,"total_duration_us":3557417},{"operation_name":"upsert","last_dispatch_duration_us":1522034,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":3430933,"total_server_duration_us":376651,"encode_duration_us":29351,"operation_id":"0x129c3","timeout_ms":10000,"last_server_duration_us":330291,"total_duration_us":3463788},{"operation_name":"upsert","last_dispatch_duration_us":2665788,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2665788,"total_server_duration_us":198750,"encode_duration_us":17538,"operation_id":"0x12a87","timeout_ms":10000,"last_server_duration_us":198750,"total_duration_us":2671576}],"total_count":41}} WARN [2022-03-13 17:18:58,717] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16134596,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16938591,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9318741,"encode_duration_us":20408,"operation_id":"0x12426","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10000944},{"operation_name":"upsert","last_dispatch_duration_us":9446252,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10042799,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9162826,"encode_duration_us":69698,"operation_id":"0x125a7","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10007252}],"total_count":2}} INFO [2022-03-13 17:18:58,742] jepsen worker 88 - jepsen.util 2428 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,742] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 11] INFO [2022-03-13 17:18:58,745] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,745] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,758] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,758] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:18:58,763] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:58,763] jepsen worker nemesis - couchbase.util Rebalance check count: 645 INFO [2022-03-13 17:18:58,763] jepsen worker nemesis - couchbase.util Rebalance stuck count: 517 WARN [2022-03-13 17:18:58,815] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9178948,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11234813,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9200395,"encode_duration_us":19429,"operation_id":"0x12612","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10004568},{"operation_name":"upsert","last_dispatch_duration_us":9052984,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16078813,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9334189,"encode_duration_us":18589,"operation_id":"0x12354","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10008963}],"total_count":2}} INFO [2022-03-13 17:18:58,816] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,816] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:18:58,818] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,818] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,819] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:18:58,819] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 4] INFO [2022-03-13 17:18:58,824] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,825] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] WARN [2022-03-13 17:18:58,831] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":217964,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":217964,"total_server_duration_us":30823,"encode_duration_us":18368,"operation_id":"0x127b2","timeout_ms":10000,"last_server_duration_us":30823,"total_duration_us":10010689},{"operation_name":"upsert","last_dispatch_duration_us":29401,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":657317,"total_server_duration_us":395608,"encode_duration_us":21575,"operation_id":"0x128a6","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10004779},{"operation_name":"upsert","last_dispatch_duration_us":8432782,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10180400,"total_server_duration_us":439336,"encode_duration_us":21484,"operation_id":"0x126e7","timeout_ms":10000,"last_server_duration_us":92438,"total_duration_us":10004400},{"operation_name":"upsert","last_dispatch_duration_us":30289,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":745498,"total_server_duration_us":427513,"encode_duration_us":20190,"operation_id":"0x1274f","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10004042},{"operation_name":"upsert","last_dispatch_duration_us":8698838,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10596333,"total_server_duration_us":328317,"encode_duration_us":18960,"operation_id":"0x126e8","timeout_ms":10000,"last_server_duration_us":62412,"total_duration_us":10003791},{"operation_name":"upsert","last_dispatch_duration_us":806952,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9028718,"total_server_duration_us":147278,"encode_duration_us":24217,"operation_id":"0x127db","timeout_ms":10000,"last_server_duration_us":25101,"total_duration_us":10003552},{"operation_name":"upsert","last_dispatch_duration_us":8605571,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10175484,"total_server_duration_us":429217,"encode_duration_us":20807,"operation_id":"0x126e6","timeout_ms":10000,"last_server_duration_us":148276,"total_duration_us":10002897},{"operation_name":"upsert","last_dispatch_duration_us":9277629,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9829179,"total_server_duration_us":9176947,"encode_duration_us":19333,"operation_id":"0x126e2","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":9883484},{"operation_name":"upsert","last_dispatch_duration_us":2955665,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4169200,"total_server_duration_us":502985,"encode_duration_us":17405,"operation_id":"0x12a2f","timeout_ms":10000,"last_server_duration_us":355632,"total_duration_us":5773423},{"operation_name":"upsert","last_dispatch_duration_us":3152706,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":4922055,"total_server_duration_us":962308,"encode_duration_us":18000,"operation_id":"0x12a46","timeout_ms":10000,"last_server_duration_us":152740,"total_duration_us":4980981}],"total_count":39}} INFO [2022-03-13 17:18:58,880] jepsen worker 25 - jepsen.util 565 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,880] jepsen worker 76 - jepsen.util 796 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,881] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,881] jepsen worker 76 - jepsen.util 796 :invoke :write [25 0] INFO [2022-03-13 17:18:58,881] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,881] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:18:58,881] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:58,882] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:18:58,905] jepsen worker 89 - jepsen.util 2789 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,906] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:58,918] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,919] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:18:58,929] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,930] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:58,931] jepsen worker 62 - jepsen.util 512 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,933] jepsen worker 62 - jepsen.util 512 :invoke :write [20 5] INFO [2022-03-13 17:18:58,992] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,992] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,993] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:18:58,994] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,996] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,996] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,997] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:58,998] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:58,999] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,000] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:18:59,037] jepsen worker 37 - jepsen.util 2467 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,038] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,053] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,053] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,061] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,062] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,077] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,077] jepsen worker 52 - jepsen.util 682 :fail :write [17 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,078] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:59,079] jepsen worker 52 - jepsen.util 682 :invoke :write [17 24] INFO [2022-03-13 17:18:59,079] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,080] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:18:59,135] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,136] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:18:59,152] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,152] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,152] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 40] INFO [2022-03-13 17:18:59,153] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:18:59,169] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,169] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,169] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,169] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,171] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:18:59,171] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:18:59,175] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,176] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,183] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,183] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:18:59,238] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,238] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:18:59,262] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,263] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:18:59,274] jepsen worker 1 - jepsen.util 4051 :info :write [0 5] :SyncWriteAmbiguous INFO [2022-03-13 17:18:59,274] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,275] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:18:59,275] jepsen worker 1 - jepsen.util 4141 :invoke :write [0 20] INFO [2022-03-13 17:18:59,288] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,288] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,289] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:59,289] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:18:59,293] jepsen worker 80 - jepsen.util 800 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,294] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:59,297] jepsen worker 75 - jepsen.util 525 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,297] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,298] jepsen worker 75 - jepsen.util 525 :invoke :write [25 33] INFO [2022-03-13 17:18:59,298] jepsen worker 13 - jepsen.util 733 :invoke :write [4 40] INFO [2022-03-13 17:18:59,333] jepsen worker 88 - jepsen.util 2428 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,333] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,333] jepsen worker 60 - jepsen.util 690 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,334] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 18] INFO [2022-03-13 17:18:59,335] jepsen worker 60 - jepsen.util 690 :invoke :write [20 3] INFO [2022-03-13 17:18:59,335] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,336] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,337] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,347] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,348] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:18:59,365] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,366] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,373] jepsen worker 38 - jepsen.util 2828 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,373] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,443] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,443] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,450] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,451] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:18:59,461] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,461] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,462] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,463] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:18:59,470] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,471] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,472] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,473] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,475] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,475] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,477] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,479] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:18:59,484] jepsen worker 77 - jepsen.util 707 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,485] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,485] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,485] jepsen worker 77 - jepsen.util 707 :invoke :write [25 46] INFO [2022-03-13 17:18:59,487] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:18:59,488] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,489] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:18:59,490] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:18:59,494] jepsen worker 14 - jepsen.util 374 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,495] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,494] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,495] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,495] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:18:59,495] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:59,496] jepsen worker 35 - jepsen.util 935 :invoke :write [11 8] INFO [2022-03-13 17:18:59,497] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:18:59,537] jepsen worker 88 - jepsen.util 2428 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,537] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,547] jepsen worker 76 - jepsen.util 796 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,547] jepsen worker 76 - jepsen.util 796 :invoke :write [25 23] INFO [2022-03-13 17:18:59,574] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,574] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,575] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,576] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,601] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,601] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,602] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,602] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,649] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,650] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:18:59,686] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,687] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,689] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,689] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,689] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,690] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:18:59,690] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,691] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:18:59,692] jepsen worker 33 - jepsen.util 753 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,692] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,692] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,692] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,693] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,693] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,695] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:18:59,696] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,696] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:18:59,697] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:18:59,698] jepsen worker 14 - jepsen.util 374 :invoke :write [4 6] INFO [2022-03-13 17:18:59,699] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:18:59,700] jepsen worker 33 - jepsen.util 753 :invoke :write [11 49] INFO [2022-03-13 17:18:59,703] jepsen worker 34 - jepsen.util 934 :invoke :write [11 29] INFO [2022-03-13 17:18:59,703] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,706] jepsen worker 8 - jepsen.util 728 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,707] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,707] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,707] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:18:59,708] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:18:59,708] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,709] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:18:59,727] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,727] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,731] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,732] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:18:59,741] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,741] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,741] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,742] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:18:59,743] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,744] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 40] INFO [2022-03-13 17:18:59,744] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:18:59,746] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,748] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,748] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,750] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,750] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:18:59,769] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:18:59,769] jepsen worker nemesis - couchbase.util Rebalance check count: 646 INFO [2022-03-13 17:18:59,769] jepsen worker nemesis - couchbase.util Rebalance stuck count: 518 INFO [2022-03-13 17:18:59,805] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,805] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,805] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,806] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,825] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,825] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,826] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:18:59,826] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:18:59,877] jepsen worker 88 - jepsen.util 2428 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,878] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 35] INFO [2022-03-13 17:18:59,883] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,883] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,883] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,884] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,884] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,884] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:18:59,886] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,886] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,886] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,887] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:18:59,887] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 6] INFO [2022-03-13 17:18:59,888] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:18:59,899] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,900] jepsen worker 21 - jepsen.util 831 :invoke :write [7 24] INFO [2022-03-13 17:18:59,911] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,911] jepsen worker 22 - jepsen.util 652 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,912] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,912] jepsen worker 53 - jepsen.util 593 :invoke :write [17 20] INFO [2022-03-13 17:18:59,913] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,914] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:18:59,915] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,916] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:18:59,955] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,956] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:18:59,983] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,984] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:18:59,986] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:18:59,986] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:19:00,100] jepsen worker 12 - jepsen.util 912 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,102] jepsen worker 12 - jepsen.util 912 :invoke :write [4 7] INFO [2022-03-13 17:19:00,103] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,103] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,103] jepsen worker 35 - jepsen.util 935 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,195] jepsen worker 80 - jepsen.util 800 :invoke :write [26 14] INFO [2022-03-13 17:19:00,195] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:00,104] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,195] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:00,118] jepsen worker 13 - jepsen.util 733 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,138] jepsen worker 14 - jepsen.util 374 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,195] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:00,195] jepsen worker 51 - jepsen.util 771 :invoke :write [17 43] INFO [2022-03-13 17:19:00,195] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:00,304] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,304] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,304] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,305] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,306] jepsen worker 78 - jepsen.util 708 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,307] jepsen worker 78 - jepsen.util 708 :invoke :write [26 47] INFO [2022-03-13 17:19:00,316] jepsen worker 43 - jepsen.util 673 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,316] jepsen worker 43 - jepsen.util 673 :invoke :write [14 46] INFO [2022-03-13 17:19:00,338] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,338] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,339] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:19:00,341] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 41] INFO [2022-03-13 17:19:00,347] jepsen worker 36 - jepsen.util 2466 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,347] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,363] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,363] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:19:00,364] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,364] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,365] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,365] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:19:00,371] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,372] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,373] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,374] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,375] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,375] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,377] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,377] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:19:00,443] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,443] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 44] INFO [2022-03-13 17:19:00,448] jepsen worker 38 - jepsen.util 2828 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,448] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,450] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,450] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,537] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,537] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:00,538] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,538] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,539] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:19:00,540] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,543] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,543] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,543] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:19:00,544] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,546] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,547] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,548] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:19:00,548] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,552] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,553] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,554] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,555] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,556] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,556] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 40] INFO [2022-03-13 17:19:00,617] jepsen worker 70 - jepsen.util 880 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,617] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:00,617] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,618] jepsen worker 42 - jepsen.util 402 :invoke :write [14 20] INFO [2022-03-13 17:19:00,622] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,622] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:19:00,623] jepsen worker 75 - jepsen.util 525 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,623] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:00,639] jepsen worker 61 - jepsen.util 781 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,640] jepsen worker 61 - jepsen.util 781 :invoke :write [20 38] INFO [2022-03-13 17:19:00,660] jepsen worker 4 - jepsen.util 2614 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,660] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,661] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,661] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 7] INFO [2022-03-13 17:19:00,680] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,680] jepsen worker 88 - jepsen.util 2428 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,681] jepsen worker 38 - jepsen.util 2828 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,681] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,682] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,683] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:19:00,698] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,698] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,726] jepsen worker 20 - jepsen.util 4700 :info :write [6 29] :RequestTimeoutException INFO [2022-03-13 17:19:00,726] jepsen worker 20 - jepsen.util 4790 :invoke :read [6 nil] INFO [2022-03-13 17:19:00,737] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,738] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:19:00,739] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,740] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,746] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,746] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,747] jepsen worker 18 - jepsen.util 4338 :info :write [6 47] :RequestTimeoutException INFO [2022-03-13 17:19:00,748] jepsen worker 18 - jepsen.util 4428 :invoke :write [6 23] INFO [2022-03-13 17:19:00,750] jepsen worker 27 - jepsen.util 4617 :info :write [9 23] :RequestTimeoutException INFO [2022-03-13 17:19:00,750] jepsen worker 19 - jepsen.util 4249 :info :write [6 23] :RequestTimeoutException INFO [2022-03-13 17:19:00,750] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,751] jepsen worker 27 - jepsen.util 4707 :invoke :write [9 21] INFO [2022-03-13 17:19:00,752] jepsen worker 19 - jepsen.util 4339 :invoke :read [6 nil] INFO [2022-03-13 17:19:00,754] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,755] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,756] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 48] INFO [2022-03-13 17:19:00,763] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,763] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,764] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,765] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,769] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,771] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,775] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,775] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,776] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:19:00,776] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,780] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,781] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:00,781] jepsen worker nemesis - couchbase.util Rebalance check count: 647 INFO [2022-03-13 17:19:00,781] jepsen worker nemesis - couchbase.util Rebalance stuck count: 519 INFO [2022-03-13 17:19:00,782] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,782] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,783] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,783] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:19:00,785] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,787] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,788] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,790] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,791] jepsen worker 19 - jepsen.util 4339 :ok :read [6 16] INFO [2022-03-13 17:19:00,791] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:19:00,791] jepsen worker 19 - jepsen.util 4339 :invoke :write [6 27] INFO [2022-03-13 17:19:00,806] jepsen worker 39 - jepsen.util 4719 :info :write [13 42] :RequestTimeoutException INFO [2022-03-13 17:19:00,806] jepsen worker 39 - jepsen.util 4809 :invoke :read [13 nil] INFO [2022-03-13 17:19:00,822] jepsen worker 26 - jepsen.util 566 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,823] jepsen worker 26 - jepsen.util 566 :invoke :write [8 7] INFO [2022-03-13 17:19:00,828] jepsen worker 44 - jepsen.util 764 :fail :write [14 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,828] jepsen worker 44 - jepsen.util 764 :invoke :write [14 15] INFO [2022-03-13 17:19:00,832] jepsen worker 33 - jepsen.util 753 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,833] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:00,836] jepsen worker 45 - jepsen.util 4725 :info :write [15 49] :RequestTimeoutException INFO [2022-03-13 17:19:00,837] jepsen worker 45 - jepsen.util 4815 :invoke :read [15 nil] INFO [2022-03-13 17:19:00,846] jepsen worker 36 - jepsen.util 2466 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,847] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,848] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,848] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,851] jepsen worker 39 - jepsen.util 4809 :ok :read [13 39] INFO [2022-03-13 17:19:00,852] jepsen worker 39 - jepsen.util 4809 :invoke :read [13 nil] INFO [2022-03-13 17:19:00,865] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,865] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,866] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:00,867] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:19:00,893] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,893] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:00,897] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,898] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:00,911] jepsen worker 39 - jepsen.util 4809 :ok :read [13 39] INFO [2022-03-13 17:19:00,912] jepsen worker 39 - jepsen.util 4809 :invoke :write [13 9] INFO [2022-03-13 17:19:00,957] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,957] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,958] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,958] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:19:00,961] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,962] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,964] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,964] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,965] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,965] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,965] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,965] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,966] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,966] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:19:00,967] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:19:00,967] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,968] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,969] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,969] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,970] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:00,970] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,971] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,971] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:00,972] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 5] INFO [2022-03-13 17:19:00,972] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,973] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:19:00,974] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,975] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:19:00,978] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:00,979] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,025] jepsen worker 76 - jepsen.util 796 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,027] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:01,028] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,028] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,028] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:01,029] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:01,031] jepsen worker 79 - jepsen.util 1159 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,031] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,033] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:01,033] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:19:01,038] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,038] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,038] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,039] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:01,040] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:01,041] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,041] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,042] jepsen worker 13 - jepsen.util 733 :invoke :write [4 27] INFO [2022-03-13 17:19:01,042] jepsen worker 76 - jepsen.util 796 :invoke :write [25 13] INFO [2022-03-13 17:19:01,042] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:01,046] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,047] jepsen worker 75 - jepsen.util 525 :invoke :write [25 33] INFO [2022-03-13 17:19:01,053] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,054] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,055] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,056] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,094] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,095] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,113] jepsen worker 37 - jepsen.util 2467 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,113] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,113] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,113] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,114] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,114] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,114] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,115] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:19:01,130] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,130] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,130] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,131] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,132] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:19:01,132] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 25] INFO [2022-03-13 17:19:01,132] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,133] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,140] jepsen worker 84 - jepsen.util 4764 :info :write [28 23] :RequestTimeoutException INFO [2022-03-13 17:19:01,141] jepsen worker 84 - jepsen.util 4854 :invoke :write [28 39] INFO [2022-03-13 17:19:01,142] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,143] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:19:01,205] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,205] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,208] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,208] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:19:01,217] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,218] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,222] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,223] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,224] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,225] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,229] jepsen worker 34 - jepsen.util 934 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,229] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,230] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:01,230] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,232] jepsen worker 52 - jepsen.util 682 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,233] jepsen worker 52 - jepsen.util 682 :invoke :write [17 20] INFO [2022-03-13 17:19:01,234] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,235] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,237] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,237] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:19:01,264] jepsen worker 38 - jepsen.util 2828 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,265] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,276] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,276] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,277] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:19:01,278] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,283] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,283] jepsen worker 62 - jepsen.util 512 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,283] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,284] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,284] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:01,285] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:01,285] jepsen worker 7 - jepsen.util 637 :invoke :write [2 38] INFO [2022-03-13 17:19:01,286] jepsen worker 6 - jepsen.util 636 :invoke :write [2 48] INFO [2022-03-13 17:19:01,353] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,353] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,354] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,354] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,355] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,355] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,356] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,356] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,362] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,362] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,362] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,363] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:19:01,364] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,365] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:19:01,386] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,386] jepsen worker 88 - jepsen.util 2428 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,387] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 11] INFO [2022-03-13 17:19:01,387] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:19:01,414] jepsen worker 47 - jepsen.util 4367 :info :write [15 21] :RequestTimeoutException INFO [2022-03-13 17:19:01,415] jepsen worker 47 - jepsen.util 4457 :invoke :read [15 nil] INFO [2022-03-13 17:19:01,437] jepsen worker 77 - jepsen.util 707 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,437] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,438] jepsen worker 77 - jepsen.util 707 :invoke :write [25 43] INFO [2022-03-13 17:19:01,438] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,439] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,439] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,439] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:01,438] jepsen worker 21 - jepsen.util 831 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,439] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:01,440] jepsen worker 24 - jepsen.util 654 :invoke :write [8 6] INFO [2022-03-13 17:19:01,441] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:19:01,442] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:19:01,444] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,444] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,444] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:19:01,445] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:19:01,450] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,456] jepsen worker 59 - jepsen.util 4469 :info :write [19 35] :RequestTimeoutException INFO [2022-03-13 17:19:01,457] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,459] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:19:01,459] jepsen worker 59 - jepsen.util 4559 :invoke :write [19 30] INFO [2022-03-13 17:19:01,462] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,463] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,464] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,464] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,464] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,465] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:19:01,470] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,477] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,479] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:01,486] jepsen worker 81 - jepsen.util 5031 :info :write [27 25] :RequestTimeoutException INFO [2022-03-13 17:19:01,487] jepsen worker 81 - jepsen.util 5121 :invoke :read [27 nil] INFO [2022-03-13 17:19:01,505] jepsen worker 80 - jepsen.util 800 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,507] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:01,545] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,546] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 18] INFO [2022-03-13 17:19:01,571] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,571] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,572] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:19:01,573] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,581] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,582] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:19:01,588] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,589] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,627] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,627] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,630] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,630] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,631] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,632] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,641] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,642] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:01,643] jepsen worker 60 - jepsen.util 690 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,644] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:01,646] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,646] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:19:01,647] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,648] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:19:01,675] jepsen worker 88 - jepsen.util 2428 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,676] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 20] INFO [2022-03-13 17:19:01,685] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,686] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 48] INFO [2022-03-13 17:19:01,688] jepsen worker 51 - jepsen.util 771 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,688] jepsen worker 51 - jepsen.util 771 :invoke :write [17 34] INFO [2022-03-13 17:19:01,709] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,709] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,710] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,710] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:01,711] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:19:01,711] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:19:01,715] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,715] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,715] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,716] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:01,718] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,719] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:01,720] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,722] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:19:01,786] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:01,786] jepsen worker nemesis - couchbase.util Rebalance check count: 648 INFO [2022-03-13 17:19:01,786] jepsen worker nemesis - couchbase.util Rebalance stuck count: 520 INFO [2022-03-13 17:19:01,792] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,793] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 5] INFO [2022-03-13 17:19:01,815] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,815] jepsen worker 37 - jepsen.util 2467 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,816] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:19:01,816] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,838] jepsen worker 54 - jepsen.util 4194 :info :write [18 17] :RequestTimeoutException INFO [2022-03-13 17:19:01,839] jepsen worker 54 - jepsen.util 4284 :invoke :read [18 nil] INFO [2022-03-13 17:19:01,844] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,844] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:19:01,846] jepsen worker 12 - jepsen.util 912 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,846] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,847] jepsen worker 12 - jepsen.util 912 :invoke :write [4 35] INFO [2022-03-13 17:19:01,847] jepsen worker 70 - jepsen.util 880 :invoke :write [23 21] INFO [2022-03-13 17:19:01,851] jepsen worker 13 - jepsen.util 733 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,851] jepsen worker 30 - jepsen.util 4440 :info :write [10 26] :RequestTimeoutException INFO [2022-03-13 17:19:01,852] jepsen worker 30 - jepsen.util 4530 :invoke :read [10 nil] INFO [2022-03-13 17:19:01,853] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:01,875] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,876] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:01,879] jepsen worker 89 - jepsen.util 2789 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,880] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:19:01,900] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,900] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:01,946] jepsen worker 38 - jepsen.util 2828 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,947] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:19:01,950] jepsen worker 31 - jepsen.util 4531 :info :write [10 44] :RequestTimeoutException INFO [2022-03-13 17:19:01,951] jepsen worker 31 - jepsen.util 4621 :invoke :write [10 6] INFO [2022-03-13 17:19:01,960] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:01,961] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,018] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,018] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:19:02,022] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,022] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,022] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,023] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:02,024] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,024] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:02,024] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,025] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,025] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:02,025] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:19:02,027] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,027] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:19:02,052] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,052] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,052] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:19:02,053] jepsen worker 71 - jepsen.util 611 :invoke :write [23 21] INFO [2022-03-13 17:19:02,056] jepsen worker 78 - jepsen.util 708 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,057] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:02,062] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,062] jepsen worker 88 - jepsen.util 2428 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,062] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,064] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,064] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,064] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,065] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,065] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,065] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,066] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:19:02,066] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,067] jepsen worker 34 - jepsen.util 934 :invoke :write [11 11] INFO [2022-03-13 17:19:02,068] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:02,068] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,069] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:02,069] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:19:02,070] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:19:02,074] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:02,075] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,076] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:02,078] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,079] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:19:02,114] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,114] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:02,160] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,161] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 33] INFO [2022-03-13 17:19:02,185] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,186] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:19:02,195] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,197] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:02,214] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,214] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,215] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:02,216] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:02,216] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,217] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:19:02,219] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,219] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:02,221] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,221] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:19:02,244] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,245] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,253] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,254] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 22] INFO [2022-03-13 17:19:02,277] jepsen worker 53 - jepsen.util 593 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,277] jepsen worker 75 - jepsen.util 525 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,277] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:02,279] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:02,279] jepsen worker 42 - jepsen.util 402 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,280] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:02,287] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,288] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:02,315] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,315] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,316] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:19:02,316] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:19:02,361] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,362] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:19:02,365] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,366] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:02,420] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,421] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 33] INFO [2022-03-13 17:19:02,435] jepsen worker 4 - jepsen.util 2614 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,435] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,435] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,436] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:19:02,437] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,437] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:02,438] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,439] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:19:02,447] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,448] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:19:02,478] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,479] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:02,479] jepsen worker 61 - jepsen.util 781 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,480] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,480] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,480] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,482] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:02,483] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:19:02,484] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 43] INFO [2022-03-13 17:19:02,485] jepsen worker 23 - jepsen.util 563 :invoke :write [7 23] INFO [2022-03-13 17:19:02,499] jepsen worker 71 - jepsen.util 611 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,499] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,500] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:19:02,500] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:02,502] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,502] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:19:02,522] jepsen worker 44 - jepsen.util 764 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,523] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,523] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,523] jepsen worker 44 - jepsen.util 764 :invoke :write [14 26] INFO [2022-03-13 17:19:02,524] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:02,530] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:02,543] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,544] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:02,554] jepsen worker 77 - jepsen.util 707 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,554] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,555] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,556] jepsen worker 77 - jepsen.util 707 :invoke :write [25 34] INFO [2022-03-13 17:19:02,556] jepsen worker 61 - jepsen.util 781 :invoke :write [20 41] INFO [2022-03-13 17:19:02,557] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:02,591] jepsen worker 7 - jepsen.util 637 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,592] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:02,595] jepsen worker 88 - jepsen.util 2428 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,595] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:02,601] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,602] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 7] INFO [2022-03-13 17:19:02,629] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,629] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:02,632] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,633] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:19:02,642] jepsen worker 55 - jepsen.util 4105 :ok :read [18 21] INFO [2022-03-13 17:19:02,642] jepsen worker 55 - jepsen.util 4105 :invoke :read [18 nil] INFO [2022-03-13 17:19:02,662] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,663] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:02,679] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,680] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:02,684] jepsen worker 79 - jepsen.util 1159 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,684] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:02,691] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,691] jepsen worker 24 - jepsen.util 654 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,692] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:02,693] jepsen worker 80 - jepsen.util 800 :invoke :write [26 33] INFO [2022-03-13 17:19:02,712] jepsen worker 16 - jepsen.util 4336 :ok :read [5 43] INFO [2022-03-13 17:19:02,712] jepsen worker 16 - jepsen.util 4336 :invoke :write [5 12] INFO [2022-03-13 17:19:02,742] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,743] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:02,746] jepsen worker 6 - jepsen.util 636 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,747] jepsen worker 6 - jepsen.util 636 :invoke :write [2 41] INFO [2022-03-13 17:19:02,754] jepsen worker 37 - jepsen.util 2467 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,754] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,754] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:19:02,755] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 19] INFO [2022-03-13 17:19:02,774] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,774] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:19:02,777] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,778] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:19:02,783] jepsen worker 23 - jepsen.util 563 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,783] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,784] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:19:02,785] jepsen worker 78 - jepsen.util 708 :invoke :write [26 29] INFO [2022-03-13 17:19:02,791] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:02,792] jepsen worker nemesis - couchbase.util Rebalance check count: 649 INFO [2022-03-13 17:19:02,792] jepsen worker nemesis - couchbase.util Rebalance stuck count: 521 INFO [2022-03-13 17:19:02,804] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,804] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,806] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:19:02,807] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:02,808] jepsen worker 76 - jepsen.util 796 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,808] jepsen worker 43 - jepsen.util 673 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,808] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,808] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,808] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,809] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:02,812] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:02,812] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:02,813] jepsen worker 43 - jepsen.util 673 :invoke :write [14 30] INFO [2022-03-13 17:19:02,815] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:19:02,839] jepsen worker 4 - jepsen.util 2614 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,840] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,859] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,860] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:19:02,879] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,880] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:19:02,920] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,920] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,921] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:19:02,921] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 6] INFO [2022-03-13 17:19:02,935] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,935] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:02,949] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,949] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:02,988] jepsen worker 36 - jepsen.util 2466 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:02,989] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,010] jepsen worker 51 - jepsen.util 771 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,010] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,010] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,011] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,011] jepsen worker 51 - jepsen.util 771 :invoke :write [17 33] INFO [2022-03-13 17:19:03,011] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,011] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:03,012] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:19:03,013] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:03,013] jepsen worker 25 - jepsen.util 565 :invoke :write [8 20] INFO [2022-03-13 17:19:03,017] jepsen worker 65 - jepsen.util 4745 :info :write [21 2] :SyncWriteAmbiguous INFO [2022-03-13 17:19:03,017] jepsen worker 65 - jepsen.util 4835 :invoke :read [21 nil] INFO [2022-03-13 17:19:03,019] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,020] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,020] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,020] jepsen worker 76 - jepsen.util 796 :invoke :write [25 28] INFO [2022-03-13 17:19:03,023] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:03,024] jepsen worker 13 - jepsen.util 733 :invoke :write [4 31] INFO [2022-03-13 17:19:03,030] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,031] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:03,032] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,032] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,032] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,033] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,033] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,034] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:19:03,034] jepsen worker 17 - jepsen.util 4697 :info :write [5 19] :RequestTimeoutException INFO [2022-03-13 17:19:03,035] jepsen worker 75 - jepsen.util 525 :invoke :write [25 12] INFO [2022-03-13 17:19:03,035] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,035] jepsen worker 17 - jepsen.util 4787 :invoke :read [5 nil] INFO [2022-03-13 17:19:03,036] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 11] INFO [2022-03-13 17:19:03,039] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,040] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,047] jepsen worker 12 - jepsen.util 912 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,049] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,052] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:03,053] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,053] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,054] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,055] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:19:03,057] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,057] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,058] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,059] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:03,060] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,060] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,062] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,062] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:19:03,099] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,100] jepsen worker 69 - jepsen.util 789 :invoke :write [23 0] INFO [2022-03-13 17:19:03,107] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,108] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,113] jepsen worker 26 - jepsen.util 566 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,114] jepsen worker 26 - jepsen.util 566 :invoke :write [8 25] INFO [2022-03-13 17:19:03,153] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,153] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,184] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,185] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,186] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,186] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:19:03,190] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,191] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,192] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,192] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,193] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,194] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:19:03,208] jepsen worker 77 - jepsen.util 707 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,209] jepsen worker 77 - jepsen.util 707 :invoke :write [25 15] INFO [2022-03-13 17:19:03,223] jepsen worker 4 - jepsen.util 2614 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,223] jepsen worker 34 - jepsen.util 934 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,223] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,223] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,224] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:03,224] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:03,269] jepsen worker 88 - jepsen.util 2428 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,269] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,269] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,270] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 38] INFO [2022-03-13 17:19:03,279] jepsen worker 76 - jepsen.util 796 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,280] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:03,281] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,281] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:19:03,286] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,287] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,294] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,294] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,294] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,294] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:19:03,295] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:19:03,295] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,297] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,298] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,299] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,300] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:19:03,306] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,306] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,307] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,307] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,307] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,308] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:19:03,308] jepsen worker 24 - jepsen.util 654 :invoke :write [8 1] INFO [2022-03-13 17:19:03,308] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,308] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,309] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:19:03,395] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,395] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:19:03,411] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,412] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,416] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,416] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,417] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,418] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:19:03,418] jepsen worker 70 - jepsen.util 880 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,418] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,419] jepsen worker 70 - jepsen.util 880 :invoke :write [23 21] INFO [2022-03-13 17:19:03,420] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:03,420] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,421] jepsen worker 62 - jepsen.util 512 :invoke :write [20 22] INFO [2022-03-13 17:19:03,421] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,422] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:19:03,423] jepsen worker 69 - jepsen.util 789 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,424] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,427] jepsen worker 69 - jepsen.util 789 :invoke :write [23 2] INFO [2022-03-13 17:19:03,428] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:03,428] jepsen worker 52 - jepsen.util 682 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,429] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:03,435] jepsen worker 46 - jepsen.util 4546 :info :write [15 23] :SyncWriteAmbiguous INFO [2022-03-13 17:19:03,437] jepsen worker 46 - jepsen.util 4636 :invoke :write [15 22] INFO [2022-03-13 17:19:03,462] jepsen worker 37 - jepsen.util 2467 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,462] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,463] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 10] INFO [2022-03-13 17:19:03,464] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,466] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,467] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,469] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,470] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:19:03,480] jepsen worker 80 - jepsen.util 800 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,481] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,481] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,481] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:03,482] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:03,483] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:03,485] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,485] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:03,487] jepsen worker 63 - jepsen.util 4563 :info :write [21 36] :RequestTimeoutException INFO [2022-03-13 17:19:03,489] jepsen worker 63 - jepsen.util 4653 :invoke :write [21 30] INFO [2022-03-13 17:19:03,496] jepsen worker 30 - jepsen.util 4530 :ok :read [10 :nil] INFO [2022-03-13 17:19:03,497] jepsen worker 30 - jepsen.util 4530 :invoke :write [10 10] INFO [2022-03-13 17:19:03,504] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,504] jepsen worker 61 - jepsen.util 781 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,505] jepsen worker 61 - jepsen.util 781 :invoke :write [20 20] INFO [2022-03-13 17:19:03,507] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:03,513] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,514] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:03,521] jepsen worker 9 - jepsen.util 4059 :info :write [3 48] :RequestTimeoutException INFO [2022-03-13 17:19:03,521] jepsen worker 9 - jepsen.util 4149 :invoke :write [3 45] INFO [2022-03-13 17:19:03,540] jepsen worker 45 - jepsen.util 4815 :ok :read [15 48] INFO [2022-03-13 17:19:03,541] jepsen worker 45 - jepsen.util 4815 :invoke :read [15 nil] INFO [2022-03-13 17:19:03,556] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,557] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,558] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,559] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,566] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,566] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,567] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:19:03,567] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:19:03,615] jepsen worker 32 - jepsen.util 4712 :ok :read [10 :nil] INFO [2022-03-13 17:19:03,616] jepsen worker 32 - jepsen.util 4712 :invoke :write [10 34] INFO [2022-03-13 17:19:03,616] jepsen worker 13 - jepsen.util 733 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,617] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:03,621] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,621] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,623] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,624] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:03,625] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,625] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:19:03,635] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,635] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,649] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,649] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,650] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,650] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,651] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,652] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:19:03,654] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,654] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:03,657] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,657] jepsen worker 69 - jepsen.util 789 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,658] jepsen worker 71 - jepsen.util 611 :invoke :write [23 37] INFO [2022-03-13 17:19:03,658] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:03,691] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,691] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,709] jepsen worker 37 - jepsen.util 2467 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,710] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,760] jepsen worker 64 - jepsen.util 4384 :ok :read [21 :nil] INFO [2022-03-13 17:19:03,761] jepsen worker 64 - jepsen.util 4384 :invoke :write [21 9] INFO [2022-03-13 17:19:03,762] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,762] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,762] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,763] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,764] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,764] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,764] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:19:03,765] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,765] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:19:03,765] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:19:03,797] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,798] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,800] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:03,800] jepsen worker nemesis - couchbase.util Rebalance check count: 650 INFO [2022-03-13 17:19:03,800] jepsen worker nemesis - couchbase.util Rebalance stuck count: 522 INFO [2022-03-13 17:19:03,810] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,810] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,811] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:03,811] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 2] INFO [2022-03-13 17:19:03,823] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,823] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,823] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,824] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:03,824] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:03,824] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:19:03,831] jepsen worker 51 - jepsen.util 771 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,835] jepsen worker 44 - jepsen.util 764 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,835] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:03,835] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,838] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:03,839] jepsen worker 44 - jepsen.util 764 :invoke :write [14 27] INFO [2022-03-13 17:19:03,840] jepsen worker 7 - jepsen.util 637 :invoke :write [2 0] INFO [2022-03-13 17:19:03,840] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:19:03,860] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,860] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:19:03,863] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,863] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,864] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:03,864] jepsen worker 25 - jepsen.util 565 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,864] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,865] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,865] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:03,866] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:03,867] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:03,868] jepsen worker 34 - jepsen.util 934 :invoke :write [11 35] INFO [2022-03-13 17:19:03,870] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,871] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:19:03,894] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,895] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:19:03,895] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,896] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:19:03,938] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,938] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,939] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,939] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:03,940] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:19:03,941] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 8] INFO [2022-03-13 17:19:03,942] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,943] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:03,991] jepsen worker 29 - jepsen.util 4979 :ok :read [9 26] INFO [2022-03-13 17:19:03,991] jepsen worker 29 - jepsen.util 4979 :invoke :write [9 13] INFO [2022-03-13 17:19:03,993] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:03,994] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,012] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,014] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:19:04,031] jepsen worker 71 - jepsen.util 611 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,031] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,032] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:04,032] jepsen worker 75 - jepsen.util 525 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,032] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:04,033] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,033] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:04,033] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,033] jepsen worker 76 - jepsen.util 796 :invoke :write [25 41] INFO [2022-03-13 17:19:04,034] jepsen worker 13 - jepsen.util 733 :invoke :write [4 27] INFO [2022-03-13 17:19:04,036] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,036] jepsen worker 69 - jepsen.util 789 :invoke :write [23 41] INFO [2022-03-13 17:19:04,040] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,041] jepsen worker 78 - jepsen.util 708 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,041] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:19:04,042] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:04,064] jepsen worker 24 - jepsen.util 654 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,064] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,064] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,065] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,065] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:04,066] jepsen worker 21 - jepsen.util 831 :invoke :write [7 1] INFO [2022-03-13 17:19:04,066] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:04,067] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:19:04,068] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,069] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,069] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,070] jepsen worker 14 - jepsen.util 374 :invoke :write [4 20] INFO [2022-03-13 17:19:04,071] jepsen worker 51 - jepsen.util 771 :invoke :write [17 12] INFO [2022-03-13 17:19:04,072] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:04,074] jepsen worker 88 - jepsen.util 2428 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,076] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 42] INFO [2022-03-13 17:19:04,078] jepsen worker 89 - jepsen.util 2789 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,079] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:19:04,108] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,109] jepsen worker 75 - jepsen.util 525 :invoke :write [25 35] INFO [2022-03-13 17:19:04,114] jepsen worker 43 - jepsen.util 673 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,115] jepsen worker 43 - jepsen.util 673 :invoke :write [14 45] INFO [2022-03-13 17:19:04,148] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,148] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:19:04,151] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,151] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,162] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,162] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,163] jepsen worker 6 - jepsen.util 636 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,163] jepsen worker 12 - jepsen.util 912 :invoke :write [4 25] INFO [2022-03-13 17:19:04,164] jepsen worker 8 - jepsen.util 728 :invoke :write [2 27] INFO [2022-03-13 17:19:04,164] jepsen worker 6 - jepsen.util 636 :invoke :write [2 10] INFO [2022-03-13 17:19:04,172] jepsen worker 34 - jepsen.util 934 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,173] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:04,175] jepsen worker 74 - jepsen.util 4304 :info :write [24 27] :RequestTimeoutException INFO [2022-03-13 17:19:04,176] jepsen worker 74 - jepsen.util 4394 :invoke :read [24 nil] INFO [2022-03-13 17:19:04,181] jepsen worker 40 - jepsen.util 4720 :info :write [13 19] :SyncWriteAmbiguous INFO [2022-03-13 17:19:04,182] jepsen worker 40 - jepsen.util 4810 :invoke :write [13 1] INFO [2022-03-13 17:19:04,221] jepsen worker 54 - jepsen.util 4284 :ok :read [18 21] INFO [2022-03-13 17:19:04,222] jepsen worker 54 - jepsen.util 4284 :invoke :write [18 40] INFO [2022-03-13 17:19:04,255] jepsen worker 69 - jepsen.util 789 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,256] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:19:04,277] jepsen worker 89 - jepsen.util 2789 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,278] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,286] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,286] jepsen worker 37 - jepsen.util 2467 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,286] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,287] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:04,288] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 38] INFO [2022-03-13 17:19:04,288] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:19:04,309] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,310] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,312] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,313] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,313] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:19:04,314] jepsen worker 81 - jepsen.util 5121 :ok :read [27 4] INFO [2022-03-13 17:19:04,314] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,315] jepsen worker 81 - jepsen.util 5121 :invoke :write [27 31] INFO [2022-03-13 17:19:04,315] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,316] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:19:04,319] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,319] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,322] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,323] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:19:04,360] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,360] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,361] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,361] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,363] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,363] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,364] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,365] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:19:04,366] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,367] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,367] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,368] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:19:04,373] jepsen worker 88 - jepsen.util 2428 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,374] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,464] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,464] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:19:04,473] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,474] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:04,497] jepsen worker 68 - jepsen.util 4568 :info :write [22 42] :RequestTimeoutException INFO [2022-03-13 17:19:04,497] jepsen worker 68 - jepsen.util 4658 :invoke :write [22 39] INFO [2022-03-13 17:19:04,530] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,531] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,583] jepsen worker 37 - jepsen.util 2467 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,583] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,583] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,584] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,584] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 46] INFO [2022-03-13 17:19:04,585] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:04,586] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,589] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,590] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:19:04,590] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:19:04,595] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,596] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:19:04,598] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,599] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,600] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,601] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,602] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,602] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,603] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,605] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:19:04,635] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,637] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,643] jepsen worker 72 - jepsen.util 4572 :info :write [24 5] :RequestTimeoutException INFO [2022-03-13 17:19:04,644] jepsen worker 72 - jepsen.util 4662 :invoke :read [24 nil] INFO [2022-03-13 17:19:04,692] jepsen worker 37 - jepsen.util 2467 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,693] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:19:04,740] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,740] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:19:04,782] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,783] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,806] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:04,806] jepsen worker nemesis - couchbase.util Rebalance check count: 651 INFO [2022-03-13 17:19:04,806] jepsen worker nemesis - couchbase.util Rebalance stuck count: 523 INFO [2022-03-13 17:19:04,824] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,824] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,876] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,876] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,877] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:19:04,877] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,880] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,880] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:19:04,890] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,891] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,892] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,892] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,893] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,893] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,893] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:04,894] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,898] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,898] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,898] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,899] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:19:04,903] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,903] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:19:04,904] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,905] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:19:04,943] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,943] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,943] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,944] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,944] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,945] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,946] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,946] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,947] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,947] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,948] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,948] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,949] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,949] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:04,950] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,950] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,950] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,951] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:19:04,952] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,952] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:04,977] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,977] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,977] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:04,978] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:19:04,979] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 38] INFO [2022-03-13 17:19:04,979] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 48] INFO [2022-03-13 17:19:05,026] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,027] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:19:05,028] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,029] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:05,031] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,032] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:19:05,042] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,043] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:19:05,107] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,108] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,118] jepsen worker 56 - jepsen.util 4376 :info :write [18 38] :RequestTimeoutException INFO [2022-03-13 17:19:05,120] jepsen worker 11 - jepsen.util 4331 :info :write [3 17] :RequestTimeoutException INFO [2022-03-13 17:19:05,196] jepsen worker 56 - jepsen.util 4466 :invoke :write [18 0] INFO [2022-03-13 17:19:05,138] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,197] jepsen worker 11 - jepsen.util 4421 :invoke :write [3 1] INFO [2022-03-13 17:19:05,197] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,172] jepsen worker 37 - jepsen.util 2467 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,172] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,173] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,198] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,198] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:05,198] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 17] INFO [2022-03-13 17:19:05,206] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,206] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:19:05,208] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,209] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:19:05,210] jepsen worker 26 - jepsen.util 566 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,210] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,211] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,213] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,214] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:05,214] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,215] jepsen worker 26 - jepsen.util 566 :invoke :write [8 43] INFO [2022-03-13 17:19:05,216] jepsen worker 52 - jepsen.util 682 :invoke :write [17 44] INFO [2022-03-13 17:19:05,220] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:05,223] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,224] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:19:05,226] jepsen worker 60 - jepsen.util 690 :invoke :write [20 22] INFO [2022-03-13 17:19:05,231] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,232] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:19:05,234] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,235] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,235] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,236] jepsen worker 24 - jepsen.util 654 :invoke :write [8 38] INFO [2022-03-13 17:19:05,237] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:19:05,238] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:05,242] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,244] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,250] jepsen worker 88 - jepsen.util 2428 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,252] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:05,254] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,257] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,259] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,260] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,260] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,261] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 7] INFO [2022-03-13 17:19:05,271] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,272] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,274] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,275] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:19:05,312] jepsen worker 0 - jepsen.util 4320 :ok :read [0 41] INFO [2022-03-13 17:19:05,313] jepsen worker 0 - jepsen.util 4320 :invoke :write [0 41] INFO [2022-03-13 17:19:05,322] jepsen worker 41 - jepsen.util 4541 :ok :read [13 39] INFO [2022-03-13 17:19:05,323] jepsen worker 41 - jepsen.util 4541 :invoke :read [13 nil] INFO [2022-03-13 17:19:05,324] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,325] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:19:05,330] jepsen worker 15 - jepsen.util 4425 :ok :read [5 43] INFO [2022-03-13 17:19:05,331] jepsen worker 15 - jepsen.util 4425 :invoke :write [5 25] INFO [2022-03-13 17:19:05,335] jepsen worker 89 - jepsen.util 2789 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,336] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,345] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,345] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,346] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,347] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,349] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,349] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,352] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,352] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,357] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,357] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:19:05,357] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,357] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,358] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,358] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:05,359] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:19:05,359] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,363] jepsen worker 13 - jepsen.util 733 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,363] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:05,364] jepsen worker 28 - jepsen.util 4528 :info :write [9 44] :RequestTimeoutException INFO [2022-03-13 17:19:05,364] jepsen worker 28 - jepsen.util 4618 :invoke :read [9 nil] INFO [2022-03-13 17:19:05,389] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,389] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,390] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:19:05,391] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,392] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,392] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,399] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,400] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,401] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,402] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,403] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,404] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,406] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,407] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,408] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,408] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,408] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,408] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,409] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:19:05,410] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:05,411] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,411] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,412] jepsen worker 69 - jepsen.util 789 :invoke :write [23 4] INFO [2022-03-13 17:19:05,412] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,412] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,413] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:19:05,413] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,414] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,414] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:19:05,422] jepsen worker 80 - jepsen.util 800 :invoke :write [26 38] INFO [2022-03-13 17:19:05,443] jepsen worker 37 - jepsen.util 2467 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,443] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,444] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,444] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,455] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,456] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 49] INFO [2022-03-13 17:19:05,456] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,457] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:19:05,492] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,492] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,493] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:05,493] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:19:05,515] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,515] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,523] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,524] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,526] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,527] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:19:05,532] jepsen worker 51 - jepsen.util 771 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,533] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:05,556] jepsen worker 62 - jepsen.util 512 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,556] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,556] jepsen worker 77 - jepsen.util 707 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,559] jepsen worker 62 - jepsen.util 512 :invoke :write [20 38] INFO [2022-03-13 17:19:05,559] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 26] INFO [2022-03-13 17:19:05,560] jepsen worker 77 - jepsen.util 707 :invoke :write [25 1] INFO [2022-03-13 17:19:05,560] jepsen worker 41 - jepsen.util 4541 :ok :read [13 39] INFO [2022-03-13 17:19:05,561] jepsen worker 41 - jepsen.util 4541 :invoke :read [13 nil] INFO [2022-03-13 17:19:05,564] jepsen worker 58 - jepsen.util 4198 :ok :read [19 6] INFO [2022-03-13 17:19:05,565] jepsen worker 58 - jepsen.util 4198 :invoke :read [19 nil] INFO [2022-03-13 17:19:05,568] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,569] jepsen worker 22 - jepsen.util 652 :invoke :write [7 49] INFO [2022-03-13 17:19:05,576] jepsen worker 57 - jepsen.util 4557 :ok :read [19 6] INFO [2022-03-13 17:19:05,577] jepsen worker 57 - jepsen.util 4557 :invoke :write [19 32] INFO [2022-03-13 17:19:05,581] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,582] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:19:05,586] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,587] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:19:05,593] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,594] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,596] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,597] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,606] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,606] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,607] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:05,612] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,613] jepsen worker 2 - jepsen.util 4772 :info :write [0 29] :RequestTimeoutException INFO [2022-03-13 17:19:05,614] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,615] jepsen worker 2 - jepsen.util 4862 :invoke :write [0 47] INFO [2022-03-13 17:19:05,616] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,616] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,616] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,617] jepsen worker 82 - jepsen.util 4672 :info :write [27 28] :RequestTimeoutException INFO [2022-03-13 17:19:05,618] jepsen worker 83 - jepsen.util 4133 :info :write [27 7] :RequestTimeoutException INFO [2022-03-13 17:19:05,618] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,618] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,619] jepsen worker 83 - jepsen.util 4223 :invoke :read [27 nil] INFO [2022-03-13 17:19:05,619] jepsen worker 82 - jepsen.util 4762 :invoke :write [27 49] INFO [2022-03-13 17:19:05,619] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,620] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:19:05,622] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,622] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 49] INFO [2022-03-13 17:19:05,649] jepsen worker 76 - jepsen.util 796 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,649] jepsen worker 76 - jepsen.util 796 :invoke :write [25 14] INFO [2022-03-13 17:19:05,650] jepsen worker 14 - jepsen.util 374 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,650] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:05,698] jepsen worker 70 - jepsen.util 880 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,699] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,699] jepsen worker 69 - jepsen.util 789 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,700] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,706] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,706] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,707] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,707] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:19:05,708] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,709] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:19:05,727] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,729] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,752] jepsen worker 43 - jepsen.util 673 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,753] jepsen worker 43 - jepsen.util 673 :invoke :write [14 18] INFO [2022-03-13 17:19:05,767] jepsen worker 41 - jepsen.util 4541 :ok :read [13 39] INFO [2022-03-13 17:19:05,768] jepsen worker 41 - jepsen.util 4541 :invoke :read [13 nil] INFO [2022-03-13 17:19:05,775] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,776] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,776] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:19:05,781] jepsen worker 61 - jepsen.util 781 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,781] jepsen worker 21 - jepsen.util 831 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,781] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,781] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,781] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,781] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,781] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,782] jepsen worker 61 - jepsen.util 781 :invoke :write [20 6] INFO [2022-03-13 17:19:05,782] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,789] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,789] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,792] jepsen worker 25 - jepsen.util 565 :invoke :write [8 35] INFO [2022-03-13 17:19:05,793] jepsen worker 78 - jepsen.util 708 :invoke :write [26 11] INFO [2022-03-13 17:19:05,795] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:19:05,797] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:05,797] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:05,798] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:05,799] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:05,806] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:05,814] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:05,814] jepsen worker nemesis - couchbase.util Rebalance check count: 652 INFO [2022-03-13 17:19:05,814] jepsen worker nemesis - couchbase.util Rebalance stuck count: 524 INFO [2022-03-13 17:19:05,819] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,819] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,819] jepsen worker 37 - jepsen.util 2467 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,820] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,820] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,820] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,821] jepsen worker 75 - jepsen.util 525 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,821] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:05,822] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,822] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 22] INFO [2022-03-13 17:19:05,823] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:05,824] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:19:05,824] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:05,825] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,826] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,827] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:19:05,887] jepsen worker 41 - jepsen.util 4541 :ok :read [13 39] INFO [2022-03-13 17:19:05,887] jepsen worker 41 - jepsen.util 4541 :invoke :write [13 17] INFO [2022-03-13 17:19:05,891] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,892] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:05,897] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,897] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,898] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:19:05,898] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,898] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:19:05,899] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:19:05,902] jepsen worker 24 - jepsen.util 654 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,903] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:05,927] jepsen worker 7 - jepsen.util 637 :fail :write [2 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,928] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,928] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,928] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:19:05,929] jepsen worker 7 - jepsen.util 637 :invoke :write [2 27] INFO [2022-03-13 17:19:05,930] jepsen worker 69 - jepsen.util 789 :invoke :write [23 6] INFO [2022-03-13 17:19:05,963] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,964] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:05,967] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,968] jepsen worker 88 - jepsen.util 2428 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,968] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 40] INFO [2022-03-13 17:19:05,969] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 38] INFO [2022-03-13 17:19:05,998] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:05,998] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,002] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,003] jepsen worker 23 - jepsen.util 563 :invoke :write [7 5] INFO [2022-03-13 17:19:06,012] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,013] jepsen worker 35 - jepsen.util 935 :invoke :write [11 7] INFO [2022-03-13 17:19:06,037] jepsen worker 26 - jepsen.util 566 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,037] jepsen worker 76 - jepsen.util 796 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,038] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,040] jepsen worker 76 - jepsen.util 796 :invoke :write [25 26] INFO [2022-03-13 17:19:06,040] jepsen worker 13 - jepsen.util 733 :invoke :write [4 8] INFO [2022-03-13 17:19:06,042] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:06,047] jepsen worker 73 - jepsen.util 4753 :info :write [24 33] :RequestTimeoutException INFO [2022-03-13 17:19:06,048] jepsen worker 73 - jepsen.util 4843 :invoke :write [24 5] INFO [2022-03-13 17:19:06,058] jepsen worker 38 - jepsen.util 2828 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,058] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,059] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:19:06,060] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,061] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,063] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:19:06,070] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,071] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:19:06,072] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,073] jepsen worker 71 - jepsen.util 611 :invoke :write [23 22] INFO [2022-03-13 17:19:06,085] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,086] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,127] jepsen worker 80 - jepsen.util 800 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,127] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,127] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,127] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,127] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,128] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,128] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:06,129] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:06,129] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:06,130] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:06,130] jepsen worker 80 - jepsen.util 800 :invoke :write [26 9] INFO [2022-03-13 17:19:06,131] jepsen worker 21 - jepsen.util 831 :invoke :write [7 22] INFO [2022-03-13 17:19:06,159] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,160] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:19:06,160] jepsen worker 66 - jepsen.util 4206 :info :write [22 19] :RequestTimeoutException INFO [2022-03-13 17:19:06,161] jepsen worker 66 - jepsen.util 4296 :invoke :read [22 nil] INFO [2022-03-13 17:19:06,171] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,171] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,172] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,172] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,172] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:19:06,173] jepsen worker 69 - jepsen.util 789 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,173] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:06,174] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,178] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,178] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,179] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 2] INFO [2022-03-13 17:19:06,179] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,179] jepsen worker 60 - jepsen.util 690 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,180] jepsen worker 60 - jepsen.util 690 :invoke :write [20 34] INFO [2022-03-13 17:19:06,180] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,180] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,181] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,182] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,184] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,184] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,185] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,186] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:19:06,197] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,197] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,197] jepsen worker 44 - jepsen.util 764 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,198] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:06,199] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:06,199] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,220] jepsen worker 25 - jepsen.util 565 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,220] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,221] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:19:06,221] jepsen worker 24 - jepsen.util 654 :invoke :write [8 9] INFO [2022-03-13 17:19:06,251] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,252] jepsen worker 88 - jepsen.util 2428 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,252] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:19:06,255] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,255] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 20] INFO [2022-03-13 17:19:06,256] jepsen worker 75 - jepsen.util 525 :invoke :write [25 31] INFO [2022-03-13 17:19:06,266] jepsen worker 79 - jepsen.util 1159 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,267] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 20] INFO [2022-03-13 17:19:06,291] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,291] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,291] jepsen worker 71 - jepsen.util 611 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,291] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,293] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:19:06,295] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:19:06,295] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:19:06,296] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:19:06,306] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,308] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:06,315] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,316] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:06,320] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,321] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 46] INFO [2022-03-13 17:19:06,328] jepsen worker 66 - jepsen.util 4296 :ok :read [22 30] INFO [2022-03-13 17:19:06,329] jepsen worker 66 - jepsen.util 4296 :invoke :write [22 28] INFO [2022-03-13 17:19:06,339] jepsen worker 36 - jepsen.util 2466 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,339] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,343] jepsen worker 35 - jepsen.util 935 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,343] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,344] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,344] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,344] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,345] jepsen worker 34 - jepsen.util 934 :invoke :write [11 7] INFO [2022-03-13 17:19:06,353] jepsen worker 14 - jepsen.util 374 :invoke :write [4 14] INFO [2022-03-13 17:19:06,354] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:06,354] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:06,355] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:06,366] jepsen worker 52 - jepsen.util 682 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,368] jepsen worker 52 - jepsen.util 682 :invoke :write [17 40] INFO [2022-03-13 17:19:06,403] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,404] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:19:06,413] jepsen worker 89 - jepsen.util 2789 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,413] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 8] INFO [2022-03-13 17:19:06,420] jepsen worker 37 - jepsen.util 2467 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,421] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 42] INFO [2022-03-13 17:19:06,430] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,431] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,459] jepsen worker 80 - jepsen.util 800 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,460] jepsen worker 80 - jepsen.util 800 :invoke :write [26 32] INFO [2022-03-13 17:19:06,480] jepsen worker 76 - jepsen.util 796 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,481] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:06,485] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,486] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:06,491] jepsen worker 77 - jepsen.util 707 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,491] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,491] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:06,492] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:06,499] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,500] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:19:06,545] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,545] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,545] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,546] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:19:06,546] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,547] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,548] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,548] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,549] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,549] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,550] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,551] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:19:06,551] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,552] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,553] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,554] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:19:06,558] jepsen worker 21 - jepsen.util 831 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,559] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,559] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:06,559] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,560] jepsen worker 89 - jepsen.util 2789 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,561] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,561] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:06,561] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:19:06,643] jepsen worker 88 - jepsen.util 2428 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,644] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 37] INFO [2022-03-13 17:19:06,649] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,649] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 24] INFO [2022-03-13 17:19:06,652] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,652] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:19:06,665] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,666] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,668] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,669] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:06,671] jepsen worker 22 - jepsen.util 652 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,672] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:06,673] jepsen worker 62 - jepsen.util 512 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,673] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,674] jepsen worker 70 - jepsen.util 880 :invoke :write [23 25] INFO [2022-03-13 17:19:06,675] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:06,701] jepsen worker 61 - jepsen.util 781 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,701] jepsen worker 61 - jepsen.util 781 :invoke :write [20 39] INFO [2022-03-13 17:19:06,704] jepsen worker 67 - jepsen.util 3757 :info :write [22 43] :RequestTimeoutException INFO [2022-03-13 17:19:06,704] jepsen worker 67 - jepsen.util 3847 :invoke :write [22 36] INFO [2022-03-13 17:19:06,705] jepsen worker 87 - jepsen.util 2877 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,706] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:06,713] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,714] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:19:06,741] jepsen worker 24 - jepsen.util 654 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,742] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:06,753] jepsen worker 12 - jepsen.util 912 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,753] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:06,775] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,775] jepsen worker 37 - jepsen.util 2467 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,776] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:19:06,783] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,799] jepsen worker 38 - jepsen.util 2828 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,800] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:19:06,820] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:06,820] jepsen worker nemesis - couchbase.util Rebalance check count: 653 INFO [2022-03-13 17:19:06,820] jepsen worker nemesis - couchbase.util Rebalance stuck count: 525 INFO [2022-03-13 17:19:06,828] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,829] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,837] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,837] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,839] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,839] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:06,840] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,841] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:19:06,857] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:19:06,858] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] INFO [2022-03-13 17:19:06,871] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,871] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,871] jepsen worker 13 - jepsen.util 733 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,872] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:06,874] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:06,874] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:06,875] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,875] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,876] jepsen worker 69 - jepsen.util 789 :invoke :write [23 22] INFO [2022-03-13 17:19:06,877] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:19:06,884] jepsen worker 78 - jepsen.util 708 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,885] jepsen worker 78 - jepsen.util 708 :invoke :write [26 34] INFO [2022-03-13 17:19:06,910] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,910] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:19:06,912] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,912] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:19:06,916] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,916] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:19:06,975] jepsen worker 88 - jepsen.util 2428 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,976] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,975] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,976] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:06,977] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:06,977] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 40] INFO [2022-03-13 17:19:06,983] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,983] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,984] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 0] INFO [2022-03-13 17:19:06,984] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:06,992] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,992] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,993] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,993] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:06,993] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:19:06,994] jepsen worker 51 - jepsen.util 771 :invoke :write [17 22] INFO [2022-03-13 17:19:06,996] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:06,996] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:19:07,002] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,003] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:07,009] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,010] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:19:07,021] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,021] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,031] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,031] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 24] INFO [2022-03-13 17:19:07,080] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,080] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:07,095] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,095] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,096] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,097] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,098] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,098] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,102] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,102] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,103] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,104] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:19:07,121] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,122] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:07,129] jepsen worker 6 - jepsen.util 636 :fail :write [2 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,130] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:07,131] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,131] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] WARN [2022-03-13 17:19:07,131] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7995177,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10887425,"total_server_duration_us":415614,"encode_duration_us":19771,"operation_id":"0x128ec","timeout_ms":10000,"last_server_duration_us":25450,"total_duration_us":10008908},{"operation_name":"upsert","last_dispatch_duration_us":36472,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":3695520,"total_server_duration_us":731510,"encode_duration_us":18232,"operation_id":"0x129ae","timeout_ms":10000,"last_server_duration_us":20921,"total_duration_us":10008220},{"operation_name":"upsert","last_dispatch_duration_us":38760,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8911251,"total_server_duration_us":269101,"encode_duration_us":18769,"operation_id":"0x12b6e","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10005235},{"operation_name":"upsert","last_dispatch_duration_us":89202,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2157965,"total_server_duration_us":659871,"encode_duration_us":21568,"operation_id":"0x12a9a","timeout_ms":10000,"last_server_duration_us":49719,"total_duration_us":10003993},{"operation_name":"upsert","last_dispatch_duration_us":44868,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":3423724,"total_server_duration_us":687004,"encode_duration_us":19431,"operation_id":"0x12a1d","timeout_ms":10000,"last_server_duration_us":27312,"total_duration_us":10002389},{"operation_name":"upsert","last_dispatch_duration_us":8317073,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8909322,"total_server_duration_us":395060,"encode_duration_us":19306,"operation_id":"0x129bf","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10001344},{"operation_name":"upsert","last_dispatch_duration_us":9060435,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9060435,"total_server_duration_us":9045859,"encode_duration_us":19978,"operation_id":"0x12b46","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":9067010},{"operation_name":"get","last_dispatch_duration_us":2805227,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2805227,"total_server_duration_us":7962,"operation_id":"0x12e64","timeout_ms":30000,"last_server_duration_us":7962,"total_duration_us":2807486},{"operation_name":"upsert","last_dispatch_duration_us":893948,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":893948,"total_server_duration_us":295516,"encode_duration_us":18537,"operation_id":"0x12b2b","timeout_ms":10000,"last_server_duration_us":295516,"total_duration_us":2496013},{"operation_name":"upsert","last_dispatch_duration_us":2005816,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2005816,"total_server_duration_us":157260,"encode_duration_us":23873,"operation_id":"0x12c40","timeout_ms":10000,"last_server_duration_us":157260,"total_duration_us":2015313}],"total_count":83}} INFO [2022-03-13 17:19:07,141] jepsen worker 43 - jepsen.util 673 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,141] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,142] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,141] jepsen worker 75 - jepsen.util 525 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,142] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,142] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,143] jepsen worker 26 - jepsen.util 566 :invoke :write [8 36] INFO [2022-03-13 17:19:07,144] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:07,145] jepsen worker 53 - jepsen.util 593 :invoke :write [17 48] INFO [2022-03-13 17:19:07,145] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,146] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:07,148] jepsen worker 75 - jepsen.util 525 :invoke :write [25 8] INFO [2022-03-13 17:19:07,150] jepsen worker 43 - jepsen.util 673 :invoke :write [14 19] INFO [2022-03-13 17:19:07,150] jepsen worker 76 - jepsen.util 796 :invoke :write [25 41] INFO [2022-03-13 17:19:07,165] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,166] jepsen worker 13 - jepsen.util 733 :invoke :write [4 29] INFO [2022-03-13 17:19:07,171] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,173] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 29] INFO [2022-03-13 17:19:07,171] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,182] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:19:07,171] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,182] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:07,184] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,185] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:07,186] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,188] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:19:07,202] jepsen worker 38 - jepsen.util 2828 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,202] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,203] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:19:07,205] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:07,208] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,208] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:19:07,218] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,218] jepsen worker 88 - jepsen.util 2428 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,219] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:07,219] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:19:07,255] jepsen worker 60 - jepsen.util 690 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,255] jepsen worker 60 - jepsen.util 690 :invoke :write [20 9] INFO [2022-03-13 17:19:07,259] jepsen worker 23 - jepsen.util 563 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,260] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:07,263] jepsen worker 34 - jepsen.util 934 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,264] jepsen worker 34 - jepsen.util 934 :invoke :write [11 4] INFO [2022-03-13 17:19:07,294] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,294] jepsen worker 69 - jepsen.util 789 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,295] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:07,295] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:19:07,299] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,300] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:19:07,305] jepsen worker 3 - jepsen.util 2253 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,305] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,308] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,308] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 10] INFO [2022-03-13 17:19:07,320] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,320] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:19:07,353] jepsen worker 75 - jepsen.util 525 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,353] jepsen worker 75 - jepsen.util 525 :invoke :write [25 19] INFO [2022-03-13 17:19:07,368] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,369] jepsen worker 37 - jepsen.util 2467 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,369] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 10] INFO [2022-03-13 17:19:07,371] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 22] INFO [2022-03-13 17:19:07,400] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,400] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 37] INFO [2022-03-13 17:19:07,403] jepsen worker 14 - jepsen.util 374 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,403] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:19:07,437] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,439] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:07,443] jepsen worker 38 - jepsen.util 2828 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,444] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:19:07,448] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:07,448] jepsen worker 8 - jepsen.util 728 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,449] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:07,450] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:07,450] jepsen worker 44 - jepsen.util 764 :invoke :write [14 31] INFO [2022-03-13 17:19:07,451] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:07,470] jepsen worker 52 - jepsen.util 682 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,470] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:07,474] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,474] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,475] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:07,475] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:07,477] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,477] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:19:07,498] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,498] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,500] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:07,500] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,545] jepsen worker 89 - jepsen.util 2789 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,545] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:07,555] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,556] jepsen worker 80 - jepsen.util 800 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,556] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,556] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,556] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:07,557] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:07,557] jepsen worker 80 - jepsen.util 800 :invoke :write [26 14] INFO [2022-03-13 17:19:07,558] jepsen worker 24 - jepsen.util 654 :invoke :write [8 22] INFO [2022-03-13 17:19:07,560] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,561] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 29] INFO [2022-03-13 17:19:07,564] jepsen worker 79 - jepsen.util 1159 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,566] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,566] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 3] INFO [2022-03-13 17:19:07,566] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:07,567] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,567] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,569] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:07,569] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:07,570] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,571] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,571] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:07,571] jepsen worker 71 - jepsen.util 611 :invoke :write [23 5] INFO [2022-03-13 17:19:07,572] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,572] jepsen worker 76 - jepsen.util 796 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,573] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:07,573] jepsen worker 76 - jepsen.util 796 :invoke :write [25 6] INFO [2022-03-13 17:19:07,580] jepsen worker 10 - jepsen.util 4780 :info :write [3 35] :RequestTimeoutException INFO [2022-03-13 17:19:07,584] jepsen worker 10 - jepsen.util 4870 :invoke :read [3 nil] INFO [2022-03-13 17:19:07,586] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,588] jepsen worker 69 - jepsen.util 789 :invoke :write [23 49] INFO [2022-03-13 17:19:07,602] jepsen worker 53 - jepsen.util 593 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,603] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:07,620] jepsen worker 4 - jepsen.util 2614 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,620] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,621] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,622] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,622] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:19:07,622] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,622] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,623] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,623] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,623] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,623] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,624] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:07,624] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:19:07,625] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:19:07,625] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:19:07,625] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:19:07,668] jepsen worker 13 - jepsen.util 733 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,669] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:07,694] jepsen worker 26 - jepsen.util 566 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,694] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:07,706] jepsen worker 37 - jepsen.util 2467 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,707] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:19:07,723] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,723] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:07,724] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,725] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:19:07,739] jepsen worker 71 - jepsen.util 611 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,740] jepsen worker 71 - jepsen.util 611 :invoke :write [23 23] INFO [2022-03-13 17:19:07,741] jepsen worker 7 - jepsen.util 637 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,741] jepsen worker 7 - jepsen.util 637 :invoke :write [2 37] INFO [2022-03-13 17:19:07,744] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,745] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:07,756] jepsen worker 75 - jepsen.util 525 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,757] jepsen worker 75 - jepsen.util 525 :invoke :write [25 45] INFO [2022-03-13 17:19:07,761] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,762] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:19:07,788] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,791] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:07,800] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,800] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:07,805] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,806] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:07,807] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,807] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:19:07,819] jepsen worker 51 - jepsen.util 771 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,820] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:07,827] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:07,827] jepsen worker nemesis - couchbase.util Rebalance check count: 654 INFO [2022-03-13 17:19:07,827] jepsen worker nemesis - couchbase.util Rebalance stuck count: 526 INFO [2022-03-13 17:19:07,840] jepsen worker 61 - jepsen.util 781 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,840] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:07,849] jepsen worker 43 - jepsen.util 673 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,851] jepsen worker 43 - jepsen.util 673 :invoke :write [14 20] INFO [2022-03-13 17:19:07,854] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,855] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,866] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,867] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:07,896] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,897] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,897] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:07,897] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:19:07,901] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,902] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:07,903] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,903] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:19:07,923] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,924] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:19:07,984] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,984] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:07,985] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:07,985] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 24] INFO [2022-03-13 17:19:08,001] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,002] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:19:08,003] jepsen worker 88 - jepsen.util 2428 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,003] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,003] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 29] INFO [2022-03-13 17:19:08,004] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:19:08,066] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,066] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,067] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:19:08,067] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,083] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,083] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,111] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,112] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,144] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,145] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:19:08,173] jepsen worker 36 - jepsen.util 2466 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,173] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,174] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,174] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:08,182] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,182] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,183] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 5] INFO [2022-03-13 17:19:08,183] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 10] INFO [2022-03-13 17:19:08,211] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,211] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,212] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,213] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:08,228] jepsen worker 85 - jepsen.util 4135 :info :write [28 25] :RequestTimeoutException INFO [2022-03-13 17:19:08,229] jepsen worker 85 - jepsen.util 4225 :invoke :read [28 nil] INFO [2022-03-13 17:19:08,264] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,265] jepsen worker 88 - jepsen.util 2428 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,266] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:19:08,266] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,275] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,276] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 8] INFO [2022-03-13 17:19:08,278] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,279] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:19:08,283] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,283] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,319] jepsen worker 38 - jepsen.util 2828 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,320] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:19:08,349] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,350] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,355] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,355] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:08,363] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,363] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,364] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,364] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:19:08,373] jepsen worker 69 - jepsen.util 789 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,373] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:08,381] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,382] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:08,425] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,426] jepsen worker 89 - jepsen.util 2789 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,426] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,427] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] WARN [2022-03-13 17:19:08,430] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9517971,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19127858,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9284022,"encode_duration_us":26868,"operation_id":"0x1274a","timeout_ms":10000,"last_server_duration_us":9073483,"total_duration_us":10008308},{"operation_name":"upsert","last_dispatch_duration_us":9561679,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11692116,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9228233,"encode_duration_us":39063,"operation_id":"0x12a18","timeout_ms":10000,"last_server_duration_us":9080926,"total_duration_us":10010970}],"total_count":2}} INFO [2022-03-13 17:19:08,431] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,431] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,433] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,433] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,434] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,435] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:19:08,445] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,445] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,445] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,446] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,464] jepsen worker 76 - jepsen.util 796 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,464] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:08,464] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,465] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,465] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:08,465] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,465] jepsen worker 6 - jepsen.util 636 :invoke :write [2 36] INFO [2022-03-13 17:19:08,466] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:08,466] jepsen worker 70 - jepsen.util 880 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,466] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,466] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:08,467] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:08,468] jepsen worker 13 - jepsen.util 733 :invoke :write [4 19] INFO [2022-03-13 17:19:08,468] jepsen worker 70 - jepsen.util 880 :invoke :write [23 47] INFO [2022-03-13 17:19:08,492] jepsen worker 34 - jepsen.util 934 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,493] jepsen worker 34 - jepsen.util 934 :invoke :write [11 8] INFO [2022-03-13 17:19:08,494] jepsen worker 71 - jepsen.util 611 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,494] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,494] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,495] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,495] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:08,496] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:08,499] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,499] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,500] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:19:08,501] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:08,502] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,503] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:08,504] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,505] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:08,506] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,506] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:08,507] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,508] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:19:08,511] jepsen worker 78 - jepsen.util 708 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,511] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,511] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,513] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:08,513] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:08,514] jepsen worker 78 - jepsen.util 708 :invoke :write [26 9] INFO [2022-03-13 17:19:08,531] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,532] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,545] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,545] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,545] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,546] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,546] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,547] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,547] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,547] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,548] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:19:08,548] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,548] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:08,549] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:19:08,550] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,551] jepsen worker 60 - jepsen.util 690 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,551] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:08,552] jepsen worker 60 - jepsen.util 690 :invoke :write [20 22] INFO [2022-03-13 17:19:08,553] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,553] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,554] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:08,555] jepsen worker 53 - jepsen.util 593 :invoke :write [17 38] INFO [2022-03-13 17:19:08,558] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,558] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,559] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:19:08,560] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:19:08,562] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,563] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:08,584] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,585] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:19:08,596] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,599] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,602] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,603] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,607] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,608] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,610] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,610] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,613] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,614] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,619] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,619] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,619] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:08,620] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] WARN [2022-03-13 17:19:08,634] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15551716,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":15551716,"total_server_duration_us":17290,"operation_id":"0x1296f","timeout_ms":30000,"last_server_duration_us":17290,"total_duration_us":15554552},{"operation_name":"get","last_dispatch_duration_us":12515843,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":12515843,"total_server_duration_us":24755,"operation_id":"0x12a6e","timeout_ms":30000,"last_server_duration_us":24755,"total_duration_us":12518372},{"operation_name":"upsert","last_dispatch_duration_us":9112970,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9891928,"total_server_duration_us":9312954,"encode_duration_us":21522,"operation_id":"0x129b0","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10005802},{"operation_name":"upsert","last_dispatch_duration_us":9386111,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10215378,"total_server_duration_us":9626711,"encode_duration_us":17353,"operation_id":"0x12a21","timeout_ms":10000,"last_server_duration_us":9099014,"total_duration_us":10004980},{"operation_name":"upsert","last_dispatch_duration_us":161705,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9947313,"total_server_duration_us":183137,"encode_duration_us":19026,"operation_id":"0x12ba6","timeout_ms":10000,"last_server_duration_us":32161,"total_duration_us":10004969},{"operation_name":"upsert","last_dispatch_duration_us":8347888,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8347888,"total_server_duration_us":24154,"encode_duration_us":17864,"operation_id":"0x12b49","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10003002},{"operation_name":"upsert","last_dispatch_duration_us":9098109,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9802556,"total_server_duration_us":9437710,"encode_duration_us":28359,"operation_id":"0x12a9b","timeout_ms":10000,"last_server_duration_us":9072420,"total_duration_us":9897960},{"operation_name":"get","last_dispatch_duration_us":9176355,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9176355,"total_server_duration_us":23,"operation_id":"0x12ba4","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":9178864},{"operation_name":"get","last_dispatch_duration_us":2679379,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":2679379,"total_server_duration_us":30918,"operation_id":"0x12e06","timeout_ms":30000,"last_server_duration_us":30918,"total_duration_us":2682110},{"operation_name":"upsert","last_dispatch_duration_us":2463531,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2463531,"total_server_duration_us":357510,"encode_duration_us":22206,"operation_id":"0x12da9","timeout_ms":10000,"last_server_duration_us":357510,"total_duration_us":2468168}],"total_count":49}} INFO [2022-03-13 17:19:08,638] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,638] jepsen worker 44 - jepsen.util 764 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,638] jepsen worker 44 - jepsen.util 764 :invoke :write [14 48] INFO [2022-03-13 17:19:08,639] jepsen worker 62 - jepsen.util 512 :invoke :write [20 31] INFO [2022-03-13 17:19:08,643] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,644] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] WARN [2022-03-13 17:19:08,648] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17370049,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19160026,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9302682,"encode_duration_us":21259,"operation_id":"0x127c9","timeout_ms":10000,"last_server_duration_us":9027816,"total_duration_us":10007521}],"total_count":1}} WARN [2022-03-13 17:19:08,648] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":232845,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":13295160,"total_server_duration_us":49837,"operation_id":"0x12411","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":30006863},{"operation_name":"get","last_dispatch_duration_us":8956431,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8956431,"total_server_duration_us":48563,"operation_id":"0x12b53","timeout_ms":30000,"last_server_duration_us":48563,"total_duration_us":10047784},{"operation_name":"upsert","last_dispatch_duration_us":92006,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1472354,"total_server_duration_us":615544,"encode_duration_us":20136,"operation_id":"0x129a8","timeout_ms":10000,"last_server_duration_us":68015,"total_duration_us":10008091},{"operation_name":"upsert","last_dispatch_duration_us":13919562,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":13919562,"total_server_duration_us":29415,"encode_duration_us":20433,"operation_id":"0x129c9","timeout_ms":10000,"last_server_duration_us":29415,"total_duration_us":10007788},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":18808,"operation_id":"0x129ba","timeout_ms":10000,"total_duration_us":10007381},{"operation_name":"upsert","last_dispatch_duration_us":8932576,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8932576,"total_server_duration_us":30633,"encode_duration_us":17420,"operation_id":"0x12afc","timeout_ms":10000,"last_server_duration_us":30633,"total_duration_us":10003518},{"operation_name":"upsert","last_dispatch_duration_us":9065536,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9508517,"total_server_duration_us":9301407,"encode_duration_us":23062,"operation_id":"0x12a8a","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":9556639},{"operation_name":"get","last_dispatch_duration_us":8519865,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8519865,"total_server_duration_us":135,"operation_id":"0x12c16","timeout_ms":30000,"last_server_duration_us":135,"total_duration_us":8523843},{"operation_name":"upsert","last_dispatch_duration_us":2024587,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2024587,"total_server_duration_us":197496,"encode_duration_us":19044,"operation_id":"0x12c19","timeout_ms":10000,"last_server_duration_us":197496,"total_duration_us":2029021},{"operation_name":"upsert","last_dispatch_duration_us":1142906,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1142906,"total_server_duration_us":958747,"encode_duration_us":41647,"operation_id":"0x12fe0","timeout_ms":10000,"last_server_duration_us":958747,"total_duration_us":1147940}],"total_count":21}} WARN [2022-03-13 17:19:08,654] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17430200,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18175698,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9473372,"encode_duration_us":20190,"operation_id":"0x1274f","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10004042},{"operation_name":"upsert","last_dispatch_duration_us":16965893,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17183858,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9090490,"encode_duration_us":18368,"operation_id":"0x127b2","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":10010689}],"total_count":2}} INFO [2022-03-13 17:19:08,665] jepsen worker 75 - jepsen.util 525 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,665] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,666] jepsen worker 75 - jepsen.util 525 :invoke :write [25 8] INFO [2022-03-13 17:19:08,667] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:19:08,673] jepsen worker 88 - jepsen.util 2428 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,673] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,674] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 2] INFO [2022-03-13 17:19:08,674] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,677] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,677] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,685] jepsen worker 24 - jepsen.util 654 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,686] jepsen worker 24 - jepsen.util 654 :invoke :write [8 19] INFO [2022-03-13 17:19:08,705] jepsen worker 79 - jepsen.util 1159 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,705] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,705] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,706] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,707] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:19:08,707] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:08,708] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:08,709] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:08,719] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,720] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:08,724] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,724] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,726] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:08,726] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] WARN [2022-03-13 17:19:08,730] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9561679,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":11692116,"total_server_duration_us":9228233,"encode_duration_us":39063,"operation_id":"0x12a18","timeout_ms":10000,"last_server_duration_us":9080926,"total_duration_us":10010970},{"operation_name":"upsert","last_dispatch_duration_us":437425,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9936561,"total_server_duration_us":102274,"encode_duration_us":16949,"operation_id":"0x12ad9","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10008462},{"operation_name":"upsert","last_dispatch_duration_us":43037,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":676971,"total_server_duration_us":411353,"encode_duration_us":18406,"operation_id":"0x12cae","timeout_ms":10000,"last_server_duration_us":27492,"total_duration_us":10007709},{"operation_name":"upsert","last_dispatch_duration_us":9142339,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":13088271,"total_server_duration_us":314981,"encode_duration_us":17178,"operation_id":"0x129ad","timeout_ms":10000,"last_server_duration_us":41767,"total_duration_us":10007323},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":19628,"operation_id":"0x12b99","timeout_ms":10000,"total_duration_us":10005702},{"operation_name":"upsert","last_dispatch_duration_us":47107,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":940262,"total_server_duration_us":482393,"encode_duration_us":17554,"operation_id":"0x12b6f","timeout_ms":10000,"last_server_duration_us":31298,"total_duration_us":10003293},{"operation_name":"upsert","last_dispatch_duration_us":201505,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9105487,"total_server_duration_us":61700,"encode_duration_us":18650,"operation_id":"0x12b48","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10001788},{"operation_name":"upsert","last_dispatch_duration_us":40612,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1035488,"total_server_duration_us":540583,"encode_duration_us":17401,"operation_id":"0x12b76","timeout_ms":10000,"last_server_duration_us":29045,"total_duration_us":10001329},{"operation_name":"get","last_dispatch_duration_us":9361663,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9361663,"total_server_duration_us":257473,"operation_id":"0x12b1b","timeout_ms":30000,"last_server_duration_us":257473,"total_duration_us":9363725},{"operation_name":"upsert","last_dispatch_duration_us":2399131,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2399131,"total_server_duration_us":191893,"encode_duration_us":20239,"operation_id":"0x12c8e","timeout_ms":10000,"last_server_duration_us":191893,"total_duration_us":2404249}],"total_count":44}} WARN [2022-03-13 17:19:08,739] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":8852176,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8852176,"total_server_duration_us":5860,"operation_id":"0x129a7","timeout_ms":30000,"last_server_duration_us":5860,"total_duration_us":11979294},{"operation_name":"upsert","last_dispatch_duration_us":8949190,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10099747,"total_server_duration_us":192249,"encode_duration_us":24951,"operation_id":"0x12a8b","timeout_ms":10000,"last_server_duration_us":32743,"total_duration_us":10009042},{"operation_name":"upsert","last_dispatch_duration_us":9048006,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9048006,"total_server_duration_us":9033121,"encode_duration_us":29617,"operation_id":"0x129d0","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10006839},{"operation_name":"upsert","last_dispatch_duration_us":8893924,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":11222578,"total_server_duration_us":1541662,"encode_duration_us":16757,"operation_id":"0x129bb","timeout_ms":10000,"last_server_duration_us":52420,"total_duration_us":10001528},{"operation_name":"get","last_dispatch_duration_us":8863449,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8863449,"total_server_duration_us":12119,"operation_id":"0x12b30","timeout_ms":30000,"last_server_duration_us":12119,"total_duration_us":8863788},{"operation_name":"get","last_dispatch_duration_us":8184093,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8184093,"total_server_duration_us":74510,"operation_id":"0x12b66","timeout_ms":30000,"last_server_duration_us":74510,"total_duration_us":8184455},{"operation_name":"upsert","last_dispatch_duration_us":1730465,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1730465,"total_server_duration_us":99346,"encode_duration_us":30536,"operation_id":"0x12e2c","timeout_ms":10000,"last_server_duration_us":99346,"total_duration_us":1734342},{"operation_name":"upsert","last_dispatch_duration_us":1687998,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1687998,"total_server_duration_us":66037,"encode_duration_us":17865,"operation_id":"0x12fe7","timeout_ms":10000,"last_server_duration_us":66037,"total_duration_us":1692335},{"operation_name":"upsert","last_dispatch_duration_us":1580605,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1580605,"total_server_duration_us":77290,"encode_duration_us":30313,"operation_id":"0x12fdc","timeout_ms":10000,"last_server_duration_us":77290,"total_duration_us":1584656},{"operation_name":"upsert","last_dispatch_duration_us":1448018,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1448018,"total_server_duration_us":182305,"encode_duration_us":24633,"operation_id":"0x12d5d","timeout_ms":10000,"last_server_duration_us":182305,"total_duration_us":1452340}],"total_count":31}} INFO [2022-03-13 17:19:08,760] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,761] jepsen worker 71 - jepsen.util 611 :invoke :write [23 40] INFO [2022-03-13 17:19:08,769] jepsen worker 80 - jepsen.util 800 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,770] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,770] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,770] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,770] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,770] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,771] jepsen worker 43 - jepsen.util 673 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,771] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,771] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,771] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,773] jepsen worker 26 - jepsen.util 566 :invoke :write [8 28] INFO [2022-03-13 17:19:08,774] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:08,775] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:08,776] jepsen worker 25 - jepsen.util 565 :invoke :write [8 26] INFO [2022-03-13 17:19:08,777] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:19:08,778] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:08,779] jepsen worker 43 - jepsen.util 673 :invoke :write [14 9] INFO [2022-03-13 17:19:08,780] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,780] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,781] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:08,781] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,782] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,783] jepsen worker 52 - jepsen.util 682 :invoke :write [17 27] INFO [2022-03-13 17:19:08,784] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,785] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,785] jepsen worker 33 - jepsen.util 753 :invoke :write [11 44] INFO [2022-03-13 17:19:08,785] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,786] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,786] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,787] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,787] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,788] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 36] INFO [2022-03-13 17:19:08,788] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,789] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,794] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,795] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,796] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,797] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,801] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,802] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,806] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,807] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,808] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,809] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,822] jepsen worker 7 - jepsen.util 637 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,822] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,822] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,823] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:08,823] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,823] jepsen worker 42 - jepsen.util 402 :invoke :write [14 38] INFO [2022-03-13 17:19:08,825] jepsen worker 12 - jepsen.util 912 :invoke :write [4 22] INFO [2022-03-13 17:19:08,826] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:19:08,827] jepsen worker 7 - jepsen.util 637 :invoke :write [2 19] INFO [2022-03-13 17:19:08,827] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:08,833] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:08,833] jepsen worker nemesis - couchbase.util Rebalance check count: 655 INFO [2022-03-13 17:19:08,833] jepsen worker nemesis - couchbase.util Rebalance stuck count: 527 WARN [2022-03-13 17:19:08,835] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9386111,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10215378,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9626711,"encode_duration_us":17353,"operation_id":"0x12a21","timeout_ms":10000,"last_server_duration_us":9099014,"total_duration_us":10004980},{"operation_name":"upsert","last_dispatch_duration_us":9112970,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9891928,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9312954,"encode_duration_us":21522,"operation_id":"0x129b0","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10005802},{"operation_name":"upsert","last_dispatch_duration_us":17713252,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25438408,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9303145,"encode_duration_us":18070,"operation_id":"0x124a0","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10009542}],"total_count":3}} INFO [2022-03-13 17:19:08,836] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,837] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] WARN [2022-03-13 17:19:08,846] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9593583,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11343268,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9617273,"encode_duration_us":17992,"operation_id":"0x128c0","timeout_ms":10000,"last_server_duration_us":9067105,"total_duration_us":10005534},{"operation_name":"upsert","last_dispatch_duration_us":9048006,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9048006,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9033121,"encode_duration_us":29617,"operation_id":"0x129d0","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10006839}],"total_count":2}} INFO [2022-03-13 17:19:08,853] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException WARN [2022-03-13 17:19:08,854] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17406,"operation_id":"0x12b4a","timeout_ms":10000,"total_duration_us":10010861},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":16993,"operation_id":"0x12be6","timeout_ms":10000,"total_duration_us":10009497},{"operation_name":"upsert","last_dispatch_duration_us":15380022,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":15380022,"total_server_duration_us":26685,"encode_duration_us":17492,"operation_id":"0x12a7d","timeout_ms":10000,"last_server_duration_us":26685,"total_duration_us":10009005},{"operation_name":"upsert","last_dispatch_duration_us":8016556,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8016556,"total_server_duration_us":27945,"encode_duration_us":23784,"operation_id":"0x12b26","timeout_ms":10000,"last_server_duration_us":27945,"total_duration_us":10003271},{"operation_name":"upsert","last_dispatch_duration_us":7659129,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8676697,"total_server_duration_us":45257,"encode_duration_us":20917,"operation_id":"0x129c7","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10001691},{"operation_name":"upsert","last_dispatch_duration_us":9047912,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9047912,"total_server_duration_us":9029938,"encode_duration_us":16789,"operation_id":"0x1297f","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":9051647},{"operation_name":"get","last_dispatch_duration_us":8001570,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8001570,"total_server_duration_us":63568,"operation_id":"0x12b22","timeout_ms":30000,"last_server_duration_us":63568,"total_duration_us":8009859},{"operation_name":"upsert","last_dispatch_duration_us":1649435,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1649435,"total_server_duration_us":101696,"encode_duration_us":23647,"operation_id":"0x12fef","timeout_ms":10000,"last_server_duration_us":101696,"total_duration_us":3260569},{"operation_name":"upsert","last_dispatch_duration_us":1334529,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1334529,"total_server_duration_us":176692,"encode_duration_us":20933,"operation_id":"0x12fee","timeout_ms":10000,"last_server_duration_us":176692,"total_duration_us":2943321},{"operation_name":"upsert","last_dispatch_duration_us":2708913,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2708913,"total_server_duration_us":226973,"encode_duration_us":32649,"operation_id":"0x12c8d","timeout_ms":10000,"last_server_duration_us":226973,"total_duration_us":2737131}],"total_count":59}} INFO [2022-03-13 17:19:08,854] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,869] jepsen worker 34 - jepsen.util 934 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,869] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,869] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:19:08,870] jepsen worker 34 - jepsen.util 934 :invoke :write [11 8] INFO [2022-03-13 17:19:08,883] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,883] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,884] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,884] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,885] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,885] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,888] jepsen worker 78 - jepsen.util 708 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,888] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,888] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,888] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,889] jepsen worker 78 - jepsen.util 708 :invoke :write [26 0] INFO [2022-03-13 17:19:08,889] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:08,890] jepsen worker 23 - jepsen.util 563 :invoke :write [7 32] INFO [2022-03-13 17:19:08,891] jepsen worker 13 - jepsen.util 733 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,891] jepsen worker 77 - jepsen.util 707 :invoke :write [25 2] INFO [2022-03-13 17:19:08,891] jepsen worker 13 - jepsen.util 733 :invoke :write [4 4] INFO [2022-03-13 17:19:08,893] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,894] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,895] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:19:08,897] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:19:08,898] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,907] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:19:08,912] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,912] jepsen worker 88 - jepsen.util 2428 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,912] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,913] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,915] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,915] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,916] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,916] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,916] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,917] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,921] jepsen worker 38 - jepsen.util 2828 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,922] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 37] INFO [2022-03-13 17:19:08,963] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,963] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,963] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,963] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:08,964] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:08,964] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,964] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,965] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 11] INFO [2022-03-13 17:19:08,965] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,965] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,966] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 23] INFO [2022-03-13 17:19:08,967] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:19:08,971] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,972] jepsen worker 14 - jepsen.util 374 :invoke :write [4 4] INFO [2022-03-13 17:19:08,991] jepsen worker 71 - jepsen.util 611 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,991] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:08,992] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:08,992] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:09,009] jepsen worker 6 - jepsen.util 636 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,009] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:09,020] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,021] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:19:09,023] jepsen worker 53 - jepsen.util 593 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,023] jepsen worker 53 - jepsen.util 593 :invoke :write [17 42] INFO [2022-03-13 17:19:09,033] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,034] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:09,045] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,047] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,069] jepsen worker 24 - jepsen.util 654 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,070] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,070] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,070] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,070] jepsen worker 24 - jepsen.util 654 :invoke :write [8 48] INFO [2022-03-13 17:19:09,072] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:09,073] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:09,074] jepsen worker 80 - jepsen.util 800 :invoke :write [26 19] INFO [2022-03-13 17:19:09,083] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,083] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,084] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 29] INFO [2022-03-13 17:19:09,086] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:09,099] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,099] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,099] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:09,100] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:19:09,105] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,106] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:19:09,110] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,110] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,110] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,111] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,112] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,112] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:19:09,119] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,119] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,120] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,121] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,126] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,127] jepsen worker 88 - jepsen.util 2428 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,127] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:19:09,128] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 23] INFO [2022-03-13 17:19:09,140] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,140] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,141] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,142] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,142] jepsen worker 38 - jepsen.util 2828 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,143] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,144] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:09,145] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,146] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,146] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,147] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:19:09,147] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 38] INFO [2022-03-13 17:19:09,155] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,156] jepsen worker 75 - jepsen.util 525 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,159] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:09,159] jepsen worker 35 - jepsen.util 935 :invoke :write [11 41] INFO [2022-03-13 17:19:09,169] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,169] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:19:09,199] jepsen worker 70 - jepsen.util 880 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,200] jepsen worker 70 - jepsen.util 880 :invoke :write [23 39] INFO [2022-03-13 17:19:09,207] jepsen worker 37 - jepsen.util 2467 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,208] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,243] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,243] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,243] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,243] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,244] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:19:09,244] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,248] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,248] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,248] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,248] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,250] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,250] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,251] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:19:09,251] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:19:09,258] jepsen worker 33 - jepsen.util 753 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,259] jepsen worker 33 - jepsen.util 753 :invoke :write [11 32] INFO [2022-03-13 17:19:09,264] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,264] jepsen worker 60 - jepsen.util 690 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,265] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:09,265] jepsen worker 60 - jepsen.util 690 :invoke :write [20 9] INFO [2022-03-13 17:19:09,267] jepsen worker 78 - jepsen.util 708 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,268] jepsen worker 78 - jepsen.util 708 :invoke :write [26 5] INFO [2022-03-13 17:19:09,273] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,273] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,274] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,274] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:19:09,274] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,275] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 4] INFO [2022-03-13 17:19:09,279] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,280] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,282] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,283] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,285] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,285] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 19] INFO [2022-03-13 17:19:09,310] jepsen worker 1 - jepsen.util 4141 :info :write [0 20] :RequestTimeoutException INFO [2022-03-13 17:19:09,311] jepsen worker 1 - jepsen.util 4231 :invoke :write [0 46] INFO [2022-03-13 17:19:09,358] jepsen worker 25 - jepsen.util 565 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,358] jepsen worker 88 - jepsen.util 2428 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,359] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,359] jepsen worker 25 - jepsen.util 565 :invoke :write [8 43] INFO [2022-03-13 17:19:09,360] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 9] INFO [2022-03-13 17:19:09,361] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:09,376] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,378] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,380] jepsen worker 38 - jepsen.util 2828 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,381] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:09,410] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,410] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:09,417] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,418] jepsen worker 76 - jepsen.util 796 :invoke :write [25 49] INFO [2022-03-13 17:19:09,424] jepsen worker 62 - jepsen.util 512 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,425] jepsen worker 62 - jepsen.util 512 :invoke :write [20 22] INFO [2022-03-13 17:19:09,434] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,434] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,434] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:19:09,435] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 36] INFO [2022-03-13 17:19:09,447] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,448] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,469] jepsen worker 89 - jepsen.util 2789 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,469] jepsen worker 34 - jepsen.util 934 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,469] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:19:09,470] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:09,493] jepsen worker 23 - jepsen.util 563 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,494] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:09,517] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,517] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,517] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,517] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,518] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,519] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,519] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,520] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,520] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:19:09,520] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:19:09,521] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,522] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,523] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,523] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,523] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,524] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:09,524] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:19:09,525] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,526] jepsen worker 26 - jepsen.util 566 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,526] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,526] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:09,527] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:09,530] jepsen worker 5 - jepsen.util 2705 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,530] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,531] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,532] jepsen worker 69 - jepsen.util 789 :invoke :write [23 42] INFO [2022-03-13 17:19:09,563] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,563] jepsen worker 14 - jepsen.util 374 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,563] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,564] jepsen worker 14 - jepsen.util 374 :invoke :write [4 43] INFO [2022-03-13 17:19:09,564] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:09,565] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:09,580] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,580] jepsen worker 37 - jepsen.util 2467 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,581] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:19:09,581] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,583] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,584] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,585] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,585] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,586] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:19:09,586] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,603] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,603] jepsen worker 44 - jepsen.util 764 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,604] jepsen worker 8 - jepsen.util 728 :invoke :write [2 47] INFO [2022-03-13 17:19:09,605] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:09,607] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,608] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 35] INFO [2022-03-13 17:19:09,635] jepsen worker 13 - jepsen.util 733 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,635] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,636] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:09,636] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:09,656] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,657] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:19:09,658] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,659] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:19:09,659] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,662] jepsen worker 24 - jepsen.util 654 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,663] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:19:09,664] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:09,667] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,668] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:19:09,684] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,684] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,686] jepsen worker 77 - jepsen.util 707 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,686] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,687] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:09,688] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:09,747] jepsen worker 43 - jepsen.util 673 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,747] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:09,753] jepsen worker 88 - jepsen.util 2428 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,754] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 3] INFO [2022-03-13 17:19:09,756] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,756] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:19:09,824] jepsen worker 7 - jepsen.util 637 :fail :write [2 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,825] jepsen worker 35 - jepsen.util 935 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,826] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:19:09,826] jepsen worker 7 - jepsen.util 637 :invoke :write [2 6] INFO [2022-03-13 17:19:09,828] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,829] jepsen worker 69 - jepsen.util 789 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,829] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,830] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:19:09,838] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,840] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:09,841] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,842] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:09,843] jepsen worker 76 - jepsen.util 796 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,843] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,844] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,844] jepsen worker 76 - jepsen.util 796 :invoke :write [25 15] INFO [2022-03-13 17:19:09,845] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:09,845] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,845] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:09,846] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,846] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:09,846] jepsen worker nemesis - couchbase.util Rebalance check count: 656 INFO [2022-03-13 17:19:09,846] jepsen worker nemesis - couchbase.util Rebalance stuck count: 528 INFO [2022-03-13 17:19:09,846] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,847] jepsen worker 13 - jepsen.util 733 :invoke :write [4 26] INFO [2022-03-13 17:19:09,847] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,848] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:09,850] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,851] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:09,858] jepsen worker 3 - jepsen.util 2253 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,858] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,858] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,859] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,859] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:19:09,860] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,860] jepsen worker 75 - jepsen.util 525 :invoke :write [25 35] INFO [2022-03-13 17:19:09,862] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,862] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:09,863] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:09,864] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,865] jepsen worker 79 - jepsen.util 1159 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,865] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:19:09,867] jepsen worker 33 - jepsen.util 753 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,868] jepsen worker 33 - jepsen.util 753 :invoke :write [11 46] INFO [2022-03-13 17:19:09,870] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:09,893] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,893] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:19:09,907] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,907] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 36] INFO [2022-03-13 17:19:09,958] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,958] jepsen worker 25 - jepsen.util 565 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,958] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,959] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,959] jepsen worker 51 - jepsen.util 771 :invoke :write [17 1] INFO [2022-03-13 17:19:09,959] jepsen worker 21 - jepsen.util 831 :invoke :write [7 8] INFO [2022-03-13 17:19:09,960] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:09,961] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,961] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,961] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:19:09,961] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:09,963] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:19:09,963] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,964] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:19:09,968] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,968] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:09,970] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,970] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:19:09,982] jepsen worker 52 - jepsen.util 682 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:09,984] jepsen worker 52 - jepsen.util 682 :invoke :write [17 26] INFO [2022-03-13 17:19:09,999] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,000] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,012] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,013] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,024] jepsen worker 12 - jepsen.util 912 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,024] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,024] jepsen worker 12 - jepsen.util 912 :invoke :write [4 36] INFO [2022-03-13 17:19:10,025] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:10,041] jepsen worker 88 - jepsen.util 2428 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,042] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 40] INFO [2022-03-13 17:19:10,078] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,079] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:10,093] jepsen worker 14 - jepsen.util 374 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,093] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:10,106] jepsen worker 78 - jepsen.util 708 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,106] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,106] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,107] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,107] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,108] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:19:10,109] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:10,109] jepsen worker 61 - jepsen.util 781 :invoke :write [20 29] INFO [2022-03-13 17:19:10,111] jepsen worker 77 - jepsen.util 707 :invoke :write [25 30] INFO [2022-03-13 17:19:10,118] jepsen worker 78 - jepsen.util 708 :invoke :write [26 0] INFO [2022-03-13 17:19:10,126] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,127] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:10,128] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,128] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,128] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,129] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,131] jepsen worker 53 - jepsen.util 593 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,131] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,131] jepsen worker 53 - jepsen.util 593 :invoke :write [17 14] INFO [2022-03-13 17:19:10,132] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,132] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:10,132] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,133] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,133] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 41] INFO [2022-03-13 17:19:10,134] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,135] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:19:10,139] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,140] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,151] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,152] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:19:10,152] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,153] jepsen worker 69 - jepsen.util 789 :invoke :write [23 48] INFO [2022-03-13 17:19:10,199] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,200] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,231] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,231] jepsen worker 80 - jepsen.util 800 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,232] jepsen worker 80 - jepsen.util 800 :invoke :write [26 12] INFO [2022-03-13 17:19:10,233] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:10,238] jepsen worker 76 - jepsen.util 796 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,238] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:10,242] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,242] jepsen worker 37 - jepsen.util 2467 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,243] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,243] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 7] INFO [2022-03-13 17:19:10,244] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,244] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:19:10,246] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,247] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,248] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,248] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,248] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,249] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,250] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,250] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,251] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,252] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,254] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,254] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:19:10,281] jepsen worker 42 - jepsen.util 402 :fail :write [14 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,283] jepsen worker 42 - jepsen.util 402 :invoke :write [14 19] INFO [2022-03-13 17:19:10,287] jepsen worker 38 - jepsen.util 2828 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,287] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,287] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,288] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:19:10,288] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,289] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:19:10,318] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,319] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:10,341] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,341] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,342] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 1] INFO [2022-03-13 17:19:10,343] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:19:10,343] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,344] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,364] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,364] jepsen worker 88 - jepsen.util 2428 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,364] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:19:10,366] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 0] INFO [2022-03-13 17:19:10,371] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,373] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,373] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,375] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:10,428] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,428] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,429] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,429] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,430] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,430] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,431] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,431] jepsen worker 61 - jepsen.util 781 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,431] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,432] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,432] jepsen worker 61 - jepsen.util 781 :invoke :write [20 24] INFO [2022-03-13 17:19:10,432] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,433] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,434] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,434] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,435] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,435] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:10,435] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,437] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:19:10,438] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:19:10,443] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,443] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:10,454] jepsen worker 60 - jepsen.util 690 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,454] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,455] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:10,455] jepsen worker 60 - jepsen.util 690 :invoke :write [20 27] INFO [2022-03-13 17:19:10,455] jepsen worker 26 - jepsen.util 566 :invoke :write [8 5] INFO [2022-03-13 17:19:10,456] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:10,495] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,496] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:19:10,512] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,512] jepsen worker 69 - jepsen.util 789 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,513] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,514] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:19:10,514] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,515] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,516] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,516] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,518] jepsen worker 34 - jepsen.util 934 :invoke :write [11 41] INFO [2022-03-13 17:19:10,518] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:19:10,520] jepsen worker 33 - jepsen.util 753 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,520] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:10,530] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,531] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:19:10,549] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,550] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:19:10,561] jepsen worker 70 - jepsen.util 880 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,561] jepsen worker 70 - jepsen.util 880 :invoke :write [23 2] INFO [2022-03-13 17:19:10,562] jepsen worker 37 - jepsen.util 2467 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,563] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,572] jepsen worker 13 - jepsen.util 733 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,573] jepsen worker 13 - jepsen.util 733 :invoke :write [4 20] INFO [2022-03-13 17:19:10,619] jepsen worker 3 - jepsen.util 2253 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,619] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,620] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:19:10,620] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:10,629] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,630] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,636] jepsen worker 51 - jepsen.util 771 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,637] jepsen worker 51 - jepsen.util 771 :invoke :write [17 32] INFO [2022-03-13 17:19:10,655] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,656] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:10,667] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,668] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:19:10,677] jepsen worker 77 - jepsen.util 707 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,677] jepsen worker 77 - jepsen.util 707 :invoke :write [25 34] INFO [2022-03-13 17:19:10,729] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,729] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,730] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:19:10,730] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:19:10,736] jepsen worker 52 - jepsen.util 682 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,736] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,736] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:10,737] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:10,757] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,757] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,759] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,759] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:10,762] jepsen worker 62 - jepsen.util 512 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,763] jepsen worker 62 - jepsen.util 512 :invoke :write [20 28] INFO [2022-03-13 17:19:10,776] jepsen worker 18 - jepsen.util 4428 :info :write [6 23] :RequestTimeoutException INFO [2022-03-13 17:19:10,777] jepsen worker 18 - jepsen.util 4518 :invoke :write [6 35] INFO [2022-03-13 17:19:10,781] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,782] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,786] jepsen worker 27 - jepsen.util 4707 :info :write [9 21] :RequestTimeoutException INFO [2022-03-13 17:19:10,787] jepsen worker 27 - jepsen.util 4797 :invoke :read [9 nil] INFO [2022-03-13 17:19:10,803] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,803] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,803] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,804] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,810] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,810] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,811] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,811] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:19:10,812] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,813] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,814] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,814] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,815] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:19:10,815] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,815] jepsen worker 88 - jepsen.util 2428 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,816] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:19:10,821] jepsen worker 19 - jepsen.util 4339 :info :write [6 27] :RequestTimeoutException INFO [2022-03-13 17:19:10,821] jepsen worker 19 - jepsen.util 4429 :invoke :read [6 nil] INFO [2022-03-13 17:19:10,835] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,836] jepsen worker 76 - jepsen.util 796 :invoke :write [25 20] INFO [2022-03-13 17:19:10,838] jepsen worker 75 - jepsen.util 525 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,839] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:10,851] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:10,851] jepsen worker nemesis - couchbase.util Rebalance check count: 657 INFO [2022-03-13 17:19:10,851] jepsen worker nemesis - couchbase.util Rebalance stuck count: 529 INFO [2022-03-13 17:19:10,852] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,852] jepsen worker 21 - jepsen.util 831 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,852] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:10,853] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:10,866] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,866] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,866] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,867] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,867] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:19:10,868] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:19:10,869] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,870] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,872] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,873] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:19:10,874] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,875] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,878] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,878] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,879] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,879] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,880] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,880] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,880] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:19:10,880] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,881] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,881] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:10,882] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:19:10,883] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,883] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:19:10,885] jepsen worker 78 - jepsen.util 708 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,885] jepsen worker 78 - jepsen.util 708 :invoke :write [26 4] INFO [2022-03-13 17:19:10,896] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,897] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,898] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:10,899] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,900] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:19:10,938] jepsen worker 39 - jepsen.util 4809 :info :write [13 9] :RequestTimeoutException INFO [2022-03-13 17:19:10,939] jepsen worker 39 - jepsen.util 4899 :invoke :write [13 21] INFO [2022-03-13 17:19:10,939] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,940] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:19:10,956] jepsen worker 80 - jepsen.util 800 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,956] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,956] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,957] jepsen worker 33 - jepsen.util 753 :invoke :write [11 31] INFO [2022-03-13 17:19:10,958] jepsen worker 35 - jepsen.util 935 :invoke :write [11 32] INFO [2022-03-13 17:19:10,958] jepsen worker 80 - jepsen.util 800 :invoke :write [26 10] INFO [2022-03-13 17:19:10,964] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,965] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:10,973] jepsen worker 8 - jepsen.util 728 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:10,974] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:11,000] jepsen worker 53 - jepsen.util 593 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,000] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:11,000] jepsen worker 53 - jepsen.util 593 :invoke :write [17 45] INFO [2022-03-13 17:19:11,002] jepsen worker 43 - jepsen.util 673 :invoke :write [14 9] INFO [2022-03-13 17:19:11,017] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,017] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:11,028] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,029] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:19:11,044] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,045] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:19:11,053] jepsen worker 34 - jepsen.util 934 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,053] jepsen worker 34 - jepsen.util 934 :invoke :write [11 37] INFO [2022-03-13 17:19:11,062] jepsen worker 13 - jepsen.util 733 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,063] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,063] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:11,064] jepsen worker 75 - jepsen.util 525 :invoke :write [25 36] INFO [2022-03-13 17:19:11,074] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,075] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:11,076] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,077] jepsen worker 69 - jepsen.util 789 :invoke :write [23 43] INFO [2022-03-13 17:19:11,077] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,078] jepsen worker 71 - jepsen.util 611 :invoke :write [23 11] INFO [2022-03-13 17:19:11,082] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,082] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:11,084] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,085] jepsen worker 23 - jepsen.util 563 :invoke :write [7 46] INFO [2022-03-13 17:19:11,099] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,100] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,101] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:19:11,102] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:19:11,109] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,110] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,122] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,122] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:11,143] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,144] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:19:11,145] jepsen worker 51 - jepsen.util 771 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,146] jepsen worker 51 - jepsen.util 771 :invoke :write [17 5] INFO [2022-03-13 17:19:11,164] jepsen worker 7 - jepsen.util 637 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,164] jepsen worker 84 - jepsen.util 4854 :info :write [28 39] :RequestTimeoutException INFO [2022-03-13 17:19:11,164] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,165] jepsen worker 84 - jepsen.util 4944 :invoke :write [28 14] INFO [2022-03-13 17:19:11,166] jepsen worker 7 - jepsen.util 637 :invoke :write [2 18] INFO [2022-03-13 17:19:11,167] jepsen worker 6 - jepsen.util 636 :invoke :write [2 23] INFO [2022-03-13 17:19:11,172] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,172] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,174] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:11,174] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,203] jepsen worker 71 - jepsen.util 611 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,204] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:11,206] jepsen worker 60 - jepsen.util 690 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,207] jepsen worker 60 - jepsen.util 690 :invoke :write [20 15] INFO [2022-03-13 17:19:11,228] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,229] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:11,232] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,233] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:19:11,250] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,251] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:11,265] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,266] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,266] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:19:11,266] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,267] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:11,267] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:19:11,284] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,284] jepsen worker 76 - jepsen.util 796 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,285] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:11,285] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:11,297] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,297] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,298] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 24] INFO [2022-03-13 17:19:11,298] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:11,304] jepsen worker 61 - jepsen.util 781 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,304] jepsen worker 61 - jepsen.util 781 :invoke :write [20 42] INFO [2022-03-13 17:19:11,330] jepsen worker 69 - jepsen.util 789 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,331] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:19:11,331] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,331] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:19:11,336] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,337] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:19:11,346] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,346] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,347] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 4] INFO [2022-03-13 17:19:11,348] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:11,394] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,394] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,394] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,394] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,395] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:11,396] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:19:11,396] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:11,396] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,410] jepsen worker 12 - jepsen.util 912 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,411] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:11,435] jepsen worker 26 - jepsen.util 566 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,436] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:11,436] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,436] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,437] jepsen worker 52 - jepsen.util 682 :invoke :write [17 47] INFO [2022-03-13 17:19:11,437] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:11,443] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,444] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,444] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:19:11,445] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:11,453] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,453] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:11,492] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,493] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:19:11,499] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,499] jepsen worker 35 - jepsen.util 935 :fail :write [11 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,499] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:19:11,500] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:11,507] jepsen worker 59 - jepsen.util 4559 :info :write [19 30] :RequestTimeoutException INFO [2022-03-13 17:19:11,507] jepsen worker 59 - jepsen.util 4649 :invoke :write [19 4] INFO [2022-03-13 17:19:11,516] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,516] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,517] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,518] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,518] jepsen worker 75 - jepsen.util 525 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,518] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,519] jepsen worker 77 - jepsen.util 707 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,519] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:11,520] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,521] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:19:11,521] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,521] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,522] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:11,522] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,523] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,523] jepsen worker 77 - jepsen.util 707 :invoke :write [25 48] INFO [2022-03-13 17:19:11,523] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,524] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:19:11,527] jepsen worker 76 - jepsen.util 796 :invoke :write [25 37] INFO [2022-03-13 17:19:11,528] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,531] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,531] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,532] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,541] jepsen worker 13 - jepsen.util 733 :invoke :write [4 23] INFO [2022-03-13 17:19:11,541] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,542] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:11,543] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,543] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:19:11,554] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,556] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:11,556] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,557] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:11,563] jepsen worker 37 - jepsen.util 2467 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,563] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:11,564] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,564] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,565] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:11,566] jepsen worker 69 - jepsen.util 789 :invoke :write [23 41] INFO [2022-03-13 17:19:11,570] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,579] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:19:11,596] jepsen worker 42 - jepsen.util 402 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,596] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:11,597] jepsen worker 42 - jepsen.util 402 :invoke :write [14 10] INFO [2022-03-13 17:19:11,597] jepsen worker 44 - jepsen.util 764 :invoke :write [14 39] INFO [2022-03-13 17:19:11,600] jepsen worker 33 - jepsen.util 753 :fail :write [11 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,601] jepsen worker 33 - jepsen.util 753 :invoke :write [11 38] INFO [2022-03-13 17:19:11,644] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,644] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,649] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,649] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:19:11,658] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,658] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,659] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:19:11,659] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:11,673] jepsen worker 53 - jepsen.util 593 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,675] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:11,684] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,685] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:11,697] jepsen worker 69 - jepsen.util 789 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,697] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:19:11,701] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,702] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:19:11,711] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,711] jepsen worker 78 - jepsen.util 708 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,712] jepsen worker 78 - jepsen.util 708 :invoke :write [26 46] INFO [2022-03-13 17:19:11,713] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:11,713] jepsen worker 86 - jepsen.util 4406 :ok :read [28 44] INFO [2022-03-13 17:19:11,714] jepsen worker 86 - jepsen.util 4406 :invoke :write [28 16] INFO [2022-03-13 17:19:11,716] jepsen worker 80 - jepsen.util 800 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,717] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:11,760] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,760] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,760] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:19:11,761] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:19:11,761] jepsen worker 88 - jepsen.util 2428 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,761] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,762] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:11,763] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 37] INFO [2022-03-13 17:19:11,795] jepsen worker 17 - jepsen.util 4787 :ok :read [5 43] INFO [2022-03-13 17:19:11,796] jepsen worker 17 - jepsen.util 4787 :invoke :write [5 21] INFO [2022-03-13 17:19:11,798] jepsen worker 83 - jepsen.util 4223 :fail :read [27 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,799] jepsen worker 83 - jepsen.util 4223 :invoke :read [27 nil] INFO [2022-03-13 17:19:11,804] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,805] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:11,806] jepsen worker 70 - jepsen.util 880 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,806] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,806] jepsen worker 76 - jepsen.util 796 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,807] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:11,808] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:11,808] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:11,811] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,812] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:19:11,823] jepsen worker 51 - jepsen.util 771 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,823] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,823] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,823] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,824] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,824] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:11,824] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:11,825] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:11,826] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:19:11,826] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:11,831] jepsen worker 85 - jepsen.util 4225 :ok :read [28 44] INFO [2022-03-13 17:19:11,833] jepsen worker 85 - jepsen.util 4225 :invoke :read [28 nil] INFO [2022-03-13 17:19:11,857] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:11,857] jepsen worker nemesis - couchbase.util Rebalance check count: 658 INFO [2022-03-13 17:19:11,857] jepsen worker nemesis - couchbase.util Rebalance stuck count: 530 INFO [2022-03-13 17:19:11,865] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,866] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:19:11,906] jepsen worker 34 - jepsen.util 934 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,906] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,907] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,908] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:19:11,908] jepsen worker 34 - jepsen.util 934 :invoke :write [11 10] INFO [2022-03-13 17:19:11,909] jepsen worker 43 - jepsen.util 673 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,909] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:11,909] jepsen worker 23 - jepsen.util 563 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,910] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:11,910] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:19:11,928] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,928] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,929] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:19:11,934] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:19:11,951] jepsen worker 37 - jepsen.util 2467 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,951] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,952] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 38] INFO [2022-03-13 17:19:11,953] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 11] INFO [2022-03-13 17:19:11,959] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,959] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:11,974] jepsen worker 31 - jepsen.util 4621 :info :write [10 6] :RequestTimeoutException INFO [2022-03-13 17:19:11,975] jepsen worker 31 - jepsen.util 4711 :invoke :read [10 nil] INFO [2022-03-13 17:19:11,976] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,976] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:11,977] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:19:11,979] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:19:11,987] jepsen worker 20 - jepsen.util 4790 :ok :read [6 16] INFO [2022-03-13 17:19:11,988] jepsen worker 20 - jepsen.util 4790 :invoke :read [6 nil] INFO [2022-03-13 17:19:12,014] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,015] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,015] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:12,016] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:12,020] jepsen worker 13 - jepsen.util 733 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,020] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,020] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,022] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:12,023] jepsen worker 76 - jepsen.util 796 :invoke :write [25 9] INFO [2022-03-13 17:19:12,024] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:19:12,031] jepsen worker 62 - jepsen.util 512 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,031] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,032] jepsen worker 62 - jepsen.util 512 :invoke :write [20 46] INFO [2022-03-13 17:19:12,033] jepsen worker 8 - jepsen.util 728 :invoke :write [2 27] INFO [2022-03-13 17:19:12,068] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,069] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:19:12,073] jepsen worker 55 - jepsen.util 4105 :ok :read [18 21] INFO [2022-03-13 17:19:12,074] jepsen worker 55 - jepsen.util 4105 :invoke :read [18 nil] INFO [2022-03-13 17:19:12,076] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,077] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:19:12,086] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,086] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,087] jepsen worker 74 - jepsen.util 4394 :ok :read [24 24] INFO [2022-03-13 17:19:12,088] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:12,089] jepsen worker 74 - jepsen.util 4394 :invoke :write [24 37] INFO [2022-03-13 17:19:12,089] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,090] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:19:12,091] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:19:12,104] jepsen worker 28 - jepsen.util 4618 :ok :read [9 26] INFO [2022-03-13 17:19:12,106] jepsen worker 28 - jepsen.util 4618 :invoke :write [9 9] INFO [2022-03-13 17:19:12,112] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,112] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:19:12,112] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,114] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:12,123] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,124] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:19:12,126] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,126] jepsen worker 60 - jepsen.util 690 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,127] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,128] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:12,128] jepsen worker 26 - jepsen.util 566 :invoke :write [8 14] INFO [2022-03-13 17:19:12,128] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:12,131] jepsen worker 19 - jepsen.util 4429 :ok :read [6 16] INFO [2022-03-13 17:19:12,131] jepsen worker 19 - jepsen.util 4429 :invoke :read [6 nil] INFO [2022-03-13 17:19:12,142] jepsen worker 20 - jepsen.util 4790 :ok :read [6 16] INFO [2022-03-13 17:19:12,143] jepsen worker 20 - jepsen.util 4790 :invoke :write [6 47] INFO [2022-03-13 17:19:12,161] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,162] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,162] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,163] jepsen worker 80 - jepsen.util 800 :invoke :write [26 3] INFO [2022-03-13 17:19:12,164] jepsen worker 33 - jepsen.util 753 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,165] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,166] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:12,174] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:12,181] jepsen worker 83 - jepsen.util 4223 :ok :read [27 4] INFO [2022-03-13 17:19:12,182] jepsen worker 83 - jepsen.util 4223 :invoke :read [27 nil] INFO [2022-03-13 17:19:12,191] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,192] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 15] INFO [2022-03-13 17:19:12,196] jepsen worker 55 - jepsen.util 4105 :ok :read [18 21] INFO [2022-03-13 17:19:12,197] jepsen worker 47 - jepsen.util 4457 :ok :read [15 48] INFO [2022-03-13 17:19:12,197] jepsen worker 55 - jepsen.util 4105 :invoke :read [18 nil] INFO [2022-03-13 17:19:12,198] jepsen worker 47 - jepsen.util 4457 :invoke :read [15 nil] INFO [2022-03-13 17:19:12,199] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,199] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,204] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,204] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,209] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,214] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:19:12,216] jepsen worker 61 - jepsen.util 781 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,219] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:12,224] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,225] jepsen worker 19 - jepsen.util 4429 :ok :read [6 16] INFO [2022-03-13 17:19:12,225] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:19:12,226] jepsen worker 19 - jepsen.util 4429 :invoke :read [6 nil] INFO [2022-03-13 17:19:12,256] jepsen worker 85 - jepsen.util 4225 :ok :read [28 44] INFO [2022-03-13 17:19:12,257] jepsen worker 85 - jepsen.util 4225 :invoke :read [28 nil] INFO [2022-03-13 17:19:12,259] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,260] jepsen worker 71 - jepsen.util 611 :invoke :write [23 6] INFO [2022-03-13 17:19:12,269] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,269] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,270] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,270] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,270] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:19:12,271] jepsen worker 24 - jepsen.util 654 :invoke :write [8 27] INFO [2022-03-13 17:19:12,272] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:12,276] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,286] jepsen worker 52 - jepsen.util 682 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,286] jepsen worker 88 - jepsen.util 2428 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,287] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:12,288] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:12,289] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:12,289] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,289] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,289] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,290] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:12,291] jepsen worker 43 - jepsen.util 673 :invoke :write [14 13] INFO [2022-03-13 17:19:12,292] jepsen worker 22 - jepsen.util 652 :invoke :write [7 4] INFO [2022-03-13 17:19:12,293] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:12,312] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,313] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:12,340] jepsen worker 7 - jepsen.util 637 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,340] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:12,356] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,357] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,357] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,357] jepsen worker 4 - jepsen.util 2614 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,358] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:12,359] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,360] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:19:12,360] jepsen worker 55 - jepsen.util 4105 :ok :read [18 21] INFO [2022-03-13 17:19:12,361] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:19:12,362] jepsen worker 55 - jepsen.util 4105 :invoke :write [18 6] INFO [2022-03-13 17:19:12,367] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,367] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,369] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,369] jepsen worker 19 - jepsen.util 4429 :ok :read [6 16] INFO [2022-03-13 17:19:12,370] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,371] jepsen worker 19 - jepsen.util 4429 :invoke :read [6 nil] INFO [2022-03-13 17:19:12,371] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,372] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,373] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,373] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,374] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,375] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:12,376] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,376] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:19:12,383] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,384] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:19:12,385] jepsen worker 83 - jepsen.util 4223 :ok :read [27 4] INFO [2022-03-13 17:19:12,386] jepsen worker 83 - jepsen.util 4223 :invoke :write [27 11] INFO [2022-03-13 17:19:12,414] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,414] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,416] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:12,416] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:19:12,427] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,428] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,428] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:19:12,428] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:12,439] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,440] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:12,441] jepsen worker 26 - jepsen.util 566 :fail :write [8 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,441] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,441] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,441] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,442] jepsen worker 26 - jepsen.util 566 :invoke :write [8 21] INFO [2022-03-13 17:19:12,443] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:12,444] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:12,445] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:12,454] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,454] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:12,457] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,457] jepsen worker 37 - jepsen.util 2467 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,458] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:12,459] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:12,461] jepsen worker 85 - jepsen.util 4225 :ok :read [28 44] INFO [2022-03-13 17:19:12,461] jepsen worker 85 - jepsen.util 4225 :invoke :write [28 15] INFO [2022-03-13 17:19:12,496] jepsen worker 77 - jepsen.util 707 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,496] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,497] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:12,498] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:12,515] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,516] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:19:12,516] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,516] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,517] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 47] INFO [2022-03-13 17:19:12,518] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 11] INFO [2022-03-13 17:19:12,525] jepsen worker 76 - jepsen.util 796 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,525] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,525] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,526] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,526] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:12,527] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 40] INFO [2022-03-13 17:19:12,527] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:12,528] jepsen worker 76 - jepsen.util 796 :invoke :write [25 25] INFO [2022-03-13 17:19:12,536] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,537] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:12,538] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,539] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:12,542] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,542] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,542] jepsen worker 75 - jepsen.util 525 :invoke :write [25 39] INFO [2022-03-13 17:19:12,546] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:12,549] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,549] jepsen worker 71 - jepsen.util 611 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,550] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:19:12,551] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:12,557] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,558] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 47] INFO [2022-03-13 17:19:12,560] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,561] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:12,563] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,563] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:19:12,574] jepsen worker 34 - jepsen.util 934 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,574] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,575] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,575] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,575] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,575] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:12,576] jepsen worker 51 - jepsen.util 771 :invoke :write [17 7] INFO [2022-03-13 17:19:12,576] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:12,577] jepsen worker 33 - jepsen.util 753 :invoke :write [11 24] INFO [2022-03-13 17:19:12,580] jepsen worker 34 - jepsen.util 934 :invoke :write [11 22] INFO [2022-03-13 17:19:12,612] jepsen worker 6 - jepsen.util 636 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,612] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,613] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:12,613] jepsen worker 6 - jepsen.util 636 :invoke :write [2 32] INFO [2022-03-13 17:19:12,628] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,629] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:12,629] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,630] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:19:12,632] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,633] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:19:12,645] jepsen worker 43 - jepsen.util 673 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,645] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,645] jepsen worker 43 - jepsen.util 673 :invoke :write [14 27] INFO [2022-03-13 17:19:12,646] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:12,668] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,669] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:19:12,669] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,670] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:19:12,713] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,714] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:19:12,731] jepsen worker 78 - jepsen.util 708 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,732] jepsen worker 78 - jepsen.util 708 :invoke :write [26 9] INFO [2022-03-13 17:19:12,736] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,737] jepsen worker 38 - jepsen.util 2828 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,737] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:12,738] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:12,743] jepsen worker 80 - jepsen.util 800 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,744] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:12,746] jepsen worker 16 - jepsen.util 4336 :info :write [5 12] :RequestTimeoutException INFO [2022-03-13 17:19:12,748] jepsen worker 16 - jepsen.util 4426 :invoke :write [5 31] INFO [2022-03-13 17:19:12,756] jepsen worker 3 - jepsen.util 2253 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,757] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:19:12,805] jepsen worker 88 - jepsen.util 2428 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,806] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:12,808] jepsen worker 46 - jepsen.util 4636 :info :write [15 22] :SyncWriteAmbiguous INFO [2022-03-13 17:19:12,809] jepsen worker 46 - jepsen.util 4726 :invoke :write [15 47] INFO [2022-03-13 17:19:12,835] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,836] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:19:12,842] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,842] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:19:12,863] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:12,863] jepsen worker nemesis - couchbase.util Rebalance check count: 659 INFO [2022-03-13 17:19:12,863] jepsen worker nemesis - couchbase.util Rebalance stuck count: 531 INFO [2022-03-13 17:19:12,865] jepsen worker 44 - jepsen.util 764 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,866] jepsen worker 44 - jepsen.util 764 :invoke :write [14 35] INFO [2022-03-13 17:19:12,868] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,868] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,871] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:19:12,872] jepsen worker 69 - jepsen.util 789 :invoke :write [23 23] INFO [2022-03-13 17:19:12,880] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,880] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:12,886] jepsen worker 22 - jepsen.util 652 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,886] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,886] jepsen worker 76 - jepsen.util 796 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,886] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,886] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,887] jepsen worker 76 - jepsen.util 796 :invoke :write [25 45] INFO [2022-03-13 17:19:12,887] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:19:12,888] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:12,888] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:12,889] jepsen worker 52 - jepsen.util 682 :invoke :write [17 7] INFO [2022-03-13 17:19:12,921] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,922] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:19:12,931] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,931] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 37] INFO [2022-03-13 17:19:12,938] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,939] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:12,986] jepsen worker 45 - jepsen.util 4815 :ok :read [15 48] INFO [2022-03-13 17:19:12,987] jepsen worker 45 - jepsen.util 4815 :invoke :write [15 20] INFO [2022-03-13 17:19:12,988] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,989] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,988] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,989] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:12,989] jepsen worker 23 - jepsen.util 563 :invoke :write [7 5] INFO [2022-03-13 17:19:12,991] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:12,991] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:19:12,998] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:13,001] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,002] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,020] jepsen worker 19 - jepsen.util 4429 :ok :read [6 16] INFO [2022-03-13 17:19:13,022] jepsen worker 72 - jepsen.util 4662 :ok :read [24 24] INFO [2022-03-13 17:19:13,022] jepsen worker 19 - jepsen.util 4429 :invoke :read [6 nil] INFO [2022-03-13 17:19:13,023] jepsen worker 72 - jepsen.util 4662 :invoke :write [24 44] INFO [2022-03-13 17:19:13,037] jepsen worker 37 - jepsen.util 2467 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,038] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,038] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,038] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,039] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:19:13,042] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,044] jepsen worker 71 - jepsen.util 611 :invoke :write [23 45] INFO [2022-03-13 17:19:13,046] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:13,076] jepsen worker 24 - jepsen.util 654 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,077] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,077] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,077] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,078] jepsen worker 24 - jepsen.util 654 :invoke :write [8 38] INFO [2022-03-13 17:19:13,078] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,078] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:19:13,079] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:13,080] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,080] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:19:13,081] jepsen worker 25 - jepsen.util 565 :invoke :write [8 48] INFO [2022-03-13 17:19:13,081] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,084] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,085] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 9] INFO [2022-03-13 17:19:13,103] jepsen worker 42 - jepsen.util 402 :fail :write [14 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,103] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,104] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:13,104] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:13,117] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,117] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,118] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:19:13,119] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,123] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 48] INFO [2022-03-13 17:19:13,124] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,128] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,129] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,131] jepsen worker 64 - jepsen.util 4384 :info :write [21 9] :SyncWriteAmbiguous INFO [2022-03-13 17:19:13,131] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,131] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,132] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:13,133] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,134] jepsen worker 64 - jepsen.util 4474 :invoke :write [21 36] INFO [2022-03-13 17:19:13,135] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,135] jepsen worker 75 - jepsen.util 525 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,136] jepsen worker 75 - jepsen.util 525 :invoke :write [25 41] INFO [2022-03-13 17:19:13,137] jepsen worker 26 - jepsen.util 566 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,138] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:13,139] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,139] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:13,139] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,141] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,142] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,145] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,146] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:19:13,166] jepsen worker 33 - jepsen.util 753 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,167] jepsen worker 33 - jepsen.util 753 :invoke :write [11 19] INFO [2022-03-13 17:19:13,168] jepsen worker 19 - jepsen.util 4429 :ok :read [6 16] INFO [2022-03-13 17:19:13,169] jepsen worker 19 - jepsen.util 4429 :invoke :write [6 8] INFO [2022-03-13 17:19:13,196] jepsen worker 4 - jepsen.util 2614 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,196] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,196] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,197] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:19:13,198] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:13,198] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 12] INFO [2022-03-13 17:19:13,204] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,205] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:19:13,212] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,213] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:19:13,215] jepsen worker 89 - jepsen.util 2789 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,216] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,228] jepsen worker 78 - jepsen.util 708 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,228] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,228] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,231] jepsen worker 32 - jepsen.util 4712 :info :write [10 34] :SyncWriteAmbiguous INFO [2022-03-13 17:19:13,231] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:13,231] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:13,232] jepsen worker 32 - jepsen.util 4802 :invoke :read [10 nil] INFO [2022-03-13 17:19:13,232] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:13,281] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,281] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,281] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:19:13,282] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:13,284] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,285] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:19:13,303] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,304] jepsen worker 80 - jepsen.util 800 :invoke :write [26 31] INFO [2022-03-13 17:19:13,305] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,305] jepsen worker 76 - jepsen.util 796 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,306] jepsen worker 13 - jepsen.util 733 :invoke :write [4 3] INFO [2022-03-13 17:19:13,306] jepsen worker 51 - jepsen.util 771 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,307] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:13,307] jepsen worker 51 - jepsen.util 771 :invoke :write [17 41] INFO [2022-03-13 17:19:13,311] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,311] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,312] jepsen worker 60 - jepsen.util 690 :invoke :write [20 21] INFO [2022-03-13 17:19:13,316] jepsen worker 53 - jepsen.util 593 :invoke :write [17 0] INFO [2022-03-13 17:19:13,320] jepsen worker 32 - jepsen.util 4802 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,311] jepsen worker 43 - jepsen.util 673 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,322] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:13,326] jepsen worker 32 - jepsen.util 4802 :invoke :read [10 nil] INFO [2022-03-13 17:19:13,330] jepsen worker 32 - jepsen.util 4802 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,330] jepsen worker 62 - jepsen.util 512 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,331] jepsen worker 32 - jepsen.util 4802 :invoke :read [10 nil] INFO [2022-03-13 17:19:13,335] jepsen worker 62 - jepsen.util 512 :invoke :write [20 47] INFO [2022-03-13 17:19:13,336] jepsen worker 32 - jepsen.util 4802 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,342] jepsen worker 32 - jepsen.util 4802 :invoke :read [10 nil] INFO [2022-03-13 17:19:13,356] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,356] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:13,369] jepsen worker 32 - jepsen.util 4802 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,370] jepsen worker 32 - jepsen.util 4802 :invoke :read [10 nil] INFO [2022-03-13 17:19:13,373] jepsen worker 32 - jepsen.util 4802 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,374] jepsen worker 32 - jepsen.util 4802 :invoke :write [10 21] INFO [2022-03-13 17:19:13,379] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,380] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:19:13,390] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,390] jepsen worker 34 - jepsen.util 934 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,390] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,391] jepsen worker 21 - jepsen.util 831 :invoke :write [7 1] INFO [2022-03-13 17:19:13,391] jepsen worker 34 - jepsen.util 934 :invoke :write [11 45] INFO [2022-03-13 17:19:13,392] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:19:13,435] jepsen worker 3 - jepsen.util 2253 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,436] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:19:13,443] jepsen worker 23 - jepsen.util 563 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,444] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:13,463] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,464] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:19:13,477] jepsen worker 24 - jepsen.util 654 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,478] jepsen worker 24 - jepsen.util 654 :invoke :write [8 27] INFO [2022-03-13 17:19:13,510] jepsen worker 88 - jepsen.util 2428 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,510] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,510] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,511] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 44] INFO [2022-03-13 17:19:13,512] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:19:13,512] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:19:13,518] jepsen worker 63 - jepsen.util 4653 :info :write [21 30] :RequestTimeoutException INFO [2022-03-13 17:19:13,519] jepsen worker 63 - jepsen.util 4743 :invoke :write [21 15] INFO [2022-03-13 17:19:13,522] jepsen worker 75 - jepsen.util 525 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,522] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,523] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:13,523] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:13,523] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,524] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:13,531] jepsen worker 30 - jepsen.util 4530 :info :write [10 10] :RequestTimeoutException INFO [2022-03-13 17:19:13,531] jepsen worker 30 - jepsen.util 4620 :invoke :read [10 nil] INFO [2022-03-13 17:19:13,550] jepsen worker 9 - jepsen.util 4149 :info :write [3 45] :RequestTimeoutException INFO [2022-03-13 17:19:13,551] jepsen worker 9 - jepsen.util 4239 :invoke :read [3 nil] INFO [2022-03-13 17:19:13,561] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,561] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:13,562] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,563] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:19:13,565] jepsen worker 8 - jepsen.util 728 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,566] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,566] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,566] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:13,567] jepsen worker 12 - jepsen.util 912 :invoke :write [4 39] INFO [2022-03-13 17:19:13,567] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:13,577] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,577] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,578] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:13,579] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 3] INFO [2022-03-13 17:19:13,581] jepsen worker 69 - jepsen.util 789 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,582] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:19:13,613] jepsen worker 25 - jepsen.util 565 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,614] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:19:13,671] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,672] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,672] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,672] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,672] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,673] jepsen worker 77 - jepsen.util 707 :invoke :write [25 46] INFO [2022-03-13 17:19:13,673] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:19:13,674] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:13,675] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:13,676] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,679] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:13,681] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:19:13,697] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,697] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,698] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,698] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,699] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:19:13,703] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:19:13,703] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 42] INFO [2022-03-13 17:19:13,704] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,704] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:19:13,705] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:19:13,726] jepsen worker 13 - jepsen.util 733 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,726] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,728] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:13,728] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:13,730] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,730] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,731] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:13,731] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:13,732] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,732] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,734] jepsen worker 76 - jepsen.util 796 :invoke :write [25 8] INFO [2022-03-13 17:19:13,734] jepsen worker 13 - jepsen.util 733 :invoke :write [4 21] INFO [2022-03-13 17:19:13,734] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,735] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:19:13,749] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,749] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:19:13,756] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,756] jepsen worker 52 - jepsen.util 682 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,756] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:13,756] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:13,757] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:13,757] jepsen worker 52 - jepsen.util 682 :invoke :write [17 46] INFO [2022-03-13 17:19:13,767] jepsen worker 6 - jepsen.util 636 :fail :write [2 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,768] jepsen worker 6 - jepsen.util 636 :invoke :write [2 15] INFO [2022-03-13 17:19:13,776] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,776] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,777] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 45] INFO [2022-03-13 17:19:13,778] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:13,780] jepsen worker 53 - jepsen.util 593 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,782] jepsen worker 33 - jepsen.util 753 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,783] jepsen worker 33 - jepsen.util 753 :invoke :write [11 0] INFO [2022-03-13 17:19:13,784] jepsen worker 53 - jepsen.util 593 :invoke :write [17 42] INFO [2022-03-13 17:19:13,811] jepsen worker 88 - jepsen.util 2428 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,811] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,811] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,812] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:13,813] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:19:13,814] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 30] INFO [2022-03-13 17:19:13,831] jepsen worker 31 - jepsen.util 4711 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,831] jepsen worker 30 - jepsen.util 4620 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,831] jepsen worker 31 - jepsen.util 4711 :invoke :write [10 3] INFO [2022-03-13 17:19:13,832] jepsen worker 30 - jepsen.util 4620 :invoke :read [10 nil] INFO [2022-03-13 17:19:13,852] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,853] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 4] INFO [2022-03-13 17:19:13,862] jepsen worker 3 - jepsen.util 2253 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,863] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:19:13,868] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:13,868] jepsen worker nemesis - couchbase.util Rebalance check count: 660 INFO [2022-03-13 17:19:13,868] jepsen worker nemesis - couchbase.util Rebalance stuck count: 532 INFO [2022-03-13 17:19:13,873] jepsen worker 51 - jepsen.util 771 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,874] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:13,894] jepsen worker 37 - jepsen.util 2467 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,895] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 45] INFO [2022-03-13 17:19:13,915] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,915] jepsen worker 44 - jepsen.util 764 :fail :write [14 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,915] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:13,916] jepsen worker 70 - jepsen.util 880 :invoke :write [23 28] INFO [2022-03-13 17:19:13,916] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:13,917] jepsen worker 42 - jepsen.util 402 :invoke :write [14 20] INFO [2022-03-13 17:19:13,926] jepsen worker 60 - jepsen.util 690 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,926] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:13,926] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,927] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,928] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:13,928] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:13,928] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:13,929] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:13,954] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,955] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:13,967] jepsen worker 80 - jepsen.util 800 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,968] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:13,970] jepsen worker 30 - jepsen.util 4620 :ok :read [10 :nil] INFO [2022-03-13 17:19:13,971] jepsen worker 30 - jepsen.util 4620 :invoke :write [10 8] INFO [2022-03-13 17:19:13,971] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,972] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:13,982] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,983] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:13,993] jepsen worker 76 - jepsen.util 796 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:13,993] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:14,017] jepsen worker 29 - jepsen.util 4979 :info :write [9 13] :RequestTimeoutException INFO [2022-03-13 17:19:14,018] jepsen worker 29 - jepsen.util 5069 :invoke :read [9 nil] INFO [2022-03-13 17:19:14,043] jepsen worker 89 - jepsen.util 2789 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,043] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,043] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,044] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:14,048] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,049] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,049] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,049] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:14,050] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,051] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,052] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,052] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,053] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:14,053] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:19:14,054] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,055] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:19:14,087] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,088] jepsen worker 24 - jepsen.util 654 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,089] jepsen worker 24 - jepsen.util 654 :invoke :write [8 12] INFO [2022-03-13 17:19:14,089] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:19:14,097] jepsen worker 62 - jepsen.util 512 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,098] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,098] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,098] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:14,099] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:14,099] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:14,121] jepsen worker 52 - jepsen.util 682 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,121] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:19:14,123] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,123] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,124] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,124] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:19:14,147] jepsen worker 88 - jepsen.util 2428 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,147] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,147] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,148] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 31] INFO [2022-03-13 17:19:14,148] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,149] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,152] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,152] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,153] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:14,159] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:19:14,160] jepsen worker 77 - jepsen.util 707 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,161] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:14,171] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,172] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:14,183] jepsen worker 13 - jepsen.util 733 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,184] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:19:14,206] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,206] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,206] jepsen worker 40 - jepsen.util 4810 :info :write [13 1] :RequestTimeoutException INFO [2022-03-13 17:19:14,206] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,208] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:19:14,208] jepsen worker 40 - jepsen.util 4900 :invoke :read [13 nil] INFO [2022-03-13 17:19:14,212] jepsen worker 49 - jepsen.util 3559 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:19:14,213] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 39] INFO [2022-03-13 17:19:14,214] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,214] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,248] jepsen worker 54 - jepsen.util 4284 :info :write [18 40] :RequestTimeoutException INFO [2022-03-13 17:19:14,248] jepsen worker 54 - jepsen.util 4374 :invoke :write [18 11] INFO [2022-03-13 17:19:14,269] jepsen worker 12 - jepsen.util 912 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,270] jepsen worker 12 - jepsen.util 912 :invoke :write [4 34] INFO [2022-03-13 17:19:14,281] jepsen worker 34 - jepsen.util 934 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,281] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:14,287] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,287] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,288] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:14,290] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,296] jepsen worker 37 - jepsen.util 2467 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,297] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:14,313] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,313] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,314] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:14,315] jepsen worker 61 - jepsen.util 781 :invoke :write [20 30] INFO [2022-03-13 17:19:14,336] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,336] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,337] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:14,337] jepsen worker 75 - jepsen.util 525 :invoke :write [25 23] INFO [2022-03-13 17:19:14,338] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,339] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:14,339] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,339] jepsen worker 53 - jepsen.util 593 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,339] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,339] jepsen worker 22 - jepsen.util 652 :invoke :write [7 10] INFO [2022-03-13 17:19:14,339] jepsen worker 60 - jepsen.util 690 :invoke :write [20 37] INFO [2022-03-13 17:19:14,340] jepsen worker 26 - jepsen.util 566 :invoke :write [8 6] INFO [2022-03-13 17:19:14,341] jepsen worker 81 - jepsen.util 5121 :info :write [27 31] :RequestTimeoutException INFO [2022-03-13 17:19:14,342] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:14,342] jepsen worker 81 - jepsen.util 5211 :invoke :write [27 30] INFO [2022-03-13 17:19:14,343] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:14,345] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,346] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,346] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,348] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 37] INFO [2022-03-13 17:19:14,378] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,378] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,379] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:19:14,380] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 2] INFO [2022-03-13 17:19:14,406] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,407] jepsen worker 14 - jepsen.util 374 :invoke :write [4 41] INFO [2022-03-13 17:19:14,421] jepsen worker 49 - jepsen.util 3559 :fail :write [16 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,422] jepsen worker 49 - jepsen.util 3559 :invoke :read [16 nil] INFO [2022-03-13 17:19:14,472] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:14,472] jepsen worker 6 - jepsen.util 636 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,473] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,473] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,472] jepsen worker 79 - jepsen.util 1159 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,473] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,474] jepsen worker 23 - jepsen.util 563 :invoke :write [7 35] INFO [2022-03-13 17:19:14,475] jepsen worker 58 - jepsen.util 4198 :fail :read [19 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,476] jepsen worker 88 - jepsen.util 2428 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,476] jepsen worker 44 - jepsen.util 764 :invoke :write [14 13] INFO [2022-03-13 17:19:14,476] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,477] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:14,478] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:14,479] jepsen worker 6 - jepsen.util 636 :invoke :write [2 47] INFO [2022-03-13 17:19:14,480] jepsen worker 58 - jepsen.util 4198 :invoke :read [19 nil] INFO [2022-03-13 17:19:14,480] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:14,481] jepsen worker 78 - jepsen.util 708 :invoke :write [26 21] INFO [2022-03-13 17:19:14,482] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,483] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,483] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,484] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:19:14,485] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,487] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:14,487] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,488] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,488] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,488] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,489] jepsen worker 89 - jepsen.util 2789 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,489] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:14,491] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:19:14,497] jepsen worker 76 - jepsen.util 796 :invoke :write [25 26] INFO [2022-03-13 17:19:14,499] jepsen worker 13 - jepsen.util 733 :invoke :write [4 32] INFO [2022-03-13 17:19:14,505] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:14,505] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:14,506] jepsen worker 21 - jepsen.util 831 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,506] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,506] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,507] jepsen worker 21 - jepsen.util 831 :invoke :write [7 7] INFO [2022-03-13 17:19:14,508] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:19:14,509] jepsen worker 51 - jepsen.util 771 :invoke :write [17 14] INFO [2022-03-13 17:19:14,537] jepsen worker 68 - jepsen.util 4658 :info :write [22 39] :RequestTimeoutException INFO [2022-03-13 17:19:14,539] jepsen worker 68 - jepsen.util 4748 :invoke :read [22 nil] INFO [2022-03-13 17:19:14,549] jepsen worker 3 - jepsen.util 2253 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,549] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,550] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:19:14,550] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,581] jepsen worker 42 - jepsen.util 402 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,582] jepsen worker 42 - jepsen.util 402 :invoke :write [14 18] INFO [2022-03-13 17:19:14,587] jepsen worker 60 - jepsen.util 690 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,588] jepsen worker 60 - jepsen.util 690 :invoke :write [20 26] INFO [2022-03-13 17:19:14,606] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,607] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:19:14,650] jepsen worker 61 - jepsen.util 781 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,650] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,651] jepsen worker 61 - jepsen.util 781 :invoke :write [20 32] INFO [2022-03-13 17:19:14,651] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:14,653] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,654] jepsen worker 75 - jepsen.util 525 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,655] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 33] INFO [2022-03-13 17:19:14,659] jepsen worker 75 - jepsen.util 525 :invoke :write [25 38] INFO [2022-03-13 17:19:14,666] jepsen worker 33 - jepsen.util 753 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,666] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,667] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:14,667] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:19:14,687] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,688] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:19:14,694] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,694] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,695] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 22] INFO [2022-03-13 17:19:14,695] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:14,698] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,699] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,699] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,699] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,700] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:19:14,700] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 33] INFO [2022-03-13 17:19:14,702] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,702] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,703] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,703] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 49] INFO [2022-03-13 17:19:14,703] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,706] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:14,707] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,708] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:19:14,748] jepsen worker 71 - jepsen.util 611 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,749] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:19:14,773] jepsen worker 15 - jepsen.util 4425 :info :write [5 25] :SyncWriteAmbiguous INFO [2022-03-13 17:19:14,773] jepsen worker 15 - jepsen.util 4515 :invoke :read [5 nil] INFO [2022-03-13 17:19:14,785] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,786] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:19:14,796] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,796] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,798] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,799] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:19:14,830] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,830] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:14,837] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,838] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:19:14,874] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:14,874] jepsen worker nemesis - couchbase.util Rebalance check count: 661 INFO [2022-03-13 17:19:14,874] jepsen worker nemesis - couchbase.util Rebalance stuck count: 533 INFO [2022-03-13 17:19:14,917] jepsen worker 89 - jepsen.util 2789 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,918] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:19:14,926] jepsen worker 3 - jepsen.util 2253 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,926] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:14,927] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:19:14,927] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:19:15,005] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,006] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,029] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,030] jepsen worker 88 - jepsen.util 2428 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,030] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 21] INFO [2022-03-13 17:19:15,031] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:19:15,043] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,044] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:19:15,096] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,096] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:19:15,110] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:19:15,110] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:19:15,112] jepsen worker 22 - jepsen.util 652 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,112] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:15,124] jepsen worker 70 - jepsen.util 880 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,124] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,124] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,124] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,124] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:15,124] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,125] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:15,125] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,126] jepsen worker 62 - jepsen.util 512 :invoke :write [20 31] INFO [2022-03-13 17:19:15,126] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:15,127] jepsen worker 24 - jepsen.util 654 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,127] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,128] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:15,129] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:15,136] jepsen worker 13 - jepsen.util 733 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,137] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:15,140] jepsen worker 37 - jepsen.util 2467 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,141] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 32] INFO [2022-03-13 17:19:15,194] jepsen worker 76 - jepsen.util 796 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,194] jepsen worker 26 - jepsen.util 566 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,194] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,194] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,194] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,194] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:15,195] jepsen worker 43 - jepsen.util 673 :invoke :write [14 39] INFO [2022-03-13 17:19:15,196] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:15,198] jepsen worker 53 - jepsen.util 593 :invoke :write [17 14] INFO [2022-03-13 17:19:15,198] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:15,199] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:15,199] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:15,218] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,218] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,219] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,219] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,221] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,221] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,221] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:19:15,222] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,224] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,224] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,225] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,225] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,227] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,227] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,228] jepsen worker 11 - jepsen.util 4421 :info :write [3 1] :RequestTimeoutException INFO [2022-03-13 17:19:15,228] jepsen worker 11 - jepsen.util 4511 :invoke :write [3 34] INFO [2022-03-13 17:19:15,229] jepsen worker 56 - jepsen.util 4466 :info :write [18 0] :RequestTimeoutException INFO [2022-03-13 17:19:15,230] jepsen worker 56 - jepsen.util 4556 :invoke :read [18 nil] INFO [2022-03-13 17:19:15,234] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,235] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:19:15,242] jepsen worker 23 - jepsen.util 563 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,243] jepsen worker 23 - jepsen.util 563 :invoke :write [7 36] INFO [2022-03-13 17:19:15,263] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,263] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:15,282] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,282] jepsen worker 60 - jepsen.util 690 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,282] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,282] jepsen worker 75 - jepsen.util 525 :fail :write [25 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,282] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,283] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,283] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,283] jepsen worker 76 - jepsen.util 796 :invoke :write [25 11] INFO [2022-03-13 17:19:15,285] jepsen worker 26 - jepsen.util 566 :invoke :write [8 36] INFO [2022-03-13 17:19:15,286] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,287] jepsen worker 52 - jepsen.util 682 :invoke :write [17 49] INFO [2022-03-13 17:19:15,287] jepsen worker 75 - jepsen.util 525 :invoke :write [25 13] INFO [2022-03-13 17:19:15,287] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:15,288] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:15,289] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:15,289] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,293] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,293] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,293] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,294] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:19:15,294] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 42] INFO [2022-03-13 17:19:15,295] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,297] jepsen worker 13 - jepsen.util 733 :invoke :write [4 0] INFO [2022-03-13 17:19:15,300] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:15,317] jepsen worker 88 - jepsen.util 2428 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,318] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,318] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,319] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:19:15,340] jepsen worker 0 - jepsen.util 4320 :info :write [0 41] :RequestTimeoutException INFO [2022-03-13 17:19:15,341] jepsen worker 0 - jepsen.util 4410 :invoke :write [0 12] INFO [2022-03-13 17:19:15,349] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,350] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:15,353] jepsen worker 14 - jepsen.util 374 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,355] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:15,363] jepsen worker 53 - jepsen.util 593 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,363] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:15,382] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,382] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,384] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,384] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,385] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,386] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:15,408] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,408] jepsen worker 37 - jepsen.util 2467 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,409] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 25] INFO [2022-03-13 17:19:15,409] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,410] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,410] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,412] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,413] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 33] INFO [2022-03-13 17:19:15,421] jepsen worker 3 - jepsen.util 2253 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,421] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:19:15,437] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,438] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:15,455] jepsen worker 21 - jepsen.util 831 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,455] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:15,465] jepsen worker 12 - jepsen.util 912 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,465] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,466] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:19:15,466] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:15,494] jepsen worker 88 - jepsen.util 2428 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,495] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,526] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,527] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 37] INFO [2022-03-13 17:19:15,533] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,534] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,536] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,536] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:19:15,553] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,553] jepsen worker 80 - jepsen.util 800 :invoke :write [26 0] INFO [2022-03-13 17:19:15,579] jepsen worker 43 - jepsen.util 673 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,579] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,579] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,580] jepsen worker 43 - jepsen.util 673 :invoke :write [14 6] INFO [2022-03-13 17:19:15,583] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:15,583] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:15,593] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,594] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:15,594] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,594] jepsen worker 37 - jepsen.util 2467 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,595] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,595] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:19:15,597] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,597] jepsen worker 78 - jepsen.util 708 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,597] jepsen worker 77 - jepsen.util 707 :invoke :write [25 28] INFO [2022-03-13 17:19:15,598] jepsen worker 78 - jepsen.util 708 :invoke :write [26 43] INFO [2022-03-13 17:19:15,616] jepsen worker 57 - jepsen.util 4557 :info :write [19 32] :RequestTimeoutException INFO [2022-03-13 17:19:15,617] jepsen worker 57 - jepsen.util 4647 :invoke :write [19 34] INFO [2022-03-13 17:19:15,624] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,624] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,631] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,631] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,632] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:19:15,632] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,640] jepsen worker 82 - jepsen.util 4762 :info :write [27 49] :RequestTimeoutException INFO [2022-03-13 17:19:15,641] jepsen worker 82 - jepsen.util 4852 :invoke :write [27 21] INFO [2022-03-13 17:19:15,656] jepsen worker 2 - jepsen.util 4862 :info :write [0 47] :RequestTimeoutException INFO [2022-03-13 17:19:15,656] jepsen worker 51 - jepsen.util 771 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,656] jepsen worker 2 - jepsen.util 4952 :invoke :write [0 16] INFO [2022-03-13 17:19:15,656] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,657] jepsen worker 33 - jepsen.util 753 :invoke :write [11 41] INFO [2022-03-13 17:19:15,670] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:15,698] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,699] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:19:15,715] jepsen worker 75 - jepsen.util 525 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,717] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:15,725] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,726] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,739] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,740] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,741] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,741] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,741] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:15,742] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,742] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,742] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,742] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,743] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:15,744] jepsen worker 21 - jepsen.util 831 :invoke :write [7 47] INFO [2022-03-13 17:19:15,745] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:15,746] jepsen worker 34 - jepsen.util 934 :invoke :write [11 7] INFO [2022-03-13 17:19:15,746] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:15,754] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,754] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,755] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,755] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 46] INFO [2022-03-13 17:19:15,759] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,760] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:19:15,828] jepsen worker 4 - jepsen.util 2614 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,829] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,831] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,831] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,832] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,832] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,832] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,832] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,832] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,833] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,833] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:19:15,833] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,834] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,834] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:15,835] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,835] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,835] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:19:15,835] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 45] INFO [2022-03-13 17:19:15,847] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,847] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,848] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:15,848] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,854] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,854] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,854] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:19:15,855] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:19:15,872] jepsen worker 52 - jepsen.util 682 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,872] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:15,875] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,875] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:19:15,880] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:15,880] jepsen worker nemesis - couchbase.util Rebalance check count: 662 INFO [2022-03-13 17:19:15,880] jepsen worker nemesis - couchbase.util Rebalance stuck count: 534 INFO [2022-03-13 17:19:15,891] jepsen worker 80 - jepsen.util 800 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,891] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,892] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:15,892] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:15,902] jepsen worker 44 - jepsen.util 764 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,903] jepsen worker 79 - jepsen.util 1159 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,903] jepsen worker 44 - jepsen.util 764 :invoke :write [14 23] INFO [2022-03-13 17:19:15,903] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:15,920] jepsen worker 41 - jepsen.util 4541 :info :write [13 17] :RequestTimeoutException INFO [2022-03-13 17:19:15,921] jepsen worker 41 - jepsen.util 4631 :invoke :write [13 5] INFO [2022-03-13 17:19:15,959] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,959] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,960] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,960] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:15,961] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,962] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:19:15,967] jepsen worker 76 - jepsen.util 796 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,967] jepsen worker 76 - jepsen.util 796 :invoke :write [25 33] INFO [2022-03-13 17:19:15,988] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,989] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:19:15,991] jepsen worker 6 - jepsen.util 636 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,991] jepsen worker 6 - jepsen.util 636 :invoke :write [2 7] INFO [2022-03-13 17:19:15,996] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,996] jepsen worker 33 - jepsen.util 753 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,996] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,996] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,997] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:15,996] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:15,997] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:15,998] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:19:15,998] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:15,999] jepsen worker 25 - jepsen.util 565 :invoke :write [8 32] INFO [2022-03-13 17:19:16,006] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,007] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:16,044] jepsen worker 61 - jepsen.util 781 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,045] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:16,052] jepsen worker 26 - jepsen.util 566 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,052] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,053] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:16,053] jepsen worker 53 - jepsen.util 593 :invoke :write [17 48] INFO [2022-03-13 17:19:16,055] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,055] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:16,056] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,056] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:16,057] jepsen worker 21 - jepsen.util 831 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,057] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:19:16,058] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,058] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,059] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 41] INFO [2022-03-13 17:19:16,059] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:19:16,061] jepsen worker 88 - jepsen.util 2428 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,061] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 23] INFO [2022-03-13 17:19:16,078] jepsen worker 73 - jepsen.util 4843 :info :write [24 5] :RequestTimeoutException INFO [2022-03-13 17:19:16,079] jepsen worker 73 - jepsen.util 4933 :invoke :read [24 nil] INFO [2022-03-13 17:19:16,080] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,081] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:19:16,087] jepsen worker 13 - jepsen.util 733 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,088] jepsen worker 13 - jepsen.util 733 :invoke :write [4 2] INFO [2022-03-13 17:19:16,089] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,090] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:16,096] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,097] jepsen worker 42 - jepsen.util 402 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,097] jepsen worker 8 - jepsen.util 728 :invoke :write [2 9] INFO [2022-03-13 17:19:16,097] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,098] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,098] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,099] jepsen worker 70 - jepsen.util 880 :invoke :write [23 12] INFO [2022-03-13 17:19:16,099] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:16,114] jepsen worker 34 - jepsen.util 934 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,114] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,114] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,115] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,115] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,115] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:16,115] jepsen worker 33 - jepsen.util 753 :invoke :write [11 20] INFO [2022-03-13 17:19:16,115] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,116] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,116] jepsen worker 4 - jepsen.util 2614 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,116] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:16,117] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:19:16,117] jepsen worker 34 - jepsen.util 934 :invoke :write [11 43] INFO [2022-03-13 17:19:16,117] jepsen worker 24 - jepsen.util 654 :invoke :write [8 28] INFO [2022-03-13 17:19:16,122] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,122] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:16,140] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,141] jepsen worker 23 - jepsen.util 563 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,141] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:16,141] jepsen worker 23 - jepsen.util 563 :invoke :write [7 30] INFO [2022-03-13 17:19:16,168] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,169] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:19:16,172] jepsen worker 76 - jepsen.util 796 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,172] jepsen worker 43 - jepsen.util 673 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,172] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,172] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,172] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,172] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,172] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,173] jepsen worker 60 - jepsen.util 690 :invoke :write [20 18] INFO [2022-03-13 17:19:16,174] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,175] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,175] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,176] jepsen worker 76 - jepsen.util 796 :invoke :write [25 9] INFO [2022-03-13 17:19:16,177] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:19:16,179] jepsen worker 75 - jepsen.util 525 :invoke :write [25 17] INFO [2022-03-13 17:19:16,182] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,182] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:16,188] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,191] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:19:16,192] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,192] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:19:16,197] jepsen worker 38 - jepsen.util 2828 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,202] jepsen worker 62 - jepsen.util 512 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,202] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,203] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:16,203] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:16,205] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:19:16,226] jepsen worker 77 - jepsen.util 707 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,227] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:16,228] jepsen worker 25 - jepsen.util 565 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,228] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:16,231] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,231] jepsen worker 53 - jepsen.util 593 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,231] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,232] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,232] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,232] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,233] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,233] jepsen worker 26 - jepsen.util 566 :invoke :write [8 20] INFO [2022-03-13 17:19:16,246] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,247] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,255] jepsen worker 13 - jepsen.util 733 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,255] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:16,256] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,256] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:19:16,257] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,257] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:19:16,279] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,279] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,279] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,279] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,280] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:19:16,280] jepsen worker 80 - jepsen.util 800 :invoke :write [26 10] INFO [2022-03-13 17:19:16,281] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,281] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:16,290] jepsen worker 60 - jepsen.util 690 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,290] jepsen worker 60 - jepsen.util 690 :invoke :write [20 34] INFO [2022-03-13 17:19:16,291] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,292] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,312] jepsen worker 44 - jepsen.util 764 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,313] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,336] jepsen worker 76 - jepsen.util 796 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,337] jepsen worker 76 - jepsen.util 796 :invoke :write [25 43] INFO [2022-03-13 17:19:16,339] jepsen worker 24 - jepsen.util 654 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,340] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:16,343] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,343] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,344] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,344] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,344] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,344] jepsen worker 78 - jepsen.util 708 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,344] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:19:16,344] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,345] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:16,346] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:19:16,346] jepsen worker 61 - jepsen.util 781 :invoke :write [20 6] INFO [2022-03-13 17:19:16,347] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,348] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:19:16,350] jepsen worker 66 - jepsen.util 4296 :info :write [22 28] :RequestTimeoutException INFO [2022-03-13 17:19:16,351] jepsen worker 66 - jepsen.util 4386 :invoke :read [22 nil] INFO [2022-03-13 17:19:16,355] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:16,368] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,368] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:16,381] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,382] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:16,388] jepsen worker 33 - jepsen.util 753 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,390] jepsen worker 33 - jepsen.util 753 :invoke :write [11 41] INFO [2022-03-13 17:19:16,397] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,397] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,397] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,397] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,398] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,398] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:19:16,398] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,399] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,399] jepsen worker 6 - jepsen.util 636 :fail :write [2 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,400] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,401] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,401] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:16,401] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,401] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:16,413] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,413] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,413] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:16,413] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,426] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,426] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,426] jepsen worker 23 - jepsen.util 563 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,426] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,426] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:16,427] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:16,427] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:16,427] jepsen worker 78 - jepsen.util 708 :invoke :write [26 45] INFO [2022-03-13 17:19:16,429] jepsen worker 75 - jepsen.util 525 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,429] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,430] jepsen worker 75 - jepsen.util 525 :invoke :write [25 25] INFO [2022-03-13 17:19:16,430] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:16,431] jepsen worker 34 - jepsen.util 934 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,431] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:16,460] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,461] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,460] jepsen worker 88 - jepsen.util 2428 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,461] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,461] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,461] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 47] INFO [2022-03-13 17:19:16,462] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:16,462] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,463] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 12] INFO [2022-03-13 17:19:16,463] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,464] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:19:16,469] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,469] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:19:16,470] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:16,476] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,477] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:19:16,502] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,502] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,502] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,503] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,504] jepsen worker 70 - jepsen.util 880 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,504] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:16,505] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:16,505] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:16,506] jepsen worker 51 - jepsen.util 771 :invoke :write [17 43] INFO [2022-03-13 17:19:16,506] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:16,515] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,516] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:16,536] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,536] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,537] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,537] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:16,538] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:16,538] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:16,543] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,543] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,543] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:16,544] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:16,545] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,546] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,545] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,545] jepsen worker 61 - jepsen.util 781 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,546] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,547] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,547] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,547] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 1] INFO [2022-03-13 17:19:16,548] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:16,549] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:16,549] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 29] INFO [2022-03-13 17:19:16,550] jepsen worker 61 - jepsen.util 781 :invoke :write [20 38] INFO [2022-03-13 17:19:16,550] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:19:16,551] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:16,574] jepsen worker 80 - jepsen.util 800 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,575] jepsen worker 80 - jepsen.util 800 :invoke :write [26 14] INFO [2022-03-13 17:19:16,583] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,584] jepsen worker 37 - jepsen.util 2467 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,584] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,585] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 25] INFO [2022-03-13 17:19:16,595] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,595] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:19:16,604] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,604] jepsen worker 76 - jepsen.util 796 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,604] jepsen worker 26 - jepsen.util 566 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,605] jepsen worker 13 - jepsen.util 733 :invoke :write [4 46] INFO [2022-03-13 17:19:16,605] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:16,605] jepsen worker 26 - jepsen.util 566 :invoke :write [8 26] INFO [2022-03-13 17:19:16,617] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,617] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:19:16,645] jepsen worker 38 - jepsen.util 2828 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,646] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:16,649] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,649] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,649] jepsen worker 21 - jepsen.util 831 :invoke :write [7 37] INFO [2022-03-13 17:19:16,650] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:16,650] jepsen worker 8 - jepsen.util 728 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,651] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,651] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,651] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,651] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,652] jepsen worker 8 - jepsen.util 728 :invoke :write [2 41] INFO [2022-03-13 17:19:16,653] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:16,653] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:16,653] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:16,654] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,655] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,655] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,655] jepsen worker 7 - jepsen.util 637 :invoke :write [2 16] INFO [2022-03-13 17:19:16,655] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,656] jepsen worker 6 - jepsen.util 636 :invoke :write [2 13] INFO [2022-03-13 17:19:16,663] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,706] jepsen worker 3 - jepsen.util 2253 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,706] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:16,709] jepsen worker 78 - jepsen.util 708 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,710] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,710] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,710] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:16,711] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:16,712] jepsen worker 77 - jepsen.util 707 :invoke :write [25 23] INFO [2022-03-13 17:19:16,712] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,712] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,713] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,713] jepsen worker 60 - jepsen.util 690 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,714] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,714] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,714] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,714] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,714] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,714] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:16,714] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,715] jepsen worker 44 - jepsen.util 764 :invoke :write [14 12] INFO [2022-03-13 17:19:16,718] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,719] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:19:16,720] jepsen worker 12 - jepsen.util 912 :invoke :write [4 15] INFO [2022-03-13 17:19:16,720] jepsen worker 33 - jepsen.util 753 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,721] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,721] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,721] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,721] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,722] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:16,723] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,723] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:16,724] jepsen worker 33 - jepsen.util 753 :invoke :write [11 43] INFO [2022-03-13 17:19:16,725] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:19:16,727] jepsen worker 34 - jepsen.util 934 :invoke :write [11 14] INFO [2022-03-13 17:19:16,728] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:19:16,734] jepsen worker 35 - jepsen.util 935 :invoke :write [11 37] INFO [2022-03-13 17:19:16,734] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,743] jepsen worker 67 - jepsen.util 3847 :info :write [22 36] :RequestTimeoutException INFO [2022-03-13 17:19:16,744] jepsen worker 62 - jepsen.util 512 :invoke :write [20 34] INFO [2022-03-13 17:19:16,745] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 37] INFO [2022-03-13 17:19:16,749] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:19:16,750] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:19:16,758] jepsen worker 67 - jepsen.util 3937 :invoke :write [22 18] INFO [2022-03-13 17:19:16,758] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:16,758] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:16,759] jepsen worker 43 - jepsen.util 673 :invoke :write [14 3] INFO [2022-03-13 17:19:16,774] jepsen worker 88 - jepsen.util 2428 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,774] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 31] INFO [2022-03-13 17:19:16,793] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,793] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 48] INFO [2022-03-13 17:19:16,797] jepsen worker 51 - jepsen.util 771 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,798] jepsen worker 75 - jepsen.util 525 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,798] jepsen worker 51 - jepsen.util 771 :invoke :write [17 18] INFO [2022-03-13 17:19:16,798] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,799] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:16,803] jepsen worker 76 - jepsen.util 796 :invoke :write [25 19] INFO [2022-03-13 17:19:16,818] jepsen worker 79 - jepsen.util 1159 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,818] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:16,871] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,871] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:19:16,877] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,878] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,878] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 31] INFO [2022-03-13 17:19:16,878] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:19:16,886] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:16,886] jepsen worker nemesis - couchbase.util Rebalance check count: 663 INFO [2022-03-13 17:19:16,886] jepsen worker nemesis - couchbase.util Rebalance stuck count: 535 INFO [2022-03-13 17:19:16,890] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,890] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,891] jepsen worker 71 - jepsen.util 611 :invoke :write [23 24] INFO [2022-03-13 17:19:16,893] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:19:16,912] jepsen worker 8 - jepsen.util 728 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,913] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:16,924] jepsen worker 38 - jepsen.util 2828 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,924] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,925] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:16,925] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:19:16,926] jepsen worker 80 - jepsen.util 800 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,926] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,927] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:16,927] jepsen worker 24 - jepsen.util 654 :invoke :write [8 34] INFO [2022-03-13 17:19:16,941] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,941] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 15] INFO [2022-03-13 17:19:16,986] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,987] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:19:16,989] jepsen worker 4 - jepsen.util 2614 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:16,990] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,009] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,010] jepsen worker 13 - jepsen.util 733 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,010] jepsen worker 13 - jepsen.util 733 :invoke :write [4 34] INFO [2022-03-13 17:19:17,011] jepsen worker 75 - jepsen.util 525 :invoke :write [25 4] INFO [2022-03-13 17:19:17,023] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,023] jepsen worker 69 - jepsen.util 789 :invoke :write [23 42] INFO [2022-03-13 17:19:17,044] jepsen worker 21 - jepsen.util 831 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,045] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:17,050] jepsen worker 36 - jepsen.util 2466 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,051] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:19:17,059] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,059] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,073] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,073] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,073] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,074] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:19:17,082] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,083] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,084] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,084] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:19:17,098] jepsen worker 6 - jepsen.util 636 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,098] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,098] jepsen worker 61 - jepsen.util 781 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,098] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,099] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:17,099] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:17,099] jepsen worker 6 - jepsen.util 636 :invoke :write [2 49] INFO [2022-03-13 17:19:17,103] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:17,120] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,120] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,120] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:19:17,121] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:19:17,122] jepsen worker 71 - jepsen.util 611 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,123] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:19:17,134] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,134] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,134] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,135] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:19:17,142] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,142] jepsen worker 26 - jepsen.util 566 :fail :write [8 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,142] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,142] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,143] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:17,144] jepsen worker 88 - jepsen.util 2428 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,144] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:17,145] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 19] INFO [2022-03-13 17:19:17,145] jepsen worker 60 - jepsen.util 690 :invoke :write [20 43] INFO [2022-03-13 17:19:17,146] jepsen worker 26 - jepsen.util 566 :invoke :write [8 28] INFO [2022-03-13 17:19:17,151] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,155] jepsen worker 25 - jepsen.util 565 :invoke :write [8 46] WARN [2022-03-13 17:19:17,159] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8648412,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9361221,"total_server_duration_us":375659,"encode_duration_us":17739,"operation_id":"0x12e03","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10011073},{"operation_name":"upsert","last_dispatch_duration_us":8394645,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9559174,"total_server_duration_us":549155,"encode_duration_us":22189,"operation_id":"0x12c61","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10009426},{"operation_name":"upsert","last_dispatch_duration_us":78054,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1084304,"total_server_duration_us":530020,"encode_duration_us":18231,"operation_id":"0x12ffa","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10007966},{"operation_name":"upsert","last_dispatch_duration_us":37630,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":815741,"total_server_duration_us":497150,"encode_duration_us":18617,"operation_id":"0x130ca","timeout_ms":10000,"last_server_duration_us":24669,"total_duration_us":10001387},{"operation_name":"upsert","last_dispatch_duration_us":9164716,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9291123,"total_server_duration_us":9152797,"encode_duration_us":21999,"operation_id":"0x12f70","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":9347061},{"operation_name":"get","last_dispatch_duration_us":8356137,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8356137,"total_server_duration_us":114289,"operation_id":"0x13032","timeout_ms":30000,"last_server_duration_us":114289,"total_duration_us":8358740},{"operation_name":"upsert","last_dispatch_duration_us":946917,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":946917,"total_server_duration_us":74648,"encode_duration_us":24599,"operation_id":"0x135f7","timeout_ms":10000,"last_server_duration_us":74648,"total_duration_us":1531465},{"operation_name":"upsert","last_dispatch_duration_us":1351702,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1351702,"total_server_duration_us":534196,"encode_duration_us":25431,"operation_id":"0x13197","timeout_ms":10000,"last_server_duration_us":534196,"total_duration_us":1361261},{"operation_name":"get","last_dispatch_duration_us":909525,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":909525,"total_server_duration_us":15,"operation_id":"0x131e2","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":1212922},{"operation_name":"get","last_dispatch_duration_us":909605,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":909605,"total_server_duration_us":15,"operation_id":"0x131e3","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":1212390}],"total_count":102}} INFO [2022-03-13 17:19:17,180] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,180] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,181] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 31] INFO [2022-03-13 17:19:17,182] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,224] jepsen worker 76 - jepsen.util 796 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,225] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,237] jepsen worker 69 - jepsen.util 789 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,237] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:17,239] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,239] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:19:17,253] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,254] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,254] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,255] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,258] jepsen worker 37 - jepsen.util 2467 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,259] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,264] jepsen worker 34 - jepsen.util 934 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,265] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:17,298] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,299] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:19:17,318] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,319] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,319] jepsen worker 7 - jepsen.util 637 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,319] jepsen worker 77 - jepsen.util 707 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,319] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,319] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,319] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,320] jepsen worker 7 - jepsen.util 637 :invoke :write [2 46] INFO [2022-03-13 17:19:17,321] jepsen worker 78 - jepsen.util 708 :invoke :write [26 26] INFO [2022-03-13 17:19:17,321] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:17,323] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 39] INFO [2022-03-13 17:19:17,323] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,323] jepsen worker 70 - jepsen.util 880 :invoke :write [23 35] INFO [2022-03-13 17:19:17,324] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,326] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:19:17,328] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,329] jepsen worker 77 - jepsen.util 707 :invoke :write [25 2] INFO [2022-03-13 17:19:17,330] jepsen worker 61 - jepsen.util 781 :invoke :write [20 31] INFO [2022-03-13 17:19:17,339] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,340] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,341] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:19:17,346] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:17,348] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,349] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:19:17,354] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,354] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:19:17,363] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,363] jepsen worker 36 - jepsen.util 2466 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,363] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,364] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,365] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:19:17,365] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,366] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,366] jepsen worker 35 - jepsen.util 935 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,368] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:19:17,368] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,368] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:17,368] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,370] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,370] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:19:17,372] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,372] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,373] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,374] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:19:17,374] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,375] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:19:17,393] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,393] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,393] jepsen worker 43 - jepsen.util 673 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,393] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,394] jepsen worker 75 - jepsen.util 525 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,395] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:17,395] jepsen worker 52 - jepsen.util 682 :invoke :write [17 28] INFO [2022-03-13 17:19:17,397] jepsen worker 75 - jepsen.util 525 :invoke :write [25 16] INFO [2022-03-13 17:19:17,397] jepsen worker 22 - jepsen.util 652 :invoke :write [7 42] INFO [2022-03-13 17:19:17,398] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:17,418] jepsen worker 88 - jepsen.util 2428 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,418] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,419] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,420] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:17,426] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,426] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,427] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,427] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:17,428] jepsen worker 12 - jepsen.util 912 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,428] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:17,428] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,429] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,430] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:17,432] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:17,433] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:19:17,434] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:19:17,446] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,446] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,469] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,471] jepsen worker 69 - jepsen.util 789 :invoke :write [23 46] INFO [2022-03-13 17:19:17,487] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,488] jepsen worker 14 - jepsen.util 374 :invoke :write [4 23] INFO [2022-03-13 17:19:17,509] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,509] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:19:17,511] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,511] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,511] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:19:17,512] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:19:17,548] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,548] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:19:17,561] jepsen worker 60 - jepsen.util 690 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,561] jepsen worker 60 - jepsen.util 690 :invoke :write [20 1] INFO [2022-03-13 17:19:17,585] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,585] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,586] jepsen worker 44 - jepsen.util 764 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,586] jepsen worker 79 - jepsen.util 1159 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,586] jepsen worker 44 - jepsen.util 764 :invoke :write [14 6] INFO [2022-03-13 17:19:17,587] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:17,594] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,595] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:19:17,596] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,597] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,599] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,600] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:19:17,608] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,608] jepsen worker 33 - jepsen.util 753 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,609] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:17,610] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:19:17,630] jepsen worker 13 - jepsen.util 733 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,630] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,630] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:17,631] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,632] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,632] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:19:17,639] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,640] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:19:17,646] jepsen worker 69 - jepsen.util 789 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,646] jepsen worker 69 - jepsen.util 789 :invoke :write [23 46] INFO [2022-03-13 17:19:17,684] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,685] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:17,716] jepsen worker 25 - jepsen.util 565 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,716] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,716] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:19:17,717] jepsen worker 21 - jepsen.util 831 :invoke :write [7 47] INFO [2022-03-13 17:19:17,718] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,719] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:19:17,721] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,721] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,722] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,723] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,752] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,753] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:19:17,778] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:17,778] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,778] jepsen worker 26 - jepsen.util 566 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,778] jepsen worker 53 - jepsen.util 593 :invoke :write [17 24] INFO [2022-03-13 17:19:17,779] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:17,779] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:17,781] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,781] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,789] jepsen worker 62 - jepsen.util 512 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,789] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,789] jepsen worker 78 - jepsen.util 708 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,789] jepsen worker 8 - jepsen.util 728 :invoke :write [2 49] INFO [2022-03-13 17:19:17,791] jepsen worker 62 - jepsen.util 512 :invoke :write [20 35] INFO [2022-03-13 17:19:17,795] jepsen worker 78 - jepsen.util 708 :invoke :write [26 16] INFO [2022-03-13 17:19:17,799] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,799] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,801] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:17,801] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,818] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,818] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,818] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,818] jepsen worker 24 - jepsen.util 654 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,818] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,818] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,819] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,819] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,820] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,821] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:19:17,822] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,822] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,822] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,822] jepsen worker 24 - jepsen.util 654 :invoke :write [8 37] INFO [2022-03-13 17:19:17,822] jepsen worker 34 - jepsen.util 934 :invoke :write [11 13] INFO [2022-03-13 17:19:17,823] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,823] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:19:17,824] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,825] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,826] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,827] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,828] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,829] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,830] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:17,831] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,831] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,832] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,833] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:19:17,851] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,851] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,852] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:17,852] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,854] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,855] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,855] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:19:17,856] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:17,857] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,858] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:19:17,887] jepsen worker 69 - jepsen.util 789 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,888] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:19:17,892] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:17,892] jepsen worker nemesis - couchbase.util Rebalance check count: 664 INFO [2022-03-13 17:19:17,892] jepsen worker nemesis - couchbase.util Rebalance stuck count: 536 INFO [2022-03-13 17:19:17,894] jepsen worker 75 - jepsen.util 525 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,894] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,895] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,895] jepsen worker 13 - jepsen.util 733 :invoke :write [4 20] INFO [2022-03-13 17:19:17,896] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,896] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,901] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,901] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,902] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:17,903] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,903] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,903] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] INFO [2022-03-13 17:19:17,904] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,905] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:19:17,905] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,906] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,907] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,908] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,909] jepsen worker 75 - jepsen.util 525 :invoke :write [25 0] INFO [2022-03-13 17:19:17,910] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,910] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 27] INFO [2022-03-13 17:19:17,911] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:17,924] jepsen worker 51 - jepsen.util 771 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,924] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:17,938] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,938] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:17,944] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,945] jepsen worker 76 - jepsen.util 796 :invoke :write [25 11] INFO [2022-03-13 17:19:17,983] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:17,984] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:19:18,011] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,011] jepsen worker 6 - jepsen.util 636 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,012] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:18,013] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:18,034] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,035] jepsen worker 71 - jepsen.util 611 :invoke :write [23 32] INFO [2022-03-13 17:19:18,043] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,044] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,053] jepsen worker 14 - jepsen.util 374 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,053] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,053] jepsen worker 80 - jepsen.util 800 :invoke :write [26 19] INFO [2022-03-13 17:19:18,054] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:18,055] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,055] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:19:18,086] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,086] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:18,087] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,088] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:18,100] jepsen worker 52 - jepsen.util 682 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,101] jepsen worker 52 - jepsen.util 682 :invoke :write [17 30] INFO [2022-03-13 17:19:18,107] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,107] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,108] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,109] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,110] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,110] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,110] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,111] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,111] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,111] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:19:18,111] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,112] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,113] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,114] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,114] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,115] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,116] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,116] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:19:18,125] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,125] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 42] INFO [2022-03-13 17:19:18,148] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,149] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:18,155] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,155] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:18,174] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,174] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:18,200] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,200] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:18,210] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,210] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:19:18,215] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,216] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,221] jepsen worker 7 - jepsen.util 637 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,221] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,221] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,222] jepsen worker 77 - jepsen.util 707 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,222] jepsen worker 7 - jepsen.util 637 :invoke :write [2 15] INFO [2022-03-13 17:19:18,223] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:19:18,225] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:19:18,228] jepsen worker 77 - jepsen.util 707 :invoke :write [25 3] INFO [2022-03-13 17:19:18,251] jepsen worker 13 - jepsen.util 733 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,253] jepsen worker 13 - jepsen.util 733 :invoke :write [4 40] INFO [2022-03-13 17:19:18,279] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,280] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,307] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,307] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:18,318] jepsen worker 71 - jepsen.util 611 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,318] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,319] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:19:18,319] jepsen worker 69 - jepsen.util 789 :invoke :write [23 48] INFO [2022-03-13 17:19:18,336] jepsen worker 3 - jepsen.util 2253 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,336] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,337] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,337] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:18,337] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,338] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,338] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,338] jepsen worker 37 - jepsen.util 2467 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,339] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 17] INFO [2022-03-13 17:19:18,339] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 31] INFO [2022-03-13 17:19:18,342] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,342] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,346] jepsen worker 22 - jepsen.util 652 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,346] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:19:18,352] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,352] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,353] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,353] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,354] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,354] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,355] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 12] INFO [2022-03-13 17:19:18,355] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:19:18,390] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,391] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:18,399] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,399] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,400] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 34] INFO [2022-03-13 17:19:18,400] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,401] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,402] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 40] INFO [2022-03-13 17:19:18,414] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,415] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:19:18,417] jepsen worker 70 - jepsen.util 880 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,418] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:18,417] jepsen worker 61 - jepsen.util 781 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,417] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,417] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,418] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:18,419] jepsen worker 42 - jepsen.util 402 :invoke :write [14 17] INFO [2022-03-13 17:19:18,420] jepsen worker 70 - jepsen.util 880 :invoke :write [23 29] INFO [2022-03-13 17:19:18,421] jepsen worker 12 - jepsen.util 912 :invoke :write [4 33] INFO [2022-03-13 17:19:18,421] jepsen worker 61 - jepsen.util 781 :invoke :write [20 42] WARN [2022-03-13 17:19:18,446] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9145059,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18250546,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9137309,"encode_duration_us":18650,"operation_id":"0x12b48","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10001788},{"operation_name":"upsert","last_dispatch_duration_us":17066021,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17066021,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9069231,"encode_duration_us":19628,"operation_id":"0x12b99","timeout_ms":10000,"last_server_duration_us":9069231,"total_duration_us":10005702},{"operation_name":"upsert","last_dispatch_duration_us":9040232,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9980495,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9509149,"encode_duration_us":17554,"operation_id":"0x12b6f","timeout_ms":10000,"last_server_duration_us":9026756,"total_duration_us":10003293}],"total_count":3}} INFO [2022-03-13 17:19:18,455] jepsen worker 21 - jepsen.util 831 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,456] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:18,463] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,464] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:18,466] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,466] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:19:18,507] jepsen worker 75 - jepsen.util 525 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,508] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:18,526] jepsen worker 89 - jepsen.util 2789 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,526] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,526] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:19:18,527] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:19:18,547] jepsen worker 34 - jepsen.util 934 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,547] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:18,555] jepsen worker 69 - jepsen.util 789 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,555] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,555] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:19:18,556] jepsen worker 69 - jepsen.util 789 :invoke :write [23 46] INFO [2022-03-13 17:19:18,566] jepsen worker 3 - jepsen.util 2253 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,567] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:19:18,587] jepsen worker 60 - jepsen.util 690 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,587] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:18,588] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,588] jepsen worker 60 - jepsen.util 690 :invoke :write [20 19] INFO [2022-03-13 17:19:18,589] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:18,589] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:18,606] jepsen worker 37 - jepsen.util 2467 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,606] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,612] jepsen worker 38 - jepsen.util 2828 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,612] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:19:18,618] jepsen worker 44 - jepsen.util 764 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,618] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,619] jepsen worker 78 - jepsen.util 708 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,620] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:18,620] jepsen worker 78 - jepsen.util 708 :invoke :write [26 14] INFO [2022-03-13 17:19:18,629] jepsen worker 44 - jepsen.util 764 :invoke :write [14 16] INFO [2022-03-13 17:19:18,638] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,639] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,660] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,661] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,661] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:19:18,661] jepsen worker 14 - jepsen.util 374 :invoke :write [4 23] WARN [2022-03-13 17:19:18,666] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10743465,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10743465,"total_server_duration_us":20282,"operation_id":"0x12e52","timeout_ms":30000,"last_server_duration_us":20282,"total_duration_us":10749761},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":17652,"operation_id":"0x12df2","timeout_ms":10000,"total_duration_us":10009940},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":29882,"operation_id":"0x130cc","timeout_ms":10000,"total_duration_us":10009920},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":31371,"operation_id":"0x130bd","timeout_ms":10000,"total_duration_us":10007157},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":17879,"operation_id":"0x12ff0","timeout_ms":10000,"total_duration_us":10005283},{"operation_name":"get","last_dispatch_duration_us":8909116,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8909116,"total_server_duration_us":60885,"operation_id":"0x130ae","timeout_ms":30000,"last_server_duration_us":60885,"total_duration_us":8909421},{"operation_name":"upsert","last_dispatch_duration_us":2110529,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2110529,"total_server_duration_us":63053,"encode_duration_us":26269,"operation_id":"0x136a1","timeout_ms":10000,"last_server_duration_us":63053,"total_duration_us":2114617},{"operation_name":"upsert","last_dispatch_duration_us":1981117,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1981117,"total_server_duration_us":108723,"encode_duration_us":23684,"operation_id":"0x135e5","timeout_ms":10000,"last_server_duration_us":108723,"total_duration_us":1986674},{"operation_name":"upsert","last_dispatch_duration_us":1668426,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1668426,"total_server_duration_us":98723,"encode_duration_us":31298,"operation_id":"0x13581","timeout_ms":10000,"last_server_duration_us":98723,"total_duration_us":1672777},{"operation_name":"get","last_dispatch_duration_us":1295912,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":1295912,"total_server_duration_us":32,"operation_id":"0x1370f","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":1296096}],"total_count":11}} WARN [2022-03-13 17:19:18,666] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":23568581,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":23568581,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9062854,"encode_duration_us":18808,"operation_id":"0x129ba","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10007381},{"operation_name":"upsert","last_dispatch_duration_us":17206158,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18678512,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9652910,"encode_duration_us":20136,"operation_id":"0x129a8","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10008091}],"total_count":2}} INFO [2022-03-13 17:19:18,668] jepsen worker 69 - jepsen.util 789 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,668] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] WARN [2022-03-13 17:19:18,670] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":20751662,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":20751662,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9029938,"encode_duration_us":16993,"operation_id":"0x12be6","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10009497},{"operation_name":"upsert","last_dispatch_duration_us":16418449,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16418449,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9054355,"encode_duration_us":17406,"operation_id":"0x12b4a","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10010861}],"total_count":2}} INFO [2022-03-13 17:19:18,692] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,692] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:19:18,703] jepsen worker 76 - jepsen.util 796 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,704] jepsen worker 76 - jepsen.util 796 :invoke :write [25 44] INFO [2022-03-13 17:19:18,711] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,712] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] WARN [2022-03-13 17:19:18,747] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":99063,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":7238556,"total_server_duration_us":9589,"operation_id":"0x127a2","timeout_ms":30000,"last_server_duration_us":534,"total_duration_us":30010788},{"operation_name":"upsert","last_dispatch_duration_us":156861,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9081863,"total_server_duration_us":320621,"encode_duration_us":17923,"operation_id":"0x12ff1","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10007730},{"operation_name":"upsert","last_dispatch_duration_us":67637,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":701712,"total_server_duration_us":94202,"encode_duration_us":21015,"operation_id":"0x12f83","timeout_ms":10000,"last_server_duration_us":29045,"total_duration_us":10007422},{"operation_name":"upsert","last_dispatch_duration_us":8282023,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8821863,"total_server_duration_us":321423,"encode_duration_us":18343,"operation_id":"0x12e10","timeout_ms":10000,"last_server_duration_us":25712,"total_duration_us":10006764},{"operation_name":"upsert","last_dispatch_duration_us":7927555,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8939219,"total_server_duration_us":333499,"encode_duration_us":19664,"operation_id":"0x12fd4","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10005934},{"operation_name":"upsert","last_dispatch_duration_us":220259,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9274389,"total_server_duration_us":216831,"encode_duration_us":25126,"operation_id":"0x1305f","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10005153},{"operation_name":"upsert","last_dispatch_duration_us":348260,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8416735,"total_server_duration_us":121793,"encode_duration_us":24834,"operation_id":"0x13103","timeout_ms":10000,"last_server_duration_us":33132,"total_duration_us":10003893},{"operation_name":"get","last_dispatch_duration_us":262449,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":262449,"total_server_duration_us":25537,"operation_id":"0x134c9","timeout_ms":30000,"last_server_duration_us":25537,"total_duration_us":1854505},{"operation_name":"upsert","last_dispatch_duration_us":1360293,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1360293,"total_server_duration_us":52064,"encode_duration_us":26888,"operation_id":"0x13693","timeout_ms":10000,"last_server_duration_us":52064,"total_duration_us":1366167},{"operation_name":"upsert","last_dispatch_duration_us":1222624,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1222624,"total_server_duration_us":75201,"encode_duration_us":18175,"operation_id":"0x13692","timeout_ms":10000,"last_server_duration_us":75201,"total_duration_us":1228811}],"total_count":50}} WARN [2022-03-13 17:19:18,752] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":162782,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":9941244,"total_server_duration_us":15484,"operation_id":"0x12735","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":30010418},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":22389,"operation_id":"0x1305d","timeout_ms":10000,"total_duration_us":10009706},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":19325,"operation_id":"0x12f91","timeout_ms":10000,"total_duration_us":10008176},{"operation_name":"upsert","last_dispatch_duration_us":7163856,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":14447259,"total_server_duration_us":288824,"encode_duration_us":26755,"operation_id":"0x12d2f","timeout_ms":10000,"last_server_duration_us":28768,"total_duration_us":10007053},{"operation_name":"upsert","last_dispatch_duration_us":7298161,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8305129,"total_server_duration_us":308184,"encode_duration_us":27453,"operation_id":"0x130f3","timeout_ms":10000,"last_server_duration_us":50652,"total_duration_us":10004645},{"operation_name":"upsert","last_dispatch_duration_us":9090960,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9090960,"total_server_duration_us":25101,"encode_duration_us":28329,"operation_id":"0x12f89","timeout_ms":10000,"last_server_duration_us":25101,"total_duration_us":10004555},{"operation_name":"upsert","last_dispatch_duration_us":34253,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1271534,"total_server_duration_us":266179,"encode_duration_us":23052,"operation_id":"0x13075","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10003425},{"operation_name":"upsert","last_dispatch_duration_us":7188859,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7952211,"total_server_duration_us":255247,"encode_duration_us":19086,"operation_id":"0x1312d","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10001185},{"operation_name":"get","last_dispatch_duration_us":9421632,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9421632,"total_server_duration_us":332632,"operation_id":"0x12f90","timeout_ms":30000,"last_server_duration_us":332632,"total_duration_us":9423720},{"operation_name":"upsert","last_dispatch_duration_us":9186649,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9406092,"total_server_duration_us":9072412,"encode_duration_us":19382,"operation_id":"0x1307a","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9420205}],"total_count":65}} WARN [2022-03-13 17:19:18,752] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9097410,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17445299,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9081696,"encode_duration_us":17864,"operation_id":"0x12b49","timeout_ms":10000,"last_server_duration_us":9057542,"total_duration_us":10003002}],"total_count":1}} INFO [2022-03-13 17:19:18,762] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,763] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:19:18,766] jepsen worker 88 - jepsen.util 2428 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,766] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,767] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 38] INFO [2022-03-13 17:19:18,768] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,768] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:18,768] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,769] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:19:18,771] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:18,772] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,773] jepsen worker 71 - jepsen.util 611 :invoke :write [23 9] INFO [2022-03-13 17:19:18,780] jepsen worker 53 - jepsen.util 593 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,781] jepsen worker 53 - jepsen.util 593 :invoke :write [17 47] INFO [2022-03-13 17:19:18,791] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,791] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,791] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,791] jepsen worker 80 - jepsen.util 800 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,792] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:18,793] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:18,794] jepsen worker 80 - jepsen.util 800 :invoke :write [26 32] INFO [2022-03-13 17:19:18,794] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:18,808] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,809] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:18,841] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,841] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,843] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,843] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 28] INFO [2022-03-13 17:19:18,847] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,847] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,852] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,852] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:19:18,857] jepsen worker 62 - jepsen.util 512 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,857] jepsen worker 77 - jepsen.util 707 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,858] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,859] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] WARN [2022-03-13 17:19:18,860] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":14640047,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":14640047,"total_server_duration_us":5907,"operation_id":"0x12c1a","timeout_ms":30000,"last_server_duration_us":5907,"total_duration_us":14642233},{"operation_name":"upsert","last_dispatch_duration_us":10224140,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10224140,"total_server_duration_us":19966,"encode_duration_us":35638,"operation_id":"0x12e66","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10011130},{"operation_name":"upsert","last_dispatch_duration_us":8547062,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":12183377,"total_server_duration_us":341673,"encode_duration_us":24962,"operation_id":"0x12df4","timeout_ms":10000,"last_server_duration_us":33524,"total_duration_us":10009488},{"operation_name":"upsert","last_dispatch_duration_us":48567,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8717139,"total_server_duration_us":234587,"encode_duration_us":31847,"operation_id":"0x13013","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10006317},{"operation_name":"upsert","last_dispatch_duration_us":289191,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":10118362,"total_server_duration_us":169355,"encode_duration_us":31734,"operation_id":"0x12f14","timeout_ms":10000,"last_server_duration_us":22223,"total_duration_us":10002166},{"operation_name":"upsert","last_dispatch_duration_us":9155740,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9552881,"total_server_duration_us":9146338,"encode_duration_us":31028,"operation_id":"0x12fa6","timeout_ms":10000,"last_server_duration_us":9074546,"total_duration_us":9580867},{"operation_name":"upsert","last_dispatch_duration_us":9168456,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9334391,"total_server_duration_us":9080085,"encode_duration_us":17719,"operation_id":"0x12fb2","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9350658},{"operation_name":"get","last_dispatch_duration_us":8734693,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8734693,"total_server_duration_us":8069,"operation_id":"0x12f37","timeout_ms":30000,"last_server_duration_us":8069,"total_duration_us":8743066},{"operation_name":"get","last_dispatch_duration_us":6178865,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":6178865,"total_server_duration_us":76,"operation_id":"0x130c8","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":6179133},{"operation_name":"get","last_dispatch_duration_us":3578590,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":3578590,"total_server_duration_us":38,"operation_id":"0x13233","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":3581226}],"total_count":26}} WARN [2022-03-13 17:19:18,860] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17869654,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17869654,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9035244,"encode_duration_us":22085,"operation_id":"0x127df","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10005907}],"total_count":1}} INFO [2022-03-13 17:19:18,860] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:18,861] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 33] INFO [2022-03-13 17:19:18,863] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,864] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,864] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,865] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:18,880] jepsen worker 24 - jepsen.util 654 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,880] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,881] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,881] jepsen worker 24 - jepsen.util 654 :invoke :write [8 48] INFO [2022-03-13 17:19:18,881] jepsen worker 21 - jepsen.util 831 :invoke :write [7 49] INFO [2022-03-13 17:19:18,882] jepsen worker 34 - jepsen.util 934 :invoke :write [11 19] INFO [2022-03-13 17:19:18,898] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:18,898] jepsen worker nemesis - couchbase.util Rebalance check count: 665 INFO [2022-03-13 17:19:18,898] jepsen worker nemesis - couchbase.util Rebalance stuck count: 537 INFO [2022-03-13 17:19:18,901] jepsen worker 13 - jepsen.util 733 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,901] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,901] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,902] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,903] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:18,903] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:18,904] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:19:18,905] jepsen worker 75 - jepsen.util 525 :invoke :write [25 13] INFO [2022-03-13 17:19:18,905] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,906] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:18,911] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,911] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:19:18,924] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,925] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:19:18,940] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,943] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] WARN [2022-03-13 17:19:18,974] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11239987,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":11239987,"total_server_duration_us":180894,"operation_id":"0x12dea","timeout_ms":30000,"last_server_duration_us":180894,"total_duration_us":11243284},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":28867,"operation_id":"0x13173","timeout_ms":10000,"total_duration_us":10009437},{"operation_name":"upsert","last_dispatch_duration_us":10025161,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10025161,"total_server_duration_us":29788,"encode_duration_us":19483,"operation_id":"0x12eac","timeout_ms":10000,"last_server_duration_us":29788,"total_duration_us":10003114},{"operation_name":"upsert","last_dispatch_duration_us":16312091,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":16312091,"total_server_duration_us":19574,"encode_duration_us":20869,"operation_id":"0x12e35","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10001534},{"operation_name":"get","last_dispatch_duration_us":9407252,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9407252,"total_server_duration_us":33426,"operation_id":"0x12f0b","timeout_ms":30000,"last_server_duration_us":33426,"total_duration_us":9410086},{"operation_name":"get","last_dispatch_duration_us":7884884,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7884884,"total_server_duration_us":10451,"operation_id":"0x12fec","timeout_ms":30000,"last_server_duration_us":10451,"total_duration_us":7888976},{"operation_name":"get","last_dispatch_duration_us":6712491,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":6712491,"total_server_duration_us":14573,"operation_id":"0x1307f","timeout_ms":30000,"last_server_duration_us":14573,"total_duration_us":6715338},{"operation_name":"get","last_dispatch_duration_us":852087,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":852087,"total_server_duration_us":69,"operation_id":"0x1334d","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":1990487},{"operation_name":"upsert","last_dispatch_duration_us":1490504,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1490504,"total_server_duration_us":97638,"encode_duration_us":29721,"operation_id":"0x134d3","timeout_ms":10000,"last_server_duration_us":97638,"total_duration_us":1502598},{"operation_name":"upsert","last_dispatch_duration_us":1468034,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1468034,"total_server_duration_us":15693,"encode_duration_us":24538,"operation_id":"0x13687","timeout_ms":10000,"last_server_duration_us":15693,"total_duration_us":1489386}],"total_count":79}} INFO [2022-03-13 17:19:18,980] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,980] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,980] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,981] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,982] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:19:18,982] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:18,983] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:19:18,985] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:19:18,987] jepsen worker 52 - jepsen.util 682 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:18,988] jepsen worker 52 - jepsen.util 682 :invoke :write [17 10] INFO [2022-03-13 17:19:18,999] jepsen worker 37 - jepsen.util 2467 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,000] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:19,001] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,001] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 36] INFO [2022-03-13 17:19:19,025] jepsen worker 71 - jepsen.util 611 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,026] jepsen worker 71 - jepsen.util 611 :invoke :write [23 9] INFO [2022-03-13 17:19:19,057] jepsen worker 61 - jepsen.util 781 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,057] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,057] jepsen worker 8 - jepsen.util 728 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,058] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:19,059] jepsen worker 61 - jepsen.util 781 :invoke :write [20 13] INFO [2022-03-13 17:19:19,059] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:19,059] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,060] jepsen worker 23 - jepsen.util 563 :invoke :write [7 48] INFO [2022-03-13 17:19:19,094] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,095] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:19:19,098] jepsen worker 14 - jepsen.util 374 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,098] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,099] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,099] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,099] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:19:19,100] jepsen worker 51 - jepsen.util 771 :invoke :write [17 26] INFO [2022-03-13 17:19:19,100] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:19:19,100] jepsen worker 33 - jepsen.util 753 :invoke :write [11 49] INFO [2022-03-13 17:19:19,107] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,107] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:19,117] jepsen worker 76 - jepsen.util 796 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,117] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,119] jepsen worker 76 - jepsen.util 796 :invoke :write [25 4] INFO [2022-03-13 17:19:19,119] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:19,131] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,132] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,177] jepsen worker 88 - jepsen.util 2428 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,177] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,177] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,178] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 22] INFO [2022-03-13 17:19:19,179] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,179] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,183] jepsen worker 38 - jepsen.util 2828 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,183] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:19,205] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,205] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,205] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:19,206] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:19:19,207] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:19,207] jepsen worker 43 - jepsen.util 673 :invoke :write [14 0] INFO [2022-03-13 17:19:19,216] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,217] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,222] jepsen worker 80 - jepsen.util 800 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,223] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:19,238] jepsen worker 71 - jepsen.util 611 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,238] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,238] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,239] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,239] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,240] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,240] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:19:19,241] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,242] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,243] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,244] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,245] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,246] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,247] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,248] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,250] jepsen worker 71 - jepsen.util 611 :invoke :write [23 37] INFO [2022-03-13 17:19:19,250] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,250] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,251] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:19:19,252] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,253] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,253] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,254] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 17] INFO [2022-03-13 17:19:19,257] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,259] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,261] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:19:19,261] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:19:19,262] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,263] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,264] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,266] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,267] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,268] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,269] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,270] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,271] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:19:19,291] jepsen worker 78 - jepsen.util 708 :fail :write [26 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,292] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,292] jepsen worker 7 - jepsen.util 637 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,293] jepsen worker 77 - jepsen.util 707 :invoke :write [25 9] INFO [2022-03-13 17:19:19,294] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:19,294] jepsen worker 78 - jepsen.util 708 :invoke :write [26 46] INFO [2022-03-13 17:19:19,297] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,297] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,298] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,299] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:19:19,302] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,303] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,307] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,307] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,307] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,308] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,309] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,309] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,309] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:19:19,310] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:19:19,330] jepsen worker 1 - jepsen.util 4231 :info :write [0 46] :RequestTimeoutException INFO [2022-03-13 17:19:19,331] jepsen worker 1 - jepsen.util 4321 :invoke :read [0 nil] INFO [2022-03-13 17:19:19,332] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,333] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:19,350] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,350] jepsen worker 75 - jepsen.util 525 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,351] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:19,352] jepsen worker 75 - jepsen.util 525 :invoke :write [25 3] INFO [2022-03-13 17:19:19,355] jepsen worker 21 - jepsen.util 831 :fail :write [7 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,355] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:19,366] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,366] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,367] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,367] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:19,368] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,369] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:19:19,369] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,369] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:19:19,377] jepsen worker 37 - jepsen.util 2467 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,377] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,378] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:19,378] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,417] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,418] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:19:19,445] jepsen worker 60 - jepsen.util 690 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,446] jepsen worker 60 - jepsen.util 690 :invoke :write [20 3] INFO [2022-03-13 17:19:19,467] jepsen worker 24 - jepsen.util 654 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,468] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:19,478] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,479] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:19:19,480] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,480] jepsen worker 71 - jepsen.util 611 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,480] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,481] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,482] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,482] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,482] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,483] jepsen worker 71 - jepsen.util 611 :invoke :write [23 21] INFO [2022-03-13 17:19:19,483] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,484] jepsen worker 69 - jepsen.util 789 :invoke :write [23 12] INFO [2022-03-13 17:19:19,508] jepsen worker 70 - jepsen.util 880 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,509] jepsen worker 79 - jepsen.util 1159 :fail :write [26 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,509] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 29] INFO [2022-03-13 17:19:19,510] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,541] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,542] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:19:19,545] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,545] jepsen worker 76 - jepsen.util 796 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,545] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:19:19,545] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,546] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:19:19,549] jepsen worker 34 - jepsen.util 934 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,549] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,550] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:19,550] jepsen worker 35 - jepsen.util 935 :invoke :write [11 38] INFO [2022-03-13 17:19:19,554] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:19,617] jepsen worker 69 - jepsen.util 789 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,618] jepsen worker 69 - jepsen.util 789 :invoke :write [23 45] INFO [2022-03-13 17:19:19,623] jepsen worker 53 - jepsen.util 593 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,624] jepsen worker 53 - jepsen.util 593 :invoke :write [17 46] INFO [2022-03-13 17:19:19,645] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,645] jepsen worker 88 - jepsen.util 2428 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,645] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,645] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,645] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,646] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 4] INFO [2022-03-13 17:19:19,647] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 37] INFO [2022-03-13 17:19:19,647] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,648] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 1] INFO [2022-03-13 17:19:19,648] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,651] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,652] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,653] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,653] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,653] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,654] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,654] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:19:19,655] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:19,656] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,657] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,658] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,658] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,660] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:19:19,661] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:19:19,730] jepsen worker 71 - jepsen.util 611 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,731] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,739] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,739] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,740] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:19:19,740] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 18] INFO [2022-03-13 17:19:19,749] jepsen worker 75 - jepsen.util 525 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,749] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,750] jepsen worker 13 - jepsen.util 733 :invoke :write [4 31] INFO [2022-03-13 17:19:19,750] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:19,757] jepsen worker 42 - jepsen.util 402 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,757] jepsen worker 23 - jepsen.util 563 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,758] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:19,759] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:19,771] jepsen worker 33 - jepsen.util 753 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,772] jepsen worker 33 - jepsen.util 753 :invoke :write [11 12] INFO [2022-03-13 17:19:19,807] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,808] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,825] jepsen worker 37 - jepsen.util 2467 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,826] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:19,832] jepsen worker 69 - jepsen.util 789 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,833] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,833] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,833] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,834] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,834] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:19:19,835] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,835] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,836] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,837] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:19,838] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,839] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:19:19,840] jepsen worker 89 - jepsen.util 2789 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,841] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,846] jepsen worker 52 - jepsen.util 682 :fail :write [17 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,846] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,846] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:19,847] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:19:19,884] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,884] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,895] jepsen worker 51 - jepsen.util 771 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,895] jepsen worker 51 - jepsen.util 771 :invoke :write [17 37] INFO [2022-03-13 17:19:19,903] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:19,903] jepsen worker nemesis - couchbase.util Rebalance check count: 666 INFO [2022-03-13 17:19:19,903] jepsen worker nemesis - couchbase.util Rebalance stuck count: 538 INFO [2022-03-13 17:19:19,907] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,907] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,939] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,939] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,939] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,940] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,940] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,940] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:19,943] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,943] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,944] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:19,944] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,944] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:19:19,946] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,946] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,947] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 45] INFO [2022-03-13 17:19:19,947] jepsen worker 38 - jepsen.util 2828 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,948] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:19:19,952] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,952] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,953] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,954] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:19,955] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:19:19,958] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,970] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,971] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:19,972] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,972] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 43] INFO [2022-03-13 17:19:19,987] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:19,988] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:19:20,003] jepsen worker 88 - jepsen.util 2428 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,003] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,003] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 44] INFO [2022-03-13 17:19:20,004] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:20,006] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,007] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:20,008] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,008] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 24] INFO [2022-03-13 17:19:20,019] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,020] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:20,021] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,022] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:20,034] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,034] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,035] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,035] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,035] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,036] jepsen worker 80 - jepsen.util 800 :invoke :write [26 48] INFO [2022-03-13 17:19:20,037] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:19:20,037] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:20,039] jepsen worker 24 - jepsen.util 654 :invoke :write [8 33] INFO [2022-03-13 17:19:20,040] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:20,048] jepsen worker 12 - jepsen.util 912 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,048] jepsen worker 61 - jepsen.util 781 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,048] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,049] jepsen worker 61 - jepsen.util 781 :invoke :write [20 40] INFO [2022-03-13 17:19:20,050] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:20,050] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:19:20,063] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,063] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:20,083] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,084] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:19:20,099] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,100] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:20,112] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,113] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:20,141] jepsen worker 89 - jepsen.util 2789 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,142] jepsen worker 35 - jepsen.util 935 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,371] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:19:20,142] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,371] jepsen worker 35 - jepsen.util 935 :invoke :write [11 23] INFO [2022-03-13 17:19:20,150] jepsen worker 4 - jepsen.util 2614 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,168] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,371] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:20,371] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:19:20,372] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:19:20,168] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,220] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,373] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:20,373] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:20,220] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,374] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:19:20,375] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,376] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,379] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:20,381] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,382] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:19:20,384] jepsen worker 71 - jepsen.util 611 :invoke :write [23 48] INFO [2022-03-13 17:19:20,223] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,386] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:20,227] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,232] jepsen worker 37 - jepsen.util 2467 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,386] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:20,259] jepsen worker 33 - jepsen.util 753 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,386] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:20,386] jepsen worker 33 - jepsen.util 753 :invoke :write [11 43] INFO [2022-03-13 17:19:20,263] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,263] jepsen worker 13 - jepsen.util 733 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,263] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,386] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:19:20,386] jepsen worker 76 - jepsen.util 796 :invoke :write [25 25] INFO [2022-03-13 17:19:20,387] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:20,387] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,391] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,396] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:19:20,397] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:19:20,263] jepsen worker 88 - jepsen.util 2428 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,398] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 48] INFO [2022-03-13 17:19:20,398] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,275] jepsen worker 78 - jepsen.util 708 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,406] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:20,407] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:19:20,276] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,409] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:20,414] jepsen worker 60 - jepsen.util 690 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,416] jepsen worker 60 - jepsen.util 690 :invoke :write [20 42] INFO [2022-03-13 17:19:20,417] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,418] jepsen worker 77 - jepsen.util 707 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,276] jepsen worker 44 - jepsen.util 764 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,276] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,418] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:20,418] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,276] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,276] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,418] jepsen worker 7 - jepsen.util 637 :invoke :write [2 28] INFO [2022-03-13 17:19:20,418] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:20,276] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:20,418] jepsen worker 70 - jepsen.util 880 :invoke :write [23 39] INFO [2022-03-13 17:19:20,418] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:20,424] jepsen worker 75 - jepsen.util 525 :invoke :write [25 23] INFO [2022-03-13 17:19:20,425] jepsen worker 77 - jepsen.util 707 :invoke :write [25 25] INFO [2022-03-13 17:19:20,277] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,426] jepsen worker 12 - jepsen.util 912 :invoke :write [4 47] INFO [2022-03-13 17:19:20,338] jepsen worker 43 - jepsen.util 673 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,351] jepsen worker 51 - jepsen.util 771 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,427] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:20,351] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,351] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,428] jepsen worker 8 - jepsen.util 728 :invoke :write [2 30] INFO [2022-03-13 17:19:20,428] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:20,428] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:20,418] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,427] jepsen worker 51 - jepsen.util 771 :invoke :write [17 49] INFO [2022-03-13 17:19:20,434] jepsen worker 62 - jepsen.util 512 :invoke :write [20 36] INFO [2022-03-13 17:19:20,440] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:20,440] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:20,440] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,444] jepsen worker 42 - jepsen.util 402 :invoke :write [14 19] INFO [2022-03-13 17:19:20,445] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:20,446] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:20,465] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,467] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:20,513] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,513] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:20,517] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,518] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:19:20,526] jepsen worker 79 - jepsen.util 1159 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,526] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,526] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:20,526] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,527] jepsen worker 44 - jepsen.util 764 :invoke :write [14 43] INFO [2022-03-13 17:19:20,528] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:20,529] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:19:20,533] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:20,533] jepsen worker 53 - jepsen.util 593 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,533] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,534] jepsen worker 53 - jepsen.util 593 :invoke :write [17 41] INFO [2022-03-13 17:19:20,535] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:20,536] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,537] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:20,564] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,564] jepsen worker 69 - jepsen.util 789 :invoke :write [23 12] INFO [2022-03-13 17:19:20,579] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,579] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,580] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:19:20,580] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:20,589] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,592] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,592] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:20,594] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,594] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:20,594] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:19:20,602] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,603] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 24] INFO [2022-03-13 17:19:20,604] jepsen worker 80 - jepsen.util 800 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,605] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:20,618] jepsen worker 59 - jepsen.util 4649 :info :write [19 4] :SyncWriteAmbiguous INFO [2022-03-13 17:19:20,619] jepsen worker 59 - jepsen.util 4739 :invoke :read [19 nil] INFO [2022-03-13 17:19:20,645] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,646] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:19:20,660] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,660] jepsen worker 61 - jepsen.util 781 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,660] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:20,661] jepsen worker 61 - jepsen.util 781 :invoke :write [20 36] INFO [2022-03-13 17:19:20,665] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,666] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 9] INFO [2022-03-13 17:19:20,671] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,671] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,672] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:20,673] jepsen worker 71 - jepsen.util 611 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,673] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:20,673] jepsen worker 43 - jepsen.util 673 :invoke :write [14 41] INFO [2022-03-13 17:19:20,674] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:20,675] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:20,726] jepsen worker 24 - jepsen.util 654 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,727] jepsen worker 12 - jepsen.util 912 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,728] jepsen worker 12 - jepsen.util 912 :invoke :write [4 15] INFO [2022-03-13 17:19:20,728] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,728] jepsen worker 24 - jepsen.util 654 :invoke :write [8 6] INFO [2022-03-13 17:19:20,728] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:19:20,733] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,734] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:19:20,749] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,749] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:20,773] jepsen worker 88 - jepsen.util 2428 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,774] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:20,781] jepsen worker 69 - jepsen.util 789 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,781] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,782] jepsen worker 69 - jepsen.util 789 :invoke :write [23 24] INFO [2022-03-13 17:19:20,783] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:20,788] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,789] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:19:20,797] jepsen worker 38 - jepsen.util 2828 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,797] jepsen worker 18 - jepsen.util 4518 :info :write [6 35] :RequestTimeoutException INFO [2022-03-13 17:19:20,798] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:19:20,798] jepsen worker 18 - jepsen.util 4608 :invoke :write [6 44] INFO [2022-03-13 17:19:20,824] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,824] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:20,838] jepsen worker 35 - jepsen.util 935 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,839] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:19:20,860] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,860] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:20,865] jepsen worker 77 - jepsen.util 707 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,865] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,865] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:20,866] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 9] INFO [2022-03-13 17:19:20,890] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,890] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,890] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,891] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:19:20,891] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:19:20,892] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:20,893] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,893] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 7] INFO [2022-03-13 17:19:20,907] jepsen worker 69 - jepsen.util 789 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,908] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:20,908] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:20,908] jepsen worker nemesis - couchbase.util Rebalance check count: 667 INFO [2022-03-13 17:19:20,908] jepsen worker nemesis - couchbase.util Rebalance stuck count: 539 INFO [2022-03-13 17:19:20,909] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,910] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,911] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 27] INFO [2022-03-13 17:19:20,914] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,914] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,915] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:20,916] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:20,917] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,917] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,918] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:19:20,918] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:19:20,922] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:19:20,935] jepsen worker 8 - jepsen.util 728 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,935] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,936] jepsen worker 8 - jepsen.util 728 :invoke :write [2 25] INFO [2022-03-13 17:19:20,937] jepsen worker 6 - jepsen.util 636 :invoke :write [2 1] INFO [2022-03-13 17:19:20,952] jepsen worker 60 - jepsen.util 690 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,953] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,952] jepsen worker 76 - jepsen.util 796 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,953] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:20,954] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:20,954] jepsen worker 76 - jepsen.util 796 :invoke :write [25 7] INFO [2022-03-13 17:19:20,958] jepsen worker 33 - jepsen.util 753 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,958] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,958] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:20,959] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:20,973] jepsen worker 86 - jepsen.util 4406 :info :write [28 16] :SyncWriteAmbiguous INFO [2022-03-13 17:19:20,973] jepsen worker 86 - jepsen.util 4496 :invoke :write [28 27] INFO [2022-03-13 17:19:20,976] jepsen worker 39 - jepsen.util 4899 :info :write [13 21] :RequestTimeoutException INFO [2022-03-13 17:19:20,976] jepsen worker 4 - jepsen.util 2614 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:20,977] jepsen worker 39 - jepsen.util 4989 :invoke :read [13 nil] INFO [2022-03-13 17:19:20,978] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:21,025] jepsen worker 68 - jepsen.util 4748 :ok :read [22 30] INFO [2022-03-13 17:19:21,026] jepsen worker 68 - jepsen.util 4748 :invoke :read [22 nil] INFO [2022-03-13 17:19:21,027] jepsen worker 89 - jepsen.util 2789 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,028] jepsen worker 37 - jepsen.util 2467 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,028] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,028] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:19:21,029] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 1] INFO [2022-03-13 17:19:21,029] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,032] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,032] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,040] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,041] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 31] INFO [2022-03-13 17:19:21,043] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,043] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,043] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,044] jepsen worker 69 - jepsen.util 789 :invoke :write [23 23] INFO [2022-03-13 17:19:21,053] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,054] jepsen worker 71 - jepsen.util 611 :invoke :write [23 19] INFO [2022-03-13 17:19:21,056] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,062] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:19:21,087] jepsen worker 1 - jepsen.util 4321 :ok :read [0 41] INFO [2022-03-13 17:19:21,089] jepsen worker 1 - jepsen.util 4321 :invoke :write [0 39] INFO [2022-03-13 17:19:21,094] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,095] jepsen worker 68 - jepsen.util 4748 :ok :read [22 30] INFO [2022-03-13 17:19:21,095] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:21,095] jepsen worker 68 - jepsen.util 4748 :invoke :write [22 12] INFO [2022-03-13 17:19:21,111] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,111] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,111] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:21,111] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,112] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:21,112] jepsen worker 21 - jepsen.util 831 :invoke :write [7 46] INFO [2022-03-13 17:19:21,124] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,124] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,124] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:19:21,125] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:21,140] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,140] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,140] jepsen worker 75 - jepsen.util 525 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,140] jepsen worker 53 - jepsen.util 593 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,141] jepsen worker 22 - jepsen.util 652 :invoke :write [7 8] INFO [2022-03-13 17:19:21,142] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,142] jepsen worker 52 - jepsen.util 682 :invoke :write [17 1] INFO [2022-03-13 17:19:21,142] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:21,144] jepsen worker 65 - jepsen.util 4835 :ok :read [21 :nil] INFO [2022-03-13 17:19:21,145] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,145] jepsen worker 65 - jepsen.util 4835 :invoke :read [21 nil] INFO [2022-03-13 17:19:21,145] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,145] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,146] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:19:21,147] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,147] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,148] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,149] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 2] INFO [2022-03-13 17:19:21,157] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,159] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:19:21,163] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,163] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,164] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:21,165] jepsen worker 78 - jepsen.util 708 :invoke :write [26 21] INFO [2022-03-13 17:19:21,169] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,169] jepsen worker 42 - jepsen.util 402 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,169] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,170] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:21,171] jepsen worker 42 - jepsen.util 402 :invoke :write [14 31] INFO [2022-03-13 17:19:21,173] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:21,185] jepsen worker 10 - jepsen.util 4870 :ok :read [3 25] INFO [2022-03-13 17:19:21,186] jepsen worker 84 - jepsen.util 4944 :info :write [28 14] :RequestTimeoutException INFO [2022-03-13 17:19:21,186] jepsen worker 10 - jepsen.util 4870 :invoke :read [3 nil] INFO [2022-03-13 17:19:21,187] jepsen worker 84 - jepsen.util 5034 :invoke :read [28 nil] INFO [2022-03-13 17:19:21,193] jepsen worker 69 - jepsen.util 789 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,195] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,208] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,209] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:19:21,224] jepsen worker 39 - jepsen.util 4989 :ok :read [13 39] INFO [2022-03-13 17:19:21,225] jepsen worker 39 - jepsen.util 4989 :invoke :write [13 25] INFO [2022-03-13 17:19:21,226] jepsen worker 66 - jepsen.util 4386 :ok :read [22 30] INFO [2022-03-13 17:19:21,227] jepsen worker 66 - jepsen.util 4386 :invoke :write [22 8] INFO [2022-03-13 17:19:21,232] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,232] jepsen worker 51 - jepsen.util 771 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,235] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:21,235] jepsen worker 47 - jepsen.util 4457 :ok :read [15 48] INFO [2022-03-13 17:19:21,237] jepsen worker 80 - jepsen.util 800 :invoke :write [26 28] INFO [2022-03-13 17:19:21,238] jepsen worker 47 - jepsen.util 4457 :invoke :read [15 nil] INFO [2022-03-13 17:19:21,245] jepsen worker 73 - jepsen.util 4933 :ok :read [24 24] INFO [2022-03-13 17:19:21,246] jepsen worker 73 - jepsen.util 4933 :invoke :write [24 43] INFO [2022-03-13 17:19:21,263] jepsen worker 65 - jepsen.util 4835 :ok :read [21 :nil] INFO [2022-03-13 17:19:21,264] jepsen worker 65 - jepsen.util 4835 :invoke :write [21 23] INFO [2022-03-13 17:19:21,268] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,268] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,269] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 25] INFO [2022-03-13 17:19:21,270] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:21,279] jepsen worker 56 - jepsen.util 4556 :ok :read [18 21] INFO [2022-03-13 17:19:21,279] jepsen worker 56 - jepsen.util 4556 :invoke :write [18 3] INFO [2022-03-13 17:19:21,280] jepsen worker 10 - jepsen.util 4870 :ok :read [3 25] INFO [2022-03-13 17:19:21,281] jepsen worker 10 - jepsen.util 4870 :invoke :read [3 nil] INFO [2022-03-13 17:19:21,310] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,310] jepsen worker 43 - jepsen.util 673 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,310] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,310] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,310] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,310] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:21,311] jepsen worker 60 - jepsen.util 690 :invoke :write [20 41] INFO [2022-03-13 17:19:21,312] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:21,312] jepsen worker 26 - jepsen.util 566 :invoke :write [8 4] INFO [2022-03-13 17:19:21,313] jepsen worker 43 - jepsen.util 673 :invoke :write [14 31] INFO [2022-03-13 17:19:21,321] jepsen worker 37 - jepsen.util 2467 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,321] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:19:21,327] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,328] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:19:21,333] jepsen worker 71 - jepsen.util 611 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,333] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,334] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,334] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:19:21,335] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,336] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,338] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,338] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,340] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,343] jepsen worker 69 - jepsen.util 789 :invoke :write [23 16] INFO [2022-03-13 17:19:21,371] jepsen worker 28 - jepsen.util 4618 :info :write [9 9] :SyncWriteAmbiguous INFO [2022-03-13 17:19:21,372] jepsen worker 28 - jepsen.util 4708 :invoke :write [9 43] INFO [2022-03-13 17:19:21,376] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,376] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,377] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:19:21,377] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:19:21,377] jepsen worker 47 - jepsen.util 4457 :ok :read [15 48] INFO [2022-03-13 17:19:21,378] jepsen worker 47 - jepsen.util 4457 :invoke :write [15 17] INFO [2022-03-13 17:19:21,385] jepsen worker 10 - jepsen.util 4870 :ok :read [3 25] INFO [2022-03-13 17:19:21,386] jepsen worker 10 - jepsen.util 4870 :invoke :read [3 nil] INFO [2022-03-13 17:19:21,407] jepsen worker 24 - jepsen.util 654 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,408] jepsen worker 24 - jepsen.util 654 :invoke :write [8 18] INFO [2022-03-13 17:19:21,411] jepsen worker 89 - jepsen.util 2789 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,412] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,419] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,420] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:19:21,440] jepsen worker 22 - jepsen.util 652 :fail :write [7 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,441] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:19:21,442] jepsen worker 3 - jepsen.util 2253 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,442] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:21,461] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,462] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:19:21,476] jepsen worker 61 - jepsen.util 781 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,476] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,476] jepsen worker 62 - jepsen.util 512 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,477] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,478] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:21,479] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:21,479] jepsen worker 88 - jepsen.util 2428 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,480] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,509] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,509] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,509] jepsen worker 76 - jepsen.util 796 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,510] jepsen worker 75 - jepsen.util 525 :invoke :write [25 24] INFO [2022-03-13 17:19:21,510] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,511] jepsen worker 13 - jepsen.util 733 :invoke :write [4 15] INFO [2022-03-13 17:19:21,512] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:19:21,514] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,520] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,521] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,521] jepsen worker 10 - jepsen.util 4870 :ok :read [3 25] INFO [2022-03-13 17:19:21,522] jepsen worker 10 - jepsen.util 4870 :invoke :read [3 nil] INFO [2022-03-13 17:19:21,523] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,523] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,524] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,524] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,524] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,525] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:19:21,526] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,527] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,528] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,529] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,531] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,531] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,531] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,531] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,531] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,532] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,532] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,532] jepsen worker 35 - jepsen.util 935 :invoke :write [11 29] INFO [2022-03-13 17:19:21,533] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,533] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:19:21,534] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,534] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:19:21,535] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:21,537] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:21,537] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:19:21,538] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:21,543] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,545] jepsen worker 76 - jepsen.util 796 :invoke :write [25 37] INFO [2022-03-13 17:19:21,579] jepsen worker 69 - jepsen.util 789 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,580] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,599] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,599] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,600] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:21,601] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:21,617] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,618] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,619] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:21,619] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 12] INFO [2022-03-13 17:19:21,638] jepsen worker 52 - jepsen.util 682 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,638] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,639] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:21,639] jepsen worker 53 - jepsen.util 593 :invoke :write [17 12] INFO [2022-03-13 17:19:21,642] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,643] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,650] jepsen worker 79 - jepsen.util 1159 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,650] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,651] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:21,651] jepsen worker 23 - jepsen.util 563 :invoke :write [7 19] INFO [2022-03-13 17:19:21,663] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,664] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:19:21,666] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,666] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:19:21,668] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,668] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,669] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,669] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:19:21,671] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:19:21,672] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:21,676] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,676] jepsen worker 21 - jepsen.util 831 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,678] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:21,678] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:19:21,680] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,681] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:21,682] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,683] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:21,705] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,706] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,706] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,707] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:21,711] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,711] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,712] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:19:21,712] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,712] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,713] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:19:21,714] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,715] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 2] INFO [2022-03-13 17:19:21,723] jepsen worker 10 - jepsen.util 4870 :ok :read [3 25] INFO [2022-03-13 17:19:21,726] jepsen worker 10 - jepsen.util 4870 :invoke :read [3 nil] INFO [2022-03-13 17:19:21,747] jepsen worker 44 - jepsen.util 764 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,748] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:21,763] jepsen worker 13 - jepsen.util 733 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,764] jepsen worker 13 - jepsen.util 733 :invoke :write [4 42] INFO [2022-03-13 17:19:21,796] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,796] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,798] jepsen worker 80 - jepsen.util 800 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,799] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,802] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:19:21,803] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:21,827] jepsen worker 17 - jepsen.util 4787 :info :write [5 21] :RequestTimeoutException INFO [2022-03-13 17:19:21,828] jepsen worker 17 - jepsen.util 4877 :invoke :read [5 nil] INFO [2022-03-13 17:19:21,833] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,833] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,833] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:19:21,834] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:19:21,870] jepsen worker 60 - jepsen.util 690 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,872] jepsen worker 88 - jepsen.util 2428 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,872] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,872] jepsen worker 60 - jepsen.util 690 :invoke :write [20 12] INFO [2022-03-13 17:19:21,873] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 47] INFO [2022-03-13 17:19:21,873] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 41] INFO [2022-03-13 17:19:21,898] jepsen worker 10 - jepsen.util 4870 :ok :read [3 25] INFO [2022-03-13 17:19:21,899] jepsen worker 10 - jepsen.util 4870 :invoke :read [3 nil] INFO [2022-03-13 17:19:21,901] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,901] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,902] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,902] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,904] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,904] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,905] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:21,906] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:19:21,908] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,908] jepsen worker 78 - jepsen.util 708 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,908] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,908] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,908] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,909] jepsen worker 77 - jepsen.util 707 :invoke :write [25 13] INFO [2022-03-13 17:19:21,909] jepsen worker 69 - jepsen.util 789 :invoke :write [23 8] INFO [2022-03-13 17:19:21,911] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:21,912] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:21,913] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:21,915] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:21,915] jepsen worker nemesis - couchbase.util Rebalance check count: 668 INFO [2022-03-13 17:19:21,915] jepsen worker nemesis - couchbase.util Rebalance stuck count: 540 INFO [2022-03-13 17:19:21,929] jepsen worker 24 - jepsen.util 654 :fail :write [8 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,930] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,930] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:21,935] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:21,935] jepsen worker 89 - jepsen.util 2789 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,935] jepsen worker 46 - jepsen.util 4726 :info :write [15 47] :SyncWriteAmbiguous INFO [2022-03-13 17:19:21,935] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,935] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,936] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:21,937] jepsen worker 46 - jepsen.util 4816 :invoke :write [15 31] INFO [2022-03-13 17:19:21,938] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:21,938] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:21,946] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,946] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:21,950] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,950] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:19:21,985] jepsen worker 75 - jepsen.util 525 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,985] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:21,989] jepsen worker 7 - jepsen.util 637 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:21,990] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:22,001] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,002] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,004] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,005] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:19:22,027] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,027] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:22,029] jepsen worker 35 - jepsen.util 935 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,030] jepsen worker 35 - jepsen.util 935 :invoke :write [11 18] INFO [2022-03-13 17:19:22,035] jepsen worker 27 - jepsen.util 4797 :ok :read [9 26] INFO [2022-03-13 17:19:22,036] jepsen worker 27 - jepsen.util 4797 :invoke :write [9 7] INFO [2022-03-13 17:19:22,036] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,036] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,037] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,041] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,041] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,042] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:22,044] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,044] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,045] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,046] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:19:22,049] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,050] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,051] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,052] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,053] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,054] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,055] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,056] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,057] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,058] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 42] INFO [2022-03-13 17:19:22,084] jepsen worker 26 - jepsen.util 566 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,085] jepsen worker 26 - jepsen.util 566 :invoke :write [8 12] INFO [2022-03-13 17:19:22,105] jepsen worker 88 - jepsen.util 2428 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,106] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,107] jepsen worker 69 - jepsen.util 789 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,108] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,108] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:19:22,109] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:22,111] jepsen worker 10 - jepsen.util 4870 :ok :read [3 25] INFO [2022-03-13 17:19:22,112] jepsen worker 10 - jepsen.util 4870 :invoke :write [3 1] INFO [2022-03-13 17:19:22,120] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,121] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:22,122] jepsen worker 9 - jepsen.util 4239 :ok :read [3 25] INFO [2022-03-13 17:19:22,122] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,123] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:22,123] jepsen worker 9 - jepsen.util 4239 :invoke :write [3 23] INFO [2022-03-13 17:19:22,123] jepsen worker 74 - jepsen.util 4394 :info :write [24 37] :RequestTimeoutException INFO [2022-03-13 17:19:22,124] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,124] jepsen worker 74 - jepsen.util 4484 :invoke :read [24 nil] INFO [2022-03-13 17:19:22,124] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:22,125] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,126] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:22,127] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,127] jepsen worker 71 - jepsen.util 611 :invoke :write [23 17] INFO [2022-03-13 17:19:22,145] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,145] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,146] jepsen worker 33 - jepsen.util 753 :invoke :write [11 34] INFO [2022-03-13 17:19:22,146] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:22,151] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,151] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:19:22,154] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,155] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 7] INFO [2022-03-13 17:19:22,186] jepsen worker 20 - jepsen.util 4790 :info :write [6 47] :RequestTimeoutException INFO [2022-03-13 17:19:22,187] jepsen worker 20 - jepsen.util 4880 :invoke :write [6 32] INFO [2022-03-13 17:19:22,188] jepsen worker 76 - jepsen.util 796 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,190] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:19:22,192] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,195] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:19:22,215] jepsen worker 23 - jepsen.util 563 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,216] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,216] jepsen worker 12 - jepsen.util 912 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,216] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,217] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:22,218] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:22,219] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:22,220] jepsen worker 23 - jepsen.util 563 :invoke :write [7 9] INFO [2022-03-13 17:19:22,228] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,229] jepsen worker 78 - jepsen.util 708 :invoke :write [26 8] INFO [2022-03-13 17:19:22,232] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,233] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:19:22,244] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,244] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,244] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:19:22,245] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,250] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,250] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,250] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,250] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,250] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,250] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,251] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:22,251] jepsen worker 24 - jepsen.util 654 :invoke :write [8 33] INFO [2022-03-13 17:19:22,252] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:19:22,253] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:22,254] jepsen worker 34 - jepsen.util 934 :invoke :write [11 5] INFO [2022-03-13 17:19:22,258] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:22,265] jepsen worker 74 - jepsen.util 4484 :ok :read [24 24] INFO [2022-03-13 17:19:22,266] jepsen worker 74 - jepsen.util 4484 :invoke :write [24 27] INFO [2022-03-13 17:19:22,279] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,279] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:22,286] jepsen worker 43 - jepsen.util 673 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,287] jepsen worker 43 - jepsen.util 673 :invoke :write [14 3] INFO [2022-03-13 17:19:22,309] jepsen worker 4 - jepsen.util 2614 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,309] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,311] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:19:22,312] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:19:22,315] jepsen worker 89 - jepsen.util 2789 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,315] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,348] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,348] jepsen worker 35 - jepsen.util 935 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,349] jepsen worker 51 - jepsen.util 771 :invoke :write [17 14] INFO [2022-03-13 17:19:22,349] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:19:22,353] jepsen worker 71 - jepsen.util 611 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,354] jepsen worker 71 - jepsen.util 611 :invoke :write [23 38] INFO [2022-03-13 17:19:22,370] jepsen worker 37 - jepsen.util 2467 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,370] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,371] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:19:22,372] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:19:22,390] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,390] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,391] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,393] jepsen worker 55 - jepsen.util 4105 :info :write [18 6] :SyncWriteAmbiguous INFO [2022-03-13 17:19:22,394] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:22,395] jepsen worker 55 - jepsen.util 4195 :invoke :write [18 13] INFO [2022-03-13 17:19:22,398] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,398] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:19:22,398] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,398] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,399] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,400] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:19:22,401] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,402] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,404] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,405] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 8] INFO [2022-03-13 17:19:22,407] jepsen worker 83 - jepsen.util 4223 :info :write [27 11] :RequestTimeoutException INFO [2022-03-13 17:19:22,408] jepsen worker 83 - jepsen.util 4313 :invoke :read [27 nil] INFO [2022-03-13 17:19:22,414] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,414] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:22,443] jepsen worker 83 - jepsen.util 4313 :ok :read [27 4] INFO [2022-03-13 17:19:22,443] jepsen worker 83 - jepsen.util 4313 :invoke :read [27 nil] INFO [2022-03-13 17:19:22,449] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,450] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,452] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,453] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,454] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,455] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,456] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,457] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,458] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,459] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:19:22,478] jepsen worker 83 - jepsen.util 4313 :ok :read [27 4] INFO [2022-03-13 17:19:22,479] jepsen worker 83 - jepsen.util 4313 :invoke :read [27 nil] INFO [2022-03-13 17:19:22,484] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,485] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:19:22,496] jepsen worker 85 - jepsen.util 4225 :info :write [28 15] :RequestTimeoutException INFO [2022-03-13 17:19:22,497] jepsen worker 85 - jepsen.util 4315 :invoke :write [28 48] INFO [2022-03-13 17:19:22,519] jepsen worker 83 - jepsen.util 4313 :ok :read [27 4] INFO [2022-03-13 17:19:22,519] jepsen worker 83 - jepsen.util 4313 :invoke :write [27 20] INFO [2022-03-13 17:19:22,539] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,540] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:19:22,547] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,547] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,555] jepsen worker 32 - jepsen.util 4802 :info :write [10 21] :SyncWriteAmbiguous INFO [2022-03-13 17:19:22,555] jepsen worker 32 - jepsen.util 4892 :invoke :write [10 17] INFO [2022-03-13 17:19:22,585] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,586] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:22,608] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,609] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:19:22,610] jepsen worker 29 - jepsen.util 5069 :ok :read [9 26] INFO [2022-03-13 17:19:22,610] jepsen worker 29 - jepsen.util 5069 :invoke :read [9 nil] INFO [2022-03-13 17:19:22,612] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,613] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:22,628] jepsen worker 13 - jepsen.util 733 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,628] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,632] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:22,632] jepsen worker 75 - jepsen.util 525 :invoke :write [25 33] INFO [2022-03-13 17:19:22,647] jepsen worker 77 - jepsen.util 707 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,647] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,647] jepsen worker 8 - jepsen.util 728 :fail :write [2 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,647] jepsen worker 77 - jepsen.util 707 :invoke :write [25 8] INFO [2022-03-13 17:19:22,647] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,648] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 42] INFO [2022-03-13 17:19:22,648] jepsen worker 8 - jepsen.util 728 :invoke :write [2 29] INFO [2022-03-13 17:19:22,654] jepsen worker 61 - jepsen.util 781 :invoke :write [20 0] INFO [2022-03-13 17:19:22,687] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,689] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:19:22,698] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,698] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,699] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:22,700] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:22,727] jepsen worker 89 - jepsen.util 2789 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,729] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,729] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,730] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:22,732] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,732] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,733] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:19:22,734] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:19:22,738] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,738] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,738] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,738] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:19:22,739] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:19:22,739] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 45] INFO [2022-03-13 17:19:22,762] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,763] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,763] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,765] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:19:22,767] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:22,768] jepsen worker 23 - jepsen.util 563 :fail :write [7 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,768] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:19:22,769] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:19:22,774] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,775] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:19:22,780] jepsen worker 16 - jepsen.util 4426 :info :write [5 31] :RequestTimeoutException INFO [2022-03-13 17:19:22,781] jepsen worker 16 - jepsen.util 4516 :invoke :write [5 45] INFO [2022-03-13 17:19:22,799] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,799] jepsen worker 33 - jepsen.util 753 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,799] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,800] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,801] jepsen worker 14 - jepsen.util 374 :invoke :write [4 16] INFO [2022-03-13 17:19:22,802] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:22,803] jepsen worker 33 - jepsen.util 753 :invoke :write [11 35] INFO [2022-03-13 17:19:22,803] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:22,837] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,837] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:22,863] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,864] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:22,873] jepsen worker 6 - jepsen.util 636 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,873] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:22,874] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:22,875] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:22,886] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,887] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 49] INFO [2022-03-13 17:19:22,892] jepsen worker 71 - jepsen.util 611 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,893] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:22,912] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,913] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:22,919] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,920] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,920] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,920] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:22,920] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:22,920] jepsen worker nemesis - couchbase.util Rebalance check count: 669 INFO [2022-03-13 17:19:22,920] jepsen worker nemesis - couchbase.util Rebalance stuck count: 541 INFO [2022-03-13 17:19:22,921] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:22,922] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:22,973] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,974] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:22,994] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,994] jepsen worker 4 - jepsen.util 2614 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:22,995] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:22,996] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:23,015] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,016] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,016] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:19:23,016] jepsen worker 53 - jepsen.util 593 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,017] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,018] jepsen worker 53 - jepsen.util 593 :invoke :write [17 5] INFO [2022-03-13 17:19:23,024] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,024] jepsen worker 78 - jepsen.util 708 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,024] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:19:23,025] jepsen worker 78 - jepsen.util 708 :invoke :write [26 8] INFO [2022-03-13 17:19:23,030] jepsen worker 45 - jepsen.util 4815 :info :write [15 20] :RequestTimeoutException INFO [2022-03-13 17:19:23,031] jepsen worker 45 - jepsen.util 4905 :invoke :read [15 nil] INFO [2022-03-13 17:19:23,044] jepsen worker 24 - jepsen.util 654 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,045] jepsen worker 24 - jepsen.util 654 :invoke :write [8 15] INFO [2022-03-13 17:19:23,046] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,047] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,049] jepsen worker 89 - jepsen.util 2789 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,050] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:19:23,050] jepsen worker 72 - jepsen.util 4662 :info :write [24 44] :RequestTimeoutException INFO [2022-03-13 17:19:23,051] jepsen worker 72 - jepsen.util 4752 :invoke :write [24 46] INFO [2022-03-13 17:19:23,056] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,057] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:23,058] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,059] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:23,060] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,061] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:19:23,085] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,085] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,086] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:23,086] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:19:23,090] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,090] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:23,092] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,093] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:19:23,116] jepsen worker 42 - jepsen.util 402 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,116] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,116] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,117] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:23,118] jepsen worker 45 - jepsen.util 4905 :ok :read [15 48] INFO [2022-03-13 17:19:23,118] jepsen worker 7 - jepsen.util 637 :invoke :write [2 29] INFO [2022-03-13 17:19:23,119] jepsen worker 45 - jepsen.util 4905 :invoke :write [15 31] INFO [2022-03-13 17:19:23,119] jepsen worker 42 - jepsen.util 402 :invoke :write [14 43] INFO [2022-03-13 17:19:23,125] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,133] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:19:23,138] jepsen worker 34 - jepsen.util 934 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,139] jepsen worker 34 - jepsen.util 934 :invoke :write [11 36] INFO [2022-03-13 17:19:23,143] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,144] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:19:23,150] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,151] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,176] jepsen worker 64 - jepsen.util 4474 :info :write [21 36] :RequestTimeoutException INFO [2022-03-13 17:19:23,177] jepsen worker 64 - jepsen.util 4564 :invoke :read [21 nil] INFO [2022-03-13 17:19:23,179] jepsen worker 75 - jepsen.util 525 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,179] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,179] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,179] jepsen worker 75 - jepsen.util 525 :invoke :write [25 37] INFO [2022-03-13 17:19:23,180] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:23,181] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:23,191] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,192] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,193] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:19:23,193] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:19:23,195] jepsen worker 19 - jepsen.util 4429 :info :write [6 8] :RequestTimeoutException INFO [2022-03-13 17:19:23,195] jepsen worker 19 - jepsen.util 4519 :invoke :write [6 41] INFO [2022-03-13 17:19:23,198] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,199] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,200] jepsen worker 64 - jepsen.util 4564 :ok :read [21 :nil] INFO [2022-03-13 17:19:23,200] jepsen worker 64 - jepsen.util 4564 :invoke :write [21 7] INFO [2022-03-13 17:19:23,219] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,220] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 35] INFO [2022-03-13 17:19:23,228] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,229] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,229] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 37] INFO [2022-03-13 17:19:23,230] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,235] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,236] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:23,236] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,237] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:19:23,262] jepsen worker 60 - jepsen.util 690 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,265] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,266] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,266] jepsen worker 60 - jepsen.util 690 :invoke :write [20 30] INFO [2022-03-13 17:19:23,267] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,268] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,268] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,268] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:19:23,269] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:23,270] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,271] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,272] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,272] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,273] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:19:23,274] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,275] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,276] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,277] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,278] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:19:23,279] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,282] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,282] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,283] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,284] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,284] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,285] jepsen worker 71 - jepsen.util 611 :invoke :write [23 45] INFO [2022-03-13 17:19:23,286] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,287] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:19:23,330] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,330] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:19:23,358] jepsen worker 51 - jepsen.util 771 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,358] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:23,377] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,377] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] INFO [2022-03-13 17:19:23,391] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,391] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:23,427] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,428] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,429] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,429] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:23,433] jepsen worker 79 - jepsen.util 1159 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,433] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,434] jepsen worker 70 - jepsen.util 880 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,434] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 22] INFO [2022-03-13 17:19:23,436] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:23,436] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,451] jepsen worker 88 - jepsen.util 2428 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,451] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:23,453] jepsen worker 14 - jepsen.util 374 :fail :write [4 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,454] jepsen worker 14 - jepsen.util 374 :invoke :write [4 44] INFO [2022-03-13 17:19:23,467] jepsen worker 26 - jepsen.util 566 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,467] jepsen worker 26 - jepsen.util 566 :invoke :write [8 21] INFO [2022-03-13 17:19:23,484] jepsen worker 36 - jepsen.util 2466 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,485] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,510] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,511] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,511] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,512] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:19:23,517] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,517] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:19:23,541] jepsen worker 75 - jepsen.util 525 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,541] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:23,551] jepsen worker 63 - jepsen.util 4743 :info :write [21 15] :RequestTimeoutException INFO [2022-03-13 17:19:23,552] jepsen worker 63 - jepsen.util 4833 :invoke :write [21 47] INFO [2022-03-13 17:19:23,564] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,565] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:23,566] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,567] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 15] INFO [2022-03-13 17:19:23,579] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,580] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:19:23,583] jepsen worker 71 - jepsen.util 611 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,584] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,584] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:19:23,585] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:23,588] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,588] jepsen worker 69 - jepsen.util 789 :invoke :write [23 5] INFO [2022-03-13 17:19:23,618] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,618] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:19:23,647] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,647] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,648] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 26] INFO [2022-03-13 17:19:23,648] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 26] INFO [2022-03-13 17:19:23,674] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,674] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,674] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:19:23,675] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,715] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,716] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:19:23,718] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,718] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:19:23,767] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:23,767] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,767] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,767] jepsen worker 44 - jepsen.util 764 :invoke :write [14 45] INFO [2022-03-13 17:19:23,768] jepsen worker 6 - jepsen.util 636 :invoke :write [2 26] INFO [2022-03-13 17:19:23,768] jepsen worker 62 - jepsen.util 512 :invoke :write [20 25] INFO [2022-03-13 17:19:23,782] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,782] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,783] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:19:23,783] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 31] INFO [2022-03-13 17:19:23,796] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,797] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:19:23,868] jepsen worker 31 - jepsen.util 4711 :info :write [10 3] :RequestTimeoutException INFO [2022-03-13 17:19:23,868] jepsen worker 31 - jepsen.util 4801 :invoke :write [10 35] INFO [2022-03-13 17:19:23,877] jepsen worker 5 - jepsen.util 2705 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,877] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,877] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:23,878] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:23,880] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,881] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,906] jepsen worker 52 - jepsen.util 682 :ok :read [17 20] INFO [2022-03-13 17:19:23,907] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:23,920] jepsen worker 88 - jepsen.util 2428 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,920] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,921] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 3] INFO [2022-03-13 17:19:23,923] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:19:23,926] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:23,926] jepsen worker nemesis - couchbase.util Rebalance check count: 670 INFO [2022-03-13 17:19:23,926] jepsen worker nemesis - couchbase.util Rebalance stuck count: 542 INFO [2022-03-13 17:19:23,940] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,940] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,940] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:19:23,941] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:19:23,964] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,965] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:19:23,995] jepsen worker 36 - jepsen.util 2466 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,996] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:23,998] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:23,999] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:24,000] jepsen worker 30 - jepsen.util 4620 :info :write [10 8] :RequestTimeoutException INFO [2022-03-13 17:19:24,002] jepsen worker 30 - jepsen.util 4710 :invoke :write [10 3] INFO [2022-03-13 17:19:24,009] jepsen worker 15 - jepsen.util 4515 :ok :read [5 43] INFO [2022-03-13 17:19:24,010] jepsen worker 15 - jepsen.util 4515 :invoke :read [5 nil] INFO [2022-03-13 17:19:24,034] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,034] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,035] jepsen worker 80 - jepsen.util 800 :invoke :write [26 29] INFO [2022-03-13 17:19:24,039] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:24,046] jepsen worker 37 - jepsen.util 2467 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,047] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,048] jepsen worker 51 - jepsen.util 771 :ok :read [17 20] INFO [2022-03-13 17:19:24,048] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:24,079] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,080] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:19:24,095] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,095] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,095] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,096] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:19:24,097] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:19:24,098] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,099] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,100] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:24,119] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,120] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,123] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,123] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,125] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,126] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:19:24,134] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,135] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,137] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,143] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:19:24,158] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,159] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 11] INFO [2022-03-13 17:19:24,161] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,162] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:24,203] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,203] jepsen worker 88 - jepsen.util 2428 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,204] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,204] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 46] INFO [2022-03-13 17:19:24,218] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,219] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,236] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,236] jepsen worker 23 - jepsen.util 563 :invoke :write [7 16] INFO [2022-03-13 17:19:24,250] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,250] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,251] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,251] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:19:24,259] jepsen worker 33 - jepsen.util 753 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,260] jepsen worker 33 - jepsen.util 753 :invoke :write [11 25] INFO [2022-03-13 17:19:24,262] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,263] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,276] jepsen worker 54 - jepsen.util 4374 :info :write [18 11] :RequestTimeoutException INFO [2022-03-13 17:19:24,277] jepsen worker 54 - jepsen.util 4464 :invoke :read [18 nil] INFO [2022-03-13 17:19:24,285] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,286] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 40] INFO [2022-03-13 17:19:24,324] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,325] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:24,329] jepsen worker 7 - jepsen.util 637 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,329] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,329] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,330] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:24,330] jepsen worker 70 - jepsen.util 880 :invoke :write [23 18] INFO [2022-03-13 17:19:24,339] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:24,341] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,342] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,365] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,366] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,366] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,367] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,369] jepsen worker 89 - jepsen.util 2789 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,370] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,370] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 2] INFO [2022-03-13 17:19:24,371] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:24,380] jepsen worker 24 - jepsen.util 654 :fail :write [8 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,381] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,381] jepsen worker 81 - jepsen.util 5211 :info :write [27 30] :RequestTimeoutException INFO [2022-03-13 17:19:24,381] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:24,382] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:24,382] jepsen worker 81 - jepsen.util 5301 :invoke :read [27 nil] INFO [2022-03-13 17:19:24,390] jepsen worker 88 - jepsen.util 2428 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,391] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,392] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,392] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:24,397] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,397] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,398] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,398] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:19:24,399] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,405] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,405] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 20] INFO [2022-03-13 17:19:24,406] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 25] INFO [2022-03-13 17:19:24,410] jepsen worker 81 - jepsen.util 5301 :ok :read [27 4] INFO [2022-03-13 17:19:24,411] jepsen worker 81 - jepsen.util 5301 :invoke :read [27 nil] INFO [2022-03-13 17:19:24,437] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,437] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,437] jepsen worker 53 - jepsen.util 593 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,438] jepsen worker 81 - jepsen.util 5301 :ok :read [27 4] INFO [2022-03-13 17:19:24,438] jepsen worker 81 - jepsen.util 5301 :invoke :read [27 nil] INFO [2022-03-13 17:19:24,439] jepsen worker 75 - jepsen.util 525 :invoke :write [25 46] INFO [2022-03-13 17:19:24,439] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:24,440] jepsen worker 53 - jepsen.util 593 :invoke :write [17 42] INFO [2022-03-13 17:19:24,464] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,465] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:24,468] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,468] jepsen worker 81 - jepsen.util 5301 :ok :read [27 4] INFO [2022-03-13 17:19:24,469] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:24,470] jepsen worker 81 - jepsen.util 5301 :invoke :read [27 nil] INFO [2022-03-13 17:19:24,475] jepsen worker 89 - jepsen.util 2789 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,476] jepsen worker 3 - jepsen.util 2253 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,476] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,476] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:19:24,476] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,476] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,477] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,477] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:24,478] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:19:24,480] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,482] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,482] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,482] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,483] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:24,483] jepsen worker 34 - jepsen.util 934 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,483] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:24,484] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,484] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:24,485] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,485] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,486] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:24,487] jepsen worker 51 - jepsen.util 771 :invoke :write [17 27] INFO [2022-03-13 17:19:24,491] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,492] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,492] jepsen worker 69 - jepsen.util 789 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,493] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:24,494] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:19:24,494] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:24,500] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,501] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:19:24,515] jepsen worker 81 - jepsen.util 5301 :ok :read [27 4] INFO [2022-03-13 17:19:24,517] jepsen worker 81 - jepsen.util 5301 :invoke :write [27 30] INFO [2022-03-13 17:19:24,538] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,538] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 32] INFO [2022-03-13 17:19:24,549] jepsen worker 8 - jepsen.util 728 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,549] jepsen worker 78 - jepsen.util 708 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,550] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:24,551] jepsen worker 8 - jepsen.util 728 :invoke :write [2 49] INFO [2022-03-13 17:19:24,601] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,601] jepsen worker 88 - jepsen.util 2428 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,601] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:19:24,602] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:24,604] jepsen worker 80 - jepsen.util 800 :fail :write [26 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,604] jepsen worker 80 - jepsen.util 800 :invoke :write [26 7] INFO [2022-03-13 17:19:24,604] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,605] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:19:24,615] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,617] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,650] jepsen worker 60 - jepsen.util 690 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,651] jepsen worker 60 - jepsen.util 690 :invoke :write [20 1] INFO [2022-03-13 17:19:24,681] jepsen worker 38 - jepsen.util 2828 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,682] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:19:24,684] jepsen worker 37 - jepsen.util 2467 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,684] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,685] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 36] INFO [2022-03-13 17:19:24,685] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:24,710] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,711] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:19:24,720] jepsen worker 14 - jepsen.util 374 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,721] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:24,722] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,723] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,727] jepsen worker 79 - jepsen.util 1159 :fail :write [26 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,727] jepsen worker 44 - jepsen.util 764 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,728] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:24,729] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:24,742] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,742] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,742] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:24,743] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:24,747] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,747] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:24,749] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,750] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:24,761] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,761] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:19:24,819] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,819] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:19:24,822] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,823] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,823] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,823] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,824] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,825] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:24,825] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,825] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,825] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,826] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,826] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,826] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:19:24,827] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:19:24,827] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,828] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:19:24,828] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,828] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,829] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,830] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,831] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:19:24,871] jepsen worker 77 - jepsen.util 707 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,872] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:24,876] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,876] jepsen worker 37 - jepsen.util 2467 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,877] jepsen worker 69 - jepsen.util 789 :invoke :write [23 45] INFO [2022-03-13 17:19:24,877] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,878] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 2] INFO [2022-03-13 17:19:24,879] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,885] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,886] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:19:24,896] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,896] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,896] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:19:24,897] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:24,898] jepsen worker 26 - jepsen.util 566 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,898] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:24,932] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:24,932] jepsen worker nemesis - couchbase.util Rebalance check count: 671 INFO [2022-03-13 17:19:24,932] jepsen worker nemesis - couchbase.util Rebalance stuck count: 543 INFO [2022-03-13 17:19:24,934] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,935] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:24,946] jepsen worker 62 - jepsen.util 512 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,946] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:24,952] jepsen worker 33 - jepsen.util 753 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,952] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,952] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,952] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,952] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,952] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:24,953] jepsen worker 21 - jepsen.util 831 :invoke :write [7 0] INFO [2022-03-13 17:19:24,954] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:24,955] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:24,957] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:24,987] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:24,988] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:19:25,005] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,006] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,012] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,012] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,012] jepsen worker 75 - jepsen.util 525 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,013] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,014] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,015] jepsen worker 13 - jepsen.util 733 :invoke :write [4 35] INFO [2022-03-13 17:19:25,015] jepsen worker 75 - jepsen.util 525 :invoke :write [25 15] INFO [2022-03-13 17:19:25,019] jepsen worker 76 - jepsen.util 796 :invoke :write [25 13] INFO [2022-03-13 17:19:25,036] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,037] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:25,042] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,043] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,045] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,046] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,047] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,049] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,058] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,059] jepsen worker 51 - jepsen.util 771 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,059] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:19:25,060] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:25,069] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,069] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,069] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,070] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 4] INFO [2022-03-13 17:19:25,071] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:19:25,072] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,073] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,074] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:19:25,085] jepsen worker 88 - jepsen.util 2428 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,086] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 39] INFO [2022-03-13 17:19:25,097] jepsen worker 61 - jepsen.util 781 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,098] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:25,115] jepsen worker 69 - jepsen.util 789 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,115] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,116] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:19:25,158] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,160] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,160] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,161] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,161] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:19:25,162] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,163] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:19:25,171] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,171] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,172] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:19:25,172] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:19:25,204] jepsen worker 80 - jepsen.util 800 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,205] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,205] jepsen worker 14 - jepsen.util 374 :invoke :write [4 0] INFO [2022-03-13 17:19:25,206] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:25,211] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,212] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,236] jepsen worker 6 - jepsen.util 636 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,236] jepsen worker 6 - jepsen.util 636 :invoke :write [2 34] INFO [2022-03-13 17:19:25,249] jepsen worker 43 - jepsen.util 673 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,249] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,249] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:25,250] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:25,252] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,253] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 24] INFO [2022-03-13 17:19:25,258] jepsen worker 11 - jepsen.util 4511 :info :write [3 34] :RequestTimeoutException INFO [2022-03-13 17:19:25,259] jepsen worker 11 - jepsen.util 4601 :invoke :read [3 nil] INFO [2022-03-13 17:19:25,286] jepsen worker 36 - jepsen.util 2466 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,286] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:19:25,292] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,293] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:25,310] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,310] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,311] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,311] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,311] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,311] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:25,312] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:25,312] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:25,313] jepsen worker 34 - jepsen.util 934 :invoke :write [11 36] INFO [2022-03-13 17:19:25,314] jepsen worker 24 - jepsen.util 654 :invoke :write [8 1] INFO [2022-03-13 17:19:25,354] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,355] jepsen worker 69 - jepsen.util 789 :invoke :write [23 17] INFO [2022-03-13 17:19:25,358] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,359] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,366] jepsen worker 0 - jepsen.util 4410 :info :write [0 12] :RequestTimeoutException INFO [2022-03-13 17:19:25,367] jepsen worker 0 - jepsen.util 4500 :invoke :write [0 19] INFO [2022-03-13 17:19:25,374] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,374] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,374] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,375] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:19:25,376] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,376] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,383] jepsen worker 76 - jepsen.util 796 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,383] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,384] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:19:25,384] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:25,401] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,401] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,402] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 38] INFO [2022-03-13 17:19:25,403] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:25,404] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,405] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:25,410] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,410] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:19:25,426] jepsen worker 21 - jepsen.util 831 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,426] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,426] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,427] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:19:25,428] jepsen worker 51 - jepsen.util 771 :invoke :write [17 20] INFO [2022-03-13 17:19:25,428] jepsen worker 80 - jepsen.util 800 :invoke :write [26 23] INFO [2022-03-13 17:19:25,445] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,446] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,448] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,449] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,449] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:25,450] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 36] INFO [2022-03-13 17:19:25,458] jepsen worker 42 - jepsen.util 402 :fail :write [14 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,458] jepsen worker 23 - jepsen.util 563 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,458] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,458] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,458] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,458] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,459] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:19:25,460] jepsen worker 88 - jepsen.util 2428 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,460] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:19:25,461] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:25,462] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 9] INFO [2022-03-13 17:19:25,463] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:19:25,463] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:25,464] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:25,469] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,469] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,469] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,470] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:25,470] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:25,471] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:25,474] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,475] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,475] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,476] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:25,478] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:19:25,478] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,479] jepsen worker 77 - jepsen.util 707 :invoke :write [25 35] INFO [2022-03-13 17:19:25,480] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:25,512] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,512] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:25,516] jepsen worker 89 - jepsen.util 2789 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,516] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 26] INFO [2022-03-13 17:19:25,521] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,522] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:25,524] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,525] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:19:25,530] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,530] jepsen worker 14 - jepsen.util 374 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,530] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,530] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,531] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:25,532] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:19:25,535] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,535] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,536] jepsen worker 25 - jepsen.util 565 :invoke :write [8 39] INFO [2022-03-13 17:19:25,537] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:25,538] jepsen worker 14 - jepsen.util 374 :invoke :write [4 23] INFO [2022-03-13 17:19:25,539] jepsen worker 69 - jepsen.util 789 :fail :write [23 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,539] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,544] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,544] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,545] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,545] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,545] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,546] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,547] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,548] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:19:25,548] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,550] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,550] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,551] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,552] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,553] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,554] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:19:25,603] jepsen worker 37 - jepsen.util 2467 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,603] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 36] INFO [2022-03-13 17:19:25,606] jepsen worker 70 - jepsen.util 880 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,606] jepsen worker 70 - jepsen.util 880 :invoke :write [23 19] INFO [2022-03-13 17:19:25,615] jepsen worker 75 - jepsen.util 525 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,616] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:19:25,619] jepsen worker 34 - jepsen.util 934 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,620] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:25,620] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,621] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:19:25,630] jepsen worker 4 - jepsen.util 2614 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,630] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,630] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,630] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:19:25,631] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:25,631] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:19:25,641] jepsen worker 57 - jepsen.util 4647 :info :write [19 34] :RequestTimeoutException INFO [2022-03-13 17:19:25,642] jepsen worker 57 - jepsen.util 4737 :invoke :read [19 nil] INFO [2022-03-13 17:19:25,668] jepsen worker 82 - jepsen.util 4852 :info :write [27 21] :RequestTimeoutException INFO [2022-03-13 17:19:25,669] jepsen worker 82 - jepsen.util 4942 :invoke :write [27 6] INFO [2022-03-13 17:19:25,675] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,676] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:19:25,680] jepsen worker 2 - jepsen.util 4952 :info :write [0 16] :RequestTimeoutException INFO [2022-03-13 17:19:25,681] jepsen worker 2 - jepsen.util 5042 :invoke :read [0 nil] INFO [2022-03-13 17:19:25,689] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,690] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,700] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,701] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:19:25,705] jepsen worker 89 - jepsen.util 2789 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,706] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:19:25,714] jepsen worker 53 - jepsen.util 593 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,715] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:25,720] jepsen worker 24 - jepsen.util 654 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,721] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:25,722] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,723] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:19:25,783] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,784] jepsen worker 51 - jepsen.util 771 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,785] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:19:25,786] jepsen worker 51 - jepsen.util 771 :invoke :write [17 34] INFO [2022-03-13 17:19:25,799] jepsen worker 13 - jepsen.util 733 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,799] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:25,814] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:25,814] jepsen worker 8 - jepsen.util 728 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,814] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,814] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,815] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,815] jepsen worker 7 - jepsen.util 637 :invoke :write [2 5] INFO [2022-03-13 17:19:25,816] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:25,816] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:25,817] jepsen worker 8 - jepsen.util 728 :invoke :write [2 35] INFO [2022-03-13 17:19:25,817] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:25,820] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,820] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:25,822] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,823] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:25,852] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,853] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 44] INFO [2022-03-13 17:19:25,855] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,856] jepsen worker 33 - jepsen.util 753 :invoke :write [11 43] INFO [2022-03-13 17:19:25,857] jepsen worker 80 - jepsen.util 800 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,858] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:25,878] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,879] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,879] jepsen worker 60 - jepsen.util 690 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,879] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,879] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:25,879] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,880] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,880] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:25,880] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,881] jepsen worker 60 - jepsen.util 690 :invoke :write [20 40] INFO [2022-03-13 17:19:25,881] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,882] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:19:25,882] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:25,883] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,883] jepsen worker 52 - jepsen.util 682 :invoke :write [17 8] INFO [2022-03-13 17:19:25,884] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:25,884] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:19:25,884] jepsen worker 53 - jepsen.util 593 :invoke :write [17 8] INFO [2022-03-13 17:19:25,885] jepsen worker 43 - jepsen.util 673 :invoke :write [14 11] INFO [2022-03-13 17:19:25,887] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:25,887] jepsen worker 79 - jepsen.util 1159 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,888] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,888] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 5] INFO [2022-03-13 17:19:25,889] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:25,896] jepsen worker 25 - jepsen.util 565 :fail :write [8 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,898] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:25,911] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,912] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:25,923] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,924] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,924] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,924] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:25,925] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:25,925] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:25,926] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,927] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,927] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:19:25,929] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:19:25,937] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:25,937] jepsen worker nemesis - couchbase.util Rebalance check count: 672 INFO [2022-03-13 17:19:25,937] jepsen worker nemesis - couchbase.util Rebalance stuck count: 544 INFO [2022-03-13 17:19:25,944] jepsen worker 41 - jepsen.util 4631 :info :write [13 5] :RequestTimeoutException INFO [2022-03-13 17:19:25,944] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,945] jepsen worker 41 - jepsen.util 4721 :invoke :write [13 8] INFO [2022-03-13 17:19:25,946] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:25,948] jepsen worker 14 - jepsen.util 374 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,948] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,949] jepsen worker 14 - jepsen.util 374 :invoke :write [4 44] INFO [2022-03-13 17:19:25,950] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:19:25,950] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,950] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,950] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:25,951] jepsen worker 34 - jepsen.util 934 :invoke :write [11 3] INFO [2022-03-13 17:19:25,953] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,954] jepsen worker 76 - jepsen.util 796 :invoke :write [25 17] INFO [2022-03-13 17:19:25,955] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,955] jepsen worker 26 - jepsen.util 566 :invoke :write [8 21] INFO [2022-03-13 17:19:25,960] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,961] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:25,996] jepsen worker 6 - jepsen.util 636 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,996] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:25,997] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:25,998] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,043] jepsen worker 51 - jepsen.util 771 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,043] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,043] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,045] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:26,046] jepsen worker 37 - jepsen.util 2467 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,046] jepsen worker 21 - jepsen.util 831 :invoke :write [7 28] INFO [2022-03-13 17:19:26,047] jepsen worker 80 - jepsen.util 800 :invoke :write [26 18] INFO [2022-03-13 17:19:26,048] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 23] INFO [2022-03-13 17:19:26,070] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,071] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:26,092] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,093] jepsen worker 61 - jepsen.util 781 :invoke :write [20 43] INFO [2022-03-13 17:19:26,115] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,115] jepsen worker 33 - jepsen.util 753 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,115] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,116] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:26,117] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:26,117] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,118] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:26,118] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:19:26,120] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,120] jepsen worker 4 - jepsen.util 2614 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,120] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:26,121] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,122] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,122] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,123] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:26,123] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:26,134] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,134] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:19:26,174] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,174] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,174] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,175] jepsen worker 42 - jepsen.util 402 :invoke :write [14 20] INFO [2022-03-13 17:19:26,175] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:26,176] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:19:26,178] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,179] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:26,180] jepsen worker 60 - jepsen.util 690 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,180] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,180] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,180] jepsen worker 75 - jepsen.util 525 :invoke :write [25 27] INFO [2022-03-13 17:19:26,181] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:26,181] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:26,182] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,184] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:26,185] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,185] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 49] INFO [2022-03-13 17:19:26,198] jepsen worker 14 - jepsen.util 374 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,199] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:26,211] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,211] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 35] INFO [2022-03-13 17:19:26,248] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,249] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:19:26,269] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,270] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,271] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,272] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,283] jepsen worker 77 - jepsen.util 707 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,284] jepsen worker 77 - jepsen.util 707 :invoke :write [25 18] INFO [2022-03-13 17:19:26,288] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,289] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,295] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,295] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,299] jepsen worker 34 - jepsen.util 934 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,299] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,299] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,299] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,300] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:26,301] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:26,301] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,302] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:19:26,320] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,320] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:19:26,323] jepsen worker 89 - jepsen.util 2789 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,324] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:26,348] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,348] jepsen worker 52 - jepsen.util 682 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,348] jepsen worker 76 - jepsen.util 796 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,349] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:19:26,350] jepsen worker 52 - jepsen.util 682 :invoke :write [17 8] INFO [2022-03-13 17:19:26,350] jepsen worker 13 - jepsen.util 733 :invoke :write [4 43] INFO [2022-03-13 17:19:26,352] jepsen worker 70 - jepsen.util 880 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,352] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,352] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,352] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,353] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:26,354] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:26,355] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:26,356] jepsen worker 44 - jepsen.util 764 :invoke :write [14 3] INFO [2022-03-13 17:19:26,362] jepsen worker 21 - jepsen.util 831 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,362] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:26,392] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,393] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:19:26,405] jepsen worker 37 - jepsen.util 2467 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,405] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 47] INFO [2022-03-13 17:19:26,406] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,406] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:19:26,410] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,411] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:19:26,434] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,435] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,435] jepsen worker 23 - jepsen.util 563 :invoke :write [7 14] INFO [2022-03-13 17:19:26,436] jepsen worker 78 - jepsen.util 708 :invoke :write [26 5] INFO [2022-03-13 17:19:26,442] jepsen worker 80 - jepsen.util 800 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,442] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,442] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,443] jepsen worker 80 - jepsen.util 800 :invoke :write [26 0] INFO [2022-03-13 17:19:26,443] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,444] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:19:26,444] jepsen worker 33 - jepsen.util 753 :invoke :write [11 7] INFO [2022-03-13 17:19:26,445] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:26,468] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,469] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:19:26,472] jepsen worker 43 - jepsen.util 673 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,472] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,473] jepsen worker 43 - jepsen.util 673 :invoke :write [14 34] INFO [2022-03-13 17:19:26,473] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:19:26,476] jepsen worker 75 - jepsen.util 525 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,477] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:26,489] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,489] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:19:26,505] jepsen worker 7 - jepsen.util 637 :fail :write [2 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,507] jepsen worker 7 - jepsen.util 637 :invoke :write [2 35] INFO [2022-03-13 17:19:26,517] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,517] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,517] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,517] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,517] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,517] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,519] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:26,520] jepsen worker 51 - jepsen.util 771 :invoke :write [17 5] INFO [2022-03-13 17:19:26,521] jepsen worker 14 - jepsen.util 374 :invoke :write [4 41] INFO [2022-03-13 17:19:26,522] jepsen worker 34 - jepsen.util 934 :invoke :write [11 48] INFO [2022-03-13 17:19:26,523] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:26,523] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:26,533] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,534] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,537] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,538] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:19:26,575] jepsen worker 79 - jepsen.util 1159 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,576] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:26,585] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,585] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:19:26,601] jepsen worker 80 - jepsen.util 800 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,603] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:26,657] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,657] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,658] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,659] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,669] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,669] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,669] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,670] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,677] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,677] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:19:26,678] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,678] jepsen worker 33 - jepsen.util 753 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,679] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:26,679] jepsen worker 21 - jepsen.util 831 :invoke :write [7 39] INFO [2022-03-13 17:19:26,685] jepsen worker 26 - jepsen.util 566 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,685] jepsen worker 8 - jepsen.util 728 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,685] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,685] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,686] jepsen worker 8 - jepsen.util 728 :invoke :write [2 18] INFO [2022-03-13 17:19:26,687] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:26,688] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:26,689] jepsen worker 26 - jepsen.util 566 :invoke :write [8 2] INFO [2022-03-13 17:19:26,723] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,723] jepsen worker 88 - jepsen.util 2428 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,724] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,725] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:19:26,725] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,726] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,739] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,740] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,741] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,742] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,752] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,752] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,753] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:26,753] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:19:26,755] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,755] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,755] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,756] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,757] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,757] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,757] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:26,757] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:19:26,762] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,763] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 4] INFO [2022-03-13 17:19:26,766] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,766] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:19:26,775] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,775] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,775] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,775] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:26,776] jepsen worker 25 - jepsen.util 565 :invoke :write [8 10] INFO [2022-03-13 17:19:26,776] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:26,780] jepsen worker 67 - jepsen.util 3937 :info :write [22 18] :RequestTimeoutException INFO [2022-03-13 17:19:26,781] jepsen worker 67 - jepsen.util 4027 :invoke :write [22 34] INFO [2022-03-13 17:19:26,782] jepsen worker 61 - jepsen.util 781 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,783] jepsen worker 61 - jepsen.util 781 :invoke :write [20 39] INFO [2022-03-13 17:19:26,809] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,810] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:26,811] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,811] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,811] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,812] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,854] jepsen worker 51 - jepsen.util 771 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,855] jepsen worker 51 - jepsen.util 771 :invoke :write [17 32] INFO [2022-03-13 17:19:26,859] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,860] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:26,864] jepsen worker 42 - jepsen.util 402 :fail :write [14 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,865] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,864] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,865] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,865] jepsen worker 70 - jepsen.util 880 :invoke :write [23 37] INFO [2022-03-13 17:19:26,866] jepsen worker 42 - jepsen.util 402 :invoke :write [14 27] INFO [2022-03-13 17:19:26,868] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:26,873] jepsen worker 13 - jepsen.util 733 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,873] jepsen worker 62 - jepsen.util 512 :invoke :write [20 40] INFO [2022-03-13 17:19:26,873] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,875] jepsen worker 13 - jepsen.util 733 :invoke :write [4 7] INFO [2022-03-13 17:19:26,876] jepsen worker 76 - jepsen.util 796 :invoke :write [25 5] INFO [2022-03-13 17:19:26,873] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,880] jepsen worker 53 - jepsen.util 593 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,881] jepsen worker 75 - jepsen.util 525 :invoke :write [25 9] INFO [2022-03-13 17:19:26,880] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,881] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,882] jepsen worker 53 - jepsen.util 593 :invoke :write [17 27] INFO [2022-03-13 17:19:26,883] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:26,884] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:26,903] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,907] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,909] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,909] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,910] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,910] jepsen worker 37 - jepsen.util 2467 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,910] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:26,911] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 49] INFO [2022-03-13 17:19:26,912] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,912] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:19:26,912] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,913] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:19:26,939] jepsen worker 14 - jepsen.util 374 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,940] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,941] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:26,941] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,941] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] INFO [2022-03-13 17:19:26,942] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:26,944] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:26,944] jepsen worker nemesis - couchbase.util Rebalance check count: 673 INFO [2022-03-13 17:19:26,944] jepsen worker nemesis - couchbase.util Rebalance stuck count: 545 INFO [2022-03-13 17:19:26,949] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,950] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:19:26,976] jepsen worker 77 - jepsen.util 707 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,976] jepsen worker 52 - jepsen.util 682 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:26,976] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:26,977] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:27,016] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,016] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,016] jepsen worker 34 - jepsen.util 934 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,018] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:27,018] jepsen worker 33 - jepsen.util 753 :invoke :write [11 5] INFO [2022-03-13 17:19:27,019] jepsen worker 34 - jepsen.util 934 :invoke :write [11 37] INFO [2022-03-13 17:19:27,033] jepsen worker 36 - jepsen.util 2466 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,034] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:27,041] jepsen worker 44 - jepsen.util 764 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,042] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:27,047] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,048] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 27] INFO [2022-03-13 17:19:27,070] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,071] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,089] jepsen worker 21 - jepsen.util 831 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,090] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,090] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,090] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:27,091] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,091] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:27,114] jepsen worker 78 - jepsen.util 708 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,114] jepsen worker 78 - jepsen.util 708 :invoke :write [26 34] INFO [2022-03-13 17:19:27,116] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,117] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:27,125] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,125] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,132] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,133] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:27,147] jepsen worker 25 - jepsen.util 565 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,147] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:19:27,156] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,157] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:19:27,165] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,165] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,166] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,167] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,168] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,168] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,168] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:19:27,169] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:19:27,174] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,175] jepsen worker 7 - jepsen.util 637 :fail :write [2 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,177] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:27,177] jepsen worker 7 - jepsen.util 637 :invoke :write [2 46] INFO [2022-03-13 17:19:27,204] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,204] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,233] jepsen worker 75 - jepsen.util 525 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,233] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:27,244] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,244] jepsen worker 51 - jepsen.util 771 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,244] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,245] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:27,246] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,250] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:27,251] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:27,251] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,275] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,275] jepsen worker 23 - jepsen.util 563 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,276] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:27,277] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] WARN [2022-03-13 17:19:27,281] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":80015,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1684447,"total_server_duration_us":620015,"encode_duration_us":20246,"operation_id":"0x135d5","timeout_ms":10000,"last_server_duration_us":63053,"total_duration_us":10010328},{"operation_name":"upsert","last_dispatch_duration_us":65964,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6443080,"total_server_duration_us":42903,"encode_duration_us":30380,"operation_id":"0x13653","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10007478},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":18386,"operation_id":"0x1371d","timeout_ms":10000,"total_duration_us":10004805},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","encode_duration_us":17967,"operation_id":"0x1380c","timeout_ms":10000,"total_duration_us":10003373},{"operation_name":"upsert","last_dispatch_duration_us":8568590,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9086302,"total_server_duration_us":323211,"encode_duration_us":17835,"operation_id":"0x13311","timeout_ms":10000,"last_server_duration_us":23559,"total_duration_us":10000906},{"operation_name":"upsert","last_dispatch_duration_us":40821,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":1548675,"total_server_duration_us":616254,"encode_duration_us":25677,"operation_id":"0x1357f","timeout_ms":10000,"last_server_duration_us":22887,"total_duration_us":10000818},{"operation_name":"upsert","last_dispatch_duration_us":9075579,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9075579,"total_server_duration_us":9023573,"encode_duration_us":23575,"operation_id":"0x13477","timeout_ms":10000,"last_server_duration_us":9023573,"total_duration_us":9082473},{"operation_name":"upsert","last_dispatch_duration_us":644197,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1412902,"total_server_duration_us":160563,"encode_duration_us":21155,"operation_id":"0x13c0b","timeout_ms":10000,"last_server_duration_us":88246,"total_duration_us":1434797},{"operation_name":"upsert","last_dispatch_duration_us":473629,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1306238,"total_server_duration_us":174055,"encode_duration_us":18117,"operation_id":"0x13c2f","timeout_ms":10000,"last_server_duration_us":67750,"total_duration_us":1325660},{"operation_name":"upsert","last_dispatch_duration_us":553182,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1292543,"total_server_duration_us":225080,"encode_duration_us":20824,"operation_id":"0x13c21","timeout_ms":10000,"last_server_duration_us":85742,"total_duration_us":1314253}],"total_count":82}} INFO [2022-03-13 17:19:27,283] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,284] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,284] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,284] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:19:27,284] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:19:27,285] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,286] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,287] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,288] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,288] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:19:27,299] jepsen worker 37 - jepsen.util 2467 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,299] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,299] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,301] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,301] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 30] INFO [2022-03-13 17:19:27,302] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,303] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,304] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,304] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,306] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,306] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,307] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:19:27,310] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,310] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,311] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,312] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 1] INFO [2022-03-13 17:19:27,355] jepsen worker 33 - jepsen.util 753 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,355] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,363] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,364] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:19:27,390] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,390] jepsen worker 8 - jepsen.util 728 :fail :write [2 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,391] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:27,391] jepsen worker 8 - jepsen.util 728 :invoke :write [2 15] INFO [2022-03-13 17:19:27,406] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,407] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:19:27,409] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,410] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,417] jepsen worker 43 - jepsen.util 673 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,418] jepsen worker 43 - jepsen.util 673 :invoke :write [14 4] INFO [2022-03-13 17:19:27,456] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,457] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:27,484] jepsen worker 34 - jepsen.util 934 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,484] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,484] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,484] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,486] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:19:27,486] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,487] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:27,488] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,495] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,495] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,495] jepsen worker 38 - jepsen.util 2828 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,495] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,496] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,496] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,499] jepsen worker 76 - jepsen.util 796 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,500] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:19:27,502] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,503] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,503] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,504] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:19:27,504] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,504] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 9] INFO [2022-03-13 17:19:27,519] jepsen worker 61 - jepsen.util 781 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,519] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,520] jepsen worker 61 - jepsen.util 781 :invoke :write [20 20] INFO [2022-03-13 17:19:27,521] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:27,538] jepsen worker 88 - jepsen.util 2428 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,539] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 12] INFO [2022-03-13 17:19:27,624] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,624] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,627] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,628] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,628] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,628] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:19:27,629] jepsen worker 51 - jepsen.util 771 :invoke :write [17 13] INFO [2022-03-13 17:19:27,629] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:19:27,630] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,630] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,631] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,631] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,631] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:27,632] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,632] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,633] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,633] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,634] jepsen worker 24 - jepsen.util 654 :invoke :write [8 21] INFO [2022-03-13 17:19:27,642] jepsen worker 42 - jepsen.util 402 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,643] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:27,653] jepsen worker 89 - jepsen.util 2789 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,653] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:19:27,674] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,675] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:19:27,696] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,696] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,697] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:19:27,698] jepsen worker 69 - jepsen.util 789 :invoke :write [23 4] INFO [2022-03-13 17:19:27,727] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,727] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,728] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,728] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:19:27,728] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,729] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:19:27,732] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,732] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,732] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,732] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,732] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,733] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,733] jepsen worker 33 - jepsen.util 753 :invoke :write [11 43] INFO [2022-03-13 17:19:27,733] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,734] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:27,735] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,735] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:27,736] jepsen worker 80 - jepsen.util 800 :invoke :write [26 23] INFO [2022-03-13 17:19:27,736] jepsen worker 35 - jepsen.util 935 :invoke :write [11 27] INFO [2022-03-13 17:19:27,746] jepsen worker 37 - jepsen.util 2467 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,747] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 2] INFO [2022-03-13 17:19:27,750] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:19:27,759] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,760] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:27,765] jepsen worker 13 - jepsen.util 733 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,765] jepsen worker 26 - jepsen.util 566 :fail :write [8 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,765] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,765] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,766] jepsen worker 60 - jepsen.util 690 :invoke :write [20 30] INFO [2022-03-13 17:19:27,767] jepsen worker 75 - jepsen.util 525 :invoke :write [25 6] INFO [2022-03-13 17:19:27,769] jepsen worker 13 - jepsen.util 733 :invoke :write [4 34] INFO [2022-03-13 17:19:27,778] jepsen worker 26 - jepsen.util 566 :invoke :write [8 22] INFO [2022-03-13 17:19:27,833] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,834] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,833] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,834] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,835] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:19:27,835] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,836] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:27,836] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,837] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,837] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:19:27,839] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 0] INFO [2022-03-13 17:19:27,839] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,843] jepsen worker 78 - jepsen.util 708 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,843] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,843] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,843] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,844] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:27,845] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 5] INFO [2022-03-13 17:19:27,845] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:27,846] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:27,853] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,854] jepsen worker 14 - jepsen.util 374 :invoke :write [4 28] INFO [2022-03-13 17:19:27,870] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,870] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:27,926] jepsen worker 70 - jepsen.util 880 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,926] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:27,949] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,949] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:27,950] jepsen worker nemesis - couchbase.util Rebalance check count: 674 INFO [2022-03-13 17:19:27,950] jepsen worker nemesis - couchbase.util Rebalance stuck count: 546 INFO [2022-03-13 17:19:27,950] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:27,951] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,952] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:19:27,956] jepsen worker 88 - jepsen.util 2428 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,957] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:19:27,967] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,967] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,967] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,967] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:19:27,968] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:19:27,968] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:19:27,996] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,996] jepsen worker 53 - jepsen.util 593 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,997] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,997] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:27,997] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:27,998] jepsen worker 53 - jepsen.util 593 :invoke :write [17 15] INFO [2022-03-13 17:19:27,999] jepsen worker 22 - jepsen.util 652 :invoke :write [7 1] INFO [2022-03-13 17:19:28,000] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:28,003] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,003] jepsen worker 61 - jepsen.util 781 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,003] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,003] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,004] jepsen worker 78 - jepsen.util 708 :invoke :write [26 8] INFO [2022-03-13 17:19:28,005] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:28,006] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,006] jepsen worker 23 - jepsen.util 563 :invoke :write [7 37] INFO [2022-03-13 17:19:28,006] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:28,007] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:28,029] jepsen worker 89 - jepsen.util 2789 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,030] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:19:28,069] jepsen worker 69 - jepsen.util 789 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,069] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,069] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:19:28,070] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:19:28,079] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,080] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,080] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:19:28,081] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:19:28,152] jepsen worker 51 - jepsen.util 771 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,152] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:28,154] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,155] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,155] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,155] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,161] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,162] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,163] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,164] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:19:28,166] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,166] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:19:28,176] jepsen worker 62 - jepsen.util 512 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,176] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,176] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:28,177] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:28,177] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:28,178] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:28,183] jepsen worker 24 - jepsen.util 654 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,183] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,183] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,183] jepsen worker 24 - jepsen.util 654 :invoke :write [8 30] INFO [2022-03-13 17:19:28,184] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:28,185] jepsen worker 21 - jepsen.util 831 :invoke :write [7 27] INFO [2022-03-13 17:19:28,221] jepsen worker 37 - jepsen.util 2467 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,221] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:19:28,239] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,240] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:28,262] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,263] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 5] INFO [2022-03-13 17:19:28,311] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,312] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,316] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,316] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:28,325] jepsen worker 43 - jepsen.util 673 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,326] jepsen worker 43 - jepsen.util 673 :invoke :write [14 1] INFO [2022-03-13 17:19:28,327] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,328] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,330] jepsen worker 33 - jepsen.util 753 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,331] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:28,334] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,335] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:28,340] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,341] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,341] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:28,342] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,367] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,367] jepsen worker 79 - jepsen.util 1159 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,368] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,368] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:28,369] jepsen worker 61 - jepsen.util 781 :invoke :write [20 2] INFO [2022-03-13 17:19:28,369] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:28,380] jepsen worker 75 - jepsen.util 525 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,381] jepsen worker 75 - jepsen.util 525 :invoke :write [25 18] INFO [2022-03-13 17:19:28,388] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,388] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,388] jepsen worker 38 - jepsen.util 2828 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,389] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,389] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,390] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:19:28,390] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,391] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,397] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,397] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,398] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,399] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,400] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,400] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 31] INFO [2022-03-13 17:19:28,403] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,403] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,404] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:28,405] jepsen worker 69 - jepsen.util 789 :invoke :write [23 18] INFO [2022-03-13 17:19:28,405] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,405] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,406] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:28,406] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,407] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,407] jepsen worker 35 - jepsen.util 935 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,407] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:28,408] jepsen worker 35 - jepsen.util 935 :invoke :write [11 4] INFO [2022-03-13 17:19:28,409] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,409] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:19:28,410] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,422] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,457] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,457] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,457] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,458] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:19:28,458] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:19:28,459] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,462] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,463] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,464] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,464] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,465] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:19:28,465] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,499] jepsen worker 7 - jepsen.util 637 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,499] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,500] jepsen worker 12 - jepsen.util 912 :invoke :write [4 21] INFO [2022-03-13 17:19:28,502] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:28,517] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,517] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:28,520] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,520] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 14] INFO [2022-03-13 17:19:28,547] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,547] jepsen worker 14 - jepsen.util 374 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,547] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:19:28,548] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] WARN [2022-03-13 17:19:28,574] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17019472,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18054961,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9588566,"encode_duration_us":17401,"operation_id":"0x12b76","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10001329},{"operation_name":"upsert","last_dispatch_duration_us":16867280,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17568993,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9167685,"encode_duration_us":21015,"operation_id":"0x12f83","timeout_ms":10000,"last_server_duration_us":9073483,"total_duration_us":10007422}],"total_count":2}} INFO [2022-03-13 17:19:28,588] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,589] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,602] jepsen worker 38 - jepsen.util 2828 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,603] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,671] jepsen worker 78 - jepsen.util 708 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,672] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,672] jepsen worker 78 - jepsen.util 708 :invoke :write [26 44] INFO [2022-03-13 17:19:28,673] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:19:28,678] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,678] jepsen worker 89 - jepsen.util 2789 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,679] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:19:28,680] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:19:28,684] jepsen worker 60 - jepsen.util 690 :fail :write [20 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,685] jepsen worker 60 - jepsen.util 690 :invoke :write [20 16] INFO [2022-03-13 17:19:28,686] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException WARN [2022-03-13 17:19:28,686] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":18125629,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":18125629,"total_server_duration_us":500,"operation_id":"0x12f33","timeout_ms":30000,"last_server_duration_us":500,"total_duration_us":18125995},{"operation_name":"get","last_dispatch_duration_us":13583145,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":13583145,"total_server_duration_us":9227,"operation_id":"0x131f0","timeout_ms":30000,"last_server_duration_us":9227,"total_duration_us":13586090},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":29145,"operation_id":"0x13428","timeout_ms":10000,"total_duration_us":10007534},{"operation_name":"upsert","last_dispatch_duration_us":245542,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9218778,"total_server_duration_us":230630,"encode_duration_us":35497,"operation_id":"0x13596","timeout_ms":10000,"last_server_duration_us":28218,"total_duration_us":10006043},{"operation_name":"get","last_dispatch_duration_us":9013905,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9013905,"total_server_duration_us":38,"operation_id":"0x134ea","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":9017002},{"operation_name":"get","last_dispatch_duration_us":6014367,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6014367,"total_server_duration_us":11118,"operation_id":"0x136cb","timeout_ms":30000,"last_server_duration_us":11118,"total_duration_us":6017730},{"operation_name":"get","last_dispatch_duration_us":5143474,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":5143474,"total_server_duration_us":23,"operation_id":"0x1375e","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":5147463},{"operation_name":"upsert","last_dispatch_duration_us":968407,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":968407,"total_server_duration_us":52658,"encode_duration_us":20017,"operation_id":"0x13dc1","timeout_ms":10000,"last_server_duration_us":52658,"total_duration_us":972394},{"operation_name":"upsert","last_dispatch_duration_us":310921,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":857390,"total_server_duration_us":264743,"encode_duration_us":19163,"operation_id":"0x13c7c","timeout_ms":10000,"last_server_duration_us":66957,"total_duration_us":883810},{"operation_name":"get","last_dispatch_duration_us":756231,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":756231,"total_server_duration_us":38,"operation_id":"0x13de4","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":756415}],"total_count":15}} INFO [2022-03-13 17:19:28,687] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:28,687] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,687] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,688] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,689] jepsen worker 71 - jepsen.util 611 :invoke :write [23 16] INFO [2022-03-13 17:19:28,696] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,696] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 44] WARN [2022-03-13 17:19:28,697] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9079799,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9778195,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9236271,"encode_duration_us":26018,"operation_id":"0x134da","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007949},{"operation_name":"upsert","last_dispatch_duration_us":19829218,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19829218,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9075609,"encode_duration_us":28867,"operation_id":"0x13173","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10009437}],"total_count":2}} INFO [2022-03-13 17:19:28,710] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,711] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,750] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,750] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:19:28,754] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,755] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,757] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,757] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,758] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,758] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] WARN [2022-03-13 17:19:28,775] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":11225382,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":11225382,"total_server_duration_us":282264,"operation_id":"0x13403","timeout_ms":30000,"last_server_duration_us":282264,"total_duration_us":11227706},{"operation_name":"upsert","last_dispatch_duration_us":11179401,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11179401,"total_server_duration_us":21082,"encode_duration_us":23418,"operation_id":"0x1355e","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10008103},{"operation_name":"upsert","last_dispatch_duration_us":84613,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9459518,"total_server_duration_us":277179,"encode_duration_us":32581,"operation_id":"0x1357a","timeout_ms":10000,"last_server_duration_us":62668,"total_duration_us":10006797},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18175,"operation_id":"0x13721","timeout_ms":10000,"total_duration_us":10004882},{"operation_name":"get","last_dispatch_duration_us":9200105,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9200105,"total_server_duration_us":12375,"operation_id":"0x136a0","timeout_ms":30000,"last_server_duration_us":12375,"total_duration_us":9208457},{"operation_name":"get","last_dispatch_duration_us":8552399,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8552399,"total_server_duration_us":43,"operation_id":"0x135d7","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":8554438},{"operation_name":"upsert","last_dispatch_duration_us":1392734,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2922997,"total_server_duration_us":209612,"encode_duration_us":19333,"operation_id":"0x13baa","timeout_ms":10000,"last_server_duration_us":111819,"total_duration_us":2941951},{"operation_name":"upsert","last_dispatch_duration_us":1336349,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2696246,"total_server_duration_us":148083,"encode_duration_us":25889,"operation_id":"0x13c02","timeout_ms":10000,"last_server_duration_us":68680,"total_duration_us":2716701},{"operation_name":"upsert","last_dispatch_duration_us":1406707,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1406707,"total_server_duration_us":63182,"encode_duration_us":18362,"operation_id":"0x13c60","timeout_ms":10000,"last_server_duration_us":63182,"total_duration_us":1411766},{"operation_name":"upsert","last_dispatch_duration_us":1391859,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":1391859,"total_server_duration_us":72862,"encode_duration_us":22019,"operation_id":"0x13c1c","timeout_ms":10000,"last_server_duration_us":72862,"total_duration_us":1396755}],"total_count":62}} WARN [2022-03-13 17:19:28,775] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":55734,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":6307775,"total_server_duration_us":435176,"encode_duration_us":17968,"operation_id":"0x1371f","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10008554},{"operation_name":"upsert","last_dispatch_duration_us":44253,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9292438,"total_server_duration_us":248261,"encode_duration_us":31734,"operation_id":"0x13612","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10004275},{"operation_name":"upsert","last_dispatch_duration_us":260252,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":593737,"total_server_duration_us":48174,"encode_duration_us":24746,"operation_id":"0x13623","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10004223},{"operation_name":"upsert","last_dispatch_duration_us":8107364,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8406740,"total_server_duration_us":65006,"encode_duration_us":18151,"operation_id":"0x13405","timeout_ms":10000,"last_server_duration_us":31776,"total_duration_us":10001774},{"operation_name":"get","last_dispatch_duration_us":6458289,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":6458289,"total_server_duration_us":14367,"operation_id":"0x13678","timeout_ms":30000,"last_server_duration_us":14367,"total_duration_us":6463170},{"operation_name":"upsert","last_dispatch_duration_us":1089236,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2395149,"total_server_duration_us":157841,"encode_duration_us":29403,"operation_id":"0x13bf1","timeout_ms":10000,"last_server_duration_us":71366,"total_duration_us":2411493},{"operation_name":"upsert","last_dispatch_duration_us":1089537,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2171617,"total_server_duration_us":187972,"encode_duration_us":32728,"operation_id":"0x13bf0","timeout_ms":10000,"last_server_duration_us":81828,"total_duration_us":2190935},{"operation_name":"get","last_dispatch_duration_us":1733686,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":1733686,"total_server_duration_us":10213,"operation_id":"0x139aa","timeout_ms":30000,"last_server_duration_us":10213,"total_duration_us":1736707},{"operation_name":"upsert","last_dispatch_duration_us":1497239,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1497239,"total_server_duration_us":75616,"encode_duration_us":22256,"operation_id":"0x13c1e","timeout_ms":10000,"last_server_duration_us":75616,"total_duration_us":1501255},{"operation_name":"upsert","last_dispatch_duration_us":1443141,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1443141,"total_server_duration_us":73684,"encode_duration_us":19199,"operation_id":"0x13c00","timeout_ms":10000,"last_server_duration_us":73684,"total_duration_us":1447144}],"total_count":55}} WARN [2022-03-13 17:19:28,782] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":59561,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1057513,"total_server_duration_us":161658,"encode_duration_us":24060,"operation_id":"0x13520","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10009947},{"operation_name":"upsert","last_dispatch_duration_us":44635,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8723662,"total_server_duration_us":426761,"encode_duration_us":17205,"operation_id":"0x13642","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10009325},{"operation_name":"upsert","last_dispatch_duration_us":155362,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":155362,"total_server_duration_us":22141,"encode_duration_us":20409,"operation_id":"0x136e1","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10003939},{"operation_name":"upsert","last_dispatch_duration_us":49042,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1365916,"total_server_duration_us":183202,"encode_duration_us":27378,"operation_id":"0x134b6","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10002341},{"operation_name":"upsert","last_dispatch_duration_us":7317342,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9286100,"total_server_duration_us":341039,"encode_duration_us":18202,"operation_id":"0x13442","timeout_ms":10000,"last_server_duration_us":24841,"total_duration_us":10001896},{"operation_name":"upsert","last_dispatch_duration_us":50157,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1051340,"total_server_duration_us":160754,"encode_duration_us":18225,"operation_id":"0x1350d","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10001794},{"operation_name":"upsert","last_dispatch_duration_us":9231328,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9231328,"total_server_duration_us":9079863,"encode_duration_us":19694,"operation_id":"0x134a6","timeout_ms":10000,"last_server_duration_us":9079863,"total_duration_us":9236459},{"operation_name":"upsert","last_dispatch_duration_us":9149350,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9149350,"total_server_duration_us":9087309,"encode_duration_us":20667,"operation_id":"0x135b4","timeout_ms":10000,"last_server_duration_us":9087309,"total_duration_us":9158044},{"operation_name":"get","last_dispatch_duration_us":186504,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":186504,"total_server_duration_us":19,"operation_id":"0x13793","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":4843228},{"operation_name":"upsert","last_dispatch_duration_us":551687,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1441466,"total_server_duration_us":161755,"encode_duration_us":31073,"operation_id":"0x13c39","timeout_ms":10000,"last_server_duration_us":58004,"total_duration_us":1517034}],"total_count":32}} WARN [2022-03-13 17:19:28,786] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":19238211,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":19238211,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9056479,"encode_duration_us":17879,"operation_id":"0x12ff0","timeout_ms":10000,"last_server_duration_us":9056479,"total_duration_us":10005283},{"operation_name":"upsert","last_dispatch_duration_us":20373158,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":20373158,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9058604,"encode_duration_us":17652,"operation_id":"0x12df2","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10009940}],"total_count":2}} INFO [2022-03-13 17:19:28,803] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,804] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:19:28,822] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,822] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,823] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:19:28,823] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,872] jepsen worker 80 - jepsen.util 800 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,872] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,872] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,873] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:28,873] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:28,874] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:28,875] jepsen worker 13 - jepsen.util 733 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,875] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:28,875] jepsen worker 8 - jepsen.util 728 :fail :write [2 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,875] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException WARN [2022-03-13 17:19:28,875] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16177992,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17449527,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9299300,"encode_duration_us":23052,"operation_id":"0x13075","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10003425},{"operation_name":"upsert","last_dispatch_duration_us":18123610,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18123610,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9036305,"encode_duration_us":19325,"operation_id":"0x12f91","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10008176}],"total_count":2}} INFO [2022-03-13 17:19:28,876] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,876] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:28,876] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,876] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,876] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,876] jepsen worker 62 - jepsen.util 512 :invoke :write [20 6] INFO [2022-03-13 17:19:28,877] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:28,878] jepsen worker 13 - jepsen.util 733 :invoke :write [4 26] INFO [2022-03-13 17:19:28,878] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:28,880] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:28,880] jepsen worker 44 - jepsen.util 764 :invoke :write [14 17] INFO [2022-03-13 17:19:28,881] jepsen worker 8 - jepsen.util 728 :invoke :write [2 4] INFO [2022-03-13 17:19:28,882] jepsen worker 42 - jepsen.util 402 :invoke :write [14 46] INFO [2022-03-13 17:19:28,887] jepsen worker 76 - jepsen.util 796 :invoke :write [25 6] WARN [2022-03-13 17:19:28,897] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":121892,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":896095,"total_server_duration_us":262214,"encode_duration_us":33603,"operation_id":"0x134e6","timeout_ms":10000,"last_server_duration_us":31489,"total_duration_us":10008788},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":20529,"operation_id":"0x136ce","timeout_ms":10000,"total_duration_us":10008606},{"operation_name":"upsert","last_dispatch_duration_us":9079799,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9778195,"total_server_duration_us":9236271,"encode_duration_us":26018,"operation_id":"0x134da","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10007949},{"operation_name":"upsert","last_dispatch_duration_us":8866571,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9207893,"total_server_duration_us":120410,"encode_duration_us":18273,"operation_id":"0x13598","timeout_ms":10000,"last_server_duration_us":30069,"total_duration_us":10007078},{"operation_name":"upsert","last_dispatch_duration_us":10654942,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":10654942,"total_server_duration_us":19418,"encode_duration_us":17705,"operation_id":"0x13749","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10003780},{"operation_name":"upsert","last_dispatch_duration_us":9123665,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9831571,"total_server_duration_us":9222500,"encode_duration_us":32584,"operation_id":"0x13509","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":9997368},{"operation_name":"upsert","last_dispatch_duration_us":9129460,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9209873,"total_server_duration_us":9074456,"encode_duration_us":22507,"operation_id":"0x134de","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":9238592},{"operation_name":"upsert","last_dispatch_duration_us":9095146,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9095146,"total_server_duration_us":9024634,"encode_duration_us":20885,"operation_id":"0x13567","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":9099573},{"operation_name":"get","last_dispatch_duration_us":8572761,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8572761,"total_server_duration_us":69,"operation_id":"0x13624","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":8575225},{"operation_name":"upsert","last_dispatch_duration_us":1353771,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1353771,"total_server_duration_us":125429,"encode_duration_us":54448,"operation_id":"0x13a58","timeout_ms":10000,"last_server_duration_us":125429,"total_duration_us":2960093}],"total_count":77}} INFO [2022-03-13 17:19:28,905] jepsen worker 37 - jepsen.util 2467 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,905] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,906] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,906] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,907] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,908] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,909] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,909] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,910] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,910] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,915] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,916] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,923] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,924] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,924] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,924] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,925] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 32] INFO [2022-03-13 17:19:28,925] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,928] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:28,929] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,955] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:28,956] jepsen worker nemesis - couchbase.util Rebalance check count: 675 INFO [2022-03-13 17:19:28,956] jepsen worker nemesis - couchbase.util Rebalance stuck count: 547 INFO [2022-03-13 17:19:28,970] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,970] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,971] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,971] jepsen worker 23 - jepsen.util 563 :fail :write [7 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,971] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,971] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,972] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:28,972] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,973] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:28,974] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 15] INFO [2022-03-13 17:19:28,975] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:28,975] jepsen worker 23 - jepsen.util 563 :invoke :write [7 5] INFO [2022-03-13 17:19:28,975] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,976] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:19:28,981] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,982] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:28,982] jepsen worker 69 - jepsen.util 789 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,982] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,984] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:19:28,984] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:19:28,992] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:28,992] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 43] INFO [2022-03-13 17:19:29,011] jepsen worker 24 - jepsen.util 654 :fail :write [8 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,012] jepsen worker 24 - jepsen.util 654 :invoke :write [8 1] INFO [2022-03-13 17:19:29,013] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,013] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,014] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:19:29,014] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,015] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:29,016] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,024] jepsen worker 88 - jepsen.util 2428 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,024] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,025] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,026] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,026] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,027] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,027] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 6] INFO [2022-03-13 17:19:29,028] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,031] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,031] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,032] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,032] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,033] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,034] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:19:29,035] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,036] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:19:29,041] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,041] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 47] INFO [2022-03-13 17:19:29,077] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,078] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 20] INFO [2022-03-13 17:19:29,101] jepsen worker 71 - jepsen.util 611 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,102] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:29,103] jepsen worker 12 - jepsen.util 912 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,104] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,104] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,104] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,105] jepsen worker 70 - jepsen.util 880 :invoke :write [23 9] INFO [2022-03-13 17:19:29,105] jepsen worker 6 - jepsen.util 636 :invoke :write [2 46] INFO [2022-03-13 17:19:29,106] jepsen worker 12 - jepsen.util 912 :invoke :write [4 3] INFO [2022-03-13 17:19:29,107] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:29,125] jepsen worker 21 - jepsen.util 831 :fail :write [7 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,125] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,127] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:29,127] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:29,129] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,129] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:19:29,131] jepsen worker 75 - jepsen.util 525 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,132] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,132] jepsen worker 26 - jepsen.util 566 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,133] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:29,133] jepsen worker 75 - jepsen.util 525 :invoke :write [25 32] INFO [2022-03-13 17:19:29,134] jepsen worker 26 - jepsen.util 566 :invoke :write [8 21] INFO [2022-03-13 17:19:29,178] jepsen worker 38 - jepsen.util 2828 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,179] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,193] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,193] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,194] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:29,194] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:19:29,223] jepsen worker 84 - jepsen.util 5034 :ok :read [28 44] INFO [2022-03-13 17:19:29,223] jepsen worker 84 - jepsen.util 5034 :invoke :write [28 42] INFO [2022-03-13 17:19:29,226] jepsen worker 2 - jepsen.util 5042 :ok :read [0 41] INFO [2022-03-13 17:19:29,226] jepsen worker 2 - jepsen.util 5042 :invoke :write [0 39] INFO [2022-03-13 17:19:29,244] jepsen worker 89 - jepsen.util 2789 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,244] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,246] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,246] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:19:29,247] jepsen worker 37 - jepsen.util 2467 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,247] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,247] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,248] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:19:29,262] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,263] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:19:29,303] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,304] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,304] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:19:29,304] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:29,311] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,312] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:19:29,318] jepsen worker 61 - jepsen.util 781 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,319] jepsen worker 61 - jepsen.util 781 :invoke :write [20 39] INFO [2022-03-13 17:19:29,347] jepsen worker 88 - jepsen.util 2428 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,347] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,348] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 23] INFO [2022-03-13 17:19:29,348] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,354] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,355] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,385] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,386] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:19:29,410] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,410] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,411] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:19:29,412] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:19:29,431] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,431] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,431] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,431] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,432] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:19:29,432] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:19:29,460] jepsen worker 37 - jepsen.util 2467 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,461] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:19:29,461] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,461] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,463] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,463] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 38] INFO [2022-03-13 17:19:29,482] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,482] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:29,508] jepsen worker 35 - jepsen.util 935 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,508] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,508] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:19:29,509] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:29,534] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,534] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,535] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:19:29,536] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,537] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,538] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:19:29,567] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,568] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,581] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,582] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:19:29,597] jepsen worker 53 - jepsen.util 593 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,597] jepsen worker 13 - jepsen.util 733 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,597] jepsen worker 13 - jepsen.util 733 :invoke :write [4 29] INFO [2022-03-13 17:19:29,598] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:29,608] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,608] jepsen worker 88 - jepsen.util 2428 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,609] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,609] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,617] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,618] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:19:29,651] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,651] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,651] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,652] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,652] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,653] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,653] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 12] INFO [2022-03-13 17:19:29,653] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,653] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:29,653] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,654] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:19:29,654] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 45] INFO [2022-03-13 17:19:29,671] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,671] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,671] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:29,672] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:29,736] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,736] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,736] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,737] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,738] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 14] INFO [2022-03-13 17:19:29,738] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:19:29,746] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,747] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:19:29,749] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,750] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:29,809] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,810] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,811] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,811] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,812] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:19:29,812] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,813] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,814] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 45] INFO [2022-03-13 17:19:29,823] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,824] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:19:29,839] jepsen worker 4 - jepsen.util 2614 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,840] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:29,890] jepsen worker 88 - jepsen.util 2428 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,890] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:29,898] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,899] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:19:29,953] jepsen worker 89 - jepsen.util 2789 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,954] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:19:29,961] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:29,961] jepsen worker nemesis - couchbase.util Rebalance check count: 676 INFO [2022-03-13 17:19:29,961] jepsen worker nemesis - couchbase.util Rebalance stuck count: 548 INFO [2022-03-13 17:19:29,967] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:29,968] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:19:30,010] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,011] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,012] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,013] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:19:30,050] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,051] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 14] INFO [2022-03-13 17:19:30,092] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,092] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,093] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:30,093] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,111] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,112] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,128] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,128] jepsen worker 37 - jepsen.util 2467 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,128] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 28] INFO [2022-03-13 17:19:30,131] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 37] INFO [2022-03-13 17:19:30,185] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,185] jepsen worker 38 - jepsen.util 2828 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,186] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:19:30,187] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:19:30,202] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,202] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,203] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,203] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,223] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,223] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:19:30,282] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,282] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,282] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,282] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,283] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:19:30,284] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:30,287] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,287] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,288] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:30,288] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,290] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,290] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,291] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,292] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:19:30,298] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,299] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:19:30,335] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,335] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,404] jepsen worker 88 - jepsen.util 2428 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,404] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,404] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,405] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,410] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,411] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:19:30,460] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,461] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:19:30,481] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,482] jepsen worker 62 - jepsen.util 512 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,483] jepsen worker 71 - jepsen.util 611 :invoke :write [23 22] INFO [2022-03-13 17:19:30,483] jepsen worker 62 - jepsen.util 512 :invoke :write [20 38] INFO [2022-03-13 17:19:30,487] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,488] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,488] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,489] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:30,489] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:19:30,490] jepsen worker 80 - jepsen.util 800 :invoke :write [26 43] INFO [2022-03-13 17:19:30,491] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,493] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,495] jepsen worker 51 - jepsen.util 771 :invoke :write [17 0] INFO [2022-03-13 17:19:30,495] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 47] INFO [2022-03-13 17:19:30,508] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,508] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,512] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,513] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,519] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,520] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 19] INFO [2022-03-13 17:19:30,558] jepsen worker 22 - jepsen.util 652 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,559] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:19:30,571] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,571] jepsen worker 37 - jepsen.util 2467 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,572] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 25] INFO [2022-03-13 17:19:30,572] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 20] INFO [2022-03-13 17:19:30,573] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,573] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,576] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,577] jepsen worker 69 - jepsen.util 789 :invoke :write [23 36] INFO [2022-03-13 17:19:30,594] jepsen worker 78 - jepsen.util 708 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,594] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,595] jepsen worker 24 - jepsen.util 654 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,596] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,596] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,596] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,597] jepsen worker 77 - jepsen.util 707 :invoke :write [25 11] INFO [2022-03-13 17:19:30,598] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:30,600] jepsen worker 33 - jepsen.util 753 :invoke :write [11 49] INFO [2022-03-13 17:19:30,604] jepsen worker 78 - jepsen.util 708 :invoke :write [26 13] INFO [2022-03-13 17:19:30,606] jepsen worker 14 - jepsen.util 374 :invoke :write [4 49] INFO [2022-03-13 17:19:30,607] jepsen worker 24 - jepsen.util 654 :invoke :write [8 21] INFO [2022-03-13 17:19:30,612] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,613] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:19:30,627] jepsen worker 76 - jepsen.util 796 :fail :write [25 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,628] jepsen worker 76 - jepsen.util 796 :invoke :write [25 39] INFO [2022-03-13 17:19:30,671] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,672] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:30,673] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,673] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,674] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,674] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:19:30,675] jepsen worker 71 - jepsen.util 611 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,675] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:30,679] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:30,680] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:30,681] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,681] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 45] INFO [2022-03-13 17:19:30,715] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,715] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 14] INFO [2022-03-13 17:19:30,718] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,718] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,718] jepsen worker 34 - jepsen.util 934 :invoke :write [11 25] INFO [2022-03-13 17:19:30,719] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:30,728] jepsen worker 42 - jepsen.util 402 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,728] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:30,734] jepsen worker 43 - jepsen.util 673 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,735] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:30,750] jepsen worker 5 - jepsen.util 2705 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,751] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,752] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,753] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:19:30,766] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,766] jepsen worker 69 - jepsen.util 789 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,766] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:30,767] jepsen worker 71 - jepsen.util 611 :invoke :write [23 22] INFO [2022-03-13 17:19:30,771] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,772] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:30,773] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,773] jepsen worker 69 - jepsen.util 789 :invoke :write [23 48] INFO [2022-03-13 17:19:30,783] jepsen worker 3 - jepsen.util 2253 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,783] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,790] jepsen worker 38 - jepsen.util 2828 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,791] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 7] INFO [2022-03-13 17:19:30,798] jepsen worker 75 - jepsen.util 525 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,799] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:30,824] jepsen worker 18 - jepsen.util 4608 :info :write [6 44] :RequestTimeoutException INFO [2022-03-13 17:19:30,825] jepsen worker 18 - jepsen.util 4698 :invoke :write [6 19] INFO [2022-03-13 17:19:30,831] jepsen worker 88 - jepsen.util 2428 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,832] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 2] INFO [2022-03-13 17:19:30,863] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,864] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:30,871] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,872] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,903] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,904] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,910] jepsen worker 23 - jepsen.util 563 :fail :write [7 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,911] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:19:30,914] jepsen worker 80 - jepsen.util 800 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,915] jepsen worker 80 - jepsen.util 800 :invoke :write [26 38] INFO [2022-03-13 17:19:30,916] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,917] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:19:30,944] jepsen worker 47 - jepsen.util 4457 :info :write [15 17] :SyncWriteAmbiguous INFO [2022-03-13 17:19:30,945] jepsen worker 47 - jepsen.util 4547 :invoke :write [15 3] INFO [2022-03-13 17:19:30,953] jepsen worker 71 - jepsen.util 611 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,954] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:30,956] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,956] jepsen worker 38 - jepsen.util 2828 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,957] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:30,957] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 1] INFO [2022-03-13 17:19:30,958] jepsen worker 4 - jepsen.util 2614 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,958] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,959] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,959] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,960] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,960] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:30,961] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,963] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 40] INFO [2022-03-13 17:19:30,964] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,965] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 33] INFO [2022-03-13 17:19:30,965] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,966] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,968] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,968] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,970] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,970] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:30,970] jepsen worker nemesis - couchbase.util Rebalance check count: 677 INFO [2022-03-13 17:19:30,970] jepsen worker nemesis - couchbase.util Rebalance stuck count: 549 INFO [2022-03-13 17:19:30,971] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,972] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,973] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,974] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,975] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,976] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,977] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:30,979] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,980] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:19:30,981] jepsen worker 8 - jepsen.util 728 :fail :write [2 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:30,982] jepsen worker 8 - jepsen.util 728 :invoke :write [2 34] INFO [2022-03-13 17:19:31,007] jepsen worker 86 - jepsen.util 4496 :info :write [28 27] :RequestTimeoutException INFO [2022-03-13 17:19:31,007] jepsen worker 86 - jepsen.util 4586 :invoke :read [28 nil] INFO [2022-03-13 17:19:31,011] jepsen worker 37 - jepsen.util 2467 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,012] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 27] INFO [2022-03-13 17:19:31,021] jepsen worker 38 - jepsen.util 2828 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,021] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,074] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,074] jepsen worker 89 - jepsen.util 2789 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,075] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,075] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,076] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,076] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,077] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,077] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,077] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,078] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:31,078] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,079] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,080] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,080] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,081] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,082] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:19:31,083] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,083] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:19:31,118] jepsen worker 1 - jepsen.util 4321 :info :write [0 39] :RequestTimeoutException INFO [2022-03-13 17:19:31,119] jepsen worker 1 - jepsen.util 4411 :invoke :read [0 nil] INFO [2022-03-13 17:19:31,128] jepsen worker 68 - jepsen.util 4748 :info :write [22 12] :RequestTimeoutException INFO [2022-03-13 17:19:31,128] jepsen worker 68 - jepsen.util 4838 :invoke :read [22 nil] INFO [2022-03-13 17:19:31,129] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,129] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,141] jepsen worker 27 - jepsen.util 4797 :info :write [9 7] :SyncWriteAmbiguous INFO [2022-03-13 17:19:31,141] jepsen worker 27 - jepsen.util 4887 :invoke :read [9 nil] INFO [2022-03-13 17:19:31,146] jepsen worker 1 - jepsen.util 4411 :ok :read [0 41] INFO [2022-03-13 17:19:31,147] jepsen worker 1 - jepsen.util 4411 :invoke :read [0 nil] INFO [2022-03-13 17:19:31,172] jepsen worker 1 - jepsen.util 4411 :ok :read [0 41] INFO [2022-03-13 17:19:31,172] jepsen worker 1 - jepsen.util 4411 :invoke :read [0 nil] INFO [2022-03-13 17:19:31,191] jepsen worker 88 - jepsen.util 2428 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,191] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,191] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 35] INFO [2022-03-13 17:19:31,192] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 17] INFO [2022-03-13 17:19:31,196] jepsen worker 4 - jepsen.util 2614 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,196] jepsen worker 1 - jepsen.util 4411 :ok :read [0 41] INFO [2022-03-13 17:19:31,197] jepsen worker 1 - jepsen.util 4411 :invoke :read [0 nil] INFO [2022-03-13 17:19:31,197] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:19:31,223] jepsen worker 1 - jepsen.util 4411 :ok :read [0 41] INFO [2022-03-13 17:19:31,224] jepsen worker 1 - jepsen.util 4411 :invoke :read [0 nil] INFO [2022-03-13 17:19:31,236] jepsen worker 73 - jepsen.util 4933 :info :write [24 43] :SyncWriteAmbiguous INFO [2022-03-13 17:19:31,236] jepsen worker 9 - jepsen.util 4239 :info :write [3 23] :SyncWriteAmbiguous INFO [2022-03-13 17:19:31,237] jepsen worker 9 - jepsen.util 4329 :invoke :read [3 nil] INFO [2022-03-13 17:19:31,238] jepsen worker 73 - jepsen.util 5023 :invoke :read [24 nil] INFO [2022-03-13 17:19:31,245] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,245] jepsen worker 39 - jepsen.util 4989 :info :write [13 25] :RequestTimeoutException INFO [2022-03-13 17:19:31,246] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,247] jepsen worker 39 - jepsen.util 5079 :invoke :write [13 49] INFO [2022-03-13 17:19:31,252] jepsen worker 1 - jepsen.util 4411 :ok :read [0 41] INFO [2022-03-13 17:19:31,253] jepsen worker 1 - jepsen.util 4411 :invoke :read [0 nil] INFO [2022-03-13 17:19:31,266] jepsen worker 66 - jepsen.util 4386 :info :write [22 8] :RequestTimeoutException INFO [2022-03-13 17:19:31,267] jepsen worker 66 - jepsen.util 4476 :invoke :write [22 21] INFO [2022-03-13 17:19:31,270] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,271] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:19:31,274] jepsen worker 1 - jepsen.util 4411 :ok :read [0 41] INFO [2022-03-13 17:19:31,276] jepsen worker 1 - jepsen.util 4411 :invoke :write [0 23] INFO [2022-03-13 17:19:31,304] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,304] jepsen worker 38 - jepsen.util 2828 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,305] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,305] jepsen worker 69 - jepsen.util 789 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,305] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,305] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,306] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,307] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:31,308] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:19:31,308] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:19:31,309] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,310] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,310] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,311] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,311] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,312] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,312] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:19:31,313] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,314] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,315] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,315] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,316] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,316] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,316] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,316] jepsen worker 65 - jepsen.util 4835 :info :write [21 23] :RequestTimeoutException INFO [2022-03-13 17:19:31,317] jepsen worker 56 - jepsen.util 4556 :info :write [18 3] :RequestTimeoutException INFO [2022-03-13 17:19:31,317] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:19:31,317] jepsen worker 56 - jepsen.util 4646 :invoke :read [18 nil] INFO [2022-03-13 17:19:31,318] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:19:31,318] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,319] jepsen worker 65 - jepsen.util 4925 :invoke :write [21 39] INFO [2022-03-13 17:19:31,320] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,324] jepsen worker 60 - jepsen.util 690 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,325] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:31,360] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,360] jepsen worker 51 - jepsen.util 771 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,361] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:31,361] jepsen worker 51 - jepsen.util 771 :invoke :write [17 16] INFO [2022-03-13 17:19:31,379] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,380] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:19:31,381] jepsen worker 79 - jepsen.util 1159 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,381] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:31,398] jepsen worker 28 - jepsen.util 4708 :info :write [9 43] :RequestTimeoutException INFO [2022-03-13 17:19:31,399] jepsen worker 28 - jepsen.util 4798 :invoke :write [9 23] INFO [2022-03-13 17:19:31,411] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,411] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,411] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,412] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:31,412] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,412] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,415] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,415] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,416] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:19:31,417] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,419] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,420] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:31,432] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,433] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,454] jepsen worker 37 - jepsen.util 2467 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,455] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 29] INFO [2022-03-13 17:19:31,468] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,468] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:31,483] jepsen worker 13 - jepsen.util 733 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,483] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,483] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:19:31,484] jepsen worker 52 - jepsen.util 682 :invoke :write [17 2] INFO [2022-03-13 17:19:31,498] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,498] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,499] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,499] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,500] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,500] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,501] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:19:31,501] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:19:31,516] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,517] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:19:31,535] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,535] jepsen worker 33 - jepsen.util 753 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,535] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:31,536] jepsen worker 33 - jepsen.util 753 :invoke :write [11 6] INFO [2022-03-13 17:19:31,553] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,553] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,554] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:31,555] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,555] jepsen worker 71 - jepsen.util 611 :invoke :write [23 5] INFO [2022-03-13 17:19:31,556] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:19:31,576] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,577] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:19:31,586] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,586] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,620] jepsen worker 55 - jepsen.util 4195 :info :write [18 13] :SyncWriteAmbiguous INFO [2022-03-13 17:19:31,620] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,621] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,621] jepsen worker 55 - jepsen.util 4285 :invoke :write [18 21] INFO [2022-03-13 17:19:31,627] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,627] jepsen worker 44 - jepsen.util 764 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,629] jepsen worker 7 - jepsen.util 637 :invoke :write [2 28] INFO [2022-03-13 17:19:31,630] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:31,685] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,686] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,691] jepsen worker 61 - jepsen.util 781 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,691] jepsen worker 61 - jepsen.util 781 :invoke :write [20 47] INFO [2022-03-13 17:19:31,709] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,709] jepsen worker 88 - jepsen.util 2428 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,710] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:31,710] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 35] INFO [2022-03-13 17:19:31,714] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,714] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,715] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 29] INFO [2022-03-13 17:19:31,715] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,717] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,718] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:19:31,785] jepsen worker 37 - jepsen.util 2467 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,786] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,788] jepsen worker 26 - jepsen.util 566 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,788] jepsen worker 76 - jepsen.util 796 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,788] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,788] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,789] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:31,790] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:31,790] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:31,791] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:31,791] jepsen worker 14 - jepsen.util 374 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,792] jepsen worker 14 - jepsen.util 374 :invoke :write [4 22] INFO [2022-03-13 17:19:31,800] jepsen worker 71 - jepsen.util 611 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,801] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:19:31,812] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,813] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,814] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,815] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,815] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,815] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,816] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,816] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,817] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:19:31,817] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,817] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,818] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,818] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,819] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,820] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,821] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:19:31,821] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,822] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 29] INFO [2022-03-13 17:19:31,836] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,836] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,836] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:31,837] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:31,838] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:31,838] jepsen worker 22 - jepsen.util 652 :invoke :write [7 41] INFO [2022-03-13 17:19:31,854] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,855] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,857] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,857] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:19:31,881] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,882] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:19:31,909] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,910] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,915] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,916] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,954] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,954] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:31,964] jepsen worker 24 - jepsen.util 654 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,964] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:31,966] jepsen worker 6 - jepsen.util 636 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,967] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:31,968] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,968] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,968] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,969] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:31,969] jepsen worker 38 - jepsen.util 2828 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,970] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:31,970] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:31,970] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,971] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 37] INFO [2022-03-13 17:19:31,972] jepsen worker 76 - jepsen.util 796 :invoke :write [25 21] INFO [2022-03-13 17:19:31,974] jepsen worker 46 - jepsen.util 4816 :info :write [15 31] :RequestTimeoutException INFO [2022-03-13 17:19:31,975] jepsen worker 46 - jepsen.util 4906 :invoke :write [15 22] INFO [2022-03-13 17:19:31,976] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:31,976] jepsen worker nemesis - couchbase.util Rebalance check count: 678 INFO [2022-03-13 17:19:31,976] jepsen worker nemesis - couchbase.util Rebalance stuck count: 550 INFO [2022-03-13 17:19:31,979] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,979] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,979] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:31,980] jepsen worker 13 - jepsen.util 733 :invoke :write [4 2] INFO [2022-03-13 17:19:31,982] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,984] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,984] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,984] jepsen worker 88 - jepsen.util 2428 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,987] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 27] INFO [2022-03-13 17:19:31,988] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:31,989] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,989] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:31,990] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,991] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,992] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:19:31,992] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:31,993] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:31,995] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 9] INFO [2022-03-13 17:19:32,019] jepsen worker 73 - jepsen.util 5023 :ok :read [24 24] INFO [2022-03-13 17:19:32,020] jepsen worker 73 - jepsen.util 5023 :invoke :write [24 8] INFO [2022-03-13 17:19:32,044] jepsen worker 3 - jepsen.util 2253 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,045] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:19:32,071] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,072] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:32,073] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,074] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:19:32,094] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,095] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:32,111] jepsen worker 34 - jepsen.util 934 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,112] jepsen worker 34 - jepsen.util 934 :invoke :write [11 43] INFO [2022-03-13 17:19:32,114] jepsen worker 77 - jepsen.util 707 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,115] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,120] jepsen worker 52 - jepsen.util 682 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,120] jepsen worker 52 - jepsen.util 682 :invoke :write [17 34] INFO [2022-03-13 17:19:32,121] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,121] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,121] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,121] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:32,122] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:32,122] jepsen worker 53 - jepsen.util 593 :invoke :write [17 39] INFO [2022-03-13 17:19:32,123] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,123] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:32,123] jepsen worker 69 - jepsen.util 789 :invoke :write [23 14] INFO [2022-03-13 17:19:32,125] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:32,128] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,129] jepsen worker 75 - jepsen.util 525 :invoke :write [25 24] INFO [2022-03-13 17:19:32,132] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,132] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,133] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,133] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:32,137] jepsen worker 10 - jepsen.util 4870 :info :write [3 1] :RequestTimeoutException INFO [2022-03-13 17:19:32,137] jepsen worker 10 - jepsen.util 4960 :invoke :write [3 2] INFO [2022-03-13 17:19:32,154] jepsen worker 38 - jepsen.util 2828 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,154] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,155] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:19:32,155] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 16] INFO [2022-03-13 17:19:32,174] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,175] jepsen worker 37 - jepsen.util 2467 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,175] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,175] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 19] INFO [2022-03-13 17:19:32,187] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,187] jepsen worker 22 - jepsen.util 652 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,189] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:32,189] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,192] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,192] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,192] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,193] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,194] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:32,194] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:32,202] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,203] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:32,226] jepsen worker 20 - jepsen.util 4880 :info :write [6 32] :RequestTimeoutException INFO [2022-03-13 17:19:32,227] jepsen worker 20 - jepsen.util 4970 :invoke :write [6 29] INFO [2022-03-13 17:19:32,234] jepsen worker 38 - jepsen.util 2828 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,235] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:19:32,241] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,241] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,241] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,241] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,242] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:19:32,242] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:32,244] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,245] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,245] jepsen worker 76 - jepsen.util 796 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,245] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:32,245] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:32,246] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,246] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,247] jepsen worker 43 - jepsen.util 673 :invoke :write [14 15] INFO [2022-03-13 17:19:32,247] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,247] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,248] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,251] jepsen worker 60 - jepsen.util 690 :invoke :write [20 22] INFO [2022-03-13 17:19:32,255] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:32,255] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,257] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,257] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,258] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,258] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:32,259] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,260] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,260] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:32,261] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:19:32,271] jepsen worker 70 - jepsen.util 880 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,271] jepsen worker 51 - jepsen.util 771 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,272] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,272] jepsen worker 70 - jepsen.util 880 :invoke :write [23 35] INFO [2022-03-13 17:19:32,275] jepsen worker 51 - jepsen.util 771 :invoke :write [17 15] INFO [2022-03-13 17:19:32,278] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,279] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:19:32,286] jepsen worker 74 - jepsen.util 4484 :info :write [24 27] :RequestTimeoutException INFO [2022-03-13 17:19:32,286] jepsen worker 35 - jepsen.util 935 :invoke :write [11 21] INFO [2022-03-13 17:19:32,290] jepsen worker 74 - jepsen.util 4574 :invoke :write [24 3] INFO [2022-03-13 17:19:32,291] jepsen worker 88 - jepsen.util 2428 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,292] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,293] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:32,293] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:32,365] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,366] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,366] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:32,367] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:19:32,378] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,378] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,379] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 12] INFO [2022-03-13 17:19:32,381] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:32,425] jepsen worker 37 - jepsen.util 2467 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,425] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,425] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:19:32,426] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 11] INFO [2022-03-13 17:19:32,436] jepsen worker 52 - jepsen.util 682 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,436] jepsen worker 13 - jepsen.util 733 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,436] jepsen worker 33 - jepsen.util 753 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,436] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:32,437] jepsen worker 33 - jepsen.util 753 :invoke :write [11 14] INFO [2022-03-13 17:19:32,438] jepsen worker 52 - jepsen.util 682 :invoke :write [17 16] INFO [2022-03-13 17:19:32,443] jepsen worker 78 - jepsen.util 708 :fail :write [26 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,444] jepsen worker 78 - jepsen.util 708 :invoke :write [26 38] INFO [2022-03-13 17:19:32,453] jepsen worker 69 - jepsen.util 789 :fail :write [23 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,454] jepsen worker 69 - jepsen.util 789 :invoke :write [23 42] INFO [2022-03-13 17:19:32,471] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,472] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,472] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,473] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,473] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,474] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,476] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,477] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:32,497] jepsen worker 89 - jepsen.util 2789 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,498] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:32,505] jepsen worker 62 - jepsen.util 512 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,506] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:32,510] jepsen worker 14 - jepsen.util 374 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,510] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,511] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:32,511] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:32,540] jepsen worker 85 - jepsen.util 4315 :info :write [28 48] :RequestTimeoutException INFO [2022-03-13 17:19:32,541] jepsen worker 85 - jepsen.util 4405 :invoke :write [28 37] INFO [2022-03-13 17:19:32,548] jepsen worker 83 - jepsen.util 4313 :info :write [27 20] :RequestTimeoutException INFO [2022-03-13 17:19:32,549] jepsen worker 83 - jepsen.util 4403 :invoke :read [27 nil] INFO [2022-03-13 17:19:32,567] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,567] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,567] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,568] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:32,568] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:19:32,569] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:19:32,584] jepsen worker 32 - jepsen.util 4892 :info :write [10 17] :RequestTimeoutException INFO [2022-03-13 17:19:32,584] jepsen worker 32 - jepsen.util 4982 :invoke :read [10 nil] INFO [2022-03-13 17:19:32,587] jepsen worker 32 - jepsen.util 4982 :ok :read [10 :nil] INFO [2022-03-13 17:19:32,587] jepsen worker 32 - jepsen.util 4982 :invoke :read [10 nil] INFO [2022-03-13 17:19:32,589] jepsen worker 32 - jepsen.util 4982 :ok :read [10 :nil] INFO [2022-03-13 17:19:32,590] jepsen worker 32 - jepsen.util 4982 :invoke :write [10 42] INFO [2022-03-13 17:19:32,609] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,610] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:19:32,616] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,616] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,617] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,617] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:32,618] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:19:32,619] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,621] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,621] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,622] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 19] INFO [2022-03-13 17:19:32,622] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,623] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,624] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 40] INFO [2022-03-13 17:19:32,642] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,643] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:19:32,658] jepsen worker 40 - jepsen.util 4900 :ok :read [13 39] INFO [2022-03-13 17:19:32,659] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,659] jepsen worker 40 - jepsen.util 4900 :invoke :write [13 0] INFO [2022-03-13 17:19:32,660] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:32,695] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,695] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,695] jepsen worker 23 - jepsen.util 563 :invoke :write [7 33] INFO [2022-03-13 17:19:32,699] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 30] INFO [2022-03-13 17:19:32,700] jepsen worker 58 - jepsen.util 4198 :ok :read [19 34] INFO [2022-03-13 17:19:32,701] jepsen worker 58 - jepsen.util 4198 :invoke :write [19 36] INFO [2022-03-13 17:19:32,703] jepsen worker 80 - jepsen.util 800 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,704] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:32,706] jepsen worker 53 - jepsen.util 593 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,706] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,706] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,706] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,706] jepsen worker 75 - jepsen.util 525 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,707] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,708] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,709] jepsen worker 26 - jepsen.util 566 :invoke :write [8 45] INFO [2022-03-13 17:19:32,710] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:32,711] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,711] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:32,712] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,712] jepsen worker 37 - jepsen.util 2467 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,713] jepsen worker 13 - jepsen.util 733 :invoke :write [4 40] INFO [2022-03-13 17:19:32,714] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,714] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,714] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,714] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,715] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 23] INFO [2022-03-13 17:19:32,715] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,717] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:19:32,717] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,718] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,718] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 8] INFO [2022-03-13 17:19:32,724] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,725] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,725] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,726] jepsen worker 76 - jepsen.util 796 :invoke :write [25 0] INFO [2022-03-13 17:19:32,731] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,731] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,738] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,738] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:32,741] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,742] jepsen worker 75 - jepsen.util 525 :invoke :write [25 9] INFO [2022-03-13 17:19:32,744] jepsen worker 36 - jepsen.util 2466 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,746] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 24] INFO [2022-03-13 17:19:32,758] jepsen worker 69 - jepsen.util 789 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,758] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:19:32,813] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,813] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,813] jepsen worker 34 - jepsen.util 934 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,814] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:32,814] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:32,815] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:19:32,816] jepsen worker 16 - jepsen.util 4516 :info :write [5 45] :RequestTimeoutException INFO [2022-03-13 17:19:32,817] jepsen worker 16 - jepsen.util 4606 :invoke :read [5 nil] INFO [2022-03-13 17:19:32,822] jepsen worker 43 - jepsen.util 673 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,822] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:32,825] jepsen worker 8 - jepsen.util 728 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,826] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:32,826] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:32,826] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:32,831] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,832] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,847] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,848] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,853] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,853] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:19:32,858] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,858] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 41] INFO [2022-03-13 17:19:32,896] jepsen worker 3 - jepsen.util 2253 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,896] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,938] jepsen worker 88 - jepsen.util 2428 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,939] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:32,948] jepsen worker 36 - jepsen.util 2466 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,948] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,948] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,949] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:32,949] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,950] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:19:32,957] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,957] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:32,961] jepsen worker 89 - jepsen.util 2789 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,961] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:32,962] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,962] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:19:32,967] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,968] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:32,983] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:32,983] jepsen worker nemesis - couchbase.util Rebalance check count: 679 INFO [2022-03-13 17:19:32,983] jepsen worker nemesis - couchbase.util Rebalance stuck count: 551 INFO [2022-03-13 17:19:32,994] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,995] jepsen worker 51 - jepsen.util 771 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:32,995] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:19:32,996] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:33,056] jepsen worker 38 - jepsen.util 2828 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,056] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,056] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,057] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,058] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,058] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,058] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 4] INFO [2022-03-13 17:19:33,059] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:19:33,062] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,062] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 46] INFO [2022-03-13 17:19:33,081] jepsen worker 72 - jepsen.util 4752 :info :write [24 46] :RequestTimeoutException INFO [2022-03-13 17:19:33,081] jepsen worker 72 - jepsen.util 4842 :invoke :write [24 6] INFO [2022-03-13 17:19:33,096] jepsen worker 37 - jepsen.util 2467 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,096] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,097] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,098] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,108] jepsen worker 61 - jepsen.util 781 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,108] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,109] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:33,109] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:33,116] jepsen worker 13 - jepsen.util 733 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,116] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:33,126] jepsen worker 59 - jepsen.util 4739 :ok :read [19 34] INFO [2022-03-13 17:19:33,126] jepsen worker 59 - jepsen.util 4739 :invoke :read [19 nil] INFO [2022-03-13 17:19:33,133] jepsen worker 17 - jepsen.util 4877 :ok :read [5 43] INFO [2022-03-13 17:19:33,133] jepsen worker 17 - jepsen.util 4877 :invoke :read [5 nil] INFO [2022-03-13 17:19:33,134] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,135] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:19:33,146] jepsen worker 45 - jepsen.util 4905 :info :write [15 31] :RequestTimeoutException INFO [2022-03-13 17:19:33,146] jepsen worker 45 - jepsen.util 4995 :invoke :write [15 29] INFO [2022-03-13 17:19:33,149] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,149] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,150] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,150] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,151] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,152] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,152] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,152] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,153] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,153] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,155] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,155] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,155] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,156] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 19] INFO [2022-03-13 17:19:33,157] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,157] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,159] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,159] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,160] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,160] jepsen worker 15 - jepsen.util 4515 :ok :read [5 43] INFO [2022-03-13 17:19:33,161] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,162] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,163] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,163] jepsen worker 15 - jepsen.util 4515 :invoke :read [5 nil] INFO [2022-03-13 17:19:33,164] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 13] INFO [2022-03-13 17:19:33,164] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,173] jepsen worker 35 - jepsen.util 935 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,173] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:33,174] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,175] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,175] jepsen worker 38 - jepsen.util 2828 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,176] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 24] INFO [2022-03-13 17:19:33,192] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,192] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,193] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:33,194] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:19:33,198] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,199] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:33,200] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,201] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:33,202] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,203] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:33,204] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,204] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:19:33,209] jepsen worker 59 - jepsen.util 4739 :ok :read [19 34] INFO [2022-03-13 17:19:33,209] jepsen worker 59 - jepsen.util 4739 :invoke :write [19 36] INFO [2022-03-13 17:19:33,217] jepsen worker 17 - jepsen.util 4877 :ok :read [5 43] INFO [2022-03-13 17:19:33,218] jepsen worker 17 - jepsen.util 4877 :invoke :write [5 37] INFO [2022-03-13 17:19:33,220] jepsen worker 60 - jepsen.util 690 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,221] jepsen worker 60 - jepsen.util 690 :invoke :write [20 18] INFO [2022-03-13 17:19:33,286] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,284] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,284] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,284] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,287] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,284] jepsen worker 12 - jepsen.util 912 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,284] jepsen worker 19 - jepsen.util 4519 :info :write [6 41] :RequestTimeoutException INFO [2022-03-13 17:19:33,284] jepsen worker 64 - jepsen.util 4564 :info :write [21 7] :RequestTimeoutException INFO [2022-03-13 17:19:33,284] jepsen worker 33 - jepsen.util 753 :fail :write [11 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,285] jepsen worker 15 - jepsen.util 4515 :ok :read [5 43] INFO [2022-03-13 17:19:33,285] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,288] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:33,285] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,285] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,285] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,288] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:19:33,289] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:33,290] jepsen worker 15 - jepsen.util 4515 :invoke :read [5 nil] INFO [2022-03-13 17:19:33,285] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,290] jepsen worker 12 - jepsen.util 912 :invoke :write [4 43] INFO [2022-03-13 17:19:33,292] jepsen worker 19 - jepsen.util 4609 :invoke :read [6 nil] INFO [2022-03-13 17:19:33,292] jepsen worker 64 - jepsen.util 4654 :invoke :read [21 nil] INFO [2022-03-13 17:19:33,293] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:19:33,294] jepsen worker 34 - jepsen.util 934 :invoke :write [11 42] INFO [2022-03-13 17:19:33,295] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:19:33,298] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,298] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 16] INFO [2022-03-13 17:19:33,301] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,336] jepsen worker 76 - jepsen.util 796 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,338] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:33,345] jepsen worker 4 - jepsen.util 2614 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,346] jepsen worker 38 - jepsen.util 2828 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,346] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 46] INFO [2022-03-13 17:19:33,346] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,355] jepsen worker 15 - jepsen.util 4515 :ok :read [5 43] INFO [2022-03-13 17:19:33,355] jepsen worker 15 - jepsen.util 4515 :invoke :write [5 9] INFO [2022-03-13 17:19:33,361] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,361] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,361] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,362] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,363] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,363] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,363] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:19:33,364] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,365] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,365] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,366] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,367] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 48] INFO [2022-03-13 17:19:33,404] jepsen worker 37 - jepsen.util 2467 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,404] jepsen worker 5 - jepsen.util 2705 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,405] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,406] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 36] INFO [2022-03-13 17:19:33,410] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,410] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,410] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,411] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,412] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,412] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,412] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:19:33,413] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:33,428] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,428] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,428] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,428] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,429] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,429] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,429] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,429] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,431] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:33,431] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,431] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:33,432] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:33,433] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:33,433] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:33,435] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,436] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:33,437] jepsen worker 78 - jepsen.util 708 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,437] jepsen worker 78 - jepsen.util 708 :invoke :write [26 32] INFO [2022-03-13 17:19:33,459] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,460] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,476] jepsen worker 34 - jepsen.util 934 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,476] jepsen worker 34 - jepsen.util 934 :invoke :write [11 6] INFO [2022-03-13 17:19:33,507] jepsen worker 38 - jepsen.util 2828 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,507] jepsen worker 88 - jepsen.util 2428 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,507] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,507] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 11] INFO [2022-03-13 17:19:33,508] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,508] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 3] INFO [2022-03-13 17:19:33,509] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,509] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:19:33,519] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,519] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,519] jepsen worker 4 - jepsen.util 2614 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,520] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:19:33,521] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,522] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,522] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,523] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,523] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:33,524] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:19:33,576] jepsen worker 52 - jepsen.util 682 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,577] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,577] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:33,577] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,578] jepsen worker 53 - jepsen.util 593 :invoke :write [17 45] INFO [2022-03-13 17:19:33,578] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:19:33,580] jepsen worker 7 - jepsen.util 637 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,580] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:33,580] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,581] jepsen worker 7 - jepsen.util 637 :invoke :write [2 30] INFO [2022-03-13 17:19:33,582] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:33,582] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:33,583] jepsen worker 63 - jepsen.util 4833 :info :write [21 47] :RequestTimeoutException INFO [2022-03-13 17:19:33,584] jepsen worker 63 - jepsen.util 4923 :invoke :read [21 nil] INFO [2022-03-13 17:19:33,588] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,588] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] INFO [2022-03-13 17:19:33,590] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,591] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:19:33,621] jepsen worker 38 - jepsen.util 2828 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,622] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,640] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,640] jepsen worker 75 - jepsen.util 525 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,640] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,641] jepsen worker 76 - jepsen.util 796 :invoke :write [25 14] INFO [2022-03-13 17:19:33,642] jepsen worker 13 - jepsen.util 733 :invoke :write [4 42] INFO [2022-03-13 17:19:33,643] jepsen worker 75 - jepsen.util 525 :invoke :write [25 29] INFO [2022-03-13 17:19:33,655] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,656] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 22] INFO [2022-03-13 17:19:33,686] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,686] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,687] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 0] INFO [2022-03-13 17:19:33,687] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,688] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,689] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:19:33,726] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,726] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,726] jepsen worker 37 - jepsen.util 2467 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,727] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,728] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:33,730] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,731] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,732] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:19:33,738] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,739] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,740] jepsen worker 26 - jepsen.util 566 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,741] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:33,741] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:33,742] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:33,746] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,747] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:19:33,752] jepsen worker 38 - jepsen.util 2828 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,753] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 21] INFO [2022-03-13 17:19:33,753] jepsen worker 23 - jepsen.util 563 :fail :write [7 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,754] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:19:33,788] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,789] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:19:33,822] jepsen worker 38 - jepsen.util 2828 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,823] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:19:33,833] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,833] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:19:33,836] jepsen worker 88 - jepsen.util 2428 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,837] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 49] INFO [2022-03-13 17:19:33,837] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,838] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,850] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,850] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 21] INFO [2022-03-13 17:19:33,881] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,882] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,887] jepsen worker 60 - jepsen.util 690 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,888] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,888] jepsen worker 60 - jepsen.util 690 :invoke :write [20 46] INFO [2022-03-13 17:19:33,888] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:19:33,894] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,894] jepsen worker 70 - jepsen.util 880 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,894] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:33,894] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,894] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:33,895] jepsen worker 62 - jepsen.util 512 :invoke :write [20 0] INFO [2022-03-13 17:19:33,896] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:33,896] jepsen worker 31 - jepsen.util 4801 :info :write [10 35] :RequestTimeoutException INFO [2022-03-13 17:19:33,896] jepsen worker 8 - jepsen.util 728 :invoke :write [2 40] INFO [2022-03-13 17:19:33,897] jepsen worker 31 - jepsen.util 4891 :invoke :write [10 41] INFO [2022-03-13 17:19:33,913] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,914] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:33,929] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,929] jepsen worker 79 - jepsen.util 1159 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,929] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,932] jepsen worker 77 - jepsen.util 707 :invoke :write [25 47] INFO [2022-03-13 17:19:33,933] jepsen worker 61 - jepsen.util 781 :invoke :write [20 11] INFO [2022-03-13 17:19:33,933] jepsen worker 3 - jepsen.util 2253 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,933] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,934] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 16] INFO [2022-03-13 17:19:33,935] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:33,936] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:33,969] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,970] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,981] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:33,982] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:33,992] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:33,992] jepsen worker nemesis - couchbase.util Rebalance check count: 680 INFO [2022-03-13 17:19:33,993] jepsen worker nemesis - couchbase.util Rebalance stuck count: 552 INFO [2022-03-13 17:19:34,000] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,000] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,001] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 26] INFO [2022-03-13 17:19:34,001] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,006] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,007] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,008] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,009] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 8] INFO [2022-03-13 17:19:34,018] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,018] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,018] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,019] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:19:34,019] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:34,020] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,021] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,022] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:19:34,026] jepsen worker 30 - jepsen.util 4710 :info :write [10 3] :RequestTimeoutException INFO [2022-03-13 17:19:34,027] jepsen worker 30 - jepsen.util 4800 :invoke :read [10 nil] INFO [2022-03-13 17:19:34,028] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,029] jepsen worker 12 - jepsen.util 912 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,029] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:34,029] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:34,029] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:34,030] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:34,035] jepsen worker 76 - jepsen.util 796 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,036] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:34,040] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,040] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,040] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,040] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,040] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,040] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,040] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,041] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,042] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,042] jepsen worker 14 - jepsen.util 374 :invoke :write [4 42] INFO [2022-03-13 17:19:34,043] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:34,044] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:34,044] jepsen worker 51 - jepsen.util 771 :invoke :write [17 7] INFO [2022-03-13 17:19:34,047] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:34,047] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:19:34,048] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:34,050] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,053] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,099] jepsen worker 38 - jepsen.util 2828 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,100] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,109] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,109] jepsen worker 88 - jepsen.util 2428 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,110] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:19:34,110] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 10] INFO [2022-03-13 17:19:34,124] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,124] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,124] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:19:34,125] jepsen worker 71 - jepsen.util 611 :invoke :write [23 45] INFO [2022-03-13 17:19:34,157] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,157] jepsen worker 89 - jepsen.util 2789 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,157] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,158] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 44] INFO [2022-03-13 17:19:34,158] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,159] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,160] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,161] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,162] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,163] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,164] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,164] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 27] INFO [2022-03-13 17:19:34,172] jepsen worker 37 - jepsen.util 2467 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,172] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,172] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,173] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,188] jepsen worker 78 - jepsen.util 708 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,189] jepsen worker 78 - jepsen.util 708 :invoke :write [26 39] INFO [2022-03-13 17:19:34,196] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,197] jepsen worker 33 - jepsen.util 753 :invoke :write [11 22] INFO [2022-03-13 17:19:34,204] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,205] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:34,210] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,211] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:19:34,229] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,230] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:34,255] jepsen worker 89 - jepsen.util 2789 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,255] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,273] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,274] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:19:34,282] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,282] jepsen worker 13 - jepsen.util 733 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,282] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,285] jepsen worker 13 - jepsen.util 733 :invoke :write [4 32] INFO [2022-03-13 17:19:34,285] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,285] jepsen worker 7 - jepsen.util 637 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,285] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:34,286] jepsen worker 7 - jepsen.util 637 :invoke :write [2 43] INFO [2022-03-13 17:19:34,286] jepsen worker 42 - jepsen.util 402 :invoke :write [14 28] INFO [2022-03-13 17:19:34,287] jepsen worker 70 - jepsen.util 880 :invoke :write [23 11] INFO [2022-03-13 17:19:34,295] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,295] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:34,308] jepsen worker 34 - jepsen.util 934 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,311] jepsen worker 34 - jepsen.util 934 :invoke :write [11 8] INFO [2022-03-13 17:19:34,311] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,311] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 48] INFO [2022-03-13 17:19:34,311] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,312] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,314] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,315] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,318] jepsen worker 38 - jepsen.util 2828 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,318] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,319] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,320] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,321] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,321] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,322] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 4] INFO [2022-03-13 17:19:34,322] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,323] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,324] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:19:34,344] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,344] jepsen worker 88 - jepsen.util 2428 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,348] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,349] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 37] INFO [2022-03-13 17:19:34,374] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,374] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,393] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,393] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,394] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:19:34,394] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:19:34,408] jepsen worker 71 - jepsen.util 611 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,409] jepsen worker 71 - jepsen.util 611 :invoke :write [23 27] INFO [2022-03-13 17:19:34,415] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,415] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,417] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,417] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,419] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,420] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,424] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,425] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,425] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:34,425] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,426] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,426] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:34,426] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,427] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:34,427] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:34,427] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:34,443] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,443] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,444] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,444] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:19:34,445] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,446] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:34,468] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,468] jepsen worker 37 - jepsen.util 2467 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,470] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,470] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:19:34,476] jepsen worker 89 - jepsen.util 2789 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,476] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,478] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,479] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,480] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,480] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,481] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,482] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:19:34,517] jepsen worker 51 - jepsen.util 771 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,517] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:34,523] jepsen worker 36 - jepsen.util 2466 :fail :write [12 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,523] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 31] INFO [2022-03-13 17:19:34,532] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,532] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:34,548] jepsen worker 81 - jepsen.util 5301 :info :write [27 30] :RequestTimeoutException INFO [2022-03-13 17:19:34,549] jepsen worker 81 - jepsen.util 5391 :invoke :write [27 9] INFO [2022-03-13 17:19:34,599] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,599] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,599] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 5] INFO [2022-03-13 17:19:34,599] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,601] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,601] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 16] INFO [2022-03-13 17:19:34,612] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,612] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,612] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,613] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:19:34,613] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,614] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,615] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,615] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:19:34,624] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,625] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:19:34,671] jepsen worker 53 - jepsen.util 593 :fail :write [17 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,671] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,672] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,672] jepsen worker 75 - jepsen.util 525 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,672] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:34,672] jepsen worker 43 - jepsen.util 673 :invoke :write [14 8] INFO [2022-03-13 17:19:34,673] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:34,673] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:34,674] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:34,674] jepsen worker 53 - jepsen.util 593 :invoke :write [17 13] INFO [2022-03-13 17:19:34,684] jepsen worker 8 - jepsen.util 728 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,684] jepsen worker 8 - jepsen.util 728 :invoke :write [2 9] INFO [2022-03-13 17:19:34,694] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,694] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:19:34,707] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,708] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 29] INFO [2022-03-13 17:19:34,711] jepsen worker 38 - jepsen.util 2828 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,711] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:19:34,732] jepsen worker 13 - jepsen.util 733 :fail :write [4 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,733] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,733] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,733] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:34,734] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:34,735] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:34,735] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,735] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,736] jepsen worker 60 - jepsen.util 690 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,736] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,736] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:34,737] jepsen worker 76 - jepsen.util 796 :invoke :write [25 2] INFO [2022-03-13 17:19:34,738] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,738] jepsen worker 60 - jepsen.util 690 :invoke :write [20 23] INFO [2022-03-13 17:19:34,739] jepsen worker 13 - jepsen.util 733 :invoke :write [4 44] INFO [2022-03-13 17:19:34,739] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:34,741] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,741] jepsen worker 75 - jepsen.util 525 :invoke :write [25 7] INFO [2022-03-13 17:19:34,773] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,773] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,774] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,775] jepsen worker 22 - jepsen.util 652 :invoke :write [7 39] INFO [2022-03-13 17:19:34,776] jepsen worker 52 - jepsen.util 682 :invoke :write [17 22] INFO [2022-03-13 17:19:34,776] jepsen worker 26 - jepsen.util 566 :invoke :write [8 24] INFO [2022-03-13 17:19:34,788] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,789] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 34] INFO [2022-03-13 17:19:34,805] jepsen worker 36 - jepsen.util 2466 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,806] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,822] jepsen worker 89 - jepsen.util 2789 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,823] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,843] jepsen worker 14 - jepsen.util 374 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,843] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:19:34,851] jepsen worker 61 - jepsen.util 781 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,851] jepsen worker 71 - jepsen.util 611 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,852] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,852] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:34,853] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:34,854] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:34,854] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,855] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:34,855] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,856] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,856] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:34,857] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:19:34,858] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,858] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:34,859] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,860] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:19:34,862] jepsen worker 53 - jepsen.util 593 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,863] jepsen worker 53 - jepsen.util 593 :invoke :write [17 9] INFO [2022-03-13 17:19:34,894] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,895] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,908] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,908] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,908] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,909] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:19:34,910] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,911] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,912] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,913] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,914] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,915] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,915] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,915] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 49] INFO [2022-03-13 17:19:34,916] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,917] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:34,917] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,918] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:34,937] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,937] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,938] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:34,939] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:34,957] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,957] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,957] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,958] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:19:34,959] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,960] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:19:34,961] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,962] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,963] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,964] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,965] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,966] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,967] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,968] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:34,969] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:34,970] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:19:34,999] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:34,999] jepsen worker nemesis - couchbase.util Rebalance check count: 681 INFO [2022-03-13 17:19:34,999] jepsen worker nemesis - couchbase.util Rebalance stuck count: 553 INFO [2022-03-13 17:19:35,001] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,001] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,009] jepsen worker 88 - jepsen.util 2428 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,010] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 34] INFO [2022-03-13 17:19:35,027] jepsen worker 76 - jepsen.util 796 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,027] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:35,063] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,064] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:35,075] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,075] jepsen worker 38 - jepsen.util 2828 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,076] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,076] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 15] INFO [2022-03-13 17:19:35,077] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,078] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:19:35,118] jepsen worker 37 - jepsen.util 2467 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,119] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,158] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,158] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:19:35,162] jepsen worker 62 - jepsen.util 512 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,162] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,196] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:35,162] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,163] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:35,168] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,197] jepsen worker 44 - jepsen.util 764 :invoke :write [14 36] INFO [2022-03-13 17:19:35,197] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:35,178] jepsen worker 38 - jepsen.util 2828 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,198] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 35] INFO [2022-03-13 17:19:35,183] jepsen worker 75 - jepsen.util 525 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,183] jepsen worker 43 - jepsen.util 673 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,203] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:35,196] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,196] jepsen worker 6 - jepsen.util 636 :invoke :write [2 26] INFO [2022-03-13 17:19:35,204] jepsen worker 43 - jepsen.util 673 :invoke :write [14 3] INFO [2022-03-13 17:19:35,204] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:35,197] jepsen worker 12 - jepsen.util 912 :invoke :write [4 44] INFO [2022-03-13 17:19:35,237] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,237] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,238] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:19:35,239] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,239] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:19:35,240] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,279] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,279] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,320] jepsen worker 13 - jepsen.util 733 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,320] jepsen worker 60 - jepsen.util 690 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,320] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,320] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,320] jepsen worker 76 - jepsen.util 796 :invoke :write [25 17] INFO [2022-03-13 17:19:35,321] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:35,322] jepsen worker 75 - jepsen.util 525 :invoke :write [25 35] INFO [2022-03-13 17:19:35,322] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:19:35,324] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,325] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 15] INFO [2022-03-13 17:19:35,331] jepsen worker 77 - jepsen.util 707 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,332] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:35,353] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,353] jepsen worker 88 - jepsen.util 2428 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,354] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 29] INFO [2022-03-13 17:19:35,354] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,356] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,356] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 28] INFO [2022-03-13 17:19:35,357] jepsen worker 22 - jepsen.util 652 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,358] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:19:35,359] jepsen worker 38 - jepsen.util 2828 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,359] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,359] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,360] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 30] INFO [2022-03-13 17:19:35,363] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,364] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 8] INFO [2022-03-13 17:19:35,394] jepsen worker 0 - jepsen.util 4500 :info :write [0 19] :RequestTimeoutException INFO [2022-03-13 17:19:35,395] jepsen worker 0 - jepsen.util 4590 :invoke :write [0 23] INFO [2022-03-13 17:19:35,424] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,424] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,425] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 42] INFO [2022-03-13 17:19:35,425] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,468] jepsen worker 38 - jepsen.util 2828 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,469] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,485] jepsen worker 33 - jepsen.util 753 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,485] jepsen worker 33 - jepsen.util 753 :invoke :write [11 21] INFO [2022-03-13 17:19:35,493] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,493] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,494] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,494] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:35,495] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,496] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:19:35,496] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,497] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:19:35,498] jepsen worker 4 - jepsen.util 2614 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,499] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,499] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:35,500] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:19:35,501] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,502] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:35,503] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,503] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,503] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,504] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:35,505] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,505] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:35,507] jepsen worker 71 - jepsen.util 611 :invoke :write [23 35] INFO [2022-03-13 17:19:35,508] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:35,510] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,510] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:35,511] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,512] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:35,513] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,514] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:35,515] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,521] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:35,521] jepsen worker 52 - jepsen.util 682 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,522] jepsen worker 52 - jepsen.util 682 :invoke :write [17 27] INFO [2022-03-13 17:19:35,522] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,523] jepsen worker 69 - jepsen.util 789 :invoke :write [23 20] INFO [2022-03-13 17:19:35,575] jepsen worker 89 - jepsen.util 2789 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,575] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,575] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,576] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,577] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,577] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,577] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,577] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:19:35,578] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,579] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 36] INFO [2022-03-13 17:19:35,582] jepsen worker 88 - jepsen.util 2428 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,583] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,585] jepsen worker 3 - jepsen.util 2253 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,585] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:35,636] jepsen worker 76 - jepsen.util 796 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,636] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:35,639] jepsen worker 7 - jepsen.util 637 :fail :write [2 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,639] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,640] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,639] jepsen worker 34 - jepsen.util 934 :fail :write [11 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,639] jepsen worker 7 - jepsen.util 637 :invoke :write [2 48] INFO [2022-03-13 17:19:35,640] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,640] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,640] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,640] jepsen worker 34 - jepsen.util 934 :invoke :write [11 36] INFO [2022-03-13 17:19:35,641] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:35,642] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:19:35,643] jepsen worker 21 - jepsen.util 831 :invoke :write [7 32] INFO [2022-03-13 17:19:35,643] jepsen worker 80 - jepsen.util 800 :invoke :write [26 15] INFO [2022-03-13 17:19:35,645] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:35,657] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,658] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:19:35,667] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,667] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,694] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,695] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,696] jepsen worker 82 - jepsen.util 4942 :info :write [27 6] :RequestTimeoutException INFO [2022-03-13 17:19:35,697] jepsen worker 82 - jepsen.util 5032 :invoke :write [27 35] INFO [2022-03-13 17:19:35,697] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,698] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:19:35,712] jepsen worker 71 - jepsen.util 611 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,713] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:19:35,721] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,721] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:35,721] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,722] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:35,749] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,749] jepsen worker 89 - jepsen.util 2789 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,750] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 31] INFO [2022-03-13 17:19:35,750] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,752] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,753] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,754] jepsen worker 79 - jepsen.util 1159 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,755] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,755] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 37] INFO [2022-03-13 17:19:35,755] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,757] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,757] jepsen worker 37 - jepsen.util 2467 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,757] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,758] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 7] INFO [2022-03-13 17:19:35,758] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,759] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,760] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,760] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 28] INFO [2022-03-13 17:19:35,784] jepsen worker 26 - jepsen.util 566 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,784] jepsen worker 75 - jepsen.util 525 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,784] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,784] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:35,785] jepsen worker 75 - jepsen.util 525 :invoke :write [25 11] INFO [2022-03-13 17:19:35,786] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:19:35,790] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,790] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:19:35,794] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,794] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,796] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 26] INFO [2022-03-13 17:19:35,797] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:35,873] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,873] jepsen worker 89 - jepsen.util 2789 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,873] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:19:35,874] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,907] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,907] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,930] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,930] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:19:35,951] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,951] jepsen worker 38 - jepsen.util 2828 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,952] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,953] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 13] INFO [2022-03-13 17:19:35,954] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,955] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:35,957] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,957] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:35,961] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,961] jepsen worker 88 - jepsen.util 2428 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,961] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,961] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:35,967] jepsen worker 41 - jepsen.util 4721 :info :write [13 8] :RequestTimeoutException INFO [2022-03-13 17:19:35,968] jepsen worker 41 - jepsen.util 4811 :invoke :read [13 nil] INFO [2022-03-13 17:19:35,974] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:35,975] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] INFO [2022-03-13 17:19:36,004] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:36,050] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,081] jepsen worker nemesis - couchbase.util Rebalance check count: 682 INFO [2022-03-13 17:19:36,081] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,081] jepsen worker nemesis - couchbase.util Rebalance stuck count: 554 INFO [2022-03-13 17:19:36,095] jepsen worker 89 - jepsen.util 2789 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,095] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 7] INFO [2022-03-13 17:19:36,128] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,128] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,167] jepsen worker 33 - jepsen.util 753 :fail :write [11 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,167] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,167] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,168] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,169] jepsen worker 51 - jepsen.util 771 :invoke :write [17 26] INFO [2022-03-13 17:19:36,170] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:36,171] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,172] jepsen worker 33 - jepsen.util 753 :invoke :write [11 36] INFO [2022-03-13 17:19:36,176] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,176] jepsen worker 37 - jepsen.util 2467 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,176] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,177] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:36,177] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,178] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,182] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,183] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,184] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,184] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,200] jepsen worker 69 - jepsen.util 789 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,201] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,204] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,204] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,204] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,205] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:19:36,206] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,206] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,207] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,207] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,207] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,208] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,208] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,209] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,209] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,210] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,210] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:19:36,211] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,212] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,213] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 37] INFO [2022-03-13 17:19:36,222] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,222] jepsen worker 89 - jepsen.util 2789 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,223] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,224] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:19:36,227] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,229] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:19:36,244] jepsen worker 53 - jepsen.util 593 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,244] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,246] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,246] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:36,247] jepsen worker 76 - jepsen.util 796 :invoke :write [25 21] INFO [2022-03-13 17:19:36,247] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:36,249] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,250] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,250] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:19:36,251] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,253] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,254] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,255] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,256] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:19:36,262] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,263] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,274] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,275] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 1] INFO [2022-03-13 17:19:36,286] jepsen worker 42 - jepsen.util 402 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,286] jepsen worker 42 - jepsen.util 402 :invoke :write [14 45] INFO [2022-03-13 17:19:36,299] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,300] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,301] jepsen worker 34 - jepsen.util 934 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,302] jepsen worker 34 - jepsen.util 934 :invoke :write [11 24] INFO [2022-03-13 17:19:36,316] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,316] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,317] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:19:36,317] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,318] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,319] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:19:36,354] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,355] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,355] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:19:36,357] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:36,370] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,370] jepsen worker 43 - jepsen.util 673 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,371] jepsen worker 78 - jepsen.util 708 :fail :write [26 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,371] jepsen worker 21 - jepsen.util 831 :fail :write [7 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,371] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,371] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:36,372] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] INFO [2022-03-13 17:19:36,373] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,373] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,373] jepsen worker 78 - jepsen.util 708 :invoke :write [26 1] INFO [2022-03-13 17:19:36,374] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:19:36,381] jepsen worker 60 - jepsen.util 690 :invoke :write [20 9] INFO [2022-03-13 17:19:36,381] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:36,381] jepsen worker 61 - jepsen.util 781 :invoke :write [20 49] INFO [2022-03-13 17:19:36,382] jepsen worker 4 - jepsen.util 2614 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,387] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,446] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,446] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,446] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,446] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,447] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,447] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:19:36,448] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:36,448] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,448] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,448] jepsen worker 75 - jepsen.util 525 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,450] jepsen worker 13 - jepsen.util 733 :invoke :write [4 5] INFO [2022-03-13 17:19:36,450] jepsen worker 75 - jepsen.util 525 :invoke :write [25 0] INFO [2022-03-13 17:19:36,452] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:19:36,455] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,456] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,458] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,466] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,466] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,467] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 5] INFO [2022-03-13 17:19:36,468] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,469] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,471] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:19:36,473] jepsen worker 70 - jepsen.util 880 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,474] jepsen worker 80 - jepsen.util 800 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,477] jepsen worker 70 - jepsen.util 880 :invoke :write [23 1] INFO [2022-03-13 17:19:36,477] jepsen worker 80 - jepsen.util 800 :invoke :write [26 40] INFO [2022-03-13 17:19:36,529] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,529] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,530] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,531] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 28] INFO [2022-03-13 17:19:36,553] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,554] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,569] jepsen worker 79 - jepsen.util 1159 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,569] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,569] jepsen worker 77 - jepsen.util 707 :invoke :write [25 25] INFO [2022-03-13 17:19:36,570] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:36,572] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,573] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,573] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,573] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,574] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:19:36,574] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,576] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,576] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,577] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,577] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,578] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,578] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,578] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,579] jepsen worker 25 - jepsen.util 565 :invoke :write [8 8] INFO [2022-03-13 17:19:36,579] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:19:36,580] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:36,580] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,581] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,582] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:36,583] jepsen worker 24 - jepsen.util 654 :invoke :write [8 32] INFO [2022-03-13 17:19:36,591] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,592] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,593] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,594] jepsen worker 89 - jepsen.util 2789 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,594] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,595] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,598] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,598] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,599] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,599] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,600] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,601] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,601] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,602] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 35] INFO [2022-03-13 17:19:36,603] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,603] jepsen worker 88 - jepsen.util 2428 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,604] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,604] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 44] INFO [2022-03-13 17:19:36,605] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,606] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,608] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,609] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 23] INFO [2022-03-13 17:19:36,645] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,645] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,650] jepsen worker 76 - jepsen.util 796 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,650] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:36,670] jepsen worker 8 - jepsen.util 728 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,670] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,671] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,671] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:19:36,671] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:36,671] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,675] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,675] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:19:36,683] jepsen worker 33 - jepsen.util 753 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,683] jepsen worker 33 - jepsen.util 753 :invoke :write [11 7] INFO [2022-03-13 17:19:36,685] jepsen worker 89 - jepsen.util 2789 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,686] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:19:36,700] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,700] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,701] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,701] jepsen worker 71 - jepsen.util 611 :invoke :write [23 48] INFO [2022-03-13 17:19:36,704] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,705] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,706] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,706] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:36,716] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,716] jepsen worker 69 - jepsen.util 789 :invoke :write [23 0] INFO [2022-03-13 17:19:36,737] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,738] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:19:36,747] jepsen worker 38 - jepsen.util 2828 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,747] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,749] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,749] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,750] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,750] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:19:36,751] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:19:36,752] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,753] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,754] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 22] INFO [2022-03-13 17:19:36,768] jepsen worker 78 - jepsen.util 708 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,769] jepsen worker 78 - jepsen.util 708 :invoke :write [26 9] INFO [2022-03-13 17:19:36,770] jepsen worker 37 - jepsen.util 2467 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,771] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,787] jepsen worker 52 - jepsen.util 682 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,787] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,787] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,788] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:36,788] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:36,789] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:36,789] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:36,790] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:36,801] jepsen worker 51 - jepsen.util 771 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,802] jepsen worker 51 - jepsen.util 771 :invoke :write [17 22] INFO [2022-03-13 17:19:36,810] jepsen worker 67 - jepsen.util 4027 :info :write [22 34] :RequestTimeoutException INFO [2022-03-13 17:19:36,810] jepsen worker 71 - jepsen.util 611 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,811] jepsen worker 67 - jepsen.util 4117 :invoke :read [22 nil] INFO [2022-03-13 17:19:36,811] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:19:36,840] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,840] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:19:36,843] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,843] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,843] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 36] INFO [2022-03-13 17:19:36,844] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,845] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,846] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,847] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,847] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:36,849] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,849] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 49] INFO [2022-03-13 17:19:36,868] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:19:36,868] jepsen worker 50 - jepsen.util 3470 :invoke :write [16 29] INFO [2022-03-13 17:19:36,870] jepsen worker 6 - jepsen.util 636 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,870] jepsen worker 13 - jepsen.util 733 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,870] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:36,871] jepsen worker 13 - jepsen.util 733 :invoke :write [4 14] INFO [2022-03-13 17:19:36,875] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,876] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:36,884] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,884] jepsen worker 88 - jepsen.util 2428 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,884] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,885] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 31] INFO [2022-03-13 17:19:36,899] jepsen worker 34 - jepsen.util 934 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,900] jepsen worker 34 - jepsen.util 934 :invoke :write [11 23] INFO [2022-03-13 17:19:36,924] jepsen worker 69 - jepsen.util 789 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,925] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:19:36,945] jepsen worker 38 - jepsen.util 2828 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,947] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,953] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,953] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,957] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,958] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,959] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:36,959] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:19:36,974] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:36,975] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:37,011] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,012] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:19:37,024] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,024] jepsen worker 89 - jepsen.util 2789 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,025] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:19:37,025] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 5] INFO [2022-03-13 17:19:37,031] jepsen worker 3 - jepsen.util 2253 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,032] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:19:37,033] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,033] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:37,076] jepsen worker 60 - jepsen.util 690 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,076] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,076] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,076] jepsen worker 75 - jepsen.util 525 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,076] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,076] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,076] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,076] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:37,077] jepsen worker 60 - jepsen.util 690 :invoke :write [20 15] INFO [2022-03-13 17:19:37,077] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:37,078] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:37,079] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:37,079] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,080] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:37,081] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,081] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:37,081] jepsen worker 52 - jepsen.util 682 :invoke :write [17 17] INFO [2022-03-13 17:19:37,082] jepsen worker 43 - jepsen.util 673 :invoke :write [14 11] INFO [2022-03-13 17:19:37,083] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:37,084] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:37,085] jepsen worker 76 - jepsen.util 796 :invoke :write [25 0] INFO [2022-03-13 17:19:37,085] jepsen worker 26 - jepsen.util 566 :invoke :write [8 0] INFO [2022-03-13 17:19:37,087] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:37,087] jepsen worker nemesis - couchbase.util Rebalance check count: 683 INFO [2022-03-13 17:19:37,087] jepsen worker nemesis - couchbase.util Rebalance stuck count: 555 INFO [2022-03-13 17:19:37,088] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,088] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,090] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,091] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:19:37,091] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:37,092] jepsen worker 53 - jepsen.util 593 :invoke :write [17 34] INFO [2022-03-13 17:19:37,092] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,093] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:19:37,111] jepsen worker 44 - jepsen.util 764 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,112] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,112] jepsen worker 44 - jepsen.util 764 :invoke :write [14 30] INFO [2022-03-13 17:19:37,112] jepsen worker 13 - jepsen.util 733 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,112] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,113] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:37,114] jepsen worker 75 - jepsen.util 525 :invoke :write [25 20] INFO [2022-03-13 17:19:37,114] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:37,116] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,117] jepsen worker 13 - jepsen.util 733 :invoke :write [4 45] INFO [2022-03-13 17:19:37,138] jepsen worker 89 - jepsen.util 2789 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,139] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:37,152] jepsen worker 60 - jepsen.util 690 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,153] jepsen worker 60 - jepsen.util 690 :invoke :write [20 29] INFO [2022-03-13 17:19:37,156] jepsen worker 50 - jepsen.util 3470 :fail :write [16 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,156] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,156] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,156] jepsen worker 88 - jepsen.util 2428 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,156] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,158] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:37,159] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:37,160] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:37,160] jepsen worker 50 - jepsen.util 3470 :invoke :write [16 36] INFO [2022-03-13 17:19:37,161] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 33] INFO [2022-03-13 17:19:37,162] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,162] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 18] WARN [2022-03-13 17:19:37,199] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":13358416,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13358416,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":8230810,"encode_duration_us":18386,"operation_id":"0x1371d","timeout_ms":10000,"last_server_duration_us":8230810,"total_duration_us":10004805}],"total_count":1}} INFO [2022-03-13 17:19:37,201] jepsen worker 61 - jepsen.util 781 :fail :write [20 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,201] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,201] jepsen worker 80 - jepsen.util 800 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,201] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,202] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:37,203] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:37,204] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 24] INFO [2022-03-13 17:19:37,205] jepsen worker 80 - jepsen.util 800 :invoke :write [26 25] INFO [2022-03-13 17:19:37,218] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,218] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,219] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:37,219] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:37,247] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,247] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,248] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:19:37,248] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] WARN [2022-03-13 17:19:37,299] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8631597,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8631597,"total_server_duration_us":20203,"encode_duration_us":25801,"operation_id":"0x13c23","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10003483},{"operation_name":"upsert","last_dispatch_duration_us":9199469,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9199469,"total_server_duration_us":9090500,"encode_duration_us":19354,"operation_id":"0x13bb8","timeout_ms":10000,"last_server_duration_us":9090500,"total_duration_us":9203835},{"operation_name":"get","last_dispatch_duration_us":8007511,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8007511,"total_server_duration_us":124227,"operation_id":"0x13ada","timeout_ms":30000,"last_server_duration_us":124227,"total_duration_us":8009811},{"operation_name":"get","last_dispatch_duration_us":3521582,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":3521582,"total_server_duration_us":32,"operation_id":"0x13dbe","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":3524277},{"operation_name":"upsert","last_dispatch_duration_us":1918500,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1918500,"total_server_duration_us":948957,"encode_duration_us":17193,"operation_id":"0x13f88","timeout_ms":10000,"last_server_duration_us":948957,"total_duration_us":1922153},{"operation_name":"upsert","last_dispatch_duration_us":676818,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":676818,"total_server_duration_us":169781,"encode_duration_us":18983,"operation_id":"0x140b9","timeout_ms":10000,"last_server_duration_us":169781,"total_duration_us":1768285},{"operation_name":"get","last_dispatch_duration_us":1613694,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1613694,"total_server_duration_us":38,"operation_id":"0x13fae","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":1613948},{"operation_name":"get","last_dispatch_duration_us":1613390,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1613390,"total_server_duration_us":19,"operation_id":"0x13faf","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":1613555},{"operation_name":"get","last_dispatch_duration_us":1612985,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1612985,"total_server_duration_us":19,"operation_id":"0x13fb0","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":1613122},{"operation_name":"upsert","last_dispatch_duration_us":1558905,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1558905,"total_server_duration_us":62796,"encode_duration_us":17976,"operation_id":"0x13fd4","timeout_ms":10000,"last_server_duration_us":62796,"total_duration_us":1564824}],"total_count":85}} INFO [2022-03-13 17:19:37,403] jepsen worker 43 - jepsen.util 673 :fail :write [14 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,404] jepsen worker 12 - jepsen.util 912 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,404] jepsen worker 43 - jepsen.util 673 :invoke :write [14 15] INFO [2022-03-13 17:19:37,404] jepsen worker 12 - jepsen.util 912 :invoke :write [4 27] INFO [2022-03-13 17:19:37,411] jepsen worker 25 - jepsen.util 565 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,412] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,412] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:37,413] jepsen worker 35 - jepsen.util 935 :invoke :write [11 30] INFO [2022-03-13 17:19:37,420] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,420] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,421] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:19:37,421] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:37,423] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,423] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:19:37,452] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,453] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 24] INFO [2022-03-13 17:19:37,464] jepsen worker 33 - jepsen.util 753 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,465] jepsen worker 33 - jepsen.util 753 :invoke :write [11 25] INFO [2022-03-13 17:19:37,500] jepsen worker 76 - jepsen.util 796 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,500] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:37,538] jepsen worker 77 - jepsen.util 707 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,538] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,539] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:19:37,539] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:37,632] jepsen worker 51 - jepsen.util 771 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,632] jepsen worker 51 - jepsen.util 771 :invoke :write [17 13] INFO [2022-03-13 17:19:37,649] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,649] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,650] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:37,650] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:19:37,708] jepsen worker 53 - jepsen.util 593 :fail :write [17 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,708] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:37,719] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,720] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:37,723] jepsen worker 42 - jepsen.util 402 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,724] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:37,755] jepsen worker 34 - jepsen.util 934 :fail :write [11 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,756] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:37,783] jepsen worker 75 - jepsen.util 525 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,784] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:37,805] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,805] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:37,806] jepsen worker 71 - jepsen.util 611 :invoke :write [23 22] INFO [2022-03-13 17:19:37,806] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:19:38,038] jepsen worker 80 - jepsen.util 800 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,038] jepsen worker 80 - jepsen.util 800 :invoke :write [26 23] INFO [2022-03-13 17:19:38,060] jepsen worker 71 - jepsen.util 611 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,060] jepsen worker 71 - jepsen.util 611 :invoke :write [23 36] INFO [2022-03-13 17:19:38,077] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,077] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,077] jepsen worker 78 - jepsen.util 708 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,078] jepsen worker 78 - jepsen.util 708 :invoke :write [26 7] INFO [2022-03-13 17:19:38,078] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:38,079] jepsen worker 23 - jepsen.util 563 :invoke :write [7 2] INFO [2022-03-13 17:19:38,087] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,088] jepsen worker 69 - jepsen.util 789 :invoke :write [23 26] INFO [2022-03-13 17:19:38,092] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:38,092] jepsen worker nemesis - couchbase.util Rebalance check count: 684 INFO [2022-03-13 17:19:38,092] jepsen worker nemesis - couchbase.util Rebalance stuck count: 556 INFO [2022-03-13 17:19:38,132] jepsen worker 26 - jepsen.util 566 :fail :write [8 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,133] jepsen worker 26 - jepsen.util 566 :invoke :write [8 34] INFO [2022-03-13 17:19:38,150] jepsen worker 70 - jepsen.util 880 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,150] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,179] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,179] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,180] jepsen worker 21 - jepsen.util 831 :invoke :write [7 35] INFO [2022-03-13 17:19:38,180] jepsen worker 25 - jepsen.util 565 :invoke :write [8 25] INFO [2022-03-13 17:19:38,226] jepsen worker 13 - jepsen.util 733 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,226] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,226] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,227] jepsen worker 13 - jepsen.util 733 :invoke :write [4 49] INFO [2022-03-13 17:19:38,228] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:38,228] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:38,228] jepsen worker 71 - jepsen.util 611 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,229] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,229] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,229] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:19:38,230] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:38,231] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:38,244] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,244] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,244] jepsen worker 75 - jepsen.util 525 :invoke :write [25 1] INFO [2022-03-13 17:19:38,245] jepsen worker 76 - jepsen.util 796 :invoke :write [25 19] INFO [2022-03-13 17:19:38,249] jepsen worker 79 - jepsen.util 1159 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,250] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:38,279] jepsen worker 35 - jepsen.util 935 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,280] jepsen worker 35 - jepsen.util 935 :invoke :write [11 44] INFO [2022-03-13 17:19:38,287] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,288] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:38,301] jepsen worker 69 - jepsen.util 789 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,303] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,307] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,307] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:19:38,320] jepsen worker 37 - jepsen.util 2467 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,320] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:19:38,327] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,327] jepsen worker 52 - jepsen.util 682 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,328] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:38,328] jepsen worker 52 - jepsen.util 682 :invoke :write [17 40] INFO [2022-03-13 17:19:38,333] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,333] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,333] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:38,334] jepsen worker 6 - jepsen.util 636 :invoke :write [2 34] INFO [2022-03-13 17:19:38,357] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,358] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:19:38,375] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,376] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,376] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,376] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,377] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,378] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,378] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,379] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:19:38,379] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,380] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,381] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,382] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,383] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,384] jepsen worker 71 - jepsen.util 611 :invoke :write [23 1] INFO [2022-03-13 17:19:38,386] jepsen worker 89 - jepsen.util 2789 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,386] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:38,387] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,387] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,387] jepsen worker 33 - jepsen.util 753 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,388] jepsen worker 61 - jepsen.util 781 :invoke :write [20 35] INFO [2022-03-13 17:19:38,389] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:38,389] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:38,393] jepsen worker 50 - jepsen.util 3470 :fail :write [16 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,394] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] INFO [2022-03-13 17:19:38,432] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,432] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:38,473] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,473] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:38,475] jepsen worker 60 - jepsen.util 690 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,475] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:38,476] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,476] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,476] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:38,477] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:19:38,481] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,481] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:19:38,488] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,488] jepsen worker 51 - jepsen.util 771 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,488] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:38,489] jepsen worker 51 - jepsen.util 771 :invoke :write [17 5] INFO [2022-03-13 17:19:38,489] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,490] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] WARN [2022-03-13 17:19:38,501] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17198925,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17792663,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9087663,"encode_duration_us":24746,"operation_id":"0x13623","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10004223},{"operation_name":"upsert","last_dispatch_duration_us":9044460,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10038507,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9417727,"encode_duration_us":21732,"operation_id":"0x13aca","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009904}],"total_count":2}} INFO [2022-03-13 17:19:38,506] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,506] jepsen worker 44 - jepsen.util 764 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,507] jepsen worker 44 - jepsen.util 764 :invoke :write [14 8] INFO [2022-03-13 17:19:38,507] jepsen worker 8 - jepsen.util 728 :invoke :write [2 34] INFO [2022-03-13 17:19:38,511] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,513] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,543] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,544] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:19:38,551] jepsen worker 13 - jepsen.util 733 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,552] jepsen worker 13 - jepsen.util 733 :invoke :write [4 39] INFO [2022-03-13 17:19:38,557] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,558] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 36] INFO [2022-03-13 17:19:38,563] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,564] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:38,568] jepsen worker 80 - jepsen.util 800 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,569] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:38,616] jepsen worker 71 - jepsen.util 611 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,616] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,617] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:38,617] jepsen worker 78 - jepsen.util 708 :fail :write [26 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,617] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:19:38,618] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:38,623] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,624] jepsen worker 71 - jepsen.util 611 :invoke :write [23 21] INFO [2022-03-13 17:19:38,629] jepsen worker 88 - jepsen.util 2428 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,629] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,630] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:38,630] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:19:38,631] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,632] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 35] INFO [2022-03-13 17:19:38,663] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,663] jepsen worker 21 - jepsen.util 831 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,664] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:38,665] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:38,665] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,666] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,666] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:38,667] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 3] INFO [2022-03-13 17:19:38,667] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,668] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,668] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,668] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:38,668] jepsen worker 43 - jepsen.util 673 :fail :write [14 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,668] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:38,668] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,669] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,669] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:19:38,670] jepsen worker 53 - jepsen.util 593 :invoke :write [17 12] INFO [2022-03-13 17:19:38,672] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:38,672] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 6] INFO [2022-03-13 17:19:38,680] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,680] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:38,685] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,686] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] WARN [2022-03-13 17:19:38,702] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8357964,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8570839,"total_server_duration_us":127569,"encode_duration_us":21308,"operation_id":"0x13c38","timeout_ms":10000,"last_server_duration_us":26241,"total_duration_us":10061977},{"operation_name":"upsert","last_dispatch_duration_us":8215912,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8776556,"total_server_duration_us":321131,"encode_duration_us":20017,"operation_id":"0x13cf4","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10010830},{"operation_name":"upsert","last_dispatch_duration_us":7997965,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8751158,"total_server_duration_us":454701,"encode_duration_us":17857,"operation_id":"0x13bd2","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":10010198},{"operation_name":"upsert","last_dispatch_duration_us":9044460,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10038507,"total_server_duration_us":9417727,"encode_duration_us":21732,"operation_id":"0x13aca","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10009904},{"operation_name":"upsert","last_dispatch_duration_us":8358138,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9526607,"total_server_duration_us":473108,"encode_duration_us":21269,"operation_id":"0x13ac7","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10007218},{"operation_name":"upsert","last_dispatch_duration_us":46181,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":484482,"total_server_duration_us":252800,"encode_duration_us":17668,"operation_id":"0x13df0","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10004161},{"operation_name":"upsert","last_dispatch_duration_us":39932,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":39932,"total_server_duration_us":22389,"encode_duration_us":23467,"operation_id":"0x13af9","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10001347},{"operation_name":"upsert","last_dispatch_duration_us":2372624,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2372624,"total_server_duration_us":390093,"encode_duration_us":28089,"operation_id":"0x13fca","timeout_ms":10000,"last_server_duration_us":390093,"total_duration_us":2377716},{"operation_name":"upsert","last_dispatch_duration_us":2348290,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2348290,"total_server_duration_us":35312,"encode_duration_us":18120,"operation_id":"0x13ff5","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":2353271},{"operation_name":"upsert","last_dispatch_duration_us":2149447,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2149447,"total_server_duration_us":57509,"encode_duration_us":27959,"operation_id":"0x142ed","timeout_ms":10000,"last_server_duration_us":57509,"total_duration_us":2153069}],"total_count":47}} WARN [2022-03-13 17:19:38,702] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":109714,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":6748889,"total_server_duration_us":16747,"operation_id":"0x13183","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":30007872},{"operation_name":"get","last_dispatch_duration_us":18424868,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18424868,"total_server_duration_us":32,"operation_id":"0x1363e","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":18429306},{"operation_name":"get","last_dispatch_duration_us":18202992,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18202992,"total_server_duration_us":15,"operation_id":"0x1365d","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":18204273},{"operation_name":"upsert","last_dispatch_duration_us":9328361,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10292913,"total_server_duration_us":9215771,"encode_duration_us":41415,"operation_id":"0x13aea","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10010576},{"operation_name":"upsert","last_dispatch_duration_us":366291,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":1368091,"total_server_duration_us":211054,"encode_duration_us":27478,"operation_id":"0x13aeb","timeout_ms":10000,"last_server_duration_us":60254,"total_duration_us":10009233},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","encode_duration_us":19681,"operation_id":"0x13c9a","timeout_ms":10000,"total_duration_us":10007296},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","encode_duration_us":28784,"operation_id":"0x13c09","timeout_ms":10000,"total_duration_us":10005637},{"operation_name":"upsert","last_dispatch_duration_us":11678925,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":11678925,"total_server_duration_us":21731,"encode_duration_us":26025,"operation_id":"0x13abb","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10005568},{"operation_name":"upsert","last_dispatch_duration_us":150571,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10006554,"total_server_duration_us":207327,"encode_duration_us":22068,"operation_id":"0x13b90","timeout_ms":10000,"last_server_duration_us":22059,"total_duration_us":10002092},{"operation_name":"upsert","last_dispatch_duration_us":71705,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9974320,"total_server_duration_us":233097,"encode_duration_us":18850,"operation_id":"0x13ba7","timeout_ms":10000,"last_server_duration_us":24154,"total_duration_us":10000877}],"total_count":25}} INFO [2022-03-13 17:19:38,721] jepsen worker 7 - jepsen.util 637 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,722] jepsen worker 7 - jepsen.util 637 :invoke :write [2 49] WARN [2022-03-13 17:19:38,725] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17836071,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18732166,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9332508,"encode_duration_us":33603,"operation_id":"0x134e6","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":10008788}],"total_count":1}} INFO [2022-03-13 17:19:38,740] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,741] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:19:38,745] jepsen worker 76 - jepsen.util 796 :fail :write [25 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,746] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:38,761] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,761] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:38,764] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,765] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:38,774] jepsen worker 89 - jepsen.util 2789 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,774] jepsen worker 4 - jepsen.util 2614 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,774] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,775] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 22] INFO [2022-03-13 17:19:38,775] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:38,776] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:19:38,784] jepsen worker 25 - jepsen.util 565 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,790] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] WARN [2022-03-13 17:19:38,798] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12488334,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":12488334,"total_server_duration_us":32,"operation_id":"0x13a86","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":12489968},{"operation_name":"get","last_dispatch_duration_us":11286324,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11286324,"total_server_duration_us":19,"operation_id":"0x13b5e","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":11288876},{"operation_name":"upsert","last_dispatch_duration_us":8061262,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8061262,"total_server_duration_us":19887,"encode_duration_us":31444,"operation_id":"0x13c3a","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10056390},{"operation_name":"upsert","last_dispatch_duration_us":10602860,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":10602860,"total_server_duration_us":19262,"encode_duration_us":33067,"operation_id":"0x13bd1","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10009172},{"operation_name":"upsert","last_dispatch_duration_us":76023,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":6645050,"total_server_duration_us":114665,"encode_duration_us":26906,"operation_id":"0x13e81","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10000634},{"operation_name":"get","last_dispatch_duration_us":9133263,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9133263,"total_server_duration_us":43,"operation_id":"0x13ca3","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":9135121},{"operation_name":"upsert","last_dispatch_duration_us":9085213,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9085213,"total_server_duration_us":9059667,"encode_duration_us":19884,"operation_id":"0x13b91","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":9091134},{"operation_name":"upsert","last_dispatch_duration_us":9064870,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9064870,"total_server_duration_us":9032060,"encode_duration_us":28211,"operation_id":"0x13b84","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":9074839},{"operation_name":"upsert","last_dispatch_duration_us":2630429,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2630429,"total_server_duration_us":105184,"encode_duration_us":18058,"operation_id":"0x13fe7","timeout_ms":10000,"last_server_duration_us":105184,"total_duration_us":2635540},{"operation_name":"upsert","last_dispatch_duration_us":2602703,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2602703,"total_server_duration_us":492211,"encode_duration_us":31421,"operation_id":"0x13f9f","timeout_ms":10000,"last_server_duration_us":492211,"total_duration_us":2607070}],"total_count":58}} WARN [2022-03-13 17:19:38,802] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":21609622,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":21609622,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9058604,"encode_duration_us":29145,"operation_id":"0x13428","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10007534},{"operation_name":"upsert","last_dispatch_duration_us":9328361,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10292913,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9215771,"encode_duration_us":41415,"operation_id":"0x13aea","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10010576}],"total_count":2}} WARN [2022-03-13 17:19:38,807] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":36367,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1613106,"total_server_duration_us":616637,"encode_duration_us":28784,"operation_id":"0x13b98","timeout_ms":10000,"last_server_duration_us":21243,"total_duration_us":10009850},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":17736,"operation_id":"0x13dbf","timeout_ms":10000,"total_duration_us":10008661},{"operation_name":"upsert","last_dispatch_duration_us":46698,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":1064308,"total_server_duration_us":591817,"encode_duration_us":18093,"operation_id":"0x13c2b","timeout_ms":10000,"last_server_duration_us":22141,"total_duration_us":10008404},{"operation_name":"upsert","last_dispatch_duration_us":45780,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":870980,"total_server_duration_us":523444,"encode_duration_us":18484,"operation_id":"0x13ca4","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10004482},{"operation_name":"upsert","last_dispatch_duration_us":8526409,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8826664,"total_server_duration_us":148320,"encode_duration_us":34920,"operation_id":"0x13ae5","timeout_ms":10000,"last_server_duration_us":21649,"total_duration_us":10003951},{"operation_name":"upsert","last_dispatch_duration_us":8515099,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8854645,"total_server_duration_us":163201,"encode_duration_us":18870,"operation_id":"0x13ae3","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10001538},{"operation_name":"upsert","last_dispatch_duration_us":1862807,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1862807,"total_server_duration_us":161835,"encode_duration_us":17782,"operation_id":"0x14028","timeout_ms":10000,"last_server_duration_us":161835,"total_duration_us":1866924},{"operation_name":"upsert","last_dispatch_duration_us":1705878,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1705878,"total_server_duration_us":55790,"encode_duration_us":27243,"operation_id":"0x13fbe","timeout_ms":10000,"last_server_duration_us":55790,"total_duration_us":1711259},{"operation_name":"upsert","last_dispatch_duration_us":1641458,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1641458,"total_server_duration_us":50535,"encode_duration_us":18392,"operation_id":"0x13fe6","timeout_ms":10000,"last_server_duration_us":50535,"total_duration_us":1646039},{"operation_name":"upsert","last_dispatch_duration_us":76327,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000006/000000005DE126B4","last_local_socket":"172.23.104.79:33480","total_dispatch_duration_us":955531,"total_server_duration_us":199554,"encode_duration_us":26640,"operation_id":"0x144cd","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":1167004}],"total_count":31}} INFO [2022-03-13 17:19:38,832] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,832] jepsen worker 23 - jepsen.util 563 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,832] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,833] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:38,834] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:38,834] jepsen worker 77 - jepsen.util 707 :invoke :write [25 10] INFO [2022-03-13 17:19:38,854] jepsen worker 26 - jepsen.util 566 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,855] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,855] jepsen worker 26 - jepsen.util 566 :invoke :write [8 6] INFO [2022-03-13 17:19:38,856] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:38,857] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,857] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 38] INFO [2022-03-13 17:19:38,859] jepsen worker 57 - jepsen.util 4737 :ok :read [19 34] INFO [2022-03-13 17:19:38,860] jepsen worker 57 - jepsen.util 4737 :invoke :write [19 8] INFO [2022-03-13 17:19:38,875] jepsen worker 83 - jepsen.util 4403 :ok :read [27 4] INFO [2022-03-13 17:19:38,875] jepsen worker 71 - jepsen.util 611 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,875] jepsen worker 83 - jepsen.util 4403 :invoke :read [27 nil] INFO [2022-03-13 17:19:38,876] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] WARN [2022-03-13 17:19:38,899] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17418846,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17418846,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9042674,"encode_duration_us":18175,"operation_id":"0x13721","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10004882}],"total_count":1}} INFO [2022-03-13 17:19:38,913] jepsen worker 24 - jepsen.util 654 :fail :write [8 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,913] jepsen worker 24 - jepsen.util 654 :invoke :write [8 28] WARN [2022-03-13 17:19:38,925] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","encode_duration_us":20869,"operation_id":"0x13c72","timeout_ms":10000,"total_duration_us":10009965},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":27738,"operation_id":"0x13b73","timeout_ms":10000,"total_duration_us":10008040},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":20670,"operation_id":"0x13bdb","timeout_ms":10000,"total_duration_us":10007002},{"operation_name":"upsert","last_dispatch_duration_us":51256,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":2038959,"total_server_duration_us":509641,"encode_duration_us":21015,"operation_id":"0x13d69","timeout_ms":10000,"last_server_duration_us":29045,"total_duration_us":10005623},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":19164,"operation_id":"0x13aa0","timeout_ms":10000,"total_duration_us":10005233},{"operation_name":"upsert","last_dispatch_duration_us":2044813,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2044813,"total_server_duration_us":120142,"encode_duration_us":21586,"operation_id":"0x13fe3","timeout_ms":10000,"last_server_duration_us":120142,"total_duration_us":4155244},{"operation_name":"upsert","last_dispatch_duration_us":3138762,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3138762,"total_server_duration_us":86328,"encode_duration_us":20140,"operation_id":"0x13fe0","timeout_ms":10000,"last_server_duration_us":86328,"total_duration_us":3145197},{"operation_name":"upsert","last_dispatch_duration_us":1463959,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1463959,"total_server_duration_us":79545,"encode_duration_us":18306,"operation_id":"0x143de","timeout_ms":10000,"last_server_duration_us":79545,"total_duration_us":3062506},{"operation_name":"upsert","last_dispatch_duration_us":2837726,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2837726,"total_server_duration_us":253514,"encode_duration_us":17496,"operation_id":"0x13fdf","timeout_ms":10000,"last_server_duration_us":253514,"total_duration_us":2843236},{"operation_name":"upsert","last_dispatch_duration_us":2705074,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2705074,"total_server_duration_us":218622,"encode_duration_us":22793,"operation_id":"0x13fb8","timeout_ms":10000,"last_server_duration_us":218622,"total_duration_us":2723676}],"total_count":53}} INFO [2022-03-13 17:19:38,931] jepsen worker 38 - jepsen.util 2828 :fail :write [12 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,932] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:19:38,938] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,938] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:38,946] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,946] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:19:38,966] jepsen worker 75 - jepsen.util 525 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,968] jepsen worker 75 - jepsen.util 525 :invoke :write [25 21] INFO [2022-03-13 17:19:38,972] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,972] jepsen worker 12 - jepsen.util 912 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,972] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:38,972] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,974] jepsen worker 12 - jepsen.util 912 :invoke :write [4 27] INFO [2022-03-13 17:19:38,975] jepsen worker 62 - jepsen.util 512 :invoke :write [20 47] INFO [2022-03-13 17:19:38,976] jepsen worker 70 - jepsen.util 880 :invoke :write [23 42] INFO [2022-03-13 17:19:38,977] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:38,988] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,988] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:38,989] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:38,989] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:39,000] jepsen worker 83 - jepsen.util 4403 :ok :read [27 4] INFO [2022-03-13 17:19:39,000] jepsen worker 83 - jepsen.util 4403 :invoke :write [27 45] WARN [2022-03-13 17:19:39,008] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16352448,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17403789,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9212985,"encode_duration_us":18225,"operation_id":"0x1350d","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10001794},{"operation_name":"upsert","last_dispatch_duration_us":16365433,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17422946,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9199024,"encode_duration_us":24060,"operation_id":"0x13520","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10009947},{"operation_name":"upsert","last_dispatch_duration_us":16319742,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17685659,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9216323,"encode_duration_us":27378,"operation_id":"0x134b6","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10002341}],"total_count":3}} INFO [2022-03-13 17:19:39,055] jepsen worker 51 - jepsen.util 771 :fail :write [17 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,056] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,056] jepsen worker 51 - jepsen.util 771 :invoke :write [17 23] INFO [2022-03-13 17:19:39,057] jepsen worker 80 - jepsen.util 800 :invoke :write [26 25] INFO [2022-03-13 17:19:39,059] jepsen worker 89 - jepsen.util 2789 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,060] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,069] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,069] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,076] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,076] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,084] jepsen worker 61 - jepsen.util 781 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,086] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,086] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,086] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,087] jepsen worker 61 - jepsen.util 781 :invoke :write [20 33] INFO [2022-03-13 17:19:39,087] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:39,088] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,088] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:39,089] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,089] jepsen worker 23 - jepsen.util 563 :invoke :write [7 0] INFO [2022-03-13 17:19:39,099] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:39,099] jepsen worker nemesis - couchbase.util Rebalance check count: 685 INFO [2022-03-13 17:19:39,099] jepsen worker nemesis - couchbase.util Rebalance stuck count: 557 INFO [2022-03-13 17:19:39,122] jepsen worker 35 - jepsen.util 935 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,122] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,122] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,122] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:39,123] jepsen worker 33 - jepsen.util 753 :invoke :write [11 34] INFO [2022-03-13 17:19:39,123] jepsen worker 21 - jepsen.util 831 :invoke :write [7 21] INFO [2022-03-13 17:19:39,125] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,125] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:19:39,145] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,146] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,149] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,149] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,150] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,151] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,152] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,152] jepsen worker 37 - jepsen.util 2467 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,153] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,153] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,153] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,154] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,154] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 17] INFO [2022-03-13 17:19:39,155] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,156] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,157] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:19:39,157] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,158] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 46] INFO [2022-03-13 17:19:39,163] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,163] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,165] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,165] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,165] jepsen worker 6 - jepsen.util 636 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,166] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:39,168] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:39,168] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,168] jepsen worker 13 - jepsen.util 733 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,168] jepsen worker 52 - jepsen.util 682 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,169] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,169] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:39,170] jepsen worker 52 - jepsen.util 682 :invoke :write [17 9] INFO [2022-03-13 17:19:39,170] jepsen worker 76 - jepsen.util 796 :invoke :write [25 9] INFO [2022-03-13 17:19:39,171] jepsen worker 60 - jepsen.util 690 :invoke :write [20 36] INFO [2022-03-13 17:19:39,171] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:39,178] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,185] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,185] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,186] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,186] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:19:39,186] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,187] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,187] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:39,191] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,191] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,191] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,192] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,193] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,194] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,194] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,196] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,196] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,197] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,197] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,198] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:19:39,198] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,199] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,200] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,201] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,202] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,203] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:39,216] jepsen worker 24 - jepsen.util 654 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,216] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,216] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,217] jepsen worker 24 - jepsen.util 654 :invoke :write [8 38] INFO [2022-03-13 17:19:39,220] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:39,221] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:39,251] jepsen worker 84 - jepsen.util 5034 :info :write [28 42] :RequestTimeoutException INFO [2022-03-13 17:19:39,251] jepsen worker 2 - jepsen.util 5042 :info :write [0 39] :RequestTimeoutException INFO [2022-03-13 17:19:39,252] jepsen worker 2 - jepsen.util 5132 :invoke :write [0 1] INFO [2022-03-13 17:19:39,252] jepsen worker 84 - jepsen.util 5124 :invoke :write [28 28] INFO [2022-03-13 17:19:39,256] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,256] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,258] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,259] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,270] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,271] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:19:39,280] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,281] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,292] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,293] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:19:39,296] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,296] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,297] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:39,297] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 31] INFO [2022-03-13 17:19:39,301] jepsen worker 77 - jepsen.util 707 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,302] jepsen worker 77 - jepsen.util 707 :invoke :write [25 26] INFO [2022-03-13 17:19:39,307] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,307] jepsen worker 53 - jepsen.util 593 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,308] jepsen worker 22 - jepsen.util 652 :invoke :write [7 6] INFO [2022-03-13 17:19:39,308] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:39,321] jepsen worker 51 - jepsen.util 771 :fail :write [17 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,322] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:39,358] jepsen worker 88 - jepsen.util 2428 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,358] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 3] INFO [2022-03-13 17:19:39,368] jepsen worker 75 - jepsen.util 525 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,368] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,369] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:39,370] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:39,396] jepsen worker 80 - jepsen.util 800 :fail :write [26 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,396] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,397] jepsen worker 80 - jepsen.util 800 :invoke :write [26 9] INFO [2022-03-13 17:19:39,397] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:39,403] jepsen worker 8 - jepsen.util 728 :fail :write [2 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,403] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:39,412] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,412] jepsen worker 38 - jepsen.util 2828 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,412] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 47] INFO [2022-03-13 17:19:39,413] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,418] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,419] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,420] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,421] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,422] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,423] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:39,425] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,426] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:19:39,473] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,474] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,475] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,475] jepsen worker 61 - jepsen.util 781 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,475] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,476] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:39,479] jepsen worker 33 - jepsen.util 753 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,480] jepsen worker 33 - jepsen.util 753 :invoke :write [11 6] INFO [2022-03-13 17:19:39,485] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,486] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,503] jepsen worker 26 - jepsen.util 566 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,504] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:39,504] jepsen worker 26 - jepsen.util 566 :invoke :write [8 5] INFO [2022-03-13 17:19:39,505] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:39,532] jepsen worker 89 - jepsen.util 2789 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,533] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,548] jepsen worker 37 - jepsen.util 2467 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,549] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 8] INFO [2022-03-13 17:19:39,555] jepsen worker 44 - jepsen.util 764 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,556] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:39,562] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,562] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,562] jepsen worker 76 - jepsen.util 796 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,563] jepsen worker 13 - jepsen.util 733 :invoke :write [4 27] INFO [2022-03-13 17:19:39,564] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:39,564] jepsen worker 75 - jepsen.util 525 :invoke :write [25 31] INFO [2022-03-13 17:19:39,567] jepsen worker 21 - jepsen.util 831 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,567] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,568] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,568] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:39,570] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,570] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:19:39,570] jepsen worker 34 - jepsen.util 934 :invoke :write [11 35] INFO [2022-03-13 17:19:39,578] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 42] INFO [2022-03-13 17:19:39,578] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,579] jepsen worker 76 - jepsen.util 796 :invoke :write [25 22] INFO [2022-03-13 17:19:39,616] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,617] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,618] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,618] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:19:39,625] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,625] jepsen worker 23 - jepsen.util 563 :fail :write [7 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,626] jepsen worker 78 - jepsen.util 708 :invoke :write [26 21] INFO [2022-03-13 17:19:39,627] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:39,629] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,629] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,630] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,630] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,631] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:19:39,631] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:19:39,633] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,634] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:19:39,640] jepsen worker 86 - jepsen.util 4586 :ok :read [28 44] INFO [2022-03-13 17:19:39,641] jepsen worker 86 - jepsen.util 4586 :invoke :write [28 19] INFO [2022-03-13 17:19:39,673] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,673] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,674] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,675] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:19:39,678] jepsen worker 67 - jepsen.util 4117 :ok :read [22 30] INFO [2022-03-13 17:19:39,679] jepsen worker 67 - jepsen.util 4117 :invoke :write [22 46] INFO [2022-03-13 17:19:39,706] jepsen worker 88 - jepsen.util 2428 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,707] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 4] INFO [2022-03-13 17:19:39,731] jepsen worker 60 - jepsen.util 690 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,732] jepsen worker 60 - jepsen.util 690 :invoke :write [20 18] INFO [2022-03-13 17:19:39,751] jepsen worker 24 - jepsen.util 654 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,752] jepsen worker 24 - jepsen.util 654 :invoke :write [8 11] INFO [2022-03-13 17:19:39,770] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,771] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,771] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,771] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,772] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:39,772] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:39,780] jepsen worker 3 - jepsen.util 2253 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,780] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,780] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,781] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,843] jepsen worker 7 - jepsen.util 637 :fail :write [2 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,843] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:39,844] jepsen worker 42 - jepsen.util 402 :invoke :write [14 0] INFO [2022-03-13 17:19:39,844] jepsen worker 7 - jepsen.util 637 :invoke :write [2 38] INFO [2022-03-13 17:19:39,868] jepsen worker 37 - jepsen.util 2467 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,869] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 16] INFO [2022-03-13 17:19:39,886] jepsen worker 75 - jepsen.util 525 :fail :write [25 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,886] jepsen worker 75 - jepsen.util 525 :invoke :write [25 26] INFO [2022-03-13 17:19:39,894] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,894] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,895] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,895] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 6] INFO [2022-03-13 17:19:39,896] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,896] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,896] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,897] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,897] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 39] INFO [2022-03-13 17:19:39,897] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,898] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:19:39,898] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,899] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:39,900] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:19:39,904] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,905] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:39,921] jepsen worker 79 - jepsen.util 1159 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,922] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:39,922] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,923] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,923] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,924] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:19:39,925] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:39,926] jepsen worker 25 - jepsen.util 565 :invoke :write [8 10] INFO [2022-03-13 17:19:39,962] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,962] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,963] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,963] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:39,964] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,965] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,965] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,966] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:39,967] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,968] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,970] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,972] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:39,977] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,977] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,977] jepsen worker 52 - jepsen.util 682 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:39,978] jepsen worker 52 - jepsen.util 682 :invoke :write [17 30] INFO [2022-03-13 17:19:39,979] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:19:39,980] jepsen worker 53 - jepsen.util 593 :invoke :write [17 26] INFO [2022-03-13 17:19:40,004] jepsen worker 80 - jepsen.util 800 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,005] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:40,010] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,010] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,011] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:40,012] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:40,017] jepsen worker 89 - jepsen.util 2789 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,018] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:19:40,031] jepsen worker 70 - jepsen.util 880 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,031] jepsen worker 70 - jepsen.util 880 :invoke :write [23 5] INFO [2022-03-13 17:19:40,053] jepsen worker 13 - jepsen.util 733 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,054] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,054] jepsen worker 13 - jepsen.util 733 :invoke :write [4 43] INFO [2022-03-13 17:19:40,057] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:40,073] jepsen worker 88 - jepsen.util 2428 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,074] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,105] jepsen worker 38 - jepsen.util 2828 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,106] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:19:40,106] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:40,106] jepsen worker nemesis - couchbase.util Rebalance check count: 686 INFO [2022-03-13 17:19:40,106] jepsen worker nemesis - couchbase.util Rebalance stuck count: 558 INFO [2022-03-13 17:19:40,112] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,112] jepsen worker 33 - jepsen.util 753 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,113] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:40,114] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:40,151] jepsen worker 34 - jepsen.util 934 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,152] jepsen worker 34 - jepsen.util 934 :invoke :write [11 7] INFO [2022-03-13 17:19:40,166] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,166] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 18] INFO [2022-03-13 17:19:40,168] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,190] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,207] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,210] jepsen worker 22 - jepsen.util 652 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,249] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 36] INFO [2022-03-13 17:19:40,249] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:40,211] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:40,215] jepsen worker 77 - jepsen.util 707 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,250] jepsen worker 43 - jepsen.util 673 :invoke :write [14 0] INFO [2022-03-13 17:19:40,250] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:40,251] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,254] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,215] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,215] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,254] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:40,254] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:40,232] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,234] jepsen worker 24 - jepsen.util 654 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,254] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:40,234] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,255] jepsen worker 24 - jepsen.util 654 :invoke :write [8 45] INFO [2022-03-13 17:19:40,249] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:40,255] jepsen worker 51 - jepsen.util 771 :invoke :write [17 39] INFO [2022-03-13 17:19:40,255] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:40,255] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:19:40,256] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,256] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,256] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:40,257] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:19:40,260] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:40,270] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,271] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,272] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,273] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:19:40,274] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 20] INFO [2022-03-13 17:19:40,274] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:40,319] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,319] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,320] jepsen worker 37 - jepsen.util 2467 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,320] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,321] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:40,322] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:19:40,324] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:40,325] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,325] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,325] jepsen worker 12 - jepsen.util 912 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,326] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,327] jepsen worker 11 - jepsen.util 4601 :ok :read [3 25] INFO [2022-03-13 17:19:40,327] jepsen worker 12 - jepsen.util 912 :invoke :write [4 28] INFO [2022-03-13 17:19:40,328] jepsen worker 8 - jepsen.util 728 :invoke :write [2 6] INFO [2022-03-13 17:19:40,328] jepsen worker 11 - jepsen.util 4601 :invoke :write [3 4] INFO [2022-03-13 17:19:40,329] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:40,330] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:40,330] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,331] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:40,331] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 10] INFO [2022-03-13 17:19:40,332] jepsen worker 27 - jepsen.util 4887 :ok :read [9 26] INFO [2022-03-13 17:19:40,333] jepsen worker 27 - jepsen.util 4887 :invoke :write [9 10] INFO [2022-03-13 17:19:40,334] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,335] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,347] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,348] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,349] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,350] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,352] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,352] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 15] INFO [2022-03-13 17:19:40,363] jepsen worker 78 - jepsen.util 708 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,363] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,363] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,363] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,363] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,364] jepsen worker 77 - jepsen.util 707 :invoke :write [25 4] INFO [2022-03-13 17:19:40,365] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:40,365] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:40,366] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:40,366] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,366] jepsen worker 61 - jepsen.util 781 :invoke :write [20 19] INFO [2022-03-13 17:19:40,367] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,367] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:40,367] jepsen worker 23 - jepsen.util 563 :invoke :write [7 21] INFO [2022-03-13 17:19:40,368] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,368] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 0] INFO [2022-03-13 17:19:40,373] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,373] jepsen worker 78 - jepsen.util 708 :invoke :write [26 31] INFO [2022-03-13 17:19:40,390] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,391] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:40,396] jepsen worker 62 - jepsen.util 512 :fail :write [20 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,397] jepsen worker 62 - jepsen.util 512 :invoke :write [20 20] INFO [2022-03-13 17:19:40,404] jepsen worker 89 - jepsen.util 2789 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,405] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 29] INFO [2022-03-13 17:19:40,409] jepsen worker 76 - jepsen.util 796 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,409] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,410] jepsen worker 76 - jepsen.util 796 :invoke :write [25 23] INFO [2022-03-13 17:19:40,410] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] INFO [2022-03-13 17:19:40,429] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,430] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:19:40,488] jepsen worker 4 - jepsen.util 2614 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,489] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:19:40,493] jepsen worker 38 - jepsen.util 2828 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,493] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 33] INFO [2022-03-13 17:19:40,542] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,542] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,543] jepsen worker 26 - jepsen.util 566 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,543] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:40,544] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:19:40,544] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:40,558] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,558] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,559] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:19:40,560] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 46] INFO [2022-03-13 17:19:40,574] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,574] jepsen worker 25 - jepsen.util 565 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,574] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,575] jepsen worker 21 - jepsen.util 831 :invoke :write [7 1] INFO [2022-03-13 17:19:40,575] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:40,576] jepsen worker 33 - jepsen.util 753 :invoke :write [11 20] INFO [2022-03-13 17:19:40,583] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,584] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:19:40,603] jepsen worker 89 - jepsen.util 2789 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,604] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 37] INFO [2022-03-13 17:19:40,611] jepsen worker 75 - jepsen.util 525 :fail :write [25 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,611] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:40,616] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,616] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:40,642] jepsen worker 37 - jepsen.util 2467 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,643] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 41] INFO [2022-03-13 17:19:40,685] jepsen worker 38 - jepsen.util 2828 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,685] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:19:40,686] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,687] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:19:40,699] jepsen worker 42 - jepsen.util 402 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,700] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:40,701] jepsen worker 61 - jepsen.util 781 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,701] jepsen worker 61 - jepsen.util 781 :invoke :write [20 23] INFO [2022-03-13 17:19:40,712] jepsen worker 54 - jepsen.util 4464 :ok :read [18 21] INFO [2022-03-13 17:19:40,713] jepsen worker 54 - jepsen.util 4464 :invoke :write [18 38] INFO [2022-03-13 17:19:40,777] jepsen worker 88 - jepsen.util 2428 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,778] jepsen worker 88 - jepsen.util 2428 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,782] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,783] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:40,791] jepsen worker 60 - jepsen.util 690 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,792] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:40,793] jepsen worker 89 - jepsen.util 2789 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,793] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,799] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,799] jepsen worker 34 - jepsen.util 934 :fail :write [11 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,799] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,800] jepsen worker 34 - jepsen.util 934 :invoke :write [11 19] INFO [2022-03-13 17:19:40,800] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:40,801] jepsen worker 80 - jepsen.util 800 :invoke :write [26 48] INFO [2022-03-13 17:19:40,849] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,849] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,849] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,849] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,850] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:40,850] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 12] INFO [2022-03-13 17:19:40,851] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,851] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:19:40,853] jepsen worker 18 - jepsen.util 4698 :info :write [6 19] :RequestTimeoutException INFO [2022-03-13 17:19:40,854] jepsen worker 18 - jepsen.util 4788 :invoke :read [6 nil] INFO [2022-03-13 17:19:40,859] jepsen worker 7 - jepsen.util 637 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,859] jepsen worker 77 - jepsen.util 707 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,860] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:40,860] jepsen worker 7 - jepsen.util 637 :invoke :write [2 36] INFO [2022-03-13 17:19:40,870] jepsen worker 13 - jepsen.util 733 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,871] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:40,887] jepsen worker 51 - jepsen.util 771 :fail :write [17 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,887] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:40,930] jepsen worker 56 - jepsen.util 4646 :ok :read [18 21] INFO [2022-03-13 17:19:40,930] jepsen worker 56 - jepsen.util 4646 :invoke :read [18 nil] INFO [2022-03-13 17:19:40,935] jepsen worker 89 - jepsen.util 2789 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,935] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 15] INFO [2022-03-13 17:19:40,939] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,940] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,939] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,940] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:40,941] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:40,941] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 1] INFO [2022-03-13 17:19:40,941] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,942] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:40,951] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,952] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,952] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:19:40,953] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:19:40,963] jepsen worker 53 - jepsen.util 593 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,964] jepsen worker 53 - jepsen.util 593 :invoke :write [17 14] INFO [2022-03-13 17:19:40,966] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,966] jepsen worker 87 - jepsen.util 2877 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,967] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:40,968] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:19:40,971] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,972] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:19:40,975] jepsen worker 47 - jepsen.util 4547 :info :write [15 3] :RequestTimeoutException INFO [2022-03-13 17:19:40,977] jepsen worker 47 - jepsen.util 4637 :invoke :read [15 nil] INFO [2022-03-13 17:19:40,984] jepsen worker 38 - jepsen.util 2828 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,985] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 2] INFO [2022-03-13 17:19:40,998] jepsen worker 24 - jepsen.util 654 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:40,999] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:41,050] jepsen worker 76 - jepsen.util 796 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,051] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,051] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,051] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,051] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,055] jepsen worker 13 - jepsen.util 733 :invoke :write [4 11] INFO [2022-03-13 17:19:41,056] jepsen worker 35 - jepsen.util 935 :invoke :write [11 35] INFO [2022-03-13 17:19:41,057] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:41,062] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:41,062] jepsen worker 76 - jepsen.util 796 :invoke :write [25 34] INFO [2022-03-13 17:19:41,071] jepsen worker 37 - jepsen.util 2467 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,071] jepsen worker 88 - jepsen.util 2428 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,072] jepsen worker 88 - jepsen.util 2428 :invoke :write [29 28] INFO [2022-03-13 17:19:41,072] jepsen worker 37 - jepsen.util 2467 :invoke :read [12 nil] INFO [2022-03-13 17:19:41,074] jepsen worker 79 - jepsen.util 1159 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,074] jepsen worker 70 - jepsen.util 880 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,074] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:41,074] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,074] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,075] jepsen worker 70 - jepsen.util 880 :invoke :write [23 10] INFO [2022-03-13 17:19:41,076] jepsen worker 44 - jepsen.util 764 :invoke :write [14 48] INFO [2022-03-13 17:19:41,078] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,078] jepsen worker 3 - jepsen.util 2253 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,079] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:41,079] jepsen worker 6 - jepsen.util 636 :invoke :write [2 39] INFO [2022-03-13 17:19:41,081] jepsen worker 89 - jepsen.util 2789 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,081] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,082] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 21] INFO [2022-03-13 17:19:41,109] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,110] jepsen worker 75 - jepsen.util 525 :invoke :write [25 5] INFO [2022-03-13 17:19:41,115] jepsen worker 52 - jepsen.util 682 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,116] jepsen worker 52 - jepsen.util 682 :invoke :write [17 16] INFO [2022-03-13 17:19:41,117] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:41,118] jepsen worker nemesis - couchbase.util Rebalance check count: 687 INFO [2022-03-13 17:19:41,118] jepsen worker nemesis - couchbase.util Rebalance stuck count: 559 INFO [2022-03-13 17:19:41,118] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,119] jepsen worker 22 - jepsen.util 652 :invoke :write [7 35] INFO [2022-03-13 17:19:41,123] jepsen worker 21 - jepsen.util 831 :fail :write [7 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,124] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:19:41,143] jepsen worker 38 - jepsen.util 2828 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,144] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 30] INFO [2022-03-13 17:19:41,151] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,151] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,198] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,199] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:19:41,205] jepsen worker 43 - jepsen.util 673 :fail :write [14 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,205] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,205] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,206] jepsen worker 33 - jepsen.util 753 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,206] jepsen worker 26 - jepsen.util 566 :invoke :write [8 44] INFO [2022-03-13 17:19:41,207] jepsen worker 43 - jepsen.util 673 :invoke :write [14 12] INFO [2022-03-13 17:19:41,208] jepsen worker 60 - jepsen.util 690 :invoke :write [20 45] INFO [2022-03-13 17:19:41,208] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:41,213] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,214] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:41,220] jepsen worker 89 - jepsen.util 2789 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,221] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 25] INFO [2022-03-13 17:19:41,229] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,229] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:41,254] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,254] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,255] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:19:41,256] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,270] jepsen worker 13 - jepsen.util 733 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,270] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:41,276] jepsen worker 39 - jepsen.util 5079 :info :write [13 49] :RequestTimeoutException INFO [2022-03-13 17:19:41,277] jepsen worker 8 - jepsen.util 728 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,277] jepsen worker 23 - jepsen.util 563 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,277] jepsen worker 39 - jepsen.util 5169 :invoke :write [13 13] INFO [2022-03-13 17:19:41,283] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:41,284] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:41,289] jepsen worker 34 - jepsen.util 934 :fail :write [11 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,289] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,290] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:41,291] jepsen worker 34 - jepsen.util 934 :invoke :write [11 44] INFO [2022-03-13 17:19:41,300] jepsen worker 66 - jepsen.util 4476 :info :write [22 21] :RequestTimeoutException INFO [2022-03-13 17:19:41,301] jepsen worker 1 - jepsen.util 4411 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:19:41,301] jepsen worker 66 - jepsen.util 4566 :invoke :write [22 46] INFO [2022-03-13 17:19:41,302] jepsen worker 1 - jepsen.util 4501 :invoke :write [0 44] INFO [2022-03-13 17:19:41,305] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,305] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,305] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:19:41,306] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,308] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,308] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,309] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:19:41,309] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,311] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,311] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:19:41,321] jepsen worker 38 - jepsen.util 2828 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,322] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 42] INFO [2022-03-13 17:19:41,351] jepsen worker 65 - jepsen.util 4925 :info :write [21 39] :RequestTimeoutException INFO [2022-03-13 17:19:41,352] jepsen worker 64 - jepsen.util 4654 :ok :read [21 :nil] INFO [2022-03-13 17:19:41,352] jepsen worker 63 - jepsen.util 4923 :ok :read [21 :nil] INFO [2022-03-13 17:19:41,352] jepsen worker 65 - jepsen.util 5015 :invoke :write [21 38] INFO [2022-03-13 17:19:41,352] jepsen worker 64 - jepsen.util 4654 :invoke :read [21 nil] INFO [2022-03-13 17:19:41,353] jepsen worker 63 - jepsen.util 4923 :invoke :read [21 nil] INFO [2022-03-13 17:19:41,370] jepsen worker 53 - jepsen.util 593 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,371] jepsen worker 80 - jepsen.util 800 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,371] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,371] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,372] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:41,372] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:41,372] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:41,373] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,411] jepsen worker 63 - jepsen.util 4923 :ok :read [21 :nil] INFO [2022-03-13 17:19:41,411] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,411] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,411] jepsen worker 63 - jepsen.util 4923 :invoke :write [21 30] INFO [2022-03-13 17:19:41,412] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,412] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:19:41,417] jepsen worker 76 - jepsen.util 796 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,417] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:41,423] jepsen worker 28 - jepsen.util 4798 :info :write [9 23] :RequestTimeoutException INFO [2022-03-13 17:19:41,424] jepsen worker 28 - jepsen.util 4888 :invoke :read [9 nil] INFO [2022-03-13 17:19:41,447] jepsen worker 35 - jepsen.util 935 :fail :write [11 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,447] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:41,462] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,462] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,462] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,463] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,464] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:41,464] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,464] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:19:41,465] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,466] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,467] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,475] jepsen worker 89 - jepsen.util 2789 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,476] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:41,490] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,490] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,492] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:41,493] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,493] jepsen worker 78 - jepsen.util 708 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,494] jepsen worker 12 - jepsen.util 912 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,494] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,495] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,495] jepsen worker 12 - jepsen.util 912 :invoke :write [4 41] INFO [2022-03-13 17:19:41,496] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,497] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:19:41,497] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:19:41,552] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,552] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,552] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,552] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,552] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:41,553] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:41,554] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:41,554] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:41,555] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,555] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,556] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:41,556] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,571] jepsen worker 22 - jepsen.util 652 :fail :write [7 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,571] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:19:41,585] jepsen worker 59 - jepsen.util 4739 :ok :write [19 36] INFO [2022-03-13 17:19:41,586] jepsen worker 59 - jepsen.util 4739 :invoke :read [19 nil] INFO [2022-03-13 17:19:41,629] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,629] jepsen worker 38 - jepsen.util 2828 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,630] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:41,630] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:41,631] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,632] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,632] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:19:41,632] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:41,633] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,634] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 44] INFO [2022-03-13 17:19:41,636] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,637] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,642] jepsen worker 75 - jepsen.util 525 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,643] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,643] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:19:41,644] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,644] jepsen worker 13 - jepsen.util 733 :invoke :write [4 41] INFO [2022-03-13 17:19:41,645] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:41,645] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,645] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:41,653] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,654] jepsen worker 71 - jepsen.util 611 :invoke :write [23 37] INFO [2022-03-13 17:19:41,654] jepsen worker 55 - jepsen.util 4285 :info :write [18 21] :RequestTimeoutException INFO [2022-03-13 17:19:41,655] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,655] jepsen worker 55 - jepsen.util 4375 :invoke :read [18 nil] INFO [2022-03-13 17:19:41,655] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,655] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,655] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,655] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,655] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:19:41,656] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,656] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,656] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,656] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,657] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:41,657] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:41,659] jepsen worker 33 - jepsen.util 753 :invoke :write [11 3] INFO [2022-03-13 17:19:41,661] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,661] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:41,666] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:41,666] jepsen worker 14 - jepsen.util 374 :invoke :write [4 26] INFO [2022-03-13 17:19:41,666] jepsen worker 34 - jepsen.util 934 :fail :write [11 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,667] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,668] jepsen worker 34 - jepsen.util 934 :invoke :write [11 0] INFO [2022-03-13 17:19:41,668] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:41,666] jepsen worker 24 - jepsen.util 654 :invoke :write [8 36] INFO [2022-03-13 17:19:41,680] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,681] jepsen worker 76 - jepsen.util 796 :invoke :write [25 44] INFO [2022-03-13 17:19:41,683] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,683] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,684] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:41,685] jepsen worker 21 - jepsen.util 831 :invoke :write [7 26] INFO [2022-03-13 17:19:41,694] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,694] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,694] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,695] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:19:41,695] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,696] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 44] INFO [2022-03-13 17:19:41,699] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,700] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:19:41,716] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,716] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,717] jepsen worker 80 - jepsen.util 800 :invoke :write [26 23] INFO [2022-03-13 17:19:41,718] jepsen worker 51 - jepsen.util 771 :invoke :write [17 37] INFO [2022-03-13 17:19:41,731] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,731] jepsen worker 62 - jepsen.util 512 :fail :write [20 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,732] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,732] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:41,731] jepsen worker 61 - jepsen.util 781 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,731] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,732] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,732] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:41,732] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,733] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:41,734] jepsen worker 61 - jepsen.util 781 :invoke :write [20 28] INFO [2022-03-13 17:19:41,735] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:41,736] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,736] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:41,737] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,737] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:41,738] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,739] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:19:41,747] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,748] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,748] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,748] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:41,748] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,749] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,749] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,749] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:41,750] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,750] jepsen worker 77 - jepsen.util 707 :invoke :write [25 8] INFO [2022-03-13 17:19:41,751] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,751] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,751] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:41,752] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,755] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,755] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,755] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,756] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 8] INFO [2022-03-13 17:19:41,757] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,757] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:19:41,758] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,759] jepsen worker 52 - jepsen.util 682 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,759] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:41,760] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:41,769] jepsen worker 38 - jepsen.util 2828 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,770] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:41,819] jepsen worker 7 - jepsen.util 637 :fail :write [2 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,820] jepsen worker 7 - jepsen.util 637 :invoke :write [2 8] INFO [2022-03-13 17:19:41,825] jepsen worker 59 - jepsen.util 4739 :ok :read [19 36] INFO [2022-03-13 17:19:41,825] jepsen worker 4 - jepsen.util 2614 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,825] jepsen worker 59 - jepsen.util 4739 :invoke :write [19 0] INFO [2022-03-13 17:19:41,826] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,831] jepsen worker 33 - jepsen.util 753 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,831] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,832] jepsen worker 35 - jepsen.util 935 :invoke :write [11 22] INFO [2022-03-13 17:19:41,832] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:41,835] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,835] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:41,839] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,840] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:41,858] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,858] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 32] INFO [2022-03-13 17:19:41,882] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,883] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:41,908] jepsen worker 60 - jepsen.util 690 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,908] jepsen worker 60 - jepsen.util 690 :invoke :write [20 34] INFO [2022-03-13 17:19:41,957] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,957] jepsen worker 25 - jepsen.util 565 :invoke :write [8 40] INFO [2022-03-13 17:19:41,979] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,979] jepsen worker 61 - jepsen.util 781 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,981] jepsen worker 78 - jepsen.util 708 :invoke :write [26 2] INFO [2022-03-13 17:19:41,981] jepsen worker 61 - jepsen.util 781 :invoke :write [20 38] INFO [2022-03-13 17:19:41,984] jepsen worker 13 - jepsen.util 733 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,985] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:41,988] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,989] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:41,995] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:41,995] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:19:42,002] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,002] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:42,010] jepsen worker 15 - jepsen.util 4515 :fail :write [5 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,010] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,012] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 9] INFO [2022-03-13 17:19:42,013] jepsen worker 15 - jepsen.util 4515 :invoke :write [5 22] INFO [2022-03-13 17:19:42,016] jepsen worker 46 - jepsen.util 4906 :info :write [15 22] :RequestTimeoutException INFO [2022-03-13 17:19:42,017] jepsen worker 46 - jepsen.util 4996 :invoke :read [15 nil] INFO [2022-03-13 17:19:42,046] jepsen worker 73 - jepsen.util 5023 :info :write [24 8] :RequestTimeoutException INFO [2022-03-13 17:19:42,047] jepsen worker 73 - jepsen.util 5113 :invoke :write [24 17] INFO [2022-03-13 17:19:42,048] jepsen worker 14 - jepsen.util 374 :fail :write [4 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,049] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:42,060] jepsen worker 46 - jepsen.util 4996 :ok :read [15 48] INFO [2022-03-13 17:19:42,061] jepsen worker 46 - jepsen.util 4996 :invoke :read [15 nil] INFO [2022-03-13 17:19:42,084] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,084] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,084] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:42,085] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:42,087] jepsen worker 17 - jepsen.util 4877 :fail :write [5 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,088] jepsen worker 17 - jepsen.util 4877 :invoke :write [5 35] INFO [2022-03-13 17:19:42,088] jepsen worker 46 - jepsen.util 4996 :ok :read [15 48] INFO [2022-03-13 17:19:42,089] jepsen worker 46 - jepsen.util 4996 :invoke :read [15 nil] INFO [2022-03-13 17:19:42,106] jepsen worker 44 - jepsen.util 764 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,107] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:42,121] jepsen worker 89 - jepsen.util 2789 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,122] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:42,122] jepsen worker 46 - jepsen.util 4996 :ok :read [15 48] INFO [2022-03-13 17:19:42,123] jepsen worker 46 - jepsen.util 4996 :invoke :write [15 10] INFO [2022-03-13 17:19:42,124] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:42,124] jepsen worker nemesis - couchbase.util Rebalance check count: 688 INFO [2022-03-13 17:19:42,124] jepsen worker nemesis - couchbase.util Rebalance stuck count: 560 INFO [2022-03-13 17:19:42,128] jepsen worker 26 - jepsen.util 566 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,130] jepsen worker 26 - jepsen.util 566 :invoke :write [8 49] INFO [2022-03-13 17:19:42,155] jepsen worker 38 - jepsen.util 2828 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,156] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 47] INFO [2022-03-13 17:19:42,156] jepsen worker 77 - jepsen.util 707 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,157] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:42,169] jepsen worker 34 - jepsen.util 934 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,170] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,170] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:42,170] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,171] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:19:42,171] jepsen worker 10 - jepsen.util 4960 :info :write [3 2] :RequestTimeoutException INFO [2022-03-13 17:19:42,171] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:42,172] jepsen worker 10 - jepsen.util 5050 :invoke :write [3 11] INFO [2022-03-13 17:19:42,195] jepsen worker 70 - jepsen.util 880 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,197] jepsen worker 70 - jepsen.util 880 :invoke :write [23 36] INFO [2022-03-13 17:19:42,216] jepsen worker 15 - jepsen.util 4515 :fail :write [5 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,216] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,217] jepsen worker 15 - jepsen.util 4515 :invoke :write [5 26] INFO [2022-03-13 17:19:42,218] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 41] INFO [2022-03-13 17:19:42,258] jepsen worker 20 - jepsen.util 4970 :info :write [6 29] :RequestTimeoutException INFO [2022-03-13 17:19:42,259] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,259] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,259] jepsen worker 20 - jepsen.util 5060 :invoke :write [6 10] INFO [2022-03-13 17:19:42,260] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:19:42,260] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:42,271] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,271] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:42,278] jepsen worker 38 - jepsen.util 2828 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,278] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,279] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 25] INFO [2022-03-13 17:19:42,279] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:19:42,330] jepsen worker 74 - jepsen.util 4574 :info :write [24 3] :RequestTimeoutException INFO [2022-03-13 17:19:42,331] jepsen worker 74 - jepsen.util 4664 :invoke :write [24 16] INFO [2022-03-13 17:19:42,338] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,339] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:19:42,353] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,353] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:42,353] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:19:42,354] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:19:42,564] jepsen worker 85 - jepsen.util 4405 :info :write [28 37] :RequestTimeoutException INFO [2022-03-13 17:19:42,564] jepsen worker 85 - jepsen.util 4495 :invoke :write [28 7] INFO [2022-03-13 17:19:42,617] jepsen worker 32 - jepsen.util 4982 :info :write [10 42] :RequestTimeoutException INFO [2022-03-13 17:19:42,617] jepsen worker 32 - jepsen.util 5072 :invoke :read [10 nil] INFO [2022-03-13 17:19:42,686] jepsen worker 40 - jepsen.util 4900 :info :write [13 0] :RequestTimeoutException INFO [2022-03-13 17:19:42,687] jepsen worker 40 - jepsen.util 4990 :invoke :read [13 nil] INFO [2022-03-13 17:19:42,708] jepsen worker 40 - jepsen.util 4990 :ok :read [13 39] INFO [2022-03-13 17:19:42,709] jepsen worker 40 - jepsen.util 4990 :invoke :read [13 nil] INFO [2022-03-13 17:19:42,731] jepsen worker 40 - jepsen.util 4990 :ok :read [13 39] INFO [2022-03-13 17:19:42,731] jepsen worker 40 - jepsen.util 4990 :invoke :write [13 32] INFO [2022-03-13 17:19:42,736] jepsen worker 58 - jepsen.util 4198 :info :write [19 36] :RequestTimeoutException INFO [2022-03-13 17:19:42,736] jepsen worker 58 - jepsen.util 4288 :invoke :write [19 15] INFO [2022-03-13 17:19:43,107] jepsen worker 72 - jepsen.util 4842 :info :write [24 6] :RequestTimeoutException INFO [2022-03-13 17:19:43,108] jepsen worker 72 - jepsen.util 4932 :invoke :read [24 nil] INFO [2022-03-13 17:19:43,129] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:43,129] jepsen worker nemesis - couchbase.util Rebalance check count: 689 INFO [2022-03-13 17:19:43,129] jepsen worker nemesis - couchbase.util Rebalance stuck count: 561 INFO [2022-03-13 17:19:43,170] jepsen worker 45 - jepsen.util 4995 :info :write [15 29] :RequestTimeoutException INFO [2022-03-13 17:19:43,171] jepsen worker 45 - jepsen.util 5085 :invoke :read [15 nil] INFO [2022-03-13 17:19:43,490] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,490] jepsen worker 17 - jepsen.util 4877 :fail :write [5 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,490] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,491] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:19:43,491] jepsen worker 17 - jepsen.util 4877 :invoke :read [5 nil] INFO [2022-03-13 17:19:43,492] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:43,575] jepsen worker 89 - jepsen.util 2789 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,575] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:43,581] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,583] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:43,609] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,609] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:43,648] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,649] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:43,651] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,651] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:43,672] jepsen worker 15 - jepsen.util 4515 :fail :write [5 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,672] jepsen worker 15 - jepsen.util 4515 :invoke :write [5 21] INFO [2022-03-13 17:19:43,688] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,688] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:19:43,753] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,754] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:19:43,756] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,757] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:43,759] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,759] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:19:43,764] jepsen worker 38 - jepsen.util 2828 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,765] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:43,767] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,767] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 23] INFO [2022-03-13 17:19:43,772] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,773] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:43,774] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,775] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 13] INFO [2022-03-13 17:19:43,826] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,826] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,827] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:43,828] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:43,836] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,837] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:43,842] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,843] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:43,872] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,872] jepsen worker 75 - jepsen.util 525 :invoke :write [25 23] INFO [2022-03-13 17:19:43,889] jepsen worker 89 - jepsen.util 2789 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,890] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 19] INFO [2022-03-13 17:19:43,898] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,898] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,898] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:43,899] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:19:43,920] jepsen worker 31 - jepsen.util 4891 :info :write [10 41] :RequestTimeoutException INFO [2022-03-13 17:19:43,921] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,921] jepsen worker 31 - jepsen.util 4981 :invoke :write [10 0] INFO [2022-03-13 17:19:43,921] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:43,924] jepsen worker 71 - jepsen.util 611 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,925] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:43,925] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,926] jepsen worker 69 - jepsen.util 789 :invoke :write [23 2] INFO [2022-03-13 17:19:43,945] jepsen worker 24 - jepsen.util 654 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,945] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,946] jepsen worker 24 - jepsen.util 654 :invoke :write [8 25] INFO [2022-03-13 17:19:43,950] jepsen worker 38 - jepsen.util 2828 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,956] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:43,956] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 34] INFO [2022-03-13 17:19:43,959] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,959] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:43,964] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:43,964] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:19:44,003] jepsen worker 89 - jepsen.util 2789 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,004] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 46] INFO [2022-03-13 17:19:44,011] jepsen worker 6 - jepsen.util 636 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,012] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:44,014] jepsen worker 21 - jepsen.util 831 :fail :write [7 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,015] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:44,039] jepsen worker 43 - jepsen.util 673 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,040] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,040] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,040] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,040] jepsen worker 43 - jepsen.util 673 :invoke :write [14 45] INFO [2022-03-13 17:19:44,041] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:19:44,041] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:19:44,041] jepsen worker 53 - jepsen.util 593 :invoke :write [17 20] INFO [2022-03-13 17:19:44,054] jepsen worker 80 - jepsen.util 800 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,054] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,055] jepsen worker 33 - jepsen.util 753 :invoke :write [11 47] INFO [2022-03-13 17:19:44,055] jepsen worker 80 - jepsen.util 800 :invoke :write [26 49] INFO [2022-03-13 17:19:44,068] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,069] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,071] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,072] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,074] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,075] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:44,078] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,079] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:19:44,099] jepsen worker 25 - jepsen.util 565 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,101] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:44,105] jepsen worker 38 - jepsen.util 2828 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,105] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 3] INFO [2022-03-13 17:19:44,120] jepsen worker 76 - jepsen.util 796 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,120] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,121] jepsen worker 13 - jepsen.util 733 :invoke :write [4 47] INFO [2022-03-13 17:19:44,121] jepsen worker 76 - jepsen.util 796 :invoke :write [25 23] INFO [2022-03-13 17:19:44,134] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:44,134] jepsen worker nemesis - couchbase.util Rebalance check count: 690 INFO [2022-03-13 17:19:44,134] jepsen worker nemesis - couchbase.util Rebalance stuck count: 562 INFO [2022-03-13 17:19:44,140] jepsen worker 51 - jepsen.util 771 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,140] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,140] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,141] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,144] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:44,145] jepsen worker 51 - jepsen.util 771 :invoke :write [17 44] INFO [2022-03-13 17:19:44,145] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,154] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:44,156] jepsen worker 79 - jepsen.util 1159 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,156] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:44,185] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,185] jepsen worker 35 - jepsen.util 935 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,185] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,186] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:44,187] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,187] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:19:44,216] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,216] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,220] jepsen worker 89 - jepsen.util 2789 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,220] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,250] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,250] jepsen worker 12 - jepsen.util 912 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,250] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:44,250] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:44,251] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:44,251] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:44,277] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,277] jepsen worker 38 - jepsen.util 2828 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,278] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,278] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 40] INFO [2022-03-13 17:19:44,279] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,279] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,281] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,281] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,282] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,283] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,284] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,285] jepsen worker 38 - jepsen.util 2828 :invoke :write [12 19] INFO [2022-03-13 17:19:44,292] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,293] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:44,301] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,302] jepsen worker 21 - jepsen.util 831 :invoke :write [7 44] INFO [2022-03-13 17:19:44,304] jepsen worker 24 - jepsen.util 654 :fail :write [8 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,305] jepsen worker 24 - jepsen.util 654 :invoke :write [8 47] INFO [2022-03-13 17:19:44,325] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,325] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,325] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,327] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:19:44,327] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,329] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,330] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:19:44,332] jepsen worker 69 - jepsen.util 789 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,333] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:19:44,340] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:44,340] jepsen worker 75 - jepsen.util 525 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,340] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,341] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,341] jepsen worker 75 - jepsen.util 525 :invoke :write [25 47] INFO [2022-03-13 17:19:44,341] jepsen worker 22 - jepsen.util 652 :invoke :write [7 25] INFO [2022-03-13 17:19:44,342] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:44,374] jepsen worker 89 - jepsen.util 2789 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,375] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,384] jepsen worker 78 - jepsen.util 708 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,384] jepsen worker 78 - jepsen.util 708 :invoke :write [26 45] INFO [2022-03-13 17:19:44,385] jepsen worker 80 - jepsen.util 800 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,386] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:44,415] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,415] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,417] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,417] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:44,417] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,418] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:19:44,431] jepsen worker 49 - jepsen.util 3559 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:19:44,432] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 0] INFO [2022-03-13 17:19:44,440] jepsen worker 7 - jepsen.util 637 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,440] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:44,440] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,441] jepsen worker 44 - jepsen.util 764 :invoke :write [14 40] INFO [2022-03-13 17:19:44,441] jepsen worker 7 - jepsen.util 637 :invoke :write [2 28] INFO [2022-03-13 17:19:44,443] jepsen worker 8 - jepsen.util 728 :invoke :write [2 2] INFO [2022-03-13 17:19:44,467] jepsen worker 33 - jepsen.util 753 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,467] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,467] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,467] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,467] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:44,468] jepsen worker 60 - jepsen.util 690 :fail :write [20 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,468] jepsen worker 38 - jepsen.util 2828 :fail :write [12 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,468] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,469] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:19:44,470] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,470] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,471] jepsen worker 60 - jepsen.util 690 :invoke :write [20 37] INFO [2022-03-13 17:19:44,472] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:19:44,472] jepsen worker 34 - jepsen.util 934 :invoke :write [11 28] INFO [2022-03-13 17:19:44,497] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,497] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:19:44,500] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,501] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,503] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,503] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,505] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,505] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 30] INFO [2022-03-13 17:19:44,512] jepsen worker 13 - jepsen.util 733 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,513] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:44,560] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,560] jepsen worker 49 - jepsen.util 3559 :fail :write [16 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,560] jepsen worker 49 - jepsen.util 3559 :invoke :write [16 9] INFO [2022-03-13 17:19:44,562] jepsen worker 89 - jepsen.util 2789 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,563] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,563] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,570] jepsen worker 81 - jepsen.util 5391 :info :write [27 9] :RequestTimeoutException INFO [2022-03-13 17:19:44,570] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,570] jepsen worker 51 - jepsen.util 771 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,571] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:44,572] jepsen worker 35 - jepsen.util 935 :invoke :write [11 34] INFO [2022-03-13 17:19:44,572] jepsen worker 81 - jepsen.util 5481 :invoke :read [27 nil] INFO [2022-03-13 17:19:44,581] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,582] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:19:44,587] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,588] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,592] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,592] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:19:44,612] jepsen worker 61 - jepsen.util 781 :fail :write [20 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,612] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,614] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,614] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,615] jepsen worker 77 - jepsen.util 707 :invoke :write [25 14] INFO [2022-03-13 17:19:44,616] jepsen worker 23 - jepsen.util 563 :invoke :write [7 22] INFO [2022-03-13 17:19:44,616] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:44,617] jepsen worker 61 - jepsen.util 781 :invoke :write [20 43] INFO [2022-03-13 17:19:44,626] jepsen worker 26 - jepsen.util 566 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,626] jepsen worker 26 - jepsen.util 566 :invoke :write [8 9] INFO [2022-03-13 17:19:44,641] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,641] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 25] INFO [2022-03-13 17:19:44,671] jepsen worker 5 - jepsen.util 2705 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,672] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,679] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,679] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,681] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,681] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:44,683] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,684] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:19:44,687] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,688] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:19:44,689] jepsen worker 21 - jepsen.util 831 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,690] jepsen worker 21 - jepsen.util 831 :invoke :write [7 43] INFO [2022-03-13 17:19:44,691] jepsen worker 76 - jepsen.util 796 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,692] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:44,697] jepsen worker 70 - jepsen.util 880 :fail :write [23 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,697] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,699] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,699] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:44,699] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,699] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,700] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:19:44,701] jepsen worker 62 - jepsen.util 512 :invoke :write [20 5] INFO [2022-03-13 17:19:44,701] jepsen worker 12 - jepsen.util 912 :invoke :write [4 6] INFO [2022-03-13 17:19:44,701] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:44,702] jepsen worker 70 - jepsen.util 880 :invoke :write [23 8] INFO [2022-03-13 17:19:44,704] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:44,756] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,757] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:19:44,762] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,763] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 1] INFO [2022-03-13 17:19:44,773] jepsen worker 24 - jepsen.util 654 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,773] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,774] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,774] jepsen worker 24 - jepsen.util 654 :invoke :write [8 17] INFO [2022-03-13 17:19:44,775] jepsen worker 80 - jepsen.util 800 :invoke :write [26 35] INFO [2022-03-13 17:19:44,776] jepsen worker 33 - jepsen.util 753 :invoke :write [11 24] INFO [2022-03-13 17:19:44,787] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,787] jepsen worker 78 - jepsen.util 708 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,787] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:44,788] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:44,819] jepsen worker 36 - jepsen.util 2466 :fail :write [12 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,820] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:19:44,834] jepsen worker 44 - jepsen.util 764 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,834] jepsen worker 43 - jepsen.util 673 :fail :write [14 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,835] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:44,835] jepsen worker 43 - jepsen.util 673 :invoke :write [14 9] INFO [2022-03-13 17:19:44,841] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,842] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:19:44,845] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,845] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:44,874] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,874] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:44,884] jepsen worker 75 - jepsen.util 525 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,884] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,884] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,885] jepsen worker 75 - jepsen.util 525 :invoke :write [25 18] INFO [2022-03-13 17:19:44,885] jepsen worker 13 - jepsen.util 733 :invoke :write [4 23] INFO [2022-03-13 17:19:44,885] jepsen worker 76 - jepsen.util 796 :invoke :write [25 21] INFO [2022-03-13 17:19:44,913] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,913] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,917] jepsen worker 23 - jepsen.util 563 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,917] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,917] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:44,918] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:44,920] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,921] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:44,938] jepsen worker 3 - jepsen.util 2253 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,938] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:19:44,949] jepsen worker 34 - jepsen.util 934 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,949] jepsen worker 34 - jepsen.util 934 :invoke :write [11 36] INFO [2022-03-13 17:19:44,979] jepsen worker 53 - jepsen.util 593 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,980] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,981] jepsen worker 53 - jepsen.util 593 :invoke :write [17 18] INFO [2022-03-13 17:19:44,982] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:44,988] jepsen worker 8 - jepsen.util 728 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,989] jepsen worker 8 - jepsen.util 728 :invoke :write [2 23] INFO [2022-03-13 17:19:44,990] jepsen worker 68 - jepsen.util 4838 :ok :read [22 30] INFO [2022-03-13 17:19:44,991] jepsen worker 68 - jepsen.util 4838 :invoke :read [22 nil] INFO [2022-03-13 17:19:44,995] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:44,996] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:44,999] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,000] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,003] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,004] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,007] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,008] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,014] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,015] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,016] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,017] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:19:45,052] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,053] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:19:45,090] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,091] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,098] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,098] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,099] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,099] jepsen worker 14 - jepsen.util 374 :invoke :write [4 38] INFO [2022-03-13 17:19:45,099] jepsen worker 51 - jepsen.util 771 :invoke :write [17 12] INFO [2022-03-13 17:19:45,100] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,108] jepsen worker 68 - jepsen.util 4838 :ok :read [22 30] INFO [2022-03-13 17:19:45,108] jepsen worker 68 - jepsen.util 4838 :invoke :write [22 40] INFO [2022-03-13 17:19:45,115] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,116] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,117] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,123] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:19:45,124] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:19:45,125] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:19:45,125] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,126] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:45,135] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,136] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 35] INFO [2022-03-13 17:19:45,137] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,138] jepsen worker 69 - jepsen.util 789 :invoke :write [23 16] INFO [2022-03-13 17:19:45,149] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:45,149] jepsen worker nemesis - couchbase.util Rebalance check count: 691 INFO [2022-03-13 17:19:45,149] jepsen worker nemesis - couchbase.util Rebalance stuck count: 563 INFO [2022-03-13 17:19:45,158] jepsen worker 22 - jepsen.util 652 :fail :write [7 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,159] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:45,163] jepsen worker 77 - jepsen.util 707 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,163] jepsen worker 77 - jepsen.util 707 :invoke :write [25 25] INFO [2022-03-13 17:19:45,193] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,194] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:19:45,196] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,196] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,197] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:19:45,197] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:45,250] jepsen worker 7 - jepsen.util 637 :fail :write [2 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,250] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:45,250] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,251] jepsen worker 6 - jepsen.util 636 :invoke :write [2 11] INFO [2022-03-13 17:19:45,252] jepsen worker 7 - jepsen.util 637 :invoke :write [2 16] INFO [2022-03-13 17:19:45,253] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:45,256] jepsen worker 3 - jepsen.util 2253 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,257] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,268] jepsen worker 35 - jepsen.util 935 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,269] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:45,284] jepsen worker 75 - jepsen.util 525 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,284] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:45,285] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,285] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:19:45,318] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,318] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:45,344] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,345] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,345] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,344] jepsen worker 61 - jepsen.util 781 :fail :write [20 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,346] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:45,347] jepsen worker 23 - jepsen.util 563 :invoke :write [7 10] INFO [2022-03-13 17:19:45,348] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 9] INFO [2022-03-13 17:19:45,350] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:45,381] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,381] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,381] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,382] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,383] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:19:45,383] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,383] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:19:45,384] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,385] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,386] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,387] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,388] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:45,389] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,389] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,393] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,393] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:19:45,401] jepsen worker 70 - jepsen.util 880 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,402] jepsen worker 21 - jepsen.util 831 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,402] jepsen worker 70 - jepsen.util 880 :invoke :write [23 22] INFO [2022-03-13 17:19:45,403] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:45,423] jepsen worker 0 - jepsen.util 4590 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:19:45,424] jepsen worker 0 - jepsen.util 4680 :invoke :write [0 28] INFO [2022-03-13 17:19:45,436] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,436] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:45,441] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,442] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,444] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,447] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:19:45,451] jepsen worker 60 - jepsen.util 690 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,451] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:45,488] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,489] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:45,494] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,495] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,497] jepsen worker 76 - jepsen.util 796 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,497] jepsen worker 76 - jepsen.util 796 :invoke :write [25 32] INFO [2022-03-13 17:19:45,536] jepsen worker 77 - jepsen.util 707 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,536] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,536] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,536] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:45,537] jepsen worker 78 - jepsen.util 708 :invoke :write [26 15] INFO [2022-03-13 17:19:45,538] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:45,549] jepsen worker 33 - jepsen.util 753 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,549] jepsen worker 33 - jepsen.util 753 :invoke :write [11 39] INFO [2022-03-13 17:19:45,571] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,572] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:19:45,579] jepsen worker 12 - jepsen.util 912 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,579] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,580] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:19:45,581] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:45,614] jepsen worker 26 - jepsen.util 566 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,615] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:45,639] jepsen worker 79 - jepsen.util 1159 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,639] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 26] INFO [2022-03-13 17:19:45,645] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,647] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:19:45,662] jepsen worker 24 - jepsen.util 654 :fail :write [8 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,666] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:45,677] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,677] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,677] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,678] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:19:45,726] jepsen worker 82 - jepsen.util 5032 :info :write [27 35] :RequestTimeoutException INFO [2022-03-13 17:19:45,727] jepsen worker 82 - jepsen.util 5122 :invoke :write [27 23] INFO [2022-03-13 17:19:45,730] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,731] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:19:45,740] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:45,740] jepsen worker 62 - jepsen.util 512 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,741] jepsen worker 44 - jepsen.util 764 :invoke :write [14 48] INFO [2022-03-13 17:19:45,741] jepsen worker 62 - jepsen.util 512 :invoke :write [20 10] INFO [2022-03-13 17:19:45,760] jepsen worker 13 - jepsen.util 733 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,760] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,761] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:45,761] jepsen worker 13 - jepsen.util 733 :invoke :write [4 9] INFO [2022-03-13 17:19:45,773] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,773] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,774] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:19:45,775] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:19:45,803] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,803] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:19:45,818] jepsen worker 80 - jepsen.util 800 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,819] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:45,840] jepsen worker 23 - jepsen.util 563 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,840] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,840] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,841] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:19:45,841] jepsen worker 77 - jepsen.util 707 :invoke :write [25 48] INFO [2022-03-13 17:19:45,842] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:45,849] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,849] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,861] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,861] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:19:45,869] jepsen worker 43 - jepsen.util 673 :fail :write [14 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,870] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:45,917] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,918] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,920] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,920] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,924] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,924] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,926] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,927] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:19:45,932] jepsen worker 8 - jepsen.util 728 :fail :write [2 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,932] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:45,933] jepsen worker 42 - jepsen.util 402 :invoke :write [14 33] INFO [2022-03-13 17:19:45,934] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:45,935] jepsen worker 14 - jepsen.util 374 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,936] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:45,936] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,936] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:45,947] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,948] jepsen worker 76 - jepsen.util 796 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,948] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,948] jepsen worker 52 - jepsen.util 682 :invoke :write [17 32] INFO [2022-03-13 17:19:45,949] jepsen worker 75 - jepsen.util 525 :invoke :write [25 20] INFO [2022-03-13 17:19:45,950] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:45,977] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,978] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,980] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,981] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,983] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,984] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,994] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,995] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:45,996] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:45,997] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:46,014] jepsen worker 51 - jepsen.util 771 :fail :write [17 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,014] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:46,033] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,033] jepsen worker 69 - jepsen.util 789 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,034] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:19:46,034] jepsen worker 71 - jepsen.util 611 :invoke :write [23 9] INFO [2022-03-13 17:19:46,042] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,042] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:19:46,077] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,077] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,077] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,078] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:46,078] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:19:46,078] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:19:46,086] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,086] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:19:46,120] jepsen worker 78 - jepsen.util 708 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,120] jepsen worker 53 - jepsen.util 593 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,120] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,120] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:46,120] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,121] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:46,121] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:19:46,121] jepsen worker 60 - jepsen.util 690 :invoke :write [20 15] INFO [2022-03-13 17:19:46,123] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,123] jepsen worker 26 - jepsen.util 566 :invoke :write [8 1] INFO [2022-03-13 17:19:46,125] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:46,126] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:46,142] jepsen worker 6 - jepsen.util 636 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,143] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:46,144] jepsen worker 33 - jepsen.util 753 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,144] jepsen worker 33 - jepsen.util 753 :invoke :write [11 20] INFO [2022-03-13 17:19:46,144] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,145] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,145] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:46,146] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,146] jepsen worker 24 - jepsen.util 654 :invoke :write [8 34] INFO [2022-03-13 17:19:46,146] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,146] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,146] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:46,146] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,146] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,147] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:46,147] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:46,147] jepsen worker 51 - jepsen.util 771 :invoke :write [17 36] INFO [2022-03-13 17:19:46,148] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:46,154] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:46,155] jepsen worker nemesis - couchbase.util Rebalance check count: 692 INFO [2022-03-13 17:19:46,155] jepsen worker nemesis - couchbase.util Rebalance stuck count: 564 INFO [2022-03-13 17:19:46,156] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,157] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,159] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,160] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,161] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,162] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,163] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,164] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:19:46,171] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,171] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,207] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,207] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,207] jepsen worker 34 - jepsen.util 934 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,207] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,207] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,208] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,208] jepsen worker 21 - jepsen.util 831 :invoke :write [7 45] INFO [2022-03-13 17:19:46,210] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:46,210] jepsen worker 14 - jepsen.util 374 :invoke :write [4 27] INFO [2022-03-13 17:19:46,211] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:46,212] jepsen worker 35 - jepsen.util 935 :invoke :write [11 1] INFO [2022-03-13 17:19:46,213] jepsen worker 34 - jepsen.util 934 :invoke :write [11 39] INFO [2022-03-13 17:19:46,218] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,219] jepsen worker 13 - jepsen.util 733 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,219] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:46,220] jepsen worker 13 - jepsen.util 733 :invoke :write [4 22] INFO [2022-03-13 17:19:46,229] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,229] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,235] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,236] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,237] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,238] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:19:46,239] jepsen worker 79 - jepsen.util 1159 :fail :write [26 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,239] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,240] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 6] INFO [2022-03-13 17:19:46,242] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:46,285] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,285] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,285] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,286] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:19:46,286] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,286] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:19:46,290] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,291] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:19:46,301] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,301] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:46,312] jepsen worker 24 - jepsen.util 654 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,312] jepsen worker 24 - jepsen.util 654 :invoke :write [8 45] INFO [2022-03-13 17:19:46,373] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,374] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 35] INFO [2022-03-13 17:19:46,387] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,388] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,395] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,395] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:19:46,434] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,435] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,480] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,481] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,483] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,483] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:19:46,495] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,496] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,525] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,525] jepsen worker 36 - jepsen.util 2466 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,525] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,526] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:46,526] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:19:46,527] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:19:46,529] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,529] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:46,530] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,531] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:46,532] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,532] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:46,533] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,534] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:46,535] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,535] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:46,536] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,537] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:19:46,562] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,563] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:46,614] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,614] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:19:46,620] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,621] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,622] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,622] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:19:46,635] jepsen worker 52 - jepsen.util 682 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,635] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,636] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:46,636] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:46,636] jepsen worker 53 - jepsen.util 593 :invoke :write [17 49] INFO [2022-03-13 17:19:46,637] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:46,670] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,670] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:19:46,674] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,674] jepsen worker 75 - jepsen.util 525 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,675] jepsen worker 76 - jepsen.util 796 :invoke :write [25 30] INFO [2022-03-13 17:19:46,675] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:46,680] jepsen worker 7 - jepsen.util 637 :fail :write [2 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,681] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:46,700] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,700] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,704] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,704] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,705] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,706] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,706] jepsen worker 51 - jepsen.util 771 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,707] jepsen worker 51 - jepsen.util 771 :invoke :write [17 7] INFO [2022-03-13 17:19:46,707] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,708] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:19:46,759] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,759] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,760] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:19:46,760] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:19:46,762] jepsen worker 77 - jepsen.util 707 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,763] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:46,767] jepsen worker 60 - jepsen.util 690 :fail :write [20 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,768] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:46,791] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,791] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:19:46,808] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,808] jepsen worker 33 - jepsen.util 753 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,809] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,809] jepsen worker 25 - jepsen.util 565 :invoke :write [8 36] INFO [2022-03-13 17:19:46,810] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:46,810] jepsen worker 80 - jepsen.util 800 :invoke :write [26 34] INFO [2022-03-13 17:19:46,811] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,811] jepsen worker 12 - jepsen.util 912 :invoke :write [4 42] INFO [2022-03-13 17:19:46,812] jepsen worker 70 - jepsen.util 880 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,813] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:46,818] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,818] jepsen worker 13 - jepsen.util 733 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,819] jepsen worker 13 - jepsen.util 733 :invoke :write [4 46] INFO [2022-03-13 17:19:46,820] jepsen worker 75 - jepsen.util 525 :invoke :write [25 16] INFO [2022-03-13 17:19:46,834] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,834] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,835] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:19:46,836] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:19:46,843] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,844] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:19:46,869] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,870] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:19:46,905] jepsen worker 14 - jepsen.util 374 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,905] jepsen worker 14 - jepsen.util 374 :invoke :write [4 46] INFO [2022-03-13 17:19:46,908] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,911] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:46,913] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,914] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:19:46,946] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,947] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,960] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,961] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:19:46,967] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,967] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,967] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,967] jepsen worker 23 - jepsen.util 563 :invoke :write [7 41] INFO [2022-03-13 17:19:46,968] jepsen worker 61 - jepsen.util 781 :invoke :write [20 35] INFO [2022-03-13 17:19:46,970] jepsen worker 78 - jepsen.util 708 :invoke :write [26 20] INFO [2022-03-13 17:19:46,972] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,973] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:46,974] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:46,975] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:47,012] jepsen worker 76 - jepsen.util 796 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,013] jepsen worker 76 - jepsen.util 796 :invoke :write [25 1] INFO [2022-03-13 17:19:47,020] jepsen worker 21 - jepsen.util 831 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,021] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:47,022] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,023] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,032] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,033] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:47,039] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,039] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:47,040] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,041] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:47,067] jepsen worker 44 - jepsen.util 764 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,068] jepsen worker 44 - jepsen.util 764 :invoke :write [14 4] INFO [2022-03-13 17:19:47,078] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,080] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,083] jepsen worker 26 - jepsen.util 566 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,084] jepsen worker 26 - jepsen.util 566 :invoke :write [8 20] INFO [2022-03-13 17:19:47,085] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,085] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,086] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:47,086] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:47,087] jepsen worker 43 - jepsen.util 673 :invoke :write [14 8] INFO [2022-03-13 17:19:47,088] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:47,099] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,099] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,099] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,099] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:19:47,100] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 2] INFO [2022-03-13 17:19:47,100] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:19:47,126] jepsen worker 13 - jepsen.util 733 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,126] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:47,137] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,138] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:19:47,139] jepsen worker 34 - jepsen.util 934 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,140] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:47,160] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:47,160] jepsen worker nemesis - couchbase.util Rebalance check count: 693 INFO [2022-03-13 17:19:47,160] jepsen worker nemesis - couchbase.util Rebalance stuck count: 565 INFO [2022-03-13 17:19:47,169] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,169] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,178] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,178] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,179] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,180] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:47,200] jepsen worker 79 - jepsen.util 1159 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,201] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 40] INFO [2022-03-13 17:19:47,201] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,201] jepsen worker 71 - jepsen.util 611 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,202] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:47,203] jepsen worker 77 - jepsen.util 707 :invoke :write [25 22] INFO [2022-03-13 17:19:47,212] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,213] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] WARN [2022-03-13 17:19:47,216] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17794329,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19343005,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9665299,"encode_duration_us":25677,"operation_id":"0x1357f","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10000818},{"operation_name":"upsert","last_dispatch_duration_us":9039516,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9523864,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9260948,"encode_duration_us":17371,"operation_id":"0x13feb","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10009539}],"total_count":2}} INFO [2022-03-13 17:19:47,220] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,221] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:47,236] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,238] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:19:47,241] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,241] jepsen worker 53 - jepsen.util 593 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,242] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:47,242] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:47,247] jepsen worker 24 - jepsen.util 654 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,248] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:47,281] jepsen worker 3 - jepsen.util 2253 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,281] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 20] INFO [2022-03-13 17:19:47,288] jepsen worker 62 - jepsen.util 512 :fail :write [20 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,289] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:47,292] jepsen worker 75 - jepsen.util 525 :fail :write [25 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,293] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:47,301] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,302] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] WARN [2022-03-13 17:19:47,317] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16415253,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16415253,"total_server_duration_us":6143,"operation_id":"0x13cbc","timeout_ms":30000,"last_server_duration_us":6143,"total_duration_us":16417532},{"operation_name":"upsert","last_dispatch_duration_us":52365,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8304243,"total_server_duration_us":323176,"encode_duration_us":21456,"operation_id":"0x140e8","timeout_ms":10000,"last_server_duration_us":33230,"total_duration_us":10010669},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":22761,"operation_id":"0x14180","timeout_ms":10000,"total_duration_us":10009964},{"operation_name":"upsert","last_dispatch_duration_us":9039516,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9523864,"total_server_duration_us":9260948,"encode_duration_us":17371,"operation_id":"0x13feb","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10009539},{"operation_name":"upsert","last_dispatch_duration_us":38648,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8150421,"total_server_duration_us":260245,"encode_duration_us":17726,"operation_id":"0x140ea","timeout_ms":10000,"last_server_duration_us":25888,"total_duration_us":10006350},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","encode_duration_us":18652,"operation_id":"0x142be","timeout_ms":10000,"total_duration_us":10004527},{"operation_name":"upsert","last_dispatch_duration_us":7451684,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8060369,"total_server_duration_us":315517,"encode_duration_us":21871,"operation_id":"0x13fee","timeout_ms":10000,"last_server_duration_us":19730,"total_duration_us":10002211},{"operation_name":"get","last_dispatch_duration_us":2831921,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":2831921,"total_server_duration_us":137185,"operation_id":"0x144a2","timeout_ms":30000,"last_server_duration_us":137185,"total_duration_us":2841146},{"operation_name":"upsert","last_dispatch_duration_us":2071961,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":2071961,"total_server_duration_us":20362,"encode_duration_us":37130,"operation_id":"0x14780","timeout_ms":10000,"last_server_duration_us":20362,"total_duration_us":2384302},{"operation_name":"upsert","last_dispatch_duration_us":770948,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":770948,"total_server_duration_us":47762,"encode_duration_us":27947,"operation_id":"0x14479","timeout_ms":10000,"last_server_duration_us":47762,"total_duration_us":2380345}],"total_count":78}} INFO [2022-03-13 17:19:47,346] jepsen worker 51 - jepsen.util 771 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,346] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,346] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,347] jepsen worker 51 - jepsen.util 771 :invoke :write [17 19] INFO [2022-03-13 17:19:47,347] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:47,348] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,349] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,349] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,350] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,351] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,352] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,352] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,354] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,355] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,361] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,361] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,362] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,362] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:47,383] jepsen worker 23 - jepsen.util 563 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,383] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,383] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,384] jepsen worker 23 - jepsen.util 563 :invoke :write [7 3] INFO [2022-03-13 17:19:47,384] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:19:47,384] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 7] INFO [2022-03-13 17:19:47,420] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,421] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,421] jepsen worker 22 - jepsen.util 652 :invoke :write [7 30] INFO [2022-03-13 17:19:47,421] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,421] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,422] jepsen worker 3 - jepsen.util 2253 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,422] jepsen worker 52 - jepsen.util 682 :invoke :write [17 7] INFO [2022-03-13 17:19:47,423] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:47,423] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:19:47,425] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,426] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,426] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,426] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:47,427] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 15] INFO [2022-03-13 17:19:47,430] jepsen worker 42 - jepsen.util 402 :fail :write [14 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,431] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,431] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,431] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:47,432] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:47,432] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:47,433] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,434] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,434] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,434] jepsen worker 7 - jepsen.util 637 :invoke :write [2 11] INFO [2022-03-13 17:19:47,436] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:47,437] jepsen worker 70 - jepsen.util 880 :invoke :write [23 5] INFO [2022-03-13 17:19:47,438] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,439] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:19:47,439] jepsen worker 25 - jepsen.util 565 :fail :write [8 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,441] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:19:47,453] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,454] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:47,455] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,455] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,455] jepsen worker 76 - jepsen.util 796 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,456] jepsen worker 75 - jepsen.util 525 :invoke :write [25 40] INFO [2022-03-13 17:19:47,457] jepsen worker 76 - jepsen.util 796 :invoke :write [25 22] INFO [2022-03-13 17:19:47,458] jepsen worker 13 - jepsen.util 733 :invoke :write [4 13] INFO [2022-03-13 17:19:47,495] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,495] jepsen worker 14 - jepsen.util 374 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,496] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:19:47,497] jepsen worker 21 - jepsen.util 831 :invoke :write [7 3] INFO [2022-03-13 17:19:47,517] jepsen worker 26 - jepsen.util 566 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,518] jepsen worker 26 - jepsen.util 566 :invoke :write [8 48] INFO [2022-03-13 17:19:47,519] jepsen worker 87 - jepsen.util 2877 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,520] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:47,525] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,526] jepsen worker 4 - jepsen.util 2614 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,526] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:19:47,531] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:19:47,569] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,570] jepsen worker 78 - jepsen.util 708 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,570] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,571] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:47,600] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,600] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,601] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:47,601] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,601] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:19:47,602] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:47,614] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,614] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,616] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,617] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:19:47,627] jepsen worker 80 - jepsen.util 800 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,627] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,627] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,628] jepsen worker 80 - jepsen.util 800 :invoke :write [26 8] INFO [2022-03-13 17:19:47,629] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:47,629] jepsen worker 34 - jepsen.util 934 :invoke :write [11 9] INFO [2022-03-13 17:19:47,665] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,665] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,665] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,666] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:47,667] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:19:47,667] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,668] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,668] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:47,669] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,669] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,670] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 40] INFO [2022-03-13 17:19:47,670] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 10] INFO [2022-03-13 17:19:47,672] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,673] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:19:47,708] jepsen worker 12 - jepsen.util 912 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,709] jepsen worker 12 - jepsen.util 912 :invoke :write [4 31] INFO [2022-03-13 17:19:47,712] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,713] jepsen worker 69 - jepsen.util 789 :invoke :write [23 4] INFO [2022-03-13 17:19:47,729] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,730] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,731] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,732] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:19:47,768] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,768] jepsen worker 35 - jepsen.util 935 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,769] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:19:47,769] jepsen worker 33 - jepsen.util 753 :invoke :write [11 26] INFO [2022-03-13 17:19:47,782] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,783] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,799] jepsen worker 43 - jepsen.util 673 :fail :write [14 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,799] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,800] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,800] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,800] jepsen worker 75 - jepsen.util 525 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,801] jepsen worker 60 - jepsen.util 690 :invoke :write [20 42] INFO [2022-03-13 17:19:47,802] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:19:47,803] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:47,804] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:47,804] jepsen worker 53 - jepsen.util 593 :invoke :write [17 37] INFO [2022-03-13 17:19:47,806] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,806] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:47,820] jepsen worker 61 - jepsen.util 781 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,821] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:47,866] jepsen worker 36 - jepsen.util 2466 :fail :write [12 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,867] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,867] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 44] INFO [2022-03-13 17:19:47,868] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:19:47,886] jepsen worker 51 - jepsen.util 771 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,886] jepsen worker 51 - jepsen.util 771 :invoke :write [17 6] INFO [2022-03-13 17:19:47,896] jepsen worker 44 - jepsen.util 764 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,897] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,897] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:47,897] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,897] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,897] jepsen worker 44 - jepsen.util 764 :invoke :write [14 40] INFO [2022-03-13 17:19:47,898] jepsen worker 8 - jepsen.util 728 :invoke :write [2 6] INFO [2022-03-13 17:19:47,899] jepsen worker 6 - jepsen.util 636 :invoke :write [2 30] INFO [2022-03-13 17:19:47,899] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:19:47,900] jepsen worker 62 - jepsen.util 512 :invoke :write [20 12] INFO [2022-03-13 17:19:47,925] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,926] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:19:47,947] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,947] jepsen worker 4 - jepsen.util 2614 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,947] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:19:47,948] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:19:47,960] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,961] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:47,988] jepsen worker 22 - jepsen.util 652 :fail :write [7 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,988] jepsen worker 76 - jepsen.util 796 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,989] jepsen worker 22 - jepsen.util 652 :invoke :write [7 15] INFO [2022-03-13 17:19:47,989] jepsen worker 76 - jepsen.util 796 :invoke :write [25 20] INFO [2022-03-13 17:19:47,994] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,995] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:47,997] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,998] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:47,998] jepsen worker 77 - jepsen.util 707 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:47,999] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:48,001] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,001] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:48,033] jepsen worker 36 - jepsen.util 2466 :fail :write [12 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,034] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:19:48,062] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,063] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:48,068] jepsen worker 7 - jepsen.util 637 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,069] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:48,090] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,091] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,096] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,097] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:48,126] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,127] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:19:48,146] jepsen worker 13 - jepsen.util 733 :fail :write [4 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,146] jepsen worker 52 - jepsen.util 682 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,149] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:48,150] jepsen worker 13 - jepsen.util 733 :invoke :write [4 25] INFO [2022-03-13 17:19:48,157] jepsen worker 79 - jepsen.util 1159 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,158] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 38] INFO [2022-03-13 17:19:48,166] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:48,166] jepsen worker nemesis - couchbase.util Rebalance check count: 694 INFO [2022-03-13 17:19:48,166] jepsen worker nemesis - couchbase.util Rebalance stuck count: 566 INFO [2022-03-13 17:19:48,174] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,175] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 30] INFO [2022-03-13 17:19:48,185] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,185] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:19:48,207] jepsen worker 21 - jepsen.util 831 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,207] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,207] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:48,208] jepsen worker 24 - jepsen.util 654 :invoke :write [8 27] INFO [2022-03-13 17:19:48,209] jepsen worker 70 - jepsen.util 880 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,209] jepsen worker 70 - jepsen.util 880 :invoke :write [23 10] INFO [2022-03-13 17:19:48,240] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,241] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,243] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,243] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,243] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 4] INFO [2022-03-13 17:19:48,244] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,246] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,247] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 21] INFO [2022-03-13 17:19:48,262] jepsen worker 26 - jepsen.util 566 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,263] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:48,263] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:48,264] jepsen worker 43 - jepsen.util 673 :invoke :write [14 37] INFO [2022-03-13 17:19:48,266] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,266] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:48,272] jepsen worker 23 - jepsen.util 563 :fail :write [7 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,272] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,272] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,272] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,273] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:19:48,273] jepsen worker 77 - jepsen.util 707 :invoke :write [25 48] INFO [2022-03-13 17:19:48,274] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:48,274] jepsen worker 61 - jepsen.util 781 :invoke :write [20 24] INFO [2022-03-13 17:19:48,278] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,279] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:19:48,310] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,310] jepsen worker 4 - jepsen.util 2614 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,312] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,312] jepsen worker 80 - jepsen.util 800 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,312] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:19:48,313] jepsen worker 80 - jepsen.util 800 :invoke :write [26 36] INFO [2022-03-13 17:19:48,339] jepsen worker 36 - jepsen.util 2466 :fail :write [12 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,339] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,339] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,339] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:19:48,358] jepsen worker 12 - jepsen.util 912 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,359] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:48,361] jepsen worker 5 - jepsen.util 2705 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,362] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,372] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,372] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,373] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 11] INFO [2022-03-13 17:19:48,373] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,373] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:19:48,374] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:19:48,374] jepsen worker 34 - jepsen.util 934 :fail :write [11 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,374] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,375] jepsen worker 25 - jepsen.util 565 :invoke :write [8 21] INFO [2022-03-13 17:19:48,375] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:48,408] jepsen worker 51 - jepsen.util 771 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,409] jepsen worker 51 - jepsen.util 771 :invoke :write [17 28] INFO [2022-03-13 17:19:48,415] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,416] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:19:48,454] jepsen worker 4 - jepsen.util 2614 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,455] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:19:48,457] jepsen worker 41 - jepsen.util 4811 :ok :read [13 39] INFO [2022-03-13 17:19:48,457] jepsen worker 41 - jepsen.util 4811 :invoke :read [13 nil] INFO [2022-03-13 17:19:48,478] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,478] jepsen worker 76 - jepsen.util 796 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,478] jepsen worker 60 - jepsen.util 690 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,479] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:19:48,480] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:48,481] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:48,484] jepsen worker 79 - jepsen.util 1159 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,484] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,485] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:48,486] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:48,492] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,493] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:19:48,510] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,510] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,510] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,511] jepsen worker 35 - jepsen.util 935 :invoke :write [11 16] INFO [2022-03-13 17:19:48,519] jepsen worker 14 - jepsen.util 374 :invoke :write [4 21] INFO [2022-03-13 17:19:48,519] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] WARN [2022-03-13 17:19:48,523] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17361151,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17401083,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9067186,"encode_duration_us":23467,"operation_id":"0x13af9","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10001347},{"operation_name":"upsert","last_dispatch_duration_us":17370566,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17855049,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9290166,"encode_duration_us":17668,"operation_id":"0x13df0","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10004161}],"total_count":2}} INFO [2022-03-13 17:19:48,526] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,527] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:48,541] jepsen worker 44 - jepsen.util 764 :fail :write [14 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,541] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:48,563] jepsen worker 81 - jepsen.util 5481 :ok :read [27 4] INFO [2022-03-13 17:19:48,563] jepsen worker 81 - jepsen.util 5481 :invoke :write [27 22] INFO [2022-03-13 17:19:48,576] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,580] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,586] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,587] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,588] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,589] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,590] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,590] jepsen worker 41 - jepsen.util 4811 :ok :read [13 39] INFO [2022-03-13 17:19:48,591] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:19:48,592] jepsen worker 41 - jepsen.util 4811 :invoke :write [13 42] INFO [2022-03-13 17:19:48,612] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,613] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:48,641] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,641] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,642] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:19:48,643] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,643] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,644] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,644] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:19:48,645] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 49] INFO [2022-03-13 17:19:48,657] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,657] jepsen worker 24 - jepsen.util 654 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,658] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:48,659] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:48,693] jepsen worker 6 - jepsen.util 636 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,694] jepsen worker 6 - jepsen.util 636 :invoke :write [2 46] INFO [2022-03-13 17:19:48,701] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,702] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:19:48,708] jepsen worker 13 - jepsen.util 733 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,708] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,709] jepsen worker 53 - jepsen.util 593 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,709] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,709] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:48,710] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:48,710] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,711] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:48,711] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:48,712] jepsen worker 75 - jepsen.util 525 :invoke :write [25 47] WARN [2022-03-13 17:19:48,718] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13198841,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":13198841,"total_server_duration_us":9341,"operation_id":"0x13db9","timeout_ms":30000,"last_server_duration_us":9341,"total_duration_us":13201667},{"operation_name":"upsert","last_dispatch_duration_us":9166762,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":12543743,"total_server_duration_us":9071139,"encode_duration_us":18729,"operation_id":"0x143e6","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10009983},{"operation_name":"upsert","last_dispatch_duration_us":8692733,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":15593035,"total_server_duration_us":133495,"encode_duration_us":24585,"operation_id":"0x141f0","timeout_ms":10000,"last_server_duration_us":25188,"total_duration_us":10009835},{"operation_name":"upsert","last_dispatch_duration_us":8684902,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":15602471,"total_server_duration_us":139016,"encode_duration_us":18473,"operation_id":"0x141e5","timeout_ms":10000,"last_server_duration_us":20760,"total_duration_us":10007643},{"operation_name":"upsert","last_dispatch_duration_us":8340811,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8470039,"total_server_duration_us":43743,"encode_duration_us":22999,"operation_id":"0x140c0","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10006702},{"operation_name":"upsert","last_dispatch_duration_us":89663,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":358924,"total_server_duration_us":70260,"encode_duration_us":19271,"operation_id":"0x14138","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10003864},{"operation_name":"upsert","last_dispatch_duration_us":8359287,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8812308,"total_server_duration_us":175788,"encode_duration_us":24222,"operation_id":"0x14171","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10001646},{"operation_name":"get","last_dispatch_duration_us":8058502,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8058502,"total_server_duration_us":310037,"operation_id":"0x14243","timeout_ms":30000,"last_server_duration_us":310037,"total_duration_us":8058805},{"operation_name":"get","last_dispatch_duration_us":6305302,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":6305302,"total_server_duration_us":43,"operation_id":"0x141c8","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":6309650},{"operation_name":"upsert","last_dispatch_duration_us":2231895,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":2231895,"total_server_duration_us":41123,"encode_duration_us":32648,"operation_id":"0x14767","timeout_ms":10000,"last_server_duration_us":41123,"total_duration_us":2236885}],"total_count":26}} INFO [2022-03-13 17:19:48,730] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,730] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,731] jepsen worker 76 - jepsen.util 796 :invoke :write [25 22] INFO [2022-03-13 17:19:48,731] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:48,736] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,736] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,737] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:19:48,737] jepsen worker 13 - jepsen.util 733 :invoke :write [4 39] WARN [2022-03-13 17:19:48,743] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9051403,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11090363,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9541701,"encode_duration_us":21015,"operation_id":"0x13d69","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10005623},{"operation_name":"upsert","last_dispatch_duration_us":25364926,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25364926,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9053293,"encode_duration_us":20529,"operation_id":"0x136ce","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10008606}],"total_count":2}} INFO [2022-03-13 17:19:48,770] jepsen worker 80 - jepsen.util 800 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,771] jepsen worker 80 - jepsen.util 800 :invoke :write [26 12] INFO [2022-03-13 17:19:48,774] jepsen worker 77 - jepsen.util 707 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,775] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,775] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:48,776] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:48,803] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,803] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] WARN [2022-03-13 17:19:48,819] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16002416,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16002416,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":6166527,"encode_duration_us":28784,"operation_id":"0x13c09","timeout_ms":10000,"last_server_duration_us":6166527,"total_duration_us":10005637},{"operation_name":"upsert","last_dispatch_duration_us":18046024,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27264803,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9270119,"encode_duration_us":35497,"operation_id":"0x13596","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10006043},{"operation_name":"upsert","last_dispatch_duration_us":18006330,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":19374422,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9246298,"encode_duration_us":27478,"operation_id":"0x13aeb","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10009233},{"operation_name":"upsert","last_dispatch_duration_us":17041442,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17041442,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9053293,"encode_duration_us":19681,"operation_id":"0x13c9a","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10007296},{"operation_name":"upsert","last_dispatch_duration_us":9166762,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":12543743,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9071139,"encode_duration_us":18729,"operation_id":"0x143e6","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10009983}],"total_count":5}} WARN [2022-03-13 17:19:48,819] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":6957,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":6968258,"total_server_duration_us":10715,"operation_id":"0x13656","timeout_ms":30000,"last_server_duration_us":376,"total_duration_us":30007547},{"operation_name":"get","last_dispatch_duration_us":15032888,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":15032888,"total_server_duration_us":255,"operation_id":"0x13d5c","timeout_ms":30000,"last_server_duration_us":255,"total_duration_us":15037143},{"operation_name":"upsert","last_dispatch_duration_us":9036162,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9036162,"total_server_duration_us":21082,"encode_duration_us":21022,"operation_id":"0x140fa","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10010104},{"operation_name":"upsert","last_dispatch_duration_us":8090870,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8090870,"total_server_duration_us":21001,"encode_duration_us":31032,"operation_id":"0x141a2","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10008443},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","encode_duration_us":19192,"operation_id":"0x14232","timeout_ms":10000,"total_duration_us":10003512},{"operation_name":"upsert","last_dispatch_duration_us":31878,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":7643514,"total_server_duration_us":293926,"encode_duration_us":17303,"operation_id":"0x1432d","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10002815},{"operation_name":"get","last_dispatch_duration_us":9165229,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9165229,"total_server_duration_us":19652,"operation_id":"0x140db","timeout_ms":30000,"last_server_duration_us":19652,"total_duration_us":9169901},{"operation_name":"upsert","last_dispatch_duration_us":291450,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":8614438,"total_server_duration_us":142468,"encode_duration_us":94574,"operation_id":"0x14248","timeout_ms":10000,"last_server_duration_us":51946,"total_duration_us":8773993},{"operation_name":"upsert","last_dispatch_duration_us":305636,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":8553631,"total_server_duration_us":199347,"encode_duration_us":18868,"operation_id":"0x14256","timeout_ms":10000,"last_server_duration_us":137543,"total_duration_us":8635840},{"operation_name":"upsert","last_dispatch_duration_us":8279520,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":8279520,"total_server_duration_us":8241013,"encode_duration_us":91646,"operation_id":"0x14245","timeout_ms":10000,"last_server_duration_us":8241013,"total_duration_us":8283754}],"total_count":69}} WARN [2022-03-13 17:19:48,823] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":36706,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":2608738,"total_server_duration_us":31880,"operation_id":"0x136b4","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":30006266},{"operation_name":"get","last_dispatch_duration_us":13842045,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":13842045,"total_server_duration_us":56,"operation_id":"0x140d9","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":13844193},{"operation_name":"upsert","last_dispatch_duration_us":8718677,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":8718677,"total_server_duration_us":19966,"encode_duration_us":19635,"operation_id":"0x14191","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10010396},{"operation_name":"upsert","last_dispatch_duration_us":7900655,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7900655,"total_server_duration_us":26418,"encode_duration_us":20785,"operation_id":"0x14220","timeout_ms":10000,"last_server_duration_us":26418,"total_duration_us":10004570},{"operation_name":"get","last_dispatch_duration_us":9592883,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9592883,"total_server_duration_us":32,"operation_id":"0x140f2","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":9594724},{"operation_name":"upsert","last_dispatch_duration_us":2592589,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2592589,"total_server_duration_us":78132,"encode_duration_us":27106,"operation_id":"0x1479e","timeout_ms":10000,"last_server_duration_us":78132,"total_duration_us":2602741},{"operation_name":"get","last_dispatch_duration_us":2454353,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2454353,"total_server_duration_us":30823,"operation_id":"0x147af","timeout_ms":30000,"last_server_duration_us":30823,"total_duration_us":2454509},{"operation_name":"upsert","last_dispatch_duration_us":2377040,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2377040,"total_server_duration_us":69347,"encode_duration_us":21570,"operation_id":"0x1479d","timeout_ms":10000,"last_server_duration_us":69347,"total_duration_us":2380952},{"operation_name":"upsert","last_dispatch_duration_us":2357712,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2357712,"total_server_duration_us":81398,"encode_duration_us":27806,"operation_id":"0x14789","timeout_ms":10000,"last_server_duration_us":81398,"total_duration_us":2371261},{"operation_name":"upsert","last_dispatch_duration_us":2130311,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2130311,"total_server_duration_us":1706350,"encode_duration_us":26081,"operation_id":"0x14788","timeout_ms":10000,"last_server_duration_us":1706350,"total_duration_us":2136832}],"total_count":44}} WARN [2022-03-13 17:19:48,830] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12470415,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":12470415,"total_server_duration_us":37138,"operation_id":"0x14408","timeout_ms":30000,"last_server_duration_us":37138,"total_duration_us":12472711},{"operation_name":"upsert","last_dispatch_duration_us":8966239,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":17191243,"total_server_duration_us":41845,"encode_duration_us":17908,"operation_id":"0x140e7","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10010877},{"operation_name":"upsert","last_dispatch_duration_us":41565,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8793222,"total_server_duration_us":265218,"encode_duration_us":36029,"operation_id":"0x1416c","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10004369},{"operation_name":"upsert","last_dispatch_duration_us":8159357,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8206378,"total_server_duration_us":54268,"encode_duration_us":23466,"operation_id":"0x141d5","timeout_ms":10000,"last_server_duration_us":26685,"total_duration_us":10002899},{"operation_name":"get","last_dispatch_duration_us":8605788,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8605788,"total_server_duration_us":172137,"operation_id":"0x140cc","timeout_ms":30000,"last_server_duration_us":172137,"total_duration_us":8614597},{"operation_name":"get","last_dispatch_duration_us":7766827,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7766827,"total_server_duration_us":460639,"operation_id":"0x1428e","timeout_ms":30000,"last_server_duration_us":460639,"total_duration_us":7767185},{"operation_name":"get","last_dispatch_duration_us":3972116,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":3972116,"total_server_duration_us":76,"operation_id":"0x1488e","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":3974532},{"operation_name":"upsert","last_dispatch_duration_us":2379317,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2379317,"total_server_duration_us":78837,"encode_duration_us":31138,"operation_id":"0x1476e","timeout_ms":10000,"last_server_duration_us":78837,"total_duration_us":2406580},{"operation_name":"upsert","last_dispatch_duration_us":2178788,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2178788,"total_server_duration_us":1741500,"encode_duration_us":38935,"operation_id":"0x14765","timeout_ms":10000,"last_server_duration_us":1741500,"total_duration_us":2189295},{"operation_name":"get","last_dispatch_duration_us":2133570,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2133570,"total_server_duration_us":38,"operation_id":"0x1479b","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":2133855}],"total_count":27}} INFO [2022-03-13 17:19:48,850] jepsen worker 22 - jepsen.util 652 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,851] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,851] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,851] jepsen worker 22 - jepsen.util 652 :invoke :write [7 11] INFO [2022-03-13 17:19:48,852] jepsen worker 26 - jepsen.util 566 :invoke :write [8 23] INFO [2022-03-13 17:19:48,852] jepsen worker 52 - jepsen.util 682 :invoke :write [17 13] INFO [2022-03-13 17:19:48,857] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,857] jepsen worker 33 - jepsen.util 753 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,858] jepsen worker 33 - jepsen.util 753 :invoke :write [11 33] INFO [2022-03-13 17:19:48,859] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:48,873] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,873] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,875] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,875] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:48,881] jepsen worker 69 - jepsen.util 789 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,881] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,883] jepsen worker 71 - jepsen.util 611 :invoke :write [23 42] INFO [2022-03-13 17:19:48,884] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:48,886] jepsen worker 57 - jepsen.util 4737 :info :write [19 8] :RequestTimeoutException INFO [2022-03-13 17:19:48,887] jepsen worker 57 - jepsen.util 4827 :invoke :read [19 nil] INFO [2022-03-13 17:19:48,913] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,914] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] WARN [2022-03-13 17:19:48,919] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17645589,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27105107,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9302874,"encode_duration_us":32581,"operation_id":"0x1357a","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10006797}],"total_count":1}} WARN [2022-03-13 17:19:48,944] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","encode_duration_us":19039,"operation_id":"0x143a0","timeout_ms":10000,"total_duration_us":10008452},{"operation_name":"upsert","encode_duration_us":20097,"operation_id":"0x140b3","timeout_ms":10000,"total_duration_us":10007735},{"operation_name":"upsert","last_dispatch_duration_us":7539805,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8038069,"total_server_duration_us":316187,"encode_duration_us":20610,"operation_id":"0x141c9","timeout_ms":10000,"last_server_duration_us":35412,"total_duration_us":10000949},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":22627,"operation_id":"0x14103","timeout_ms":10000,"total_duration_us":10000734},{"operation_name":"get","last_dispatch_duration_us":2585660,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2585660,"total_server_duration_us":108,"operation_id":"0x14709","timeout_ms":30000,"last_server_duration_us":108,"total_duration_us":3155206},{"operation_name":"upsert","last_dispatch_duration_us":2891035,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2891035,"total_server_duration_us":83703,"encode_duration_us":25365,"operation_id":"0x146f5","timeout_ms":10000,"last_server_duration_us":83703,"total_duration_us":2906059},{"operation_name":"upsert","last_dispatch_duration_us":2724948,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2724948,"total_server_duration_us":76451,"encode_duration_us":24725,"operation_id":"0x14739","timeout_ms":10000,"last_server_duration_us":76451,"total_duration_us":2729102},{"operation_name":"upsert","last_dispatch_duration_us":2596245,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2596245,"total_server_duration_us":44058,"encode_duration_us":18131,"operation_id":"0x14793","timeout_ms":10000,"last_server_duration_us":44058,"total_duration_us":2601206},{"operation_name":"get","last_dispatch_duration_us":2486436,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2486436,"total_server_duration_us":43,"operation_id":"0x14772","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":2517265},{"operation_name":"get","last_dispatch_duration_us":2486584,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2486584,"total_server_duration_us":11,"operation_id":"0x14778","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":2512100}],"total_count":70}} INFO [2022-03-13 17:19:48,975] jepsen worker 3 - jepsen.util 2253 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,975] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,975] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,976] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 7] INFO [2022-03-13 17:19:48,976] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,976] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:48,979] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,979] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,979] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:19:48,980] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:19:48,980] jepsen worker 25 - jepsen.util 565 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,980] jepsen worker 25 - jepsen.util 565 :invoke :write [8 27] INFO [2022-03-13 17:19:48,991] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:48,991] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:49,015] jepsen worker 61 - jepsen.util 781 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,015] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,015] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,015] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,016] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,016] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,017] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:49,018] jepsen worker 75 - jepsen.util 525 :fail :write [25 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,018] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:19:49,020] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:49,020] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:49,022] jepsen worker 23 - jepsen.util 563 :invoke :write [7 18] INFO [2022-03-13 17:19:49,022] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:49,023] jepsen worker 75 - jepsen.util 525 :invoke :write [25 30] INFO [2022-03-13 17:19:49,024] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,024] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,024] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,024] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,025] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:49,026] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 48] INFO [2022-03-13 17:19:49,026] jepsen worker 77 - jepsen.util 707 :invoke :write [25 24] INFO [2022-03-13 17:19:49,026] jepsen worker 83 - jepsen.util 4403 :info :write [27 45] :RequestTimeoutException INFO [2022-03-13 17:19:49,027] jepsen worker 61 - jepsen.util 781 :invoke :write [20 22] INFO [2022-03-13 17:19:49,027] jepsen worker 83 - jepsen.util 4493 :invoke :write [27 8] WARN [2022-03-13 17:19:49,030] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18202905,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27029570,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9206924,"encode_duration_us":34920,"operation_id":"0x13ae5","timeout_ms":10000,"last_server_duration_us":9058604,"total_duration_us":10003951},{"operation_name":"upsert","last_dispatch_duration_us":9067240,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10680347,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9670992,"encode_duration_us":28784,"operation_id":"0x13b98","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10009850},{"operation_name":"upsert","last_dispatch_duration_us":13593492,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13593492,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9046921,"encode_duration_us":17736,"operation_id":"0x13dbf","timeout_ms":10000,"last_server_duration_us":9046921,"total_duration_us":10008661}],"total_count":3}} INFO [2022-03-13 17:19:49,085] jepsen worker 62 - jepsen.util 512 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,085] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,086] jepsen worker 62 - jepsen.util 512 :invoke :write [20 31] INFO [2022-03-13 17:19:49,086] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:49,089] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,090] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:19:49,092] jepsen worker 36 - jepsen.util 2466 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,093] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 28] INFO [2022-03-13 17:19:49,098] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,098] jepsen worker 71 - jepsen.util 611 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,099] jepsen worker 69 - jepsen.util 789 :invoke :write [23 2] INFO [2022-03-13 17:19:49,100] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:19:49,103] jepsen worker 43 - jepsen.util 673 :fail :write [14 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,103] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,104] jepsen worker 43 - jepsen.util 673 :invoke :write [14 47] INFO [2022-03-13 17:19:49,106] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:49,106] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,107] jepsen worker 60 - jepsen.util 690 :invoke :write [20 4] INFO [2022-03-13 17:19:49,113] jepsen worker 51 - jepsen.util 771 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,113] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,114] jepsen worker 51 - jepsen.util 771 :invoke :write [17 16] INFO [2022-03-13 17:19:49,115] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:49,172] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:49,172] jepsen worker nemesis - couchbase.util Rebalance check count: 695 INFO [2022-03-13 17:19:49,172] jepsen worker nemesis - couchbase.util Rebalance stuck count: 567 INFO [2022-03-13 17:19:49,175] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,176] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,177] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,178] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,179] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,180] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,184] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,185] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,188] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,188] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,189] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,190] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:49,196] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,196] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:19:49,222] jepsen worker 35 - jepsen.util 935 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,223] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,223] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:49,224] jepsen worker 24 - jepsen.util 654 :invoke :write [8 47] INFO [2022-03-13 17:19:49,241] jepsen worker 69 - jepsen.util 789 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,241] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:49,245] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,246] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:19:49,270] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,270] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,277] jepsen worker 2 - jepsen.util 5132 :info :write [0 1] :RequestTimeoutException INFO [2022-03-13 17:19:49,277] jepsen worker 76 - jepsen.util 796 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,278] jepsen worker 76 - jepsen.util 796 :invoke :write [25 43] INFO [2022-03-13 17:19:49,278] jepsen worker 2 - jepsen.util 5222 :invoke :read [0 nil] INFO [2022-03-13 17:19:49,279] jepsen worker 84 - jepsen.util 5124 :info :write [28 28] :RequestTimeoutException INFO [2022-03-13 17:19:49,280] jepsen worker 84 - jepsen.util 5214 :invoke :read [28 nil] INFO [2022-03-13 17:19:49,304] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,304] jepsen worker 8 - jepsen.util 728 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,305] jepsen worker 12 - jepsen.util 912 :invoke :write [4 46] INFO [2022-03-13 17:19:49,307] jepsen worker 8 - jepsen.util 728 :invoke :write [2 42] INFO [2022-03-13 17:19:49,311] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,311] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,312] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,313] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:19:49,337] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,337] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,338] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:49,338] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:49,339] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,339] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,340] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:19:49,340] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:49,342] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,343] jepsen worker 71 - jepsen.util 611 :invoke :write [23 43] INFO [2022-03-13 17:19:49,345] jepsen worker 36 - jepsen.util 2466 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,346] jepsen worker 14 - jepsen.util 374 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,347] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:49,347] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:49,349] jepsen worker 79 - jepsen.util 1159 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,350] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:49,389] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,389] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:49,393] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,394] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:49,395] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,396] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:19:49,399] jepsen worker 84 - jepsen.util 5214 :ok :read [28 44] INFO [2022-03-13 17:19:49,399] jepsen worker 84 - jepsen.util 5214 :invoke :write [28 18] INFO [2022-03-13 17:19:49,419] jepsen worker 22 - jepsen.util 652 :fail :write [7 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,420] jepsen worker 22 - jepsen.util 652 :invoke :write [7 48] INFO [2022-03-13 17:19:49,430] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,430] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,431] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,431] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:19:49,431] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 36] INFO [2022-03-13 17:19:49,435] jepsen worker 80 - jepsen.util 800 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,435] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:19:49,436] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:19:49,435] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,442] jepsen worker 34 - jepsen.util 934 :invoke :write [11 0] INFO [2022-03-13 17:19:49,462] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:49,462] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,462] jepsen worker 70 - jepsen.util 880 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,463] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:49,464] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:49,465] jepsen worker 70 - jepsen.util 880 :invoke :write [23 31] INFO [2022-03-13 17:19:49,469] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,469] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:19:49,473] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,474] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:19:49,556] jepsen worker 13 - jepsen.util 733 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,557] jepsen worker 13 - jepsen.util 733 :invoke :write [4 31] INFO [2022-03-13 17:19:49,561] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,562] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:49,567] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,568] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:19:49,574] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,575] jepsen worker 77 - jepsen.util 707 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,575] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,576] jepsen worker 77 - jepsen.util 707 :invoke :write [25 13] INFO [2022-03-13 17:19:49,616] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,617] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,619] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,620] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,621] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,621] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,622] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,623] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,624] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,625] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:19:49,647] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,648] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:19:49,656] jepsen worker 26 - jepsen.util 566 :fail :write [8 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,656] jepsen worker 26 - jepsen.util 566 :invoke :write [8 43] INFO [2022-03-13 17:19:49,664] jepsen worker 33 - jepsen.util 753 :fail :write [11 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,665] jepsen worker 33 - jepsen.util 753 :invoke :write [11 26] INFO [2022-03-13 17:19:49,675] jepsen worker 86 - jepsen.util 4586 :info :write [28 19] :RequestTimeoutException INFO [2022-03-13 17:19:49,676] jepsen worker 71 - jepsen.util 611 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,677] jepsen worker 86 - jepsen.util 4676 :invoke :read [28 nil] INFO [2022-03-13 17:19:49,678] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:19:49,702] jepsen worker 4 - jepsen.util 2614 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,702] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,710] jepsen worker 67 - jepsen.util 4117 :info :write [22 46] :RequestTimeoutException INFO [2022-03-13 17:19:49,711] jepsen worker 67 - jepsen.util 4207 :invoke :read [22 nil] INFO [2022-03-13 17:19:49,721] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,721] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 4] INFO [2022-03-13 17:19:49,732] jepsen worker 75 - jepsen.util 525 :fail :write [25 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,733] jepsen worker 75 - jepsen.util 525 :invoke :write [25 28] INFO [2022-03-13 17:19:49,734] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,734] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:49,744] jepsen worker 67 - jepsen.util 4207 :ok :read [22 30] INFO [2022-03-13 17:19:49,745] jepsen worker 25 - jepsen.util 565 :fail :write [8 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,745] jepsen worker 67 - jepsen.util 4207 :invoke :write [22 28] INFO [2022-03-13 17:19:49,745] jepsen worker 25 - jepsen.util 565 :invoke :write [8 33] INFO [2022-03-13 17:19:49,764] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:49,764] jepsen worker 6 - jepsen.util 636 :fail :write [2 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,765] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:49,766] jepsen worker 6 - jepsen.util 636 :invoke :write [2 33] INFO [2022-03-13 17:19:49,788] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,789] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:49,801] jepsen worker 52 - jepsen.util 682 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,801] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,802] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:49,802] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:49,810] jepsen worker 51 - jepsen.util 771 :fail :write [17 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,811] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:49,811] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,812] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:49,819] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,820] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:19:49,840] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,840] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,840] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,841] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:49,841] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,842] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,846] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,846] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,846] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,847] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:49,847] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:19:49,848] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,848] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,849] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 11] INFO [2022-03-13 17:19:49,849] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,850] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:19:49,871] jepsen worker 5 - jepsen.util 2705 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,871] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:49,873] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,873] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:49,908] jepsen worker 9 - jepsen.util 4329 :ok :read [3 25] INFO [2022-03-13 17:19:49,908] jepsen worker 9 - jepsen.util 4329 :invoke :write [3 3] INFO [2022-03-13 17:19:49,916] jepsen worker 61 - jepsen.util 781 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,916] jepsen worker 61 - jepsen.util 781 :invoke :write [20 23] INFO [2022-03-13 17:19:49,925] jepsen worker 29 - jepsen.util 5069 :ok :read [9 26] INFO [2022-03-13 17:19:49,926] jepsen worker 29 - jepsen.util 5069 :invoke :write [9 23] INFO [2022-03-13 17:19:49,929] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,929] jepsen worker 76 - jepsen.util 796 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,930] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:19:49,930] jepsen worker 28 - jepsen.util 4888 :ok :read [9 26] INFO [2022-03-13 17:19:49,932] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:19:49,934] jepsen worker 28 - jepsen.util 4888 :invoke :write [9 26] INFO [2022-03-13 17:19:49,941] jepsen worker 18 - jepsen.util 4788 :ok :read [6 16] INFO [2022-03-13 17:19:49,942] jepsen worker 18 - jepsen.util 4788 :invoke :write [6 10] INFO [2022-03-13 17:19:49,957] jepsen worker 43 - jepsen.util 673 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,957] jepsen worker 19 - jepsen.util 4609 :ok :read [6 16] INFO [2022-03-13 17:19:49,958] jepsen worker 43 - jepsen.util 673 :invoke :write [14 4] INFO [2022-03-13 17:19:49,959] jepsen worker 19 - jepsen.util 4609 :invoke :write [6 28] INFO [2022-03-13 17:19:49,969] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:49,969] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:19:49,999] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,000] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:19:50,024] jepsen worker 24 - jepsen.util 654 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,024] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,025] jepsen worker 24 - jepsen.util 654 :invoke :write [8 46] INFO [2022-03-13 17:19:50,025] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:50,026] jepsen worker 47 - jepsen.util 4637 :ok :read [15 48] INFO [2022-03-13 17:19:50,027] jepsen worker 47 - jepsen.util 4637 :invoke :write [15 17] INFO [2022-03-13 17:19:50,044] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,044] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,044] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:19:50,045] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:50,046] jepsen worker 64 - jepsen.util 4654 :ok :read [21 :nil] INFO [2022-03-13 17:19:50,047] jepsen worker 64 - jepsen.util 4654 :invoke :write [21 2] INFO [2022-03-13 17:19:50,051] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,051] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:50,052] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,053] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:19:50,062] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,063] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:19:50,068] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,069] jepsen worker 80 - jepsen.util 800 :invoke :write [26 2] INFO [2022-03-13 17:19:50,070] jepsen worker 36 - jepsen.util 2466 :fail :write [12 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,071] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 28] INFO [2022-03-13 17:19:50,111] jepsen worker 34 - jepsen.util 934 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,111] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,112] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:50,114] jepsen worker 34 - jepsen.util 934 :invoke :write [11 5] INFO [2022-03-13 17:19:50,118] jepsen worker 55 - jepsen.util 4375 :ok :read [18 21] INFO [2022-03-13 17:19:50,119] jepsen worker 55 - jepsen.util 4375 :invoke :write [18 43] INFO [2022-03-13 17:19:50,137] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,138] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,140] jepsen worker 13 - jepsen.util 733 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,140] jepsen worker 13 - jepsen.util 733 :invoke :write [4 34] INFO [2022-03-13 17:19:50,144] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,145] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,150] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,151] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,152] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,152] jepsen worker 33 - jepsen.util 753 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,153] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:19:50,154] jepsen worker 33 - jepsen.util 753 :invoke :write [11 17] INFO [2022-03-13 17:19:50,166] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,166] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,186] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:50,186] jepsen worker nemesis - couchbase.util Rebalance check count: 696 INFO [2022-03-13 17:19:50,186] jepsen worker nemesis - couchbase.util Rebalance stuck count: 568 INFO [2022-03-13 17:19:50,194] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,195] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:19:50,241] jepsen worker 30 - jepsen.util 4800 :ok :read [10 :nil] INFO [2022-03-13 17:19:50,242] jepsen worker 30 - jepsen.util 4800 :invoke :read [10 nil] INFO [2022-03-13 17:19:50,252] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,253] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,254] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,254] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:50,276] jepsen worker 60 - jepsen.util 690 :fail :write [20 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,277] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,277] jepsen worker 60 - jepsen.util 690 :invoke :write [20 9] INFO [2022-03-13 17:19:50,278] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:19:50,321] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,322] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,323] jepsen worker 25 - jepsen.util 565 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,323] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,323] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,323] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,323] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,324] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:50,324] jepsen worker 36 - jepsen.util 2466 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,324] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,324] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,325] jepsen worker 25 - jepsen.util 565 :invoke :write [8 44] INFO [2022-03-13 17:19:50,326] jepsen worker 14 - jepsen.util 374 :invoke :write [4 46] INFO [2022-03-13 17:19:50,327] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:50,327] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,327] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,328] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,328] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:50,330] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,330] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,331] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,331] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,332] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,332] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:50,332] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,333] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,333] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,334] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:19:50,334] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,334] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,335] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,335] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:19:50,335] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,337] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,338] jepsen worker 62 - jepsen.util 512 :fail :write [20 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,339] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,339] jepsen worker 23 - jepsen.util 563 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,340] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:19:50,341] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:50,342] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:50,343] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:19:50,343] jepsen worker 62 - jepsen.util 512 :invoke :write [20 13] INFO [2022-03-13 17:19:50,345] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,347] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:50,361] jepsen worker 11 - jepsen.util 4601 :info :write [3 4] :RequestTimeoutException INFO [2022-03-13 17:19:50,362] jepsen worker 11 - jepsen.util 4691 :invoke :read [3 nil] INFO [2022-03-13 17:19:50,370] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,370] jepsen worker 27 - jepsen.util 4887 :info :write [9 10] :RequestTimeoutException INFO [2022-03-13 17:19:50,370] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:50,371] jepsen worker 27 - jepsen.util 4977 :invoke :write [9 38] INFO [2022-03-13 17:19:50,392] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,392] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:19:50,401] jepsen worker 32 - jepsen.util 5072 :ok :read [10 :nil] INFO [2022-03-13 17:19:50,402] jepsen worker 32 - jepsen.util 5072 :invoke :read [10 nil] INFO [2022-03-13 17:19:50,408] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,408] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:19:50,439] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,440] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:50,462] jepsen worker 24 - jepsen.util 654 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,462] jepsen worker 24 - jepsen.util 654 :invoke :write [8 12] INFO [2022-03-13 17:19:50,465] jepsen worker 75 - jepsen.util 525 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,466] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:50,481] jepsen worker 12 - jepsen.util 912 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,482] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,482] jepsen worker 12 - jepsen.util 912 :invoke :write [4 3] INFO [2022-03-13 17:19:50,482] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,488] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,488] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,489] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,490] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:19:50,515] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,515] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:50,534] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,534] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:50,542] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,542] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,542] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,543] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,543] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:50,544] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,551] jepsen worker 32 - jepsen.util 5072 :ok :read [10 :nil] INFO [2022-03-13 17:19:50,551] jepsen worker 32 - jepsen.util 5072 :invoke :read [10 nil] INFO [2022-03-13 17:19:50,554] jepsen worker 32 - jepsen.util 5072 :ok :read [10 :nil] INFO [2022-03-13 17:19:50,554] jepsen worker 32 - jepsen.util 5072 :invoke :write [10 2] INFO [2022-03-13 17:19:50,556] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,556] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,556] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,557] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,558] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,558] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 31] INFO [2022-03-13 17:19:50,563] jepsen worker 22 - jepsen.util 652 :fail :write [7 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,564] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:50,566] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,566] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,567] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:19:50,567] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:19:50,572] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,572] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,574] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,574] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,575] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,576] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,577] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,578] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:19:50,607] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,608] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:19:50,636] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,637] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,637] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:19:50,638] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:50,639] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,640] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:50,641] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,642] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:19:50,642] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,643] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,644] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,645] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:19:50,646] jepsen worker 77 - jepsen.util 707 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,646] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:50,658] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,658] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:50,677] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,677] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:50,681] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,681] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,740] jepsen worker 54 - jepsen.util 4464 :info :write [18 38] :RequestTimeoutException INFO [2022-03-13 17:19:50,740] jepsen worker 54 - jepsen.util 4554 :invoke :read [18 nil] INFO [2022-03-13 17:19:50,741] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,742] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,760] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,761] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,762] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,763] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:19:50,767] jepsen worker 80 - jepsen.util 800 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,767] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,768] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:19:50,769] jepsen worker 80 - jepsen.util 800 :invoke :write [26 5] INFO [2022-03-13 17:19:50,787] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,788] jepsen worker 71 - jepsen.util 611 :invoke :write [23 21] INFO [2022-03-13 17:19:50,801] jepsen worker 56 - jepsen.util 4646 :ok :read [18 21] INFO [2022-03-13 17:19:50,802] jepsen worker 56 - jepsen.util 4646 :invoke :read [18 nil] INFO [2022-03-13 17:19:50,804] jepsen worker 26 - jepsen.util 566 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,804] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,804] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,805] jepsen worker 26 - jepsen.util 566 :invoke :write [8 42] INFO [2022-03-13 17:19:50,806] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:50,807] jepsen worker 52 - jepsen.util 682 :invoke :write [17 22] INFO [2022-03-13 17:19:50,817] jepsen worker 36 - jepsen.util 2466 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,818] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,818] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,818] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:50,819] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,820] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,820] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,821] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:19:50,821] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,822] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,822] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,823] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,823] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,824] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,824] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,825] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:19:50,826] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,827] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:19:50,845] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,847] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,849] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,849] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,851] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,851] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,857] jepsen worker 8 - jepsen.util 728 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,857] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:50,857] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,858] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:50,859] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:50,859] jepsen worker 8 - jepsen.util 728 :invoke :write [2 1] INFO [2022-03-13 17:19:50,863] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,864] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,865] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,866] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:50,887] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,888] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:50,888] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,888] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,889] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:50,889] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:19:50,896] jepsen worker 34 - jepsen.util 934 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,896] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:50,907] jepsen worker 13 - jepsen.util 733 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,908] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,908] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,909] jepsen worker 13 - jepsen.util 733 :invoke :write [4 9] INFO [2022-03-13 17:19:50,909] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:50,910] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:50,921] jepsen worker 61 - jepsen.util 781 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,922] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,922] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,922] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,923] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:50,923] jepsen worker 23 - jepsen.util 563 :invoke :write [7 14] INFO [2022-03-13 17:19:50,925] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 24] INFO [2022-03-13 17:19:50,926] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,927] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:50,927] jepsen worker 77 - jepsen.util 707 :invoke :write [25 44] INFO [2022-03-13 17:19:50,928] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,929] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,929] jepsen worker 61 - jepsen.util 781 :invoke :write [20 12] INFO [2022-03-13 17:19:50,933] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:50,942] jepsen worker 59 - jepsen.util 4739 :info :write [19 0] :SyncWriteAmbiguous INFO [2022-03-13 17:19:50,943] jepsen worker 59 - jepsen.util 4829 :invoke :read [19 nil] INFO [2022-03-13 17:19:50,961] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,961] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:19:50,982] jepsen worker 59 - jepsen.util 4829 :ok :read [19 36] INFO [2022-03-13 17:19:50,983] jepsen worker 59 - jepsen.util 4829 :invoke :write [19 34] INFO [2022-03-13 17:19:50,992] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,992] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,994] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,995] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:50,997] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:50,998] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:51,012] jepsen worker 43 - jepsen.util 673 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,012] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,013] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,013] jepsen worker 43 - jepsen.util 673 :invoke :write [14 31] INFO [2022-03-13 17:19:51,014] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:51,015] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,016] jepsen worker 78 - jepsen.util 708 :invoke :write [26 49] INFO [2022-03-13 17:19:51,017] jepsen worker 76 - jepsen.util 796 :invoke :write [25 3] INFO [2022-03-13 17:19:51,024] jepsen worker 33 - jepsen.util 753 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,024] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,026] jepsen worker 33 - jepsen.util 753 :invoke :write [11 48] INFO [2022-03-13 17:19:51,026] jepsen worker 51 - jepsen.util 771 :invoke :write [17 48] INFO [2022-03-13 17:19:51,029] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,030] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:19:51,039] jepsen worker 57 - jepsen.util 4827 :ok :read [19 36] INFO [2022-03-13 17:19:51,041] jepsen worker 57 - jepsen.util 4827 :invoke :write [19 28] INFO [2022-03-13 17:19:51,043] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,044] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,063] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,064] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,069] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,070] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,071] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,072] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,075] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,075] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:19:51,082] jepsen worker 70 - jepsen.util 880 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,082] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:51,083] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:19:51,083] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,100] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,100] jepsen worker 88 - jepsen.util 2428 :info :write [29 28] :RequestTimeoutException INFO [2022-03-13 17:19:51,101] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 21] INFO [2022-03-13 17:19:51,102] jepsen worker 88 - jepsen.util 2518 :invoke :write [29 38] INFO [2022-03-13 17:19:51,121] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,121] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:51,123] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,123] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:19:51,143] jepsen worker 25 - jepsen.util 565 :fail :write [8 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,144] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:19:51,155] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,156] jepsen worker 71 - jepsen.util 611 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,156] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,156] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:19:51,163] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,164] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,165] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,166] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,166] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,167] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,167] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:51,170] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,172] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,173] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,174] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,174] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,175] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,176] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,177] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,177] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:19:51,184] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,184] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:19:51,186] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,187] jepsen worker 60 - jepsen.util 690 :fail :write [20 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,188] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,188] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:51,189] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,189] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,190] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:19:51,190] jepsen worker 53 - jepsen.util 593 :invoke :write [17 28] INFO [2022-03-13 17:19:51,192] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:51,192] jepsen worker nemesis - couchbase.util Rebalance check count: 697 INFO [2022-03-13 17:19:51,193] jepsen worker nemesis - couchbase.util Rebalance stuck count: 569 INFO [2022-03-13 17:19:51,199] jepsen worker 14 - jepsen.util 374 :fail :write [4 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,200] jepsen worker 14 - jepsen.util 374 :invoke :write [4 1] INFO [2022-03-13 17:19:51,226] jepsen worker 3 - jepsen.util 2253 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,226] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,227] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 11] INFO [2022-03-13 17:19:51,228] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,256] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,257] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:51,284] jepsen worker 23 - jepsen.util 563 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,284] jepsen worker 23 - jepsen.util 563 :invoke :write [7 46] INFO [2022-03-13 17:19:51,291] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,291] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,292] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,293] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:19:51,307] jepsen worker 39 - jepsen.util 5169 :info :write [13 13] :RequestTimeoutException INFO [2022-03-13 17:19:51,307] jepsen worker 39 - jepsen.util 5259 :invoke :write [13 41] INFO [2022-03-13 17:19:51,321] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,321] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,325] jepsen worker 13 - jepsen.util 733 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,325] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,326] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:51,326] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:51,329] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,330] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:19:51,332] jepsen worker 4 - jepsen.util 2614 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,333] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 38] INFO [2022-03-13 17:19:51,338] jepsen worker 1 - jepsen.util 4501 :info :write [0 44] :RequestTimeoutException INFO [2022-03-13 17:19:51,339] jepsen worker 1 - jepsen.util 4591 :invoke :read [0 nil] INFO [2022-03-13 17:19:51,340] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,340] jepsen worker 66 - jepsen.util 4566 :info :write [22 46] :RequestTimeoutException INFO [2022-03-13 17:19:51,341] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,341] jepsen worker 66 - jepsen.util 4656 :invoke :write [22 20] INFO [2022-03-13 17:19:51,350] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,351] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,362] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,363] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:19:51,370] jepsen worker 24 - jepsen.util 654 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,370] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:51,390] jepsen worker 65 - jepsen.util 5015 :info :write [21 38] :RequestTimeoutException INFO [2022-03-13 17:19:51,391] jepsen worker 65 - jepsen.util 5105 :invoke :write [21 37] INFO [2022-03-13 17:19:51,391] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,391] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,392] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,393] jepsen worker 71 - jepsen.util 611 :invoke :write [23 32] INFO [2022-03-13 17:19:51,394] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,394] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,396] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,397] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:19:51,399] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,399] jepsen worker 1 - jepsen.util 4591 :ok :read [0 41] INFO [2022-03-13 17:19:51,399] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,400] jepsen worker 1 - jepsen.util 4591 :invoke :write [0 46] INFO [2022-03-13 17:19:51,408] jepsen worker 6 - jepsen.util 636 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,408] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:51,433] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,433] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,434] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,435] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:51,436] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,437] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,439] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,440] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,441] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,442] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:19:51,445] jepsen worker 52 - jepsen.util 682 :fail :write [17 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,445] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,445] jepsen worker 76 - jepsen.util 796 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,445] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,447] jepsen worker 63 - jepsen.util 4923 :info :write [21 30] :RequestTimeoutException INFO [2022-03-13 17:19:51,448] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:19:51,449] jepsen worker 63 - jepsen.util 5013 :invoke :read [21 nil] INFO [2022-03-13 17:19:51,450] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:51,451] jepsen worker 75 - jepsen.util 525 :invoke :write [25 13] INFO [2022-03-13 17:19:51,452] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:51,465] jepsen worker 63 - jepsen.util 5013 :ok :read [21 :nil] INFO [2022-03-13 17:19:51,466] jepsen worker 63 - jepsen.util 5013 :invoke :write [21 30] INFO [2022-03-13 17:19:51,500] jepsen worker 4 - jepsen.util 2614 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,501] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 40] INFO [2022-03-13 17:19:51,504] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,505] jepsen worker 13 - jepsen.util 733 :invoke :write [4 17] INFO [2022-03-13 17:19:51,517] jepsen worker 71 - jepsen.util 611 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,518] jepsen worker 71 - jepsen.util 611 :invoke :write [23 33] INFO [2022-03-13 17:19:51,553] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,553] jepsen worker 80 - jepsen.util 800 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,554] jepsen worker 80 - jepsen.util 800 :invoke :write [26 24] INFO [2022-03-13 17:19:51,554] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:51,556] jepsen worker 26 - jepsen.util 566 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,557] jepsen worker 26 - jepsen.util 566 :invoke :write [8 10] INFO [2022-03-13 17:19:51,558] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,558] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,559] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:19:51,559] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:19:51,563] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,564] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:51,567] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,568] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:51,593] jepsen worker 4 - jepsen.util 2614 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,593] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,616] jepsen worker 77 - jepsen.util 707 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,616] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,616] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:51,617] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,619] jepsen worker 75 - jepsen.util 525 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,620] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:51,620] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,621] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:19:51,625] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,626] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:19:51,668] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,669] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:19:51,670] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,671] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,679] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,679] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:51,690] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,691] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:51,695] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,696] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:51,696] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,697] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:51,703] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,704] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:51,723] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,723] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,724] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,724] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:19:51,743] jepsen worker 62 - jepsen.util 512 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,743] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:51,750] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,750] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,751] jepsen worker 43 - jepsen.util 673 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,751] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:19:51,752] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,753] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:51,753] jepsen worker 60 - jepsen.util 690 :invoke :write [20 40] INFO [2022-03-13 17:19:51,754] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,759] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:51,759] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:51,761] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,761] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:51,777] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,777] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,779] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,780] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:19:51,787] jepsen worker 51 - jepsen.util 771 :fail :write [17 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,787] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:51,824] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,824] jepsen worker 71 - jepsen.util 611 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,825] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,825] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:51,826] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:19:51,826] jepsen worker 79 - jepsen.util 1159 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,826] jepsen worker 71 - jepsen.util 611 :invoke :write [23 39] INFO [2022-03-13 17:19:51,826] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:51,841] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,842] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:19:51,845] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,845] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,847] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:19:51,847] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:51,893] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,893] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:19:51,899] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,899] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:51,923] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,923] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,924] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 44] INFO [2022-03-13 17:19:51,924] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:51,932] jepsen worker 12 - jepsen.util 912 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,932] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:51,932] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,933] jepsen worker 12 - jepsen.util 912 :invoke :write [4 22] INFO [2022-03-13 17:19:51,933] jepsen worker 44 - jepsen.util 764 :invoke :write [14 16] INFO [2022-03-13 17:19:51,934] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:51,940] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,941] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:19:51,945] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,946] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:51,971] jepsen worker 71 - jepsen.util 611 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:51,972] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,001] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,002] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,003] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,003] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:52,005] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,005] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:52,070] jepsen worker 73 - jepsen.util 5113 :info :write [24 17] :RequestTimeoutException INFO [2022-03-13 17:19:52,071] jepsen worker 73 - jepsen.util 5203 :invoke :write [24 16] INFO [2022-03-13 17:19:52,071] jepsen worker 4 - jepsen.util 2614 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,071] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,072] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 29] INFO [2022-03-13 17:19:52,072] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 40] INFO [2022-03-13 17:19:52,073] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,073] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,074] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,075] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:19:52,114] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,114] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,115] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,116] jepsen worker 25 - jepsen.util 565 :invoke :write [8 49] INFO [2022-03-13 17:19:52,121] jepsen worker 13 - jepsen.util 733 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,121] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,121] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,124] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:52,124] jepsen worker 13 - jepsen.util 733 :invoke :write [4 37] INFO [2022-03-13 17:19:52,125] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,125] jepsen worker 76 - jepsen.util 796 :invoke :write [25 40] INFO [2022-03-13 17:19:52,126] jepsen worker 75 - jepsen.util 525 :invoke :write [25 23] INFO [2022-03-13 17:19:52,138] jepsen worker 61 - jepsen.util 781 :fail :write [20 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,139] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:52,144] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,144] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,146] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,147] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,148] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,148] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,149] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:52,149] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,153] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,153] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,154] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,155] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,156] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,156] jepsen worker 46 - jepsen.util 4996 :info :write [15 10] :RequestTimeoutException INFO [2022-03-13 17:19:52,156] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,157] jepsen worker 46 - jepsen.util 5086 :invoke :write [15 45] INFO [2022-03-13 17:19:52,157] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,158] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 14] INFO [2022-03-13 17:19:52,198] jepsen worker 10 - jepsen.util 5050 :info :write [3 11] :RequestTimeoutException INFO [2022-03-13 17:19:52,198] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:52,198] jepsen worker nemesis - couchbase.util Rebalance check count: 698 INFO [2022-03-13 17:19:52,198] jepsen worker nemesis - couchbase.util Rebalance stuck count: 570 INFO [2022-03-13 17:19:52,198] jepsen worker 10 - jepsen.util 5140 :invoke :write [3 14] INFO [2022-03-13 17:19:52,219] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,219] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,220] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,220] jepsen worker 53 - jepsen.util 593 :fail :write [17 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,221] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,222] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,222] jepsen worker 52 - jepsen.util 682 :invoke :write [17 35] INFO [2022-03-13 17:19:52,223] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:52,223] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,223] jepsen worker 14 - jepsen.util 374 :fail :write [4 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,224] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,224] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:52,224] jepsen worker 4 - jepsen.util 2614 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,225] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,225] jepsen worker 69 - jepsen.util 789 :invoke :write [23 31] INFO [2022-03-13 17:19:52,226] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,226] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:19:52,226] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,232] jepsen worker 14 - jepsen.util 374 :invoke :write [4 49] INFO [2022-03-13 17:19:52,233] jepsen worker 71 - jepsen.util 611 :invoke :write [23 48] INFO [2022-03-13 17:19:52,247] jepsen worker 8 - jepsen.util 728 :fail :write [2 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,247] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,250] jepsen worker 8 - jepsen.util 728 :invoke :write [2 24] INFO [2022-03-13 17:19:52,251] jepsen worker 70 - jepsen.util 880 :invoke :write [23 7] INFO [2022-03-13 17:19:52,280] jepsen worker 20 - jepsen.util 5060 :info :write [6 10] :RequestTimeoutException INFO [2022-03-13 17:19:52,281] jepsen worker 20 - jepsen.util 5150 :invoke :read [6 nil] INFO [2022-03-13 17:19:52,304] jepsen worker 87 - jepsen.util 2877 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,305] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,306] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,307] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,307] jepsen worker 3 - jepsen.util 2253 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,308] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,308] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 26] INFO [2022-03-13 17:19:52,309] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,309] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,310] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:52,311] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,311] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,311] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,312] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 32] INFO [2022-03-13 17:19:52,312] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,313] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 36] INFO [2022-03-13 17:19:52,316] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,316] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,317] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,318] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,319] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,320] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,321] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,321] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,322] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,323] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,324] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,324] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,325] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,326] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:19:52,353] jepsen worker 74 - jepsen.util 4664 :info :write [24 16] :RequestTimeoutException INFO [2022-03-13 17:19:52,354] jepsen worker 74 - jepsen.util 4754 :invoke :write [24 33] INFO [2022-03-13 17:19:52,366] jepsen worker 80 - jepsen.util 800 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,366] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,366] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,366] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,366] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,366] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:52,367] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,367] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:52,368] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:52,369] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:52,369] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,370] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,375] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,375] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:52,385] jepsen worker 78 - jepsen.util 708 :fail :write [26 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,386] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:52,386] jepsen worker 69 - jepsen.util 789 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,387] jepsen worker 69 - jepsen.util 789 :invoke :write [23 40] INFO [2022-03-13 17:19:52,396] jepsen worker 26 - jepsen.util 566 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,396] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:52,397] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,398] jepsen worker 43 - jepsen.util 673 :invoke :write [14 39] INFO [2022-03-13 17:19:52,401] jepsen worker 4 - jepsen.util 2614 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,402] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 11] INFO [2022-03-13 17:19:52,429] jepsen worker 87 - jepsen.util 2877 :fail :write [29 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,430] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:19:52,475] jepsen worker 36 - jepsen.util 2466 :fail :write [12 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,475] jepsen worker 71 - jepsen.util 611 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,478] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 40] INFO [2022-03-13 17:19:52,479] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:19:52,488] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,488] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,488] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,488] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,488] jepsen worker 62 - jepsen.util 512 :invoke :write [20 0] INFO [2022-03-13 17:19:52,489] jepsen worker 42 - jepsen.util 402 :invoke :write [14 12] INFO [2022-03-13 17:19:52,490] jepsen worker 76 - jepsen.util 796 :fail :write [25 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,490] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:52,491] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:52,491] jepsen worker 76 - jepsen.util 796 :invoke :write [25 21] INFO [2022-03-13 17:19:52,519] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,520] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:19:52,528] jepsen worker 25 - jepsen.util 565 :fail :write [8 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,528] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,582] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,582] jepsen worker 60 - jepsen.util 690 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,582] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,582] jepsen worker 22 - jepsen.util 652 :invoke :write [7 2] INFO [2022-03-13 17:19:52,583] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:52,584] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:52,585] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,586] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:19:52,586] jepsen worker 85 - jepsen.util 4495 :info :write [28 7] :RequestTimeoutException INFO [2022-03-13 17:19:52,587] jepsen worker 85 - jepsen.util 4585 :invoke :write [28 2] INFO [2022-03-13 17:19:52,621] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,621] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:52,623] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,624] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:52,634] jepsen worker 13 - jepsen.util 733 :fail :write [4 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,635] jepsen worker 13 - jepsen.util 733 :invoke :write [4 19] INFO [2022-03-13 17:19:52,640] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,640] jepsen worker 23 - jepsen.util 563 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,640] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,640] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,640] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,641] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:52,642] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:19:52,642] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,642] jepsen worker 77 - jepsen.util 707 :invoke :write [25 1] INFO [2022-03-13 17:19:52,643] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:52,643] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:52,644] jepsen worker 78 - jepsen.util 708 :invoke :write [26 2] INFO [2022-03-13 17:19:52,657] jepsen worker 69 - jepsen.util 789 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,658] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,659] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,659] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,659] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 20] INFO [2022-03-13 17:19:52,660] jepsen worker 61 - jepsen.util 781 :invoke :write [20 23] INFO [2022-03-13 17:19:52,669] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,669] jepsen worker 33 - jepsen.util 753 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,670] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:52,671] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:19:52,677] jepsen worker 4 - jepsen.util 2614 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,678] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:19:52,697] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,698] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:52,741] jepsen worker 36 - jepsen.util 2466 :fail :write [12 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,742] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:52,749] jepsen worker 44 - jepsen.util 764 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,750] jepsen worker 40 - jepsen.util 4990 :info :write [13 32] :RequestTimeoutException INFO [2022-03-13 17:19:52,750] jepsen worker 44 - jepsen.util 764 :invoke :write [14 14] INFO [2022-03-13 17:19:52,753] jepsen worker 40 - jepsen.util 5080 :invoke :write [13 14] INFO [2022-03-13 17:19:52,754] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,755] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:19:52,758] jepsen worker 58 - jepsen.util 4288 :info :write [19 15] :RequestTimeoutException INFO [2022-03-13 17:19:52,759] jepsen worker 58 - jepsen.util 4378 :invoke :read [19 nil] INFO [2022-03-13 17:19:52,764] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,765] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,765] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,766] jepsen worker 69 - jepsen.util 789 :invoke :write [23 21] INFO [2022-03-13 17:19:52,766] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,767] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,769] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,770] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,771] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,772] jepsen worker 52 - jepsen.util 682 :fail :write [17 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,772] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,772] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:52,773] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,773] jepsen worker 52 - jepsen.util 682 :invoke :write [17 0] INFO [2022-03-13 17:19:52,775] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,775] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:19:52,778] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,778] jepsen worker 14 - jepsen.util 374 :fail :write [4 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,779] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,779] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,779] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,779] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,779] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,780] jepsen worker 14 - jepsen.util 374 :invoke :write [4 41] INFO [2022-03-13 17:19:52,780] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,783] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:52,784] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,784] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,785] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:52,786] jepsen worker 51 - jepsen.util 771 :invoke :write [17 32] INFO [2022-03-13 17:19:52,787] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:52,787] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:52,812] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,816] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:19:52,837] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,838] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 39] INFO [2022-03-13 17:19:52,851] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,851] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:19:52,871] jepsen worker 12 - jepsen.util 912 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,872] jepsen worker 75 - jepsen.util 525 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,873] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:52,874] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:52,881] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,882] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,883] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,884] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:52,884] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,885] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,885] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:19:52,885] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,885] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,886] jepsen worker 21 - jepsen.util 831 :invoke :write [7 23] INFO [2022-03-13 17:19:52,887] jepsen worker 24 - jepsen.util 654 :invoke :write [8 1] INFO [2022-03-13 17:19:52,888] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,888] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,888] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:52,889] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:52,890] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,891] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:52,894] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:52,898] jepsen worker 69 - jepsen.util 789 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,899] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:19:52,933] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,933] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:19:52,956] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,956] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:52,968] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,968] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,968] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,969] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:52,969] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,970] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,971] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,971] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,972] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,973] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,974] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,975] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,976] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,976] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:52,979] jepsen worker 14 - jepsen.util 374 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,979] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:52,996] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:52,997] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:53,000] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,001] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,001] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:53,003] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:53,004] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,005] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:19:53,020] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,020] jepsen worker 3 - jepsen.util 2253 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,020] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,021] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:19:53,021] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,021] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:53,022] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,023] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,023] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:19:53,023] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 0] INFO [2022-03-13 17:19:53,087] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,087] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,098] jepsen worker 51 - jepsen.util 771 :fail :write [17 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,098] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,099] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,098] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,099] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:53,099] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,099] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:53,100] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:53,100] jepsen worker 80 - jepsen.util 800 :invoke :write [26 21] INFO [2022-03-13 17:19:53,100] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:53,124] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,124] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,124] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:19:53,125] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:53,126] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,126] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:53,127] jepsen worker 43 - jepsen.util 673 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,127] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,127] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,127] jepsen worker 43 - jepsen.util 673 :invoke :write [14 25] INFO [2022-03-13 17:19:53,128] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,131] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:19:53,131] jepsen worker 53 - jepsen.util 593 :invoke :write [17 27] INFO [2022-03-13 17:19:53,132] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:53,133] jepsen worker 76 - jepsen.util 796 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,133] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,133] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:53,134] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 3] INFO [2022-03-13 17:19:53,144] jepsen worker 70 - jepsen.util 880 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,149] jepsen worker 70 - jepsen.util 880 :invoke :write [23 0] INFO [2022-03-13 17:19:53,150] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,152] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,153] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,154] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:53,193] jepsen worker 36 - jepsen.util 2466 :fail :write [12 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,193] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,193] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:53,194] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,204] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:53,204] jepsen worker nemesis - couchbase.util Rebalance check count: 699 INFO [2022-03-13 17:19:53,204] jepsen worker nemesis - couchbase.util Rebalance stuck count: 571 INFO [2022-03-13 17:19:53,243] jepsen worker 77 - jepsen.util 707 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,245] jepsen worker 77 - jepsen.util 707 :invoke :write [25 17] INFO [2022-03-13 17:19:53,249] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,250] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:53,251] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,251] jepsen worker 4 - jepsen.util 2614 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,251] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,252] jepsen worker 33 - jepsen.util 753 :invoke :write [11 39] INFO [2022-03-13 17:19:53,253] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,255] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,256] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,257] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,257] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,257] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,257] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,258] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,259] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,259] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:19:53,259] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:53,260] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,260] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:19:53,261] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,260] jepsen worker 22 - jepsen.util 652 :fail :write [7 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,261] jepsen worker 22 - jepsen.util 652 :invoke :write [7 24] INFO [2022-03-13 17:19:53,266] jepsen worker 71 - jepsen.util 611 :invoke :write [23 3] INFO [2022-03-13 17:19:53,269] jepsen worker 26 - jepsen.util 566 :invoke :write [8 33] INFO [2022-03-13 17:19:53,314] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,314] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:19:53,345] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,346] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,346] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,346] jepsen worker 13 - jepsen.util 733 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,346] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:19:53,347] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:53,347] jepsen worker 75 - jepsen.util 525 :invoke :write [25 39] INFO [2022-03-13 17:19:53,348] jepsen worker 76 - jepsen.util 796 :invoke :write [25 18] INFO [2022-03-13 17:19:53,351] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,352] jepsen worker 13 - jepsen.util 733 :invoke :write [4 10] INFO [2022-03-13 17:19:53,359] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,360] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:53,371] jepsen worker 21 - jepsen.util 831 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,371] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:19:53,373] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,373] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,373] jepsen worker 8 - jepsen.util 728 :fail :write [2 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,374] jepsen worker 6 - jepsen.util 636 :invoke :write [2 42] INFO [2022-03-13 17:19:53,375] jepsen worker 7 - jepsen.util 637 :invoke :write [2 22] INFO [2022-03-13 17:19:53,375] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:53,383] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,383] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:19:53,398] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,399] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:19:53,448] jepsen worker 78 - jepsen.util 708 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,449] jepsen worker 78 - jepsen.util 708 :invoke :write [26 20] INFO [2022-03-13 17:19:53,453] jepsen worker 71 - jepsen.util 611 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,453] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:19:53,454] jepsen worker 52 - jepsen.util 682 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,454] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,454] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,455] jepsen worker 52 - jepsen.util 682 :invoke :write [17 2] INFO [2022-03-13 17:19:53,455] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:53,461] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:53,462] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,463] jepsen worker 69 - jepsen.util 789 :invoke :write [23 8] INFO [2022-03-13 17:19:53,467] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,468] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:19:53,469] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,470] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,470] jepsen worker 24 - jepsen.util 654 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,470] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,470] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,470] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,470] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:19:53,470] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,471] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:53,473] jepsen worker 34 - jepsen.util 934 :invoke :write [11 40] INFO [2022-03-13 17:19:53,478] jepsen worker 24 - jepsen.util 654 :invoke :write [8 24] INFO [2022-03-13 17:19:53,480] jepsen worker 25 - jepsen.util 565 :invoke :write [8 24] INFO [2022-03-13 17:19:53,481] jepsen worker 14 - jepsen.util 374 :invoke :write [4 39] INFO [2022-03-13 17:19:53,486] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:53,534] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,535] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:53,540] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,540] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:19:53,584] jepsen worker 75 - jepsen.util 525 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,584] jepsen worker 75 - jepsen.util 525 :invoke :write [25 11] INFO [2022-03-13 17:19:53,587] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,588] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:53,590] jepsen worker 80 - jepsen.util 800 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,591] jepsen worker 80 - jepsen.util 800 :invoke :write [26 16] INFO [2022-03-13 17:19:53,596] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,597] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:19:53,620] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,621] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:19:53,652] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,652] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,653] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,653] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:53,653] jepsen worker 61 - jepsen.util 781 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,654] jepsen worker 42 - jepsen.util 402 :fail :write [14 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,654] jepsen worker 42 - jepsen.util 402 :invoke :write [14 13] INFO [2022-03-13 17:19:53,655] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:19:53,671] jepsen worker 43 - jepsen.util 673 :fail :write [14 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,672] jepsen worker 43 - jepsen.util 673 :invoke :write [14 18] INFO [2022-03-13 17:19:53,684] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,684] jepsen worker 71 - jepsen.util 611 :invoke :write [23 22] INFO [2022-03-13 17:19:53,697] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,698] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:19:53,700] jepsen worker 15 - jepsen.util 4515 :info :write [5 21] :RequestTimeoutException INFO [2022-03-13 17:19:53,701] jepsen worker 15 - jepsen.util 4605 :invoke :read [5 nil] INFO [2022-03-13 17:19:53,713] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,714] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,714] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,714] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,716] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:53,716] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 35] INFO [2022-03-13 17:19:53,860] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,860] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,861] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:53,861] jepsen worker 51 - jepsen.util 771 :invoke :write [17 47] INFO [2022-03-13 17:19:53,899] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:53,899] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:53,946] jepsen worker 31 - jepsen.util 4981 :info :write [10 0] :RequestTimeoutException INFO [2022-03-13 17:19:53,947] jepsen worker 31 - jepsen.util 5071 :invoke :write [10 0] INFO [2022-03-13 17:19:54,082] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,082] jepsen worker 12 - jepsen.util 912 :invoke :write [4 30] INFO [2022-03-13 17:19:54,197] jepsen worker 34 - jepsen.util 934 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,198] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:54,207] jepsen worker 22 - jepsen.util 652 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,207] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:54,209] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:54,209] jepsen worker nemesis - couchbase.util Rebalance check count: 700 INFO [2022-03-13 17:19:54,209] jepsen worker nemesis - couchbase.util Rebalance stuck count: 572 INFO [2022-03-13 17:19:54,259] jepsen worker 71 - jepsen.util 611 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,260] jepsen worker 71 - jepsen.util 611 :invoke :write [23 13] INFO [2022-03-13 17:19:54,354] jepsen worker 25 - jepsen.util 565 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,355] jepsen worker 25 - jepsen.util 565 :invoke :write [8 7] INFO [2022-03-13 17:19:54,356] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,357] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:54,358] jepsen worker 70 - jepsen.util 880 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,358] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:54,373] jepsen worker 75 - jepsen.util 525 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,374] jepsen worker 75 - jepsen.util 525 :invoke :write [25 33] INFO [2022-03-13 17:19:54,462] jepsen worker 78 - jepsen.util 708 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,462] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:54,482] jepsen worker 69 - jepsen.util 789 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,483] jepsen worker 69 - jepsen.util 789 :invoke :write [23 33] INFO [2022-03-13 17:19:54,529] jepsen worker 33 - jepsen.util 753 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,529] jepsen worker 33 - jepsen.util 753 :invoke :write [11 22] INFO [2022-03-13 17:19:54,537] jepsen worker 26 - jepsen.util 566 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,537] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,538] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:54,538] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:54,580] jepsen worker 49 - jepsen.util 3559 :info :write [16 9] :RequestTimeoutException INFO [2022-03-13 17:19:54,582] jepsen worker 49 - jepsen.util 3649 :invoke :write [16 14] INFO [2022-03-13 17:19:54,608] jepsen worker 86 - jepsen.util 4676 :ok :read [28 44] INFO [2022-03-13 17:19:54,609] jepsen worker 86 - jepsen.util 4676 :invoke :read [28 nil] INFO [2022-03-13 17:19:54,645] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,645] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,646] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:54,646] jepsen worker 35 - jepsen.util 935 :invoke :write [11 43] INFO [2022-03-13 17:19:54,964] jepsen worker 71 - jepsen.util 611 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,965] jepsen worker 71 - jepsen.util 611 :invoke :write [23 46] INFO [2022-03-13 17:19:54,970] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,970] jepsen worker 7 - jepsen.util 637 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,970] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,971] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:54,972] jepsen worker 7 - jepsen.util 637 :invoke :write [2 20] INFO [2022-03-13 17:19:54,973] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:19:54,991] jepsen worker 76 - jepsen.util 796 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:54,992] jepsen worker 76 - jepsen.util 796 :invoke :write [25 9] INFO [2022-03-13 17:19:55,028] jepsen worker 51 - jepsen.util 771 :fail :write [17 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,029] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:55,089] jepsen worker 52 - jepsen.util 682 :fail :write [17 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,090] jepsen worker 52 - jepsen.util 682 :invoke :write [17 46] INFO [2022-03-13 17:19:55,098] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,099] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,099] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,099] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,102] jepsen worker 69 - jepsen.util 789 :fail :write [23 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,103] jepsen worker 69 - jepsen.util 789 :invoke :write [23 1] INFO [2022-03-13 17:19:55,120] jepsen worker 6 - jepsen.util 636 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,121] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:19:55,127] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,127] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,128] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,129] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:19:55,138] jepsen worker 68 - jepsen.util 4838 :info :write [22 40] :RequestTimeoutException INFO [2022-03-13 17:19:55,139] jepsen worker 68 - jepsen.util 4928 :invoke :read [22 nil] INFO [2022-03-13 17:19:55,150] jepsen worker 14 - jepsen.util 374 :fail :write [4 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,150] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,151] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:55,152] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:55,158] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,159] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:55,161] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,162] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:19:55,184] jepsen worker 36 - jepsen.util 2466 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,184] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:19:55,197] jepsen worker 79 - jepsen.util 1159 :fail :write [26 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,199] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,218] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:55,219] jepsen worker 71 - jepsen.util 611 :fail :write [23 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,243] jepsen worker 13 - jepsen.util 733 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,310] jepsen worker nemesis - couchbase.util Rebalance check count: 701 INFO [2022-03-13 17:19:55,310] jepsen worker 71 - jepsen.util 611 :invoke :write [23 37] INFO [2022-03-13 17:19:55,310] jepsen worker nemesis - couchbase.util Rebalance stuck count: 573 INFO [2022-03-13 17:19:55,259] jepsen worker 49 - jepsen.util 3649 :fail :write [16 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,310] jepsen worker 13 - jepsen.util 733 :invoke :write [4 29] INFO [2022-03-13 17:19:55,310] jepsen worker 49 - jepsen.util 3649 :invoke :write [16 33] INFO [2022-03-13 17:19:55,259] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,259] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,310] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,310] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,259] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,311] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:55,312] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,312] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,313] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,313] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:19:55,314] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,314] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 48] INFO [2022-03-13 17:19:55,264] jepsen worker 24 - jepsen.util 654 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,289] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,310] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,318] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:55,310] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 44] INFO [2022-03-13 17:19:55,312] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,318] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:19:55,318] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,319] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,320] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,321] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,321] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,322] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,323] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,324] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,324] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,324] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,324] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:19:55,330] jepsen worker 42 - jepsen.util 402 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,330] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,330] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:55,335] jepsen worker 69 - jepsen.util 789 :fail :write [23 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,335] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,335] jepsen worker 69 - jepsen.util 789 :invoke :write [23 43] INFO [2022-03-13 17:19:55,337] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,338] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,340] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,341] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:19:55,355] jepsen worker 43 - jepsen.util 673 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,356] jepsen worker 43 - jepsen.util 673 :invoke :write [14 18] INFO [2022-03-13 17:19:55,363] jepsen worker 25 - jepsen.util 565 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,364] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:55,398] jepsen worker 49 - jepsen.util 3649 :fail :write [16 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,399] jepsen worker 49 - jepsen.util 3649 :invoke :write [16 45] INFO [2022-03-13 17:19:55,410] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,410] jepsen worker 77 - jepsen.util 707 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,411] jepsen worker 77 - jepsen.util 707 :invoke :write [25 33] INFO [2022-03-13 17:19:55,412] jepsen worker 78 - jepsen.util 708 :invoke :write [26 5] INFO [2022-03-13 17:19:55,413] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,414] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:55,440] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,441] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:55,443] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,443] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:55,444] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,445] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:55,450] jepsen worker 0 - jepsen.util 4680 :info :write [0 28] :RequestTimeoutException INFO [2022-03-13 17:19:55,451] jepsen worker 4 - jepsen.util 2614 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,451] jepsen worker 0 - jepsen.util 4770 :invoke :write [0 23] INFO [2022-03-13 17:19:55,452] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:19:55,454] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,455] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:19:55,469] jepsen worker 33 - jepsen.util 753 :fail :write [11 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,469] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,469] jepsen worker 71 - jepsen.util 611 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,469] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:55,470] jepsen worker 71 - jepsen.util 611 :invoke :write [23 2] INFO [2022-03-13 17:19:55,471] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:55,491] jepsen worker 75 - jepsen.util 525 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,491] jepsen worker 75 - jepsen.util 525 :invoke :write [25 49] INFO [2022-03-13 17:19:55,508] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,508] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:55,511] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,512] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,513] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,514] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,517] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,517] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,519] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,520] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,521] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,522] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:19:55,538] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,538] jepsen worker 62 - jepsen.util 512 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,538] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:19:55,539] jepsen worker 62 - jepsen.util 512 :invoke :write [20 25] INFO [2022-03-13 17:19:55,556] jepsen worker 35 - jepsen.util 935 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,556] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,556] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,557] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,557] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:19:55,558] jepsen worker 35 - jepsen.util 935 :invoke :write [11 36] INFO [2022-03-13 17:19:55,559] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:55,564] jepsen worker 34 - jepsen.util 934 :invoke :write [11 18] INFO [2022-03-13 17:19:55,568] jepsen worker 69 - jepsen.util 789 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,570] jepsen worker 49 - jepsen.util 3649 :fail :write [16 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,571] jepsen worker 49 - jepsen.util 3649 :invoke :write [16 45] INFO [2022-03-13 17:19:55,571] jepsen worker 69 - jepsen.util 789 :invoke :write [23 11] INFO [2022-03-13 17:19:55,583] jepsen worker 59 - jepsen.util 4829 :ok :write [19 34] INFO [2022-03-13 17:19:55,583] jepsen worker 59 - jepsen.util 4829 :invoke :read [19 nil] INFO [2022-03-13 17:19:55,598] jepsen worker 53 - jepsen.util 593 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,598] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,600] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:55,600] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:55,600] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,600] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,601] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:55,602] jepsen worker 60 - jepsen.util 690 :invoke :write [20 44] INFO [2022-03-13 17:19:55,610] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,610] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,611] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,611] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:55,623] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,624] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:55,625] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,625] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:19:55,647] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,648] jepsen worker 23 - jepsen.util 563 :invoke :write [7 44] INFO [2022-03-13 17:19:55,658] jepsen worker 71 - jepsen.util 611 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,659] jepsen worker 71 - jepsen.util 611 :invoke :write [23 4] INFO [2022-03-13 17:19:55,662] jepsen worker 80 - jepsen.util 800 :fail :write [26 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,662] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,662] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,662] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,663] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,663] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,664] jepsen worker 33 - jepsen.util 753 :invoke :write [11 49] INFO [2022-03-13 17:19:55,665] jepsen worker 25 - jepsen.util 565 :invoke :write [8 6] INFO [2022-03-13 17:19:55,666] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:55,667] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:55,667] jepsen worker 80 - jepsen.util 800 :invoke :write [26 42] INFO [2022-03-13 17:19:55,668] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:55,674] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,675] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,699] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,699] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,701] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,701] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:55,726] jepsen worker 49 - jepsen.util 3649 :fail :write [16 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,726] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,727] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,727] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,728] jepsen worker 49 - jepsen.util 3649 :invoke :read [16 nil] INFO [2022-03-13 17:19:55,729] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:19:55,729] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,730] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,733] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,733] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,733] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,734] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:19:55,739] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,740] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,741] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,742] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,743] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,743] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,744] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,744] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,745] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,746] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,747] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,747] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,748] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,749] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 24] INFO [2022-03-13 17:19:55,757] jepsen worker 82 - jepsen.util 5122 :info :write [27 23] :RequestTimeoutException INFO [2022-03-13 17:19:55,758] jepsen worker 82 - jepsen.util 5212 :invoke :write [27 34] INFO [2022-03-13 17:19:55,767] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,768] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:19:55,770] jepsen worker 76 - jepsen.util 796 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,771] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:55,803] jepsen worker 69 - jepsen.util 789 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,803] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:55,829] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,829] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,866] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,867] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:55,873] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,873] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:19:55,921] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,921] jepsen worker 4 - jepsen.util 2614 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,921] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,922] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,947] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,948] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:19:55,961] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,961] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,961] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,961] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:19:55,962] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,963] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,963] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,964] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,964] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,965] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,965] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:55,965] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 29] INFO [2022-03-13 17:19:55,975] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,976] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:19:55,990] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:55,990] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:19:56,058] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,058] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:56,059] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,060] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:56,061] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,061] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:19:56,089] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,090] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:19:56,141] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,142] jepsen worker 3 - jepsen.util 2253 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,142] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:19:56,143] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:19:56,188] jepsen worker 34 - jepsen.util 934 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,189] jepsen worker 34 - jepsen.util 934 :invoke :write [11 12] INFO [2022-03-13 17:19:56,199] jepsen worker 52 - jepsen.util 682 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,200] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:56,201] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,201] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,203] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,203] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,204] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:19:56,204] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:56,208] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,209] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:19:56,231] jepsen worker 71 - jepsen.util 611 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,231] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,232] jepsen worker 44 - jepsen.util 764 :fail :write [14 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,232] jepsen worker 79 - jepsen.util 1159 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,233] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,233] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,233] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,233] jepsen worker 69 - jepsen.util 789 :invoke :write [23 6] INFO [2022-03-13 17:19:56,234] jepsen worker 44 - jepsen.util 764 :invoke :write [14 4] INFO [2022-03-13 17:19:56,236] jepsen worker 8 - jepsen.util 728 :invoke :write [2 40] INFO [2022-03-13 17:19:56,238] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 48] INFO [2022-03-13 17:19:56,239] jepsen worker 71 - jepsen.util 611 :invoke :write [23 28] INFO [2022-03-13 17:19:56,239] jepsen worker 61 - jepsen.util 781 :invoke :write [20 41] INFO [2022-03-13 17:19:56,240] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,267] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,268] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:19:56,274] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,275] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,275] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,276] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:56,314] jepsen worker 12 - jepsen.util 912 :fail :write [4 30] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,315] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:56,316] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:56,316] jepsen worker nemesis - couchbase.util Rebalance check count: 702 INFO [2022-03-13 17:19:56,316] jepsen worker nemesis - couchbase.util Rebalance stuck count: 574 INFO [2022-03-13 17:19:56,333] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,333] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,334] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,334] jepsen worker 35 - jepsen.util 935 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,334] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,335] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,335] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:56,336] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:56,338] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,338] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,338] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,338] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,339] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,339] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 35] INFO [2022-03-13 17:19:56,339] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,340] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,342] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,343] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,343] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,344] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,344] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,345] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:19:56,346] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,347] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,348] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,349] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:19:56,352] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,353] jepsen worker 13 - jepsen.util 733 :fail :write [4 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,354] jepsen worker 13 - jepsen.util 733 :invoke :write [4 22] INFO [2022-03-13 17:19:56,354] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,356] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,356] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:19:56,378] jepsen worker 77 - jepsen.util 707 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,378] jepsen worker 69 - jepsen.util 789 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,379] jepsen worker 77 - jepsen.util 707 :invoke :write [25 34] INFO [2022-03-13 17:19:56,383] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,396] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,397] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:19:56,417] jepsen worker 36 - jepsen.util 2466 :fail :write [12 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,418] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:19:56,426] jepsen worker 43 - jepsen.util 673 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,427] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:56,428] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,428] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,428] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,429] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:56,429] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:19:56,430] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:56,437] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,438] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,438] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,438] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,438] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:56,439] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:56,439] jepsen worker 24 - jepsen.util 654 :invoke :write [8 42] INFO [2022-03-13 17:19:56,444] jepsen worker 14 - jepsen.util 374 :invoke :write [4 34] INFO [2022-03-13 17:19:56,476] jepsen worker 71 - jepsen.util 611 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,476] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,477] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,477] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,478] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,479] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,479] jepsen worker 71 - jepsen.util 611 :invoke :write [23 45] INFO [2022-03-13 17:19:56,479] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,481] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,482] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,482] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,482] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,483] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:56,484] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:19:56,490] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,490] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 5] INFO [2022-03-13 17:19:56,495] jepsen worker 7 - jepsen.util 637 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,496] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,496] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:56,496] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:56,496] jepsen worker 6 - jepsen.util 636 :invoke :write [2 12] INFO [2022-03-13 17:19:56,497] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:19:56,543] jepsen worker 33 - jepsen.util 753 :fail :write [11 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,543] jepsen worker 75 - jepsen.util 525 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,543] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,543] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:19:56,544] jepsen worker 76 - jepsen.util 796 :invoke :write [25 3] INFO [2022-03-13 17:19:56,544] jepsen worker 75 - jepsen.util 525 :invoke :write [25 10] INFO [2022-03-13 17:19:56,548] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,548] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,570] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,570] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:56,572] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,572] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:19:56,580] jepsen worker 78 - jepsen.util 708 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,581] jepsen worker 78 - jepsen.util 708 :invoke :write [26 32] INFO [2022-03-13 17:19:56,602] jepsen worker 71 - jepsen.util 611 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,603] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,608] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,609] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:19:56,637] jepsen worker 25 - jepsen.util 565 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,637] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:56,637] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,638] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,638] jepsen worker 60 - jepsen.util 690 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,638] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,638] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:56,639] jepsen worker 26 - jepsen.util 566 :invoke :write [8 7] INFO [2022-03-13 17:19:56,639] jepsen worker 25 - jepsen.util 565 :invoke :write [8 41] INFO [2022-03-13 17:19:56,640] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:19:56,641] jepsen worker 60 - jepsen.util 690 :invoke :write [20 44] INFO [2022-03-13 17:19:56,646] jepsen worker 52 - jepsen.util 682 :invoke :write [17 24] INFO [2022-03-13 17:19:56,659] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,661] jepsen worker 3 - jepsen.util 2253 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,661] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,661] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:56,661] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,662] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,718] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,719] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,719] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,719] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:19:56,720] jepsen worker 62 - jepsen.util 512 :fail :write [20 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,721] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,721] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,721] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,721] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,721] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 14] INFO [2022-03-13 17:19:56,722] jepsen worker 71 - jepsen.util 611 :invoke :write [23 15] INFO [2022-03-13 17:19:56,723] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:56,723] jepsen worker 70 - jepsen.util 880 :invoke :write [23 44] INFO [2022-03-13 17:19:56,725] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,725] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,726] jepsen worker 69 - jepsen.util 789 :invoke :write [23 39] INFO [2022-03-13 17:19:56,728] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,729] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:19:56,734] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,735] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:56,757] jepsen worker 80 - jepsen.util 800 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,758] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:56,787] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,788] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 24] INFO [2022-03-13 17:19:56,790] jepsen worker 13 - jepsen.util 733 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,791] jepsen worker 13 - jepsen.util 733 :invoke :write [4 12] INFO [2022-03-13 17:19:56,799] jepsen worker 23 - jepsen.util 563 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,800] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:56,811] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,811] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:19:56,849] jepsen worker 4 - jepsen.util 2614 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,851] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:56,875] jepsen worker 69 - jepsen.util 789 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,876] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:56,895] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,895] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:56,898] jepsen worker 34 - jepsen.util 934 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,898] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,898] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,899] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,899] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,899] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:56,900] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:19:56,900] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,901] jepsen worker 21 - jepsen.util 831 :invoke :write [7 16] INFO [2022-03-13 17:19:56,901] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:56,902] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:56,902] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,903] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:56,903] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:19:56,910] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,911] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 15] INFO [2022-03-13 17:19:56,918] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,918] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:56,919] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:56,919] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,919] jepsen worker 43 - jepsen.util 673 :invoke :write [14 18] INFO [2022-03-13 17:19:56,920] jepsen worker 53 - jepsen.util 593 :invoke :write [17 42] INFO [2022-03-13 17:19:56,961] jepsen worker 36 - jepsen.util 2466 :fail :write [12 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,962] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,962] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:56,963] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 45] INFO [2022-03-13 17:19:56,987] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:56,988] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:57,004] jepsen worker 71 - jepsen.util 611 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,004] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,005] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:57,006] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:57,011] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,012] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,012] jepsen worker 71 - jepsen.util 611 :invoke :write [23 21] INFO [2022-03-13 17:19:57,013] jepsen worker 69 - jepsen.util 789 :invoke :write [23 12] INFO [2022-03-13 17:19:57,026] jepsen worker 3 - jepsen.util 2253 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,027] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,027] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 8] INFO [2022-03-13 17:19:57,027] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:57,050] jepsen worker 24 - jepsen.util 654 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,051] jepsen worker 24 - jepsen.util 654 :invoke :write [8 7] INFO [2022-03-13 17:19:57,058] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,059] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:19:57,073] jepsen worker 75 - jepsen.util 525 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,074] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:57,075] jepsen worker 79 - jepsen.util 1159 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,076] jepsen worker 44 - jepsen.util 764 :fail :write [14 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,076] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:57,076] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:57,086] jepsen worker 4 - jepsen.util 2614 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,087] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,088] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,088] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 26] INFO [2022-03-13 17:19:57,136] jepsen worker 69 - jepsen.util 789 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,136] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:57,137] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,137] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,142] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,143] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,143] jepsen worker 3 - jepsen.util 2253 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,143] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:19:57,144] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,144] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] WARN [2022-03-13 17:19:57,149] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9044062,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9545386,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9294520,"encode_duration_us":17147,"operation_id":"0x147e4","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10000796},{"operation_name":"upsert","last_dispatch_duration_us":18084232,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26715829,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9124540,"encode_duration_us":25801,"operation_id":"0x13c23","timeout_ms":10000,"last_server_duration_us":9104337,"total_duration_us":10003483}],"total_count":2}} INFO [2022-03-13 17:19:57,151] jepsen worker 60 - jepsen.util 690 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,152] jepsen worker 60 - jepsen.util 690 :invoke :write [20 29] INFO [2022-03-13 17:19:57,160] jepsen worker 14 - jepsen.util 374 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,160] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,161] jepsen worker 14 - jepsen.util 374 :invoke :write [4 36] INFO [2022-03-13 17:19:57,164] jepsen worker 33 - jepsen.util 753 :invoke :write [11 46] INFO [2022-03-13 17:19:57,199] jepsen worker 36 - jepsen.util 2466 :fail :write [12 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,199] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,200] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:57,200] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,210] jepsen worker 71 - jepsen.util 611 :fail :write [23 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,211] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,211] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:19:57,212] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:57,219] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,219] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:57,222] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,223] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:19:57,243] jepsen worker 61 - jepsen.util 781 :fail :write [20 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,244] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,244] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:19:57,244] jepsen worker 8 - jepsen.util 728 :fail :write [2 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,244] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,245] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:57,246] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:19:57,246] jepsen worker 8 - jepsen.util 728 :invoke :write [2 6] INFO [2022-03-13 17:19:57,247] jepsen worker 42 - jepsen.util 402 :invoke :write [14 44] INFO [2022-03-13 17:19:57,247] jepsen worker 61 - jepsen.util 781 :invoke :write [20 28] INFO [2022-03-13 17:19:57,254] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,254] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,254] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,256] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:19:57,257] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,257] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:19:57,258] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,259] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,261] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,261] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:19:57,264] jepsen worker 76 - jepsen.util 796 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,265] jepsen worker 76 - jepsen.util 796 :invoke :write [25 9] INFO [2022-03-13 17:19:57,274] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,274] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,274] jepsen worker 25 - jepsen.util 565 :fail :write [8 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,275] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,275] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,275] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:57,276] jepsen worker 51 - jepsen.util 771 :invoke :write [17 44] INFO [2022-03-13 17:19:57,277] jepsen worker 34 - jepsen.util 934 :invoke :write [11 47] INFO [2022-03-13 17:19:57,278] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:57,284] jepsen worker 35 - jepsen.util 935 :invoke :write [11 39] INFO [2022-03-13 17:19:57,285] jepsen worker 6 - jepsen.util 636 :fail :write [2 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,286] jepsen worker 6 - jepsen.util 636 :invoke :write [2 8] INFO [2022-03-13 17:19:57,286] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,288] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,288] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,289] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,290] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,291] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:19:57,291] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,292] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,293] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,294] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,295] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,296] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,297] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,298] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:19:57,321] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:57,321] jepsen worker nemesis - couchbase.util Rebalance check count: 703 INFO [2022-03-13 17:19:57,321] jepsen worker nemesis - couchbase.util Rebalance stuck count: 575 INFO [2022-03-13 17:19:57,337] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,337] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 37] INFO [2022-03-13 17:19:57,338] jepsen worker 26 - jepsen.util 566 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,338] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:57,341] jepsen worker 77 - jepsen.util 707 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,341] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] WARN [2022-03-13 17:19:57,354] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8297869,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9096181,"total_server_duration_us":470073,"encode_duration_us":21921,"operation_id":"0x1462d","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10008648},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","encode_duration_us":20465,"operation_id":"0x146f2","timeout_ms":10000,"total_duration_us":10007499},{"operation_name":"upsert","last_dispatch_duration_us":8329958,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9004406,"total_server_duration_us":449972,"encode_duration_us":31654,"operation_id":"0x14711","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10006552},{"operation_name":"upsert","last_dispatch_duration_us":8676723,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8917526,"total_server_duration_us":166134,"encode_duration_us":19638,"operation_id":"0x14902","timeout_ms":10000,"last_server_duration_us":21001,"total_duration_us":10005923},{"operation_name":"upsert","last_dispatch_duration_us":49620,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9943962,"total_server_duration_us":239202,"encode_duration_us":19690,"operation_id":"0x147a6","timeout_ms":10000,"last_server_duration_us":34811,"total_duration_us":10002179},{"operation_name":"upsert","last_dispatch_duration_us":10888232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":10888232,"total_server_duration_us":31108,"encode_duration_us":25773,"operation_id":"0x146c3","timeout_ms":10000,"last_server_duration_us":31108,"total_duration_us":10000923},{"operation_name":"upsert","last_dispatch_duration_us":9044062,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9545386,"total_server_duration_us":9294520,"encode_duration_us":17147,"operation_id":"0x147e4","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":10000796},{"operation_name":"get","last_dispatch_duration_us":4911431,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":4911431,"total_server_duration_us":6530,"operation_id":"0x14b74","timeout_ms":30000,"last_server_duration_us":6530,"total_duration_us":4914013},{"operation_name":"upsert","last_dispatch_duration_us":437746,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":437746,"total_server_duration_us":67882,"encode_duration_us":28971,"operation_id":"0x14e0b","timeout_ms":10000,"last_server_duration_us":67882,"total_duration_us":2040902},{"operation_name":"upsert","last_dispatch_duration_us":979487,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1454378,"total_server_duration_us":133266,"encode_duration_us":22077,"operation_id":"0x14de9","timeout_ms":10000,"last_server_duration_us":67882,"total_duration_us":1763560}],"total_count":96}} INFO [2022-03-13 17:19:57,370] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,371] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,375] jepsen worker 13 - jepsen.util 733 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,376] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,376] jepsen worker 75 - jepsen.util 525 :invoke :write [25 4] INFO [2022-03-13 17:19:57,378] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:57,398] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,399] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,400] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,401] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:19:57,410] jepsen worker 52 - jepsen.util 682 :fail :write [17 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,410] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,411] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:57,411] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:57,413] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,414] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:19:57,445] jepsen worker 2 - jepsen.util 5222 :ok :read [0 41] INFO [2022-03-13 17:19:57,446] jepsen worker 2 - jepsen.util 5222 :invoke :read [0 nil] INFO [2022-03-13 17:19:57,448] jepsen worker 21 - jepsen.util 831 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,449] jepsen worker 21 - jepsen.util 831 :invoke :write [7 6] INFO [2022-03-13 17:19:57,455] jepsen worker 76 - jepsen.util 796 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,455] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,456] jepsen worker 13 - jepsen.util 733 :invoke :write [4 7] INFO [2022-03-13 17:19:57,456] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:57,473] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,473] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,473] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,474] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:57,527] jepsen worker 43 - jepsen.util 673 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,528] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:57,535] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,536] jepsen worker 69 - jepsen.util 789 :invoke :write [23 48] INFO [2022-03-13 17:19:57,541] jepsen worker 87 - jepsen.util 2877 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,542] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:19:57,548] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,548] jepsen worker 70 - jepsen.util 880 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,548] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,548] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:57,549] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:57,549] jepsen worker 44 - jepsen.util 764 :invoke :write [14 27] INFO [2022-03-13 17:19:57,550] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:57,551] jepsen worker 12 - jepsen.util 912 :invoke :write [4 42] INFO [2022-03-13 17:19:57,567] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,567] jepsen worker 75 - jepsen.util 525 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,568] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:57,569] jepsen worker 75 - jepsen.util 525 :invoke :write [25 48] INFO [2022-03-13 17:19:57,596] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,597] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:19:57,624] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,624] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:57,625] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,626] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:19:57,627] jepsen worker 53 - jepsen.util 593 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,628] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,628] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,628] jepsen worker 53 - jepsen.util 593 :invoke :write [17 4] INFO [2022-03-13 17:19:57,629] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:57,629] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,633] jepsen worker 78 - jepsen.util 708 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,633] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,634] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,635] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:57,635] jepsen worker 23 - jepsen.util 563 :invoke :write [7 45] INFO [2022-03-13 17:19:57,636] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:57,639] jepsen worker 24 - jepsen.util 654 :fail :write [8 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,640] jepsen worker 24 - jepsen.util 654 :invoke :write [8 6] INFO [2022-03-13 17:19:57,654] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,655] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:19:57,713] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,714] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:19:57,717] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,718] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,718] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,719] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:19:57,721] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:19:57,730] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:57,732] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,732] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:19:57,744] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,744] jepsen worker 13 - jepsen.util 733 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,746] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:19:57,747] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:57,752] jepsen worker 14 - jepsen.util 374 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,753] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:57,767] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,767] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,768] jepsen worker 71 - jepsen.util 611 :invoke :write [23 23] INFO [2022-03-13 17:19:57,768] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:57,810] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,810] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,811] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:19:57,811] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:19:57,824] jepsen worker 60 - jepsen.util 690 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,824] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,824] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,824] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,825] jepsen worker 61 - jepsen.util 781 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,825] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,826] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:57,827] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:19:57,827] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:57,828] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,828] jepsen worker 8 - jepsen.util 728 :fail :write [2 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,828] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,836] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:57,837] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,837] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:57,839] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 23] INFO [2022-03-13 17:19:57,840] jepsen worker 43 - jepsen.util 673 :invoke :write [14 49] INFO [2022-03-13 17:19:57,841] jepsen worker 78 - jepsen.util 708 :invoke :write [26 1] INFO [2022-03-13 17:19:57,844] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:19:57,844] jepsen worker 52 - jepsen.util 682 :invoke :write [17 18] INFO [2022-03-13 17:19:57,844] jepsen worker 60 - jepsen.util 690 :invoke :write [20 37] INFO [2022-03-13 17:19:57,846] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:57,896] jepsen worker 33 - jepsen.util 753 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,896] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,896] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,897] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:19:57,898] jepsen worker 25 - jepsen.util 565 :invoke :write [8 45] INFO [2022-03-13 17:19:57,898] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:19:57,904] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,905] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:19:57,931] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,931] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:19:57,934] jepsen worker 75 - jepsen.util 525 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,934] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,935] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:57,935] jepsen worker 13 - jepsen.util 733 :invoke :write [4 28] INFO [2022-03-13 17:19:57,969] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,970] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 43] INFO [2022-03-13 17:19:57,973] jepsen worker 42 - jepsen.util 402 :fail :write [14 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,973] jepsen worker 23 - jepsen.util 563 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,973] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,973] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,973] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,973] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,975] jepsen worker 42 - jepsen.util 402 :invoke :write [14 5] INFO [2022-03-13 17:19:57,976] jepsen worker 61 - jepsen.util 781 :invoke :write [20 45] INFO [2022-03-13 17:19:57,977] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:57,978] jepsen worker 77 - jepsen.util 707 :invoke :write [25 42] INFO [2022-03-13 17:19:57,978] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:57,979] jepsen worker 62 - jepsen.util 512 :invoke :write [20 32] INFO [2022-03-13 17:19:57,988] jepsen worker 53 - jepsen.util 593 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,988] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,988] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:57,989] jepsen worker 53 - jepsen.util 593 :invoke :write [17 33] INFO [2022-03-13 17:19:57,990] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:19:57,993] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:58,018] jepsen worker 34 - jepsen.util 934 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,019] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:58,042] jepsen worker 3 - jepsen.util 2253 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,043] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:19:58,050] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,051] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 46] INFO [2022-03-13 17:19:58,054] jepsen worker 58 - jepsen.util 4378 :ok :read [19 34] INFO [2022-03-13 17:19:58,055] jepsen worker 58 - jepsen.util 4378 :invoke :write [19 0] INFO [2022-03-13 17:19:58,059] jepsen worker 6 - jepsen.util 636 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,060] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,060] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,060] jepsen worker 6 - jepsen.util 636 :invoke :write [2 29] INFO [2022-03-13 17:19:58,061] jepsen worker 7 - jepsen.util 637 :invoke :write [2 29] INFO [2022-03-13 17:19:58,061] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:58,100] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,100] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,101] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:58,101] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:58,102] jepsen worker 78 - jepsen.util 708 :fail :write [26 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,102] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,103] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,103] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,104] jepsen worker 23 - jepsen.util 563 :invoke :write [7 16] INFO [2022-03-13 17:19:58,106] jepsen worker 78 - jepsen.util 708 :invoke :write [26 19] INFO [2022-03-13 17:19:58,106] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,106] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:58,138] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,139] jepsen worker 22 - jepsen.util 652 :invoke :write [7 22] INFO [2022-03-13 17:19:58,140] jepsen worker 36 - jepsen.util 2466 :fail :write [12 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,140] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:19:58,164] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,165] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:19:58,180] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,180] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,181] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:19:58,181] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,211] jepsen worker 35 - jepsen.util 935 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,212] jepsen worker 35 - jepsen.util 935 :invoke :write [11 5] INFO [2022-03-13 17:19:58,226] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,226] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,227] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:19:58,227] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,232] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,232] jepsen worker 13 - jepsen.util 733 :fail :write [4 28] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,232] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,234] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:58,234] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:58,235] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:58,280] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,280] jepsen worker 43 - jepsen.util 673 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,281] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,281] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,281] jepsen worker 44 - jepsen.util 764 :fail :write [14 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,282] jepsen worker 76 - jepsen.util 796 :invoke :write [25 24] INFO [2022-03-13 17:19:58,283] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:58,284] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:58,285] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:19:58,286] jepsen worker 13 - jepsen.util 733 :invoke :write [4 36] INFO [2022-03-13 17:19:58,293] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,294] jepsen worker 75 - jepsen.util 525 :invoke :write [25 35] INFO [2022-03-13 17:19:58,296] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,296] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,297] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:19:58,297] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 29] INFO [2022-03-13 17:19:58,308] jepsen worker 87 - jepsen.util 2877 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,308] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:19:58,318] jepsen worker 51 - jepsen.util 771 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,318] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:58,327] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:58,327] jepsen worker nemesis - couchbase.util Rebalance check count: 704 INFO [2022-03-13 17:19:58,327] jepsen worker nemesis - couchbase.util Rebalance stuck count: 576 INFO [2022-03-13 17:19:58,359] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,360] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:58,377] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,378] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:19:58,388] jepsen worker 52 - jepsen.util 682 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,389] jepsen worker 52 - jepsen.util 682 :invoke :write [17 13] INFO [2022-03-13 17:19:58,398] jepsen worker 61 - jepsen.util 781 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,398] jepsen worker 61 - jepsen.util 781 :invoke :write [20 39] INFO [2022-03-13 17:19:58,429] jepsen worker 3 - jepsen.util 2253 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,429] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:58,442] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,443] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,444] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,445] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:19:58,472] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,472] jepsen worker 60 - jepsen.util 690 :fail :write [20 37] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,473] jepsen worker 26 - jepsen.util 566 :invoke :write [8 48] INFO [2022-03-13 17:19:58,473] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:58,476] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,476] jepsen worker 21 - jepsen.util 831 :fail :write [7 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,478] jepsen worker 14 - jepsen.util 374 :invoke :write [4 23] INFO [2022-03-13 17:19:58,478] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:19:58,489] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,489] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,490] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,490] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 39] INFO [2022-03-13 17:19:58,514] jepsen worker 69 - jepsen.util 789 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,515] jepsen worker 69 - jepsen.util 789 :invoke :write [23 13] INFO [2022-03-13 17:19:58,522] jepsen worker 76 - jepsen.util 796 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,523] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:19:58,535] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,536] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] WARN [2022-03-13 17:19:58,545] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9066789,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10393929,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9347049,"encode_duration_us":26054,"operation_id":"0x148e1","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10002775},{"operation_name":"upsert","last_dispatch_duration_us":18094974,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26665813,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9175552,"encode_duration_us":21308,"operation_id":"0x13c38","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10061977}],"total_count":2}} INFO [2022-03-13 17:19:58,555] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,555] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,556] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:19:58,557] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:19:58,563] jepsen worker 24 - jepsen.util 654 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,563] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:58,571] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,572] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:19:58,580] jepsen worker 12 - jepsen.util 912 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,580] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,580] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:58,581] jepsen worker 12 - jepsen.util 912 :invoke :write [4 35] INFO [2022-03-13 17:19:58,596] jepsen worker 81 - jepsen.util 5481 :info :write [27 22] :RequestTimeoutException INFO [2022-03-13 17:19:58,597] jepsen worker 81 - jepsen.util 5571 :invoke :write [27 11] INFO [2022-03-13 17:19:58,614] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,615] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:19:58,616] jepsen worker 41 - jepsen.util 4811 :info :write [13 42] :RequestTimeoutException INFO [2022-03-13 17:19:58,617] jepsen worker 41 - jepsen.util 4901 :invoke :write [13 22] INFO [2022-03-13 17:19:58,658] jepsen worker 36 - jepsen.util 2466 :fail :write [12 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,659] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:58,667] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,667] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,672] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,673] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,674] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,674] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:19:58,680] jepsen worker 77 - jepsen.util 707 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,680] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,680] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:58,682] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:58,691] jepsen worker 25 - jepsen.util 565 :fail :write [8 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,692] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,692] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,692] jepsen worker 25 - jepsen.util 565 :invoke :write [8 13] INFO [2022-03-13 17:19:58,694] jepsen worker 34 - jepsen.util 934 :invoke :write [11 18] INFO [2022-03-13 17:19:58,694] jepsen worker 80 - jepsen.util 800 :invoke :write [26 24] INFO [2022-03-13 17:19:58,725] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,725] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] WARN [2022-03-13 17:19:58,739] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":87961,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":6367038,"total_server_duration_us":87177,"encode_duration_us":17329,"operation_id":"0x1482d","timeout_ms":10000,"last_server_duration_us":35011,"total_duration_us":10006854},{"operation_name":"upsert","last_dispatch_duration_us":17721907,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":18280348,"total_server_duration_us":9107114,"encode_duration_us":18864,"operation_id":"0x145a9","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10006428},{"operation_name":"upsert","last_dispatch_duration_us":8690539,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9471520,"total_server_duration_us":141960,"encode_duration_us":21211,"operation_id":"0x1459d","timeout_ms":10000,"last_server_duration_us":24755,"total_duration_us":10004048},{"operation_name":"upsert","last_dispatch_duration_us":14784547,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":14784547,"total_server_duration_us":26241,"encode_duration_us":17379,"operation_id":"0x147da","timeout_ms":10000,"last_server_duration_us":26241,"total_duration_us":10003900},{"operation_name":"get","last_dispatch_duration_us":9021926,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9021926,"total_server_duration_us":27,"operation_id":"0x146e1","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":9025006},{"operation_name":"get","last_dispatch_duration_us":8692992,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8692992,"total_server_duration_us":1406,"operation_id":"0x14715","timeout_ms":30000,"last_server_duration_us":1406,"total_duration_us":8693344},{"operation_name":"get","last_dispatch_duration_us":8439309,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8439309,"total_server_duration_us":100,"operation_id":"0x1475a","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":8443240},{"operation_name":"get","last_dispatch_duration_us":8142364,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8142364,"total_server_duration_us":100,"operation_id":"0x14b2c","timeout_ms":30000,"last_server_duration_us":100,"total_duration_us":8146178},{"operation_name":"upsert","last_dispatch_duration_us":1440137,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1440137,"total_server_duration_us":73136,"encode_duration_us":21000,"operation_id":"0x14e17","timeout_ms":10000,"last_server_duration_us":73136,"total_duration_us":1445800},{"operation_name":"get","last_dispatch_duration_us":1382859,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":1382859,"total_server_duration_us":1382290,"operation_id":"0x14e14","timeout_ms":30000,"last_server_duration_us":1382290,"total_duration_us":1383194}],"total_count":21}} INFO [2022-03-13 17:19:58,741] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,742] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] WARN [2022-03-13 17:19:58,746] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16213486,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":16213486,"total_server_duration_us":275013,"operation_id":"0x142cc","timeout_ms":30000,"last_server_duration_us":275013,"total_duration_us":16213768},{"operation_name":"upsert","last_dispatch_duration_us":7079594,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":12263159,"total_server_duration_us":419270,"encode_duration_us":26990,"operation_id":"0x14710","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10008231},{"operation_name":"upsert","last_dispatch_duration_us":7116160,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8343867,"total_server_duration_us":397419,"encode_duration_us":22235,"operation_id":"0x14934","timeout_ms":10000,"last_server_duration_us":37963,"total_duration_us":10007630},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","encode_duration_us":20774,"operation_id":"0x147b4","timeout_ms":10000,"total_duration_us":10004067},{"operation_name":"upsert","last_dispatch_duration_us":7079447,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10869727,"total_server_duration_us":456862,"encode_duration_us":16869,"operation_id":"0x147e5","timeout_ms":10000,"last_server_duration_us":21082,"total_duration_us":10003997},{"operation_name":"upsert","last_dispatch_duration_us":36382,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":7028962,"total_server_duration_us":369814,"encode_duration_us":21794,"operation_id":"0x145dc","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10002973},{"operation_name":"upsert","last_dispatch_duration_us":9066789,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10393929,"total_server_duration_us":9347049,"encode_duration_us":26054,"operation_id":"0x148e1","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10002775},{"operation_name":"get","last_dispatch_duration_us":9852946,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9852946,"total_server_duration_us":23,"operation_id":"0x146da","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":9855469},{"operation_name":"get","last_dispatch_duration_us":105110,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":105110,"total_server_duration_us":32,"operation_id":"0x14d70","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":5274752},{"operation_name":"upsert","last_dispatch_duration_us":1165878,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2197944,"total_server_duration_us":139839,"encode_duration_us":26901,"operation_id":"0x14d6b","timeout_ms":10000,"last_server_duration_us":75755,"total_duration_us":2510357}],"total_count":53}} INFO [2022-03-13 17:19:58,748] jepsen worker 13 - jepsen.util 733 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,749] jepsen worker 53 - jepsen.util 593 :fail :write [17 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,749] jepsen worker 13 - jepsen.util 733 :invoke :write [4 31] INFO [2022-03-13 17:19:58,750] jepsen worker 53 - jepsen.util 593 :invoke :write [17 18] INFO [2022-03-13 17:19:58,755] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,755] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,755] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,756] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:19:58,756] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:58,757] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] WARN [2022-03-13 17:19:58,762] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":27324762,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":36532656,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9174765,"encode_duration_us":18273,"operation_id":"0x13598","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":10007078},{"operation_name":"upsert","last_dispatch_duration_us":15905595,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15905595,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9047983,"encode_duration_us":19039,"operation_id":"0x143a0","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10008452}],"total_count":2}} INFO [2022-03-13 17:19:58,780] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,780] jepsen worker 33 - jepsen.util 753 :invoke :write [11 25] INFO [2022-03-13 17:19:58,814] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,814] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,815] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,816] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] WARN [2022-03-13 17:19:58,833] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7272989,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":7272989,"total_server_duration_us":19340,"encode_duration_us":17473,"operation_id":"0x14814","timeout_ms":10000,"last_server_duration_us":19340,"total_duration_us":10010142},{"operation_name":"upsert","last_dispatch_duration_us":11496017,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":11496017,"total_server_duration_us":9042674,"encode_duration_us":18963,"operation_id":"0x145db","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10007391},{"operation_name":"upsert","last_dispatch_duration_us":18163783,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":18163783,"total_server_duration_us":9036305,"encode_duration_us":24867,"operation_id":"0x1467f","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10006754},{"operation_name":"upsert","last_dispatch_duration_us":18183684,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":18183684,"total_server_duration_us":9030999,"encode_duration_us":30164,"operation_id":"0x14681","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10006101},{"operation_name":"upsert","last_dispatch_duration_us":9154560,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9154560,"total_server_duration_us":33918,"encode_duration_us":17519,"operation_id":"0x147ba","timeout_ms":10000,"last_server_duration_us":33918,"total_duration_us":10002943},{"operation_name":"upsert","last_dispatch_duration_us":17201057,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":17201057,"total_server_duration_us":25537,"encode_duration_us":34365,"operation_id":"0x1471b","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10002634},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","encode_duration_us":17480,"operation_id":"0x1488f","timeout_ms":10000,"total_duration_us":10001663},{"operation_name":"upsert","last_dispatch_duration_us":9081776,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9081776,"total_server_duration_us":9063917,"encode_duration_us":25469,"operation_id":"0x14794","timeout_ms":10000,"last_server_duration_us":9063917,"total_duration_us":9087964},{"operation_name":"upsert","last_dispatch_duration_us":1654629,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2434279,"total_server_duration_us":141249,"encode_duration_us":22443,"operation_id":"0x14db9","timeout_ms":10000,"last_server_duration_us":70018,"total_duration_us":2443742},{"operation_name":"get","last_dispatch_duration_us":2127034,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":2127034,"total_server_duration_us":43,"operation_id":"0x14aff","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":2132958}],"total_count":68}} WARN [2022-03-13 17:19:58,839] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17390048,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17748973,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9109749,"encode_duration_us":19271,"operation_id":"0x14138","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":10003864},{"operation_name":"upsert","last_dispatch_duration_us":17721907,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18280348,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9107114,"encode_duration_us":18864,"operation_id":"0x145a9","timeout_ms":10000,"last_server_duration_us":9032060,"total_duration_us":10006428}],"total_count":2}} INFO [2022-03-13 17:19:58,847] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException WARN [2022-03-13 17:19:58,847] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":36857,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":8351092,"total_server_duration_us":287167,"encode_duration_us":18074,"operation_id":"0x1470b","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10010750},{"operation_name":"upsert","last_dispatch_duration_us":80533,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1259930,"total_server_duration_us":365962,"encode_duration_us":21463,"operation_id":"0x14acf","timeout_ms":10000,"last_server_duration_us":52539,"total_duration_us":10010606},{"operation_name":"upsert","last_dispatch_duration_us":79202,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8359770,"total_server_duration_us":383681,"encode_duration_us":23105,"operation_id":"0x147ad","timeout_ms":10000,"last_server_duration_us":54335,"total_duration_us":10009210},{"operation_name":"upsert","last_dispatch_duration_us":89066,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9732692,"total_server_duration_us":232551,"encode_duration_us":25756,"operation_id":"0x1462a","timeout_ms":10000,"last_server_duration_us":22637,"total_duration_us":10008373},{"operation_name":"upsert","last_dispatch_duration_us":33018,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":1248092,"total_server_duration_us":289564,"encode_duration_us":17191,"operation_id":"0x14ad3","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10006349},{"operation_name":"upsert","last_dispatch_duration_us":61888,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8014689,"total_server_duration_us":339206,"encode_duration_us":31758,"operation_id":"0x14720","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10002507},{"operation_name":"get","last_dispatch_duration_us":7783079,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":7783079,"total_server_duration_us":1356,"operation_id":"0x147e1","timeout_ms":30000,"last_server_duration_us":1356,"total_duration_us":7783430},{"operation_name":"upsert","last_dispatch_duration_us":4573958,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4573958,"total_server_duration_us":4559261,"encode_duration_us":21432,"operation_id":"0x14c4d","timeout_ms":10000,"last_server_duration_us":4559261,"total_duration_us":4577461},{"operation_name":"upsert","last_dispatch_duration_us":1102576,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1844708,"total_server_duration_us":131308,"encode_duration_us":29389,"operation_id":"0x14dd4","timeout_ms":10000,"last_server_duration_us":63955,"total_duration_us":1860893},{"operation_name":"upsert","last_dispatch_duration_us":1141604,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1603831,"total_server_duration_us":462847,"encode_duration_us":21651,"operation_id":"0x14dd1","timeout_ms":10000,"last_server_duration_us":355900,"total_duration_us":1620483}],"total_count":33}} INFO [2022-03-13 17:19:58,847] jepsen worker 22 - jepsen.util 652 :fail :write [7 22] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,848] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:58,848] jepsen worker 75 - jepsen.util 525 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,848] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,848] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,848] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:19:58,849] jepsen worker 75 - jepsen.util 525 :invoke :write [25 39] INFO [2022-03-13 17:19:58,849] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:58,850] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,850] jepsen worker 76 - jepsen.util 796 :invoke :write [25 44] INFO [2022-03-13 17:19:58,856] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,858] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,859] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:19:58,865] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,866] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,883] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,884] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,885] jepsen worker 78 - jepsen.util 708 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,885] jepsen worker 42 - jepsen.util 402 :fail :write [14 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,886] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,886] jepsen worker 42 - jepsen.util 402 :invoke :write [14 48] INFO [2022-03-13 17:19:58,887] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:19:58,888] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,891] jepsen worker 52 - jepsen.util 682 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,891] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,892] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:58,893] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:19:58,893] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,894] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:19:58,920] jepsen worker 35 - jepsen.util 935 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,920] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,920] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,921] jepsen worker 21 - jepsen.util 831 :invoke :write [7 42] INFO [2022-03-13 17:19:58,922] jepsen worker 35 - jepsen.util 935 :invoke :write [11 40] INFO [2022-03-13 17:19:58,922] jepsen worker 51 - jepsen.util 771 :invoke :write [17 44] INFO [2022-03-13 17:19:58,933] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,933] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] WARN [2022-03-13 17:19:58,934] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16293447,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16293447,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9033121,"encode_duration_us":19192,"operation_id":"0x14232","timeout_ms":10000,"last_server_duration_us":9033121,"total_duration_us":10003512},{"operation_name":"upsert","last_dispatch_duration_us":18183684,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18183684,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9030999,"encode_duration_us":30164,"operation_id":"0x14681","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10006101},{"operation_name":"upsert","last_dispatch_duration_us":18163783,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18163783,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9036305,"encode_duration_us":24867,"operation_id":"0x1467f","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10006754},{"operation_name":"upsert","last_dispatch_duration_us":11496017,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":11496017,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9042674,"encode_duration_us":18963,"operation_id":"0x145db","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10007391}],"total_count":4}} INFO [2022-03-13 17:19:58,940] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,941] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:58,942] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,943] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:19:58,961] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,961] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] WARN [2022-03-13 17:19:58,962] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":8859418,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8859418,"total_server_duration_us":639,"operation_id":"0x13be3","timeout_ms":30000,"last_server_duration_us":639,"total_duration_us":27285429},{"operation_name":"get","last_dispatch_duration_us":8890256,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8890256,"total_server_duration_us":23,"operation_id":"0x140e3","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":18647915},{"operation_name":"get","last_dispatch_duration_us":8929822,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8929822,"total_server_duration_us":28127,"operation_id":"0x14242","timeout_ms":30000,"last_server_duration_us":28127,"total_duration_us":16643181},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":17706,"operation_id":"0x147be","timeout_ms":10000,"total_duration_us":10004046},{"operation_name":"get","last_dispatch_duration_us":9062661,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9062661,"total_server_duration_us":28036,"operation_id":"0x146d3","timeout_ms":30000,"last_server_duration_us":28036,"total_duration_us":9065219},{"operation_name":"get","last_dispatch_duration_us":8480127,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8480127,"total_server_duration_us":56,"operation_id":"0x1471e","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":8483301},{"operation_name":"upsert","last_dispatch_duration_us":1994697,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":3277289,"total_server_duration_us":497504,"encode_duration_us":19868,"operation_id":"0x14d74","timeout_ms":10000,"last_server_duration_us":416393,"total_duration_us":3461433},{"operation_name":"upsert","last_dispatch_duration_us":1530535,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2840055,"total_server_duration_us":154922,"encode_duration_us":20974,"operation_id":"0x14d3e","timeout_ms":10000,"last_server_duration_us":50535,"total_duration_us":3028038},{"operation_name":"upsert","last_dispatch_duration_us":1645958,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1645958,"total_server_duration_us":64343,"encode_duration_us":17767,"operation_id":"0x14e22","timeout_ms":10000,"last_server_duration_us":64343,"total_duration_us":2213617},{"operation_name":"get","last_dispatch_duration_us":1698686,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":1698686,"total_server_duration_us":69,"operation_id":"0x14e26","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":1875139}],"total_count":73}} INFO [2022-03-13 17:19:58,986] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,987] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,989] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,990] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,993] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,994] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,995] jepsen worker 13 - jepsen.util 733 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,996] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:58,997] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,998] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:58,999] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:58,999] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] WARN [2022-03-13 17:19:59,047] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9559603,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17765982,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9092696,"encode_duration_us":23466,"operation_id":"0x141d5","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10002899}],"total_count":1}} INFO [2022-03-13 17:19:59,049] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,050] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:19:59,052] jepsen worker 14 - jepsen.util 374 :fail :write [4 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,052] jepsen worker 72 - jepsen.util 4932 :ok :read [24 24] INFO [2022-03-13 17:19:59,052] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,053] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:19:59,055] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:59,056] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,056] jepsen worker 62 - jepsen.util 512 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,056] jepsen worker 72 - jepsen.util 4932 :invoke :read [24 nil] INFO [2022-03-13 17:19:59,057] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,058] jepsen worker 62 - jepsen.util 512 :invoke :write [20 19] INFO [2022-03-13 17:19:59,063] jepsen worker 83 - jepsen.util 4493 :info :write [27 8] :RequestTimeoutException INFO [2022-03-13 17:19:59,065] jepsen worker 83 - jepsen.util 4583 :invoke :read [27 nil] INFO [2022-03-13 17:19:59,072] jepsen worker 11 - jepsen.util 4691 :ok :read [3 25] INFO [2022-03-13 17:19:59,073] jepsen worker 11 - jepsen.util 4691 :invoke :read [3 nil] INFO [2022-03-13 17:19:59,079] jepsen worker 45 - jepsen.util 5085 :ok :read [15 48] INFO [2022-03-13 17:19:59,080] jepsen worker 45 - jepsen.util 5085 :invoke :write [15 35] INFO [2022-03-13 17:19:59,082] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,083] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 48] INFO [2022-03-13 17:19:59,085] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,085] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:19:59,100] jepsen worker 26 - jepsen.util 566 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,101] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:59,108] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,109] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:59,113] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,114] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:19:59,133] jepsen worker 25 - jepsen.util 565 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,134] jepsen worker 25 - jepsen.util 565 :invoke :write [8 9] INFO [2022-03-13 17:19:59,160] jepsen worker 47 - jepsen.util 4637 :info :write [15 17] :SyncWriteAmbiguous INFO [2022-03-13 17:19:59,162] jepsen worker 47 - jepsen.util 4727 :invoke :write [15 37] INFO [2022-03-13 17:19:59,163] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,163] jepsen worker 34 - jepsen.util 934 :fail :write [11 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,168] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:19:59,168] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:19:59,176] jepsen worker 23 - jepsen.util 563 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,178] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:59,180] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,181] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:59,192] jepsen worker 64 - jepsen.util 4654 :info :write [21 2] :SyncWriteAmbiguous INFO [2022-03-13 17:19:59,193] jepsen worker 64 - jepsen.util 4744 :invoke :read [21 nil] INFO [2022-03-13 17:19:59,212] jepsen worker 75 - jepsen.util 525 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,213] jepsen worker 33 - jepsen.util 753 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,213] jepsen worker 75 - jepsen.util 525 :invoke :write [25 27] INFO [2022-03-13 17:19:59,214] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:19:59,232] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,233] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:59,238] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,239] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:19:59,261] jepsen worker 64 - jepsen.util 4744 :ok :read [21 :nil] INFO [2022-03-13 17:19:59,262] jepsen worker 64 - jepsen.util 4744 :invoke :write [21 17] INFO [2022-03-13 17:19:59,262] jepsen worker 4 - jepsen.util 2614 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,263] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,290] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,291] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,293] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,293] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,296] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,297] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,299] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,299] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,300] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,301] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,302] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,302] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,303] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,304] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,305] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,306] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:19:59,311] jepsen worker 80 - jepsen.util 800 :fail :write [26 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,312] jepsen worker 80 - jepsen.util 800 :invoke :write [26 38] INFO [2022-03-13 17:19:59,319] jepsen worker 7 - jepsen.util 637 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,319] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:19:59,333] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:19:59,333] jepsen worker nemesis - couchbase.util Rebalance check count: 705 INFO [2022-03-13 17:19:59,333] jepsen worker nemesis - couchbase.util Rebalance stuck count: 577 INFO [2022-03-13 17:19:59,338] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,338] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,338] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,339] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:19:59,340] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,340] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:19:59,343] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,344] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 41] INFO [2022-03-13 17:19:59,350] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,351] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:59,353] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,353] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:59,365] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,366] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:19:59,391] jepsen worker 53 - jepsen.util 593 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,391] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,391] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:19:59,391] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,392] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,392] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,393] jepsen worker 52 - jepsen.util 682 :invoke :write [17 38] INFO [2022-03-13 17:19:59,394] jepsen worker 60 - jepsen.util 690 :invoke :write [20 5] INFO [2022-03-13 17:19:59,396] jepsen worker 43 - jepsen.util 673 :invoke :write [14 39] INFO [2022-03-13 17:19:59,397] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:19:59,398] jepsen worker 53 - jepsen.util 593 :invoke :write [17 38] INFO [2022-03-13 17:19:59,398] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:19:59,406] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,407] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:19:59,411] jepsen worker 55 - jepsen.util 4375 :info :write [18 43] :SyncWriteAmbiguous INFO [2022-03-13 17:19:59,412] jepsen worker 76 - jepsen.util 796 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,413] jepsen worker 55 - jepsen.util 4465 :invoke :read [18 nil] INFO [2022-03-13 17:19:59,412] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,414] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:19:59,413] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:19:59,420] jepsen worker 71 - jepsen.util 611 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,421] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,425] jepsen worker 61 - jepsen.util 781 :fail :write [20 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,426] jepsen worker 84 - jepsen.util 5214 :info :write [28 18] :RequestTimeoutException INFO [2022-03-13 17:19:59,426] jepsen worker 61 - jepsen.util 781 :invoke :write [20 16] INFO [2022-03-13 17:19:59,427] jepsen worker 84 - jepsen.util 5304 :invoke :write [28 22] INFO [2022-03-13 17:19:59,427] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,428] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,428] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,428] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,430] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,431] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:19:59,432] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:59,433] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:19:59,433] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:19:59,434] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,434] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,434] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,434] jepsen worker 78 - jepsen.util 708 :invoke :write [26 23] INFO [2022-03-13 17:19:59,435] jepsen worker 51 - jepsen.util 771 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,435] jepsen worker 26 - jepsen.util 566 :invoke :write [8 46] INFO [2022-03-13 17:19:59,436] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:19:59,437] jepsen worker 77 - jepsen.util 707 :invoke :write [25 3] INFO [2022-03-13 17:19:59,439] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:59,444] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,444] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,444] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 41] INFO [2022-03-13 17:19:59,445] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:59,446] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,455] jepsen worker 23 - jepsen.util 563 :invoke :write [7 17] INFO [2022-03-13 17:19:59,464] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,465] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 19] INFO [2022-03-13 17:19:59,482] jepsen worker 4 - jepsen.util 2614 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,483] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,483] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,484] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:59,515] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,516] jepsen worker 69 - jepsen.util 789 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,516] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:19:59,516] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,522] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,523] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,524] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,525] jepsen worker 69 - jepsen.util 789 :invoke :write [23 16] INFO [2022-03-13 17:19:59,534] jepsen worker 6 - jepsen.util 636 :fail :write [2 29] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,535] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:19:59,535] jepsen worker 6 - jepsen.util 636 :invoke :write [2 14] INFO [2022-03-13 17:19:59,536] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,537] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:19:59,538] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:19:59,541] jepsen worker 21 - jepsen.util 831 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,543] jepsen worker 21 - jepsen.util 831 :invoke :write [7 34] INFO [2022-03-13 17:19:59,549] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,550] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:59,553] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,553] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,553] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,553] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,555] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 23] INFO [2022-03-13 17:19:59,556] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:19:59,557] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:19:59,557] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 0] INFO [2022-03-13 17:19:59,582] jepsen worker 5 - jepsen.util 2705 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,583] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:19:59,589] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,589] jepsen worker 75 - jepsen.util 525 :fail :write [25 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,589] jepsen worker 13 - jepsen.util 733 :invoke :write [4 24] INFO [2022-03-13 17:19:59,590] jepsen worker 75 - jepsen.util 525 :invoke :write [25 37] INFO [2022-03-13 17:19:59,622] jepsen worker 36 - jepsen.util 2466 :fail :write [12 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,623] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:19:59,640] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,640] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:59,653] jepsen worker 3 - jepsen.util 2253 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,654] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,671] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,671] jepsen worker 25 - jepsen.util 565 :fail :write [8 9] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,671] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,671] jepsen worker 34 - jepsen.util 934 :invoke :write [11 24] INFO [2022-03-13 17:19:59,672] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:59,673] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:59,679] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,680] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,682] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,683] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,703] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,704] jepsen worker 33 - jepsen.util 753 :invoke :write [11 46] INFO [2022-03-13 17:19:59,711] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,711] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,711] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,712] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:59,712] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:19:59,713] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:19:59,714] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,715] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:59,716] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,717] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:19:59,718] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,719] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:59,720] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,720] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:19:59,721] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,722] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:19:59,751] jepsen worker 80 - jepsen.util 800 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,752] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,752] jepsen worker 80 - jepsen.util 800 :invoke :write [26 35] INFO [2022-03-13 17:19:59,752] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:19:59,762] jepsen worker 69 - jepsen.util 789 :fail :write [23 16] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,762] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,763] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,763] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,764] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,765] jepsen worker 71 - jepsen.util 611 :invoke :write [23 10] INFO [2022-03-13 17:19:59,767] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,768] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,768] jepsen worker 69 - jepsen.util 789 :invoke :write [23 27] INFO [2022-03-13 17:19:59,768] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,769] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,770] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,772] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,772] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,773] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,774] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:19:59,775] jepsen worker 60 - jepsen.util 690 :fail :write [20 5] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,776] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:19:59,777] jepsen worker 67 - jepsen.util 4207 :info :write [22 28] :RequestTimeoutException INFO [2022-03-13 17:19:59,778] jepsen worker 67 - jepsen.util 4297 :invoke :write [22 8] INFO [2022-03-13 17:19:59,785] jepsen worker 35 - jepsen.util 935 :fail :write [11 40] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,786] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:19:59,809] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,810] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:59,843] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,843] jepsen worker 20 - jepsen.util 5150 :ok :read [6 16] INFO [2022-03-13 17:19:59,844] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:19:59,844] jepsen worker 20 - jepsen.util 5150 :invoke :read [6 nil] INFO [2022-03-13 17:19:59,852] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,853] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:19:59,869] jepsen worker 78 - jepsen.util 708 :fail :write [26 23] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,870] jepsen worker 12 - jepsen.util 912 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,871] jepsen worker 78 - jepsen.util 708 :invoke :write [26 48] INFO [2022-03-13 17:19:59,871] jepsen worker 12 - jepsen.util 912 :invoke :write [4 27] INFO [2022-03-13 17:19:59,878] jepsen worker 54 - jepsen.util 4554 :ok :read [18 21] INFO [2022-03-13 17:19:59,879] jepsen worker 54 - jepsen.util 4554 :invoke :write [18 37] INFO [2022-03-13 17:19:59,888] jepsen worker 69 - jepsen.util 789 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,889] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,889] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,891] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:19:59,892] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:19:59,893] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:19:59,933] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,934] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:19:59,944] jepsen worker 9 - jepsen.util 4329 :info :write [3 3] :RequestTimeoutException INFO [2022-03-13 17:19:59,945] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:19:59,958] jepsen worker 21 - jepsen.util 831 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,958] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,958] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,958] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,959] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:19:59,960] jepsen worker 21 - jepsen.util 831 :invoke :write [7 12] INFO [2022-03-13 17:19:59,960] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:19:59,960] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:19:59,963] jepsen worker 28 - jepsen.util 4888 :info :write [9 26] :RequestTimeoutException INFO [2022-03-13 17:19:59,964] jepsen worker 28 - jepsen.util 4978 :invoke :write [9 9] INFO [2022-03-13 17:19:59,974] jepsen worker 18 - jepsen.util 4788 :info :write [6 10] :RequestTimeoutException INFO [2022-03-13 17:19:59,974] jepsen worker 29 - jepsen.util 5069 :info :write [9 23] :RequestTimeoutException INFO [2022-03-13 17:19:59,975] jepsen worker 18 - jepsen.util 4878 :invoke :read [6 nil] INFO [2022-03-13 17:19:59,975] jepsen worker 29 - jepsen.util 5159 :invoke :read [9 nil] INFO [2022-03-13 17:19:59,983] jepsen worker 71 - jepsen.util 611 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,984] jepsen worker 19 - jepsen.util 4609 :info :write [6 28] :RequestTimeoutException INFO [2022-03-13 17:19:59,984] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:19:59,985] jepsen worker 19 - jepsen.util 4699 :invoke :read [6 nil] INFO [2022-03-13 17:19:59,995] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:19:59,996] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:00,000] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,000] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,000] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,000] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,000] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,001] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,001] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:20:00,002] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 12] INFO [2022-03-13 17:20:00,002] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,003] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,003] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:20:00,004] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 20] INFO [2022-03-13 17:20:00,035] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,036] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,038] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,038] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:20:00,066] jepsen worker 9 - jepsen.util 4419 :ok :read [3 25] INFO [2022-03-13 17:20:00,067] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:20:00,104] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,105] jepsen worker 4 - jepsen.util 2614 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,105] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,106] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 47] INFO [2022-03-13 17:20:00,106] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,107] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,108] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,109] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:20:00,114] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,114] jepsen worker 52 - jepsen.util 682 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,114] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,114] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:20:00,115] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:00,120] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,122] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,123] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:20:00,157] jepsen worker 3 - jepsen.util 2253 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,158] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:20:00,163] jepsen worker 42 - jepsen.util 402 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,164] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,163] jepsen worker 77 - jepsen.util 707 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,164] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:00,165] jepsen worker 70 - jepsen.util 880 :invoke :write [23 9] INFO [2022-03-13 17:20:00,167] jepsen worker 34 - jepsen.util 934 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,167] jepsen worker 34 - jepsen.util 934 :invoke :write [11 10] INFO [2022-03-13 17:20:00,169] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:00,179] jepsen worker 9 - jepsen.util 4419 :ok :read [3 25] INFO [2022-03-13 17:20:00,180] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:20:00,203] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,203] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,204] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,204] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 13] INFO [2022-03-13 17:20:00,206] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,207] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,208] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,209] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,210] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,211] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,214] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,215] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,220] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,221] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 18] INFO [2022-03-13 17:20:00,263] jepsen worker 13 - jepsen.util 733 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,264] jepsen worker 13 - jepsen.util 733 :invoke :write [4 40] INFO [2022-03-13 17:20:00,267] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,268] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,285] jepsen worker 4 - jepsen.util 2614 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,286] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,290] jepsen worker 9 - jepsen.util 4419 :ok :read [3 25] INFO [2022-03-13 17:20:00,291] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:20:00,297] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,298] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,302] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,302] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,307] jepsen worker 33 - jepsen.util 753 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,307] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,308] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:00,308] jepsen worker 51 - jepsen.util 771 :invoke :write [17 9] INFO [2022-03-13 17:20:00,325] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,326] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 28] INFO [2022-03-13 17:20:00,340] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:00,340] jepsen worker nemesis - couchbase.util Rebalance check count: 706 INFO [2022-03-13 17:20:00,340] jepsen worker nemesis - couchbase.util Rebalance stuck count: 578 INFO [2022-03-13 17:20:00,359] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,359] jepsen worker 36 - jepsen.util 2466 :fail :write [12 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,360] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 4] INFO [2022-03-13 17:20:00,360] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:20:00,362] jepsen worker 53 - jepsen.util 593 :fail :write [17 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,363] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:00,371] jepsen worker 9 - jepsen.util 4419 :ok :read [3 25] INFO [2022-03-13 17:20:00,372] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:20:00,376] jepsen worker 30 - jepsen.util 4800 :ok :read [10 :nil] INFO [2022-03-13 17:20:00,377] jepsen worker 30 - jepsen.util 4800 :invoke :write [10 8] INFO [2022-03-13 17:20:00,394] jepsen worker 27 - jepsen.util 4977 :info :write [9 38] :RequestTimeoutException INFO [2022-03-13 17:20:00,394] jepsen worker 3 - jepsen.util 2253 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,395] jepsen worker 27 - jepsen.util 5067 :invoke :write [9 29] INFO [2022-03-13 17:20:00,396] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 7] INFO [2022-03-13 17:20:00,407] jepsen worker 56 - jepsen.util 4646 :ok :read [18 21] INFO [2022-03-13 17:20:00,408] jepsen worker 56 - jepsen.util 4646 :invoke :write [18 29] INFO [2022-03-13 17:20:00,425] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,426] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 2] INFO [2022-03-13 17:20:00,498] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,498] jepsen worker 80 - jepsen.util 800 :fail :write [26 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,500] jepsen worker 1 - jepsen.util 4591 :info :write [0 46] :SyncWriteAmbiguous INFO [2022-03-13 17:20:00,500] jepsen worker 79 - jepsen.util 1159 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,500] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:00,500] jepsen worker 62 - jepsen.util 512 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,500] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,501] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,503] jepsen worker 9 - jepsen.util 4419 :ok :read [3 25] INFO [2022-03-13 17:20:00,503] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,503] jepsen worker 44 - jepsen.util 764 :invoke :write [14 1] INFO [2022-03-13 17:20:00,504] jepsen worker 4 - jepsen.util 2614 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,504] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:20:00,505] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,505] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 21] INFO [2022-03-13 17:20:00,506] jepsen worker 80 - jepsen.util 800 :invoke :write [26 12] INFO [2022-03-13 17:20:00,507] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:00,507] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:00,508] jepsen worker 1 - jepsen.util 4681 :invoke :read [0 nil] INFO [2022-03-13 17:20:00,510] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:00,510] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,510] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,513] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,513] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,514] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,515] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,516] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,516] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,517] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,518] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 12] INFO [2022-03-13 17:20:00,521] jepsen worker 71 - jepsen.util 611 :invoke :write [23 44] INFO [2022-03-13 17:20:00,523] jepsen worker 69 - jepsen.util 789 :invoke :write [23 15] INFO [2022-03-13 17:20:00,527] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,527] jepsen worker 75 - jepsen.util 525 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,527] jepsen worker 55 - jepsen.util 4465 :ok :read [18 21] INFO [2022-03-13 17:20:00,528] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:00,529] jepsen worker 76 - jepsen.util 796 :invoke :write [25 7] INFO [2022-03-13 17:20:00,529] jepsen worker 55 - jepsen.util 4465 :invoke :read [18 nil] INFO [2022-03-13 17:20:00,534] jepsen worker 29 - jepsen.util 5159 :ok :read [9 26] INFO [2022-03-13 17:20:00,535] jepsen worker 29 - jepsen.util 5159 :invoke :read [9 nil] INFO [2022-03-13 17:20:00,540] jepsen worker 3 - jepsen.util 2253 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,541] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,554] jepsen worker 5 - jepsen.util 2705 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,554] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,555] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,556] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,558] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,558] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,560] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:20:00,560] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 36] INFO [2022-03-13 17:20:00,577] jepsen worker 36 - jepsen.util 2466 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,577] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,579] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 3] INFO [2022-03-13 17:20:00,579] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:20:00,581] jepsen worker 9 - jepsen.util 4419 :ok :read [3 25] INFO [2022-03-13 17:20:00,582] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:20:00,596] jepsen worker 32 - jepsen.util 5072 :info :write [10 2] :RequestTimeoutException INFO [2022-03-13 17:20:00,597] jepsen worker 32 - jepsen.util 5162 :invoke :read [10 nil] INFO [2022-03-13 17:20:00,602] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,602] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,602] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,603] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,604] jepsen worker 14 - jepsen.util 374 :invoke :write [4 4] INFO [2022-03-13 17:20:00,605] jepsen worker 25 - jepsen.util 565 :invoke :write [8 28] INFO [2022-03-13 17:20:00,605] jepsen worker 24 - jepsen.util 654 :invoke :write [8 21] INFO [2022-03-13 17:20:00,607] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:20:00,625] jepsen worker 4 - jepsen.util 2614 :fail :write [1 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,626] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,641] jepsen worker 69 - jepsen.util 789 :fail :write [23 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,642] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,645] jepsen worker 55 - jepsen.util 4465 :ok :read [18 21] INFO [2022-03-13 17:20:00,645] jepsen worker 55 - jepsen.util 4465 :invoke :read [18 nil] INFO [2022-03-13 17:20:00,660] jepsen worker 36 - jepsen.util 2466 :fail :write [12 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,661] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:20:00,673] jepsen worker 6 - jepsen.util 636 :fail :write [2 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,673] jepsen worker 23 - jepsen.util 563 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,674] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:00,675] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:00,691] jepsen worker 29 - jepsen.util 5159 :ok :read [9 26] INFO [2022-03-13 17:20:00,692] jepsen worker 29 - jepsen.util 5159 :invoke :read [9 nil] INFO [2022-03-13 17:20:00,702] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,703] jepsen worker 43 - jepsen.util 673 :fail :write [14 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,703] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 13] INFO [2022-03-13 17:20:00,704] jepsen worker 43 - jepsen.util 673 :invoke :write [14 31] INFO [2022-03-13 17:20:00,706] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,706] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,706] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,707] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:20:00,736] jepsen worker 71 - jepsen.util 611 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,736] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,737] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,737] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,760] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,760] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,761] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:20:00,761] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:00,774] jepsen worker 87 - jepsen.util 2877 :fail :write [29 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,775] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:20:00,790] jepsen worker 55 - jepsen.util 4465 :ok :read [18 21] INFO [2022-03-13 17:20:00,790] jepsen worker 55 - jepsen.util 4465 :invoke :read [18 nil] INFO [2022-03-13 17:20:00,819] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,819] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,820] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,820] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 7] INFO [2022-03-13 17:20:00,821] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,822] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,824] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,824] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,825] jepsen worker 29 - jepsen.util 5159 :ok :read [9 26] INFO [2022-03-13 17:20:00,825] jepsen worker 29 - jepsen.util 5159 :invoke :read [9 nil] INFO [2022-03-13 17:20:00,826] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,826] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,826] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,827] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:00,827] jepsen worker 21 - jepsen.util 831 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,827] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,828] jepsen worker 21 - jepsen.util 831 :invoke :write [7 20] INFO [2022-03-13 17:20:00,828] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,828] jepsen worker 71 - jepsen.util 611 :invoke :write [23 30] INFO [2022-03-13 17:20:00,829] jepsen worker 69 - jepsen.util 789 :invoke :write [23 47] INFO [2022-03-13 17:20:00,872] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,872] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,873] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 3] INFO [2022-03-13 17:20:00,873] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:00,874] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,875] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:00,879] jepsen worker 13 - jepsen.util 733 :fail :write [4 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,879] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:00,884] jepsen worker 61 - jepsen.util 781 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,884] jepsen worker 12 - jepsen.util 912 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,884] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:00,884] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:00,885] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:00,886] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:00,888] jepsen worker 5 - jepsen.util 2705 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,888] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:20:00,923] jepsen worker 55 - jepsen.util 4465 :ok :read [18 21] INFO [2022-03-13 17:20:00,924] jepsen worker 55 - jepsen.util 4465 :invoke :read [18 nil] INFO [2022-03-13 17:20:00,928] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,928] jepsen worker 36 - jepsen.util 2466 :fail :write [12 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,928] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 46] INFO [2022-03-13 17:20:00,930] jepsen worker 29 - jepsen.util 5159 :ok :read [9 26] INFO [2022-03-13 17:20:00,930] jepsen worker 29 - jepsen.util 5159 :invoke :write [9 17] INFO [2022-03-13 17:20:00,942] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:00,944] jepsen worker 34 - jepsen.util 934 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,945] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,945] jepsen worker 34 - jepsen.util 934 :invoke :write [11 37] INFO [2022-03-13 17:20:00,947] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:00,968] jepsen worker 26 - jepsen.util 566 :fail :write [8 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,969] jepsen worker 26 - jepsen.util 566 :invoke :write [8 48] INFO [2022-03-13 17:20:00,971] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,972] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:00,979] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,980] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:20:00,992] jepsen worker 3 - jepsen.util 2253 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,992] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:00,993] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:20:00,994] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,006] jepsen worker 51 - jepsen.util 771 :fail :write [17 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,007] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:01,025] jepsen worker 71 - jepsen.util 611 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,026] jepsen worker 71 - jepsen.util 611 :invoke :write [23 18] INFO [2022-03-13 17:20:01,039] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,039] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,039] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,039] jepsen worker 70 - jepsen.util 880 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,039] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,039] jepsen worker 78 - jepsen.util 708 :fail :write [26 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,040] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:01,040] jepsen worker 70 - jepsen.util 880 :invoke :write [23 40] INFO [2022-03-13 17:20:01,041] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:01,041] jepsen worker 78 - jepsen.util 708 :invoke :write [26 41] INFO [2022-03-13 17:20:01,045] jepsen worker 55 - jepsen.util 4465 :ok :read [18 21] INFO [2022-03-13 17:20:01,046] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:01,047] jepsen worker 55 - jepsen.util 4465 :invoke :write [18 26] INFO [2022-03-13 17:20:01,047] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:01,053] jepsen worker 4 - jepsen.util 2614 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,053] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,054] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:20:01,055] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:20:01,063] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,063] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:20:01,070] jepsen worker 57 - jepsen.util 4827 :info :write [19 28] :RequestTimeoutException INFO [2022-03-13 17:20:01,071] jepsen worker 57 - jepsen.util 4917 :invoke :read [19 nil] INFO [2022-03-13 17:20:01,087] jepsen worker 80 - jepsen.util 800 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,088] jepsen worker 80 - jepsen.util 800 :invoke :write [26 37] INFO [2022-03-13 17:20:01,101] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,102] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 17] INFO [2022-03-13 17:20:01,130] jepsen worker 88 - jepsen.util 2518 :info :write [29 38] :RequestTimeoutException INFO [2022-03-13 17:20:01,131] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,153] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,154] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,155] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,156] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:20:01,161] jepsen worker 76 - jepsen.util 796 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,161] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,161] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,162] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:01,162] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:01,163] jepsen worker 76 - jepsen.util 796 :invoke :write [25 24] INFO [2022-03-13 17:20:01,169] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,169] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,174] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,175] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 38] INFO [2022-03-13 17:20:01,175] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,176] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,193] jepsen worker 69 - jepsen.util 789 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,194] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:01,195] jepsen worker 14 - jepsen.util 374 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,196] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:01,225] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,225] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:20:01,243] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,243] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,309] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,309] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,309] jepsen worker 36 - jepsen.util 2466 :fail :write [12 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,309] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,310] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:20:01,311] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 31] INFO [2022-03-13 17:20:01,312] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,312] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,315] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,316] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:20:01,338] jepsen worker 39 - jepsen.util 5259 :info :write [13 41] :RequestTimeoutException INFO [2022-03-13 17:20:01,339] jepsen worker 39 - jepsen.util 5349 :invoke :write [13 33] INFO [2022-03-13 17:20:01,349] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:01,349] jepsen worker nemesis - couchbase.util Rebalance check count: 707 INFO [2022-03-13 17:20:01,349] jepsen worker nemesis - couchbase.util Rebalance stuck count: 579 INFO [2022-03-13 17:20:01,357] jepsen worker 88 - jepsen.util 2608 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,357] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,357] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 3] INFO [2022-03-13 17:20:01,358] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 17] INFO [2022-03-13 17:20:01,374] jepsen worker 66 - jepsen.util 4656 :info :write [22 20] :RequestTimeoutException INFO [2022-03-13 17:20:01,375] jepsen worker 66 - jepsen.util 4746 :invoke :read [22 nil] INFO [2022-03-13 17:20:01,405] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,406] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,407] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,408] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,409] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,410] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,411] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,412] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,414] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,414] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:20:01,415] jepsen worker 36 - jepsen.util 2466 :fail :write [12 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,415] jepsen worker 65 - jepsen.util 5105 :info :write [21 37] :RequestTimeoutException INFO [2022-03-13 17:20:01,416] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:01,416] jepsen worker 65 - jepsen.util 5195 :invoke :read [21 nil] INFO [2022-03-13 17:20:01,495] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,496] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:20:01,496] jepsen worker 63 - jepsen.util 5013 :info :write [21 30] :RequestTimeoutException INFO [2022-03-13 17:20:01,497] jepsen worker 63 - jepsen.util 5103 :invoke :write [21 38] INFO [2022-03-13 17:20:01,505] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,506] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,542] jepsen worker 88 - jepsen.util 2608 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,543] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 12] INFO [2022-03-13 17:20:01,574] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,575] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,576] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,576] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,578] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,578] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,579] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,580] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,581] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,582] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:20:01,610] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,611] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,613] jepsen worker 4 - jepsen.util 2614 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,614] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,614] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,614] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,614] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,614] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:01,615] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 6] INFO [2022-03-13 17:20:01,615] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,616] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,617] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,617] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:20:01,618] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,619] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:20:01,626] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:01,635] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,637] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:20:01,653] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,654] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:20:01,685] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,686] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,687] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,687] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,689] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,689] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:20:01,717] jepsen worker 88 - jepsen.util 2608 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,718] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,723] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,724] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 10] INFO [2022-03-13 17:20:01,745] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,746] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:20:01,791] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,794] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 9] INFO [2022-03-13 17:20:01,815] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,816] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,819] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,820] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:20:01,867] jepsen worker 3 - jepsen.util 2253 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,868] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:20:01,895] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,896] jepsen worker 88 - jepsen.util 2608 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,896] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 4] INFO [2022-03-13 17:20:01,897] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:01,898] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,899] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:20:01,929] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,929] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,931] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,931] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:20:01,959] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,959] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 11] INFO [2022-03-13 17:20:01,989] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,990] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:01,991] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:01,992] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:20:02,018] jepsen worker 5 - jepsen.util 2705 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,018] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:20:02,052] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,053] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:20:02,084] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,084] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:20:02,100] jepsen worker 73 - jepsen.util 5203 :info :write [24 16] :RequestTimeoutException INFO [2022-03-13 17:20:02,101] jepsen worker 73 - jepsen.util 5293 :invoke :read [24 nil] INFO [2022-03-13 17:20:02,113] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,114] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,115] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,116] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,117] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,117] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,118] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,119] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:20:02,123] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,123] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,124] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,124] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,124] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,124] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,124] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:02,125] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:20:02,126] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,126] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:02,126] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:02,127] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:20:02,151] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,152] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:20:02,165] jepsen worker 43 - jepsen.util 673 :fail :write [14 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,166] jepsen worker 43 - jepsen.util 673 :invoke :write [14 28] INFO [2022-03-13 17:20:02,182] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,182] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:20:02,188] jepsen worker 46 - jepsen.util 5086 :info :write [15 45] :RequestTimeoutException INFO [2022-03-13 17:20:02,189] jepsen worker 46 - jepsen.util 5176 :invoke :read [15 nil] INFO [2022-03-13 17:20:02,209] jepsen worker 76 - jepsen.util 796 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,210] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,210] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,211] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,222] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,223] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:02,224] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,225] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:20:02,229] jepsen worker 10 - jepsen.util 5140 :info :write [3 14] :RequestTimeoutException INFO [2022-03-13 17:20:02,230] jepsen worker 10 - jepsen.util 5230 :invoke :read [3 nil] INFO [2022-03-13 17:20:02,240] jepsen worker 25 - jepsen.util 565 :fail :write [8 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,241] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:02,245] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,245] jepsen worker 71 - jepsen.util 611 :fail :write [23 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,247] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,247] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:20:02,252] jepsen worker 79 - jepsen.util 1159 :fail :write [26 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,252] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,252] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,252] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,253] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:02,254] jepsen worker 77 - jepsen.util 707 :invoke :write [25 7] INFO [2022-03-13 17:20:02,256] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:02,257] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:02,259] jepsen worker 44 - jepsen.util 764 :fail :write [14 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,259] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,259] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,259] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:02,259] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,259] jepsen worker 44 - jepsen.util 764 :invoke :write [14 47] INFO [2022-03-13 17:20:02,259] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,259] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,260] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,260] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,260] jepsen worker 12 - jepsen.util 912 :invoke :write [4 19] INFO [2022-03-13 17:20:02,261] jepsen worker 42 - jepsen.util 402 :invoke :write [14 19] INFO [2022-03-13 17:20:02,262] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:02,263] jepsen worker 71 - jepsen.util 611 :invoke :write [23 31] INFO [2022-03-13 17:20:02,264] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,264] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:02,265] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:02,265] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,266] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:20:02,274] jepsen worker 62 - jepsen.util 512 :invoke :write [20 32] INFO [2022-03-13 17:20:02,276] jepsen worker 46 - jepsen.util 5176 :ok :read [15 48] INFO [2022-03-13 17:20:02,277] jepsen worker 46 - jepsen.util 5176 :invoke :read [15 nil] INFO [2022-03-13 17:20:02,318] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,319] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:02,320] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,321] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:02,322] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,323] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:20:02,327] jepsen worker 46 - jepsen.util 5176 :ok :read [15 48] INFO [2022-03-13 17:20:02,328] jepsen worker 46 - jepsen.util 5176 :invoke :write [15 3] INFO [2022-03-13 17:20:02,354] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,355] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 28] INFO [2022-03-13 17:20:02,355] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:02,355] jepsen worker nemesis - couchbase.util Rebalance check count: 708 INFO [2022-03-13 17:20:02,356] jepsen worker nemesis - couchbase.util Rebalance stuck count: 580 INFO [2022-03-13 17:20:02,359] jepsen worker 26 - jepsen.util 566 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,360] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,360] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,360] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,360] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:02,361] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:02,361] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,361] jepsen worker 53 - jepsen.util 593 :invoke :write [17 25] INFO [2022-03-13 17:20:02,362] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 24] INFO [2022-03-13 17:20:02,362] jepsen worker 22 - jepsen.util 652 :invoke :write [7 46] INFO [2022-03-13 17:20:02,363] jepsen worker 21 - jepsen.util 831 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,364] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,364] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:02,365] jepsen worker 33 - jepsen.util 753 :invoke :write [11 16] INFO [2022-03-13 17:20:02,386] jepsen worker 74 - jepsen.util 4754 :info :write [24 33] :RequestTimeoutException INFO [2022-03-13 17:20:02,386] jepsen worker 74 - jepsen.util 4844 :invoke :read [24 nil] INFO [2022-03-13 17:20:02,414] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,414] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,415] jepsen worker 88 - jepsen.util 2608 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,416] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 44] INFO [2022-03-13 17:20:02,435] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,436] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:20:02,464] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,464] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,464] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,465] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,465] jepsen worker 75 - jepsen.util 525 :invoke :write [25 23] INFO [2022-03-13 17:20:02,466] jepsen worker 13 - jepsen.util 733 :invoke :write [4 31] INFO [2022-03-13 17:20:02,466] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,467] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,476] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,477] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,478] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,479] jepsen worker 76 - jepsen.util 796 :invoke :write [25 34] INFO [2022-03-13 17:20:02,504] jepsen worker 34 - jepsen.util 934 :fail :write [11 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,504] jepsen worker 4 - jepsen.util 2614 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,505] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,505] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:02,505] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 31] INFO [2022-03-13 17:20:02,506] jepsen worker 51 - jepsen.util 771 :invoke :write [17 26] INFO [2022-03-13 17:20:02,508] jepsen worker 87 - jepsen.util 2877 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,509] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:20:02,514] jepsen worker 70 - jepsen.util 880 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,514] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,514] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,514] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,514] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,514] jepsen worker 78 - jepsen.util 708 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,515] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,515] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,515] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,515] jepsen worker 7 - jepsen.util 637 :invoke :write [2 48] INFO [2022-03-13 17:20:02,516] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 3] INFO [2022-03-13 17:20:02,518] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:02,519] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:02,520] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:20:02,521] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:02,521] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,521] jepsen worker 71 - jepsen.util 611 :fail :write [23 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,523] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,523] jepsen worker 71 - jepsen.util 611 :invoke :write [23 24] INFO [2022-03-13 17:20:02,524] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,525] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:20:02,530] jepsen worker 78 - jepsen.util 708 :invoke :write [26 47] INFO [2022-03-13 17:20:02,582] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,583] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:20:02,591] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,591] jepsen worker 52 - jepsen.util 682 :invoke :write [17 4] INFO [2022-03-13 17:20:02,613] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,614] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:02,614] jepsen worker 80 - jepsen.util 800 :fail :write [26 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,615] jepsen worker 80 - jepsen.util 800 :invoke :write [26 27] INFO [2022-03-13 17:20:02,616] jepsen worker 85 - jepsen.util 4585 :info :write [28 2] :RequestTimeoutException INFO [2022-03-13 17:20:02,618] jepsen worker 88 - jepsen.util 2608 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,619] jepsen worker 85 - jepsen.util 4675 :invoke :read [28 nil] INFO [2022-03-13 17:20:02,619] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 23] INFO [2022-03-13 17:20:02,627] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,628] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,629] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,630] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:20:02,658] jepsen worker 71 - jepsen.util 611 :fail :write [23 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,659] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,660] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,661] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:20:02,696] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,697] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:02,715] jepsen worker 75 - jepsen.util 525 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,716] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,721] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,721] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:20:02,738] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,739] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,738] jepsen worker 24 - jepsen.util 654 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,739] jepsen worker 21 - jepsen.util 831 :invoke :write [7 10] INFO [2022-03-13 17:20:02,740] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:02,740] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:02,757] jepsen worker 12 - jepsen.util 912 :fail :write [4 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,758] jepsen worker 12 - jepsen.util 912 :invoke :write [4 31] INFO [2022-03-13 17:20:02,758] jepsen worker 77 - jepsen.util 707 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,758] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,759] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:02,759] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:02,770] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,770] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,771] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,771] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,772] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,772] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,773] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,773] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:20:02,774] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,774] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:02,775] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,776] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:20:02,780] jepsen worker 40 - jepsen.util 5080 :info :write [13 14] :RequestTimeoutException INFO [2022-03-13 17:20:02,781] jepsen worker 40 - jepsen.util 5170 :invoke :write [13 11] INFO [2022-03-13 17:20:02,783] jepsen worker 36 - jepsen.util 2466 :fail :write [12 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,783] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:02,797] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,797] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:02,804] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,804] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:20:02,820] jepsen worker 43 - jepsen.util 673 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,820] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,820] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,823] jepsen worker 16 - jepsen.util 4606 :fail :read [5 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:02,824] jepsen worker 60 - jepsen.util 690 :invoke :write [20 13] INFO [2022-03-13 17:20:02,825] jepsen worker 43 - jepsen.util 673 :invoke :write [14 32] INFO [2022-03-13 17:20:02,826] jepsen worker 16 - jepsen.util 4606 :invoke :write [5 5] INFO [2022-03-13 17:20:02,826] jepsen worker 26 - jepsen.util 566 :invoke :write [8 48] INFO [2022-03-13 17:20:02,838] jepsen worker 88 - jepsen.util 2608 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,839] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 6] INFO [2022-03-13 17:20:02,857] jepsen worker 33 - jepsen.util 753 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,857] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,858] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:02,859] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:02,882] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,882] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,890] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,890] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:20:02,901] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,902] jepsen worker 69 - jepsen.util 789 :invoke :write [23 39] INFO [2022-03-13 17:20:02,912] jepsen worker 4 - jepsen.util 2614 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,913] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:02,915] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,915] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,915] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,915] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:02,916] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:02,916] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:20:02,916] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,917] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:20:02,923] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,924] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 7] INFO [2022-03-13 17:20:02,936] jepsen worker 13 - jepsen.util 733 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,936] jepsen worker 13 - jepsen.util 733 :invoke :write [4 24] INFO [2022-03-13 17:20:02,951] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,951] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:02,977] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,977] jepsen worker 51 - jepsen.util 771 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,978] jepsen worker 51 - jepsen.util 771 :invoke :write [17 44] INFO [2022-03-13 17:20:02,979] jepsen worker 14 - jepsen.util 374 :invoke :write [4 47] INFO [2022-03-13 17:20:02,988] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,989] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,991] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,991] jepsen worker 79 - jepsen.util 1159 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,991] jepsen worker 44 - jepsen.util 764 :fail :write [14 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,993] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:02,994] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:02,994] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:02,998] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:02,999] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:20:03,004] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,007] jepsen worker 88 - jepsen.util 2608 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,009] jepsen worker 71 - jepsen.util 611 :invoke :write [23 23] INFO [2022-03-13 17:20:03,011] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,013] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,013] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,016] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,017] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,018] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,019] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 42] INFO [2022-03-13 17:20:03,040] jepsen worker 53 - jepsen.util 593 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,041] jepsen worker 53 - jepsen.util 593 :invoke :write [17 18] INFO [2022-03-13 17:20:03,075] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,075] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,077] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,077] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:20:03,083] jepsen worker 80 - jepsen.util 800 :fail :write [26 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,083] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,083] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,084] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:03,085] jepsen worker 25 - jepsen.util 565 :invoke :write [8 16] INFO [2022-03-13 17:20:03,085] jepsen worker 80 - jepsen.util 800 :invoke :write [26 9] INFO [2022-03-13 17:20:03,087] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,088] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,107] jepsen worker 69 - jepsen.util 789 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,107] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:03,142] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,142] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,144] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,144] jepsen worker 76 - jepsen.util 796 :fail :write [25 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,144] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,144] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,145] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:03,146] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:03,151] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,152] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,154] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,155] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:20:03,177] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,178] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:20:03,180] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,181] jepsen worker 35 - jepsen.util 935 :invoke :write [11 47] INFO [2022-03-13 17:20:03,194] jepsen worker 62 - jepsen.util 512 :fail :write [20 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,194] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,195] jepsen worker 61 - jepsen.util 781 :invoke :write [20 40] INFO [2022-03-13 17:20:03,195] jepsen worker 62 - jepsen.util 512 :invoke :write [20 42] INFO [2022-03-13 17:20:03,202] jepsen worker 71 - jepsen.util 611 :fail :write [23 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,202] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,203] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:20:03,204] jepsen worker 69 - jepsen.util 789 :invoke :write [23 39] INFO [2022-03-13 17:20:03,214] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,214] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,218] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,219] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:20:03,222] jepsen worker 22 - jepsen.util 652 :fail :write [7 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,222] jepsen worker 22 - jepsen.util 652 :invoke :write [7 10] INFO [2022-03-13 17:20:03,231] jepsen worker 88 - jepsen.util 2608 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,232] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,234] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,234] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,238] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,239] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 15] INFO [2022-03-13 17:20:03,258] jepsen worker 21 - jepsen.util 831 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,259] jepsen worker 21 - jepsen.util 831 :invoke :write [7 38] INFO [2022-03-13 17:20:03,260] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,261] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:03,261] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,262] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:03,295] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,296] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:20:03,312] jepsen worker 4 - jepsen.util 2614 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,312] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,313] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,313] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:20:03,338] jepsen worker 69 - jepsen.util 789 :fail :write [23 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,339] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:03,343] jepsen worker 13 - jepsen.util 733 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,343] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,343] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,344] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:03,345] jepsen worker 75 - jepsen.util 525 :invoke :write [25 43] INFO [2022-03-13 17:20:03,345] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,346] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:03,346] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:03,347] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,347] jepsen worker 76 - jepsen.util 796 :invoke :write [25 43] INFO [2022-03-13 17:20:03,361] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:03,361] jepsen worker nemesis - couchbase.util Rebalance check count: 709 INFO [2022-03-13 17:20:03,361] jepsen worker nemesis - couchbase.util Rebalance stuck count: 581 INFO [2022-03-13 17:20:03,361] jepsen worker 78 - jepsen.util 708 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,361] jepsen worker 42 - jepsen.util 402 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,361] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,361] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,361] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,361] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,361] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,362] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,362] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,362] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:03,363] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,364] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 30] INFO [2022-03-13 17:20:03,364] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,364] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:03,365] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:03,366] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:03,367] jepsen worker 51 - jepsen.util 771 :fail :write [17 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,367] jepsen worker 23 - jepsen.util 563 :invoke :write [7 31] INFO [2022-03-13 17:20:03,368] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,368] jepsen worker 8 - jepsen.util 728 :invoke :write [2 47] INFO [2022-03-13 17:20:03,369] jepsen worker 42 - jepsen.util 402 :invoke :write [14 36] INFO [2022-03-13 17:20:03,370] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,372] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:20:03,373] jepsen worker 51 - jepsen.util 771 :invoke :write [17 49] INFO [2022-03-13 17:20:03,404] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,404] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,406] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:03,406] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:20:03,411] jepsen worker 52 - jepsen.util 682 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,411] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,412] jepsen worker 52 - jepsen.util 682 :invoke :write [17 40] INFO [2022-03-13 17:20:03,413] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:03,424] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,424] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,424] jepsen worker 71 - jepsen.util 611 :invoke :write [23 38] INFO [2022-03-13 17:20:03,425] jepsen worker 69 - jepsen.util 789 :invoke :write [23 30] INFO [2022-03-13 17:20:03,434] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,434] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:20:03,438] jepsen worker 14 - jepsen.util 374 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,438] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,439] jepsen worker 14 - jepsen.util 374 :invoke :write [4 25] INFO [2022-03-13 17:20:03,439] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:03,456] jepsen worker 88 - jepsen.util 2608 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,457] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,458] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,459] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,463] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,464] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 47] INFO [2022-03-13 17:20:03,505] jepsen worker 25 - jepsen.util 565 :fail :write [8 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,506] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:20:03,506] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,507] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:03,515] jepsen worker 71 - jepsen.util 611 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,516] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:03,534] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,534] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,535] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,535] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:20:03,578] jepsen worker 80 - jepsen.util 800 :fail :write [26 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,578] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:03,583] jepsen worker 69 - jepsen.util 789 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,583] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,584] jepsen worker 60 - jepsen.util 690 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,584] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:03,584] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:20:03,585] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:03,595] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,596] jepsen worker 69 - jepsen.util 789 :invoke :write [23 0] INFO [2022-03-13 17:20:03,598] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,598] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:20:03,601] jepsen worker 61 - jepsen.util 781 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,602] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,602] jepsen worker 7 - jepsen.util 637 :fail :write [2 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,602] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,603] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:03,603] jepsen worker 7 - jepsen.util 637 :invoke :write [2 38] INFO [2022-03-13 17:20:03,604] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:03,604] jepsen worker 77 - jepsen.util 707 :invoke :write [25 12] INFO [2022-03-13 17:20:03,605] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,606] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,606] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,607] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:20:03,634] jepsen worker 88 - jepsen.util 2608 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,635] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,642] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,642] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 4] INFO [2022-03-13 17:20:03,673] jepsen worker 75 - jepsen.util 525 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,673] jepsen worker 75 - jepsen.util 525 :invoke :write [25 49] INFO [2022-03-13 17:20:03,686] jepsen worker 86 - jepsen.util 4676 :ok :read [28 44] INFO [2022-03-13 17:20:03,686] jepsen worker 86 - jepsen.util 4676 :invoke :read [28 nil] INFO [2022-03-13 17:20:03,691] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,692] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,694] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,695] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:20:03,700] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,700] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,700] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:03,700] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,702] jepsen worker 35 - jepsen.util 935 :fail :write [11 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,702] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,702] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,703] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:03,703] jepsen worker 33 - jepsen.util 753 :invoke :write [11 34] INFO [2022-03-13 17:20:03,704] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:20:03,745] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,745] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:03,762] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,763] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:20:03,788] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,788] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,788] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,789] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:03,789] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,790] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:20:03,819] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:03,819] jepsen worker 12 - jepsen.util 912 :fail :write [4 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,820] jepsen worker 23 - jepsen.util 563 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,820] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:03,821] jepsen worker 44 - jepsen.util 764 :invoke :write [14 26] INFO [2022-03-13 17:20:03,822] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:03,824] jepsen worker 21 - jepsen.util 831 :fail :write [7 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,825] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:20:03,826] jepsen worker 26 - jepsen.util 566 :fail :write [8 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,827] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:03,836] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,836] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:20:03,841] jepsen worker 88 - jepsen.util 2608 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,842] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,848] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,848] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,848] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,849] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,849] jepsen worker 69 - jepsen.util 789 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,850] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:03,850] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,851] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,851] jepsen worker 69 - jepsen.util 789 :invoke :write [23 35] INFO [2022-03-13 17:20:03,855] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,859] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,864] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,864] jepsen worker 71 - jepsen.util 611 :invoke :write [23 6] INFO [2022-03-13 17:20:03,865] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,891] jepsen worker 76 - jepsen.util 796 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,891] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,892] jepsen worker 76 - jepsen.util 796 :invoke :write [25 48] INFO [2022-03-13 17:20:03,892] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:03,904] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,905] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:20:03,922] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,922] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,922] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,922] jepsen worker 51 - jepsen.util 771 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,922] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,924] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:03,925] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:03,925] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:03,926] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,927] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,927] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,927] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,928] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:03,932] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:20:03,937] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,937] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,938] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,938] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 29] INFO [2022-03-13 17:20:03,939] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 19] INFO [2022-03-13 17:20:03,940] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 25] INFO [2022-03-13 17:20:03,959] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,960] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,964] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,964] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,966] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,967] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:03,974] jepsen worker 31 - jepsen.util 5071 :info :write [10 0] :RequestTimeoutException INFO [2022-03-13 17:20:03,975] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,975] jepsen worker 31 - jepsen.util 5161 :invoke :read [10 nil] INFO [2022-03-13 17:20:03,976] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:20:03,983] jepsen worker 69 - jepsen.util 789 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:03,984] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:20:04,013] jepsen worker 62 - jepsen.util 512 :fail :write [20 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,014] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,014] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,014] jepsen worker 79 - jepsen.util 1159 :fail :write [26 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,014] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,014] jepsen worker 62 - jepsen.util 512 :invoke :write [20 6] INFO [2022-03-13 17:20:04,015] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,015] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 32] INFO [2022-03-13 17:20:04,015] jepsen worker 6 - jepsen.util 636 :invoke :write [2 33] INFO [2022-03-13 17:20:04,016] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,017] jepsen worker 78 - jepsen.util 708 :invoke :write [26 42] INFO [2022-03-13 17:20:04,018] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:04,027] jepsen worker 43 - jepsen.util 673 :fail :write [14 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,028] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:04,032] jepsen worker 14 - jepsen.util 374 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,033] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:04,050] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,050] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,051] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:04,051] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:20:04,075] jepsen worker 71 - jepsen.util 611 :fail :write [23 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,076] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:20:04,104] jepsen worker 75 - jepsen.util 525 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,104] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,104] jepsen worker 75 - jepsen.util 525 :invoke :write [25 46] INFO [2022-03-13 17:20:04,105] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:04,126] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,127] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:20:04,127] jepsen worker 88 - jepsen.util 2608 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,128] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 13] INFO [2022-03-13 17:20:04,144] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,144] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,144] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,146] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:04,147] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:04,148] jepsen worker 80 - jepsen.util 800 :invoke :write [26 10] INFO [2022-03-13 17:20:04,174] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,175] jepsen worker 69 - jepsen.util 789 :invoke :write [23 48] INFO [2022-03-13 17:20:04,178] jepsen worker 4 - jepsen.util 2614 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,179] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:20:04,198] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,199] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:20:04,210] jepsen worker 53 - jepsen.util 593 :fail :write [17 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,211] jepsen worker 53 - jepsen.util 593 :invoke :write [17 20] INFO [2022-03-13 17:20:04,213] jepsen worker 42 - jepsen.util 402 :fail :write [14 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,213] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,213] jepsen worker 77 - jepsen.util 707 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,214] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:20:04,215] jepsen worker 23 - jepsen.util 563 :invoke :write [7 23] INFO [2022-03-13 17:20:04,216] jepsen worker 77 - jepsen.util 707 :invoke :write [25 3] INFO [2022-03-13 17:20:04,218] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,219] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:04,228] jepsen worker 33 - jepsen.util 753 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,229] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:04,247] jepsen worker 3 - jepsen.util 2253 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,247] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,248] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:04,248] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:04,264] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,264] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:20:04,285] jepsen worker 88 - jepsen.util 2608 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,285] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 27] INFO [2022-03-13 17:20:04,346] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,346] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,347] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,347] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:20:04,347] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 30] INFO [2022-03-13 17:20:04,348] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:04,349] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,350] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 27] INFO [2022-03-13 17:20:04,353] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,354] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,358] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,358] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,360] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,361] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:20:04,366] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:04,366] jepsen worker nemesis - couchbase.util Rebalance check count: 710 INFO [2022-03-13 17:20:04,367] jepsen worker nemesis - couchbase.util Rebalance stuck count: 582 INFO [2022-03-13 17:20:04,401] jepsen worker 88 - jepsen.util 2608 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,402] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 45] INFO [2022-03-13 17:20:04,448] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,449] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 25] INFO [2022-03-13 17:20:04,475] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,476] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:04,516] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,517] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,526] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,527] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:20:04,539] jepsen worker 3 - jepsen.util 2253 :fail :write [1 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,540] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 10] INFO [2022-03-13 17:20:04,561] jepsen worker 76 - jepsen.util 796 :fail :write [25 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,561] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,562] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:04,563] jepsen worker 76 - jepsen.util 796 :invoke :write [25 22] INFO [2022-03-13 17:20:04,575] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,576] jepsen worker 71 - jepsen.util 611 :invoke :write [23 8] INFO [2022-03-13 17:20:04,590] jepsen worker 88 - jepsen.util 2608 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,591] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 1] INFO [2022-03-13 17:20:04,614] jepsen worker 5 - jepsen.util 2705 :fail :write [1 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,615] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:04,619] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,620] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:04,623] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,624] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:20:04,642] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,643] jepsen worker 35 - jepsen.util 935 :invoke :write [11 24] INFO [2022-03-13 17:20:04,654] jepsen worker 36 - jepsen.util 2466 :fail :write [12 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,655] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,656] jepsen worker 69 - jepsen.util 789 :fail :write [23 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,656] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 22] INFO [2022-03-13 17:20:04,658] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 26] INFO [2022-03-13 17:20:04,666] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,690] jepsen worker 22 - jepsen.util 652 :fail :write [7 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,690] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:04,703] jepsen worker 79 - jepsen.util 1159 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,703] jepsen worker 8 - jepsen.util 728 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,703] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:04,704] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:04,709] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,709] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,710] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,711] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,712] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,712] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,713] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,714] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:20:04,722] jepsen worker 3 - jepsen.util 2253 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,723] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 18] INFO [2022-03-13 17:20:04,732] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,732] jepsen worker 75 - jepsen.util 525 :fail :write [25 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,733] jepsen worker 75 - jepsen.util 525 :invoke :write [25 24] INFO [2022-03-13 17:20:04,733] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:04,734] jepsen worker 71 - jepsen.util 611 :fail :write [23 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,734] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,734] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,735] jepsen worker 69 - jepsen.util 789 :invoke :write [23 42] INFO [2022-03-13 17:20:04,736] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,737] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,738] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,739] jepsen worker 71 - jepsen.util 611 :invoke :write [23 12] INFO [2022-03-13 17:20:04,755] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,755] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,755] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,755] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,756] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,756] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,756] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:20:04,757] jepsen worker 34 - jepsen.util 934 :invoke :write [11 3] INFO [2022-03-13 17:20:04,757] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:04,758] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:04,759] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:04,760] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:04,784] jepsen worker 88 - jepsen.util 2608 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,785] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 49] INFO [2022-03-13 17:20:04,800] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,800] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 22] INFO [2022-03-13 17:20:04,851] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,851] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:04,851] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,851] jepsen worker 52 - jepsen.util 682 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,852] jepsen worker 26 - jepsen.util 566 :invoke :write [8 1] INFO [2022-03-13 17:20:04,852] jepsen worker 43 - jepsen.util 673 :invoke :write [14 46] INFO [2022-03-13 17:20:04,853] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:04,853] jepsen worker 69 - jepsen.util 789 :fail :write [23 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,853] jepsen worker 60 - jepsen.util 690 :invoke :write [20 14] INFO [2022-03-13 17:20:04,856] jepsen worker 36 - jepsen.util 2466 :fail :write [12 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,857] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:04,862] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,868] jepsen worker 80 - jepsen.util 800 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,868] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,869] jepsen worker 33 - jepsen.util 753 :invoke :write [11 3] INFO [2022-03-13 17:20:04,869] jepsen worker 80 - jepsen.util 800 :invoke :write [26 32] INFO [2022-03-13 17:20:04,876] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,877] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,878] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,879] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,884] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,885] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,886] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,887] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,889] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,890] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 47] INFO [2022-03-13 17:20:04,893] jepsen worker 78 - jepsen.util 708 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,893] jepsen worker 7 - jepsen.util 637 :fail :write [2 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,893] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,893] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,894] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:04,895] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:04,896] jepsen worker 12 - jepsen.util 912 :invoke :write [4 38] INFO [2022-03-13 17:20:04,896] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:04,928] jepsen worker 4 - jepsen.util 2614 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,928] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:04,931] jepsen worker 76 - jepsen.util 796 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,931] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,932] jepsen worker 76 - jepsen.util 796 :invoke :write [25 42] INFO [2022-03-13 17:20:04,932] jepsen worker 13 - jepsen.util 733 :invoke :write [4 44] INFO [2022-03-13 17:20:04,959] jepsen worker 35 - jepsen.util 935 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,959] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,960] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,960] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,960] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,959] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,959] jepsen worker 71 - jepsen.util 611 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,960] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:04,961] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,962] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:04,962] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,963] jepsen worker 14 - jepsen.util 374 :invoke :write [4 11] INFO [2022-03-13 17:20:04,965] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,965] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:04,965] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,965] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:20:04,965] jepsen worker 88 - jepsen.util 2608 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,966] jepsen worker 5 - jepsen.util 2705 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,966] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,967] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,967] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:04,968] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,969] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,969] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:04,970] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,970] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,970] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 22] INFO [2022-03-13 17:20:04,971] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,971] jepsen worker 69 - jepsen.util 789 :invoke :write [23 29] INFO [2022-03-13 17:20:04,971] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,971] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,972] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:20:04,973] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,974] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:04,975] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:04,976] jepsen worker 71 - jepsen.util 611 :invoke :write [23 9] INFO [2022-03-13 17:20:05,001] jepsen worker 3 - jepsen.util 2253 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,001] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,002] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,003] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:05,003] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 46] INFO [2022-03-13 17:20:05,004] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 9] INFO [2022-03-13 17:20:05,005] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,005] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:05,021] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,022] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:05,058] jepsen worker 53 - jepsen.util 593 :fail :write [17 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,059] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,059] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,059] jepsen worker 88 - jepsen.util 2608 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,059] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,060] jepsen worker 87 - jepsen.util 2877 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,061] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:05,061] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:05,062] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:05,063] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:20:05,064] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 7] INFO [2022-03-13 17:20:05,065] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 32] INFO [2022-03-13 17:20:05,070] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,070] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:05,128] jepsen worker 44 - jepsen.util 764 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,128] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,128] jepsen worker 23 - jepsen.util 563 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,129] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:05,129] jepsen worker 23 - jepsen.util 563 :invoke :write [7 28] INFO [2022-03-13 17:20:05,131] jepsen worker 70 - jepsen.util 880 :invoke :write [23 7] INFO [2022-03-13 17:20:05,135] jepsen worker 69 - jepsen.util 789 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,135] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,142] jepsen worker 4 - jepsen.util 2614 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,142] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 11] INFO [2022-03-13 17:20:05,143] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,143] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:20:05,150] jepsen worker 75 - jepsen.util 525 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,151] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:05,173] jepsen worker 34 - jepsen.util 934 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,174] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:05,220] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,221] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:20:05,224] jepsen worker 3 - jepsen.util 2253 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,238] jepsen worker 71 - jepsen.util 611 :fail :write [23 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,239] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,283] jepsen worker 71 - jepsen.util 611 :invoke :write [23 5] INFO [2022-03-13 17:20:05,261] jepsen worker 43 - jepsen.util 673 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,263] jepsen worker 80 - jepsen.util 800 :fail :write [26 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,283] jepsen worker 69 - jepsen.util 789 :invoke :write [23 37] INFO [2022-03-13 17:20:05,267] jepsen worker 88 - jepsen.util 2608 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,283] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 1] INFO [2022-03-13 17:20:05,283] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:05,283] jepsen worker 80 - jepsen.util 800 :invoke :write [26 45] INFO [2022-03-13 17:20:05,283] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,285] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,286] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 0] INFO [2022-03-13 17:20:05,298] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,299] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:20:05,309] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,309] jepsen worker 77 - jepsen.util 707 :fail :write [25 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,309] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,309] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,310] jepsen worker 6 - jepsen.util 636 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,310] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 5] INFO [2022-03-13 17:20:05,311] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:05,312] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:05,313] jepsen worker 77 - jepsen.util 707 :invoke :write [25 25] INFO [2022-03-13 17:20:05,314] jepsen worker 61 - jepsen.util 781 :invoke :write [20 28] INFO [2022-03-13 17:20:05,321] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,321] jepsen worker 33 - jepsen.util 753 :fail :write [11 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,322] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,323] jepsen worker 33 - jepsen.util 753 :invoke :write [11 38] INFO [2022-03-13 17:20:05,326] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,329] jepsen worker 76 - jepsen.util 796 :fail :write [25 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,329] jepsen worker 51 - jepsen.util 771 :invoke :write [17 42] INFO [2022-03-13 17:20:05,330] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:05,331] jepsen worker 25 - jepsen.util 565 :invoke :write [8 11] INFO [2022-03-13 17:20:05,335] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:05,371] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:05,372] jepsen worker nemesis - couchbase.util Rebalance check count: 711 INFO [2022-03-13 17:20:05,372] jepsen worker nemesis - couchbase.util Rebalance stuck count: 583 INFO [2022-03-13 17:20:05,378] jepsen worker 36 - jepsen.util 2466 :fail :write [12 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,379] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:05,428] jepsen worker 14 - jepsen.util 374 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,429] jepsen worker 14 - jepsen.util 374 :invoke :write [4 3] INFO [2022-03-13 17:20:05,432] jepsen worker 69 - jepsen.util 789 :fail :write [23 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,432] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,437] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,438] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 46] INFO [2022-03-13 17:20:05,480] jepsen worker 0 - jepsen.util 4770 :info :write [0 23] :RequestTimeoutException INFO [2022-03-13 17:20:05,481] jepsen worker 26 - jepsen.util 566 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,482] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:05,482] jepsen worker 0 - jepsen.util 4860 :invoke :read [0 nil] INFO [2022-03-13 17:20:05,509] jepsen worker 4 - jepsen.util 2614 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,509] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,511] jepsen worker 23 - jepsen.util 563 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,511] jepsen worker 62 - jepsen.util 512 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,511] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:05,511] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,512] jepsen worker 62 - jepsen.util 512 :invoke :write [20 33] INFO [2022-03-13 17:20:05,513] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:05,520] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,520] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:05,525] jepsen worker 71 - jepsen.util 611 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,525] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,525] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,526] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,526] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,526] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:05,527] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:05,528] jepsen worker 71 - jepsen.util 611 :invoke :write [23 43] INFO [2022-03-13 17:20:05,529] jepsen worker 21 - jepsen.util 831 :invoke :write [7 16] INFO [2022-03-13 17:20:05,529] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,548] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,548] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,548] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,549] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,549] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:05,549] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:05,550] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,550] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,551] jepsen worker 75 - jepsen.util 525 :invoke :write [25 41] INFO [2022-03-13 17:20:05,553] jepsen worker 13 - jepsen.util 733 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,554] jepsen worker 13 - jepsen.util 733 :invoke :write [4 0] INFO [2022-03-13 17:20:05,555] jepsen worker 76 - jepsen.util 796 :invoke :write [25 22] INFO [2022-03-13 17:20:05,587] jepsen worker 88 - jepsen.util 2608 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,587] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,588] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,589] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,596] jepsen worker 2 - jepsen.util 5222 :ok :read [0 41] INFO [2022-03-13 17:20:05,596] jepsen worker 2 - jepsen.util 5222 :invoke :read [0 nil] INFO [2022-03-13 17:20:05,597] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,597] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,598] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,599] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 19] INFO [2022-03-13 17:20:05,602] jepsen worker 87 - jepsen.util 2877 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,603] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,606] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,606] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 20] INFO [2022-03-13 17:20:05,644] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,644] jepsen worker 60 - jepsen.util 690 :fail :write [20 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,644] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,645] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,645] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,645] jepsen worker 71 - jepsen.util 611 :fail :write [23 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,645] jepsen worker 69 - jepsen.util 789 :invoke :write [23 11] INFO [2022-03-13 17:20:05,646] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,647] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:05,648] jepsen worker 22 - jepsen.util 652 :invoke :write [7 20] INFO [2022-03-13 17:20:05,649] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,650] jepsen worker 26 - jepsen.util 566 :invoke :write [8 43] INFO [2022-03-13 17:20:05,651] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,651] jepsen worker 53 - jepsen.util 593 :invoke :write [17 29] INFO [2022-03-13 17:20:05,651] jepsen worker 80 - jepsen.util 800 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,651] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:05,651] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,652] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:05,652] jepsen worker 80 - jepsen.util 800 :invoke :write [26 41] INFO [2022-03-13 17:20:05,655] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,655] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,656] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:05,656] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:05,657] jepsen worker 52 - jepsen.util 682 :invoke :write [17 13] INFO [2022-03-13 17:20:05,657] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,658] jepsen worker 71 - jepsen.util 611 :invoke :write [23 41] INFO [2022-03-13 17:20:05,659] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:05,662] jepsen worker 60 - jepsen.util 690 :invoke :write [20 23] INFO [2022-03-13 17:20:05,667] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:05,668] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:05,669] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:05,670] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:05,672] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:05,672] jepsen worker 43 - jepsen.util 673 :invoke :write [14 28] INFO [2022-03-13 17:20:05,686] jepsen worker 79 - jepsen.util 1159 :fail :write [26 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,686] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,686] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,687] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,687] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,687] jepsen worker 3 - jepsen.util 2253 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,688] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,688] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,688] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:05,688] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:05,688] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:05,689] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:05,689] jepsen worker 8 - jepsen.util 728 :invoke :write [2 20] INFO [2022-03-13 17:20:05,691] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,692] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,693] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:20:05,701] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,701] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,701] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,702] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,702] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,702] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 15] INFO [2022-03-13 17:20:05,722] jepsen worker 87 - jepsen.util 2877 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,723] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,724] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,725] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:20:05,741] jepsen worker 33 - jepsen.util 753 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,742] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:05,754] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,754] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,755] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,756] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,757] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,758] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:20:05,779] jepsen worker 88 - jepsen.util 2608 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,780] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,780] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,781] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,781] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 18] INFO [2022-03-13 17:20:05,781] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 46] INFO [2022-03-13 17:20:05,783] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,783] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 49] INFO [2022-03-13 17:20:05,788] jepsen worker 82 - jepsen.util 5212 :info :write [27 34] :RequestTimeoutException INFO [2022-03-13 17:20:05,789] jepsen worker 82 - jepsen.util 5302 :invoke :write [27 16] INFO [2022-03-13 17:20:05,814] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,815] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,817] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,818] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 21] INFO [2022-03-13 17:20:05,841] jepsen worker 36 - jepsen.util 2466 :fail :write [12 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,842] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:05,848] jepsen worker 75 - jepsen.util 525 :fail :write [25 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,849] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:05,849] jepsen worker 51 - jepsen.util 771 :fail :write [17 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,850] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:05,854] jepsen worker 69 - jepsen.util 789 :fail :write [23 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,855] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,880] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,881] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:20:05,909] jepsen worker 14 - jepsen.util 374 :fail :write [4 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,910] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:05,938] jepsen worker 87 - jepsen.util 2877 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,938] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:20:05,951] jepsen worker 71 - jepsen.util 611 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,951] jepsen worker 12 - jepsen.util 912 :fail :write [4 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,951] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,951] jepsen worker 77 - jepsen.util 707 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,952] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:05,952] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:05,952] jepsen worker 69 - jepsen.util 789 :invoke :write [23 45] INFO [2022-03-13 17:20:05,953] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:05,953] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:05,954] jepsen worker 44 - jepsen.util 764 :invoke :write [14 41] INFO [2022-03-13 17:20:05,958] jepsen worker 88 - jepsen.util 2608 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,959] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:05,960] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,960] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 24] INFO [2022-03-13 17:20:05,967] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,967] jepsen worker 4 - jepsen.util 2614 :fail :write [1 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,968] jepsen worker 71 - jepsen.util 611 :invoke :write [23 0] INFO [2022-03-13 17:20:05,969] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:05,976] jepsen worker 26 - jepsen.util 566 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,976] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:05,992] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,992] jepsen worker 25 - jepsen.util 565 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,993] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:05,994] jepsen worker 25 - jepsen.util 565 :invoke :write [8 19] INFO [2022-03-13 17:20:05,995] jepsen worker 34 - jepsen.util 934 :invoke :write [11 36] INFO [2022-03-13 17:20:05,998] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:06,031] jepsen worker 83 - jepsen.util 4583 :ok :read [27 4] INFO [2022-03-13 17:20:06,032] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,032] jepsen worker 83 - jepsen.util 4583 :invoke :read [27 nil] INFO [2022-03-13 17:20:06,033] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,034] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,035] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,038] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,041] jepsen worker 1 - jepsen.util 4681 :ok :read [0 41] INFO [2022-03-13 17:20:06,042] jepsen worker 1 - jepsen.util 4681 :invoke :write [0 36] INFO [2022-03-13 17:20:06,051] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,055] jepsen worker 85 - jepsen.util 4675 :ok :read [28 44] INFO [2022-03-13 17:20:06,056] jepsen worker 85 - jepsen.util 4675 :invoke :read [28 nil] INFO [2022-03-13 17:20:06,056] jepsen worker 2 - jepsen.util 5222 :ok :read [0 41] INFO [2022-03-13 17:20:06,057] jepsen worker 2 - jepsen.util 5222 :invoke :write [0 47] INFO [2022-03-13 17:20:06,062] jepsen worker 83 - jepsen.util 4583 :ok :read [27 4] INFO [2022-03-13 17:20:06,063] jepsen worker 83 - jepsen.util 4583 :invoke :write [27 23] INFO [2022-03-13 17:20:06,068] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,068] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,068] jepsen worker 3 - jepsen.util 2253 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,069] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:20:06,074] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 47] INFO [2022-03-13 17:20:06,075] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:20:06,096] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,096] jepsen worker 21 - jepsen.util 831 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,097] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:06,098] jepsen worker 21 - jepsen.util 831 :invoke :write [7 28] INFO [2022-03-13 17:20:06,099] jepsen worker 85 - jepsen.util 4675 :ok :read [28 44] INFO [2022-03-13 17:20:06,099] jepsen worker 85 - jepsen.util 4675 :invoke :write [28 10] INFO [2022-03-13 17:20:06,108] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,108] jepsen worker 70 - jepsen.util 880 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,108] jepsen worker 61 - jepsen.util 781 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,109] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:06,110] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,110] jepsen worker 70 - jepsen.util 880 :invoke :write [23 13] INFO [2022-03-13 17:20:06,110] jepsen worker 6 - jepsen.util 636 :invoke :write [2 31] INFO [2022-03-13 17:20:06,110] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,110] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,111] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,111] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:06,111] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,111] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:06,112] jepsen worker 61 - jepsen.util 781 :invoke :write [20 45] INFO [2022-03-13 17:20:06,112] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,112] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:06,112] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,113] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:06,113] jepsen worker 23 - jepsen.util 563 :invoke :write [7 36] INFO [2022-03-13 17:20:06,114] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,115] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,115] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,115] jepsen worker 78 - jepsen.util 708 :invoke :write [26 34] INFO [2022-03-13 17:20:06,115] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 42] INFO [2022-03-13 17:20:06,116] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,117] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:20:06,119] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,120] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:06,126] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,127] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:20:06,131] jepsen worker 88 - jepsen.util 2608 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,132] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,141] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,141] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,157] jepsen worker 80 - jepsen.util 800 :fail :write [26 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,157] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,158] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:06,158] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:06,161] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,161] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,161] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,162] jepsen worker 51 - jepsen.util 771 :invoke :write [17 31] INFO [2022-03-13 17:20:06,162] jepsen worker 35 - jepsen.util 935 :invoke :write [11 45] INFO [2022-03-13 17:20:06,163] jepsen worker 14 - jepsen.util 374 :invoke :write [4 43] INFO [2022-03-13 17:20:06,165] jepsen worker 13 - jepsen.util 733 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,165] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,166] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,166] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:06,166] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:06,167] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:06,170] jepsen worker 69 - jepsen.util 789 :fail :write [23 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,170] jepsen worker 69 - jepsen.util 789 :invoke :write [23 2] INFO [2022-03-13 17:20:06,174] jepsen worker 62 - jepsen.util 512 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,174] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:06,175] jepsen worker 62 - jepsen.util 512 :invoke :write [20 17] INFO [2022-03-13 17:20:06,181] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,181] jepsen worker 42 - jepsen.util 402 :invoke :write [14 26] INFO [2022-03-13 17:20:06,182] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:06,196] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,196] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 37] INFO [2022-03-13 17:20:06,227] jepsen worker 3 - jepsen.util 2253 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,227] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:20:06,256] jepsen worker 34 - jepsen.util 934 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,257] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,257] jepsen worker 34 - jepsen.util 934 :invoke :write [11 45] INFO [2022-03-13 17:20:06,258] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,261] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,261] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 9] INFO [2022-03-13 17:20:06,285] jepsen worker 53 - jepsen.util 593 :fail :write [17 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,285] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,285] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,285] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:06,286] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:06,286] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,295] jepsen worker 71 - jepsen.util 611 :fail :write [23 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,295] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:06,336] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,337] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 45] INFO [2022-03-13 17:20:06,340] jepsen worker 88 - jepsen.util 2608 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,341] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,343] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,344] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 35] INFO [2022-03-13 17:20:06,375] jepsen worker 25 - jepsen.util 565 :fail :write [8 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,376] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,376] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:06,376] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:06,378] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:06,378] jepsen worker nemesis - couchbase.util Rebalance check count: 712 INFO [2022-03-13 17:20:06,378] jepsen worker nemesis - couchbase.util Rebalance stuck count: 584 INFO [2022-03-13 17:20:06,396] jepsen worker 5 - jepsen.util 2705 :fail :write [1 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,396] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 44] INFO [2022-03-13 17:20:06,404] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,405] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,405] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,405] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:20:06,406] jepsen worker 8 - jepsen.util 728 :fail :write [2 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,406] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,407] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 49] INFO [2022-03-13 17:20:06,407] jepsen worker 23 - jepsen.util 563 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,408] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:06,408] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:06,409] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:06,409] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,415] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,415] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 33] INFO [2022-03-13 17:20:06,452] jepsen worker 69 - jepsen.util 789 :fail :write [23 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,452] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,453] jepsen worker 76 - jepsen.util 796 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,453] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,453] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,454] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:06,455] jepsen worker 75 - jepsen.util 525 :invoke :write [25 39] INFO [2022-03-13 17:20:06,455] jepsen worker 76 - jepsen.util 796 :invoke :write [25 23] INFO [2022-03-13 17:20:06,456] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:20:06,456] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,456] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:06,457] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:20:06,463] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,463] jepsen worker 69 - jepsen.util 789 :invoke :write [23 12] INFO [2022-03-13 17:20:06,470] jepsen worker 21 - jepsen.util 831 :fail :write [7 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,471] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,471] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,471] jepsen worker 21 - jepsen.util 831 :invoke :write [7 24] INFO [2022-03-13 17:20:06,472] jepsen worker 88 - jepsen.util 2608 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,472] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:06,473] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,473] jepsen worker 80 - jepsen.util 800 :invoke :write [26 34] INFO [2022-03-13 17:20:06,477] jepsen worker 87 - jepsen.util 2877 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,478] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,480] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,481] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,482] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,483] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,486] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,486] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:20:06,492] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,493] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:06,497] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,498] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 24] INFO [2022-03-13 17:20:06,547] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,548] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 22] INFO [2022-03-13 17:20:06,548] jepsen worker 5 - jepsen.util 2705 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,549] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,550] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,551] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 43] INFO [2022-03-13 17:20:06,571] jepsen worker 66 - jepsen.util 4746 :ok :read [22 30] INFO [2022-03-13 17:20:06,572] jepsen worker 66 - jepsen.util 4746 :invoke :read [22 nil] INFO [2022-03-13 17:20:06,617] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,617] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:06,635] jepsen worker 4 - jepsen.util 2614 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,636] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,652] jepsen worker 22 - jepsen.util 652 :fail :write [7 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,652] jepsen worker 22 - jepsen.util 652 :invoke :write [7 45] INFO [2022-03-13 17:20:06,654] jepsen worker 66 - jepsen.util 4746 :ok :read [22 30] INFO [2022-03-13 17:20:06,654] jepsen worker 66 - jepsen.util 4746 :invoke :write [22 30] INFO [2022-03-13 17:20:06,659] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,659] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:20:06,717] jepsen worker 88 - jepsen.util 2608 :fail :write [29 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,718] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 10] INFO [2022-03-13 17:20:06,754] jepsen worker 5 - jepsen.util 2705 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,755] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:20:06,769] jepsen worker 35 - jepsen.util 935 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,770] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:06,771] jepsen worker 44 - jepsen.util 764 :fail :write [14 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,771] jepsen worker 61 - jepsen.util 781 :fail :write [20 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,771] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:06,772] jepsen worker 61 - jepsen.util 781 :invoke :write [20 7] INFO [2022-03-13 17:20:06,796] jepsen worker 36 - jepsen.util 2466 :fail :write [12 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,797] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:06,819] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,819] jepsen worker 69 - jepsen.util 789 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,820] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:06,821] jepsen worker 71 - jepsen.util 611 :invoke :write [23 7] INFO [2022-03-13 17:20:06,827] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,828] jepsen worker 69 - jepsen.util 789 :invoke :write [23 49] INFO [2022-03-13 17:20:06,849] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,850] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:20:06,867] jepsen worker 60 - jepsen.util 690 :fail :write [20 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,867] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:06,869] jepsen worker 51 - jepsen.util 771 :fail :write [17 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,870] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:06,890] jepsen worker 3 - jepsen.util 2253 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,890] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,893] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,893] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,894] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 0] INFO [2022-03-13 17:20:06,894] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 33] INFO [2022-03-13 17:20:06,930] jepsen worker 88 - jepsen.util 2608 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,931] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 7] INFO [2022-03-13 17:20:06,936] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,937] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:06,941] jepsen worker 14 - jepsen.util 374 :fail :write [4 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,942] jepsen worker 79 - jepsen.util 1159 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,942] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:06,942] jepsen worker 70 - jepsen.util 880 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,943] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:06,943] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 11] INFO [2022-03-13 17:20:06,951] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,952] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:20:06,965] jepsen worker 76 - jepsen.util 796 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,966] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:06,976] jepsen worker 71 - jepsen.util 611 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,977] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:06,998] jepsen worker 4 - jepsen.util 2614 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:06,999] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 1] INFO [2022-03-13 17:20:07,040] jepsen worker 88 - jepsen.util 2608 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,041] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 30] INFO [2022-03-13 17:20:07,044] jepsen worker 34 - jepsen.util 934 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,044] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,044] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,044] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,045] jepsen worker 33 - jepsen.util 753 :invoke :write [11 25] INFO [2022-03-13 17:20:07,046] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:07,046] jepsen worker 25 - jepsen.util 565 :invoke :write [8 5] INFO [2022-03-13 17:20:07,047] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:07,053] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,053] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,062] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,062] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,067] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,068] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:20:07,075] jepsen worker 36 - jepsen.util 2466 :fail :write [12 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,075] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,075] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 14] INFO [2022-03-13 17:20:07,076] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,111] jepsen worker 69 - jepsen.util 789 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,112] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,112] jepsen worker 69 - jepsen.util 789 :invoke :write [23 38] INFO [2022-03-13 17:20:07,115] jepsen worker 71 - jepsen.util 611 :invoke :write [23 38] INFO [2022-03-13 17:20:07,136] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,136] jepsen worker 6 - jepsen.util 636 :fail :write [2 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,136] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,136] jepsen worker 78 - jepsen.util 708 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,136] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,137] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,137] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,137] jepsen worker 78 - jepsen.util 708 :invoke :write [26 40] INFO [2022-03-13 17:20:07,138] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:07,138] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,139] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:20:07,139] jepsen worker 6 - jepsen.util 636 :invoke :write [2 13] INFO [2022-03-13 17:20:07,146] jepsen worker 7 - jepsen.util 637 :invoke :write [2 11] INFO [2022-03-13 17:20:07,151] jepsen worker 21 - jepsen.util 831 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,152] jepsen worker 21 - jepsen.util 831 :invoke :write [7 21] INFO [2022-03-13 17:20:07,154] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,155] jepsen worker 23 - jepsen.util 563 :invoke :write [7 47] INFO [2022-03-13 17:20:07,155] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 19] INFO [2022-03-13 17:20:07,170] jepsen worker 4 - jepsen.util 2614 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,170] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,180] jepsen worker 52 - jepsen.util 682 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,180] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,180] jepsen worker 75 - jepsen.util 525 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,182] jepsen worker 75 - jepsen.util 525 :invoke :write [25 10] INFO [2022-03-13 17:20:07,183] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:07,184] jepsen worker 26 - jepsen.util 566 :invoke :write [8 11] INFO [2022-03-13 17:20:07,193] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,193] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:20:07,205] jepsen worker 88 - jepsen.util 2608 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,206] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,209] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,210] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,212] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,213] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,214] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,215] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,216] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,216] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,220] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,221] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 45] INFO [2022-03-13 17:20:07,238] jepsen worker 69 - jepsen.util 789 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,239] jepsen worker 69 - jepsen.util 789 :invoke :write [23 25] INFO [2022-03-13 17:20:07,263] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,264] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:20:07,273] jepsen worker 80 - jepsen.util 800 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,273] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,273] jepsen worker 35 - jepsen.util 935 :invoke :write [11 20] INFO [2022-03-13 17:20:07,274] jepsen worker 80 - jepsen.util 800 :invoke :write [26 45] INFO [2022-03-13 17:20:07,275] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,275] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,275] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,275] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,276] jepsen worker 51 - jepsen.util 771 :invoke :write [17 8] INFO [2022-03-13 17:20:07,276] jepsen worker 24 - jepsen.util 654 :invoke :write [8 24] INFO [2022-03-13 17:20:07,277] jepsen worker 14 - jepsen.util 374 :invoke :write [4 44] INFO [2022-03-13 17:20:07,277] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:07,294] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,294] jepsen worker 36 - jepsen.util 2466 :fail :write [12 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,296] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:20:07,296] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] WARN [2022-03-13 17:20:07,307] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":10853925,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":10853925,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9035244,"encode_duration_us":18982,"operation_id":"0x14d76","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10007178}],"total_count":1}} INFO [2022-03-13 17:20:07,344] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,345] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,346] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,346] jepsen worker 71 - jepsen.util 611 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,347] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:07,347] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:20:07,353] jepsen worker 61 - jepsen.util 781 :fail :write [20 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,353] jepsen worker 42 - jepsen.util 402 :fail :write [14 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,354] jepsen worker 61 - jepsen.util 781 :invoke :write [20 27] INFO [2022-03-13 17:20:07,354] jepsen worker 42 - jepsen.util 402 :invoke :write [14 6] INFO [2022-03-13 17:20:07,368] jepsen worker 25 - jepsen.util 565 :fail :write [8 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,369] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:07,383] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:07,383] jepsen worker nemesis - couchbase.util Rebalance check count: 713 INFO [2022-03-13 17:20:07,383] jepsen worker nemesis - couchbase.util Rebalance stuck count: 585 WARN [2022-03-13 17:20:07,407] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":6639240,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":6639240,"total_server_duration_us":43,"operation_id":"0x14205","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":30003120},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","encode_duration_us":17500,"operation_id":"0x14c5e","timeout_ms":10000,"total_duration_us":10010099},{"operation_name":"upsert","last_dispatch_duration_us":34531,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":846745,"total_server_duration_us":548167,"encode_duration_us":18627,"operation_id":"0x14cf4","timeout_ms":10000,"last_server_duration_us":21487,"total_duration_us":10009676},{"operation_name":"upsert","last_dispatch_duration_us":8215779,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8215779,"total_server_duration_us":19418,"encode_duration_us":20481,"operation_id":"0x14e8e","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10008075},{"operation_name":"upsert","last_dispatch_duration_us":10853925,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10853925,"total_server_duration_us":9035244,"encode_duration_us":18982,"operation_id":"0x14d76","timeout_ms":10000,"last_server_duration_us":9035244,"total_duration_us":10007178},{"operation_name":"get","last_dispatch_duration_us":9059869,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9059869,"total_server_duration_us":69,"operation_id":"0x14e48","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":9062144},{"operation_name":"upsert","last_dispatch_duration_us":423084,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":423084,"total_server_duration_us":61901,"encode_duration_us":35921,"operation_id":"0x151dd","timeout_ms":10000,"last_server_duration_us":61901,"total_duration_us":2535899},{"operation_name":"upsert","last_dispatch_duration_us":1533113,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1533113,"total_server_duration_us":77851,"encode_duration_us":33029,"operation_id":"0x151db","timeout_ms":10000,"last_server_duration_us":77851,"total_duration_us":2099002},{"operation_name":"upsert","last_dispatch_duration_us":1596543,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1596543,"total_server_duration_us":1015862,"encode_duration_us":32204,"operation_id":"0x151da","timeout_ms":10000,"last_server_duration_us":1015862,"total_duration_us":1602275},{"operation_name":"upsert","last_dispatch_duration_us":1531183,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":1531183,"total_server_duration_us":79120,"encode_duration_us":20970,"operation_id":"0x1520a","timeout_ms":10000,"last_server_duration_us":79120,"total_duration_us":1537131}],"total_count":85}} INFO [2022-03-13 17:20:07,421] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,421] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,421] jepsen worker 43 - jepsen.util 673 :fail :write [14 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,422] jepsen worker 43 - jepsen.util 673 :invoke :write [14 2] INFO [2022-03-13 17:20:07,422] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:07,423] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,423] jepsen worker 13 - jepsen.util 733 :invoke :write [4 14] INFO [2022-03-13 17:20:07,423] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:07,444] jepsen worker 88 - jepsen.util 2608 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,445] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 9] INFO [2022-03-13 17:20:07,448] jepsen worker 3 - jepsen.util 2253 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,448] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,448] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,449] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:07,449] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 16] INFO [2022-03-13 17:20:07,454] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,456] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,457] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:20:07,482] jepsen worker 33 - jepsen.util 753 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,483] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:07,494] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,494] jepsen worker 69 - jepsen.util 789 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,495] jepsen worker 69 - jepsen.util 789 :invoke :write [23 19] INFO [2022-03-13 17:20:07,496] jepsen worker 71 - jepsen.util 611 :invoke :write [23 20] INFO [2022-03-13 17:20:07,517] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,518] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,521] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,522] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,524] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,525] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:20:07,532] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,532] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,532] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:07,533] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,548] jepsen worker 79 - jepsen.util 1159 :fail :write [26 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,548] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,549] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,549] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:07,548] jepsen worker 62 - jepsen.util 512 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,549] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,549] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,550] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:07,551] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:07,552] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:07,553] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:07,554] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:07,555] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:07,555] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:07,573] jepsen worker 21 - jepsen.util 831 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,573] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,574] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:07,575] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:07,595] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,595] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,597] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,598] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,599] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,600] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:07,652] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,654] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:20:07,678] jepsen worker 35 - jepsen.util 935 :fail :write [11 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,679] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:07,684] jepsen worker 69 - jepsen.util 789 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,684] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:07,694] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,695] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,697] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,697] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,699] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,699] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,704] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,705] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,706] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,707] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:20:07,716] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,716] jepsen worker 3 - jepsen.util 2253 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,716] jepsen worker 88 - jepsen.util 2608 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,716] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,718] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,719] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,719] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 6] INFO [2022-03-13 17:20:07,720] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:20:07,721] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,721] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,722] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,723] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:20:07,724] jepsen worker 22 - jepsen.util 652 :fail :write [7 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,724] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,724] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,724] jepsen worker 75 - jepsen.util 525 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,724] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,726] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:07,726] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,726] jepsen worker 22 - jepsen.util 652 :invoke :write [7 29] INFO [2022-03-13 17:20:07,727] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,728] jepsen worker 60 - jepsen.util 690 :invoke :write [20 17] INFO [2022-03-13 17:20:07,728] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,728] jepsen worker 75 - jepsen.util 525 :invoke :write [25 44] INFO [2022-03-13 17:20:07,728] jepsen worker 76 - jepsen.util 796 :invoke :write [25 35] INFO [2022-03-13 17:20:07,742] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:20:07,767] jepsen worker 78 - jepsen.util 708 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,767] jepsen worker 6 - jepsen.util 636 :fail :write [2 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,768] jepsen worker 6 - jepsen.util 636 :invoke :write [2 42] INFO [2022-03-13 17:20:07,768] jepsen worker 78 - jepsen.util 708 :invoke :write [26 40] INFO [2022-03-13 17:20:07,802] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,802] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:07,812] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,813] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 5] INFO [2022-03-13 17:20:07,822] jepsen worker 24 - jepsen.util 654 :fail :write [8 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,822] jepsen worker 71 - jepsen.util 611 :fail :write [23 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,822] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,823] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:07,823] jepsen worker 69 - jepsen.util 789 :invoke :write [23 34] INFO [2022-03-13 17:20:07,824] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:07,843] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,843] jepsen worker 71 - jepsen.util 611 :invoke :write [23 25] INFO [2022-03-13 17:20:07,879] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,879] jepsen worker 26 - jepsen.util 566 :fail :write [8 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,879] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:07,880] jepsen worker 26 - jepsen.util 566 :invoke :write [8 33] INFO [2022-03-13 17:20:07,911] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,911] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:07,913] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,914] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:20:07,918] jepsen worker 14 - jepsen.util 374 :fail :write [4 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,918] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,919] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:07,920] jepsen worker 25 - jepsen.util 565 :invoke :write [8 31] INFO [2022-03-13 17:20:07,941] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,941] jepsen worker 88 - jepsen.util 2608 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,942] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:07,942] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:20:07,943] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,943] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 42] INFO [2022-03-13 17:20:07,956] jepsen worker 69 - jepsen.util 789 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,956] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:07,963] jepsen worker 13 - jepsen.util 733 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:07,967] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,004] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,004] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,005] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:08,006] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:20:08,020] jepsen worker 7 - jepsen.util 637 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,021] jepsen worker 23 - jepsen.util 563 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,021] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:08,022] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:08,027] jepsen worker 87 - jepsen.util 2877 :fail :write [29 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,028] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:20:08,053] jepsen worker 80 - jepsen.util 800 :fail :write [26 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,054] jepsen worker 80 - jepsen.util 800 :invoke :write [26 3] INFO [2022-03-13 17:20:08,060] jepsen worker 71 - jepsen.util 611 :fail :write [23 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,060] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,061] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,061] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,067] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,067] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,068] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,068] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,070] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,070] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,070] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,071] jepsen worker 71 - jepsen.util 611 :invoke :write [23 34] INFO [2022-03-13 17:20:08,075] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,076] jepsen worker 69 - jepsen.util 789 :invoke :write [23 28] INFO [2022-03-13 17:20:08,087] jepsen worker 58 - jepsen.util 4378 :info :write [19 0] :RequestTimeoutException INFO [2022-03-13 17:20:08,088] jepsen worker 58 - jepsen.util 4468 :invoke :read [19 nil] INFO [2022-03-13 17:20:08,115] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,115] jepsen worker 88 - jepsen.util 2608 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,115] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,115] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,116] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 35] INFO [2022-03-13 17:20:08,118] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:08,119] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:20:08,120] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:20:08,123] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,124] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 44] INFO [2022-03-13 17:20:08,147] jepsen worker 51 - jepsen.util 771 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,147] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,147] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,147] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,149] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:08,150] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,150] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,150] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,150] jepsen worker 21 - jepsen.util 831 :invoke :write [7 31] INFO [2022-03-13 17:20:08,152] jepsen worker 14 - jepsen.util 374 :invoke :write [4 4] INFO [2022-03-13 17:20:08,153] jepsen worker 72 - jepsen.util 4932 :ok :read [24 24] INFO [2022-03-13 17:20:08,154] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:08,154] jepsen worker 51 - jepsen.util 771 :invoke :write [17 7] INFO [2022-03-13 17:20:08,155] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:08,159] jepsen worker 72 - jepsen.util 4932 :invoke :read [24 nil] INFO [2022-03-13 17:20:08,163] jepsen worker 34 - jepsen.util 934 :invoke :write [11 43] INFO [2022-03-13 17:20:08,167] jepsen worker 11 - jepsen.util 4691 :ok :read [3 25] INFO [2022-03-13 17:20:08,171] jepsen worker 11 - jepsen.util 4691 :invoke :write [3 20] INFO [2022-03-13 17:20:08,173] jepsen worker 19 - jepsen.util 4699 :ok :read [6 16] INFO [2022-03-13 17:20:08,174] jepsen worker 19 - jepsen.util 4699 :invoke :read [6 nil] INFO [2022-03-13 17:20:08,183] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,185] jepsen worker 43 - jepsen.util 673 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,185] jepsen worker 75 - jepsen.util 525 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,185] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 17] INFO [2022-03-13 17:20:08,185] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,186] jepsen worker 75 - jepsen.util 525 :invoke :write [25 17] INFO [2022-03-13 17:20:08,190] jepsen worker 43 - jepsen.util 673 :invoke :write [14 16] INFO [2022-03-13 17:20:08,196] jepsen worker 72 - jepsen.util 4932 :ok :read [24 24] INFO [2022-03-13 17:20:08,197] jepsen worker 72 - jepsen.util 4932 :invoke :read [24 nil] INFO [2022-03-13 17:20:08,198] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:08,200] jepsen worker 10 - jepsen.util 5230 :ok :read [3 25] INFO [2022-03-13 17:20:08,201] jepsen worker 10 - jepsen.util 5230 :invoke :write [3 18] INFO [2022-03-13 17:20:08,209] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,211] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:20:08,229] jepsen worker 71 - jepsen.util 611 :fail :write [23 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,230] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,231] jepsen worker 19 - jepsen.util 4699 :ok :read [6 16] INFO [2022-03-13 17:20:08,233] jepsen worker 19 - jepsen.util 4699 :invoke :write [6 7] INFO [2022-03-13 17:20:08,242] jepsen worker 42 - jepsen.util 402 :fail :write [14 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,242] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,242] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:08,242] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,242] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,242] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,242] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,242] jepsen worker 61 - jepsen.util 781 :fail :write [20 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,243] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,243] jepsen worker 25 - jepsen.util 565 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,244] jepsen worker 70 - jepsen.util 880 :invoke :write [23 41] INFO [2022-03-13 17:20:08,245] jepsen worker 8 - jepsen.util 728 :invoke :write [2 39] INFO [2022-03-13 17:20:08,246] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:08,247] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:08,247] jepsen worker 62 - jepsen.util 512 :invoke :write [20 21] INFO [2022-03-13 17:20:08,248] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,248] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 44] INFO [2022-03-13 17:20:08,249] jepsen worker 61 - jepsen.util 781 :invoke :write [20 2] INFO [2022-03-13 17:20:08,250] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:08,250] jepsen worker 72 - jepsen.util 4932 :ok :read [24 24] INFO [2022-03-13 17:20:08,251] jepsen worker 72 - jepsen.util 4932 :invoke :write [24 10] INFO [2022-03-13 17:20:08,256] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:08,280] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,282] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:20:08,314] jepsen worker 47 - jepsen.util 4727 :info :write [15 37] :SyncWriteAmbiguous INFO [2022-03-13 17:20:08,315] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,315] jepsen worker 47 - jepsen.util 4817 :invoke :write [15 15] INFO [2022-03-13 17:20:08,316] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:20:08,329] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,329] jepsen worker 69 - jepsen.util 789 :fail :write [23 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,330] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,330] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:20:08,330] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,331] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:20:08,331] jepsen worker 88 - jepsen.util 2608 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,332] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 38] INFO [2022-03-13 17:20:08,340] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,341] jepsen worker 71 - jepsen.util 611 :invoke :write [23 4] INFO [2022-03-13 17:20:08,357] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,357] jepsen worker 80 - jepsen.util 800 :fail :write [26 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,358] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,358] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,359] jepsen worker 24 - jepsen.util 654 :invoke :write [8 35] INFO [2022-03-13 17:20:08,360] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:08,361] jepsen worker 33 - jepsen.util 753 :invoke :write [11 24] INFO [2022-03-13 17:20:08,362] jepsen worker 80 - jepsen.util 800 :invoke :write [26 38] INFO [2022-03-13 17:20:08,364] jepsen worker 64 - jepsen.util 4744 :info :write [21 17] :SyncWriteAmbiguous INFO [2022-03-13 17:20:08,371] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,373] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,373] jepsen worker 64 - jepsen.util 4834 :invoke :write [21 9] INFO [2022-03-13 17:20:08,377] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,379] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:20:08,384] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,385] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,386] jepsen worker 22 - jepsen.util 652 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,386] jepsen worker 76 - jepsen.util 796 :fail :write [25 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,387] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:08,388] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:08,390] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:08,390] jepsen worker nemesis - couchbase.util Rebalance check count: 714 INFO [2022-03-13 17:20:08,390] jepsen worker nemesis - couchbase.util Rebalance stuck count: 586 INFO [2022-03-13 17:20:08,398] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:08,399] jepsen worker 50 - jepsen.util 3470 :invoke :write [16 26] INFO [2022-03-13 17:20:08,403] jepsen worker 4 - jepsen.util 2614 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,404] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,408] jepsen worker 18 - jepsen.util 4878 :ok :read [6 16] INFO [2022-03-13 17:20:08,414] jepsen worker 18 - jepsen.util 4878 :invoke :read [6 nil] INFO [2022-03-13 17:20:08,459] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,460] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:20:08,465] jepsen worker 31 - jepsen.util 5161 :ok :read [10 :nil] INFO [2022-03-13 17:20:08,466] jepsen worker 31 - jepsen.util 5161 :invoke :read [10 nil] INFO [2022-03-13 17:20:08,467] jepsen worker 6 - jepsen.util 636 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,467] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:08,467] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,468] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:08,467] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,468] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:08,469] jepsen worker 44 - jepsen.util 764 :invoke :write [14 48] INFO [2022-03-13 17:20:08,470] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:08,471] jepsen worker 7 - jepsen.util 637 :invoke :write [2 8] INFO [2022-03-13 17:20:08,473] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,479] jepsen worker 78 - jepsen.util 708 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,479] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,480] jepsen worker 73 - jepsen.util 5293 :ok :read [24 24] INFO [2022-03-13 17:20:08,480] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,480] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:08,481] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:08,482] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:08,485] jepsen worker 73 - jepsen.util 5293 :invoke :write [24 11] INFO [2022-03-13 17:20:08,486] jepsen worker 31 - jepsen.util 5161 :ok :read [10 :nil] INFO [2022-03-13 17:20:08,487] jepsen worker 31 - jepsen.util 5161 :invoke :read [10 nil] INFO [2022-03-13 17:20:08,489] jepsen worker 3 - jepsen.util 2253 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,490] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:20:08,504] jepsen worker 18 - jepsen.util 4878 :ok :read [6 16] INFO [2022-03-13 17:20:08,504] jepsen worker 18 - jepsen.util 4878 :invoke :write [6 16] INFO [2022-03-13 17:20:08,511] jepsen worker 21 - jepsen.util 831 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,512] jepsen worker 21 - jepsen.util 831 :invoke :write [7 15] INFO [2022-03-13 17:20:08,522] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,522] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:08,526] jepsen worker 88 - jepsen.util 2608 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,527] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:08,529] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,530] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:08,535] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,536] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:08,537] jepsen worker 60 - jepsen.util 690 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,537] jepsen worker 75 - jepsen.util 525 :fail :write [25 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,537] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,537] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,537] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,537] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,537] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,538] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,538] jepsen worker 53 - jepsen.util 593 :invoke :write [17 43] INFO [2022-03-13 17:20:08,539] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,539] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 2] INFO [2022-03-13 17:20:08,540] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,541] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:20:08,542] jepsen worker 60 - jepsen.util 690 :invoke :write [20 11] INFO [2022-03-13 17:20:08,542] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,542] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:08,543] jepsen worker 76 - jepsen.util 796 :invoke :write [25 12] INFO [2022-03-13 17:20:08,543] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,548] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,549] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:08,549] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,550] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:08,552] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,552] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,553] jepsen worker 31 - jepsen.util 5161 :ok :read [10 :nil] INFO [2022-03-13 17:20:08,553] jepsen worker 13 - jepsen.util 733 :invoke :write [4 41] INFO [2022-03-13 17:20:08,554] jepsen worker 75 - jepsen.util 525 :invoke :write [25 29] INFO [2022-03-13 17:20:08,555] jepsen worker 31 - jepsen.util 5161 :invoke :read [10 nil] INFO [2022-03-13 17:20:08,569] jepsen worker 51 - jepsen.util 771 :fail :write [17 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,570] jepsen worker 51 - jepsen.util 771 :invoke :write [17 36] INFO [2022-03-13 17:20:08,605] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,605] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,605] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,606] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:20:08,620] jepsen worker 81 - jepsen.util 5571 :info :write [27 11] :RequestTimeoutException INFO [2022-03-13 17:20:08,621] jepsen worker 81 - jepsen.util 5661 :invoke :read [27 nil] INFO [2022-03-13 17:20:08,631] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,632] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:08,633] jepsen worker 34 - jepsen.util 934 :fail :write [11 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,634] jepsen worker 34 - jepsen.util 934 :invoke :write [11 41] INFO [2022-03-13 17:20:08,643] jepsen worker 41 - jepsen.util 4901 :info :write [13 22] :RequestTimeoutException INFO [2022-03-13 17:20:08,644] jepsen worker 41 - jepsen.util 4991 :invoke :write [13 4] INFO [2022-03-13 17:20:08,647] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,648] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 4] INFO [2022-03-13 17:20:08,663] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,663] jepsen worker 26 - jepsen.util 566 :fail :write [8 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,665] jepsen worker 52 - jepsen.util 682 :invoke :write [17 14] INFO [2022-03-13 17:20:08,665] jepsen worker 26 - jepsen.util 566 :invoke :write [8 8] INFO [2022-03-13 17:20:08,700] jepsen worker 79 - jepsen.util 1159 :fail :write [26 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,701] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:08,702] jepsen worker 31 - jepsen.util 5161 :ok :read [10 :nil] INFO [2022-03-13 17:20:08,703] jepsen worker 31 - jepsen.util 5161 :invoke :read [10 nil] INFO [2022-03-13 17:20:08,703] jepsen worker 8 - jepsen.util 728 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,704] jepsen worker 8 - jepsen.util 728 :invoke :write [2 9] WARN [2022-03-13 17:20:08,709] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16220397,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16637684,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9292916,"encode_duration_us":23538,"operation_id":"0x14b7e","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10008373},{"operation_name":"upsert","last_dispatch_duration_us":19360917,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":28079595,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9072197,"encode_duration_us":19635,"operation_id":"0x14191","timeout_ms":10000,"last_server_duration_us":9052231,"total_duration_us":10010396}],"total_count":2}} INFO [2022-03-13 17:20:08,713] jepsen worker 74 - jepsen.util 4844 :ok :read [24 24] INFO [2022-03-13 17:20:08,713] jepsen worker 20 - jepsen.util 5150 :ok :read [6 16] INFO [2022-03-13 17:20:08,714] jepsen worker 74 - jepsen.util 4844 :invoke :write [24 4] INFO [2022-03-13 17:20:08,714] jepsen worker 20 - jepsen.util 5150 :invoke :read [6 nil] INFO [2022-03-13 17:20:08,719] jepsen worker 14 - jepsen.util 374 :fail :write [4 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,719] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,720] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:08,721] jepsen worker 14 - jepsen.util 374 :invoke :write [4 12] INFO [2022-03-13 17:20:08,727] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,728] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,731] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,732] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:20:08,743] jepsen worker 50 - jepsen.util 3470 :fail :write [16 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,743] jepsen worker 88 - jepsen.util 2608 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,744] jepsen worker 32 - jepsen.util 5162 :ok :read [10 :nil] INFO [2022-03-13 17:20:08,744] jepsen worker 50 - jepsen.util 3470 :invoke :read [16 nil] INFO [2022-03-13 17:20:08,746] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 35] INFO [2022-03-13 17:20:08,746] jepsen worker 32 - jepsen.util 5162 :invoke :write [10 14] INFO [2022-03-13 17:20:08,776] jepsen worker 20 - jepsen.util 5150 :ok :read [6 16] INFO [2022-03-13 17:20:08,777] jepsen worker 20 - jepsen.util 5150 :invoke :write [6 38] INFO [2022-03-13 17:20:08,786] jepsen worker 43 - jepsen.util 673 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,786] jepsen worker 76 - jepsen.util 796 :fail :write [25 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,786] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,787] jepsen worker 76 - jepsen.util 796 :invoke :write [25 18] INFO [2022-03-13 17:20:08,788] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:08,788] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:08,795] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,795] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 47] INFO [2022-03-13 17:20:08,797] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,798] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,800] jepsen worker 87 - jepsen.util 2877 :fail :write [29 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,801] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] WARN [2022-03-13 17:20:08,814] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":9233546,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9233546,"total_server_duration_us":387308,"operation_id":"0x147ec","timeout_ms":30000,"last_server_duration_us":387308,"total_duration_us":15927397},{"operation_name":"upsert","last_dispatch_duration_us":15885024,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":15885024,"total_server_duration_us":9049045,"encode_duration_us":21758,"operation_id":"0x14d08","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10008486},{"operation_name":"upsert","last_dispatch_duration_us":29275,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":2639529,"total_server_duration_us":373950,"encode_duration_us":16862,"operation_id":"0x1507f","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10004809},{"operation_name":"upsert","last_dispatch_duration_us":56810,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9368147,"total_server_duration_us":298633,"encode_duration_us":27575,"operation_id":"0x14c58","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10001439},{"operation_name":"get","last_dispatch_duration_us":9074755,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9074755,"total_server_duration_us":12504,"operation_id":"0x150be","timeout_ms":30000,"last_server_duration_us":12504,"total_duration_us":9077043},{"operation_name":"upsert","last_dispatch_duration_us":9065756,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9065756,"total_server_duration_us":9028877,"encode_duration_us":26581,"operation_id":"0x14c8b","timeout_ms":10000,"last_server_duration_us":9028877,"total_duration_us":9070799},{"operation_name":"get","last_dispatch_duration_us":9048659,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9048659,"total_server_duration_us":56,"operation_id":"0x150c1","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":9051738},{"operation_name":"get","last_dispatch_duration_us":8683189,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8683189,"total_server_duration_us":398778,"operation_id":"0x14bcf","timeout_ms":30000,"last_server_duration_us":398778,"total_duration_us":8686571},{"operation_name":"get","last_dispatch_duration_us":8153229,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":8153229,"total_server_duration_us":7028,"operation_id":"0x1516a","timeout_ms":30000,"last_server_duration_us":7028,"total_duration_us":8155642},{"operation_name":"get","last_dispatch_duration_us":5915565,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":5915565,"total_server_duration_us":43,"operation_id":"0x1529a","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":5919701}],"total_count":62}} WARN [2022-03-13 17:20:08,820] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17111802,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26583323,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":8334050,"encode_duration_us":21211,"operation_id":"0x1459d","timeout_ms":10000,"last_server_duration_us":8192090,"total_duration_us":10004048},{"operation_name":"upsert","last_dispatch_duration_us":9150975,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15518014,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9162786,"encode_duration_us":17329,"operation_id":"0x1482d","timeout_ms":10000,"last_server_duration_us":9075609,"total_duration_us":10006854}],"total_count":2}} WARN [2022-03-13 17:20:08,820] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":147459,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":8951852,"total_server_duration_us":5513,"operation_id":"0x14540","timeout_ms":30000,"last_server_duration_us":362,"total_duration_us":30001748},{"operation_name":"upsert","last_dispatch_duration_us":60534,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9007624,"total_server_duration_us":227850,"encode_duration_us":19442,"operation_id":"0x14c89","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":10004012},{"operation_name":"upsert","last_dispatch_duration_us":50575,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7985272,"total_server_duration_us":184639,"encode_duration_us":27908,"operation_id":"0x14d2b","timeout_ms":10000,"last_server_duration_us":27132,"total_duration_us":10002722},{"operation_name":"upsert","last_dispatch_duration_us":9116824,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9249683,"total_server_duration_us":9090982,"encode_duration_us":23062,"operation_id":"0x14baf","timeout_ms":10000,"last_server_duration_us":9054355,"total_duration_us":9264468},{"operation_name":"get","last_dispatch_duration_us":9119826,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9119826,"total_server_duration_us":347361,"operation_id":"0x14c19","timeout_ms":30000,"last_server_duration_us":347361,"total_duration_us":9121952},{"operation_name":"upsert","last_dispatch_duration_us":9101975,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9101975,"total_server_duration_us":9030999,"encode_duration_us":35305,"operation_id":"0x14ba8","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":9107683},{"operation_name":"upsert","last_dispatch_duration_us":9096457,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9096457,"total_server_duration_us":9043735,"encode_duration_us":31132,"operation_id":"0x14ba4","timeout_ms":10000,"last_server_duration_us":9043735,"total_duration_us":9100791},{"operation_name":"get","last_dispatch_duration_us":8133171,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":8133171,"total_server_duration_us":49,"operation_id":"0x14fc9","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8135203},{"operation_name":"get","last_dispatch_duration_us":6942773,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":6942773,"total_server_duration_us":425321,"operation_id":"0x150c0","timeout_ms":30000,"last_server_duration_us":425321,"total_duration_us":6945478},{"operation_name":"get","last_dispatch_duration_us":5506198,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":5506198,"total_server_duration_us":76,"operation_id":"0x151a5","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":5509629}],"total_count":18}} INFO [2022-03-13 17:20:08,841] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,841] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,842] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,843] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,844] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,845] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,846] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,847] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:20:08,858] jepsen worker 31 - jepsen.util 5161 :fail :read [10 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,859] jepsen worker 31 - jepsen.util 5161 :invoke :write [10 3] INFO [2022-03-13 17:20:08,868] jepsen worker 53 - jepsen.util 593 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,868] jepsen worker 75 - jepsen.util 525 :fail :write [25 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,869] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:08,869] jepsen worker 75 - jepsen.util 525 :invoke :write [25 0] INFO [2022-03-13 17:20:08,882] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,882] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:20:08,888] jepsen worker 24 - jepsen.util 654 :fail :write [8 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,889] jepsen worker 24 - jepsen.util 654 :invoke :write [8 31] WARN [2022-03-13 17:20:08,909] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":10132417,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10132417,"total_server_duration_us":1209,"operation_id":"0x14bb7","timeout_ms":30000,"last_server_duration_us":1209,"total_duration_us":10132730},{"operation_name":"upsert","last_dispatch_duration_us":8323318,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":15214743,"total_server_duration_us":210531,"encode_duration_us":19912,"operation_id":"0x14c7c","timeout_ms":10000,"last_server_duration_us":21895,"total_duration_us":10009527},{"operation_name":"upsert","last_dispatch_duration_us":16220397,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":16637684,"total_server_duration_us":9292916,"encode_duration_us":23538,"operation_id":"0x14b7e","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":10008373},{"operation_name":"upsert","last_dispatch_duration_us":69959,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":358120,"total_server_duration_us":159394,"encode_duration_us":24550,"operation_id":"0x15033","timeout_ms":10000,"last_server_duration_us":25450,"total_duration_us":10007172},{"operation_name":"upsert","last_dispatch_duration_us":7981219,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8058141,"total_server_duration_us":64880,"encode_duration_us":23225,"operation_id":"0x14ee0","timeout_ms":10000,"last_server_duration_us":25015,"total_duration_us":10005562},{"operation_name":"upsert","last_dispatch_duration_us":35309,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9292662,"total_server_duration_us":257580,"encode_duration_us":26853,"operation_id":"0x14d07","timeout_ms":10000,"last_server_duration_us":21324,"total_duration_us":10003821},{"operation_name":"get","last_dispatch_duration_us":9589197,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9589197,"total_server_duration_us":7856,"operation_id":"0x14c24","timeout_ms":30000,"last_server_duration_us":7856,"total_duration_us":9591552},{"operation_name":"upsert","last_dispatch_duration_us":1696585,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1696585,"total_server_duration_us":106305,"encode_duration_us":26969,"operation_id":"0x151c0","timeout_ms":10000,"last_server_duration_us":106305,"total_duration_us":1706469},{"operation_name":"get","last_dispatch_duration_us":1577139,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1577139,"total_server_duration_us":43,"operation_id":"0x151df","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":1577330},{"operation_name":"upsert","last_dispatch_duration_us":1448568,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":1448568,"total_server_duration_us":81684,"encode_duration_us":18947,"operation_id":"0x15217","timeout_ms":10000,"last_server_duration_us":81684,"total_duration_us":1453910}],"total_count":58}} WARN [2022-03-13 17:20:08,910] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18179297,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18179297,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9036305,"encode_duration_us":25177,"operation_id":"0x14b9a","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10003900},{"operation_name":"upsert","last_dispatch_duration_us":9055931,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":13117352,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9455097,"encode_duration_us":30816,"operation_id":"0x14b14","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10004400},{"operation_name":"upsert","last_dispatch_duration_us":16613973,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16613973,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9030999,"encode_duration_us":17706,"operation_id":"0x147be","timeout_ms":10000,"last_server_duration_us":9030999,"total_duration_us":10004046}],"total_count":3}} WARN [2022-03-13 17:20:08,915] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15885024,"last_local_id":"A312307A00000003/00000000340BCE27","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":15885024,"last_local_socket":"172.23.104.79:43002","total_server_duration_us":9049045,"encode_duration_us":21758,"operation_id":"0x14d08","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10008486}],"total_count":1}} WARN [2022-03-13 17:20:08,920] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","encode_duration_us":18399,"operation_id":"0x14d58","timeout_ms":10000,"total_duration_us":10009524},{"operation_name":"upsert","last_dispatch_duration_us":9051016,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9831857,"total_server_duration_us":9271324,"encode_duration_us":17043,"operation_id":"0x14b3e","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10008774},{"operation_name":"upsert","last_dispatch_duration_us":9119773,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":10151920,"total_server_duration_us":9629424,"encode_duration_us":32414,"operation_id":"0x14c04","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10008710},{"operation_name":"upsert","last_dispatch_duration_us":8621232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9984270,"total_server_duration_us":715608,"encode_duration_us":20744,"operation_id":"0x14c9d","timeout_ms":10000,"last_server_duration_us":22389,"total_duration_us":10008674},{"operation_name":"upsert","last_dispatch_duration_us":9110297,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9110297,"total_server_duration_us":9038428,"encode_duration_us":33314,"operation_id":"0x150d1","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":9115126},{"operation_name":"get","last_dispatch_duration_us":8145774,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8145774,"total_server_duration_us":22141,"operation_id":"0x151cf","timeout_ms":30000,"last_server_duration_us":22141,"total_duration_us":8146191},{"operation_name":"upsert","last_dispatch_duration_us":1018488,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":1018488,"total_server_duration_us":42634,"encode_duration_us":23915,"operation_id":"0x15226","timeout_ms":10000,"last_server_duration_us":42634,"total_duration_us":1022041},{"operation_name":"get","last_dispatch_duration_us":960729,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":960729,"total_server_duration_us":23,"operation_id":"0x15221","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":960983},{"operation_name":"get","last_dispatch_duration_us":960552,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":960552,"total_server_duration_us":19,"operation_id":"0x15222","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":960768},{"operation_name":"upsert","last_dispatch_duration_us":929844,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":929844,"total_server_duration_us":108399,"encode_duration_us":22360,"operation_id":"0x1555d","timeout_ms":10000,"last_server_duration_us":108399,"total_duration_us":940796}],"total_count":30}} INFO [2022-03-13 17:20:08,925] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,927] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:08,928] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,929] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:20:08,930] jepsen worker 60 - jepsen.util 690 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,930] jepsen worker 13 - jepsen.util 733 :fail :write [4 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,930] jepsen worker 60 - jepsen.util 690 :invoke :write [20 19] INFO [2022-03-13 17:20:08,931] jepsen worker 13 - jepsen.util 733 :invoke :write [4 45] INFO [2022-03-13 17:20:08,932] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,932] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,933] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 10] INFO [2022-03-13 17:20:08,934] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 5] INFO [2022-03-13 17:20:08,989] jepsen worker 88 - jepsen.util 2608 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,990] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 39] INFO [2022-03-13 17:20:08,997] jepsen worker 52 - jepsen.util 682 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,997] jepsen worker 76 - jepsen.util 796 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:08,998] jepsen worker 52 - jepsen.util 682 :invoke :write [17 21] INFO [2022-03-13 17:20:08,999] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:08,999] jepsen worker 70 - jepsen.util 880 :fail :write [23 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,000] jepsen worker 61 - jepsen.util 781 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,000] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:09,000] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,001] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,001] jepsen worker 61 - jepsen.util 781 :invoke :write [20 28] INFO [2022-03-13 17:20:09,002] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:09,002] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:09,003] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,003] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,004] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:09,005] jepsen worker 77 - jepsen.util 707 :invoke :write [25 43] INFO [2022-03-13 17:20:09,016] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,017] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,019] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,019] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,019] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,019] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,020] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] WARN [2022-03-13 17:20:09,020] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9119773,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10151920,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9629424,"encode_duration_us":32414,"operation_id":"0x14c04","timeout_ms":10000,"last_server_duration_us":9040551,"total_duration_us":10008710},{"operation_name":"upsert","last_dispatch_duration_us":9051016,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9831857,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9271324,"encode_duration_us":17043,"operation_id":"0x14b3e","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10008774}],"total_count":2}} INFO [2022-03-13 17:20:09,021] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,026] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,027] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:09,027] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 31] INFO [2022-03-13 17:20:09,028] jepsen worker 23 - jepsen.util 563 :invoke :write [7 43] INFO [2022-03-13 17:20:09,032] jepsen worker 33 - jepsen.util 753 :fail :write [11 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,032] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,034] jepsen worker 33 - jepsen.util 753 :invoke :write [11 30] INFO [2022-03-13 17:20:09,034] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,039] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,040] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:20:09,056] jepsen worker 68 - jepsen.util 4928 :ok :read [22 30] INFO [2022-03-13 17:20:09,056] jepsen worker 68 - jepsen.util 4928 :invoke :write [22 4] INFO [2022-03-13 17:20:09,064] jepsen worker 3 - jepsen.util 2253 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,066] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:20:09,066] jepsen worker 59 - jepsen.util 4829 :ok :read [19 8] INFO [2022-03-13 17:20:09,071] jepsen worker 59 - jepsen.util 4829 :invoke :read [19 nil] INFO [2022-03-13 17:20:09,089] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:09,089] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,089] jepsen worker 62 - jepsen.util 512 :fail :write [20 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,089] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,091] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:09,093] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:09,094] jepsen worker 12 - jepsen.util 912 :invoke :write [4 2] INFO [2022-03-13 17:20:09,095] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:09,104] jepsen worker 45 - jepsen.util 5085 :info :write [15 35] :RequestTimeoutException INFO [2022-03-13 17:20:09,105] jepsen worker 45 - jepsen.util 5175 :invoke :read [15 nil] INFO [2022-03-13 17:20:09,106] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,107] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] WARN [2022-03-13 17:20:09,110] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":15891276,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":15891276,"total_server_duration_us":86035,"operation_id":"0x147ed","timeout_ms":30000,"last_server_duration_us":86035,"total_duration_us":15893646},{"operation_name":"upsert","last_dispatch_duration_us":7412746,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9375041,"total_server_duration_us":1488424,"encode_duration_us":20730,"operation_id":"0x14c82","timeout_ms":10000,"last_server_duration_us":19496,"total_duration_us":10011158},{"operation_name":"upsert","encode_duration_us":17344,"operation_id":"0x14e21","timeout_ms":10000,"total_duration_us":10009578},{"operation_name":"upsert","last_dispatch_duration_us":33490,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8451632,"total_server_duration_us":226042,"encode_duration_us":16809,"operation_id":"0x15086","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10009032},{"operation_name":"upsert","last_dispatch_duration_us":142620,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9512095,"total_server_duration_us":82906,"encode_duration_us":22351,"operation_id":"0x14b9b","timeout_ms":10000,"last_server_duration_us":21163,"total_duration_us":10008879},{"operation_name":"upsert","last_dispatch_duration_us":9055931,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":13117352,"total_server_duration_us":9455097,"encode_duration_us":30816,"operation_id":"0x14b14","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10004400},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","encode_duration_us":18110,"operation_id":"0x14bd3","timeout_ms":10000,"total_duration_us":10003937},{"operation_name":"upsert","last_dispatch_duration_us":18179297,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":18179297,"total_server_duration_us":9036305,"encode_duration_us":25177,"operation_id":"0x14b9a","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":10003900},{"operation_name":"upsert","last_dispatch_duration_us":208592,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9651487,"total_server_duration_us":75504,"encode_duration_us":32791,"operation_id":"0x14b97","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10002570},{"operation_name":"upsert","last_dispatch_duration_us":8629927,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":18064888,"total_server_duration_us":80173,"encode_duration_us":45718,"operation_id":"0x14b9c","timeout_ms":10000,"last_server_duration_us":25537,"total_duration_us":10002076}],"total_count":92}} INFO [2022-03-13 17:20:09,121] jepsen worker 59 - jepsen.util 4829 :ok :read [19 8] INFO [2022-03-13 17:20:09,122] jepsen worker 59 - jepsen.util 4829 :invoke :write [19 5] INFO [2022-03-13 17:20:09,134] jepsen worker 80 - jepsen.util 800 :fail :write [26 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,135] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:09,181] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,182] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,183] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,184] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:20:09,185] jepsen worker 88 - jepsen.util 2608 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,185] jepsen worker 36 - jepsen.util 2466 :fail :write [12 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,186] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,187] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 29] INFO [2022-03-13 17:20:09,192] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,193] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,199] jepsen worker 54 - jepsen.util 4554 :info :write [18 37] :SyncWriteAmbiguous INFO [2022-03-13 17:20:09,200] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,200] jepsen worker 54 - jepsen.util 4644 :invoke :read [18 nil] INFO [2022-03-13 17:20:09,201] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,203] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,204] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,241] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,241] jepsen worker 4 - jepsen.util 2614 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,241] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:09,242] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,247] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,247] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,249] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,249] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,252] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,252] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 29] INFO [2022-03-13 17:20:09,263] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,263] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:09,264] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,265] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:09,266] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,266] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:09,268] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,268] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:20:09,322] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,323] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:20:09,327] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,328] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 4] INFO [2022-03-13 17:20:09,348] jepsen worker 26 - jepsen.util 566 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,348] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,348] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:09,348] jepsen worker 75 - jepsen.util 525 :fail :write [25 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,348] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,349] jepsen worker 26 - jepsen.util 566 :invoke :write [8 13] INFO [2022-03-13 17:20:09,351] jepsen worker 53 - jepsen.util 593 :invoke :write [17 17] INFO [2022-03-13 17:20:09,351] jepsen worker 22 - jepsen.util 652 :invoke :write [7 14] INFO [2022-03-13 17:20:09,352] jepsen worker 43 - jepsen.util 673 :invoke :write [14 18] INFO [2022-03-13 17:20:09,358] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,388] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,389] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:20:09,393] jepsen worker 44 - jepsen.util 764 :fail :write [14 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,394] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:09,396] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:09,396] jepsen worker nemesis - couchbase.util Rebalance check count: 715 INFO [2022-03-13 17:20:09,396] jepsen worker nemesis - couchbase.util Rebalance stuck count: 587 INFO [2022-03-13 17:20:09,440] jepsen worker 36 - jepsen.util 2466 :fail :write [12 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,440] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,441] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,441] jepsen worker 51 - jepsen.util 771 :fail :write [17 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,441] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,441] jepsen worker 61 - jepsen.util 781 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,442] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:20:09,443] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:09,444] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 47] INFO [2022-03-13 17:20:09,445] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,446] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:09,446] jepsen worker 61 - jepsen.util 781 :invoke :write [20 22] INFO [2022-03-13 17:20:09,446] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,446] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:09,449] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,450] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 46] INFO [2022-03-13 17:20:09,470] jepsen worker 60 - jepsen.util 690 :fail :write [20 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,470] jepsen worker 13 - jepsen.util 733 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,470] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,470] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,471] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,472] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:09,472] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,473] jepsen worker 60 - jepsen.util 690 :invoke :write [20 35] INFO [2022-03-13 17:20:09,473] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,473] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,474] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,474] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,475] jepsen worker 13 - jepsen.util 733 :invoke :write [4 25] INFO [2022-03-13 17:20:09,475] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,484] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,484] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,485] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,486] jepsen worker 76 - jepsen.util 796 :invoke :write [25 23] INFO [2022-03-13 17:20:09,486] jepsen worker 84 - jepsen.util 5304 :info :write [28 22] :RequestTimeoutException INFO [2022-03-13 17:20:09,488] jepsen worker 84 - jepsen.util 5394 :invoke :write [28 45] INFO [2022-03-13 17:20:09,490] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,491] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,503] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,503] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,505] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,506] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:09,507] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,508] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:20:09,519] jepsen worker 88 - jepsen.util 2608 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,520] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,528] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,529] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 1] INFO [2022-03-13 17:20:09,537] jepsen worker 7 - jepsen.util 637 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,538] jepsen worker 7 - jepsen.util 637 :invoke :write [2 17] INFO [2022-03-13 17:20:09,544] jepsen worker 34 - jepsen.util 934 :fail :write [11 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,544] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,545] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:09,546] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:09,553] jepsen worker 52 - jepsen.util 682 :fail :write [17 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,554] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:09,572] jepsen worker 87 - jepsen.util 2877 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,573] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 32] INFO [2022-03-13 17:20:09,611] jepsen worker 77 - jepsen.util 707 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,612] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,632] jepsen worker 14 - jepsen.util 374 :fail :write [4 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,633] jepsen worker 14 - jepsen.util 374 :invoke :write [4 48] INFO [2022-03-13 17:20:09,635] jepsen worker 13 - jepsen.util 733 :fail :write [4 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,635] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,636] jepsen worker 75 - jepsen.util 525 :invoke :write [25 9] INFO [2022-03-13 17:20:09,636] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:09,637] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,637] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 23] INFO [2022-03-13 17:20:09,639] jepsen worker 3 - jepsen.util 2253 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,639] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:09,649] jepsen worker 9 - jepsen.util 4419 :fail :read [3 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,650] jepsen worker 9 - jepsen.util 4419 :invoke :read [3 nil] INFO [2022-03-13 17:20:09,657] jepsen worker 65 - jepsen.util 5195 :ok :read [21 :nil] INFO [2022-03-13 17:20:09,658] jepsen worker 65 - jepsen.util 5195 :invoke :write [21 4] INFO [2022-03-13 17:20:09,698] jepsen worker 87 - jepsen.util 2877 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,699] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:20:09,702] jepsen worker 54 - jepsen.util 4644 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,703] jepsen worker 54 - jepsen.util 4644 :invoke :read [18 nil] INFO [2022-03-13 17:20:09,720] jepsen worker 43 - jepsen.util 673 :fail :write [14 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,720] jepsen worker 43 - jepsen.util 673 :invoke :write [14 23] INFO [2022-03-13 17:20:09,725] jepsen worker 24 - jepsen.util 654 :fail :write [8 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,726] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:09,727] jepsen worker 8 - jepsen.util 728 :fail :write [2 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,727] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,727] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:09,727] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,728] jepsen worker 42 - jepsen.util 402 :invoke :write [14 16] INFO [2022-03-13 17:20:09,734] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,734] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:09,735] jepsen worker 8 - jepsen.util 728 :invoke :write [2 37] INFO [2022-03-13 17:20:09,735] jepsen worker 6 - jepsen.util 636 :invoke :write [2 47] INFO [2022-03-13 17:20:09,736] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:09,745] jepsen worker 5 - jepsen.util 2705 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,746] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:20:09,758] jepsen worker 88 - jepsen.util 2608 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,758] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,804] jepsen worker 67 - jepsen.util 4297 :info :write [22 8] :RequestTimeoutException INFO [2022-03-13 17:20:09,804] jepsen worker 67 - jepsen.util 4387 :invoke :read [22 nil] INFO [2022-03-13 17:20:09,807] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,808] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 48] INFO [2022-03-13 17:20:09,810] jepsen worker 76 - jepsen.util 796 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,811] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,811] jepsen worker 76 - jepsen.util 796 :invoke :write [25 10] INFO [2022-03-13 17:20:09,811] jepsen worker 13 - jepsen.util 733 :invoke :write [4 34] INFO [2022-03-13 17:20:09,812] jepsen worker 23 - jepsen.util 563 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,813] jepsen worker 23 - jepsen.util 563 :invoke :write [7 31] INFO [2022-03-13 17:20:09,813] jepsen worker 36 - jepsen.util 2466 :fail :write [12 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,814] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,815] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,816] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:09,816] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,817] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:09,819] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,819] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 25] INFO [2022-03-13 17:20:09,820] jepsen worker 9 - jepsen.util 4419 :ok :read [3 25] INFO [2022-03-13 17:20:09,823] jepsen worker 9 - jepsen.util 4419 :invoke :write [3 32] INFO [2022-03-13 17:20:09,853] jepsen worker 33 - jepsen.util 753 :fail :write [11 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,853] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,853] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,854] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,854] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,854] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,855] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:09,855] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:09,856] jepsen worker 34 - jepsen.util 934 :invoke :write [11 29] INFO [2022-03-13 17:20:09,858] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,859] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:09,860] jepsen worker 35 - jepsen.util 935 :invoke :write [11 46] INFO [2022-03-13 17:20:09,861] jepsen worker 33 - jepsen.util 753 :invoke :write [11 42] INFO [2022-03-13 17:20:09,862] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 42] INFO [2022-03-13 17:20:09,863] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,863] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,864] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:09,864] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 31] INFO [2022-03-13 17:20:09,867] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,868] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 9] INFO [2022-03-13 17:20:09,878] jepsen worker 26 - jepsen.util 566 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,878] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:09,890] jepsen worker 67 - jepsen.util 4387 :ok :read [22 30] INFO [2022-03-13 17:20:09,891] jepsen worker 67 - jepsen.util 4387 :invoke :write [22 30] INFO [2022-03-13 17:20:09,894] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,895] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:09,915] jepsen worker 88 - jepsen.util 2608 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,916] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 16] INFO [2022-03-13 17:20:09,959] jepsen worker 21 - jepsen.util 831 :fail :write [7 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,960] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:09,965] jepsen worker 75 - jepsen.util 525 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,966] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:09,976] jepsen worker 87 - jepsen.util 2877 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,977] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 9] INFO [2022-03-13 17:20:09,995] jepsen worker 4 - jepsen.util 2614 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:09,996] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 23] INFO [2022-03-13 17:20:09,996] jepsen worker 28 - jepsen.util 4978 :info :write [9 9] :RequestTimeoutException INFO [2022-03-13 17:20:09,998] jepsen worker 28 - jepsen.util 5068 :invoke :write [9 38] INFO [2022-03-13 17:20:10,012] jepsen worker 22 - jepsen.util 652 :fail :write [7 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,015] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:10,049] jepsen worker 12 - jepsen.util 912 :fail :write [4 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,050] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,049] jepsen worker 79 - jepsen.util 1159 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,049] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:10,051] jepsen worker 12 - jepsen.util 912 :invoke :write [4 15] INFO [2022-03-13 17:20:10,051] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,052] jepsen worker 44 - jepsen.util 764 :invoke :write [14 17] INFO [2022-03-13 17:20:10,054] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:10,054] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,055] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:10,060] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,061] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,066] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,067] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:20:10,082] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,082] jepsen worker 14 - jepsen.util 374 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,083] jepsen worker 14 - jepsen.util 374 :invoke :write [4 10] INFO [2022-03-13 17:20:10,083] jepsen worker 24 - jepsen.util 654 :invoke :write [8 29] INFO [2022-03-13 17:20:10,085] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,085] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,085] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,086] jepsen worker 25 - jepsen.util 565 :invoke :write [8 12] INFO [2022-03-13 17:20:10,086] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:10,087] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:10,104] jepsen worker 3 - jepsen.util 2253 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,105] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,136] jepsen worker 87 - jepsen.util 2877 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,136] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 43] INFO [2022-03-13 17:20:10,137] jepsen worker 76 - jepsen.util 796 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,138] jepsen worker 76 - jepsen.util 796 :invoke :write [25 28] INFO [2022-03-13 17:20:10,163] jepsen worker 7 - jepsen.util 637 :fail :write [2 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,163] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,163] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,163] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:10,164] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:10,164] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:10,171] jepsen worker 88 - jepsen.util 2608 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,172] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 48] INFO [2022-03-13 17:20:10,186] jepsen worker 35 - jepsen.util 935 :fail :write [11 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,186] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:10,211] jepsen worker 53 - jepsen.util 593 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,212] jepsen worker 53 - jepsen.util 593 :invoke :write [17 0] INFO [2022-03-13 17:20:10,222] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,237] jepsen worker 61 - jepsen.util 781 :fail :write [20 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,237] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,488] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:20:10,246] jepsen worker 36 - jepsen.util 2466 :fail :write [12 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,488] jepsen worker 61 - jepsen.util 781 :invoke :write [20 33] INFO [2022-03-13 17:20:10,259] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,488] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:10,488] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:10,259] jepsen worker 71 - jepsen.util 611 :fail :write [23 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,274] jepsen worker 87 - jepsen.util 2877 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,489] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 34] INFO [2022-03-13 17:20:10,490] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,492] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 21] INFO [2022-03-13 17:20:10,279] jepsen worker 33 - jepsen.util 753 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,321] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,321] jepsen worker 4 - jepsen.util 2614 :fail :write [1 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,492] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:10,492] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,492] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:20:10,329] jepsen worker 13 - jepsen.util 733 :fail :write [4 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,329] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,493] jepsen worker 13 - jepsen.util 733 :invoke :write [4 42] INFO [2022-03-13 17:20:10,493] jepsen worker 75 - jepsen.util 525 :invoke :write [25 15] INFO [2022-03-13 17:20:10,358] jepsen worker 42 - jepsen.util 402 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,496] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:10,498] jepsen worker 14 - jepsen.util 374 :fail :write [4 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,371] jepsen worker 88 - jepsen.util 2608 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,397] jepsen worker 30 - jepsen.util 4800 :info :write [10 8] :RequestTimeoutException INFO [2022-03-13 17:20:10,408] jepsen worker 34 - jepsen.util 934 :fail :write [11 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,408] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,499] jepsen worker 30 - jepsen.util 4890 :invoke :write [10 32] INFO [2022-03-13 17:20:10,499] jepsen worker 80 - jepsen.util 800 :invoke :write [26 6] INFO [2022-03-13 17:20:10,500] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,502] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,503] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,508] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,509] jepsen worker 14 - jepsen.util 374 :invoke :write [4 17] INFO [2022-03-13 17:20:10,408] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,510] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:10,409] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,411] jepsen worker 60 - jepsen.util 690 :fail :write [20 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,510] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:10,510] jepsen worker 60 - jepsen.util 690 :invoke :write [20 1] INFO [2022-03-13 17:20:10,512] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 41] INFO [2022-03-13 17:20:10,412] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,413] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:10,423] jepsen worker 27 - jepsen.util 5067 :info :write [9 29] :RequestTimeoutException INFO [2022-03-13 17:20:10,513] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:10,438] jepsen worker 56 - jepsen.util 4646 :info :write [18 29] :RequestTimeoutException INFO [2022-03-13 17:20:10,442] jepsen worker 23 - jepsen.util 563 :fail :write [7 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,514] jepsen worker 27 - jepsen.util 5157 :invoke :read [9 nil] INFO [2022-03-13 17:20:10,514] jepsen worker 56 - jepsen.util 4736 :invoke :read [18 nil] INFO [2022-03-13 17:20:10,514] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:20:10,442] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,514] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:10,442] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,488] jepsen worker 77 - jepsen.util 707 :invoke :write [25 33] INFO [2022-03-13 17:20:10,514] jepsen worker 78 - jepsen.util 708 :invoke :write [26 42] INFO [2022-03-13 17:20:10,489] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:10,499] jepsen worker 34 - jepsen.util 934 :invoke :write [11 15] INFO [2022-03-13 17:20:10,514] jepsen worker nemesis - couchbase.util Rebalance check count: 716 INFO [2022-03-13 17:20:10,520] jepsen worker nemesis - couchbase.util Rebalance stuck count: 588 INFO [2022-03-13 17:20:10,520] jepsen worker 8 - jepsen.util 728 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,522] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,523] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,526] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,528] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,528] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:10,529] jepsen worker 69 - jepsen.util 789 :invoke :write [23 7] INFO [2022-03-13 17:20:10,529] jepsen worker 8 - jepsen.util 728 :invoke :write [2 37] INFO [2022-03-13 17:20:10,542] jepsen worker 76 - jepsen.util 796 :fail :write [25 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,543] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:10,554] jepsen worker 87 - jepsen.util 2877 :fail :write [29 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,555] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:20:10,558] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,563] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,580] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,581] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,604] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,604] jepsen worker 24 - jepsen.util 654 :fail :write [8 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,605] jepsen worker 35 - jepsen.util 935 :invoke :write [11 26] INFO [2022-03-13 17:20:10,605] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:10,626] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,627] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,627] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,627] jepsen worker 43 - jepsen.util 673 :fail :write [14 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,628] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 36] INFO [2022-03-13 17:20:10,630] jepsen worker 43 - jepsen.util 673 :invoke :write [14 19] INFO [2022-03-13 17:20:10,631] jepsen worker 26 - jepsen.util 566 :invoke :write [8 38] INFO [2022-03-13 17:20:10,631] jepsen worker 22 - jepsen.util 652 :invoke :write [7 36] INFO [2022-03-13 17:20:10,646] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,646] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,647] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:20:10,648] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,649] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,650] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,651] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,652] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,654] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,655] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,656] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,657] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,663] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,663] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,665] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,667] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 11] INFO [2022-03-13 17:20:10,689] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,690] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,697] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,698] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,699] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,701] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,702] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,703] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,704] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,705] jepsen worker 75 - jepsen.util 525 :fail :write [25 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,705] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,705] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,706] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:10,707] jepsen worker 76 - jepsen.util 796 :invoke :write [25 43] INFO [2022-03-13 17:20:10,721] jepsen worker 36 - jepsen.util 2466 :fail :write [12 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,722] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:10,731] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,732] jepsen worker 71 - jepsen.util 611 :invoke :write [23 29] INFO [2022-03-13 17:20:10,763] jepsen worker 5 - jepsen.util 2705 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,763] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,764] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 1] INFO [2022-03-13 17:20:10,765] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] INFO [2022-03-13 17:20:10,771] jepsen worker 25 - jepsen.util 565 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,771] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,771] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,772] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,772] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:10,773] jepsen worker 33 - jepsen.util 753 :invoke :write [11 2] INFO [2022-03-13 17:20:10,774] jepsen worker 51 - jepsen.util 771 :invoke :write [17 46] INFO [2022-03-13 17:20:10,775] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:10,776] jepsen worker 53 - jepsen.util 593 :fail :write [17 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,777] jepsen worker 53 - jepsen.util 593 :invoke :write [17 30] INFO [2022-03-13 17:20:10,779] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,780] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:10,798] jepsen worker 61 - jepsen.util 781 :fail :write [20 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,798] jepsen worker 6 - jepsen.util 636 :fail :write [2 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,798] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,800] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 8] INFO [2022-03-13 17:20:10,806] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:10,806] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:10,821] jepsen worker 3 - jepsen.util 2253 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,822] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,873] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,873] jepsen worker 14 - jepsen.util 374 :fail :write [4 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,873] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,874] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:10,876] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,877] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:20:10,884] jepsen worker 88 - jepsen.util 2608 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,885] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,886] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,887] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,888] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,888] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 41] INFO [2022-03-13 17:20:10,904] jepsen worker 13 - jepsen.util 733 :fail :write [4 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,905] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,905] jepsen worker 13 - jepsen.util 733 :invoke :write [4 5] INFO [2022-03-13 17:20:10,906] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:10,912] jepsen worker 5 - jepsen.util 2705 :fail :write [1 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,913] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:20:10,952] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,952] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,952] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,953] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,954] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,954] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:10,956] jepsen worker 60 - jepsen.util 690 :fail :write [20 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,956] jepsen worker 60 - jepsen.util 690 :invoke :write [20 36] INFO [2022-03-13 17:20:10,958] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,958] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,958] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,958] jepsen worker 29 - jepsen.util 5159 :info :write [9 17] :RequestTimeoutException INFO [2022-03-13 17:20:10,959] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:10,959] jepsen worker 29 - jepsen.util 5249 :invoke :read [9 nil] INFO [2022-03-13 17:20:10,960] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,960] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 11] INFO [2022-03-13 17:20:10,961] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:10,961] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 43] INFO [2022-03-13 17:20:10,966] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,967] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:10,968] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,969] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:10,970] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,971] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,972] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 40] INFO [2022-03-13 17:20:10,972] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:10,976] jepsen worker 34 - jepsen.util 934 :fail :write [11 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,977] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,977] jepsen worker 34 - jepsen.util 934 :invoke :write [11 34] INFO [2022-03-13 17:20:10,978] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:10,981] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,981] jepsen worker 12 - jepsen.util 912 :fail :write [4 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:10,982] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:10,983] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:11,032] jepsen worker 76 - jepsen.util 796 :fail :write [25 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,032] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,032] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:11,033] jepsen worker 75 - jepsen.util 525 :invoke :write [25 39] INFO [2022-03-13 17:20:11,034] jepsen worker 88 - jepsen.util 2608 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,035] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,035] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 47] INFO [2022-03-13 17:20:11,035] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:20:11,037] jepsen worker 80 - jepsen.util 800 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,037] jepsen worker 80 - jepsen.util 800 :invoke :write [26 4] INFO [2022-03-13 17:20:11,039] jepsen worker 29 - jepsen.util 5249 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,040] jepsen worker 29 - jepsen.util 5249 :invoke :read [9 nil] INFO [2022-03-13 17:20:11,043] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,043] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,044] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,045] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,046] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,046] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,047] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,048] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,049] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,050] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:20:11,078] jepsen worker 3 - jepsen.util 2253 :fail :write [1 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,079] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,083] jepsen worker 55 - jepsen.util 4465 :info :write [18 26] :RequestTimeoutException INFO [2022-03-13 17:20:11,083] jepsen worker 37 - jepsen.util 2467 :fail :read [12 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:11,084] jepsen worker 37 - jepsen.util 2467 :invoke :write [12 4] INFO [2022-03-13 17:20:11,085] jepsen worker 29 - jepsen.util 5249 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,086] jepsen worker 29 - jepsen.util 5249 :invoke :write [9 5] INFO [2022-03-13 17:20:11,087] jepsen worker 35 - jepsen.util 935 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,087] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,088] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,089] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:11,089] jepsen worker 25 - jepsen.util 565 :invoke :write [8 38] INFO [2022-03-13 17:20:11,090] jepsen worker 55 - jepsen.util 4555 :invoke :read [18 nil] INFO [2022-03-13 17:20:11,091] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:11,113] jepsen worker 22 - jepsen.util 652 :fail :write [7 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,114] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:20:11,119] jepsen worker 4 - jepsen.util 2614 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,119] jepsen worker 51 - jepsen.util 771 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,120] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 44] INFO [2022-03-13 17:20:11,120] jepsen worker 51 - jepsen.util 771 :invoke :write [17 15] INFO [2022-03-13 17:20:11,127] jepsen worker 87 - jepsen.util 2877 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,127] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:11,141] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,141] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 6] INFO [2022-03-13 17:20:11,148] jepsen worker 88 - jepsen.util 2608 :fail :write [29 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,148] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 2] INFO [2022-03-13 17:20:11,188] jepsen worker 33 - jepsen.util 753 :fail :write [11 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,188] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,189] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,189] jepsen worker 14 - jepsen.util 374 :invoke :write [4 45] INFO [2022-03-13 17:20:11,190] jepsen worker 24 - jepsen.util 654 :invoke :write [8 8] INFO [2022-03-13 17:20:11,193] jepsen worker 33 - jepsen.util 753 :invoke :write [11 26] INFO [2022-03-13 17:20:11,209] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,210] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,211] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,212] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 2] INFO [2022-03-13 17:20:11,220] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,220] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,221] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 1] INFO [2022-03-13 17:20:11,222] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,228] jepsen worker 44 - jepsen.util 764 :fail :write [14 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,228] jepsen worker 77 - jepsen.util 707 :fail :write [25 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,228] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,229] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,229] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,229] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:11,230] jepsen worker 7 - jepsen.util 637 :invoke :write [2 41] INFO [2022-03-13 17:20:11,230] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:11,231] jepsen worker 70 - jepsen.util 880 :invoke :write [23 12] INFO [2022-03-13 17:20:11,231] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:11,232] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:11,232] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:11,234] jepsen worker 13 - jepsen.util 733 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,234] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,234] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:11,235] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:11,279] jepsen worker 87 - jepsen.util 2877 :fail :write [29 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,279] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 46] INFO [2022-03-13 17:20:11,309] jepsen worker 34 - jepsen.util 934 :fail :write [11 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,310] jepsen worker 34 - jepsen.util 934 :invoke :write [11 38] INFO [2022-03-13 17:20:11,323] jepsen worker 43 - jepsen.util 673 :fail :write [14 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,325] jepsen worker 43 - jepsen.util 673 :invoke :write [14 34] INFO [2022-03-13 17:20:11,327] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,327] jepsen worker 4 - jepsen.util 2614 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,329] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:20:11,329] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 33] INFO [2022-03-13 17:20:11,370] jepsen worker 39 - jepsen.util 5349 :info :write [13 33] :RequestTimeoutException INFO [2022-03-13 17:20:11,371] jepsen worker 39 - jepsen.util 5439 :invoke :read [13 nil] INFO [2022-03-13 17:20:11,382] jepsen worker 5 - jepsen.util 2705 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,383] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:20:11,402] jepsen worker 88 - jepsen.util 2608 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,403] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 45] INFO [2022-03-13 17:20:11,473] jepsen worker 87 - jepsen.util 2877 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,473] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:11,477] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,477] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:11,478] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,479] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:20:11,481] jepsen worker 80 - jepsen.util 800 :fail :write [26 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,481] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,481] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,482] jepsen worker 80 - jepsen.util 800 :invoke :write [26 46] INFO [2022-03-13 17:20:11,482] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:11,486] jepsen worker 21 - jepsen.util 831 :invoke :write [7 13] INFO [2022-03-13 17:20:11,487] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,487] jepsen worker 78 - jepsen.util 708 :fail :write [26 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,492] jepsen worker 61 - jepsen.util 781 :invoke :write [20 46] INFO [2022-03-13 17:20:11,493] jepsen worker 78 - jepsen.util 708 :invoke :write [26 21] INFO [2022-03-13 17:20:11,498] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,498] jepsen worker 75 - jepsen.util 525 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,499] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,499] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,499] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,498] jepsen worker 8 - jepsen.util 728 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,499] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:11,499] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,499] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:11,499] jepsen worker 8 - jepsen.util 728 :invoke :write [2 39] INFO [2022-03-13 17:20:11,500] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:11,501] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:11,501] jepsen worker 12 - jepsen.util 912 :invoke :write [4 14] INFO [2022-03-13 17:20:11,502] jepsen worker 13 - jepsen.util 733 :invoke :write [4 20] INFO [2022-03-13 17:20:11,503] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:11,504] jepsen worker 36 - jepsen.util 2466 :fail :write [12 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,504] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:11,504] jepsen worker 75 - jepsen.util 525 :invoke :write [25 25] INFO [2022-03-13 17:20:11,506] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:11,510] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:11,526] jepsen worker 63 - jepsen.util 5103 :info :write [21 38] :RequestTimeoutException INFO [2022-03-13 17:20:11,526] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:11,526] jepsen worker nemesis - couchbase.util Rebalance check count: 717 INFO [2022-03-13 17:20:11,527] jepsen worker nemesis - couchbase.util Rebalance stuck count: 589 INFO [2022-03-13 17:20:11,527] jepsen worker 63 - jepsen.util 5193 :invoke :read [21 nil] INFO [2022-03-13 17:20:11,573] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,573] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,576] jepsen worker 26 - jepsen.util 566 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,576] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,576] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,576] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:11,577] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:11,578] jepsen worker 76 - jepsen.util 796 :invoke :write [25 14] INFO [2022-03-13 17:20:11,579] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,579] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:20:11,585] jepsen worker 25 - jepsen.util 565 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,586] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:11,599] jepsen worker 88 - jepsen.util 2608 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,600] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:11,652] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,653] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:20:11,656] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,657] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,657] jepsen worker 51 - jepsen.util 771 :fail :write [17 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,657] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:20:11,658] jepsen worker 51 - jepsen.util 771 :invoke :write [17 27] INFO [2022-03-13 17:20:11,659] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 46] INFO [2022-03-13 17:20:11,684] jepsen worker 70 - jepsen.util 880 :fail :write [23 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,684] jepsen worker 79 - jepsen.util 1159 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,684] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,684] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,685] jepsen worker 70 - jepsen.util 880 :invoke :write [23 30] INFO [2022-03-13 17:20:11,685] jepsen worker 23 - jepsen.util 563 :invoke :write [7 13] INFO [2022-03-13 17:20:11,686] jepsen worker 77 - jepsen.util 707 :invoke :write [25 24] INFO [2022-03-13 17:20:11,688] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 19] INFO [2022-03-13 17:20:11,695] jepsen worker 53 - jepsen.util 593 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,695] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:11,721] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,722] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,724] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,724] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,730] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,731] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,750] jepsen worker 75 - jepsen.util 525 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,751] jepsen worker 75 - jepsen.util 525 :invoke :write [25 5] INFO [2022-03-13 17:20:11,767] jepsen worker 14 - jepsen.util 374 :fail :write [4 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,768] jepsen worker 14 - jepsen.util 374 :invoke :write [4 5] INFO [2022-03-13 17:20:11,808] jepsen worker 7 - jepsen.util 637 :fail :write [2 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,808] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:11,808] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,808] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,808] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:11,809] jepsen worker 7 - jepsen.util 637 :invoke :write [2 26] INFO [2022-03-13 17:20:11,810] jepsen worker 44 - jepsen.util 764 :invoke :write [14 3] INFO [2022-03-13 17:20:11,811] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:11,811] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:11,812] jepsen worker 6 - jepsen.util 636 :invoke :write [2 11] INFO [2022-03-13 17:20:11,830] jepsen worker 24 - jepsen.util 654 :fail :write [8 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,830] jepsen worker 24 - jepsen.util 654 :invoke :write [8 12] INFO [2022-03-13 17:20:11,843] jepsen worker 61 - jepsen.util 781 :fail :write [20 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,844] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:11,869] jepsen worker 60 - jepsen.util 690 :fail :write [20 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,870] jepsen worker 60 - jepsen.util 690 :invoke :write [20 24] INFO [2022-03-13 17:20:11,883] jepsen worker 33 - jepsen.util 753 :fail :write [11 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,883] jepsen worker 33 - jepsen.util 753 :invoke :write [11 48] INFO [2022-03-13 17:20:11,883] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,884] jepsen worker 35 - jepsen.util 935 :invoke :write [11 0] INFO [2022-03-13 17:20:11,925] jepsen worker 13 - jepsen.util 733 :fail :write [4 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,926] jepsen worker 13 - jepsen.util 733 :invoke :write [4 33] INFO [2022-03-13 17:20:11,931] jepsen worker 3 - jepsen.util 2253 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,931] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,931] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:20:11,932] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 18] INFO [2022-03-13 17:20:11,938] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,939] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,940] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,941] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:11,943] jepsen worker 12 - jepsen.util 912 :fail :write [4 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:11,943] jepsen worker 12 - jepsen.util 912 :invoke :write [4 9] INFO [2022-03-13 17:20:12,001] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,001] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,002] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,003] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,005] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,005] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,007] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,008] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:20:12,008] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,009] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:20:12,015] jepsen worker 78 - jepsen.util 708 :fail :write [26 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,016] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:12,028] jepsen worker 21 - jepsen.util 831 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,030] jepsen worker 88 - jepsen.util 2608 :fail :write [29 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,031] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,032] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 2] INFO [2022-03-13 17:20:12,034] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:12,035] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 32] INFO [2022-03-13 17:20:12,103] jepsen worker 36 - jepsen.util 2466 :fail :write [12 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,104] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:12,107] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,108] jepsen worker 22 - jepsen.util 652 :invoke :write [7 44] INFO [2022-03-13 17:20:12,120] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,121] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,126] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,127] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,128] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,128] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,129] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,130] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,131] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,132] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 39] INFO [2022-03-13 17:20:12,176] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,176] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:20:12,187] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,188] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,230] jepsen worker 5 - jepsen.util 2705 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,231] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:20:12,266] jepsen worker 88 - jepsen.util 2608 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,267] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 20] INFO [2022-03-13 17:20:12,324] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,325] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,334] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,334] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,337] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,338] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:20:12,348] jepsen worker 46 - jepsen.util 5176 :info :write [15 3] :RequestTimeoutException INFO [2022-03-13 17:20:12,348] jepsen worker 46 - jepsen.util 5266 :invoke :read [15 nil] INFO [2022-03-13 17:20:12,358] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,358] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,358] jepsen worker 4 - jepsen.util 2614 :fail :write [1 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,359] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 38] INFO [2022-03-13 17:20:12,360] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:20:12,366] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 12] INFO [2022-03-13 17:20:12,425] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,426] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:20:12,444] jepsen worker 88 - jepsen.util 2608 :fail :write [29 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,445] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,458] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,458] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,460] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,463] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,464] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,465] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 40] INFO [2022-03-13 17:20:12,517] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,518] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:20:12,534] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:12,534] jepsen worker nemesis - couchbase.util Rebalance check count: 718 INFO [2022-03-13 17:20:12,534] jepsen worker nemesis - couchbase.util Rebalance stuck count: 590 INFO [2022-03-13 17:20:12,539] jepsen worker 36 - jepsen.util 2466 :fail :write [12 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,540] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:12,601] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,601] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,602] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,603] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,604] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,604] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,612] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,613] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,614] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,614] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:20:12,656] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,656] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,658] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,658] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,664] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,665] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,666] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,666] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:20:12,709] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,710] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:12,710] jepsen worker 57 - jepsen.util 4917 :ok :read [19 8] INFO [2022-03-13 17:20:12,711] jepsen worker 57 - jepsen.util 4917 :invoke :write [19 36] INFO [2022-03-13 17:20:12,711] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,711] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 49] INFO [2022-03-13 17:20:12,714] jepsen worker 88 - jepsen.util 2608 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,715] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,718] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,719] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 45] INFO [2022-03-13 17:20:12,764] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,764] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,766] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,766] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 15] INFO [2022-03-13 17:20:12,779] jepsen worker 86 - jepsen.util 4676 :ok :read [28 44] INFO [2022-03-13 17:20:12,780] jepsen worker 86 - jepsen.util 4676 :invoke :write [28 39] INFO [2022-03-13 17:20:12,797] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,797] jepsen worker 4 - jepsen.util 2614 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,797] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,800] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:20:12,801] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 43] INFO [2022-03-13 17:20:12,801] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 12] INFO [2022-03-13 17:20:12,806] jepsen worker 40 - jepsen.util 5170 :info :write [13 11] :RequestTimeoutException INFO [2022-03-13 17:20:12,807] jepsen worker 40 - jepsen.util 5260 :invoke :read [13 nil] INFO [2022-03-13 17:20:12,830] jepsen worker 5 - jepsen.util 2705 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,830] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 37] INFO [2022-03-13 17:20:12,850] jepsen worker 16 - jepsen.util 4606 :info :write [5 5] :RequestTimeoutException INFO [2022-03-13 17:20:12,851] jepsen worker 16 - jepsen.util 4696 :invoke :read [5 nil] INFO [2022-03-13 17:20:12,870] jepsen worker 88 - jepsen.util 2608 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,875] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 48] INFO [2022-03-13 17:20:12,892] jepsen worker 87 - jepsen.util 2877 :fail :write [29 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,892] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:12,893] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,894] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 33] INFO [2022-03-13 17:20:12,913] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,913] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 27] INFO [2022-03-13 17:20:12,962] jepsen worker 5 - jepsen.util 2705 :fail :write [1 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,963] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 17] INFO [2022-03-13 17:20:12,977] jepsen worker 36 - jepsen.util 2466 :fail :write [12 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:12,977] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:13,051] jepsen worker 87 - jepsen.util 2877 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,052] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,053] jepsen worker 88 - jepsen.util 2608 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,054] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 42] INFO [2022-03-13 17:20:13,054] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,055] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:20:13,160] jepsen worker 3 - jepsen.util 2253 :fail :write [1 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,161] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 13] INFO [2022-03-13 17:20:13,175] jepsen worker 5 - jepsen.util 2705 :fail :write [1 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,175] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,181] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,182] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,192] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,192] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,195] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,195] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:20:13,211] jepsen worker 88 - jepsen.util 2608 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,211] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,214] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,214] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,283] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,283] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:20:13,300] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,300] jepsen worker 4 - jepsen.util 2614 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,301] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 46] INFO [2022-03-13 17:20:13,302] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,303] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,304] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,307] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,307] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 31] INFO [2022-03-13 17:20:13,370] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,371] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:20:13,409] jepsen worker 3 - jepsen.util 2253 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,409] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,409] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,410] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,462] jepsen worker 88 - jepsen.util 2608 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,462] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,465] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,466] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 40] INFO [2022-03-13 17:20:13,490] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,490] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 3] INFO [2022-03-13 17:20:13,500] jepsen worker 17 - jepsen.util 4877 :fail :read [5 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:13,501] jepsen worker 17 - jepsen.util 4877 :invoke :write [5 43] INFO [2022-03-13 17:20:13,525] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,526] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,527] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,528] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,529] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,529] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,531] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,531] jepsen worker 36 - jepsen.util 2466 :fail :write [12 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,531] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,531] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,532] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:13,532] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,532] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,533] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,534] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,534] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 4] INFO [2022-03-13 17:20:13,534] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,534] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,535] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 26] INFO [2022-03-13 17:20:13,535] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:13,536] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,536] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:20:13,539] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:13,539] jepsen worker nemesis - couchbase.util Rebalance check count: 719 INFO [2022-03-13 17:20:13,539] jepsen worker nemesis - couchbase.util Rebalance stuck count: 591 INFO [2022-03-13 17:20:13,540] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,540] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 41] INFO [2022-03-13 17:20:13,591] jepsen worker 87 - jepsen.util 2877 :fail :write [29 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,591] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,594] jepsen worker 88 - jepsen.util 2608 :fail :write [29 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,594] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,595] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,595] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:20:13,596] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,597] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 30] INFO [2022-03-13 17:20:13,660] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,661] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 24] INFO [2022-03-13 17:20:13,663] jepsen worker 4 - jepsen.util 2614 :fail :write [1 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,664] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 49] INFO [2022-03-13 17:20:13,720] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,721] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,729] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,730] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,731] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,732] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,735] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,735] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,736] jepsen worker 36 - jepsen.util 2466 :fail :write [12 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,736] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 42] INFO [2022-03-13 17:20:13,737] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,738] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,779] jepsen worker 88 - jepsen.util 2608 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,779] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,780] jepsen worker 5 - jepsen.util 2705 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,780] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:20:13,780] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,780] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,781] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 44] INFO [2022-03-13 17:20:13,782] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:13,783] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,783] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] INFO [2022-03-13 17:20:13,830] jepsen worker 3 - jepsen.util 2253 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,831] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:20:13,840] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,840] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,877] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,877] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,878] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 31] INFO [2022-03-13 17:20:13,878] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,880] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,881] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,883] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,883] jepsen worker 8 - jepsen.util 728 :fail :write [2 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,883] jepsen worker 80 - jepsen.util 800 :fail :write [26 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,884] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,885] jepsen worker 69 - jepsen.util 789 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,886] jepsen worker 8 - jepsen.util 728 :invoke :write [2 2] INFO [2022-03-13 17:20:13,886] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,886] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:13,887] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:20:13,888] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:13,893] jepsen worker 4 - jepsen.util 2614 :fail :write [1 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,893] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 16] INFO [2022-03-13 17:20:13,925] jepsen worker 88 - jepsen.util 2608 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,926] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 18] INFO [2022-03-13 17:20:13,929] jepsen worker 79 - jepsen.util 1159 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,930] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 6] INFO [2022-03-13 17:20:13,956] jepsen worker 36 - jepsen.util 2466 :fail :write [12 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,956] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:13,977] jepsen worker 76 - jepsen.util 796 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,978] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:13,995] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,996] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:13,997] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:13,998] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,023] jepsen worker 51 - jepsen.util 771 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,024] jepsen worker 51 - jepsen.util 771 :invoke :write [17 26] INFO [2022-03-13 17:20:14,028] jepsen worker 88 - jepsen.util 2608 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,029] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 9] INFO [2022-03-13 17:20:14,043] jepsen worker 87 - jepsen.util 2877 :fail :write [29 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,043] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,044] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,045] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:20:14,049] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,049] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,051] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,052] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,053] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,053] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,054] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,055] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:20:14,068] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,068] jepsen worker 4 - jepsen.util 2614 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,068] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,069] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:14,069] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 48] INFO [2022-03-13 17:20:14,070] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,070] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,071] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:14,071] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,072] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 34] INFO [2022-03-13 17:20:14,072] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,072] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 37] INFO [2022-03-13 17:20:14,084] jepsen worker 43 - jepsen.util 673 :fail :write [14 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,085] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:14,102] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,103] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,103] jepsen worker 14 - jepsen.util 374 :fail :write [4 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,103] jepsen worker 14 - jepsen.util 374 :invoke :write [4 21] INFO [2022-03-13 17:20:14,104] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,104] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,106] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,107] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:20:14,122] jepsen worker 88 - jepsen.util 2608 :fail :write [29 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,122] jepsen worker 70 - jepsen.util 880 :fail :write [23 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,122] jepsen worker 23 - jepsen.util 563 :fail :write [7 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,122] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,122] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:14,125] jepsen worker 70 - jepsen.util 880 :invoke :write [23 38] INFO [2022-03-13 17:20:14,127] jepsen worker 23 - jepsen.util 563 :invoke :write [7 23] INFO [2022-03-13 17:20:14,127] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 37] INFO [2022-03-13 17:20:14,128] jepsen worker 62 - jepsen.util 512 :invoke :write [20 26] INFO [2022-03-13 17:20:14,135] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:14,138] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,138] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,139] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,139] jepsen worker 75 - jepsen.util 525 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,141] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:14,142] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:14,142] jepsen worker 52 - jepsen.util 682 :invoke :write [17 41] INFO [2022-03-13 17:20:14,143] jepsen worker 53 - jepsen.util 593 :invoke :write [17 25] INFO [2022-03-13 17:20:14,155] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,155] jepsen worker 3 - jepsen.util 2253 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,156] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,157] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 44] INFO [2022-03-13 17:20:14,203] jepsen worker 36 - jepsen.util 2466 :fail :write [12 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,204] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:20:14,209] jepsen worker 24 - jepsen.util 654 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,210] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:14,226] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,227] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,229] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,229] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 19] INFO [2022-03-13 17:20:14,230] jepsen worker 88 - jepsen.util 2608 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,231] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 14] INFO [2022-03-13 17:20:14,245] jepsen worker 7 - jepsen.util 637 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,246] jepsen worker 7 - jepsen.util 637 :invoke :write [2 45] INFO [2022-03-13 17:20:14,262] jepsen worker 4 - jepsen.util 2614 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,262] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,263] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 45] INFO [2022-03-13 17:20:14,263] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 28] INFO [2022-03-13 17:20:14,286] jepsen worker 60 - jepsen.util 690 :fail :write [20 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,287] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:14,300] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,301] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 20] INFO [2022-03-13 17:20:14,311] jepsen worker 77 - jepsen.util 707 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,311] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,311] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,311] jepsen worker 77 - jepsen.util 707 :invoke :write [25 37] INFO [2022-03-13 17:20:14,312] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:14,312] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:14,317] jepsen worker 35 - jepsen.util 935 :fail :write [11 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,318] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:14,320] jepsen worker 88 - jepsen.util 2608 :fail :write [29 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,321] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 32] INFO [2022-03-13 17:20:14,326] jepsen worker 87 - jepsen.util 2877 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,327] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:20:14,354] jepsen worker 4 - jepsen.util 2614 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,356] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 8] INFO [2022-03-13 17:20:14,390] jepsen worker 3 - jepsen.util 2253 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,390] jepsen worker 13 - jepsen.util 733 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,390] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,390] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,391] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 14] INFO [2022-03-13 17:20:14,392] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:14,392] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:14,393] jepsen worker 76 - jepsen.util 796 :invoke :write [25 32] INFO [2022-03-13 17:20:14,397] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,397] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,398] jepsen worker 13 - jepsen.util 733 :invoke :write [4 48] INFO [2022-03-13 17:20:14,399] jepsen worker 75 - jepsen.util 525 :invoke :write [25 18] INFO [2022-03-13 17:20:14,411] jepsen worker 5 - jepsen.util 2705 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,411] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,411] jepsen worker 33 - jepsen.util 753 :fail :write [11 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,412] jepsen worker 88 - jepsen.util 2608 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,419] jepsen worker 80 - jepsen.util 800 :invoke :write [26 10] INFO [2022-03-13 17:20:14,420] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,421] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:20:14,425] jepsen worker 33 - jepsen.util 753 :invoke :write [11 39] INFO [2022-03-13 17:20:14,429] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,429] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,443] jepsen worker 44 - jepsen.util 764 :fail :write [14 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,444] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:14,473] jepsen worker 36 - jepsen.util 2466 :fail :write [12 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,473] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,474] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,475] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:14,475] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,476] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,477] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,478] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,479] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,480] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,481] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,482] jepsen worker 22 - jepsen.util 652 :fail :write [7 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,482] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 32] INFO [2022-03-13 17:20:14,483] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:14,485] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:14,485] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,485] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,486] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:14,486] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:14,487] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:20:14,508] jepsen worker 4 - jepsen.util 2614 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,509] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:20:14,518] jepsen worker 79 - jepsen.util 1159 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,519] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:14,523] jepsen worker 51 - jepsen.util 771 :fail :write [17 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,523] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:14,544] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:14,544] jepsen worker nemesis - couchbase.util Rebalance check count: 720 INFO [2022-03-13 17:20:14,544] jepsen worker nemesis - couchbase.util Rebalance stuck count: 592 INFO [2022-03-13 17:20:14,545] jepsen worker 3 - jepsen.util 2253 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,545] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,546] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 2] INFO [2022-03-13 17:20:14,547] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 27] INFO [2022-03-13 17:20:14,551] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,552] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,553] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,554] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 20] INFO [2022-03-13 17:20:14,565] jepsen worker 53 - jepsen.util 593 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,566] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:14,573] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:14,574] jepsen worker 89 - jepsen.util 2789 :fail :read [29 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:14,574] jepsen worker 38 - jepsen.util 2828 :invoke :read [12 nil] INFO [2022-03-13 17:20:14,575] jepsen worker 89 - jepsen.util 2789 :invoke :write [29 39] INFO [2022-03-13 17:20:14,585] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,586] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,594] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,594] jepsen worker 88 - jepsen.util 2608 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,594] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,595] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 35] INFO [2022-03-13 17:20:14,596] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,596] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 11] INFO [2022-03-13 17:20:14,599] jepsen worker 6 - jepsen.util 636 :fail :write [2 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,599] jepsen worker 6 - jepsen.util 636 :invoke :write [2 42] INFO [2022-03-13 17:20:14,611] jepsen worker 76 - jepsen.util 796 :fail :write [25 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,612] jepsen worker 76 - jepsen.util 796 :invoke :write [25 13] INFO [2022-03-13 17:20:14,630] jepsen worker 5 - jepsen.util 2705 :fail :write [1 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,631] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,632] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,633] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,634] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,634] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:20:14,635] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,636] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,662] jepsen worker 23 - jepsen.util 563 :fail :write [7 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,662] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,662] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,663] jepsen worker 23 - jepsen.util 563 :invoke :write [7 39] INFO [2022-03-13 17:20:14,663] jepsen worker 14 - jepsen.util 374 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,663] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,663] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,664] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,664] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:14,665] jepsen worker 14 - jepsen.util 374 :invoke :write [4 35] INFO [2022-03-13 17:20:14,666] jepsen worker 24 - jepsen.util 654 :invoke :write [8 12] INFO [2022-03-13 17:20:14,667] jepsen worker 61 - jepsen.util 781 :invoke :write [20 29] INFO [2022-03-13 17:20:14,668] jepsen worker 35 - jepsen.util 935 :invoke :write [11 25] INFO [2022-03-13 17:20:14,675] jepsen worker 3 - jepsen.util 2253 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,676] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,679] jepsen worker 25 - jepsen.util 565 :invoke :write [8 47] INFO [2022-03-13 17:20:14,686] jepsen worker 87 - jepsen.util 2877 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,694] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,702] jepsen worker 88 - jepsen.util 2608 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,703] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 32] INFO [2022-03-13 17:20:14,717] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,717] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,718] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:20:14,718] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 42] INFO [2022-03-13 17:20:14,740] jepsen worker 36 - jepsen.util 2466 :fail :write [12 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,740] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,740] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,741] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:14,742] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 13] INFO [2022-03-13 17:20:14,742] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,742] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,743] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 30] INFO [2022-03-13 17:20:14,746] jepsen worker 12 - jepsen.util 912 :fail :write [4 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,748] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,749] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 36] INFO [2022-03-13 17:20:14,749] jepsen worker 52 - jepsen.util 682 :fail :write [17 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,749] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:14,749] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,750] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,750] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,750] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:14,750] jepsen worker 60 - jepsen.util 690 :invoke :write [20 18] INFO [2022-03-13 17:20:14,752] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:14,752] jepsen worker 52 - jepsen.util 682 :invoke :write [17 4] INFO [2022-03-13 17:20:14,753] jepsen worker 22 - jepsen.util 652 :invoke :write [7 19] INFO [2022-03-13 17:20:14,754] jepsen worker 26 - jepsen.util 566 :invoke :write [8 21] INFO [2022-03-13 17:20:14,756] jepsen worker 80 - jepsen.util 800 :fail :write [26 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,757] jepsen worker 80 - jepsen.util 800 :invoke :write [26 47] INFO [2022-03-13 17:20:14,793] jepsen worker 87 - jepsen.util 2877 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,794] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,796] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,797] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:20:14,804] jepsen worker 88 - jepsen.util 2608 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,805] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 21] INFO [2022-03-13 17:20:14,817] jepsen worker 13 - jepsen.util 733 :fail :write [4 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,817] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,818] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:20:14,819] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,821] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,821] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:20:14,824] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:14,825] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:14,828] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,829] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:14,830] jepsen worker 77 - jepsen.util 707 :fail :write [25 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,830] jepsen worker 77 - jepsen.util 707 :invoke :write [25 11] INFO [2022-03-13 17:20:14,833] jepsen worker 33 - jepsen.util 753 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,833] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,833] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,833] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,833] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:14,834] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:14,835] jepsen worker 21 - jepsen.util 831 :invoke :write [7 41] INFO [2022-03-13 17:20:14,835] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:14,836] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,837] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:14,849] jepsen worker 36 - jepsen.util 2466 :fail :write [12 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,849] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:14,882] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,883] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:20:14,885] jepsen worker 4 - jepsen.util 2614 :fail :write [1 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,886] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,906] jepsen worker 24 - jepsen.util 654 :fail :write [8 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,907] jepsen worker 88 - jepsen.util 2608 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,907] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:14,908] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:14,909] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,911] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 42] INFO [2022-03-13 17:20:14,911] jepsen worker 75 - jepsen.util 525 :fail :write [25 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,912] jepsen worker 75 - jepsen.util 525 :invoke :write [25 22] INFO [2022-03-13 17:20:14,937] jepsen worker 3 - jepsen.util 2253 :fail :write [1 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,937] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,937] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,938] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,938] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,938] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 39] INFO [2022-03-13 17:20:14,939] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 34] INFO [2022-03-13 17:20:14,940] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,940] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,941] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:14,942] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,943] jepsen worker 8 - jepsen.util 728 :fail :write [2 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,943] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 34] INFO [2022-03-13 17:20:14,943] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:14,945] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,945] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:20:14,974] jepsen worker 14 - jepsen.util 374 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,975] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:14,976] jepsen worker 88 - jepsen.util 2608 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,977] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 19] INFO [2022-03-13 17:20:14,985] jepsen worker 52 - jepsen.util 682 :fail :write [17 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:14,986] jepsen worker 52 - jepsen.util 682 :invoke :write [17 43] INFO [2022-03-13 17:20:15,008] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,009] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:15,012] jepsen worker 23 - jepsen.util 563 :fail :write [7 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,012] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:15,017] jepsen worker 4 - jepsen.util 2614 :fail :write [1 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,018] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,018] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,019] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:15,020] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,021] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 16] INFO [2022-03-13 17:20:15,057] jepsen worker 36 - jepsen.util 2466 :fail :write [12 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,057] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:15,077] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,078] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:20:15,079] jepsen worker 76 - jepsen.util 796 :fail :write [25 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,080] jepsen worker 76 - jepsen.util 796 :invoke :write [25 9] INFO [2022-03-13 17:20:15,084] jepsen worker 25 - jepsen.util 565 :fail :write [8 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,085] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:15,087] jepsen worker 88 - jepsen.util 2608 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,087] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 44] INFO [2022-03-13 17:20:15,120] jepsen worker 3 - jepsen.util 2253 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,120] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,120] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,121] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 8] INFO [2022-03-13 17:20:15,122] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 19] INFO [2022-03-13 17:20:15,122] jepsen worker 3 - jepsen.util 2253 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,123] jepsen worker 62 - jepsen.util 512 :fail :write [20 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,124] jepsen worker 3 - jepsen.util 2253 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,124] jepsen worker 62 - jepsen.util 512 :invoke :write [20 16] INFO [2022-03-13 17:20:15,125] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 44] INFO [2022-03-13 17:20:15,130] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:15,131] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 39] INFO [2022-03-13 17:20:15,136] jepsen worker 87 - jepsen.util 2877 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,136] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:20:15,155] jepsen worker 35 - jepsen.util 935 :fail :write [11 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,156] jepsen worker 35 - jepsen.util 935 :invoke :write [11 11] INFO [2022-03-13 17:20:15,169] jepsen worker 88 - jepsen.util 2608 :fail :write [29 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,171] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 12] INFO [2022-03-13 17:20:15,180] jepsen worker 22 - jepsen.util 652 :fail :write [7 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,182] jepsen worker 61 - jepsen.util 781 :fail :write [20 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,182] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,182] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,183] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 43] INFO [2022-03-13 17:20:15,184] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:15,185] jepsen worker 78 - jepsen.util 708 :invoke :read [26 nil] INFO [2022-03-13 17:20:15,186] jepsen worker 22 - jepsen.util 652 :invoke :write [7 16] INFO [2022-03-13 17:20:15,199] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,200] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 31] INFO [2022-03-13 17:20:15,239] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,239] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,239] jepsen worker 80 - jepsen.util 800 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,240] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:15,256] jepsen worker 36 - jepsen.util 2466 :fail :write [12 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,450] jepsen worker 80 - jepsen.util 800 :invoke :write [26 31] INFO [2022-03-13 17:20:15,258] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,451] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:20:15,290] jepsen worker 5 - jepsen.util 2705 :fail :write [1 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,325] jepsen worker 48 - jepsen.util 3558 :fail :write [16 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,453] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:20:15,453] jepsen worker 48 - jepsen.util 3558 :invoke :write [16 36] INFO [2022-03-13 17:20:15,453] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,454] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:15,331] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:15,331] jepsen worker 70 - jepsen.util 880 :fail :write [23 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,457] jepsen worker 42 - jepsen.util 402 :invoke :write [14 49] INFO [2022-03-13 17:20:15,457] jepsen worker 70 - jepsen.util 880 :invoke :write [23 13] INFO [2022-03-13 17:20:15,463] jepsen worker 3 - jepsen.util 2253 :fail :write [1 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,465] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 38] INFO [2022-03-13 17:20:15,332] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:15,339] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,339] jepsen worker 26 - jepsen.util 566 :fail :write [8 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,350] jepsen worker 21 - jepsen.util 831 :fail :write [7 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,465] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:15,350] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,465] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:15,465] jepsen worker 24 - jepsen.util 654 :invoke :write [8 40] INFO [2022-03-13 17:20:15,358] jepsen worker 77 - jepsen.util 707 :fail :write [25 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,358] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,359] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,361] jepsen worker 78 - jepsen.util 708 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,474] jepsen worker 78 - jepsen.util 708 :invoke :write [26 47] INFO [2022-03-13 17:20:15,365] jepsen worker 4 - jepsen.util 2614 :fail :write [1 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,406] jepsen worker 88 - jepsen.util 2608 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,450] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:15,474] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 42] INFO [2022-03-13 17:20:15,451] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:15,453] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,453] jepsen worker 34 - jepsen.util 934 :fail :write [11 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,475] jepsen worker 14 - jepsen.util 374 :invoke :write [4 35] INFO [2022-03-13 17:20:15,465] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:15,475] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:15,465] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:15,474] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:20:15,480] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,474] jepsen worker 23 - jepsen.util 563 :invoke :write [7 29] INFO [2022-03-13 17:20:15,474] jepsen worker 61 - jepsen.util 781 :invoke :write [20 28] INFO [2022-03-13 17:20:15,474] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 6] INFO [2022-03-13 17:20:15,494] jepsen worker 83 - jepsen.util 4583 :info :write [27 23] :SyncWriteAmbiguous INFO [2022-03-13 17:20:15,510] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:15,511] jepsen worker 83 - jepsen.util 4673 :invoke :write [27 17] INFO [2022-03-13 17:20:15,514] jepsen worker 7 - jepsen.util 637 :fail :write [2 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,514] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,514] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,514] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,514] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,515] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,515] jepsen worker 35 - jepsen.util 935 :fail :write [11 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,519] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:15,520] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,521] jepsen worker 7 - jepsen.util 637 :invoke :write [2 27] INFO [2022-03-13 17:20:15,523] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:15,524] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 16] INFO [2022-03-13 17:20:15,525] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:20:15,526] jepsen worker 33 - jepsen.util 753 :invoke :write [11 45] INFO [2022-03-13 17:20:15,530] jepsen worker 34 - jepsen.util 934 :invoke :write [11 1] INFO [2022-03-13 17:20:15,530] jepsen worker 51 - jepsen.util 771 :invoke :write [17 49] INFO [2022-03-13 17:20:15,551] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:15,552] jepsen worker nemesis - couchbase.util Rebalance check count: 721 INFO [2022-03-13 17:20:15,552] jepsen worker nemesis - couchbase.util Rebalance stuck count: 593 INFO [2022-03-13 17:20:15,557] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,558] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,591] jepsen worker 60 - jepsen.util 690 :fail :write [20 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,591] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,592] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:15,592] jepsen worker 75 - jepsen.util 525 :fail :write [25 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,592] jepsen worker 53 - jepsen.util 593 :invoke :write [17 19] INFO [2022-03-13 17:20:15,593] jepsen worker 79 - jepsen.util 1159 :fail :write [26 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,593] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:15,594] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:15,595] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:15,595] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:15,597] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,597] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,598] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,598] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:15,606] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,606] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,607] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 32] INFO [2022-03-13 17:20:15,609] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:15,610] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,611] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:20:15,628] jepsen worker 3 - jepsen.util 2253 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,629] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 24] INFO [2022-03-13 17:20:15,658] jepsen worker 80 - jepsen.util 800 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,659] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:15,659] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,660] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:15,687] jepsen worker 48 - jepsen.util 3558 :fail :write [16 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,688] jepsen worker 48 - jepsen.util 3558 :invoke :read [16 nil] INFO [2022-03-13 17:20:15,704] jepsen worker 5 - jepsen.util 2705 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,705] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,713] jepsen worker 6 - jepsen.util 636 :fail :write [2 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,713] jepsen worker 76 - jepsen.util 796 :fail :write [25 9] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,713] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,713] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,713] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,713] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,714] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:15,714] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,715] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:15,716] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:15,717] jepsen worker 12 - jepsen.util 912 :invoke :read [4 nil] INFO [2022-03-13 17:20:15,717] jepsen worker 76 - jepsen.util 796 :invoke :write [25 4] INFO [2022-03-13 17:20:15,718] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,718] jepsen worker 75 - jepsen.util 525 :invoke :write [25 25] INFO [2022-03-13 17:20:15,752] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,753] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,753] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:15,754] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,755] jepsen worker 4 - jepsen.util 2614 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,756] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 5] INFO [2022-03-13 17:20:15,759] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,759] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,760] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,760] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:20:15,761] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,761] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:15,763] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,763] jepsen worker 52 - jepsen.util 682 :fail :write [17 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,764] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,764] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 48] INFO [2022-03-13 17:20:15,764] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:15,765] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:15,766] jepsen worker 24 - jepsen.util 654 :fail :write [8 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,767] jepsen worker 24 - jepsen.util 654 :invoke :write [8 20] INFO [2022-03-13 17:20:15,780] jepsen worker 23 - jepsen.util 563 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,781] jepsen worker 23 - jepsen.util 563 :invoke :write [7 12] INFO [2022-03-13 17:20:15,785] jepsen worker 88 - jepsen.util 2608 :fail :write [29 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,786] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 19] INFO [2022-03-13 17:20:15,820] jepsen worker 82 - jepsen.util 5302 :info :write [27 16] :RequestTimeoutException INFO [2022-03-13 17:20:15,821] jepsen worker 82 - jepsen.util 5392 :invoke :read [27 nil] INFO [2022-03-13 17:20:15,844] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,844] jepsen worker 14 - jepsen.util 374 :fail :write [4 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,845] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:15,845] jepsen worker 21 - jepsen.util 831 :invoke :write [7 17] INFO [2022-03-13 17:20:15,857] jepsen worker 36 - jepsen.util 2466 :fail :write [12 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,859] jepsen worker 36 - jepsen.util 2466 :invoke :write [12 45] INFO [2022-03-13 17:20:15,872] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,873] jepsen worker 22 - jepsen.util 652 :fail :write [7 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,874] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:15,874] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:15,876] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,876] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:15,877] jepsen worker 60 - jepsen.util 690 :invoke :write [20 3] INFO [2022-03-13 17:20:15,878] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:15,883] jepsen worker 5 - jepsen.util 2705 :fail :write [1 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,885] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:15,886] jepsen worker 58 - jepsen.util 4468 :ok :read [19 8] INFO [2022-03-13 17:20:15,887] jepsen worker 58 - jepsen.util 4468 :invoke :write [19 12] INFO [2022-03-13 17:20:15,887] jepsen worker 0 - jepsen.util 4860 :ok :read [0 41] INFO [2022-03-13 17:20:15,888] jepsen worker 0 - jepsen.util 4860 :invoke :write [0 8] INFO [2022-03-13 17:20:15,914] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:15,914] jepsen worker 62 - jepsen.util 512 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,915] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:15,916] jepsen worker 62 - jepsen.util 512 :invoke :write [20 16] INFO [2022-03-13 17:20:15,916] jepsen worker 33 - jepsen.util 753 :fail :write [11 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,917] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:15,924] jepsen worker 76 - jepsen.util 796 :fail :write [25 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,924] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:15,934] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,934] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 1] INFO [2022-03-13 17:20:15,949] jepsen worker 3 - jepsen.util 2253 :fail :write [1 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,949] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:20:15,995] jepsen worker 61 - jepsen.util 781 :fail :write [20 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:15,995] jepsen worker 61 - jepsen.util 781 :invoke :write [20 2] INFO [2022-03-13 17:20:16,002] jepsen worker 88 - jepsen.util 2608 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,003] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 12] INFO [2022-03-13 17:20:16,010] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,011] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 7] INFO [2022-03-13 17:20:16,029] jepsen worker 34 - jepsen.util 934 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,029] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,030] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,031] jepsen worker 34 - jepsen.util 934 :invoke :write [11 27] INFO [2022-03-13 17:20:16,031] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:16,032] jepsen worker 80 - jepsen.util 800 :invoke :write [26 40] INFO [2022-03-13 17:20:16,036] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,037] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,036] jepsen worker 53 - jepsen.util 593 :fail :write [17 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,036] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,037] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:16,038] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:16,039] jepsen worker 22 - jepsen.util 652 :invoke :write [7 18] INFO [2022-03-13 17:20:16,040] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:16,040] jepsen worker 53 - jepsen.util 593 :invoke :write [17 30] INFO [2022-03-13 17:20:16,041] jepsen worker 43 - jepsen.util 673 :invoke :write [14 16] INFO [2022-03-13 17:20:16,066] jepsen worker 1 - jepsen.util 4681 :info :write [0 36] :RequestTimeoutException INFO [2022-03-13 17:20:16,067] jepsen worker 1 - jepsen.util 4771 :invoke :read [0 nil] INFO [2022-03-13 17:20:16,078] jepsen worker 4 - jepsen.util 2614 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,078] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 45] INFO [2022-03-13 17:20:16,080] jepsen worker 87 - jepsen.util 2877 :fail :write [29 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,081] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:20:16,096] jepsen worker 2 - jepsen.util 5222 :info :write [0 47] :RequestTimeoutException INFO [2022-03-13 17:20:16,102] jepsen worker 2 - jepsen.util 5312 :invoke :write [0 14] INFO [2022-03-13 17:20:16,104] jepsen worker 75 - jepsen.util 525 :fail :write [25 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,105] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,106] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,106] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,107] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,107] jepsen worker 76 - jepsen.util 796 :invoke :write [25 20] INFO [2022-03-13 17:20:16,114] jepsen worker 70 - jepsen.util 880 :fail :write [23 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,115] jepsen worker 70 - jepsen.util 880 :invoke :write [23 19] INFO [2022-03-13 17:20:16,124] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,124] jepsen worker 35 - jepsen.util 935 :invoke :write [11 13] INFO [2022-03-13 17:20:16,125] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,125] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,126] jepsen worker 75 - jepsen.util 525 :invoke :write [25 45] INFO [2022-03-13 17:20:16,127] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,129] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,129] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,131] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,132] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,143] jepsen worker 1 - jepsen.util 4771 :ok :read [0 41] INFO [2022-03-13 17:20:16,144] jepsen worker 1 - jepsen.util 4771 :invoke :write [0 29] INFO [2022-03-13 17:20:16,145] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,146] jepsen worker 5 - jepsen.util 2705 :fail :write [1 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,146] jepsen worker 85 - jepsen.util 4675 :info :write [28 10] :RequestTimeoutException INFO [2022-03-13 17:20:16,146] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,148] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:20:16,148] jepsen worker 85 - jepsen.util 4765 :invoke :read [28 nil] INFO [2022-03-13 17:20:16,154] jepsen worker 36 - jepsen.util 2466 :fail :write [12 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,155] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:16,174] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,174] jepsen worker 78 - jepsen.util 708 :fail :write [26 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,175] jepsen worker 78 - jepsen.util 708 :invoke :write [26 36] INFO [2022-03-13 17:20:16,175] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:16,178] jepsen worker 88 - jepsen.util 2608 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,179] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:16,180] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,182] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 18] INFO [2022-03-13 17:20:16,199] jepsen worker 60 - jepsen.util 690 :fail :write [20 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,199] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,199] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,200] jepsen worker 52 - jepsen.util 682 :invoke :write [17 13] INFO [2022-03-13 17:20:16,201] jepsen worker 60 - jepsen.util 690 :invoke :write [20 0] INFO [2022-03-13 17:20:16,201] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:16,202] jepsen worker 51 - jepsen.util 771 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,203] jepsen worker 51 - jepsen.util 771 :invoke :write [17 46] INFO [2022-03-13 17:20:16,212] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,213] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 32] INFO [2022-03-13 17:20:16,234] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,235] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:20:16,259] jepsen worker 4 - jepsen.util 2614 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,259] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,259] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:16,260] jepsen worker 4 - jepsen.util 2614 :invoke :write [1 22] INFO [2022-03-13 17:20:16,284] jepsen worker 24 - jepsen.util 654 :fail :write [8 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,285] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:16,294] jepsen worker 42 - jepsen.util 402 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,294] jepsen worker 42 - jepsen.util 402 :invoke :write [14 30] INFO [2022-03-13 17:20:16,309] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,310] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:16,313] jepsen worker 88 - jepsen.util 2608 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,313] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,314] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:16,315] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:16,316] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,316] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,317] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:16,317] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:16,318] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,318] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,319] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:16,319] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:16,320] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,320] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 41] INFO [2022-03-13 17:20:16,347] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,348] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,361] jepsen worker 21 - jepsen.util 831 :fail :write [7 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,361] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,361] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,361] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,362] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,362] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,363] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,363] jepsen worker 21 - jepsen.util 831 :invoke :write [7 24] INFO [2022-03-13 17:20:16,364] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:20:16,365] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:16,366] jepsen worker 53 - jepsen.util 593 :fail :write [17 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,366] jepsen worker 76 - jepsen.util 796 :fail :write [25 20] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,366] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,367] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,368] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,369] jepsen worker 53 - jepsen.util 593 :invoke :write [17 14] INFO [2022-03-13 17:20:16,370] jepsen worker 33 - jepsen.util 753 :invoke :write [11 36] INFO [2022-03-13 17:20:16,370] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:20:16,448] jepsen worker 7 - jepsen.util 637 :fail :write [2 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,448] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,448] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,448] jepsen worker 12 - jepsen.util 912 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,449] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:16,449] jepsen worker 7 - jepsen.util 637 :invoke :write [2 30] INFO [2022-03-13 17:20:16,449] jepsen worker 12 - jepsen.util 912 :invoke :write [4 36] INFO [2022-03-13 17:20:16,451] jepsen worker 34 - jepsen.util 934 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,455] jepsen worker 34 - jepsen.util 934 :invoke :write [11 27] INFO [2022-03-13 17:20:16,456] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:16,456] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:16,457] jepsen worker 8 - jepsen.util 728 :invoke :write [2 37] INFO [2022-03-13 17:20:16,496] jepsen worker 23 - jepsen.util 563 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,496] jepsen worker 23 - jepsen.util 563 :invoke :write [7 24] INFO [2022-03-13 17:20:16,520] jepsen worker 80 - jepsen.util 800 :fail :write [26 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,521] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:16,557] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:16,557] jepsen worker nemesis - couchbase.util Rebalance check count: 722 INFO [2022-03-13 17:20:16,557] jepsen worker nemesis - couchbase.util Rebalance stuck count: 594 INFO [2022-03-13 17:20:16,561] jepsen worker 43 - jepsen.util 673 :fail :write [14 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,561] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,561] jepsen worker 75 - jepsen.util 525 :fail :write [25 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,562] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,562] jepsen worker 43 - jepsen.util 673 :invoke :write [14 46] INFO [2022-03-13 17:20:16,563] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:16,563] jepsen worker 75 - jepsen.util 525 :invoke :write [25 36] INFO [2022-03-13 17:20:16,564] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,565] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,565] jepsen worker 13 - jepsen.util 733 :invoke :write [4 7] INFO [2022-03-13 17:20:16,565] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,566] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,587] jepsen worker 35 - jepsen.util 935 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,587] jepsen worker 62 - jepsen.util 512 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,588] jepsen worker 35 - jepsen.util 935 :invoke :write [11 6] INFO [2022-03-13 17:20:16,589] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:16,595] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,596] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,632] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,632] jepsen worker 61 - jepsen.util 781 :fail :write [20 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,632] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 6] INFO [2022-03-13 17:20:16,633] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:16,663] jepsen worker 51 - jepsen.util 771 :fail :write [17 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,663] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,663] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,663] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,664] jepsen worker 14 - jepsen.util 374 :invoke :write [4 27] INFO [2022-03-13 17:20:16,665] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:16,665] jepsen worker 51 - jepsen.util 771 :invoke :write [17 17] INFO [2022-03-13 17:20:16,666] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:16,684] jepsen worker 66 - jepsen.util 4746 :info :write [22 30] :RequestTimeoutException INFO [2022-03-13 17:20:16,685] jepsen worker 66 - jepsen.util 4836 :invoke :write [22 42] INFO [2022-03-13 17:20:16,736] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,736] jepsen worker 22 - jepsen.util 652 :fail :write [7 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,737] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:16,737] jepsen worker 26 - jepsen.util 566 :invoke :write [8 10] INFO [2022-03-13 17:20:16,763] jepsen worker 70 - jepsen.util 880 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,764] jepsen worker 70 - jepsen.util 880 :invoke :write [23 27] INFO [2022-03-13 17:20:16,782] jepsen worker 21 - jepsen.util 831 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,784] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:16,820] jepsen worker 78 - jepsen.util 708 :fail :write [26 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,820] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,820] jepsen worker 78 - jepsen.util 708 :invoke :write [26 19] INFO [2022-03-13 17:20:16,821] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,843] jepsen worker 75 - jepsen.util 525 :fail :write [25 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,844] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:16,902] jepsen worker 60 - jepsen.util 690 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,902] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:16,944] jepsen worker 33 - jepsen.util 753 :fail :write [11 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:16,945] jepsen worker 33 - jepsen.util 753 :invoke :write [11 39] INFO [2022-03-13 17:20:17,008] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,008] jepsen worker 42 - jepsen.util 402 :fail :write [14 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,008] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:17,009] jepsen worker 44 - jepsen.util 764 :invoke :write [14 29] INFO [2022-03-13 17:20:17,009] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:17,010] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:17,030] jepsen worker 13 - jepsen.util 733 :fail :write [4 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,031] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,031] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:17,031] jepsen worker 76 - jepsen.util 796 :invoke :write [25 49] INFO [2022-03-13 17:20:17,032] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,032] jepsen worker 75 - jepsen.util 525 :invoke :write [25 14] INFO [2022-03-13 17:20:17,042] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,043] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:17,053] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,053] jepsen worker 23 - jepsen.util 563 :fail :write [7 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,054] jepsen worker 23 - jepsen.util 563 :invoke :write [7 42] INFO [2022-03-13 17:20:17,055] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:17,062] jepsen worker 34 - jepsen.util 934 :fail :write [11 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,062] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,063] jepsen worker 80 - jepsen.util 800 :invoke :write [26 31] INFO [2022-03-13 17:20:17,063] jepsen worker 34 - jepsen.util 934 :invoke :write [11 13] INFO [2022-03-13 17:20:17,107] jepsen worker 52 - jepsen.util 682 :fail :write [17 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,108] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,108] jepsen worker 52 - jepsen.util 682 :invoke :write [17 49] INFO [2022-03-13 17:20:17,108] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:17,163] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,163] jepsen worker 35 - jepsen.util 935 :fail :write [11 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,163] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,163] jepsen worker 24 - jepsen.util 654 :invoke :write [8 43] INFO [2022-03-13 17:20:17,164] jepsen worker 35 - jepsen.util 935 :invoke :write [11 42] INFO [2022-03-13 17:20:17,165] jepsen worker 25 - jepsen.util 565 :invoke :write [8 1] INFO [2022-03-13 17:20:17,203] jepsen worker 12 - jepsen.util 912 :fail :write [4 36] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,204] jepsen worker 12 - jepsen.util 912 :invoke :write [4 33] INFO [2022-03-13 17:20:17,225] jepsen worker 71 - jepsen.util 611 :fail :write [23 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,225] jepsen worker 79 - jepsen.util 1159 :fail :write [26 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,225] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,226] jepsen worker 71 - jepsen.util 611 :invoke :write [23 22] INFO [2022-03-13 17:20:17,227] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:17,227] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 15] INFO [2022-03-13 17:20:17,315] jepsen worker 53 - jepsen.util 593 :fail :write [17 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,316] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] WARN [2022-03-13 17:20:17,323] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18112040,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":27029567,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9210931,"encode_duration_us":19638,"operation_id":"0x14902","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":10005923},{"operation_name":"upsert","last_dispatch_duration_us":9079045,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":9925790,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9614209,"encode_duration_us":18627,"operation_id":"0x14cf4","timeout_ms":10000,"last_server_duration_us":9066042,"total_duration_us":10009676}],"total_count":2}} WARN [2022-03-13 17:20:17,323] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","operation_id":"0x146eb","timeout_ms":30000,"total_duration_us":30006950},{"operation_name":"get","last_dispatch_duration_us":11622647,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":11622647,"total_server_duration_us":27,"operation_id":"0x15218","timeout_ms":30000,"last_server_duration_us":27,"total_duration_us":11623626},{"operation_name":"upsert","last_dispatch_duration_us":9871317,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","total_dispatch_duration_us":9871317,"total_server_duration_us":20601,"encode_duration_us":19579,"operation_id":"0x15314","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10004520},{"operation_name":"upsert","last_dispatch_duration_us":9075732,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":9075732,"total_server_duration_us":9064980,"encode_duration_us":20640,"operation_id":"0x150d7","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":9079346},{"operation_name":"get","last_dispatch_duration_us":9070067,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9070067,"total_server_duration_us":49,"operation_id":"0x153a1","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":9072661},{"operation_name":"get","last_dispatch_duration_us":8408228,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":8408228,"total_server_duration_us":49,"operation_id":"0x15166","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8411819},{"operation_name":"get","last_dispatch_duration_us":6359661,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":6359661,"total_server_duration_us":32,"operation_id":"0x15287","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6362487},{"operation_name":"get","last_dispatch_duration_us":4488153,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":4488153,"total_server_duration_us":12763,"operation_id":"0x153d3","timeout_ms":30000,"last_server_duration_us":12763,"total_duration_us":4488480},{"operation_name":"upsert","last_dispatch_duration_us":487431,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":487431,"total_server_duration_us":67485,"encode_duration_us":17183,"operation_id":"0x15866","timeout_ms":10000,"last_server_duration_us":67485,"total_duration_us":4124905},{"operation_name":"get","last_dispatch_duration_us":463158,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000001/00000000A19F4D82","last_local_socket":"172.23.104.79:37462","total_dispatch_duration_us":463158,"total_server_duration_us":38,"operation_id":"0x1589b","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":3076492}],"total_count":55}} INFO [2022-03-13 17:20:17,331] jepsen worker 51 - jepsen.util 771 :fail :write [17 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,332] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:17,339] jepsen worker 7 - jepsen.util 637 :fail :write [2 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,339] jepsen worker 7 - jepsen.util 637 :invoke :write [2 22] INFO [2022-03-13 17:20:17,364] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,364] jepsen worker 78 - jepsen.util 708 :fail :write [26 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,364] jepsen worker 61 - jepsen.util 781 :invoke :write [20 17] INFO [2022-03-13 17:20:17,366] jepsen worker 78 - jepsen.util 708 :invoke :write [26 28] INFO [2022-03-13 17:20:17,400] jepsen worker 76 - jepsen.util 796 :fail :write [25 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,401] jepsen worker 76 - jepsen.util 796 :invoke :write [25 24] INFO [2022-03-13 17:20:17,411] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,411] jepsen worker 8 - jepsen.util 728 :fail :write [2 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,412] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:17,413] jepsen worker 8 - jepsen.util 728 :invoke :read [2 nil] INFO [2022-03-13 17:20:17,444] jepsen worker 14 - jepsen.util 374 :fail :write [4 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,444] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,445] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:17,446] jepsen worker 21 - jepsen.util 831 :invoke :write [7 4] INFO [2022-03-13 17:20:17,522] jepsen worker 23 - jepsen.util 563 :fail :write [7 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,522] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,523] jepsen worker 23 - jepsen.util 563 :invoke :write [7 29] INFO [2022-03-13 17:20:17,523] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:17,539] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,539] jepsen worker 43 - jepsen.util 673 :fail :write [14 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,539] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:17,540] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:17,551] jepsen worker 72 - jepsen.util 4932 :info :write [24 10] :SyncWriteAmbiguous INFO [2022-03-13 17:20:17,552] jepsen worker 72 - jepsen.util 5022 :invoke :read [24 nil] INFO [2022-03-13 17:20:17,561] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:17,561] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,561] jepsen worker 70 - jepsen.util 880 :fail :write [23 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,561] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:17,562] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:17,562] jepsen worker 42 - jepsen.util 402 :invoke :write [14 2] INFO [2022-03-13 17:20:17,562] jepsen worker nemesis - couchbase.util Rebalance check count: 723 INFO [2022-03-13 17:20:17,562] jepsen worker nemesis - couchbase.util Rebalance stuck count: 595 INFO [2022-03-13 17:20:17,562] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:17,598] jepsen worker 54 - jepsen.util 4644 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,599] jepsen worker 54 - jepsen.util 4644 :invoke :read [18 nil] INFO [2022-03-13 17:20:17,631] jepsen worker 79 - jepsen.util 1159 :fail :write [26 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,632] jepsen worker 79 - jepsen.util 1159 :invoke :write [26 2] INFO [2022-03-13 17:20:17,650] jepsen worker 75 - jepsen.util 525 :fail :write [25 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,650] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,651] jepsen worker 13 - jepsen.util 733 :invoke :write [4 24] INFO [2022-03-13 17:20:17,652] jepsen worker 75 - jepsen.util 525 :invoke :write [25 2] INFO [2022-03-13 17:20:17,653] jepsen worker 33 - jepsen.util 753 :fail :write [11 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,655] jepsen worker 33 - jepsen.util 753 :invoke :write [11 5] INFO [2022-03-13 17:20:17,665] jepsen worker 44 - jepsen.util 764 :fail :write [14 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,666] jepsen worker 44 - jepsen.util 764 :invoke :write [14 13] INFO [2022-03-13 17:20:17,687] jepsen worker 61 - jepsen.util 781 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,688] jepsen worker 61 - jepsen.util 781 :invoke :write [20 16] INFO [2022-03-13 17:20:17,725] jepsen worker 12 - jepsen.util 912 :fail :write [4 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,725] jepsen worker 12 - jepsen.util 912 :invoke :write [4 6] INFO [2022-03-13 17:20:17,745] jepsen worker 78 - jepsen.util 708 :fail :write [26 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,746] jepsen worker 78 - jepsen.util 708 :invoke :write [26 34] INFO [2022-03-13 17:20:17,747] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,748] jepsen worker 77 - jepsen.util 707 :invoke :write [25 1] INFO [2022-03-13 17:20:17,772] jepsen worker 56 - jepsen.util 4736 :fail :read [18 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,773] jepsen worker 56 - jepsen.util 4736 :invoke :write [18 11] INFO [2022-03-13 17:20:17,787] jepsen worker 34 - jepsen.util 934 :fail :write [11 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,787] jepsen worker 45 - jepsen.util 5175 :ok :read [15 48] INFO [2022-03-13 17:20:17,788] jepsen worker 34 - jepsen.util 934 :invoke :read [11 nil] INFO [2022-03-13 17:20:17,788] jepsen worker 45 - jepsen.util 5175 :invoke :write [15 48] INFO [2022-03-13 17:20:17,794] jepsen worker 46 - jepsen.util 5266 :ok :read [15 48] INFO [2022-03-13 17:20:17,794] jepsen worker 46 - jepsen.util 5266 :invoke :read [15 nil] INFO [2022-03-13 17:20:17,861] jepsen worker 26 - jepsen.util 566 :fail :write [8 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,861] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,862] jepsen worker 26 - jepsen.util 566 :invoke :write [8 13] INFO [2022-03-13 17:20:17,863] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:17,875] jepsen worker 7 - jepsen.util 637 :fail :write [2 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,875] jepsen worker 8 - jepsen.util 728 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,876] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,876] jepsen worker 8 - jepsen.util 728 :invoke :write [2 18] INFO [2022-03-13 17:20:17,877] jepsen worker 62 - jepsen.util 512 :invoke :write [20 6] INFO [2022-03-13 17:20:17,877] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:17,878] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,878] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,879] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:17,879] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:17,890] jepsen worker 80 - jepsen.util 800 :fail :write [26 31] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,891] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:17,974] jepsen worker 76 - jepsen.util 796 :fail :write [25 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,975] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:17,988] jepsen worker 23 - jepsen.util 563 :fail :write [7 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:17,988] jepsen worker 23 - jepsen.util 563 :invoke :read [7 nil] INFO [2022-03-13 17:20:18,002] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,003] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:18,022] jepsen worker 46 - jepsen.util 5266 :ok :read [15 48] INFO [2022-03-13 17:20:18,023] jepsen worker 46 - jepsen.util 5266 :invoke :write [15 7] INFO [2022-03-13 17:20:18,060] jepsen worker 24 - jepsen.util 654 :fail :write [8 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,061] jepsen worker 24 - jepsen.util 654 :invoke :write [8 38] INFO [2022-03-13 17:20:18,086] jepsen worker 3 - jepsen.util 2253 :fail :write [1 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,087] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,087] jepsen worker 3 - jepsen.util 2253 :invoke :write [1 16] INFO [2022-03-13 17:20:18,088] jepsen worker 36 - jepsen.util 2466 :invoke :read [12 nil] INFO [2022-03-13 17:20:18,089] jepsen worker 52 - jepsen.util 682 :fail :write [17 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,089] jepsen worker 13 - jepsen.util 733 :fail :write [4 24] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,089] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,090] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:18,091] jepsen worker 52 - jepsen.util 682 :invoke :read [17 nil] INFO [2022-03-13 17:20:18,091] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,091] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:18,091] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,092] jepsen worker 13 - jepsen.util 733 :invoke :write [4 47] INFO [2022-03-13 17:20:18,094] jepsen worker 60 - jepsen.util 690 :invoke :write [20 44] INFO [2022-03-13 17:20:18,098] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:18,099] jepsen worker 22 - jepsen.util 652 :invoke :write [7 12] INFO [2022-03-13 17:20:18,121] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,123] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 29] INFO [2022-03-13 17:20:18,146] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,149] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 14] INFO [2022-03-13 17:20:18,155] jepsen worker 42 - jepsen.util 402 :fail :write [14 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,156] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:18,158] jepsen worker 25 - jepsen.util 565 :fail :write [8 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,158] jepsen worker 25 - jepsen.util 565 :invoke :read [8 nil] INFO [2022-03-13 17:20:18,161] jepsen worker 27 - jepsen.util 5157 :ok :read [9 26] INFO [2022-03-13 17:20:18,161] jepsen worker 27 - jepsen.util 5157 :invoke :read [9 nil] INFO [2022-03-13 17:20:18,187] jepsen worker 87 - jepsen.util 2877 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,188] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 17] INFO [2022-03-13 17:20:18,215] jepsen worker 4 - jepsen.util 2614 :fail :write [1 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,215] jepsen worker 4 - jepsen.util 2614 :invoke :read [1 nil] INFO [2022-03-13 17:20:18,219] jepsen worker 11 - jepsen.util 4691 :info :write [3 20] :RequestTimeoutException INFO [2022-03-13 17:20:18,220] jepsen worker 11 - jepsen.util 4781 :invoke :read [3 nil] INFO [2022-03-13 17:20:18,223] jepsen worker 26 - jepsen.util 566 :fail :write [8 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,223] jepsen worker 52 - jepsen.util 682 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,223] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,223] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:18,223] jepsen worker 5 - jepsen.util 2705 :fail :write [1 14] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,225] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:18,225] jepsen worker 52 - jepsen.util 682 :invoke :write [17 25] INFO [2022-03-13 17:20:18,226] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:20:18,227] jepsen worker 53 - jepsen.util 593 :invoke :write [17 1] INFO [2022-03-13 17:20:18,227] jepsen worker 79 - jepsen.util 1159 :fail :write [26 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,228] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:18,229] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:18,230] jepsen worker 10 - jepsen.util 5230 :info :write [3 18] :RequestTimeoutException INFO [2022-03-13 17:20:18,231] jepsen worker 10 - jepsen.util 5320 :invoke :read [3 nil] INFO [2022-03-13 17:20:18,253] jepsen worker 87 - jepsen.util 2877 :fail :write [29 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,254] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,256] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,257] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,258] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,259] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,260] jepsen worker 19 - jepsen.util 4699 :info :write [6 7] :RequestTimeoutException INFO [2022-03-13 17:20:18,260] jepsen worker 19 - jepsen.util 4789 :invoke :write [6 33] INFO [2022-03-13 17:20:18,261] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,262] jepsen worker 75 - jepsen.util 525 :fail :write [25 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,262] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,262] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,263] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:18,263] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:18,292] jepsen worker 27 - jepsen.util 5157 :ok :read [9 26] INFO [2022-03-13 17:20:18,293] jepsen worker 27 - jepsen.util 5157 :invoke :write [9 29] INFO [2022-03-13 17:20:18,310] jepsen worker 54 - jepsen.util 4644 :ok :read [18 21] INFO [2022-03-13 17:20:18,310] jepsen worker 54 - jepsen.util 4644 :invoke :read [18 nil] INFO [2022-03-13 17:20:18,313] jepsen worker 11 - jepsen.util 4781 :ok :read [3 25] INFO [2022-03-13 17:20:18,315] jepsen worker 11 - jepsen.util 4781 :invoke :read [3 nil] INFO [2022-03-13 17:20:18,315] jepsen worker 88 - jepsen.util 2608 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,315] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,315] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,315] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 30] INFO [2022-03-13 17:20:18,316] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,317] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:20:18,320] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,322] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,325] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,325] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 7] INFO [2022-03-13 17:20:18,346] jepsen worker 47 - jepsen.util 4817 :info :write [15 15] :RequestTimeoutException INFO [2022-03-13 17:20:18,347] jepsen worker 47 - jepsen.util 4907 :invoke :write [15 28] INFO [2022-03-13 17:20:18,349] jepsen worker 54 - jepsen.util 4644 :ok :read [18 21] INFO [2022-03-13 17:20:18,349] jepsen worker 11 - jepsen.util 4781 :ok :read [3 25] INFO [2022-03-13 17:20:18,349] jepsen worker 11 - jepsen.util 4781 :invoke :read [3 nil] INFO [2022-03-13 17:20:18,349] jepsen worker 54 - jepsen.util 4644 :invoke :read [18 nil] INFO [2022-03-13 17:20:18,362] jepsen worker 35 - jepsen.util 935 :fail :write [11 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,363] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:18,383] jepsen worker 88 - jepsen.util 2608 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,384] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,384] jepsen worker 11 - jepsen.util 4781 :ok :read [3 25] INFO [2022-03-13 17:20:18,384] jepsen worker 11 - jepsen.util 4781 :invoke :write [3 22] INFO [2022-03-13 17:20:18,385] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,385] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 43] INFO [2022-03-13 17:20:18,396] jepsen worker 54 - jepsen.util 4644 :ok :read [18 21] INFO [2022-03-13 17:20:18,396] jepsen worker 54 - jepsen.util 4644 :invoke :read [18 nil] INFO [2022-03-13 17:20:18,416] jepsen worker 64 - jepsen.util 4834 :info :write [21 9] :RequestTimeoutException INFO [2022-03-13 17:20:18,417] jepsen worker 64 - jepsen.util 4924 :invoke :read [21 nil] INFO [2022-03-13 17:20:18,418] jepsen worker 64 - jepsen.util 4924 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,419] jepsen worker 64 - jepsen.util 4924 :invoke :read [21 nil] INFO [2022-03-13 17:20:18,420] jepsen worker 64 - jepsen.util 4924 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,421] jepsen worker 64 - jepsen.util 4924 :invoke :read [21 nil] INFO [2022-03-13 17:20:18,422] jepsen worker 64 - jepsen.util 4924 :fail :read [21 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,422] jepsen worker 64 - jepsen.util 4924 :invoke :write [21 5] INFO [2022-03-13 17:20:18,440] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,440] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 12] INFO [2022-03-13 17:20:18,444] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,444] jepsen worker 33 - jepsen.util 753 :fail :write [11 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,444] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,445] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:18,445] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,445] jepsen worker 34 - jepsen.util 934 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,446] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:18,446] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:18,447] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:18,447] jepsen worker 34 - jepsen.util 934 :invoke :write [11 4] INFO [2022-03-13 17:20:18,472] jepsen worker 44 - jepsen.util 764 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,473] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:18,474] jepsen worker 87 - jepsen.util 2877 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,474] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,477] jepsen worker 88 - jepsen.util 2608 :fail :write [29 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,477] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,478] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 45] INFO [2022-03-13 17:20:18,479] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,480] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,481] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,482] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,483] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,484] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,484] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:20:18,511] jepsen worker 73 - jepsen.util 5293 :info :write [24 11] :RequestTimeoutException INFO [2022-03-13 17:20:18,511] jepsen worker 73 - jepsen.util 5383 :invoke :read [24 nil] INFO [2022-03-13 17:20:18,514] jepsen worker 21 - jepsen.util 831 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,514] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:18,518] jepsen worker 5 - jepsen.util 2705 :fail :write [1 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,518] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:18,520] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,520] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 28] INFO [2022-03-13 17:20:18,551] jepsen worker 18 - jepsen.util 4878 :info :write [6 16] :RequestTimeoutException INFO [2022-03-13 17:20:18,551] jepsen worker 18 - jepsen.util 4968 :invoke :read [6 nil] INFO [2022-03-13 17:20:18,567] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:18,567] jepsen worker nemesis - couchbase.util Rebalance check count: 724 INFO [2022-03-13 17:20:18,567] jepsen worker nemesis - couchbase.util Rebalance stuck count: 596 INFO [2022-03-13 17:20:18,579] jepsen worker 13 - jepsen.util 733 :fail :write [4 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,579] jepsen worker 60 - jepsen.util 690 :fail :write [20 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,580] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,580] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,581] jepsen worker 13 - jepsen.util 733 :invoke :write [4 11] INFO [2022-03-13 17:20:18,581] jepsen worker 60 - jepsen.util 690 :invoke :write [20 11] INFO [2022-03-13 17:20:18,581] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:18,584] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,585] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,585] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,585] jepsen worker 75 - jepsen.util 525 :invoke :write [25 10] INFO [2022-03-13 17:20:18,586] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,586] jepsen worker 61 - jepsen.util 781 :fail :write [20 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,587] jepsen worker 76 - jepsen.util 796 :invoke :write [25 39] INFO [2022-03-13 17:20:18,587] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:18,588] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,589] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,590] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,591] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,592] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:20:18,630] jepsen worker 5 - jepsen.util 2705 :fail :write [1 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,630] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:18,632] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,633] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 15] INFO [2022-03-13 17:20:18,652] jepsen worker 25 - jepsen.util 565 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,652] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,652] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,652] jepsen worker 24 - jepsen.util 654 :fail :write [8 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,652] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,652] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,653] jepsen worker 25 - jepsen.util 565 :invoke :write [8 34] INFO [2022-03-13 17:20:18,654] jepsen worker 80 - jepsen.util 800 :invoke :write [26 18] INFO [2022-03-13 17:20:18,655] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:18,656] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:18,657] jepsen worker 33 - jepsen.util 753 :invoke :write [11 1] INFO [2022-03-13 17:20:18,657] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,658] jepsen worker 14 - jepsen.util 374 :invoke :write [4 8] INFO [2022-03-13 17:20:18,659] jepsen worker 54 - jepsen.util 4644 :ok :read [18 21] INFO [2022-03-13 17:20:18,660] jepsen worker 54 - jepsen.util 4644 :invoke :write [18 19] INFO [2022-03-13 17:20:18,660] jepsen worker 35 - jepsen.util 935 :invoke :write [11 10] INFO [2022-03-13 17:20:18,667] jepsen worker 41 - jepsen.util 4991 :info :write [13 4] :RequestTimeoutException INFO [2022-03-13 17:20:18,668] jepsen worker 41 - jepsen.util 5081 :invoke :write [13 4] INFO [2022-03-13 17:20:18,671] jepsen worker 18 - jepsen.util 4968 :fail :read [6 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,672] jepsen worker 18 - jepsen.util 4968 :invoke :read [6 nil] INFO [2022-03-13 17:20:18,682] jepsen worker 88 - jepsen.util 2608 :fail :write [29 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,683] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 32] INFO [2022-03-13 17:20:18,700] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,701] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,701] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,700] jepsen worker 12 - jepsen.util 912 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,701] jepsen worker 12 - jepsen.util 912 :invoke :write [4 22] INFO [2022-03-13 17:20:18,701] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,701] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:18,702] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:20:18,703] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:18,703] jepsen worker 6 - jepsen.util 636 :invoke :write [2 33] WARN [2022-03-13 17:20:18,724] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9216134,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10949681,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9616191,"encode_duration_us":19494,"operation_id":"0x15198","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10000708},{"operation_name":"upsert","last_dispatch_duration_us":17311382,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17669503,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9202068,"encode_duration_us":24550,"operation_id":"0x15033","timeout_ms":10000,"last_server_duration_us":9042674,"total_duration_us":10007172},{"operation_name":"upsert","last_dispatch_duration_us":9226003,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10032730,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9561624,"encode_duration_us":17667,"operation_id":"0x152b1","timeout_ms":10000,"last_server_duration_us":9113921,"total_duration_us":10001663}],"total_count":3}} INFO [2022-03-13 17:20:18,731] jepsen worker 5 - jepsen.util 2705 :fail :write [1 15] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,732] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:18,735] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,736] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 27] INFO [2022-03-13 17:20:18,753] jepsen worker 74 - jepsen.util 4844 :info :write [24 4] :RequestTimeoutException INFO [2022-03-13 17:20:18,754] jepsen worker 74 - jepsen.util 4934 :invoke :read [24 nil] INFO [2022-03-13 17:20:18,784] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,784] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 27] INFO [2022-03-13 17:20:18,786] jepsen worker 32 - jepsen.util 5162 :info :write [10 14] :RequestTimeoutException INFO [2022-03-13 17:20:18,788] jepsen worker 32 - jepsen.util 5252 :invoke :write [10 43] INFO [2022-03-13 17:20:18,807] jepsen worker 88 - jepsen.util 2608 :fail :write [29 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,808] jepsen worker 22 - jepsen.util 652 :fail :write [7 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,808] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,808] jepsen worker 22 - jepsen.util 652 :invoke :write [7 21] INFO [2022-03-13 17:20:18,809] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:18,809] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:18,809] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:18,809] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:18,811] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,811] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 0] INFO [2022-03-13 17:20:18,811] jepsen worker 71 - jepsen.util 611 :fail :write [23 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,812] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,812] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:18,812] jepsen worker 69 - jepsen.util 789 :invoke :write [23 10] INFO [2022-03-13 17:20:18,813] jepsen worker 20 - jepsen.util 5150 :info :write [6 38] :RequestTimeoutException INFO [2022-03-13 17:20:18,814] jepsen worker 20 - jepsen.util 5240 :invoke :read [6 nil] INFO [2022-03-13 17:20:18,815] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,816] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:18,817] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,817] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:18,818] jepsen worker 77 - jepsen.util 707 :fail :write [25 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,818] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,818] jepsen worker 77 - jepsen.util 707 :invoke :read [25 nil] INFO [2022-03-13 17:20:18,819] jepsen worker 71 - jepsen.util 611 :invoke :write [23 19] INFO [2022-03-13 17:20:18,826] jepsen worker 5 - jepsen.util 2705 :fail :write [1 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,827] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:18,828] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,829] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:18,830] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,830] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] WARN [2022-03-13 17:20:18,840] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":698,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":698,"total_server_duration_us":49,"operation_id":"0x147f4","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":30008012},{"operation_name":"get","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","operation_id":"0x1488b","timeout_ms":30000,"total_duration_us":30007722},{"operation_name":"get","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","operation_id":"0x1488c","timeout_ms":30000,"total_duration_us":30006479},{"operation_name":"upsert","last_dispatch_duration_us":9046799,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":9562714,"total_server_duration_us":9321454,"encode_duration_us":20390,"operation_id":"0x15231","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10010233},{"operation_name":"upsert","last_dispatch_duration_us":454358,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10019186,"total_server_duration_us":215391,"encode_duration_us":38915,"operation_id":"0x15660","timeout_ms":10000,"last_server_duration_us":45502,"total_duration_us":10008905},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":23091,"operation_id":"0x154fb","timeout_ms":10000,"total_duration_us":10007508},{"operation_name":"upsert","last_dispatch_duration_us":29647,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","total_dispatch_duration_us":1833797,"total_server_duration_us":366009,"encode_duration_us":47974,"operation_id":"0x15100","timeout_ms":10000,"last_server_duration_us":18876,"total_duration_us":10004523},{"operation_name":"upsert","last_dispatch_duration_us":420915,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10009033,"total_server_duration_us":167968,"encode_duration_us":24384,"operation_id":"0x15664","timeout_ms":10000,"last_server_duration_us":24583,"total_duration_us":10004112},{"operation_name":"upsert","last_dispatch_duration_us":416795,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":10071431,"total_server_duration_us":205748,"encode_duration_us":23491,"operation_id":"0x1566c","timeout_ms":10000,"last_server_duration_us":33230,"total_duration_us":10002405},{"operation_name":"upsert","last_dispatch_duration_us":9232984,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9232984,"total_server_duration_us":9039489,"encode_duration_us":21582,"operation_id":"0x15671","timeout_ms":10000,"last_server_duration_us":9039489,"total_duration_us":9276208}],"total_count":59}} WARN [2022-03-13 17:20:18,859] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":40129,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":3267766,"total_server_duration_us":750557,"encode_duration_us":21495,"operation_id":"0x15676","timeout_ms":10000,"last_server_duration_us":24239,"total_duration_us":10008540},{"operation_name":"upsert","last_dispatch_duration_us":179480,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9900685,"total_server_duration_us":201338,"encode_duration_us":31675,"operation_id":"0x15518","timeout_ms":10000,"last_server_duration_us":21731,"total_duration_us":10007246},{"operation_name":"upsert","last_dispatch_duration_us":119908,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9888131,"total_server_duration_us":222429,"encode_duration_us":42864,"operation_id":"0x1552c","timeout_ms":10000,"last_server_duration_us":20203,"total_duration_us":10003510},{"operation_name":"upsert","last_dispatch_duration_us":137025,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9876219,"total_server_duration_us":240519,"encode_duration_us":22445,"operation_id":"0x15517","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10001109},{"operation_name":"upsert","last_dispatch_duration_us":9146152,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":9365626,"total_server_duration_us":9086481,"encode_duration_us":30301,"operation_id":"0x1551a","timeout_ms":10000,"last_server_duration_us":9044797,"total_duration_us":9392423},{"operation_name":"upsert","last_dispatch_duration_us":9106196,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9276044,"total_server_duration_us":9084190,"encode_duration_us":20564,"operation_id":"0x1515f","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":9297457},{"operation_name":"get","last_dispatch_duration_us":9065994,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9065994,"total_server_duration_us":451697,"operation_id":"0x151ca","timeout_ms":30000,"last_server_duration_us":451697,"total_duration_us":9066376},{"operation_name":"get","last_dispatch_duration_us":8239676,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8239676,"total_server_duration_us":250965,"operation_id":"0x1523a","timeout_ms":30000,"last_server_duration_us":250965,"total_duration_us":8239978},{"operation_name":"upsert","last_dispatch_duration_us":6462026,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":6462026,"total_server_duration_us":45950,"encode_duration_us":25742,"operation_id":"0x15811","timeout_ms":10000,"last_server_duration_us":45950,"total_duration_us":6466400},{"operation_name":"get","last_dispatch_duration_us":4114583,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000005/00000000B6F00358","last_local_socket":"172.23.104.79:37438","total_dispatch_duration_us":4114583,"total_server_duration_us":21163,"operation_id":"0x15988","timeout_ms":30000,"last_server_duration_us":21163,"total_duration_us":4114886}],"total_count":17}} INFO [2022-03-13 17:20:18,890] jepsen worker 31 - jepsen.util 5161 :info :write [10 3] :RequestTimeoutException INFO [2022-03-13 17:20:18,891] jepsen worker 31 - jepsen.util 5251 :invoke :read [10 nil] INFO [2022-03-13 17:20:18,898] jepsen worker 87 - jepsen.util 2877 :fail :write [29 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,899] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 8] INFO [2022-03-13 17:20:18,923] jepsen worker 53 - jepsen.util 593 :fail :write [17 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,923] jepsen worker 8 - jepsen.util 728 :fail :write [2 18] :ServerOutOfMemoryException WARN [2022-03-13 17:20:18,924] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":37863,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000002/00000000E1F76995","last_local_socket":"172.23.104.79:33490","total_dispatch_duration_us":3440503,"total_server_duration_us":15203,"operation_id":"0x148dd","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":30003021},{"operation_name":"get","last_dispatch_duration_us":10383240,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":10383240,"total_server_duration_us":49,"operation_id":"0x1549f","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":10385824},{"operation_name":"upsert","last_dispatch_duration_us":9058893,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10975744,"total_server_duration_us":628270,"encode_duration_us":19536,"operation_id":"0x1519b","timeout_ms":10000,"last_server_duration_us":31298,"total_duration_us":10010472},{"operation_name":"upsert","last_dispatch_duration_us":8817219,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10592822,"total_server_duration_us":650909,"encode_duration_us":19843,"operation_id":"0x15209","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":10006888},{"operation_name":"upsert","last_dispatch_duration_us":9194286,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10809251,"total_server_duration_us":9658241,"encode_duration_us":25789,"operation_id":"0x15219","timeout_ms":10000,"last_server_duration_us":9107531,"total_duration_us":10004472},{"operation_name":"upsert","last_dispatch_duration_us":34138,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":8048743,"total_server_duration_us":337484,"encode_duration_us":19239,"operation_id":"0x156cd","timeout_ms":10000,"last_server_duration_us":23475,"total_duration_us":10003123},{"operation_name":"upsert","last_dispatch_duration_us":9226003,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10032730,"total_server_duration_us":9561624,"encode_duration_us":17667,"operation_id":"0x152b1","timeout_ms":10000,"last_server_duration_us":9113921,"total_duration_us":10001663},{"operation_name":"upsert","last_dispatch_duration_us":9216134,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":10949681,"total_server_duration_us":9616191,"encode_duration_us":19494,"operation_id":"0x15198","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10000708},{"operation_name":"upsert","last_dispatch_duration_us":2585452,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2585452,"total_server_duration_us":62925,"encode_duration_us":29257,"operation_id":"0x158ad","timeout_ms":10000,"last_server_duration_us":62925,"total_duration_us":2594745},{"operation_name":"get","last_dispatch_duration_us":2465879,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2465879,"total_server_duration_us":38,"operation_id":"0x158be","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":2466179}],"total_count":44}} INFO [2022-03-13 17:20:18,924] jepsen worker 8 - jepsen.util 728 :invoke :write [2 26] WARN [2022-03-13 17:20:18,925] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17644009,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":26652861,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9080876,"encode_duration_us":22729,"operation_id":"0x14b9d","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10001701},{"operation_name":"upsert","last_dispatch_duration_us":17833180,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":27484667,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9140484,"encode_duration_us":32791,"operation_id":"0x14b97","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10002570},{"operation_name":"upsert","last_dispatch_duration_us":17769722,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17769722,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9070294,"encode_duration_us":18110,"operation_id":"0x14bd3","timeout_ms":10000,"last_server_duration_us":9070294,"total_duration_us":10003937},{"operation_name":"upsert","last_dispatch_duration_us":9114458,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15944552,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9146112,"encode_duration_us":17740,"operation_id":"0x15154","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10006921}],"total_count":4}} INFO [2022-03-13 17:20:18,926] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] WARN [2022-03-13 17:20:18,938] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13896589,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":13896589,"total_server_duration_us":38,"operation_id":"0x14e63","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":13899226},{"operation_name":"get","last_dispatch_duration_us":13460751,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":13460751,"total_server_duration_us":19,"operation_id":"0x14eb3","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":13462467},{"operation_name":"upsert","last_dispatch_duration_us":16302200,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16302200,"total_server_duration_us":9053293,"encode_duration_us":19627,"operation_id":"0x1524a","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10009066},{"operation_name":"upsert","last_dispatch_duration_us":9339761,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":18018204,"total_server_duration_us":115360,"encode_duration_us":24309,"operation_id":"0x1516b","timeout_ms":10000,"last_server_duration_us":87506,"total_duration_us":10007101},{"operation_name":"upsert","last_dispatch_duration_us":9485056,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9485056,"total_server_duration_us":63053,"encode_duration_us":35799,"operation_id":"0x15685","timeout_ms":10000,"last_server_duration_us":63053,"total_duration_us":10006034},{"operation_name":"upsert","last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","encode_duration_us":34698,"operation_id":"0x156dc","timeout_ms":10000,"total_duration_us":10004738},{"operation_name":"upsert","last_dispatch_duration_us":29278,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":7128812,"total_server_duration_us":369042,"encode_duration_us":22063,"operation_id":"0x15312","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10002957},{"operation_name":"upsert","last_dispatch_duration_us":2438132,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2438132,"total_server_duration_us":99970,"encode_duration_us":21778,"operation_id":"0x158c5","timeout_ms":10000,"last_server_duration_us":99970,"total_duration_us":2442003},{"operation_name":"upsert","last_dispatch_duration_us":2377087,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2377087,"total_server_duration_us":1860679,"encode_duration_us":18200,"operation_id":"0x1589c","timeout_ms":10000,"last_server_duration_us":1860679,"total_duration_us":2380919},{"operation_name":"upsert","last_dispatch_duration_us":2362894,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2362894,"total_server_duration_us":51710,"encode_duration_us":20159,"operation_id":"0x158b1","timeout_ms":10000,"last_server_duration_us":51710,"total_duration_us":2366720}],"total_count":18}} WARN [2022-03-13 17:20:18,941] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9046799,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9562714,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9321454,"encode_duration_us":20390,"operation_id":"0x15231","timeout_ms":10000,"last_server_duration_us":9034183,"total_duration_us":10010233}],"total_count":1}} INFO [2022-03-13 17:20:18,945] jepsen worker 88 - jepsen.util 2608 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,947] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 7] INFO [2022-03-13 17:20:18,953] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,954] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:18,956] jepsen worker 69 - jepsen.util 789 :fail :write [23 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,957] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:18,959] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,961] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 26] INFO [2022-03-13 17:20:18,961] jepsen worker 63 - jepsen.util 5193 :ok :read [21 :nil] INFO [2022-03-13 17:20:18,962] jepsen worker 63 - jepsen.util 5193 :invoke :read [21 nil] INFO [2022-03-13 17:20:18,971] jepsen worker 75 - jepsen.util 525 :fail :write [25 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:18,972] jepsen worker 75 - jepsen.util 525 :invoke :write [25 21] INFO [2022-03-13 17:20:19,011] jepsen worker 87 - jepsen.util 2877 :fail :write [29 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:19,012] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 38] WARN [2022-03-13 17:20:19,038] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16302200,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16302200,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9053293,"encode_duration_us":19627,"operation_id":"0x1524a","timeout_ms":10000,"last_server_duration_us":9053293,"total_duration_us":10009066},{"operation_name":"upsert","last_dispatch_duration_us":16415477,"last_local_id":"A312307A00000006/000000006E7DCCBA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":16415477,"last_local_socket":"172.23.104.79:51150","total_server_duration_us":9045859,"encode_duration_us":18399,"operation_id":"0x14d58","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10009524}],"total_count":2}} INFO [2022-03-13 17:20:19,045] jepsen worker 18 - jepsen.util 4968 :ok :read [6 16] INFO [2022-03-13 17:20:19,045] jepsen worker 18 - jepsen.util 4968 :invoke :read [6 nil] INFO [2022-03-13 17:20:19,050] jepsen worker 10 - jepsen.util 5320 :ok :read [3 25] INFO [2022-03-13 17:20:19,051] jepsen worker 10 - jepsen.util 5320 :invoke :write [3 13] INFO [2022-03-13 17:20:19,069] jepsen worker 88 - jepsen.util 2608 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:19,070] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 35] INFO [2022-03-13 17:20:19,086] jepsen worker 68 - jepsen.util 4928 :info :write [22 4] :RequestTimeoutException INFO [2022-03-13 17:20:19,087] jepsen worker 68 - jepsen.util 5018 :invoke :write [22 13] INFO [2022-03-13 17:20:19,088] jepsen worker 5 - jepsen.util 2705 :fail :write [1 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:19,088] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:19,089] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:19,090] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 8] INFO [2022-03-13 17:20:19,092] jepsen worker 63 - jepsen.util 5193 :ok :read [21 :nil] INFO [2022-03-13 17:20:19,094] jepsen worker 63 - jepsen.util 5193 :invoke :read [21 nil] INFO [2022-03-13 17:20:19,121] jepsen worker 18 - jepsen.util 4968 :ok :read [6 16] INFO [2022-03-13 17:20:19,123] jepsen worker 18 - jepsen.util 4968 :invoke :write [6 32] INFO [2022-03-13 17:20:19,123] jepsen worker 87 - jepsen.util 2877 :fail :write [29 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:19,124] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 13] WARN [2022-03-13 17:20:19,126] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":141404,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9490809,"total_server_duration_us":321678,"encode_duration_us":26263,"operation_id":"0x156e6","timeout_ms":10000,"last_server_duration_us":35312,"total_duration_us":10009397},{"operation_name":"upsert","last_dispatch_duration_us":47783,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":595909,"total_server_duration_us":212333,"encode_duration_us":19831,"operation_id":"0x15588","timeout_ms":10000,"last_server_duration_us":31393,"total_duration_us":10008988},{"operation_name":"upsert","last_dispatch_duration_us":9114458,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":15944552,"total_server_duration_us":9146112,"encode_duration_us":17740,"operation_id":"0x15154","timeout_ms":10000,"last_server_duration_us":9064980,"total_duration_us":10006921},{"operation_name":"upsert","last_dispatch_duration_us":7892055,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8356599,"total_server_duration_us":93969,"encode_duration_us":20928,"operation_id":"0x1519a","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10006537},{"operation_name":"upsert","last_dispatch_duration_us":7857171,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8688439,"total_server_duration_us":240737,"encode_duration_us":33418,"operation_id":"0x156d9","timeout_ms":10000,"last_server_duration_us":28493,"total_duration_us":10005386},{"operation_name":"upsert","last_dispatch_duration_us":216816,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":10005928,"total_server_duration_us":318604,"encode_duration_us":18711,"operation_id":"0x150c5","timeout_ms":10000,"last_server_duration_us":80968,"total_duration_us":10004773},{"operation_name":"get","last_dispatch_duration_us":7843822,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7843822,"total_server_duration_us":8339,"operation_id":"0x15773","timeout_ms":30000,"last_server_duration_us":8339,"total_duration_us":7894892},{"operation_name":"get","last_dispatch_duration_us":7779102,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":7779102,"total_server_duration_us":76,"operation_id":"0x1564d","timeout_ms":30000,"last_server_duration_us":76,"total_duration_us":7781083},{"operation_name":"get","last_dispatch_duration_us":7621559,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":7621559,"total_server_duration_us":535152,"operation_id":"0x157d0","timeout_ms":30000,"last_server_duration_us":535152,"total_duration_us":7627705},{"operation_name":"upsert","last_dispatch_duration_us":2676669,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2676669,"total_server_duration_us":57756,"encode_duration_us":18621,"operation_id":"0x158c8","timeout_ms":10000,"last_server_duration_us":57756,"total_duration_us":2781339}],"total_count":87}} INFO [2022-03-13 17:20:19,146] jepsen worker 59 - jepsen.util 4829 :info :write [19 5] :RequestTimeoutException INFO [2022-03-13 17:20:19,147] jepsen worker 59 - jepsen.util 4919 :invoke :write [19 9] INFO [2022-03-13 17:20:19,153] jepsen worker 63 - jepsen.util 5193 :ok :read [21 :nil] INFO [2022-03-13 17:20:19,154] jepsen worker 63 - jepsen.util 5193 :invoke :write [21 8] INFO [2022-03-13 17:20:19,516] jepsen worker 84 - jepsen.util 5394 :info :write [28 45] :RequestTimeoutException INFO [2022-03-13 17:20:19,517] jepsen worker 84 - jepsen.util 5484 :invoke :write [28 43] INFO [2022-03-13 17:20:19,572] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:19,572] jepsen worker nemesis - couchbase.util Rebalance check count: 725 INFO [2022-03-13 17:20:19,572] jepsen worker nemesis - couchbase.util Rebalance stuck count: 597 INFO [2022-03-13 17:20:19,686] jepsen worker 65 - jepsen.util 5195 :info :write [21 4] :RequestTimeoutException INFO [2022-03-13 17:20:19,687] jepsen worker 65 - jepsen.util 5285 :invoke :write [21 26] INFO [2022-03-13 17:20:19,856] jepsen worker 9 - jepsen.util 4419 :info :write [3 32] :RequestTimeoutException INFO [2022-03-13 17:20:19,857] jepsen worker 9 - jepsen.util 4509 :invoke :read [3 nil] INFO [2022-03-13 17:20:19,916] jepsen worker 67 - jepsen.util 4387 :info :write [22 30] :RequestTimeoutException INFO [2022-03-13 17:20:19,917] jepsen worker 67 - jepsen.util 4477 :invoke :read [22 nil] INFO [2022-03-13 17:20:20,031] jepsen worker 28 - jepsen.util 5068 :info :write [9 38] :RequestTimeoutException INFO [2022-03-13 17:20:20,031] jepsen worker 28 - jepsen.util 5158 :invoke :write [9 41] INFO [2022-03-13 17:20:20,384] jepsen worker 78 - jepsen.util 708 :fail :write [26 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,384] jepsen worker 23 - jepsen.util 563 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,384] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,384] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,384] jepsen worker 77 - jepsen.util 707 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,385] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:20,386] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:20,386] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,387] jepsen worker 77 - jepsen.util 707 :invoke :write [25 23] INFO [2022-03-13 17:20:20,387] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:20,388] jepsen worker 23 - jepsen.util 563 :invoke :write [7 4] INFO [2022-03-13 17:20:20,389] jepsen worker 78 - jepsen.util 708 :invoke :write [26 0] INFO [2022-03-13 17:20:20,396] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,396] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,397] jepsen worker 61 - jepsen.util 781 :invoke :write [20 40] INFO [2022-03-13 17:20:20,398] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:20,422] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,423] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:20,453] jepsen worker 52 - jepsen.util 682 :fail :write [17 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,453] jepsen worker 52 - jepsen.util 682 :invoke :write [17 27] INFO [2022-03-13 17:20:20,476] jepsen worker 71 - jepsen.util 611 :fail :write [23 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,476] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,477] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:20,477] jepsen worker 71 - jepsen.util 611 :invoke :write [23 35] INFO [2022-03-13 17:20:20,492] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,493] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:20:20,497] jepsen worker 34 - jepsen.util 934 :fail :write [11 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,497] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,498] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,498] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,498] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:20,499] jepsen worker 21 - jepsen.util 831 :invoke :write [7 47] INFO [2022-03-13 17:20:20,500] jepsen worker 34 - jepsen.util 934 :invoke :write [11 1] INFO [2022-03-13 17:20:20,506] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:20,541] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:20,541] jepsen worker 62 - jepsen.util 512 :fail :write [20 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,541] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:20,543] jepsen worker 62 - jepsen.util 512 :invoke :read [20 nil] INFO [2022-03-13 17:20:20,543] jepsen worker 44 - jepsen.util 764 :invoke :write [14 21] INFO [2022-03-13 17:20:20,544] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,544] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,547] jepsen worker 70 - jepsen.util 880 :invoke :write [23 7] INFO [2022-03-13 17:20:20,548] jepsen worker 42 - jepsen.util 402 :invoke :read [14 nil] INFO [2022-03-13 17:20:20,548] jepsen worker 13 - jepsen.util 733 :fail :write [4 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,549] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:20,549] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:20,558] jepsen worker 30 - jepsen.util 4890 :info :write [10 32] :RequestTimeoutException INFO [2022-03-13 17:20:20,559] jepsen worker 30 - jepsen.util 4980 :invoke :write [10 13] INFO [2022-03-13 17:20:20,577] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:20,577] jepsen worker nemesis - couchbase.util Rebalance check count: 726 INFO [2022-03-13 17:20:20,577] jepsen worker nemesis - couchbase.util Rebalance stuck count: 598 INFO [2022-03-13 17:20:20,628] jepsen worker 25 - jepsen.util 565 :fail :write [8 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,628] jepsen worker 25 - jepsen.util 565 :invoke :write [8 42] INFO [2022-03-13 17:20:20,691] jepsen worker 71 - jepsen.util 611 :fail :write [23 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,692] jepsen worker 71 - jepsen.util 611 :invoke :write [23 5] INFO [2022-03-13 17:20:20,707] jepsen worker 23 - jepsen.util 563 :fail :write [7 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,708] jepsen worker 23 - jepsen.util 563 :invoke :write [7 34] INFO [2022-03-13 17:20:20,759] jepsen worker 76 - jepsen.util 796 :fail :write [25 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,759] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,759] jepsen worker 60 - jepsen.util 690 :fail :write [20 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,759] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:20,759] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:20,760] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:20,760] jepsen worker 60 - jepsen.util 690 :invoke :read [20 nil] INFO [2022-03-13 17:20:20,761] jepsen worker 43 - jepsen.util 673 :invoke :read [14 nil] INFO [2022-03-13 17:20:20,769] jepsen worker 80 - jepsen.util 800 :fail :write [26 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,769] jepsen worker 80 - jepsen.util 800 :invoke :read [26 nil] INFO [2022-03-13 17:20:20,790] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,791] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:20:20,866] jepsen worker 12 - jepsen.util 912 :fail :write [4 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,867] jepsen worker 12 - jepsen.util 912 :invoke :write [4 0] INFO [2022-03-13 17:20:20,922] jepsen worker 14 - jepsen.util 374 :fail :write [4 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,923] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:20,933] jepsen worker 5 - jepsen.util 2705 :fail :write [1 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,933] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:20:20,979] jepsen worker 88 - jepsen.util 2608 :fail :write [29 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,980] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 16] INFO [2022-03-13 17:20:20,989] jepsen worker 87 - jepsen.util 2877 :fail :write [29 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:20,990] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 28] INFO [2022-03-13 17:20:21,026] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,027] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,029] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,030] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,031] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,031] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,032] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,033] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 35] INFO [2022-03-13 17:20:21,035] jepsen worker 71 - jepsen.util 611 :fail :write [23 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,036] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:21,056] jepsen worker 75 - jepsen.util 525 :fail :write [25 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,056] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,057] jepsen worker 22 - jepsen.util 652 :fail :write [7 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,057] jepsen worker 53 - jepsen.util 593 :invoke :read [17 nil] INFO [2022-03-13 17:20:21,058] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:21,058] jepsen worker 22 - jepsen.util 652 :invoke :write [7 43] INFO [2022-03-13 17:20:21,059] jepsen worker 77 - jepsen.util 707 :fail :write [25 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,059] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,059] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,060] jepsen worker 13 - jepsen.util 733 :invoke :write [4 21] INFO [2022-03-13 17:20:21,063] jepsen worker 77 - jepsen.util 707 :invoke :write [25 5] INFO [2022-03-13 17:20:21,067] jepsen worker 76 - jepsen.util 796 :invoke :read [25 nil] INFO [2022-03-13 17:20:21,068] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,068] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,068] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,069] jepsen worker 71 - jepsen.util 611 :invoke :write [23 47] INFO [2022-03-13 17:20:21,069] jepsen worker 69 - jepsen.util 789 :invoke :write [23 32] INFO [2022-03-13 17:20:21,071] jepsen worker 75 - jepsen.util 525 :invoke :write [25 7] INFO [2022-03-13 17:20:21,073] jepsen worker 87 - jepsen.util 2877 :fail :write [29 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,074] jepsen worker 76 - jepsen.util 796 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,074] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 22] INFO [2022-03-13 17:20:21,074] jepsen worker 76 - jepsen.util 796 :invoke :write [25 8] INFO [2022-03-13 17:20:21,079] jepsen worker 88 - jepsen.util 2608 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,080] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,082] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,082] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 16] INFO [2022-03-13 17:20:21,107] jepsen worker 5 - jepsen.util 2705 :fail :write [1 35] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,110] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 38] INFO [2022-03-13 17:20:21,110] jepsen worker 37 - jepsen.util 2467 :info :write [12 4] :RequestTimeoutException INFO [2022-03-13 17:20:21,110] jepsen worker 37 - jepsen.util 2557 :invoke :write [12 47] INFO [2022-03-13 17:20:21,116] jepsen worker 29 - jepsen.util 5249 :info :write [9 5] :RequestTimeoutException INFO [2022-03-13 17:20:21,117] jepsen worker 29 - jepsen.util 5339 :invoke :read [9 nil] INFO [2022-03-13 17:20:21,140] jepsen worker 87 - jepsen.util 2877 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,141] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,142] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,142] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:20:21,167] jepsen worker 5 - jepsen.util 2705 :fail :write [1 38] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,168] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:20:21,192] jepsen worker 33 - jepsen.util 753 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,193] jepsen worker 33 - jepsen.util 753 :invoke :write [11 12] INFO [2022-03-13 17:20:21,197] jepsen worker 88 - jepsen.util 2608 :fail :write [29 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,198] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 29] INFO [2022-03-13 17:20:21,206] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,207] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,208] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,208] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:20:21,234] jepsen worker 35 - jepsen.util 935 :fail :write [11 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,234] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,235] jepsen worker 35 - jepsen.util 935 :invoke :read [11 nil] INFO [2022-03-13 17:20:21,235] jepsen worker 24 - jepsen.util 654 :invoke :write [8 6] INFO [2022-03-13 17:20:21,239] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,240] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,241] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,242] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 40] INFO [2022-03-13 17:20:21,269] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,270] jepsen worker 52 - jepsen.util 682 :fail :write [17 27] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,270] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,270] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,270] jepsen worker 43 - jepsen.util 673 :ok :read [14 :nil] INFO [2022-03-13 17:20:21,270] jepsen worker 60 - jepsen.util 690 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,270] jepsen worker 6 - jepsen.util 636 :fail :write [2 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,270] jepsen worker 53 - jepsen.util 593 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,271] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,271] jepsen worker 60 - jepsen.util 690 :invoke :write [20 0] INFO [2022-03-13 17:20:21,272] jepsen worker 6 - jepsen.util 636 :invoke :read [2 nil] INFO [2022-03-13 17:20:21,273] jepsen worker 26 - jepsen.util 566 :invoke :read [8 nil] INFO [2022-03-13 17:20:21,273] jepsen worker 43 - jepsen.util 673 :invoke :write [14 13] INFO [2022-03-13 17:20:21,274] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 41] INFO [2022-03-13 17:20:21,274] jepsen worker 52 - jepsen.util 682 :invoke :write [17 8] INFO [2022-03-13 17:20:21,275] jepsen worker 53 - jepsen.util 593 :invoke :write [17 37] INFO [2022-03-13 17:20:21,290] jepsen worker 88 - jepsen.util 2608 :fail :write [29 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,291] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,297] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,297] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 30] INFO [2022-03-13 17:20:21,302] jepsen worker 5 - jepsen.util 2705 :fail :write [1 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,303] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,304] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,306] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 0] INFO [2022-03-13 17:20:21,333] jepsen worker 87 - jepsen.util 2877 :fail :write [29 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,334] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,335] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,336] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 0] INFO [2022-03-13 17:20:21,356] jepsen worker 71 - jepsen.util 611 :fail :write [23 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,356] jepsen worker 71 - jepsen.util 611 :invoke :read [23 nil] INFO [2022-03-13 17:20:21,357] jepsen worker 34 - jepsen.util 934 :fail :write [11 1] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,358] jepsen worker 34 - jepsen.util 934 :invoke :write [11 16] INFO [2022-03-13 17:20:21,360] jepsen worker 5 - jepsen.util 2705 :fail :write [1 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,360] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 33] INFO [2022-03-13 17:20:21,390] jepsen worker 88 - jepsen.util 2608 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,390] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 21] INFO [2022-03-13 17:20:21,408] jepsen worker 22 - jepsen.util 652 :fail :write [7 43] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,408] jepsen worker 26 - jepsen.util 566 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,409] jepsen worker 22 - jepsen.util 652 :invoke :read [7 nil] INFO [2022-03-13 17:20:21,409] jepsen worker 26 - jepsen.util 566 :invoke :write [8 22] INFO [2022-03-13 17:20:21,415] jepsen worker 87 - jepsen.util 2877 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,416] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 11] INFO [2022-03-13 17:20:21,456] jepsen worker 5 - jepsen.util 2705 :fail :write [1 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,456] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,457] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,458] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 3] INFO [2022-03-13 17:20:21,475] jepsen worker 88 - jepsen.util 2608 :fail :write [29 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,476] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,479] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,479] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,482] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,483] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 7] INFO [2022-03-13 17:20:21,501] jepsen worker 87 - jepsen.util 2877 :fail :write [29 11] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,502] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 2] INFO [2022-03-13 17:20:21,532] jepsen worker 5 - jepsen.util 2705 :fail :write [1 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,533] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 10] INFO [2022-03-13 17:20:21,538] jepsen worker 25 - jepsen.util 565 :fail :write [8 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,538] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,538] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,539] jepsen worker 80 - jepsen.util 800 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,539] jepsen worker 71 - jepsen.util 611 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,539] jepsen worker 69 - jepsen.util 789 :fail :write [23 32] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,539] jepsen worker 25 - jepsen.util 565 :invoke :write [8 37] INFO [2022-03-13 17:20:21,543] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:21,544] jepsen worker 71 - jepsen.util 611 :invoke :write [23 49] INFO [2022-03-13 17:20:21,545] jepsen worker 69 - jepsen.util 789 :invoke :write [23 3] INFO [2022-03-13 17:20:21,546] jepsen worker 51 - jepsen.util 771 :invoke :read [17 nil] INFO [2022-03-13 17:20:21,552] jepsen worker 13 - jepsen.util 733 :fail :write [4 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,552] jepsen worker 13 - jepsen.util 733 :invoke :read [4 nil] INFO [2022-03-13 17:20:21,554] jepsen worker 80 - jepsen.util 800 :invoke :write [26 8] INFO [2022-03-13 17:20:21,554] jepsen worker 78 - jepsen.util 708 :fail :write [26 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,555] jepsen worker 78 - jepsen.util 708 :invoke :write [26 17] INFO [2022-03-13 17:20:21,571] jepsen worker 62 - jepsen.util 512 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,571] jepsen worker 8 - jepsen.util 728 :fail :write [2 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,572] jepsen worker 42 - jepsen.util 402 :ok :read [14 :nil] INFO [2022-03-13 17:20:21,572] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,572] jepsen worker 62 - jepsen.util 512 :invoke :write [20 17] INFO [2022-03-13 17:20:21,572] jepsen worker 42 - jepsen.util 402 :invoke :write [14 49] INFO [2022-03-13 17:20:21,573] jepsen worker 7 - jepsen.util 637 :invoke :read [2 nil] INFO [2022-03-13 17:20:21,573] jepsen worker 8 - jepsen.util 728 :invoke :write [2 8] INFO [2022-03-13 17:20:21,580] jepsen worker 88 - jepsen.util 2608 :fail :write [29 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,583] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,584] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:21,584] jepsen worker nemesis - couchbase.util Rebalance check count: 727 INFO [2022-03-13 17:20:21,584] jepsen worker nemesis - couchbase.util Rebalance stuck count: 599 INFO [2022-03-13 17:20:21,589] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,590] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,592] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,593] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 2] INFO [2022-03-13 17:20:21,604] jepsen worker 87 - jepsen.util 2877 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,605] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 12] INFO [2022-03-13 17:20:21,638] jepsen worker 5 - jepsen.util 2705 :fail :write [1 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,639] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 47] INFO [2022-03-13 17:20:21,672] jepsen worker 87 - jepsen.util 2877 :fail :write [29 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,672] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:20:21,695] jepsen worker 88 - jepsen.util 2608 :fail :write [29 2] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,696] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,700] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,700] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 33] INFO [2022-03-13 17:20:21,712] jepsen worker 5 - jepsen.util 2705 :fail :write [1 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,713] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,714] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,714] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:21,732] jepsen worker 21 - jepsen.util 831 :fail :write [7 47] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,732] jepsen worker 21 - jepsen.util 831 :invoke :read [7 nil] INFO [2022-03-13 17:20:21,779] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,780] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 49] INFO [2022-03-13 17:20:21,781] jepsen worker 81 - jepsen.util 5661 :ok :read [27 4] INFO [2022-03-13 17:20:21,781] jepsen worker 81 - jepsen.util 5661 :invoke :write [27 29] INFO [2022-03-13 17:20:21,788] jepsen worker 39 - jepsen.util 5439 :ok :read [13 39] INFO [2022-03-13 17:20:21,789] jepsen worker 39 - jepsen.util 5439 :invoke :read [13 nil] INFO [2022-03-13 17:20:21,800] jepsen worker 61 - jepsen.util 781 :fail :write [20 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,800] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,801] jepsen worker 79 - jepsen.util 1159 :invoke :read [26 nil] INFO [2022-03-13 17:20:21,802] jepsen worker 71 - jepsen.util 611 :fail :write [23 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,802] jepsen worker 61 - jepsen.util 781 :invoke :read [20 nil] INFO [2022-03-13 17:20:21,803] jepsen worker 71 - jepsen.util 611 :invoke :write [23 26] INFO [2022-03-13 17:20:21,810] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,811] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 29] INFO [2022-03-13 17:20:21,827] jepsen worker 33 - jepsen.util 753 :fail :write [11 12] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,827] jepsen worker 35 - jepsen.util 935 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,828] jepsen worker 33 - jepsen.util 753 :invoke :read [11 nil] INFO [2022-03-13 17:20:21,829] jepsen worker 35 - jepsen.util 935 :invoke :write [11 17] INFO [2022-03-13 17:20:21,855] jepsen worker 88 - jepsen.util 2608 :fail :write [29 33] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,855] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,857] jepsen worker 87 - jepsen.util 2877 :fail :write [29 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,857] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,858] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:20:21,858] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 23] INFO [2022-03-13 17:20:21,887] jepsen worker 82 - jepsen.util 5392 :ok :read [27 4] INFO [2022-03-13 17:20:21,887] jepsen worker 5 - jepsen.util 2705 :fail :write [1 29] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,888] jepsen worker 82 - jepsen.util 5392 :invoke :write [27 29] INFO [2022-03-13 17:20:21,888] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 6] INFO [2022-03-13 17:20:21,920] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,921] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,922] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,923] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 25] INFO [2022-03-13 17:20:21,949] jepsen worker 5 - jepsen.util 2705 :fail :write [1 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,949] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:21,950] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,951] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 16] INFO [2022-03-13 17:20:21,956] jepsen worker 39 - jepsen.util 5439 :ok :read [13 39] INFO [2022-03-13 17:20:21,956] jepsen worker 39 - jepsen.util 5439 :invoke :read [13 nil] INFO [2022-03-13 17:20:21,964] jepsen worker 88 - jepsen.util 2608 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,964] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 0] INFO [2022-03-13 17:20:21,967] jepsen worker 76 - jepsen.util 796 :fail :write [25 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,967] jepsen worker 60 - jepsen.util 690 :fail :write [20 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,968] jepsen worker 60 - jepsen.util 690 :invoke :write [20 13] INFO [2022-03-13 17:20:21,968] jepsen worker 76 - jepsen.util 796 :invoke :write [25 44] INFO [2022-03-13 17:20:21,981] jepsen worker 87 - jepsen.util 2877 :fail :write [29 25] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,982] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,983] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,984] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,986] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,987] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:21,989] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,989] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 18] INFO [2022-03-13 17:20:21,998] jepsen worker 44 - jepsen.util 764 :fail :write [14 21] :ServerOutOfMemoryException INFO [2022-03-13 17:20:21,999] jepsen worker 44 - jepsen.util 764 :invoke :read [14 nil] INFO [2022-03-13 17:20:22,002] jepsen worker 69 - jepsen.util 789 :fail :write [23 3] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,003] jepsen worker 69 - jepsen.util 789 :invoke :write [23 44] INFO [2022-03-13 17:20:22,014] jepsen worker 5 - jepsen.util 2705 :fail :write [1 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,014] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:22,055] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,055] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 34] INFO [2022-03-13 17:20:22,091] jepsen worker 87 - jepsen.util 2877 :fail :write [29 18] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,092] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 10] INFO [2022-03-13 17:20:22,124] jepsen worker 39 - jepsen.util 5439 :ok :read [13 39] INFO [2022-03-13 17:20:22,125] jepsen worker 39 - jepsen.util 5439 :invoke :read [13 nil] INFO [2022-03-13 17:20:22,128] jepsen worker 5 - jepsen.util 2705 :fail :write [1 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,128] jepsen worker 88 - jepsen.util 2608 :fail :write [29 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,128] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 45] INFO [2022-03-13 17:20:22,128] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,171] jepsen worker 24 - jepsen.util 654 :fail :write [8 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,171] jepsen worker 24 - jepsen.util 654 :invoke :read [8 nil] INFO [2022-03-13 17:20:22,172] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,172] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 39] INFO [2022-03-13 17:20:22,188] jepsen worker 87 - jepsen.util 2877 :fail :write [29 10] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,188] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,190] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,191] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 19] INFO [2022-03-13 17:20:22,209] jepsen worker 23 - jepsen.util 563 :fail :write [7 34] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,210] jepsen worker 23 - jepsen.util 563 :invoke :write [7 7] INFO [2022-03-13 17:20:22,212] jepsen worker 71 - jepsen.util 611 :fail :write [23 26] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,213] jepsen worker 71 - jepsen.util 611 :invoke :write [23 40] INFO [2022-03-13 17:20:22,232] jepsen worker 5 - jepsen.util 2705 :fail :write [1 45] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,238] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:22,240] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,241] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 5] INFO [2022-03-13 17:20:22,263] jepsen worker 87 - jepsen.util 2877 :fail :write [29 19] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,264] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 39] INFO [2022-03-13 17:20:22,272] jepsen worker 88 - jepsen.util 2608 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,273] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,275] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,276] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,277] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,277] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,278] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,279] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 22] INFO [2022-03-13 17:20:22,302] jepsen worker 5 - jepsen.util 2705 :fail :write [1 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,303] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 42] INFO [2022-03-13 17:20:22,324] jepsen worker 39 - jepsen.util 5439 :ok :read [13 39] INFO [2022-03-13 17:20:22,324] jepsen worker 39 - jepsen.util 5439 :invoke :read [13 nil] INFO [2022-03-13 17:20:22,345] jepsen worker 87 - jepsen.util 2877 :fail :write [29 39] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,346] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 23] INFO [2022-03-13 17:20:22,354] jepsen worker 88 - jepsen.util 2608 :fail :write [29 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,354] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,372] jepsen worker 51 - jepsen.util 771 :fail :read [17 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,373] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,372] jepsen worker 34 - jepsen.util 934 :fail :write [11 16] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,373] jepsen worker 51 - jepsen.util 771 :invoke :write [17 6] INFO [2022-03-13 17:20:22,374] jepsen worker 34 - jepsen.util 934 :invoke :write [11 28] INFO [2022-03-13 17:20:22,380] jepsen worker 14 - jepsen.util 374 :invoke :read [4 nil] INFO [2022-03-13 17:20:22,383] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,384] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,385] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,385] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 48] INFO [2022-03-13 17:20:22,411] jepsen worker 5 - jepsen.util 2705 :fail :write [1 42] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,411] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:22,426] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,427] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:22,428] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,428] jepsen worker 25 - jepsen.util 565 :fail :write [8 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,429] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:22,429] jepsen worker 25 - jepsen.util 565 :invoke :write [8 4] INFO [2022-03-13 17:20:22,431] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,436] jepsen worker 5 - jepsen.util 2705 :invoke :read [1 nil] INFO [2022-03-13 17:20:22,453] jepsen worker 39 - jepsen.util 5439 :ok :read [13 39] INFO [2022-03-13 17:20:22,454] jepsen worker 39 - jepsen.util 5439 :invoke :read [13 nil] INFO [2022-03-13 17:20:22,458] jepsen worker 88 - jepsen.util 2608 :fail :write [29 48] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,458] jepsen worker 88 - jepsen.util 2608 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,494] jepsen worker 88 - jepsen.util 2608 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,495] jepsen worker 88 - jepsen.util 2608 :invoke :write [29 37] INFO [2022-03-13 17:20:22,497] jepsen worker 87 - jepsen.util 2877 :fail :write [29 23] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,497] jepsen worker 5 - jepsen.util 2705 :fail :read [1 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,497] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,498] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 41] INFO [2022-03-13 17:20:22,503] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,504] jepsen worker 87 - jepsen.util 2877 :invoke :write [29 30] INFO [2022-03-13 17:20:22,558] jepsen worker 5 - jepsen.util 2705 :fail :write [1 41] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,559] jepsen worker 5 - jepsen.util 2705 :invoke :write [1 46] INFO [2022-03-13 17:20:22,571] jepsen worker 70 - jepsen.util 880 :fail :write [23 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,572] jepsen worker 70 - jepsen.util 880 :invoke :read [23 nil] INFO [2022-03-13 17:20:22,578] jepsen worker 69 - jepsen.util 789 :fail :write [23 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,579] jepsen worker 69 - jepsen.util 789 :invoke :read [23 nil] INFO [2022-03-13 17:20:22,584] jepsen worker 52 - jepsen.util 682 :fail :write [17 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,584] jepsen worker 75 - jepsen.util 525 :fail :write [25 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,584] jepsen worker 13 - jepsen.util 733 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,585] jepsen worker 87 - jepsen.util 2877 :fail :write [29 30] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,585] jepsen worker 13 - jepsen.util 733 :invoke :write [4 6] INFO [2022-03-13 17:20:22,586] jepsen worker 52 - jepsen.util 682 :invoke :write [17 40] INFO [2022-03-13 17:20:22,586] jepsen worker 75 - jepsen.util 525 :invoke :read [25 nil] INFO [2022-03-13 17:20:22,587] jepsen worker 87 - jepsen.util 2877 :invoke :read [29 nil] INFO [2022-03-13 17:20:22,590] jepsen worker nemesis - couchbase.util Rebalance status: {:status running, :ns_1@172.23.106.61 {:progress 0.3695014662756598}, :ns_1@172.23.105.187 {:progress 0.8888888888888888}, :ns_1@172.23.107.227 {:progress 0.3695014662756598}, :ns_1@172.23.107.228 {:progress 0.6529411764705882}} INFO [2022-03-13 17:20:22,590] jepsen worker nemesis - couchbase.util Rebalance check count: 728 INFO [2022-03-13 17:20:22,590] jepsen worker nemesis - couchbase.util Rebalance stuck count: 600 INFO [2022-03-13 17:20:22,590] jepsen worker nemesis - couchbase.util Rebalance stuck WARN [2022-03-13 17:20:22,597] jepsen worker nemesis - jepsen.generator.interpreter Process :nemesis crashed java.lang.RuntimeException: Rebalance stuck at couchbase.util$wait_for_rebalance_complete.invokeStatic(util.clj:218) at couchbase.util$wait_for_rebalance_complete.invoke(util.clj:204) at couchbase.util$wait_for_rebalance_complete.invokeStatic(util.clj:205) at couchbase.util$wait_for_rebalance_complete.invoke(util.clj:204) at couchbase.util$rebalance.invokeStatic(util.clj:253) at couchbase.util$rebalance.invoke(util.clj:227) at couchbase.nemesis$swap_rebalance$fn__3895.invoke(nemesis.clj:237) at couchbase.nemesis$swap_rebalance.invokeStatic(nemesis.clj:235) at couchbase.nemesis$swap_rebalance.invoke(nemesis.clj:227) at couchbase.nemesis$couchbase$reify__3984.invoke_BANG_(nemesis.clj:410) at jepsen.generator.interpreter.NemesisWorker.invoke_BANG_(interpreter.clj:74) at jepsen.generator.interpreter$spawn_worker$fn__9837$fn__9838.invoke(interpreter.clj:137) at jepsen.generator.interpreter$spawn_worker$fn__9837.invoke(interpreter.clj:120) at clojure.core$binding_conveyor_fn$fn__5772.invoke(core.clj:2034) at clojure.lang.AFn.call(AFn.java:18) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) INFO [2022-03-13 17:20:22,601] jepsen worker nemesis - jepsen.generator.interpreter Aborting test due to nemesis failure. INFO [2022-03-13 17:20:22,610] jepsen worker 39 - jepsen.util 5439 :ok :read [13 39] INFO [2022-03-13 17:20:22,615] jepsen worker 87 - jepsen.util 2877 :fail :read [29 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,615] jepsen worker 88 - jepsen.util 2608 :fail :write [29 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,649] jepsen worker 21 - jepsen.util 831 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,649] jepsen worker 80 - jepsen.util 800 :fail :write [26 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,649] jepsen worker 33 - jepsen.util 753 :fail :read [11 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,655] jepsen worker 77 - jepsen.util 707 :fail :write [25 5] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,659] jepsen worker 69 - jepsen.util 789 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,659] jepsen worker 71 - jepsen.util 611 :fail :write [23 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,685] jepsen worker 12 - jepsen.util 912 :fail :write [4 0] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,685] jepsen worker 6 - jepsen.util 636 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,686] jepsen worker 7 - jepsen.util 637 :fail :read [2 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,731] jepsen worker 57 - jepsen.util 4917 :info :write [19 36] :RequestTimeoutException INFO [2022-03-13 17:20:22,816] jepsen worker 24 - jepsen.util 654 :fail :read [8 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,816] jepsen worker 35 - jepsen.util 935 :fail :write [11 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,816] jepsen worker 14 - jepsen.util 374 :fail :read [4 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,819] jepsen worker 79 - jepsen.util 1159 :fail :read [26 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,819] jepsen worker 78 - jepsen.util 708 :fail :write [26 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,819] jepsen worker 61 - jepsen.util 781 :fail :read [20 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:22,820] jepsen worker 86 - jepsen.util 4676 :info :write [28 39] :RequestTimeoutException INFO [2022-03-13 17:20:23,021] jepsen worker 75 - jepsen.util 525 :fail :read [25 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,021] jepsen worker 43 - jepsen.util 673 :fail :write [14 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,021] jepsen worker 76 - jepsen.util 796 :fail :write [25 44] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,054] jepsen worker 51 - jepsen.util 771 :fail :write [17 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,055] jepsen worker 62 - jepsen.util 512 :fail :write [20 17] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,419] jepsen worker 23 - jepsen.util 563 :fail :write [7 7] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,442] jepsen worker 22 - jepsen.util 652 :fail :read [7 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,442] jepsen worker 53 - jepsen.util 593 :fail :write [17 37] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,444] jepsen worker 8 - jepsen.util 728 :fail :write [2 8] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,529] jepsen worker 17 - jepsen.util 4877 :info :write [5 43] :RequestTimeoutException INFO [2022-03-13 17:20:23,534] jepsen worker 5 - jepsen.util 2705 :fail :write [1 46] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,623] jepsen worker 34 - jepsen.util 934 :fail :write [11 28] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,633] jepsen worker 13 - jepsen.util 733 :fail :write [4 6] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,656] jepsen worker 25 - jepsen.util 565 :fail :write [8 4] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,711] jepsen worker 15 - jepsen.util 4605 :fail :read [5 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:23,835] jepsen worker 26 - jepsen.util 566 :fail :write [8 22] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,841] jepsen worker 42 - jepsen.util 402 :fail :write [14 49] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,841] jepsen worker 44 - jepsen.util 764 :ok :read [14 :nil] INFO [2022-03-13 17:20:23,843] jepsen worker 70 - jepsen.util 880 :fail :read [23 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:23,867] jepsen worker 60 - jepsen.util 690 :fail :write [20 13] :ServerOutOfMemoryException INFO [2022-03-13 17:20:24,061] jepsen worker 52 - jepsen.util 682 :fail :write [17 40] :ServerOutOfMemoryException INFO [2022-03-13 17:20:24,599] jepsen worker 89 - jepsen.util 2789 :info :write [29 39] :RequestTimeoutException INFO [2022-03-13 17:20:24,753] jepsen worker 83 - jepsen.util 4673 :info :write [27 17] :SyncWriteAmbiguous INFO [2022-03-13 17:20:24,877] jepsen worker 40 - jepsen.util 5260 :ok :read [13 39] INFO [2022-03-13 17:20:24,880] jepsen worker 67 - jepsen.util 4477 :ok :read [22 30] INFO [2022-03-13 17:20:24,918] jepsen worker 85 - jepsen.util 4765 :ok :read [28 44] INFO [2022-03-13 17:20:25,737] jepsen worker 49 - jepsen.util 3649 :fail :read [16 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:25,917] jepsen worker 0 - jepsen.util 4860 :info :write [0 8] :RequestTimeoutException INFO [2022-03-13 17:20:25,924] jepsen worker 58 - jepsen.util 4468 :info :write [19 12] :RequestTimeoutException INFO [2022-03-13 17:20:26,140] jepsen worker 2 - jepsen.util 5312 :info :write [0 14] :RequestTimeoutException INFO [2022-03-13 17:20:26,167] jepsen worker 1 - jepsen.util 4771 :info :write [0 29] :RequestTimeoutException INFO [2022-03-13 17:20:26,715] jepsen worker 66 - jepsen.util 4836 :info :write [22 42] :RequestTimeoutException INFO [2022-03-13 17:20:27,180] jepsen worker 55 - jepsen.util 4555 :ok :read [18 21] INFO [2022-03-13 17:20:27,196] jepsen worker 72 - jepsen.util 5022 :ok :read [24 24] INFO [2022-03-13 17:20:27,224] jepsen worker 31 - jepsen.util 5251 :ok :read [10 :nil] INFO [2022-03-13 17:20:27,239] jepsen worker 73 - jepsen.util 5383 :ok :read [24 24] INFO [2022-03-13 17:20:27,246] jepsen worker 45 - jepsen.util 5175 :info :write [15 48] :SyncWriteAmbiguous INFO [2022-03-13 17:20:27,320] jepsen worker 20 - jepsen.util 5240 :ok :read [6 16] WARN [2022-03-13 17:20:27,343] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18051311,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":26267091,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9039810,"encode_duration_us":20481,"operation_id":"0x14e8e","timeout_ms":10000,"last_server_duration_us":9020392,"total_duration_us":10008075}],"total_count":1}} WARN [2022-03-13 17:20:27,343] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":13143043,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":13143043,"total_server_duration_us":32,"operation_id":"0x156c9","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":13145040},{"operation_name":"get","last_dispatch_duration_us":10401250,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10401250,"total_server_duration_us":15,"operation_id":"0x1586a","timeout_ms":30000,"last_server_duration_us":15,"total_duration_us":10403194},{"operation_name":"upsert","last_dispatch_duration_us":9200301,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":11865350,"total_server_duration_us":9835010,"encode_duration_us":34992,"operation_id":"0x156a7","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10010657},{"operation_name":"upsert","last_dispatch_duration_us":142529,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9179730,"total_server_duration_us":60900,"encode_duration_us":31092,"operation_id":"0x15920","timeout_ms":10000,"last_server_duration_us":26685,"total_duration_us":10008588},{"operation_name":"upsert","last_dispatch_duration_us":126021,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2805049,"total_server_duration_us":985512,"encode_duration_us":18510,"operation_id":"0x15698","timeout_ms":10000,"last_server_duration_us":107592,"total_duration_us":10006306},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000001/00000000E3CF0611","last_local_socket":"172.23.104.79:33508","encode_duration_us":19771,"operation_id":"0x1584d","timeout_ms":10000,"total_duration_us":10005444},{"operation_name":"upsert","last_dispatch_duration_us":35309,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2776208,"total_server_duration_us":890192,"encode_duration_us":26807,"operation_id":"0x156ef","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10004498},{"operation_name":"upsert","last_dispatch_duration_us":105215,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":9922692,"total_server_duration_us":143951,"encode_duration_us":16748,"operation_id":"0x15914","timeout_ms":10000,"last_server_duration_us":21405,"total_duration_us":10002947},{"operation_name":"upsert","last_dispatch_duration_us":101225,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":2192798,"total_server_duration_us":814130,"encode_duration_us":29998,"operation_id":"0x157a1","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10002055},{"operation_name":"get","last_dispatch_duration_us":6046291,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":6046291,"total_server_duration_us":56,"operation_id":"0x15aca","timeout_ms":30000,"last_server_duration_us":56,"total_duration_us":6048513}],"total_count":57}} INFO [2022-03-13 17:20:27,373] jepsen worker 9 - jepsen.util 4509 :ok :read [3 25] INFO [2022-03-13 17:20:27,491] jepsen worker 29 - jepsen.util 5339 :fail :read [9 nil] :ServerOutOfMemoryException INFO [2022-03-13 17:20:27,727] jepsen worker 56 - jepsen.util 4736 :info :write [18 11] :SyncWriteAmbiguous INFO [2022-03-13 17:20:27,840] jepsen worker 74 - jepsen.util 4934 :ok :read [24 24] INFO [2022-03-13 17:20:28,050] jepsen worker 46 - jepsen.util 5266 :info :write [15 7] :RequestTimeoutException INFO [2022-03-13 17:20:28,116] jepsen worker 3 - jepsen.util 2253 :info :write [1 16] :RequestTimeoutException INFO [2022-03-13 17:20:28,229] jepsen worker 59 - jepsen.util 4919 :info :write [19 9] :SyncWriteAmbiguous INFO [2022-03-13 17:20:28,290] jepsen worker 19 - jepsen.util 4789 :info :write [6 33] :RequestTimeoutException INFO [2022-03-13 17:20:28,314] jepsen worker 27 - jepsen.util 5157 :info :write [9 29] :RequestTimeoutException INFO [2022-03-13 17:20:28,336] jepsen worker 63 - jepsen.util 5193 :info :write [21 8] :SyncWriteAmbiguous INFO [2022-03-13 17:20:28,374] jepsen worker 47 - jepsen.util 4907 :info :write [15 28] :RequestTimeoutException INFO [2022-03-13 17:20:28,410] jepsen worker 11 - jepsen.util 4781 :info :write [3 22] :RequestTimeoutException INFO [2022-03-13 17:20:28,446] jepsen worker 64 - jepsen.util 4924 :info :write [21 5] :RequestTimeoutException INFO [2022-03-13 17:20:28,684] jepsen worker 54 - jepsen.util 4644 :info :write [18 19] :RequestTimeoutException INFO [2022-03-13 17:20:28,688] jepsen worker 41 - jepsen.util 5081 :info :write [13 4] :RequestTimeoutException WARN [2022-03-13 17:20:28,737] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9194286,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10809251,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9658241,"encode_duration_us":25789,"operation_id":"0x15219","timeout_ms":10000,"last_server_duration_us":9107531,"total_duration_us":10004472}],"total_count":1}} INFO [2022-03-13 17:20:28,814] jepsen worker 32 - jepsen.util 5252 :info :write [10 43] :RequestTimeoutException WARN [2022-03-13 17:20:28,852] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":6810224,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":6810224,"total_server_duration_us":43,"operation_id":"0x14e12","timeout_ms":30000,"last_server_duration_us":43,"total_duration_us":30007732},{"operation_name":"upsert","last_dispatch_duration_us":33232,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9793001,"total_server_duration_us":233406,"encode_duration_us":19029,"operation_id":"0x15bdf","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10010182},{"operation_name":"upsert","last_dispatch_duration_us":8701478,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","total_dispatch_duration_us":8701478,"total_server_duration_us":20680,"encode_duration_us":18014,"operation_id":"0x1594d","timeout_ms":10000,"last_server_duration_us":20680,"total_duration_us":10009850},{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":28711,"operation_id":"0x15aed","timeout_ms":10000,"total_duration_us":10008433},{"operation_name":"upsert","last_dispatch_duration_us":31260,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9928967,"total_server_duration_us":193744,"encode_duration_us":21178,"operation_id":"0x15bc0","timeout_ms":10000,"last_server_duration_us":19262,"total_duration_us":10005458},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","encode_duration_us":20031,"operation_id":"0x159fc","timeout_ms":10000,"total_duration_us":10004129},{"operation_name":"upsert","last_dispatch_duration_us":9221324,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9858500,"total_server_duration_us":9220136,"encode_duration_us":25026,"operation_id":"0x15bb3","timeout_ms":10000,"last_server_duration_us":9083054,"total_duration_us":9927320},{"operation_name":"upsert","last_dispatch_duration_us":9432247,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9432247,"total_server_duration_us":9073483,"encode_duration_us":20093,"operation_id":"0x15bb4","timeout_ms":10000,"last_server_duration_us":9073483,"total_duration_us":9436374},{"operation_name":"get","last_dispatch_duration_us":9069696,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000003/00000000340BCE27","last_local_socket":"172.23.104.79:43002","total_dispatch_duration_us":9069696,"total_server_duration_us":108,"operation_id":"0x15c42","timeout_ms":30000,"last_server_duration_us":108,"total_duration_us":9071860},{"operation_name":"upsert","last_dispatch_duration_us":2403654,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000003/00000000590563BE","last_local_socket":"172.23.104.79:37454","total_dispatch_duration_us":2403654,"total_server_duration_us":173716,"encode_duration_us":17779,"operation_id":"0x15d16","timeout_ms":10000,"last_server_duration_us":173716,"total_duration_us":2407709}],"total_count":31}} WARN [2022-03-13 17:20:28,871] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":1848760,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":7248599,"total_server_duration_us":6322,"operation_id":"0x14ed2","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":30007974},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","encode_duration_us":17882,"operation_id":"0x15bc6","timeout_ms":10000,"total_duration_us":10010862},{"operation_name":"upsert","last_dispatch_duration_us":9114780,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10353166,"total_server_duration_us":9697859,"encode_duration_us":18593,"operation_id":"0x1584e","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10010341},{"operation_name":"upsert","last_dispatch_duration_us":9109508,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":11137657,"total_server_duration_us":9687782,"encode_duration_us":22927,"operation_id":"0x1578c","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":10009977},{"operation_name":"upsert","last_dispatch_duration_us":8161233,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":10383165,"total_server_duration_us":858654,"encode_duration_us":20142,"operation_id":"0x15772","timeout_ms":10000,"last_server_duration_us":32355,"total_duration_us":10007660},{"operation_name":"upsert","last_dispatch_duration_us":38544,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":918056,"total_server_duration_us":562447,"encode_duration_us":23765,"operation_id":"0x15b4d","timeout_ms":10000,"last_server_duration_us":23813,"total_duration_us":10005916},{"operation_name":"upsert","last_dispatch_duration_us":29029,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":9169063,"total_server_duration_us":312992,"encode_duration_us":17989,"operation_id":"0x15bf3","timeout_ms":10000,"last_server_duration_us":19887,"total_duration_us":10005749},{"operation_name":"upsert","last_dispatch_duration_us":31631,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000005/00000000322F1886","last_local_socket":"172.23.104.79:51120","total_dispatch_duration_us":7130198,"total_server_duration_us":399958,"encode_duration_us":21472,"operation_id":"0x1574d","timeout_ms":10000,"last_server_duration_us":19574,"total_duration_us":10005081},{"operation_name":"get","last_dispatch_duration_us":8489705,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":8489705,"total_server_duration_us":15059,"operation_id":"0x15c4f","timeout_ms":30000,"last_server_duration_us":15059,"total_duration_us":8491956},{"operation_name":"get","last_dispatch_duration_us":7497493,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000005/000000004E3205AD","last_local_socket":"172.23.104.79:43000","total_dispatch_duration_us":7497493,"total_server_duration_us":8502,"operation_id":"0x15c96","timeout_ms":30000,"last_server_duration_us":8502,"total_duration_us":7499438}],"total_count":15}} WARN [2022-03-13 17:20:28,872] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9109508,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11137657,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9687782,"encode_duration_us":22927,"operation_id":"0x1578c","timeout_ms":10000,"last_server_duration_us":9059667,"total_duration_us":10009977},{"operation_name":"upsert","last_dispatch_duration_us":9114780,"last_local_id":"A312307A00000005/000000004E3205AD","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":10353166,"last_local_socket":"172.23.104.79:43000","total_server_duration_us":9697859,"encode_duration_us":18593,"operation_id":"0x1584e","timeout_ms":10000,"last_server_duration_us":9091564,"total_duration_us":10010341}],"total_count":2}} WARN [2022-03-13 17:20:28,937] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":8755056,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9677710,"total_server_duration_us":386468,"encode_duration_us":19499,"operation_id":"0x15ad3","timeout_ms":10000,"last_server_duration_us":24069,"total_duration_us":10008818},{"operation_name":"upsert","last_dispatch_duration_us":8871970,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":17753617,"total_server_duration_us":128940,"encode_duration_us":50034,"operation_id":"0x157df","timeout_ms":10000,"last_server_duration_us":18953,"total_duration_us":10008095},{"operation_name":"upsert","last_dispatch_duration_us":8642203,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9429483,"total_server_duration_us":342664,"encode_duration_us":20501,"operation_id":"0x15af6","timeout_ms":10000,"last_server_duration_us":19185,"total_duration_us":10001986},{"operation_name":"upsert","last_dispatch_duration_us":29972,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":410460,"total_server_duration_us":252009,"encode_duration_us":17580,"operation_id":"0x15c37","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10001345},{"operation_name":"upsert","last_dispatch_duration_us":9159390,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":9159390,"total_server_duration_us":9045859,"encode_duration_us":16571,"operation_id":"0x15c6e","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":9163562},{"operation_name":"upsert","last_dispatch_duration_us":9050483,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000002/000000004309950E","last_local_socket":"172.23.104.79:51168","total_dispatch_duration_us":9050483,"total_server_duration_us":9036305,"encode_duration_us":26021,"operation_id":"0x15c6f","timeout_ms":10000,"last_server_duration_us":9036305,"total_duration_us":9054377},{"operation_name":"get","last_dispatch_duration_us":7432838,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7432838,"total_server_duration_us":94412,"operation_id":"0x1588a","timeout_ms":30000,"last_server_duration_us":94412,"total_duration_us":7433117},{"operation_name":"upsert","last_dispatch_duration_us":2609592,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2609592,"total_server_duration_us":1450815,"encode_duration_us":24043,"operation_id":"0x15baa","timeout_ms":10000,"last_server_duration_us":1450815,"total_duration_us":2613138},{"operation_name":"get","last_dispatch_duration_us":2394504,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2394504,"total_server_duration_us":38,"operation_id":"0x15bbe","timeout_ms":30000,"last_server_duration_us":38,"total_duration_us":2394735},{"operation_name":"get","last_dispatch_duration_us":2154166,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000002/0000000012D3A6D4","last_local_socket":"172.23.104.79:37400","total_dispatch_duration_us":2154166,"total_server_duration_us":19,"operation_id":"0x15bd4","timeout_ms":30000,"last_server_duration_us":19,"total_duration_us":2154305}],"total_count":25}} WARN [2022-03-13 17:20:28,942] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9682216,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18370656,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9279165,"encode_duration_us":33418,"operation_id":"0x156d9","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10005386},{"operation_name":"upsert","last_dispatch_duration_us":17142399,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":25594031,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9250676,"encode_duration_us":16809,"operation_id":"0x15086","timeout_ms":10000,"last_server_duration_us":9024634,"total_duration_us":10009032},{"operation_name":"upsert","last_dispatch_duration_us":17191743,"last_local_id":"A312307A00000004/000000008C3B62F1","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17787653,"last_local_socket":"172.23.104.79:51142","total_server_duration_us":9238028,"encode_duration_us":19831,"operation_id":"0x15588","timeout_ms":10000,"last_server_duration_us":9025695,"total_duration_us":10008988}],"total_count":3}} WARN [2022-03-13 17:20:28,950] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":16070450,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":16070450,"total_server_duration_us":6048,"operation_id":"0x1584b","timeout_ms":30000,"last_server_duration_us":6048,"total_duration_us":16072946},{"operation_name":"upsert","last_dispatch_duration_us":8840070,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9672883,"total_server_duration_us":448983,"encode_duration_us":17636,"operation_id":"0x15799","timeout_ms":10000,"last_server_duration_us":20045,"total_duration_us":10007090},{"operation_name":"upsert","last_dispatch_duration_us":8667631,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9902427,"total_server_duration_us":476663,"encode_duration_us":22047,"operation_id":"0x15715","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10001262},{"operation_name":"upsert","last_dispatch_duration_us":8879303,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9960430,"total_server_duration_us":474743,"encode_duration_us":27885,"operation_id":"0x1570c","timeout_ms":10000,"last_server_duration_us":19418,"total_duration_us":10001083},{"operation_name":"get","last_dispatch_duration_us":9626189,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":9626189,"total_server_duration_us":6432,"operation_id":"0x15b91","timeout_ms":30000,"last_server_duration_us":6432,"total_duration_us":9628983},{"operation_name":"upsert","last_dispatch_duration_us":9177058,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":9177058,"total_server_duration_us":9050106,"encode_duration_us":51403,"operation_id":"0x15a93","timeout_ms":10000,"last_server_duration_us":9050106,"total_duration_us":9188063},{"operation_name":"get","last_dispatch_duration_us":8710111,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8710111,"total_server_duration_us":49,"operation_id":"0x15c0d","timeout_ms":30000,"last_server_duration_us":49,"total_duration_us":8712083},{"operation_name":"get","last_dispatch_duration_us":8331778,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000006/0000000069FC3C82","last_local_socket":"172.23.104.79:43006","total_dispatch_duration_us":8331778,"total_server_duration_us":421,"operation_id":"0x15c5a","timeout_ms":30000,"last_server_duration_us":421,"total_duration_us":8332095},{"operation_name":"get","last_dispatch_duration_us":4946672,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000006/000000006E7DCCBA","last_local_socket":"172.23.104.79:51150","total_dispatch_duration_us":4946672,"total_server_duration_us":6727,"operation_id":"0x15c97","timeout_ms":30000,"last_server_duration_us":6727,"total_duration_us":4948810},{"operation_name":"upsert","last_dispatch_duration_us":2138698,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000006/0000000066A14478","last_local_socket":"172.23.104.79:37416","total_dispatch_duration_us":2138698,"total_server_duration_us":48334,"encode_duration_us":21546,"operation_id":"0x15c26","timeout_ms":10000,"last_server_duration_us":48334,"total_duration_us":2149751}],"total_count":20}} WARN [2022-03-13 17:20:28,953] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17121059,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18954857,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9403375,"encode_duration_us":47974,"operation_id":"0x15100","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10004523}],"total_count":1}} WARN [2022-03-13 17:20:29,050] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":18357593,"last_local_id":"A312307A00000006/0000000069FC3C82","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18357593,"last_local_socket":"172.23.104.79:43006","total_server_duration_us":9096885,"encode_duration_us":34698,"operation_id":"0x156dc","timeout_ms":10000,"last_server_duration_us":9096885,"total_duration_us":10004738}],"total_count":1}} INFO [2022-03-13 17:20:29,078] jepsen worker 10 - jepsen.util 5320 :info :write [3 13] :RequestTimeoutException INFO [2022-03-13 17:20:29,121] jepsen worker 68 - jepsen.util 5018 :info :write [22 13] :RequestTimeoutException WARN [2022-03-13 17:20:29,143] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":12052142,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":12052142,"total_server_duration_us":23,"operation_id":"0x1591f","timeout_ms":30000,"last_server_duration_us":23,"total_duration_us":12055012},{"operation_name":"upsert","last_dispatch_duration_us":8678298,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8678298,"total_server_duration_us":35613,"encode_duration_us":19396,"operation_id":"0x15c4a","timeout_ms":10000,"last_server_duration_us":35613,"total_duration_us":10006305},{"operation_name":"upsert","last_dispatch_duration_us":171326,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":8888734,"total_server_duration_us":54319,"encode_duration_us":20974,"operation_id":"0x15c34","timeout_ms":10000,"last_server_duration_us":34116,"total_duration_us":10003039},{"operation_name":"upsert","last_dispatch_duration_us":29269,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":9291226,"total_server_duration_us":233702,"encode_duration_us":34424,"operation_id":"0x15ad6","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":10002490},{"operation_name":"upsert","last_dispatch_duration_us":8701008,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9034380,"total_server_duration_us":150567,"encode_duration_us":24156,"operation_id":"0x15beb","timeout_ms":10000,"last_server_duration_us":20124,"total_duration_us":10001571},{"operation_name":"upsert","last_dispatch_duration_us":194138,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":9165160,"total_server_duration_us":148687,"encode_duration_us":18938,"operation_id":"0x15be1","timeout_ms":10000,"last_server_duration_us":27222,"total_duration_us":10001231},{"operation_name":"get","last_dispatch_duration_us":8753527,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000004/000000008C3B62F1","last_local_socket":"172.23.104.79:51142","total_dispatch_duration_us":8753527,"total_server_duration_us":69,"operation_id":"0x15af2","timeout_ms":30000,"last_server_duration_us":69,"total_duration_us":8755297},{"operation_name":"get","last_dispatch_duration_us":186979,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":186979,"total_server_duration_us":32,"operation_id":"0x15cf4","timeout_ms":30000,"last_server_duration_us":32,"total_duration_us":6372964},{"operation_name":"upsert","last_dispatch_duration_us":2636777,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2636777,"total_server_duration_us":67750,"encode_duration_us":18863,"operation_id":"0x15bbc","timeout_ms":10000,"last_server_duration_us":67750,"total_duration_us":2644868},{"operation_name":"upsert","last_dispatch_duration_us":2610558,"last_remote_socket":"172.23.107.228:11210","last_local_id":"A312307A00000004/00000000C4C33556","last_local_socket":"172.23.104.79:37436","total_dispatch_duration_us":2610558,"total_server_duration_us":19966,"encode_duration_us":30222,"operation_id":"0x15c5e","timeout_ms":10000,"last_server_duration_us":19966,"total_duration_us":2615762}],"total_count":30}} INFO [2022-03-13 17:20:29,150] jepsen worker 18 - jepsen.util 4968 :info :write [6 32] :RequestTimeoutException INFO [2022-03-13 17:20:29,540] jepsen worker 84 - jepsen.util 5484 :info :write [28 43] :RequestTimeoutException INFO [2022-03-13 17:20:29,714] jepsen worker 65 - jepsen.util 5285 :info :write [21 26] :RequestTimeoutException INFO [2022-03-13 17:20:30,060] jepsen worker 28 - jepsen.util 5158 :info :write [9 41] :RequestTimeoutException INFO [2022-03-13 17:20:30,588] jepsen worker 30 - jepsen.util 4980 :info :write [10 13] :RequestTimeoutException INFO [2022-03-13 17:20:31,130] jepsen worker 37 - jepsen.util 2557 :info :write [12 47] :RequestTimeoutException INFO [2022-03-13 17:20:31,800] jepsen worker 81 - jepsen.util 5661 :info :write [27 29] :RequestTimeoutException INFO [2022-03-13 17:20:31,911] jepsen worker 82 - jepsen.util 5392 :info :write [27 29] :RequestTimeoutException WARN [2022-03-13 17:20:37,355] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16294549,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18487347,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9859989,"encode_duration_us":29998,"operation_id":"0x157a1","timeout_ms":10000,"last_server_duration_us":9045859,"total_duration_us":10002055},{"operation_name":"upsert","last_dispatch_duration_us":16407308,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":16914732,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9206569,"encode_duration_us":20303,"operation_id":"0x15c6c","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10006549},{"operation_name":"upsert","last_dispatch_duration_us":9191997,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18371728,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9123754,"encode_duration_us":31092,"operation_id":"0x15920","timeout_ms":10000,"last_server_duration_us":9062854,"total_duration_us":10008588},{"operation_name":"upsert","last_dispatch_duration_us":9200301,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":11865350,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9835010,"encode_duration_us":34992,"operation_id":"0x156a7","timeout_ms":10000,"last_server_duration_us":9061792,"total_duration_us":10010657}],"total_count":4}} WARN [2022-03-13 17:20:37,355] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7268584,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":10996176,"total_server_duration_us":251118,"encode_duration_us":22019,"operation_id":"0x15c6a","timeout_ms":10000,"last_server_duration_us":20521,"total_duration_us":10010461},{"operation_name":"upsert","last_dispatch_duration_us":7270898,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":7270898,"total_server_duration_us":20601,"encode_duration_us":17494,"operation_id":"0x15c98","timeout_ms":10000,"last_server_duration_us":20601,"total_duration_us":10010440},{"operation_name":"upsert","last_dispatch_duration_us":16407308,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16914732,"total_server_duration_us":9206569,"encode_duration_us":20303,"operation_id":"0x15c6c","timeout_ms":10000,"last_server_duration_us":9047983,"total_duration_us":10006549},{"operation_name":"upsert","last_dispatch_duration_us":8638391,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000001/00000000D4977C97","last_local_socket":"172.23.104.79:42986","total_dispatch_duration_us":16818382,"total_server_duration_us":369251,"encode_duration_us":19894,"operation_id":"0x15bee","timeout_ms":10000,"last_server_duration_us":19108,"total_duration_us":10005410},{"operation_name":"upsert","last_dispatch_duration_us":7267252,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":8281790,"total_server_duration_us":253686,"encode_duration_us":17649,"operation_id":"0x15d66","timeout_ms":10000,"last_server_duration_us":19030,"total_duration_us":10004802},{"operation_name":"upsert","last_dispatch_duration_us":35832,"last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000001/000000001BD58C5A","last_local_socket":"172.23.104.79:51164","total_dispatch_duration_us":1059545,"total_server_duration_us":230320,"encode_duration_us":16983,"operation_id":"0x15d5b","timeout_ms":10000,"last_server_duration_us":25625,"total_duration_us":10001202}],"total_count":6}} INFO [2022-03-13 17:20:38,746] jepsen worker 50 - jepsen.util 3470 :fail :read [16 nil] :UnambiguousTimeoutException WARN [2022-03-13 17:20:38,749] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":17613895,"last_local_id":"A312307A00000002/000000004309950E","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":18024355,"last_local_socket":"172.23.104.79:51168","total_server_duration_us":9274522,"encode_duration_us":17580,"operation_id":"0x15c37","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10001345},{"operation_name":"upsert","last_dispatch_duration_us":17870973,"last_local_id":"A312307A00000002/00000000078A6C5F","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18296282,"last_local_socket":"172.23.104.79:43004","total_server_duration_us":9126416,"encode_duration_us":19988,"operation_id":"0x15c66","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10006945}],"total_count":2}} WARN [2022-03-13 17:20:38,865] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_remote_socket":"172.23.106.61:11210","last_local_id":"A312307A00000003/0000000079BC08CA","last_local_socket":"172.23.104.79:51148","encode_duration_us":18417,"operation_id":"0x15c7c","timeout_ms":10000,"total_duration_us":10004091},{"operation_name":"upsert","last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000003/00000000B2EF4A50","last_local_socket":"172.23.104.79:33492","encode_duration_us":17950,"operation_id":"0x15cf6","timeout_ms":10000,"total_duration_us":10001302}],"total_count":2}} WARN [2022-03-13 17:20:38,883] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"get","last_dispatch_duration_us":1848758,"last_remote_socket":"172.23.107.227:11210","last_local_id":"A312307A00000005/00000000F4F37B9B","last_local_socket":"172.23.104.79:33502","total_dispatch_duration_us":4322247,"total_server_duration_us":1998,"operation_id":"0x156d8","timeout_ms":30000,"last_server_duration_us":11,"total_duration_us":30000762}],"total_count":1}} WARN [2022-03-13 17:20:38,884] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9053431,"last_local_id":"A312307A00000005/00000000322F1886","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":9971488,"last_local_socket":"172.23.104.79:51120","total_server_duration_us":9599813,"encode_duration_us":23765,"operation_id":"0x15b4d","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10005916}],"total_count":1}} WARN [2022-03-13 17:20:38,950] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":7846787,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":7846787,"total_server_duration_us":20841,"encode_duration_us":17832,"operation_id":"0x15cc2","timeout_ms":10000,"last_server_duration_us":20841,"total_duration_us":10010265},{"operation_name":"upsert","last_dispatch_duration_us":17870973,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000002/00000000078A6C5F","last_local_socket":"172.23.104.79:43004","total_dispatch_duration_us":18296282,"total_server_duration_us":9126416,"encode_duration_us":19988,"operation_id":"0x15c66","timeout_ms":10000,"last_server_duration_us":9049045,"total_duration_us":10006945}],"total_count":2}} WARN [2022-03-13 17:20:38,954] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":9215783,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":18250164,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9187933,"encode_duration_us":24156,"operation_id":"0x15beb","timeout_ms":10000,"last_server_duration_us":9037366,"total_duration_us":10001571},{"operation_name":"upsert","last_dispatch_duration_us":9200466,"last_local_id":"A312307A00000004/000000002C88F08E","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":17878764,"last_local_socket":"172.23.104.79:42988","total_server_duration_us":9065551,"encode_duration_us":19396,"operation_id":"0x15c4a","timeout_ms":10000,"last_server_duration_us":9029938,"total_duration_us":10006305}],"total_count":2}} WARN [2022-03-13 17:20:38,965] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":15142506,"last_local_id":"A312307A00000003/0000000079BC08CA","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":15142506,"last_local_socket":"172.23.104.79:51148","total_server_duration_us":9041612,"encode_duration_us":28711,"operation_id":"0x15aed","timeout_ms":10000,"last_server_duration_us":9041612,"total_duration_us":10008433}],"total_count":1}} WARN [2022-03-13 17:20:39,155] cb-events - com.couchbase.tracing [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":184949,"last_remote_socket":"172.23.105.187:11210","last_local_id":"A312307A00000004/000000002C88F08E","last_local_socket":"172.23.104.79:42988","total_dispatch_duration_us":184949,"total_server_duration_us":19652,"encode_duration_us":23759,"operation_id":"0x15c8f","timeout_ms":10000,"last_server_duration_us":19652,"total_duration_us":10002245}],"total_count":1}} INFO [2022-03-13 17:20:42,856] jepsen worker 16 - jepsen.util 4696 :fail :read [5 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:44,581] jepsen worker 38 - jepsen.util 2828 :fail :read [12 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:45,699] jepsen worker 48 - jepsen.util 3558 :fail :read [16 nil] :UnambiguousTimeoutException WARN [2022-03-13 17:20:47,367] cb-events - com.couchbase.tracing [com.couchbase.tracing][OrphansRecordedEvent][10s] Orphaned requests found: {"kv":{"top_requests":[{"operation_name":"upsert","last_dispatch_duration_us":16277140,"last_local_id":"A312307A00000001/000000001BD58C5A","last_remote_socket":"172.23.106.61:11210","total_dispatch_duration_us":17336686,"last_local_socket":"172.23.104.79:51164","total_server_duration_us":9268748,"encode_duration_us":16983,"operation_id":"0x15d5b","timeout_ms":10000,"last_server_duration_us":9038428,"total_duration_us":10001202},{"operation_name":"upsert","last_dispatch_duration_us":18011590,"last_local_id":"A312307A00000001/00000000D4977C97","last_remote_socket":"172.23.105.187:11210","total_dispatch_duration_us":25282488,"last_local_socket":"172.23.104.79:42986","total_server_duration_us":9043114,"encode_duration_us":17494,"operation_id":"0x15c98","timeout_ms":10000,"last_server_duration_us":9022513,"total_duration_us":10010440}],"total_count":2}} INFO [2022-03-13 17:20:48,097] jepsen worker 36 - jepsen.util 2466 :fail :read [12 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:48,227] jepsen worker 4 - jepsen.util 2614 :fail :read [1 nil] :UnambiguousTimeoutException INFO [2022-03-13 17:20:48,296] cb-events - com.couchbase.core [com.couchbase.core][BucketClosedEvent][4509us] Closed bucket "default" {"coreId":"0xa312307a00000005"} INFO [2022-03-13 17:20:48,296] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][208us] Node disconnected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:20:48,296] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][125us] Node disconnected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:20:48,296] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][126us] Node disconnected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:20:48,296] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][167us] Node disconnected {"coreId":"0xa312307a00000005","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:20:48,297] cb-events - com.couchbase.core [com.couchbase.core][ShutdownCompletedEvent][31ms] Completed shutdown and closed all open buckets {"coreId":"0xa312307a00000005"} INFO [2022-03-13 17:20:48,361] cb-events - com.couchbase.core [com.couchbase.core][BucketClosedEvent][518us] Closed bucket "default" {"coreId":"0xa312307a00000002"} INFO [2022-03-13 17:20:48,362] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][157us] Node disconnected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:20:48,362] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][236us] Node disconnected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:20:48,362] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][138us] Node disconnected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:20:48,363] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][110us] Node disconnected {"coreId":"0xa312307a00000002","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:20:48,363] cb-events - com.couchbase.core [com.couchbase.core][ShutdownCompletedEvent][18ms] Completed shutdown and closed all open buckets {"coreId":"0xa312307a00000002"} INFO [2022-03-13 17:20:48,365] cb-events - com.couchbase.core [com.couchbase.core][BucketClosedEvent][452us] Closed bucket "default" {"coreId":"0xa312307a00000004"} INFO [2022-03-13 17:20:48,365] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][186us] Node disconnected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:20:48,365] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][197us] Node disconnected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:20:48,365] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][153us] Node disconnected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:20:48,365] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][160us] Node disconnected {"coreId":"0xa312307a00000004","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:20:48,368] cb-events - com.couchbase.core [com.couchbase.core][BucketClosedEvent][500us] Closed bucket "default" {"coreId":"0xa312307a00000001"} INFO [2022-03-13 17:20:48,368] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][140us] Node disconnected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:20:48,368] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][140us] Node disconnected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:20:48,368] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][154us] Node disconnected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:20:48,369] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][144us] Node disconnected {"coreId":"0xa312307a00000001","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:20:48,369] cb-events - com.couchbase.core [com.couchbase.core][ShutdownCompletedEvent][20ms] Completed shutdown and closed all open buckets {"coreId":"0xa312307a00000001"} INFO [2022-03-13 17:20:48,373] cb-events - com.couchbase.core [com.couchbase.core][BucketClosedEvent][273us] Closed bucket "default" {"coreId":"0xa312307a00000006"} INFO [2022-03-13 17:20:48,373] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][93us] Node disconnected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:20:48,373] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][142us] Node disconnected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:20:48,373] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][183us] Node disconnected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:20:48,374] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][104us] Node disconnected {"coreId":"0xa312307a00000006","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:20:48,374] cb-events - com.couchbase.core [com.couchbase.core][ShutdownCompletedEvent][18ms] Completed shutdown and closed all open buckets {"coreId":"0xa312307a00000006"} INFO [2022-03-13 17:20:48,378] cb-events - com.couchbase.core [com.couchbase.core][BucketClosedEvent][721us] Closed bucket "default" {"coreId":"0xa312307a00000003"} INFO [2022-03-13 17:20:48,378] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][136us] Node disconnected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.105.187"} INFO [2022-03-13 17:20:48,378] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][90us] Node disconnected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.106.61"} INFO [2022-03-13 17:20:48,378] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][130us] Node disconnected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.107.228"} INFO [2022-03-13 17:20:48,378] cb-events - com.couchbase.node [com.couchbase.node][NodeDisconnectedEvent][194us] Node disconnected {"coreId":"0xa312307a00000003","managerPort":"8091","remote":"172.23.107.227"} INFO [2022-03-13 17:20:48,378] cb-events - com.couchbase.core [com.couchbase.core][ShutdownCompletedEvent][19ms] Completed shutdown and closed all open buckets {"coreId":"0xa312307a00000003"} INFO [2022-03-13 17:20:48,466] cb-events - com.couchbase.core [com.couchbase.core][ShutdownCompletedEvent][22ms] Completed shutdown and closed all open buckets {"coreId":"0xa312307a00000004"} INFO [2022-03-13 17:20:48,576] jepsen test runner - jepsen.core Run complete, writing INFO [2022-03-13 17:20:48,580] jepsen fressian - couchbase.core Not writing fressian INFO [2022-03-13 17:20:52,428] jepsen test runner - jepsen.core Snarfing log files INFO [2022-03-13 17:20:57,844] jepsen node 172.23.106.61 - couchbase.util Running cbcollect_info on 172.23.106.61 INFO [2022-03-13 17:20:59,528] jepsen node 172.23.105.187 - couchbase.util Running cbcollect_info on 172.23.105.187 INFO [2022-03-13 17:20:59,747] jepsen node 172.23.107.227 - couchbase.util Running cbcollect_info on 172.23.107.227 INFO [2022-03-13 17:21:00,547] jepsen node 172.23.107.228 - couchbase.util Running cbcollect_info on 172.23.107.228 INFO [2022-03-13 17:27:07,335] jepsen node 172.23.106.61 - couchbase.util Creating hashtable dump on 172.23.106.61 INFO [2022-03-13 17:27:48,786] jepsen node 172.23.107.227 - couchbase.util Creating hashtable dump on 172.23.107.227 INFO [2022-03-13 17:28:05,305] jepsen node 172.23.105.187 - couchbase.util Creating hashtable dump on 172.23.105.187 INFO [2022-03-13 17:28:29,235] jepsen node 172.23.107.228 - couchbase.util Creating hashtable dump on 172.23.107.228 INFO [2022-03-13 17:31:00,128] jepsen node 172.23.106.61 - couchbase.util Collecting tcp packet capture INFO [2022-03-13 17:31:49,220] jepsen node 172.23.107.227 - couchbase.util Collecting tcp packet capture INFO [2022-03-13 17:32:07,818] jepsen node 172.23.105.187 - couchbase.util Collecting tcp packet capture INFO [2022-03-13 17:32:39,083] jepsen node 172.23.107.228 - couchbase.util Collecting tcp packet capture INFO [2022-03-13 17:37:32,571] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/systemd_journal.gz to cbcollect_info_ns_1@172.23.107.228_20220314-002118/systemd_journal.gz INFO [2022-03-13 17:37:45,255] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/users.dets to cbcollect_info_ns_1@172.23.107.228_20220314-002118/users.dets INFO [2022-03-13 17:37:46,834] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ddocs.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ddocs.log INFO [2022-03-13 17:37:48,419] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.info.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.info.log INFO [2022-03-13 17:37:50,430] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.mapreduce_errors.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.mapreduce_errors.log INFO [2022-03-13 17:37:52,104] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/couchbase.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/couchbase.log INFO [2022-03-13 17:37:54,439] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_error.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_error.log INFO [2022-03-13 17:37:55,833] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.eventing.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.eventing.log INFO [2022-03-13 17:37:57,261] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.views.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.views.log INFO [2022-03-13 17:37:58,815] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_access.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_access.log INFO [2022-03-13 17:38:00,374] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/snapshots/442/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/snapshots/442/chronicle_config_rsm.snapshot INFO [2022-03-13 17:38:01,955] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.reports.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.reports.log INFO [2022-03-13 17:38:03,573] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/cbcollect_info.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/cbcollect_info.log INFO [2022-03-13 17:38:05,016] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_cbas_debug.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_cbas_debug.log INFO [2022-03-13 17:38:06,454] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/projector_mprof.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/projector_mprof.log INFO [2022-03-13 17:38:07,930] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.indexer_stats.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.indexer_stats.log INFO [2022-03-13 17:38:09,468] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/logs/5.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/logs/5.log INFO [2022-03-13 17:38:10,952] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.babysitter.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.babysitter.log INFO [2022-03-13 17:38:12,452] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/projector_cprof.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/projector_cprof.log INFO [2022-03-13 17:38:13,907] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_debug.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_debug.log INFO [2022-03-13 17:38:15,422] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_mprof.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_mprof.log INFO [2022-03-13 17:38:16,876] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/projector_pprof.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/projector_pprof.log INFO [2022-03-13 17:38:18,441] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats.log INFO [2022-03-13 17:38:20,882] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/rebalance_report_20220314T000612.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/rebalance_report_20220314T000612.json INFO [2022-03-13 17:38:22,299] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/mdocs.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/mdocs.log INFO [2022-03-13 17:38:23,744] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log INFO [2022-03-13 17:39:15,569] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ini.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ini.log INFO [2022-03-13 17:39:17,122] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.json_rpc.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.json_rpc.log INFO [2022-03-13 17:39:18,659] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.http_access.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.http_access.log INFO [2022-03-13 17:39:20,203] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_log INFO [2022-03-13 17:39:21,732] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/snapshots/442/kv.snapshot to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/snapshots/442/kv.snapshot INFO [2022-03-13 17:39:23,189] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.stats.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.stats.log INFO [2022-03-13 17:39:24,741] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.xdcr_target.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.xdcr_target.log INFO [2022-03-13 17:39:26,280] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_warn.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_warn.log INFO [2022-03-13 17:39:27,820] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.metakv.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.metakv.log INFO [2022-03-13 17:39:29,291] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_remote_clusters.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_remote_clusters.json INFO [2022-03-13 17:39:30,960] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_info.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_info.log INFO [2022-03-13 17:39:32,450] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_trace.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_trace.json INFO [2022-03-13 17:39:33,924] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.prometheus.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.prometheus.log INFO [2022-03-13 17:39:35,550] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.http_access_internal.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.http_access_internal.log INFO [2022-03-13 17:39:36,948] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/rebalance_report_20220314T002158.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/rebalance_report_20220314T002158.json INFO [2022-03-13 17:39:38,444] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/rebalance_report_20220314T000721.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/rebalance_report_20220314T000721.json INFO [2022-03-13 17:39:39,900] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/index to cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/index INFO [2022-03-13 17:39:41,830] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_cprof.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_cprof.log INFO [2022-03-13 17:39:43,287] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_replications.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_replications.json INFO [2022-03-13 17:39:44,803] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.query.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.query.log INFO [2022-03-13 17:39:46,250] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/version to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/version INFO [2022-03-13 17:39:47,766] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.indexer.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.indexer.log INFO [2022-03-13 17:39:49,254] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/rsms/kv/incarnation to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/rsms/kv/incarnation INFO [2022-03-13 17:39:50,923] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/exported_snapshot/kv.snapshot to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/exported_snapshot/kv.snapshot INFO [2022-03-13 17:39:52,375] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/diag.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/diag.log INFO [2022-03-13 17:39:54,034] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/chunks/000001 to cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/chunks/000001 INFO [2022-03-13 17:39:55,639] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/kv_trace.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/kv_trace.json INFO [2022-03-13 17:39:57,360] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_shutdown.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_shutdown.log INFO [2022-03-13 17:39:58,956] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.error.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.error.log INFO [2022-03-13 17:40:00,450] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/local-ssl-cert.pem to cbcollect_info_ns_1@172.23.107.228_20220314-002118/local-ssl-cert.pem INFO [2022-03-13 17:40:02,122] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/tombstones to cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/tombstones INFO [2022-03-13 17:40:03,576] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/hashtable_dump.txt to hashtable_dump.txt INFO [2022-03-13 17:40:04,784] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.fts.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.fts.log INFO [2022-03-13 17:40:05,271] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/meta.json to cbcollect_info_ns_1@172.23.107.228_20220314-002118/stats_snapshot/01FY2X5RD909CHRYE6054T18KW/meta.json INFO [2022-03-13 17:40:06,708] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_pprof.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/goxdcr_pprof.log INFO [2022-03-13 17:40:08,239] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.debug.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.debug.log INFO [2022-03-13 17:40:09,247] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.query.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.query.log INFO [2022-03-13 17:40:10,078] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/master_events.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/master_events.log INFO [2022-03-13 17:40:10,744] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/rebalance_report_20220314T000721.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/rebalance_report_20220314T000721.json INFO [2022-03-13 17:40:11,546] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/logs/4.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/logs/4.log INFO [2022-03-13 17:40:12,705] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ini.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ini.log INFO [2022-03-13 17:40:13,110] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/syslog.tar.gz to cbcollect_info_ns_1@172.23.107.228_20220314-002118/syslog.tar.gz INFO [2022-03-13 17:40:14,320] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.goxdcr.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.goxdcr.log INFO [2022-03-13 17:40:15,559] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.couchdb.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.couchdb.log INFO [2022-03-13 17:40:16,570] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/couchbase.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/couchbase.log INFO [2022-03-13 17:40:17,235] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.goxdcr.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.goxdcr.log INFO [2022-03-13 17:40:18,481] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_trace.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_trace.json INFO [2022-03-13 17:40:18,695] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.projector.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.projector.log INFO [2022-03-13 17:40:20,028] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/projector_mprof.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/projector_mprof.log INFO [2022-03-13 17:40:20,300] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/exported_snapshot/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/exported_snapshot/chronicle_config_rsm.snapshot INFO [2022-03-13 17:40:21,550] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_cprof.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_cprof.log INFO [2022-03-13 17:40:21,779] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached-cert.pem to cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached-cert.pem INFO [2022-03-13 17:40:23,178] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.error.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.error.log INFO [2022-03-13 17:40:23,245] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/Couchbase-172.23.107.228.pcap.1647216345.gz to Couchbase-172.23.107.228.pcap.1647216345.gz INFO [2022-03-13 17:40:24,750] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/projector_pprof.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/projector_pprof.log INFO [2022-03-13 17:40:26,357] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/local-ssl-cert.pem to cbcollect_info_ns_1@172.23.107.227_20220314-002117/local-ssl-cert.pem INFO [2022-03-13 17:40:27,926] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.indexer_stats.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.indexer_stats.log INFO [2022-03-13 17:40:29,641] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_warn.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_warn.log INFO [2022-03-13 17:40:31,169] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_info.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_info.log INFO [2022-03-13 17:40:32,701] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_access.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_access.log INFO [2022-03-13 17:40:34,269] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.debug.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.debug.log INFO [2022-03-13 17:40:36,566] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.metakv.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.metakv.log INFO [2022-03-13 17:40:38,084] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.stats.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.stats.log INFO [2022-03-13 17:40:39,664] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/rsms/chronicle_config_rsm/incarnation to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/rsms/chronicle_config_rsm/incarnation INFO [2022-03-13 17:40:41,191] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/master_events.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/master_events.log INFO [2022-03-13 17:40:42,866] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/rsms/kv/incarnation to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/rsms/kv/incarnation INFO [2022-03-13 17:40:44,503] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.views.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.views.log INFO [2022-03-13 17:40:46,139] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_log INFO [2022-03-13 17:40:47,711] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/projector_cprof.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/projector_cprof.log INFO [2022-03-13 17:40:49,287] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.ns_couchdb.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.ns_couchdb.log INFO [2022-03-13 17:40:50,788] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/chunks/000001 to cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/chunks/000001 INFO [2022-03-13 17:40:52,345] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ddocs.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ddocs.log INFO [2022-03-13 17:40:53,867] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.json_rpc.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.json_rpc.log INFO [2022-03-13 17:40:55,400] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/snapshots/442/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/snapshots/442/chronicle_config_rsm.snapshot INFO [2022-03-13 17:40:56,895] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/memcached.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/memcached.log INFO [2022-03-13 17:41:23,238] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_error.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_error.log INFO [2022-03-13 17:41:24,845] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.reports.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.reports.log INFO [2022-03-13 17:41:26,655] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.couchdb.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.couchdb.log INFO [2022-03-13 17:41:28,175] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/version to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/version INFO [2022-03-13 17:41:29,673] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.eventing.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.eventing.log INFO [2022-03-13 17:41:31,250] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.projector.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.projector.log INFO [2022-03-13 17:41:32,883] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/logs/3.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/logs/3.log INFO [2022-03-13 17:41:34,481] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/rebalance_report_20220314T000612.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/rebalance_report_20220314T000612.json INFO [2022-03-13 17:41:35,957] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.backup_service.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.backup_service.log INFO [2022-03-13 17:41:37,513] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/Couchbase-172.23.107.227.pcap.1647216342.gz to Couchbase-172.23.107.227.pcap.1647216342.gz INFO [2022-03-13 17:42:25,767] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.json_rpc.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.json_rpc.log INFO [2022-03-13 17:42:30,894] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/.ignore to .ignore INFO [2022-03-13 17:42:31,322] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_error.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_error.log INFO [2022-03-13 17:42:32,818] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.backup_service.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.backup_service.log INFO [2022-03-13 17:42:32,841] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/couchstore_local.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/couchstore_local.log INFO [2022-03-13 17:42:34,419] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_dcpdebug.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.analytics_dcpdebug.log INFO [2022-03-13 17:42:34,455] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_mprof.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_mprof.log INFO [2022-03-13 17:42:35,928] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/couchstore_local.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/couchstore_local.log INFO [2022-03-13 17:42:35,983] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.projector.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.projector.log INFO [2022-03-13 17:42:37,454] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.ns_couchdb.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.ns_couchdb.log INFO [2022-03-13 17:42:37,612] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.couchdb.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.couchdb.log INFO [2022-03-13 17:42:39,093] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/02ab548f-a217-4202-ae2772af-a3a69ae6.dmp to cbcollect_info_ns_1@172.23.107.228_20220314-002118/02ab548f-a217-4202-ae2772af-a3a69ae6.dmp INFO [2022-03-13 17:42:39,243] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/couchbase.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/couchbase.log INFO [2022-03-13 17:42:40,768] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/rsms/chronicle_config_rsm/incarnation to cbcollect_info_ns_1@172.23.107.228_20220314-002118/chronicle/rsms/chronicle_config_rsm/incarnation INFO [2022-03-13 17:42:40,937] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/logs/4.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/logs/4.log INFO [2022-03-13 17:42:42,292] jepsen node 172.23.107.228 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.fts.log to cbcollect_info_ns_1@172.23.107.228_20220314-002118/ns_server.fts.log INFO [2022-03-13 17:42:42,517] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/tombstones to cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/tombstones INFO [2022-03-13 17:42:43,975] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.http_access_internal.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.http_access_internal.log INFO [2022-03-13 17:42:45,492] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/Couchbase-172.23.106.61.pcap.1647216344.gz to Couchbase-172.23.106.61.pcap.1647216344.gz INFO [2022-03-13 17:42:48,561] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ini.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ini.log INFO [2022-03-13 17:42:52,416] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/exported_snapshot/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/exported_snapshot/chronicle_config_rsm.snapshot INFO [2022-03-13 17:42:53,918] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/syslog.tar.gz to cbcollect_info_ns_1@172.23.105.187_20220314-002116/syslog.tar.gz INFO [2022-03-13 17:42:55,656] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.backup_service.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.backup_service.log INFO [2022-03-13 17:42:57,342] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.http_access.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.http_access.log INFO [2022-03-13 17:42:58,909] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_mprof.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_mprof.log INFO [2022-03-13 17:43:00,429] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.babysitter.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.babysitter.log INFO [2022-03-13 17:43:01,993] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/kv_trace.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/kv_trace.json INFO [2022-03-13 17:43:04,600] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/master_events.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/master_events.log INFO [2022-03-13 17:43:06,536] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/Couchbase-172.23.105.187.pcap.1647216344.gz to Couchbase-172.23.105.187.pcap.1647216344.gz INFO [2022-03-13 17:45:20,949] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.info.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.info.log INFO [2022-03-13 17:45:22,573] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_remote_clusters.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_remote_clusters.json INFO [2022-03-13 17:45:24,081] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/systemd_journal.gz to cbcollect_info_ns_1@172.23.107.227_20220314-002117/systemd_journal.gz INFO [2022-03-13 17:45:28,179] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/diag.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/diag.log INFO [2022-03-13 17:45:29,880] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/logs/4.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/logs/4.log INFO [2022-03-13 17:45:31,426] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_debug.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_debug.log INFO [2022-03-13 17:45:32,927] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/cbcollect_info.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/cbcollect_info.log INFO [2022-03-13 17:45:34,470] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.http_access_internal.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.http_access_internal.log INFO [2022-03-13 17:45:36,008] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_dcpdebug.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_dcpdebug.log INFO [2022-03-13 17:45:37,514] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/syslog.tar.gz to cbcollect_info_ns_1@172.23.107.227_20220314-002117/syslog.tar.gz INFO [2022-03-13 17:45:39,289] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/snapshots/442/kv.snapshot to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/snapshots/442/kv.snapshot INFO [2022-03-13 17:45:40,790] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/couchstore_local.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/couchstore_local.log INFO [2022-03-13 17:45:42,272] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/exported_snapshot/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/exported_snapshot/chronicle_config_rsm.snapshot INFO [2022-03-13 17:45:43,750] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/index to cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/index INFO [2022-03-13 17:45:45,271] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/memcached-cert.pem to cbcollect_info_ns_1@172.23.107.227_20220314-002117/memcached-cert.pem INFO [2022-03-13 17:45:46,781] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/hashtable_dump.txt to hashtable_dump.txt INFO [2022-03-13 17:45:48,505] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.babysitter.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.babysitter.log INFO [2022-03-13 17:45:50,060] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_replications.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_replications.json INFO [2022-03-13 17:45:51,541] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/exported_snapshot/kv.snapshot to cbcollect_info_ns_1@172.23.107.227_20220314-002117/chronicle/exported_snapshot/kv.snapshot INFO [2022-03-13 17:45:53,125] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.indexer.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.indexer.log INFO [2022-03-13 17:45:54,613] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_shutdown.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_shutdown.log INFO [2022-03-13 17:45:56,141] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.xdcr_target.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.xdcr_target.log INFO [2022-03-13 17:45:57,642] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.mapreduce_errors.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.mapreduce_errors.log INFO [2022-03-13 17:45:59,145] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/tombstones to cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/tombstones INFO [2022-03-13 17:46:00,646] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/users.dets to cbcollect_info_ns_1@172.23.107.227_20220314-002117/users.dets INFO [2022-03-13 17:46:02,210] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/.ignore to .ignore INFO [2022-03-13 17:46:03,700] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_cbas_debug.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.analytics_cbas_debug.log INFO [2022-03-13 17:46:05,229] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_mprof.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_mprof.log INFO [2022-03-13 17:46:06,726] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/mdocs.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/mdocs.log INFO [2022-03-13 17:46:08,243] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.prometheus.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.prometheus.log INFO [2022-03-13 17:46:09,846] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/meta.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats_snapshot/01FY2X534KW5D6VW2JM8Y1SN1N/meta.json INFO [2022-03-13 17:46:11,465] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_pprof.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/goxdcr_pprof.log INFO [2022-03-13 17:46:13,007] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/kv_trace.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/kv_trace.json INFO [2022-03-13 17:46:17,716] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/stats.log INFO [2022-03-13 17:46:19,408] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/rebalance_report_20220314T002158.json to cbcollect_info_ns_1@172.23.107.227_20220314-002117/rebalance_report_20220314T002158.json INFO [2022-03-13 17:46:20,909] jepsen node 172.23.107.227 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.http_access.log to cbcollect_info_ns_1@172.23.107.227_20220314-002117/ns_server.http_access.log INFO [2022-03-13 17:48:15,061] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/index to cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/index INFO [2022-03-13 17:48:16,570] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_pprof.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_pprof.log INFO [2022-03-13 17:48:18,074] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/rebalance_report_20220314T000721.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/rebalance_report_20220314T000721.json INFO [2022-03-13 17:48:19,493] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.query.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.query.log INFO [2022-03-13 17:48:20,920] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_trace.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_trace.json INFO [2022-03-13 17:48:22,280] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/systemd_journal.gz to cbcollect_info_ns_1@172.23.105.187_20220314-002116/systemd_journal.gz INFO [2022-03-13 17:48:25,299] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/rebalance_report_20220314T002158.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/rebalance_report_20220314T002158.json INFO [2022-03-13 17:48:26,048] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/memcached-cert.pem to cbcollect_info_ns_1@172.23.105.187_20220314-002116/memcached-cert.pem INFO [2022-03-13 17:48:26,837] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.eventing.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.eventing.log INFO [2022-03-13 17:48:27,459] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.fts.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.fts.log INFO [2022-03-13 17:48:28,238] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_info.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_info.log INFO [2022-03-13 17:48:28,947] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.eventing.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.eventing.log INFO [2022-03-13 17:48:29,543] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.babysitter.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.babysitter.log INFO [2022-03-13 17:48:30,286] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_cprof.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_cprof.log INFO [2022-03-13 17:48:31,018] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats.log INFO [2022-03-13 17:48:31,668] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.json_rpc.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.json_rpc.log INFO [2022-03-13 17:48:32,527] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/meta.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/meta.json INFO [2022-03-13 17:48:33,050] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.projector.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.projector.log INFO [2022-03-13 17:48:33,879] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/memcached.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/memcached.log INFO [2022-03-13 17:48:34,447] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/projector_mprof.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/projector_mprof.log INFO [2022-03-13 17:48:35,912] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.metakv.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.metakv.log INFO [2022-03-13 17:48:36,147] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/kv_trace.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/kv_trace.json INFO [2022-03-13 17:48:37,393] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_replications.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_replications.json INFO [2022-03-13 17:48:38,812] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_warn.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_warn.log INFO [2022-03-13 17:48:39,687] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_cprof.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_cprof.log INFO [2022-03-13 17:48:40,363] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/snapshots/442/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/snapshots/442/chronicle_config_rsm.snapshot INFO [2022-03-13 17:48:41,165] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.reports.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.reports.log INFO [2022-03-13 17:48:41,743] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/projector_pprof.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/projector_pprof.log INFO [2022-03-13 17:48:42,651] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/logs/5.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/logs/5.log INFO [2022-03-13 17:48:43,117] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.goxdcr.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.goxdcr.log INFO [2022-03-13 17:48:44,120] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.mapreduce_errors.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.mapreduce_errors.log INFO [2022-03-13 17:48:44,595] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/version to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/version INFO [2022-03-13 17:48:45,556] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_shutdown.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_shutdown.log INFO [2022-03-13 17:48:45,987] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_debug.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_debug.log INFO [2022-03-13 17:48:46,886] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.views.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.views.log INFO [2022-03-13 17:48:47,437] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/exported_snapshot/kv.snapshot to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/exported_snapshot/kv.snapshot INFO [2022-03-13 17:48:48,337] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_trace.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_trace.json INFO [2022-03-13 17:48:48,885] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/rsms/chronicle_config_rsm/incarnation to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/rsms/chronicle_config_rsm/incarnation INFO [2022-03-13 17:48:49,789] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_access.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_access.log INFO [2022-03-13 17:48:50,420] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_error.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_error.log INFO [2022-03-13 17:48:51,326] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.query.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.query.log INFO [2022-03-13 17:48:51,859] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_log INFO [2022-03-13 17:48:52,659] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.backup_service.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.backup_service.log INFO [2022-03-13 17:48:53,228] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/rebalance_report_20220314T002158.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/rebalance_report_20220314T002158.json INFO [2022-03-13 17:48:54,002] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/projector_pprof.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/projector_pprof.log INFO [2022-03-13 17:48:54,589] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/mdocs.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/mdocs.log INFO [2022-03-13 17:48:55,452] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/memcached-cert.pem to cbcollect_info_ns_1@172.23.106.61_20220314-002116/memcached-cert.pem INFO [2022-03-13 17:48:55,979] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/chunks/000001 to cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/chunks/000001 INFO [2022-03-13 17:48:56,804] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.http_access.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.http_access.log INFO [2022-03-13 17:48:57,349] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.error.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.error.log INFO [2022-03-13 17:48:58,171] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/chunks/000001 to cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/chunks/000001 INFO [2022-03-13 17:48:58,709] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.stats.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.stats.log INFO [2022-03-13 17:48:59,521] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/cbcollect_info.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/cbcollect_info.log INFO [2022-03-13 17:49:00,129] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.prometheus.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.prometheus.log INFO [2022-03-13 17:49:00,859] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/index to cbcollect_info_ns_1@172.23.106.61_20220314-002116/stats_snapshot/01FY2X4CJWS8KT22HPA7GCTV77/index INFO [2022-03-13 17:49:01,455] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/cbcollect_info.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/cbcollect_info.log INFO [2022-03-13 17:49:02,212] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_dcpdebug.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_dcpdebug.log INFO [2022-03-13 17:49:02,834] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_remote_clusters.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/goxdcr_remote_clusters.json INFO [2022-03-13 17:49:03,546] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/rsms/kv/incarnation to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/rsms/kv/incarnation INFO [2022-03-13 17:49:04,147] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/memcached.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/memcached.log INFO [2022-03-13 17:49:04,888] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/rebalance_report_20220314T000612.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/rebalance_report_20220314T000612.json INFO [2022-03-13 17:49:06,275] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/projector_mprof.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/projector_mprof.log INFO [2022-03-13 17:49:07,683] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/users.dets to cbcollect_info_ns_1@172.23.106.61_20220314-002116/users.dets INFO [2022-03-13 17:49:08,439] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_dcpdebug.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_dcpdebug.log INFO [2022-03-13 17:49:09,056] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.stats.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.stats.log INFO [2022-03-13 17:49:09,779] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.indexer.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.indexer.log INFO [2022-03-13 17:49:10,411] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.prometheus.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.prometheus.log INFO [2022-03-13 17:49:11,127] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/diag.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/diag.log INFO [2022-03-13 17:49:11,732] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.metakv.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.metakv.log INFO [2022-03-13 17:49:12,649] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.couchdb.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.couchdb.log INFO [2022-03-13 17:49:13,142] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_remote_clusters.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_remote_clusters.json INFO [2022-03-13 17:49:14,035] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.xdcr_target.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.xdcr_target.log INFO [2022-03-13 17:49:14,460] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_replications.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_replications.json INFO [2022-03-13 17:49:15,407] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/hashtable_dump.txt to hashtable_dump.txt INFO [2022-03-13 17:49:15,799] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/local-ssl-cert.pem to cbcollect_info_ns_1@172.23.106.61_20220314-002116/local-ssl-cert.pem INFO [2022-03-13 17:49:16,891] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_shutdown.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_shutdown.log INFO [2022-03-13 17:49:17,151] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_warn.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_warn.log INFO [2022-03-13 17:49:18,321] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/logs/4.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/logs/4.log INFO [2022-03-13 17:49:18,617] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/snapshots/442/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/snapshots/442/chronicle_config_rsm.snapshot INFO [2022-03-13 17:49:19,835] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats.log INFO [2022-03-13 17:49:20,069] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ini.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ini.log INFO [2022-03-13 17:49:21,554] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_access.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_access.log INFO [2022-03-13 17:49:21,612] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/master_events.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/master_events.log INFO [2022-03-13 17:49:23,494] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.indexer_stats.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.indexer_stats.log INFO [2022-03-13 17:49:23,502] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.debug.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.debug.log INFO [2022-03-13 17:49:24,974] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.views.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.views.log INFO [2022-03-13 17:49:25,139] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.indexer_stats.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.indexer_stats.log INFO [2022-03-13 17:49:26,319] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.debug.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.debug.log INFO [2022-03-13 17:49:26,487] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.ns_couchdb.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.ns_couchdb.log INFO [2022-03-13 17:49:27,985] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_debug.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_debug.log INFO [2022-03-13 17:49:29,005] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/local-ssl-cert.pem to cbcollect_info_ns_1@172.23.105.187_20220314-002116/local-ssl-cert.pem INFO [2022-03-13 17:49:29,636] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/version to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/version INFO [2022-03-13 17:49:30,655] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.mapreduce_errors.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.mapreduce_errors.log INFO [2022-03-13 17:49:31,230] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/snapshots/442/kv.snapshot to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/snapshots/442/kv.snapshot INFO [2022-03-13 17:49:32,237] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/.ignore to .ignore INFO [2022-03-13 17:49:32,884] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_log INFO [2022-03-13 17:49:33,957] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/rebalance_report_20220314T000612.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/rebalance_report_20220314T000612.json INFO [2022-03-13 17:49:34,554] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/rsms/chronicle_config_rsm/incarnation to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/rsms/chronicle_config_rsm/incarnation INFO [2022-03-13 17:49:35,756] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.reports.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.reports.log INFO [2022-03-13 17:49:36,419] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/syslog.tar.gz to cbcollect_info_ns_1@172.23.106.61_20220314-002116/syslog.tar.gz INFO [2022-03-13 17:49:37,701] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.http_access_internal.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.http_access_internal.log INFO [2022-03-13 17:49:38,450] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/hashtable_dump.txt to hashtable_dump.txt INFO [2022-03-13 17:49:39,474] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_info.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_info.log INFO [2022-03-13 17:49:40,680] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/mdocs.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/mdocs.log INFO [2022-03-13 17:49:41,285] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.ns_couchdb.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.ns_couchdb.log INFO [2022-03-13 17:49:42,470] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/exported_snapshot/kv.snapshot to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/exported_snapshot/kv.snapshot INFO [2022-03-13 17:49:43,112] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/projector_cprof.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/projector_cprof.log INFO [2022-03-13 17:49:44,240] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/projector_cprof.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/projector_cprof.log INFO [2022-03-13 17:49:44,969] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/logs/5.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/logs/5.log INFO [2022-03-13 17:49:46,008] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.info.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.info.log INFO [2022-03-13 17:49:46,750] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.info.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.info.log INFO [2022-03-13 17:49:47,756] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/rebalance_report_20220314T000721.json to cbcollect_info_ns_1@172.23.106.61_20220314-002116/rebalance_report_20220314T000721.json INFO [2022-03-13 17:49:48,744] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/users.dets to cbcollect_info_ns_1@172.23.105.187_20220314-002116/users.dets INFO [2022-03-13 17:49:49,535] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/exported_snapshot/chronicle_config_rsm.snapshot to cbcollect_info_ns_1@172.23.106.61_20220314-002116/chronicle/exported_snapshot/chronicle_config_rsm.snapshot INFO [2022-03-13 17:49:50,519] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_cbas_debug.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ns_server.analytics_cbas_debug.log INFO [2022-03-13 17:49:51,236] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.fts.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.fts.log INFO [2022-03-13 17:49:52,244] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/tombstones to cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/tombstones INFO [2022-03-13 17:49:52,980] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.goxdcr.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.goxdcr.log INFO [2022-03-13 17:49:54,058] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/meta.json to cbcollect_info_ns_1@172.23.105.187_20220314-002116/stats_snapshot/01FY2X61Y2C4H5MJX3C5MZSAD8/meta.json INFO [2022-03-13 17:49:54,767] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/.ignore to .ignore INFO [2022-03-13 17:49:55,856] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/couchstore_local.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/couchstore_local.log INFO [2022-03-13 17:49:56,500] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.indexer.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.indexer.log INFO [2022-03-13 17:49:57,617] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/snapshots/442/kv.snapshot to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/snapshots/442/kv.snapshot INFO [2022-03-13 17:49:58,316] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_pprof.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/goxdcr_pprof.log INFO [2022-03-13 17:49:59,423] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/couchbase.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/couchbase.log INFO [2022-03-13 17:50:00,049] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/diag.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/diag.log INFO [2022-03-13 17:50:01,408] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/ddocs.log to cbcollect_info_ns_1@172.23.105.187_20220314-002116/ddocs.log INFO [2022-03-13 17:50:01,933] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.error.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.error.log INFO [2022-03-13 17:50:03,203] jepsen node 172.23.105.187 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/rsms/kv/incarnation to cbcollect_info_ns_1@172.23.105.187_20220314-002116/chronicle/rsms/kv/incarnation INFO [2022-03-13 17:50:03,709] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_cbas_debug.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.analytics_cbas_debug.log INFO [2022-03-13 17:50:05,502] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/systemd_journal.gz to cbcollect_info_ns_1@172.23.106.61_20220314-002116/systemd_journal.gz INFO [2022-03-13 17:50:09,861] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.xdcr_target.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ns_server.xdcr_target.log INFO [2022-03-13 17:50:11,526] jepsen node 172.23.106.61 - jepsen.core downloading /tmp/jepsen-logs/cbcollect_info_ns_1@172.23.106.61_20220314-002116/ddocs.log to cbcollect_info_ns_1@172.23.106.61_20220314-002116/ddocs.log INFO [2022-03-13 17:50:13,130] jepsen test runner - jepsen.core Snarfing log files WARN [2022-03-13 17:50:13,133] jepsen node 172.23.107.227 - couchbase.core Ignoring duplicate log collection request WARN [2022-03-13 17:50:13,133] jepsen node 172.23.107.228 - couchbase.core Ignoring duplicate log collection request WARN [2022-03-13 17:50:13,133] jepsen node 172.23.105.187 - couchbase.core Ignoring duplicate log collection request WARN [2022-03-13 17:50:13,134] jepsen node 172.23.106.61 - couchbase.core Ignoring duplicate log collection request INFO [2022-03-13 17:50:13,138] jepsen node 172.23.105.187 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:50:13,139] jepsen node 172.23.106.61 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:50:13,139] jepsen node 172.23.107.228 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:50:13,139] jepsen node 172.23.107.227 - couchbase.util Tearing down couchbase node INFO [2022-03-13 17:50:19,172] jepsen node 172.23.107.228 - couchbase.util Teardown Complete INFO [2022-03-13 17:50:21,775] jepsen node 172.23.105.187 - couchbase.util Teardown Complete INFO [2022-03-13 17:50:21,981] jepsen node 172.23.106.61 - couchbase.util Teardown Complete INFO [2022-03-13 17:50:22,475] jepsen node 172.23.107.227 - couchbase.util Teardown Complete INFO [2022-03-13 17:50:22,484] jepsen test runner - jepsen.core Analyzing... INFO [2022-03-13 17:50:25,860] clojure-agent-send-pool-4 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/20/results.edn INFO [2022-03-13 17:50:26,166] clojure-agent-send-pool-4 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/20/history.edn INFO [2022-03-13 17:50:28,916] knossos reporter - knossos.search {:linear {:space 2, :cost 1.62E+11, :op {:process 373, :type :ok, :f :read, :value :nil, :index 3533, :time 119984731593}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,148] knossos reporter - knossos.search {:linear {:space 1, :cost 1.22E+12, :op {:process 334, :type :ok, :f :read, :value :nil, :index 3168, :time 107096797571}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,150] knossos reporter - knossos.search {:linear {:space 8, :cost 2.68E+15, :op {:process 361, :type :ok, :f :read, :value 41, :index 2825, :time 95993489805, :cas 1647216426121887744}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,196] knossos reporter - knossos.search {:linear {:space 32, :cost 1.90E+25, :op {:process 704, :type :ok, :f :read, :value 24, :index 3615, :time 123594825422, :cas 1647216428420890624}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,219] knossos reporter - knossos.search {:linear {:space 8, :cost 9.76E+12, :op {:process 353, :type :ok, :f :read, :value 8, :index 2779, :time 95169198094, :cas 1647216416155435008}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,250] knossos reporter - knossos.search {:linear {:space 1, :cost 1.22E+12, :op {:process 399, :type :ok, :f :read, :value :nil, :index 2567, :time 86640368991}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,253] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 471, :type :ok, :f :read, :value :nil, :index 3267, :time 110448338288}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,571] knossos reporter - knossos.search {:linear {:space 6144, :cost 7.50E+15, :op {:process 407, :type :ok, :f :read, :value 48, :index 2575, :time 86731206157, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:50:29,633] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 454, :type :ok, :f :read, :value 28, :index 2939, :time 99862058015, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:50:31,422] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 518, :type :ok, :f :read, :value 46, :index 2993, :time 101449235245, :cas 1647216414733041664}}, :wgl [:WGL]} INFO [2022-03-13 17:50:33,917] knossos reporter - knossos.search {:linear {:space 1, :cost 1.22E+12, :op {:process 194, :type :ok, :f :read, :value :nil, :index 3801, :time 128164227483}}, :wgl [:WGL]} INFO [2022-03-13 17:50:34,149] knossos reporter - knossos.search {:linear {:space 2, :cost 1.62E+11, :op {:process 425, :type :ok, :f :read, :value :nil, :index 3215, :time 109621335330}}, :wgl [:WGL]} INFO [2022-03-13 17:50:34,931] knossos reporter - knossos.search {:linear {:space 4, :cost 4.88E+12, :op {:process 442, :type :ok, :f :read, :value 8, :index 2805, :time 95600644413, :cas 1647216416155435008}}, :wgl [:WGL]} INFO [2022-03-13 17:50:34,931] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 400, :type :ok, :f :read, :value :nil, :index 2781, :time 95220120243}}, :wgl [:WGL]} INFO [2022-03-13 17:50:35,324] knossos reporter - knossos.search {:linear {:space 3072, :cost 1.03E+18, :op {:process 495, :type :ok, :f :read, :value 48, :index 2883, :time 97717990779, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:50:38,919] knossos reporter - knossos.search {:linear {:space 2, :cost 2.44E+12, :op {:process 373, :type :ok, :f :read, :value :nil, :index 3904, :time 129352739058}}, :wgl [:WGL]} INFO [2022-03-13 17:50:39,150] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 513, :type :ok, :f :read, :value :nil, :index 3258, :time 110318802883}}, :wgl [:WGL]} INFO [2022-03-13 17:50:39,919] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 471, :type :ok, :f :read, :value :nil, :index 3283, :time 110679334505}}, :wgl [:WGL]} INFO [2022-03-13 17:50:39,919] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 454, :type :ok, :f :read, :value 28, :index 2945, :time 99948317736, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:50:39,932] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 400, :type :ok, :f :read, :value :nil, :index 2797, :time 95447111686}}, :wgl [:WGL]} INFO [2022-03-13 17:50:39,932] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 353, :type :ok, :f :read, :value 4, :index 3047, :time 103430569032, :cas 1647216464123002880}}, :wgl [:WGL]} INFO [2022-03-13 17:50:40,017] knossos reporter - knossos.search {:linear {:space 8200, :cost 2.75E+18, :op {:process 361, :type :ok, :f :read, :value 41, :index 2829, :time 96122664140, :cas 1647216426121887744}}, :wgl [:WGL]} INFO [2022-03-13 17:50:40,486] knossos reporter - knossos.search {:linear {:space 10240, :cost 3.43E+18, :op {:process 495, :type :ok, :f :read, :value 48, :index 2887, :time 97826696245, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:50:41,490] knossos reporter - knossos.search {:linear {:space 4, :cost 2.42E+16, :op {:process 517, :type :ok, :f :read, :value 46, :index 3211, :time 109369476240, :cas 1647216414733041664}}, :wgl [:WGL]} INFO [2022-03-13 17:50:43,920] knossos reporter - knossos.search {:linear {:space 2, :cost 2.44E+12, :op {:process 373, :type :ok, :f :read, :value :nil, :index 3913, :time 129357303303}}, :wgl [:WGL]} INFO [2022-03-13 17:50:44,151] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 513, :type :ok, :f :read, :value :nil, :index 3291, :time 110744115435}}, :wgl [:WGL]} INFO [2022-03-13 17:50:45,431] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 454, :type :ok, :f :read, :value 28, :index 2951, :time 100017596393, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:50:45,432] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 471, :type :ok, :f :read, :value :nil, :index 3287, :time 110736899711}}, :wgl [:WGL]} INFO [2022-03-13 17:50:45,734] knossos reporter - knossos.search {:linear {:space 10240, :cost 6.19E+19, :op {:process 406, :type :ok, :f :read, :value 48, :index 2975, :time 100939549852, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:50:45,961] knossos reporter - knossos.search {:linear {:space 8200, :cost 2.75E+18, :op {:process 361, :type :ok, :f :read, :value 41, :index 2837, :time 96307504143, :cas 1647216426121887744}}, :wgl [:WGL]} INFO [2022-03-13 17:50:46,103] knossos reporter - knossos.search {:linear {:space 12288, :cost 7.31E+27, :op {:process 533, :type :ok, :f :read, :value 4, :index 4257, :time 136931160674, :cas 1647216464123002880}}, :wgl [:WGL]} INFO [2022-03-13 17:50:49,020] knossos reporter - knossos.search {:linear {:space 2, :cost 2.44E+12, :op {:process 373, :type :ok, :f :read, :value :nil, :index 3919, :time 129369087063}}, :wgl [:WGL]} INFO [2022-03-13 17:50:49,152] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 513, :type :ok, :f :read, :value :nil, :index 3309, :time 111420607097}}, :wgl [:WGL]} INFO [2022-03-13 17:50:51,022] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:persist-to 0, :mutation-token Optional[mt{vbID=413, vbUUID=108683614830760, seqno=9, bucket=default}], :index 2963, :value 39, :time 100291292356, :process 400, :replicate-to 0, :type :ok, :cas 1647216469441314816, :f :write, :durability-level 3}}, :wgl [:WGL]} INFO [2022-03-13 17:50:51,023] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 454, :type :ok, :f :read, :value 28, :index 2955, :time 100135418035, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:50:51,304] knossos reporter - knossos.search {:linear {:space 8200, :cost 8.80E+24, :op {:process 450, :type :ok, :f :read, :value 41, :index 3488, :time 118816362146, :cas 1647216426121887744}}, :wgl [:WGL]} INFO [2022-03-13 17:50:54,639] knossos reporter - knossos.search {:linear {:space 2, :cost 2.44E+12, :op {:process 194, :type :ok, :f :read, :value :nil, :index 3930, :time 129395947516}}, :wgl [:WGL]} INFO [2022-03-13 17:50:56,258] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 382, :type :ok, :f :read, :value :nil, :index 3409, :time 115916330128}}, :wgl [:WGL]} INFO [2022-03-13 17:50:59,639] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 334, :type :ok, :f :read, :value :nil, :index 3313, :time 111528257337}}, :wgl [:WGL]} INFO [2022-03-13 17:50:59,640] knossos reporter - knossos.search {:linear {:space 2, :cost 2.44E+12, :op {:process 373, :type :ok, :f :read, :value :nil, :index 3997, :time 130237987359}}, :wgl [:WGL]} INFO [2022-03-13 17:51:01,595] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 382, :type :ok, :f :read, :value :nil, :index 3415, :time 116011526908}}, :wgl [:WGL]} INFO [2022-03-13 17:51:01,598] knossos reporter - knossos.search {:linear {:space 4, :cost 7.85E+13, :op {:process 454, :type :ok, :f :read, :value 28, :index 2959, :time 100204924276, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:51:05,409] knossos reporter - knossos.search {:linear {:space 2, :cost 1.62E+11, :op {:process 373, :type :ok, :f :read, :value :nil, :index 4029, :time 130694398579}}, :wgl [:WGL]} INFO [2022-03-13 17:51:05,419] knossos reporter - knossos.search {:linear {:space 1, :cost 2.31E+18, :op {:process 605, :type :ok, :f :read, :value :nil, :index 3967, :time 129671539385}}, :wgl [:WGL]} INFO [2022-03-13 17:51:06,596] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 382, :type :ok, :f :read, :value :nil, :index 3419, :time 116085478596}}, :wgl [:WGL]} INFO [2022-03-13 17:51:06,599] knossos reporter - knossos.search {:linear {:space 4, :cost 2.42E+16, :op {:process 453, :type :ok, :f :read, :value 28, :index 3013, :time 102933333393, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:51:07,870] knossos reporter - knossos.search {:linear {:space 18432, :cost 1.11E+20, :op {:process 406, :type :ok, :f :read, :value 48, :index 2977, :time 100967796399, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:51:10,410] knossos reporter - knossos.search {:linear {:space 1, :cost 8.10E+10, :op {:process 373, :type :ok, :f :read, :value :nil, :index 4213, :time 134149950016}}, :wgl [:WGL]} INFO [2022-03-13 17:51:15,411] knossos reporter - knossos.search {:linear {:space 1, :cost 1.22E+12, :op {:process 552, :type :ok, :f :read, :value :nil, :index 4261, :time 137110247065}}, :wgl [:WGL]} INFO [2022-03-13 17:51:16,744] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 293, :type :ok, :f :read, :value :nil, :index 3457, :time 117992928626}}, :wgl [:WGL]} INFO [2022-03-13 17:51:20,413] knossos reporter - knossos.search {:linear {:space 1, :cost 1.22E+12, :op {:process 552, :type :ok, :f :read, :value :nil, :index 4395, :time 138995139261}}, :wgl [:WGL]} INFO [2022-03-13 17:51:25,414] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 194, :type :ok, :f :read, :value :nil, :index 4514, :time 142479835035}}, :wgl [:WGL]} INFO [2022-03-13 17:51:26,819] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 472, :type :ok, :f :read, :value :nil, :index 4123, :time 133150765187}}, :wgl [:WGL]} INFO [2022-03-13 17:51:30,416] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 194, :type :ok, :f :read, :value :nil, :index 4618, :time 143657800802}}, :wgl [:WGL]} INFO [2022-03-13 17:51:35,351] knossos reporter - knossos.search {:linear {:space 4, :cost 9.25E+18, :op {:process 516, :type :ok, :f :read, :value 46, :index 3345, :time 113284363498, :cas 1647216414733041664}}, :wgl [:WGL]} INFO [2022-03-13 17:51:35,417] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 4973, :time 149909905159}}, :wgl [:WGL]} INFO [2022-03-13 17:51:37,872] knossos reporter - knossos.search {:linear {:space 18432, :cost 1.11E+20, :op {:process 406, :type :ok, :f :read, :value 48, :index 2979, :time 100998909340, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:51:45,420] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5008, :time 151584525111}}, :wgl [:WGL]} INFO [2022-03-13 17:51:50,421] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5034, :time 151823489552}}, :wgl [:WGL]} INFO [2022-03-13 17:51:55,423] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5099, :time 152501135191}}, :wgl [:WGL]} INFO [2022-03-13 17:52:00,709] knossos reporter - knossos.search {:linear {:space 18432, :cost 1.11E+20, :op {:process 406, :type :ok, :f :read, :value 48, :index 2985, :time 101048807336, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:52:00,803] knossos reporter - knossos.search {:linear {:space 65536, :cost 3.19E+24, :op {:persist-to 0, :index 3430, :value 37, :time 116731102511, :process 669, :replicate-to 0, :type :invoke, :f :write, :durability-level 3}}, :wgl [:WGL]} INFO [2022-03-13 17:52:02,685] knossos reporter - knossos.search {:linear {:space 4, :cost 2.42E+16, :op {:process 453, :type :ok, :f :read, :value 28, :index 3015, :time 102958529384, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:52:07,739] knossos reporter - knossos.search {:linear {:space 65536, :cost 7.03E+25, :op {:process 581, :type :ok, :f :read, :value 39, :index 3491, :time 118823627547, :cas 1647216469441314816}}, :wgl [:WGL]} INFO [2022-03-13 17:52:12,686] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 561, :type :ok, :f :read, :value :nil, :index 4173, :time 133821139972}}, :wgl [:WGL]} INFO [2022-03-13 17:52:15,427] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5103, :time 152503622569}}, :wgl [:WGL]} INFO [2022-03-13 17:52:22,418] knossos reporter - knossos.search {:linear {:space 18432, :cost 1.11E+20, :op {:process 497, :type :ok, :f :read, :value 48, :index 3069, :time 103906486328, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:52:30,430] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 194, :type :ok, :f :read, :value :nil, :index 5129, :time 152810344722}}, :wgl [:WGL]} INFO [2022-03-13 17:52:35,431] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5207, :time 153412617206}}, :wgl [:WGL]} INFO [2022-03-13 17:52:39,169] knossos reporter - knossos.search {:linear {:space 17408, :cost 1.05E+20, :op {:process 497, :type :ok, :f :read, :value 48, :index 3175, :time 107802100925, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:52:40,432] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5246, :time 153641902650}}, :wgl [:WGL]} INFO [2022-03-13 17:52:45,433] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5287, :time 154018589143}}, :wgl [:WGL]} INFO [2022-03-13 17:52:48,043] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 561, :type :ok, :f :read, :value :nil, :index 4181, :time 133930436083}}, :wgl [:WGL]} INFO [2022-03-13 17:52:48,054] knossos reporter - knossos.search {:linear {:space 4, :cost 9.25E+18, :op {:process 455, :type :ok, :f :read, :value 28, :index 3305, :time 111221289638, :cas 1647216417530249216}}, :wgl [:WGL]} INFO [2022-03-13 17:52:50,435] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5355, :time 154648371554}}, :wgl [:WGL]} INFO [2022-03-13 17:52:54,946] knossos reporter - knossos.search {:linear {:space 17408, :cost 2.59E+29, :op {:process 586, :type :ok, :f :read, :value 48, :index 4093, :time 132713462119, :cas 1647216429117931520}}, :wgl [:WGL]} INFO [2022-03-13 17:53:10,445] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 194, :type :ok, :f :read, :value :nil, :index 5382, :time 154839748514}}, :wgl [:WGL]} INFO [2022-03-13 17:53:15,446] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5401, :time 155004773594}}, :wgl [:WGL]} INFO [2022-03-13 17:53:29,833] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 561, :type :ok, :f :read, :value :nil, :index 4197, :time 134033903674}}, :wgl [:WGL]} INFO [2022-03-13 17:53:37,558] knossos reporter - knossos.search {:linear {:space 4, :cost 1.34E+15, :op {:process 194, :type :ok, :f :read, :value :nil, :index 5466, :time 155851645737}}, :wgl [:WGL]} INFO [2022-03-13 17:53:42,739] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 642, :type :ok, :f :read, :value :nil, :index 5467, :time 155851701394}}, :wgl [:WGL]} INFO [2022-03-13 17:53:47,740] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5514, :time 156491795633}}, :wgl [:WGL]} INFO [2022-03-13 17:53:55,292] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 561, :type :ok, :f :read, :value :nil, :index 4215, :time 134171725316}}, :wgl [:WGL]} INFO [2022-03-13 17:53:57,742] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5519, :time 156497485189}}, :wgl [:WGL]} INFO [2022-03-13 17:54:07,744] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5524, :time 156504270781}}, :wgl [:WGL]} INFO [2022-03-13 17:54:24,051] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5527, :time 156508489244}}, :wgl [:WGL]} INFO [2022-03-13 17:54:34,053] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5529, :time 156511380801}}, :wgl [:WGL]} INFO [2022-03-13 17:54:36,149] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 561, :type :ok, :f :read, :value :nil, :index 4225, :time 134317611290}}, :wgl [:WGL]} INFO [2022-03-13 17:54:44,494] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5567, :time 156878510964}}, :wgl [:WGL]} INFO [2022-03-13 17:54:49,495] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 642, :type :ok, :f :read, :value :nil, :index 5600, :time 157095097556}}, :wgl [:WGL]} INFO [2022-03-13 17:54:54,496] knossos reporter - knossos.search {:linear {:space 2, :cost 4.62E+18, :op {:process 605, :type :ok, :f :read, :value :nil, :index 3969, :time 129673330512}}, :wgl [:WGL]} INFO [2022-03-13 17:54:54,496] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5661, :time 157544910527}}, :wgl [:WGL]} INFO [2022-03-13 17:54:59,497] knossos reporter - knossos.search {:linear {:space 4, :cost 9.25E+18, :op {:process 608, :type :ok, :f :read, :value 30, :index 3499, :time 118925004164, :cas 1647216482694660096}}, :wgl [:WGL]} INFO [2022-03-13 17:55:11,355] knossos reporter - knossos.search {:linear {:space 1, :cost 1.15E+17, :op {:process 561, :type :ok, :f :read, :value :nil, :index 4346, :time 138818569990}}, :wgl [:WGL]} INFO [2022-03-13 17:55:21,357] knossos reporter - knossos.search {:linear {:space 4, :cost 1.34E+15, :op {:process 642, :type :ok, :f :read, :value :nil, :index 5756, :time 158245721155}}, :wgl [:WGL]} INFO [2022-03-13 17:55:26,358] knossos reporter - knossos.search {:linear {:space 2, :cost 1.21E+16, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5779, :time 158419743328}}, :wgl [:WGL]} INFO [2022-03-13 17:55:46,361] knossos reporter - knossos.search {:linear {:space 4, :cost 2.42E+16, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5810, :time 158720794748}}, :wgl [:WGL]} INFO [2022-03-13 17:56:07,137] knossos reporter - knossos.search {:linear {:space 4, :cost 2.42E+16, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5815, :time 158723523638}}, :wgl [:WGL]} INFO [2022-03-13 17:56:22,753] knossos reporter - knossos.search {:linear {:space 2, :cost 4.62E+18, :op {:process 472, :type :ok, :f :read, :value :nil, :index 4355, :time 138843393218}}, :wgl [:WGL]} INFO [2022-03-13 17:56:32,719] knossos reporter - knossos.search {:linear {:space 4, :cost 1.34E+15, :op {:process 194, :type :ok, :f :read, :value :nil, :index 5825, :time 158751609401}}, :wgl [:WGL]} INFO [2022-03-13 17:56:42,721] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5860, :time 159089705941}}, :wgl [:WGL]} INFO [2022-03-13 17:56:47,722] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5870, :time 159097266541}}, :wgl [:WGL]} INFO [2022-03-13 17:56:57,727] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 642, :type :ok, :f :read, :value :nil, :index 5916, :time 159425515339}}, :wgl [:WGL]} INFO [2022-03-13 17:57:02,728] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5917, :time 159425544959}}, :wgl [:WGL]} INFO [2022-03-13 17:57:02,742] knossos reporter - knossos.search {:linear {:space 4, :cost 9.89E+22, :op {:process 634, :type :ok, :f :read, :value 16, :index 3589, :time 123045014980, :cas 1647216482695446528}}, :wgl [:WGL]} INFO [2022-03-13 17:57:07,729] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 5972, :time 159768915784}}, :wgl [:WGL]} INFO [2022-03-13 17:57:13,284] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6002, :time 159973059517}}, :wgl [:WGL]} INFO [2022-03-13 17:57:18,284] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6013, :time 160028952989}}, :wgl [:WGL]} INFO [2022-03-13 17:57:23,473] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6095, :time 161006512205}}, :wgl [:WGL]} INFO [2022-03-13 17:57:28,475] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6119, :time 161240005804}}, :wgl [:WGL]} INFO [2022-03-13 17:57:33,476] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6302, :time 163226500790}}, :wgl [:WGL]} INFO [2022-03-13 17:58:56,924] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6353, :time 163499123730}}, :wgl [:WGL]} INFO [2022-03-13 17:58:56,925] knossos reporter - knossos.search {:linear {:space 2, :cost 9.73E+19, :op {:process 424, :type :ok, :f :read, :value :nil, :index 4085, :time 132244811302}}, :wgl [:WGL]} INFO [2022-03-13 17:59:01,925] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6463, :time 165106867204}}, :wgl [:WGL]} INFO [2022-03-13 17:59:12,264] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 194, :type :ok, :f :read, :value :nil, :index 6533, :time 165674416025}}, :wgl [:WGL]} INFO [2022-03-13 17:59:17,265] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6557, :time 166223264994}}, :wgl [:WGL]} INFO [2022-03-13 17:59:22,265] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6589, :time 166440582202}}, :wgl [:WGL]} INFO [2022-03-13 17:59:27,266] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6597, :time 166458727060}}, :wgl [:WGL]} INFO [2022-03-13 17:59:32,271] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6731, :time 167726783132}}, :wgl [:WGL]} INFO [2022-03-13 17:59:37,272] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6733, :time 167728748979}}, :wgl [:WGL]} INFO [2022-03-13 17:59:42,272] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6737, :time 167733231064}}, :wgl [:WGL]} INFO [2022-03-13 17:59:47,273] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6739, :time 167738966057}}, :wgl [:WGL]} INFO [2022-03-13 18:01:13,074] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6817, :time 168308003048}}, :wgl [:WGL]} INFO [2022-03-13 18:01:18,065] knossos reporter - knossos.search {:linear {:space 4, :cost 9.25E+18, :op {:process 472, :type :ok, :f :read, :value :nil, :index 4359, :time 138845572084}}, :wgl [:WGL]} INFO [2022-03-13 18:01:18,075] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6853, :time 168600575860}}, :wgl [:WGL]} INFO [2022-03-13 18:01:22,246] knossos reporter - knossos.search {:linear {:space 262160, :cost 1.28E+25, :op {:process 450, :type :invoke, :f :read, :value 41, :index 3490, :time 118816883100}}, :wgl [:WGL]} INFO [2022-03-13 18:01:23,075] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6855, :time 168602921775}}, :wgl [:WGL]} INFO [2022-03-13 18:01:24,755] knossos reporter - knossos.search {:linear {:space 131072, :cost 3.03E+23, :op {:persist-to 0, :index 3518, :value 17, :time 119394199526, :process 607, :replicate-to 0, :type :invoke, :f :write, :durability-level 3}}, :wgl [:WGL]} INFO [2022-03-13 18:01:28,076] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6857, :time 168604797597}}, :wgl [:WGL]} INFO [2022-03-13 18:01:32,399] knossos reporter - knossos.search {:linear {:space 131072, :cost 1.95E+30, :op {:process 697, :type :ok, :f :read, :value 30, :index 4265, :time 137176651837, :cas 1647216482694660096}}, :wgl [:WGL]} INFO [2022-03-13 18:01:32,942] knossos reporter - knossos.search {:linear {:space 262160, :cost 2.81E+26, :op {:process 632, :type :ok, :f :read, :value 41, :index 3501, :time 118941409314, :cas 1647216426121887744}}, :wgl [:WGL]} INFO [2022-03-13 18:01:33,077] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 6929, :time 169093068478}}, :wgl [:WGL]} INFO [2022-03-13 18:01:38,078] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 642, :type :ok, :f :read, :value :nil, :index 7102, :time 170091551229}}, :wgl [:WGL]} INFO [2022-03-13 18:06:40,576] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7124, :time 170179313871}}, :wgl [:WGL]} Exception in thread "Thread-172" java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at clojure.core$deref_future.invokeStatic(core.clj:2304) at clojure.core$future_call$reify__8477.deref(core.clj:6976) at clojure.core$deref.invokeStatic(core.clj:2324) at clojure.core$pmap$step__8490$fn__8494.invoke(core.clj:7027) at clojure.lang.LazySeq.sval(LazySeq.java:42) at clojure.lang.LazySeq.seq(LazySeq.java:51) at clojure.lang.Cons.next(Cons.java:39) at clojure.lang.RT.next(RT.java:713) at clojure.core$next__5403.invokeStatic(core.clj:64) at clojure.core$concat$cat__5512$fn__5513.invoke(core.clj:744) at clojure.lang.LazySeq.sval(LazySeq.java:42) at clojure.lang.LazySeq.seq(LazySeq.java:51) at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59) at clojure.lang.ChunkedCons.next(ChunkedCons.java:43) at clojure.lang.RT.next(RT.java:713) at clojure.core$next__5403.invokeStatic(core.clj:64) at clojure.core.protocols$naive_seq_reduce.invokeStatic(protocols.clj:63) at clojure.core.protocols$interface_or_naive_reduce.invokeStatic(protocols.clj:72) at clojure.core.protocols$fn__8181.invokeStatic(protocols.clj:169) at clojure.core.protocols$fn__8181.invoke(protocols.clj:124) at clojure.core.protocols$fn__8136$G__8131__8145.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31) at clojure.core.protocols$fn__8168.invokeStatic(protocols.clj:75) at clojure.core.protocols$fn__8168.invoke(protocols.clj:75) at clojure.core.protocols$fn__8110$G__8105__8123.invoke(protocols.clj:13) at clojure.core$reduce.invokeStatic(core.clj:6830) at clojure.core$reduce.invoke(core.clj:6812) at knossos.linear$step.invokeStatic(linear.clj:201) at knossos.linear$step.invoke(linear.clj:169) at clojure.core$partial$fn__5861.invoke(core.clj:2643) at clojure.lang.PersistentVector.reduce(PersistentVector.java:343) at clojure.core$reduce.invokeStatic(core.clj:6829) at clojure.core$reduce.invoke(core.clj:6812) at knossos.linear$check.invokeStatic(linear.clj:228) at knossos.linear$check.invoke(linear.clj:225) at knossos.linear$start_analysis$fn__3319$fn__3320.invoke(linear.clj:277) at knossos.linear$start_analysis$fn__3319.invoke(linear.clj:277) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: Java heap space at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:863) at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:713) at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:649) at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:354) at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfAbsent(NonBlockingHashMap.java:321) at org.cliffc.high_scale_lib.NonBlockingHashSet.add(NonBlockingHashSet.java:32) at knossos.weak_cache_set.NBHSCacheSet.add_BANG_(weak_cache_set.clj:66) at knossos.linear$jit_linearizations.invokeStatic(linear.clj:78) at knossos.linear$jit_linearizations.invoke(linear.clj:66) at knossos.linear$jit_linearizations$recurrence__3268.invoke(linear.clj:98) at knossos.util$rkeep$fn__670$fn__671.invoke(util.clj:42) at clojure.core.reducers$filter$fn__587$fn__588.invoke(reducers.clj:161) at knossos.linear.config.ArrayProcesses$reify__3142.reduce(config.clj:231) at clojure.core.protocols$fn__8164.invokeStatic(protocols.clj:89) at clojure.core.protocols$fn__8164.invoke(protocols.clj:75) at clojure.core.protocols$fn__8110$G__8105__8123.invoke(protocols.clj:13) at clojure.core.reducers$folder$reify__554.coll_reduce(reducers.clj:93) at clojure.core.reducers$folder$reify__554.coll_reduce(reducers.clj:93) at clojure.core$reduce.invokeStatic(core.clj:6830) at clojure.core$reduce.invoke(core.clj:6812) at knossos.linear$jit_linearizations.invokeStatic(linear.clj:97) at knossos.linear$jit_linearizations.invoke(linear.clj:66) at knossos.linear$jit_linearizations$recurrence__3268.invoke(linear.clj:98) at knossos.util$rkeep$fn__670$fn__671.invoke(util.clj:42) at clojure.core.reducers$filter$fn__587$fn__588.invoke(reducers.clj:161) at knossos.linear.config.ArrayProcesses$reify__3142.reduce(config.clj:231) at clojure.core.protocols$fn__8164.invokeStatic(protocols.clj:89) at clojure.core.protocols$fn__8164.invoke(protocols.clj:75) at clojure.core.protocols$fn__8110$G__8105__8123.invoke(protocols.clj:13) at clojure.core.reducers$folder$reify__554.coll_reduce(reducers.clj:93) at clojure.core.reducers$folder$reify__554.coll_reduce(reducers.clj:93) at clojure.core$reduce.invokeStatic(core.clj:6830) INFO [2022-03-13 18:22:01,542] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7127, :time 170181204923}}, :wgl [:WGL]} INFO [2022-03-13 18:22:11,748] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7189, :time 170405122188}}, :wgl [:WGL]} INFO [2022-03-13 18:23:27,813] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7232, :time 170638969020}}, :wgl [:WGL]} INFO [2022-03-13 18:23:33,034] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7235, :time 170653467242}}, :wgl [:WGL]} INFO [2022-03-13 18:23:43,035] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7237, :time 170656514891}}, :wgl [:WGL]} INFO [2022-03-13 18:23:48,120] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7239, :time 170658704552}}, :wgl [:WGL]} INFO [2022-03-13 18:23:58,673] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7241, :time 170660333929}}, :wgl [:WGL]} INFO [2022-03-13 18:24:04,454] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7243, :time 170662215501}}, :wgl [:WGL]} INFO [2022-03-13 18:24:15,070] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 194, :type :ok, :f :read, :value :nil, :index 7286, :time 170999627098}}, :wgl [:WGL]} INFO [2022-03-13 18:24:20,071] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 642, :type :ok, :f :read, :value :nil, :index 7469, :time 174043789593}}, :wgl [:WGL]} INFO [2022-03-13 18:25:50,642] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 642, :type :ok, :f :read, :value :nil, :index 7505, :time 174340633697}}, :wgl [:WGL]} INFO [2022-03-13 18:31:50,665] knossos reporter - knossos.search {:linear {:space 2, :cost 1.21E+16, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7531, :time 174563391957}}, :wgl [:WGL]} Exception in thread "Thread-190" java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at clojure.core$deref_future.invokeStatic(core.clj:2304) at clojure.core$future_call$reify__8477.deref(core.clj:6976) at clojure.core$deref.invokeStatic(core.clj:2324) at clojure.core$pmap$step__8490$fn__8494.invoke(core.clj:7027) at clojure.lang.LazySeq.sval(LazySeq.java:42) at clojure.lang.LazySeq.seq(LazySeq.java:51) at clojure.lang.Cons.next(Cons.java:39) at clojure.lang.RT.next(RT.java:713) at clojure.core$next__5403.invokeStatic(core.clj:64) at clojure.core$concat$cat__5512$fn__5513.invoke(core.clj:744) at clojure.lang.LazySeq.sval(LazySeq.java:42) at clojure.lang.LazySeq.seq(LazySeq.java:51) at clojure.lang.Cons.next(Cons.java:39) at clojure.lang.RT.next(RT.java:713) at clojure.core$next__5403.invokeStatic(core.clj:64) at clojure.core.protocols$fn__8181.invokeStatic(protocols.clj:169) at clojure.core.protocols$fn__8181.invoke(protocols.clj:124) at clojure.core.protocols$fn__8136$G__8131__8145.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31) at clojure.core.protocols$fn__8168.invokeStatic(protocols.clj:75) at clojure.core.protocols$fn__8168.invoke(protocols.clj:75) at clojure.core.protocols$fn__8110$G__8105__8123.invoke(protocols.clj:13) at clojure.core$reduce.invokeStatic(core.clj:6830) at clojure.core$reduce.invoke(core.clj:6812) at knossos.linear$step.invokeStatic(linear.clj:201) at knossos.linear$step.invoke(linear.clj:169) at clojure.core$partial$fn__5861.invoke(core.clj:2643) at clojure.lang.PersistentVector.reduce(PersistentVector.java:343) at clojure.core$reduce.invokeStatic(core.clj:6829) at clojure.core$reduce.invoke(core.clj:6812) at knossos.linear$check.invokeStatic(linear.clj:228) at knossos.linear$check.invoke(linear.clj:225) at knossos.linear$start_analysis$fn__3319$fn__3320.invoke(linear.clj:277) at knossos.linear$start_analysis$fn__3319.invoke(linear.clj:277) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: Java heap space INFO [2022-03-13 18:42:59,288] knossos reporter - knossos.search {:linear {:space 4, :cost 4.61E+17, :op {:process 561, :type :ok, :f :read, :value :nil, :index 4393, :time 138994033273}}, :wgl [:WGL]} INFO [2022-03-13 18:43:10,202] knossos reporter - knossos.search {:linear {:space 4, :cost 1.34E+15, :op {:process 642, :type :ok, :f :read, :value :nil, :index 7535, :time 174563581899}}, :wgl [:WGL]} INFO [2022-03-13 18:43:21,875] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 194, :type :ok, :f :read, :value :nil, :index 7549, :time 174632578289}}, :wgl [:WGL]} INFO [2022-03-13 18:43:27,638] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 194, :type :ok, :f :read, :value :nil, :index 7690, :time 175584203887}}, :wgl [:WGL]} INFO [2022-03-13 18:43:39,239] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7698, :time 175598600140}}, :wgl [:WGL]} INFO [2022-03-13 18:43:49,923] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7705, :time 175603149026}}, :wgl [:WGL]} INFO [2022-03-13 18:45:12,619] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7736, :time 175791506819}}, :wgl [:WGL]} INFO [2022-03-13 18:45:24,294] knossos reporter - knossos.search {:linear {:space 2, :cost 2.30E+17, :op {:process 473, :type :ok, :f :read, :value :nil, :index 4717, :time 146403642265}}, :wgl [:WGL]} INFO [2022-03-13 18:45:24,299] knossos reporter - knossos.search {:linear {:space 2, :cost 3.92E+13, :op {:process 642, :type :ok, :f :read, :value :nil, :index 7744, :time 175871886324}}, :wgl [:WGL]} INFO [2022-03-13 18:45:30,526] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7829, :time 176311075647}}, :wgl [:WGL]} INFO [2022-03-13 18:45:36,958] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7831, :time 176314437610}}, :wgl [:WGL]} INFO [2022-03-13 18:45:41,959] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7835, :time 176316669675}}, :wgl [:WGL]} INFO [2022-03-13 18:45:51,961] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7837, :time 176318417673}}, :wgl [:WGL]} INFO [2022-03-13 18:45:57,044] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 642, :type :ok, :f :read, :value :nil, :index 7893, :time 177207211603}}, :wgl [:WGL]} INFO [2022-03-13 18:46:03,853] knossos reporter - knossos.search {:linear {:space 131072, :cost 1.41E+26, :op {:process 490, :type :ok, :f :read, :value 39, :index 3495, :time 118879429919, :cas 1647216469441314816}}, :wgl [:WGL]} INFO [2022-03-13 18:46:13,388] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 194, :type :ok, :f :read, :value :nil, :index 7905, :time 177292983473}}, :wgl [:WGL]} INFO [2022-03-13 18:47:34,408] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7907, :time 177328433741}}, :wgl [:WGL]} INFO [2022-03-13 18:47:50,743] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7929, :time 177544978782}}, :wgl [:WGL]} INFO [2022-03-13 18:48:03,031] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7932, :time 177547763444}}, :wgl [:WGL]} INFO [2022-03-13 18:48:13,032] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7958, :time 177784057039}}, :wgl [:WGL]} INFO [2022-03-13 18:48:23,033] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7963, :time 177788166794}}, :wgl [:WGL]} INFO [2022-03-13 18:48:23,048] knossos reporter - knossos.search {:linear {:space 1, :cost 1.15E+17, :op {:process 473, :type :ok, :f :read, :value :nil, :index 4727, :time 146551189802}}, :wgl [:WGL]} INFO [2022-03-13 18:48:28,487] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 194, :type :ok, :f :read, :value :nil, :index 7967, :time 177789666596}}, :wgl [:WGL]} INFO [2022-03-13 18:48:39,152] knossos reporter - knossos.search {:linear {:space 2, :cost 1.21E+16, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7989, :time 178092445095}}, :wgl [:WGL]} INFO [2022-03-13 18:50:33,937] knossos reporter - knossos.search {:linear {:space 4, :cost 2.42E+16, :op {:process 463, :type :ok, :f :read, :value :nil, :index 7991, :time 178095272698}}, :wgl [:WGL]} WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,262] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,263] Service Thread - knossos.search Out of memory; aborting search WARN [2022-03-13 18:52:09,263] Service Thread - knossos.search Out of memory; aborting search INFO [2022-03-13 18:52:09,277] clojure-agent-send-pool-0 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/1/results.edn INFO [2022-03-13 18:52:09,277] clojure-agent-send-pool-1 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/21/results.edn INFO [2022-03-13 18:52:09,292] clojure-agent-send-pool-2 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/4/results.edn INFO [2022-03-13 18:52:09,294] clojure-agent-send-pool-6 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/27/results.edn INFO [2022-03-13 18:52:09,310] clojure-agent-send-pool-4 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/22/results.edn INFO [2022-03-13 18:52:09,315] clojure-agent-send-pool-9 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/0/results.edn INFO [2022-03-13 18:52:09,330] clojure-agent-send-pool-5 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/15/results.edn INFO [2022-03-13 18:52:09,338] clojure-agent-send-pool-8 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/7/results.edn INFO [2022-03-13 18:52:09,338] clojure-agent-send-pool-3 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/13/results.edn INFO [2022-03-13 18:52:09,392] clojure-agent-send-pool-9 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/0/history.edn INFO [2022-03-13 18:52:09,409] clojure-agent-send-pool-4 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/22/history.edn INFO [2022-03-13 18:52:09,448] clojure-agent-send-pool-6 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/27/history.edn INFO [2022-03-13 18:52:09,449] clojure-agent-send-pool-5 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/15/history.edn INFO [2022-03-13 18:52:09,455] clojure-agent-send-pool-1 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/21/history.edn INFO [2022-03-13 18:52:09,506] clojure-agent-send-pool-3 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/13/history.edn INFO [2022-03-13 18:52:09,770] clojure-agent-send-pool-2 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/4/history.edn INFO [2022-03-13 18:52:09,854] clojure-agent-send-pool-8 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/7/history.edn INFO [2022-03-13 18:52:10,599] clojure-agent-send-pool-0 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/1/history.edn INFO [2022-03-13 18:52:11,608] clojure-agent-send-pool-8 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/2/results.edn INFO [2022-03-13 18:52:12,015] clojure-agent-send-pool-8 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/2/history.edn INFO [2022-03-13 18:52:13,586] clojure-agent-send-pool-5 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/25/results.edn INFO [2022-03-13 18:52:14,259] clojure-agent-send-pool-5 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/25/history.edn INFO [2022-03-13 18:52:14,843] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 560, :type :ok, :f :read, :value :nil, :index 3217, :time 109685762972}}, :wgl [:WGL]} INFO [2022-03-13 18:52:14,905] knossos reporter - knossos.search {:linear {:space 2, :cost 1.19E+24, :op {:process 682, :type :ok, :f :read, :value 20, :index 106233, :time 790019324550, :cas 1647216416087146496}}, :wgl [:WGL]} INFO [2022-03-13 18:52:15,116] knossos reporter - knossos.search {:linear {:space 512, :cost 5.90E+19, :op {:process 624, :type :ok, :f :read, :value 25, :index 3203, :time 109028701155, :cas 1647216446020714496}}, :wgl [:WGL]} INFO [2022-03-13 18:52:15,118] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 280, :type :ok, :f :read, :value :nil, :index 3279, :time 110566062908}}, :wgl [:WGL]} INFO [2022-03-13 18:52:15,486] knossos reporter - knossos.search {:linear {:space 1024, :cost 2.01E+16, :op {:process 359, :type :ok, :f :read, :value 17, :index 2929, :time 99765482821, :cas 1647216446215880704}}, :wgl [:WGL]} INFO [2022-03-13 18:52:43,234] knossos reporter - knossos.search {:linear {:space 1, :cost 8.10E+10, :op {:process 485, :type :ok, :f :read, :value :nil, :index 2897, :time 98532162369}}, :wgl [:WGL]} INFO [2022-03-13 18:52:43,267] knossos reporter - knossos.search {:linear {:space 8, :cost 1.57E+14, :op {:process 397, :type :ok, :f :read, :value 46, :index 3051, :time 103521451729, :cas 1647216415578849280}}, :wgl [:WGL]} INFO [2022-03-13 18:52:43,274] knossos reporter - knossos.search {:linear {:space 4, :cost 1.34E+15, :op {:process 418, :type :ok, :f :read, :value 35, :index 2913, :time 99295320479, :cas 1647216464745791488}}, :wgl [:WGL]} INFO [2022-03-13 18:52:43,370] clojure-agent-send-pool-0 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/23/results.edn INFO [2022-03-13 18:52:44,014] clojure-agent-send-pool-0 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/23/history.edn INFO [2022-03-13 18:52:44,227] clojure-agent-send-pool-7 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/24/results.edn INFO [2022-03-13 18:52:44,346] clojure-agent-send-pool-7 - jepsen.independent Wrote /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/store/Couchbase/20220313T170514.000-0700/independent/24/history.edn INFO [2022-03-13 18:52:48,235] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 304, :type :ok, :f :read, :value :nil, :index 3323, :time 112107721716}}, :wgl [:WGL]} INFO [2022-03-13 18:52:48,236] knossos reporter - knossos.search {:linear {:space 512, :cost 3.10E+18, :op {:process 627, :type :ok, :f :read, :value 17, :index 3197, :time 108971548611, :cas 1647216446215880704}}, :wgl [:WGL]} INFO [2022-03-13 18:52:48,237] knossos reporter - knossos.search {:linear {:space 1, :cost 2.47E+22, :op {:process 641, :type :ok, :f :read, :value :nil, :index 4327, :time 138686393928}}, :wgl [:WGL]} INFO [2022-03-13 18:52:49,566] knossos reporter - knossos.search {:linear {:space 512, :cost 2.49E+22, :op {:process 569, :type :ok, :f :read, :value 30, :index 3387, :time 114748049071, :cas 1647216443695431680}}, :wgl [:WGL]} INFO [2022-03-13 18:52:49,910] knossos reporter - knossos.search {:linear {:space 2, :cost 2.44E+12, :op {:process 375, :type :ok, :f :read, :value :nil, :index 2609, :time 89687776408}}, :wgl [:WGL]} INFO [2022-03-13 18:52:53,468] knossos reporter - knossos.search {:linear {:space 4, :cost 1.34E+15, :op {:process 488, :type :ok, :f :read, :value 46, :index 3297, :time 111036491252, :cas 1647216415578849280}}, :wgl [:WGL]} INFO [2022-03-13 18:52:53,475] knossos reporter - knossos.search {:linear {:space 2, :cost 6.70E+14, :op {:process 393, :type :ok, :f :read, :value :nil, :index 3451, :time 117991561876}}, :wgl [:WGL]} INFO [2022-03-13 18:52:53,589] knossos reporter - knossos.search {:linear {:space 16384, :cost 5.48E+18, :op {:process 509, :type :ok, :f :read, :value 35, :index 2927, :time 99740657986, :cas 1647216464745791488}}, :wgl [:WGL]} INFO [2022-03-13 18:52:54,911] knossos reporter - knossos.search {:linear {:space 1, :cost 1.96E+13, :op {:process 467, :type :ok, :f :read, :value 32, :index 2843, :time 96466526556, :cas 1647216455136313344}}, :wgl [:WGL]} INFO [2022-03-13 18:52:58,875] knossos reporter - knossos.search {:linear {:space 512, :cost 3.10E+18, :op {:process 627, :type :ok, :f :read, :value 17, :index 3199, :time 108997340402, :cas 1647216446215880704}}, :wgl [:WGL]} INFO [2022-03-13 18:52:58,899] knossos reporter - knossos.search {:linear {:space 512, :cost 5.49E+23, :op {:process 714, :type :ok, :f :read, :value 25, :index 3481, :time 118617422811, :cas 1647216446020714496}}, :wgl [:WGL]} INFO [2022-03-13 18:52:58,906] knossos reporter - knossos.search {:linear {:space 1, :cost 3.35E+14, :op {:process 560, :type :ok, :f :read, :value :nil, :index 3247, :time 110199230633}}, :wgl [:WGL]} INFO [2022-03-13 18:52:58,966] knossos reporter - knossos.search {:linear {:space 8192, :cost 1.89E+22, :op {:process 508, :type :ok, :f :read, :value 35, :index 3223, :time 109801429385, :cas 1647216464745791488}}, :wgl [:WGL]} INFO [2022-03-13 18:53:04,987] knossos reporter - knossos.search {:linear {:space 8192, :cost 3.99E+23, :op {:process 466, :type :ok, :f :read, :value 32, :index 3231, :time 109866686244, :cas 1647216455136313344}}, :wgl [:WGL]} INFO [2022-03-13 18:53:09,106] knossos reporter - knossos.search {:linear {:space 4, :cost 4.61E+17, :op {:process 486, :type :ok, :f :read, :value 46, :index 3371, :time 113670618448, :cas 1647216415578849280}}, :wgl [:WGL]} INFO [2022-03-13 18:53:14,663] knossos reporter - knossos.search {:linear {:space 512, :cost 5.90E+19, :op {:process 449, :type :ok, :f :read, :value 17, :index 3241, :time 109938103683, :cas 1647216446215880704}}, :wgl [:WGL]} INFO [2022-03-13 18:53:14,666] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 394, :type :ok, :f :read, :value :nil, :index 3581, :time 122207801923}}, :wgl [:WGL]} INFO [2022-03-13 18:53:19,687] knossos reporter - knossos.search {:linear {:space 1, :cost 1.15E+17, :op {:process 468, :type :ok, :f :read, :value :nil, :index 3329, :time 112488590723}}, :wgl [:WGL]} INFO [2022-03-13 18:53:35,411] knossos reporter - knossos.search {:linear {:space 32768, :cost 7.57E+22, :op {:process 508, :type :ok, :f :read, :value 35, :index 3229, :time 109832058904, :cas 1647216464745791488}}, :wgl [:WGL]} INFO [2022-03-13 18:53:50,458] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 394, :type :ok, :f :read, :value :nil, :index 3583, :time 122210483778}}, :wgl [:WGL]} INFO [2022-03-13 18:53:55,458] knossos reporter - knossos.search {:linear {:space 512, :cost 4.07E+36, :op {:process 898, :type :ok, :f :read, :value 33, :index 5157, :time 152994937549, :cas 1647216508747579392}}, :wgl [:WGL]} INFO [2022-03-13 18:54:14,875] knossos reporter - knossos.search {:linear {:space 32768, :cost 7.57E+22, :op {:process 508, :type :ok, :f :read, :value 35, :index 3233, :time 109870958649, :cas 1647216464745791488}}, :wgl [:WGL]} INFO [2022-03-13 18:54:31,549] knossos reporter - knossos.search {:linear {:space 1, :cost 6.05E+15, :op {:process 394, :type :ok, :f :read, :value :nil, :index 3585, :time 122212674210}}, :wgl [:WGL]} INFO [2022-03-13 18:55:23,771] knossos reporter - knossos.search {:linear {:space 8, :cost 9.22E+17, :op {:process 486, :type :ok, :f :read, :value 46, :index 3373, :time 113903194898, :cas 1647216415578849280}}, :wgl [:WGL]} INFO [2022-03-13 18:55:29,114] knossos reporter - knossos.search {:linear {:space 32768, :cost 1.59E+24, :op {:process 466, :type :ok, :f :read, :value 32, :index 3237, :time 109906086195, :cas 1647216455136313344}}, :wgl [:WGL]} INFO [2022-03-13 18:55:53,761] knossos reporter - knossos.search {:linear {:space 1, :cost 1.15E+17, :op {:process 665, :type :ok, :f :read, :value :nil, :index 3649, :time 125218398708}}, :wgl [:WGL]} INFO [2022-03-13 18:56:01,041] knossos reporter - knossos.search {:linear {:space 32768, :cost 7.57E+22, :op {:process 508, :type :ok, :f :read, :value 35, :index 3239, :time 109907477851, :cas 1647216464745791488}}, :wgl [:WGL]} INFO [2022-03-13 18:56:24,386] knossos reporter - knossos.search {:linear {:space 512, :cost 1.99E+29, :op {:process 715, :type :ok, :f :read, :value 25, :index 4067, :time 131600757185, :cas 1647216446020714496}}, :wgl [:WGL]} INFO [2022-03-13 18:57:00,270] knossos reporter - knossos.search {:linear {:space 1, :cost 1.15E+17, :op {:process 468, :type :ok, :f :read, :value :nil, :index 3337, :time 113135127277}}, :wgl [:WGL]} INFO [2022-03-13 19:03:04,257] knossos reporter - knossos.search {:linear {:space 512, :cost 5.49E+23, :op {:process 567, :type :ok, :f :read, :value 30, :index 3623, :time 123852361272, :cas 1647216443695431680}}, :wgl [:WGL]} # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fc76903a4a0, pid=114362, tid=0x00007fbde3497700 # # JRE version: OpenJDK Runtime Environment (8.0_312-b07) (build 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07) # Java VM: OpenJDK 64-Bit Server VM (25.312-b07 mixed mode linux-amd64 ) # Problematic frame: # V [libjvm.so+0xa4c4a0] # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /mnt/data-store/jenkins/workspace/kv-engine-cheshire-cat-jepsen-nightly/hs_err_pid114362.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # grep: store/latest/results.edn: No such file or directory Jepsen failed with unknown failure Test crashed! ###### Current Test Report ######### pass: 7 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 7/8 = 87.50% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #9 Test params: workload=partition,no-autofailover,replicas=1,disrupt-count=1,disrupt-time=10,kv-timeout=15,durability=0:0:100:0,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=partition --no-autofailover --replicas=1 --disrupt-count=1 --disrupt-time=10 --kv-timeout=15 --durability=0:0:100:0 --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-9.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 8 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 8/9 = 88.88% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #10 Test params: workload=kill,scenario=kill-ns-server,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=kill --scenario=kill-ns-server --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=value --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-10.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 9 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 9/10 = 90.00% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #11 Test params: workload=kill,scopes=1,collections=3,scenario=kill-babysitter,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=kill --scopes=1 --collections=3 --scenario=kill-babysitter --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=value --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-11.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 10 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 10/11 = 90.90% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #12 Test params: workload=kill,scenario=kill-memcached,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value,use-json-docs,doc-padding-size=4096 Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=kill --scenario=kill-memcached --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=value --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-12.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 11 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 11/12 = 91.66% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #13 Test params: workload=kill,scenario=hard-reboot,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=full,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=kill --scenario=hard-reboot --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=full --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-13.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 12 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 12/13 = 92.30% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #14 Test params: workload=kill,scenario=suspend-process,process-to-suspend=memcached,process-suspend-time=35,replicas=1,no-autofailover,durability=0:0:100:0,eviction-policy=full,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=kill --scenario=suspend-process --process-to-suspend=memcached --process-suspend-time=35 --replicas=1 --no-autofailover --durability=0:0:100:0 --eviction-policy=full --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-14.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 13 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 13/14 = 92.85% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #15 Test params: workload=set-kill,scenario=kill-memcached,replicas=1,no-autofailover,durability=0:0:100:0,eviction-policy=full,use-json-docs,doc-padding-size=64 Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-memcached --replicas=1 --no-autofailover --durability=0:0:100:0 --eviction-policy=full --use-json-docs --doc-padding-size=64 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-15.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 14 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 14/15 = 93.33% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #16 Test params: workload=set-kill,scenario=kill-memcached,replicas=1,no-autofailover,durability=0:0:0:100,eviction-policy=value Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-memcached --replicas=1 --no-autofailover --durability=0:0:0:100 --eviction-policy=value --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-16.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 15 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 15/16 = 93.75% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #17 Test params: workload=set-kill,scenario=kill-ns-server,replicas=2,no-autofailover,durability=0:100:0:0,eviction-policy=value Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-ns-server --replicas=2 --no-autofailover --durability=0:100:0:0 --eviction-policy=value --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-17.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 16 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 16/17 = 94.11% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #18 Test params: workload=set-kill,scenario=kill-babysitter,replicas=2,no-autofailover,durability=0:100:0:0,eviction-policy=value Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-babysitter --replicas=2 --no-autofailover --durability=0:100:0:0 --eviction-policy=value --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-18.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 17 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 17/18 = 94.44% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #19 Test params: workload=set-kill,scenario=hard-reboot,replicas=1,no-autofailover,durability=0:0:100:0,eviction-policy=value Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=hard-reboot --replicas=1 --no-autofailover --durability=0:0:100:0 --eviction-policy=value --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-19.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 18 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 18/19 = 94.73% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #20 Test params: workload=set-kill,scenario=suspend-process,process-to-suspend=memcached,process-suspend-time=10,replicas=2,autofailover,durability=0:0:100:0,eviction-policy=value Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=suspend-process --process-to-suspend=memcached --process-suspend-time=10 --replicas=2 --autofailover --durability=0:0:100:0 --eviction-policy=value --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-20.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 19 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 19/20 = 95.00% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #21 Test params: workload=set-kill,scenario=kill-memcached-on-slow-disk,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=value,manipulate-disks Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-memcached-on-slow-disk --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=value --manipulate-disks --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-21.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 20 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 20/21 = 95.23% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #22 Test params: workload=set-kill,scenario=kill-memcached-on-slow-disk,replicas=2,no-autofailover,durability=0:0:0:100,eviction-policy=full,manipulate-disks Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-memcached-on-slow-disk --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=full --manipulate-disks --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-22.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 21 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 21/22 = 95.45% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #23 Test params: workload=register,replicas=2,no-autofailover,rate=0,durability=50:50:0:0,bucket-type=ephemeral,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=register --replicas=2 --no-autofailover --rate=0 --durability=50:50:0:0 --bucket-type=ephemeral --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-23.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 22 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 22/23 = 95.65% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #24 Test params: workload=failover,failover-type=hard,recovery-type=delta,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0,bucket-type=ephemeral,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=failover --failover-type=hard --recovery-type=delta --replicas=2 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:100:0:0 --bucket-type=ephemeral --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-24.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 23 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 23/24 = 95.83% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #25 Test params: workload=rebalance,scenario=sequential-rebalance-out-in,replicas=2,no-autofailover,disrupt-count=1,rate=0,durability=0:100:0:0,bucket-type=ephemeral,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=rebalance --scenario=sequential-rebalance-out-in --replicas=2 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:100:0:0 --bucket-type=ephemeral --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-25.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 24 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 24/25 = 96.00% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #26 Test params: workload=set-kill,scenario=kill-memcached,replicas=2,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-memcached --replicas=2 --no-autofailover --durability=0:100:0:0 --bucket-type=ephemeral --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-26.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 25 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 25/26 = 96.15% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #27 Test params: workload=set-kill,scenario=kill-memcached,replicas=1,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral,use-json-docs,use-subdoc Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-memcached --replicas=1 --no-autofailover --durability=0:100:0:0 --bucket-type=ephemeral --use-json-docs --use-subdoc --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-27.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 26 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 26/27 = 96.29% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #28 Test params: workload=set-kill,scenario=kill-ns-server,replicas=2,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral,use-json-docs,doc-padding-size=64 Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-ns-server --replicas=2 --no-autofailover --durability=0:100:0:0 --bucket-type=ephemeral --use-json-docs --doc-padding-size=64 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-28.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 27 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 27/28 = 96.42% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #29 Test params: workload=set-kill,scenario=kill-babysitter,replicas=2,no-autofailover,durability=0:100:0:0,bucket-type=ephemeral Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=set-kill --scenario=kill-babysitter --replicas=2 --no-autofailover --durability=0:100:0:0 --bucket-type=ephemeral --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-29.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 28 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 28/29 = 96.55% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #30 Test params: workload=disk-failure,manipulate-disks,replicas=1,durability=0:100:0:0,recovery-type=full,eviction-policy=value,use-json-docs,use-subdoc Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=disk-failure --manipulate-disks --replicas=1 --durability=0:100:0:0 --recovery-type=full --eviction-policy=value --use-json-docs --use-subdoc --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-30.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 29 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 29/30 = 96.66% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #31 Test params: workload=disk-failure,manipulate-disks,replicas=1,durability=0:0:100:0,recovery-type=delta,eviction-policy=full,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=disk-failure --manipulate-disks --replicas=1 --durability=0:0:100:0 --recovery-type=delta --eviction-policy=full --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-31.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 30 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 30/31 = 96.77% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #32 Test params: workload=disk-failure,manipulate-disks,replicas=1,durability=0:0:0:100,recovery-type=delta,eviction-policy=full,cas Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=disk-failure --manipulate-disks --replicas=1 --durability=0:0:0:100 --recovery-type=delta --eviction-policy=full --cas --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-32.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 31 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 31/32 = 96.87% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #33 Test params: workload=disk-failure,manipulate-disks,disrupt-count=2,replicas=2,durability=0:0:0:100,recovery-type=delta,eviction-policy=full Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=disk-failure --manipulate-disks --disrupt-count=2 --replicas=2 --durability=0:0:0:100 --recovery-type=delta --eviction-policy=full --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-33.log file to use error_msg_from_memcached.log Test passed! ###### Current Test Report ######### pass: 32 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 32/33 = 96.96% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ################################################################ Running test: #34 Test params: workload=kill-n-disk-failure,manipulate-disks,replicas=1,durability=0:0:0:100,recovery-type=delta,eviction-policy=full Test command: lein trampoline run test --nodes-file ./nodes --username root --ssh-private-key=./resources/my.key --workload=kill-n-disk-failure --manipulate-disks --replicas=1 --durability=0:0:0:100 --recovery-type=delta --eviction-policy=full --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 &> jepsen-output-34.log file to use error_msg_from_memcached_filtered.log Test passed! ###### Current Test Report ######### pass: 33 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 33/34 = 97.05% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ ###### Finished Running Suite ######### Jepsen tests complete! ###### Final Test Report ######### pass: 33 unknown: 0 fail: 0 crash: 1 memcached errors (not reported as total): 1 33/34 = 97.05% ###### Crashed Tests ######## #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ###### Memcached errors ###### #8 :: --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0 ################################################################ Error log messages where found in memached.log's ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:05.314399-07:00 CRITICAL *** Fatal error encountered during exception handling *** ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:05.385595-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:983 ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511398-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.4-7218). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/02ab548f-a217-4202-ae2772af-a3a69ae6.dmp before terminating. ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511441-07:00 CRITICAL Stack backtrace of crashed thread: ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511693-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x67f213] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511760-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x6a03ca] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511808-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x6a0708] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511833-07:00 CRITICAL #3 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f7e2ac2e000+0x12980] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511877-07:00 CRITICAL #4 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f7e285cc000+0x3ee87] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.511914-07:00 CRITICAL #5 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f7e285cc000+0x407f1] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512014-07:00 CRITICAL #6 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xa1efc] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512075-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x68cfef] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512152-07:00 CRITICAL #8 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xacf96] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512245-07:00 CRITICAL #9 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xad001] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512336-07:00 CRITICAL #10 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xad294] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512381-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x8f847] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512499-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x4b0c35] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512562-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x4b216c] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512610-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x4a4772] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512645-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x4a48f7] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512684-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x4ab945] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512728-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x2aaf37] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512819-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x2a6080] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512889-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x71ad70] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512936-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x702b2a] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.512980-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x71dd29] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513025-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x2a475d] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513120-07:00 CRITICAL #23 /opt/couchbase/lib/libstdc++.so.6() [0x7f7e28f73000+0xd62f0] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513133-07:00 CRITICAL #24 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f7e2ac2e000+0x76db] ./20220313T170514.000-0700/172.23.107.228/cbcollect_info_ns_1@172.23.107.228_20220314-002118/memcached.log:2022-03-13T17:21:09.513189-07:00 CRITICAL #25 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f7e285cc000+0x12161f] EXIT CODE IS 6 JEPSEN CRASHED AND ERRORS FOUND IN MEMCACHED LOGS Build step 'Execute shell' marked build as failure Archiving artifacts Sending e-mails to: kv_engine_dev@couchbase.com asad.zaidi@couchbase.com Finished: FAILURE